[PATCH] Add a timer mode that disables pending missed ticks

[PATCH] Add a timer mode that disables pending missed ticks

[Shan, Haitao]

[-- Attachment #1.1: Type: text/plain, Size: 245 bytes --]

Hi, Keir,

This patch adds a new timer mode, in which no missed ticks is
calculated. This can be used with latest x86_64 linux guest, since it
can pick up missed ticks themselves.
 <<no_missed_ticks.patch>> 

Best Regards
Haitao Shan


[-- Attachment #1.2: Type: text/html, Size: 924 bytes --]

[-- Attachment #2: no_missed_ticks.patch --]
[-- Type: application/octet-stream, Size: 2348 bytes --]

diff -r 4b4c75cb6c0f xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Tue Oct 30 11:05:36 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Tue Oct 30 05:45:57 2007 +0800
@@ -25,8 +25,16 @@
 
 static int pt_support_time_frozen(struct domain *d)
 {
-    return (d->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == 
-            HVMPTM_delay_for_missed_ticks);
+    return ( d->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] ==
+            HVMPTM_delay_for_missed_ticks
+          || d->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] ==
+            HVMPTM_delay_for_missed_ticks );
+}
+
+static int pt_support_no_missed_ticks(struct domain *d)
+{
+    return (d->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] ==
+            HVMPTM_no_missed_ticks_pending);
 }
 
 static void pt_lock(struct periodic_time *pt)
@@ -115,7 +123,8 @@ void pt_restore_timer(struct vcpu *v)
 
     list_for_each_entry ( pt, head, list )
     {
-        missed_ticks(pt);
+        if ( !pt_support_no_missed_ticks(v->domain) )
+            missed_ticks(pt);
         set_timer(&pt->timer, pt->scheduled);
     }
 
@@ -136,7 +145,13 @@ static void pt_timer_fn(void *data)
     if ( !pt->one_shot )
     {
         pt->scheduled += pt->period;
-        missed_ticks(pt);
+        if ( !pt_support_no_missed_ticks(pt->vcpu->domain) )
+            missed_ticks(pt);
+        else if ( NOW() - pt->scheduled >= 0  )
+        {
+            pt->pending_intr_nr++;
+            pt->scheduled = NOW() + pt->period;
+        }
         set_timer(&pt->timer, pt->scheduled);
     }
 
@@ -233,7 +248,10 @@ void pt_intr_post(struct vcpu *v, struct
     else
     {
         pt->pending_intr_nr--;
-        pt->last_plt_gtime += pt->period_cycles;
+        if ( pt_support_no_missed_ticks(v->domain) )
+            pt->last_plt_gtime = hvm_get_guest_time(v);
+        else
+            pt->last_plt_gtime += pt->period_cycles;
     }
 
     if ( pt_support_time_frozen(v->domain) &&
diff -r 4b4c75cb6c0f xen/include/public/hvm/params.h
--- a/xen/include/public/hvm/params.h	Tue Oct 30 11:05:36 2007 +0000
+++ b/xen/include/public/hvm/params.h	Tue Oct 30 05:13:32 2007 +0800
@@ -70,6 +70,7 @@
 #define HVM_PARAM_TIMER_MODE   10
 #define HVMPTM_delay_for_missed_ticks    0
 #define HVMPTM_no_delay_for_missed_ticks 1
+#define HVMPTM_no_missed_ticks_pending   2
 
 #define HVM_NR_PARAMS          11
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 574 bytes --]


Applied as c/s 16274. Please take a look and make sure the mode works as you
expect.

 -- Keir

On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:

> Hi, Keir, 
> 
> This patch adds a new timer mode, in which no missed ticks is calculated. This
> can be used with latest x86_64 linux guest, since it can pick up missed ticks
> themselves.
> 
>  <<no_missed_ticks.patch>>
> 
> Best Regards 
> Haitao Shan 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



[-- Attachment #1.2: Type: text/html, Size: 1583 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

Keir,

Here are my comments on your change.
I've attached an updated vpt.c with the changes.

1. For the no_missed_tick_accounting method, we still need the update
    to pt->scheduled taking into account the time that has elapsed when
    missed ticks are calculated. missed_ticks (pt_process_missed_ticks)
    is called from pt_restore_timer() and pt_timer_fn() and, thus, its
    easiest to put the check for no_missed_tick_accounting method in
    missed_ticks() itself.

2. In pt_timer_fn you don't want to increment pending_intr_nr beyond 1
    for no_missed_tick_accounting option.

thanks,
Dave


Keir Fraser wrote:

>
> Applied as c/s 16274. Please take a look and make sure the mode works 
> as you expect.
>
>  -- Keir
>
> On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:
>
>     Hi, Keir,
>
>     This patch adds a new timer mode, in which no missed ticks is
>     calculated. This can be used with latest x86_64 linux guest, since
>     it can pick up missed ticks themselves.
>
>      <<no_missed_ticks.patch>>
>
>     Best Regards
>     Haitao Shan
>
>     ------------------------------------------------------------------------
>     _______________________________________________
>     Xen-devel mailing list
>     Xen-devel@lists.xensource.com
>     http://lists.xensource.com/xen-devel
>
>


[-- Attachment #2: diff.vpt --]
[-- Type: text/plain, Size: 2208 bytes --]

*** vpt.c.new.c	2007-10-30 16:45:26.000000000 -0400
--- vpt.c	2007-10-30 15:30:57.000000000 -0400
***************
*** 57,73 ****
          return;
  
      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
! 
!     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
! 	if ( missed_ticks > 1000 )
! 	    {
! 		/* TODO: Adjust guest time together */
! 		pt->pending_intr_nr++;
! 	    }
! 	else
! 	    {
! 		pt->pending_intr_nr += missed_ticks;
! 	    }
      }
  
      pt->scheduled += missed_ticks * pt->period;
--- 57,70 ----
          return;
  
      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
!     if ( missed_ticks > 1000 )
!     {
!         /* TODO: Adjust guest time together */
!         pt->pending_intr_nr++;
!     }
!     else
!     {
!         pt->pending_intr_nr += missed_ticks;
      }
  
      pt->scheduled += missed_ticks * pt->period;
***************
*** 120,126 ****
  
      list_for_each_entry ( pt, head, list )
      {
! 	pt_process_missed_ticks(pt);
          set_timer(&pt->timer, pt->scheduled);
      }
  
--- 117,124 ----
  
      list_for_each_entry ( pt, head, list )
      {
!         if ( !mode_is(v->domain, no_missed_tick_accounting) )
!             pt_process_missed_ticks(pt);
          set_timer(&pt->timer, pt->scheduled);
      }
  
***************
*** 135,151 ****
  
      pt_lock(pt);
  
!     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
! 	if(!pt->pending_intr_nr)
! 	    pt->pending_intr_nr++;
!     }
!     else
! 	pt->pending_intr_nr++;
  
      if ( !pt->one_shot )
      {
          pt->scheduled += pt->period;
! 	pt_process_missed_ticks(pt);
          set_timer(&pt->timer, pt->scheduled);
      }
  
--- 133,152 ----
  
      pt_lock(pt);
  
!     pt->pending_intr_nr++;
  
      if ( !pt->one_shot )
      {
          pt->scheduled += pt->period;
!         if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
!         {
!             pt_process_missed_ticks(pt);
!         }
!         else if ( (NOW() - pt->scheduled) >= 0 )
!         {
!             pt->pending_intr_nr++;
!             pt->scheduled = NOW() + pt->period;
!         }
          set_timer(&pt->timer, pt->scheduled);
      }
  

[-- Attachment #3: vpt.c --]
[-- Type: text/x-csrc, Size: 8507 bytes --]

/*
 * vpt.c: Virtual Platform Timer
 *
 * Copyright (c) 2006, Xiaowei Yang, Intel Corporation.
 *
 * This program is free software; you can redistribute it and/or modify it
 * under the terms and conditions of the GNU General Public License,
 * version 2, as published by the Free Software Foundation.
 *
 * This program is distributed in the hope it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
 * more details.
 *
 * You should have received a copy of the GNU General Public License along with
 * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
 * Place - Suite 330, Boston, MA 02111-1307 USA.
 *
 */

#include <xen/time.h>
#include <asm/hvm/support.h>
#include <asm/hvm/vpt.h>
#include <asm/event.h>

#define mode_is(d, name) \
    ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name)

static void pt_lock(struct periodic_time *pt)
{
    struct vcpu *v;

    for ( ; ; )
    {
        v = pt->vcpu;
        spin_lock(&v->arch.hvm_vcpu.tm_lock);
        if ( likely(pt->vcpu == v) )
            break;
        spin_unlock(&v->arch.hvm_vcpu.tm_lock);
    }
}

static void pt_unlock(struct periodic_time *pt)
{
    spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
}

static void pt_process_missed_ticks(struct periodic_time *pt)
{
    s_time_t missed_ticks;

    if ( pt->one_shot )
        return;

    missed_ticks = NOW() - pt->scheduled;
    if ( missed_ticks <= 0 )
        return;

    missed_ticks = missed_ticks / (s_time_t) pt->period + 1;

    if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
	if ( missed_ticks > 1000 )
	    {
		/* TODO: Adjust guest time together */
		pt->pending_intr_nr++;
	    }
	else
	    {
		pt->pending_intr_nr += missed_ticks;
	    }
    }

    pt->scheduled += missed_ticks * pt->period;
}

static void pt_freeze_time(struct vcpu *v)
{
    if ( !mode_is(v->domain, delay_for_missed_ticks) )
        return;

    v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v);
}

static void pt_thaw_time(struct vcpu *v)
{
    if ( !mode_is(v->domain, delay_for_missed_ticks) )
        return;

    if ( v->arch.hvm_vcpu.guest_time == 0 )
        return;

    hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time);
    v->arch.hvm_vcpu.guest_time = 0;
}

void pt_save_timer(struct vcpu *v)
{
    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
    struct periodic_time *pt;

    if ( test_bit(_VPF_blocked, &v->pause_flags) )
        return;

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    list_for_each_entry ( pt, head, list )
        stop_timer(&pt->timer);

    pt_freeze_time(v);

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
}

void pt_restore_timer(struct vcpu *v)
{
    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
    struct periodic_time *pt;

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    list_for_each_entry ( pt, head, list )
    {
	pt_process_missed_ticks(pt);
        set_timer(&pt->timer, pt->scheduled);
    }

    pt_thaw_time(v);

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
}

static void pt_timer_fn(void *data)
{
    struct periodic_time *pt = data;

    pt_lock(pt);

    if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
	if(!pt->pending_intr_nr)
	    pt->pending_intr_nr++;
    }
    else
	pt->pending_intr_nr++;

    if ( !pt->one_shot )
    {
        pt->scheduled += pt->period;
	pt_process_missed_ticks(pt);
        set_timer(&pt->timer, pt->scheduled);
    }

    vcpu_kick(pt->vcpu);

    pt_unlock(pt);
}

void pt_update_irq(struct vcpu *v)
{
    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
    struct periodic_time *pt;
    uint64_t max_lag = -1ULL;
    int irq = -1;

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    list_for_each_entry ( pt, head, list )
    {
        if ( !is_isa_irq_masked(v, pt->irq) && pt->pending_intr_nr &&
             ((pt->last_plt_gtime + pt->period_cycles) < max_lag) )
        {
            max_lag = pt->last_plt_gtime + pt->period_cycles;
            irq = pt->irq;
        }
    }

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);

    if ( is_lvtt(v, irq) )
    {
        vlapic_set_irq(vcpu_vlapic(v), irq, 0);
    }
    else if ( irq >= 0 )
    {
        hvm_isa_irq_deassert(v->domain, irq);
        hvm_isa_irq_assert(v->domain, irq);
    }
}

static struct periodic_time *is_pt_irq(
    struct vcpu *v, struct hvm_intack intack)
{
    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
    struct periodic_time *pt;
    struct RTCState *rtc = &v->domain->arch.hvm_domain.pl_time.vrtc;
    int vector;

    list_for_each_entry ( pt, head, list )
    {
        if ( !pt->pending_intr_nr )
            continue;

        if ( is_lvtt(v, pt->irq) )
        {
            if ( pt->irq != intack.vector )
                continue;
            return pt;
        }

        vector = get_isa_irq_vector(v, pt->irq, intack.source);

        /* RTC irq need special care */
        if ( (intack.vector != vector) ||
             ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) )
            continue;

        return pt;
    }

    return NULL;
}

void pt_intr_post(struct vcpu *v, struct hvm_intack intack)
{
    struct periodic_time *pt;
    time_cb *cb;
    void *cb_priv;

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    pt = is_pt_irq(v, intack);
    if ( pt == NULL )
    {
        spin_unlock(&v->arch.hvm_vcpu.tm_lock);
        return;
    }

    if ( pt->one_shot )
    {
        pt->enabled = 0;
        list_del(&pt->list);
    }
    else
    {
        pt->pending_intr_nr--;
        if ( mode_is(v->domain, no_missed_tick_accounting) )
            pt->last_plt_gtime = hvm_get_guest_time(v);
        else
            pt->last_plt_gtime += pt->period_cycles;
    }

    if ( mode_is(v->domain, delay_for_missed_ticks) &&
         (hvm_get_guest_time(v) < pt->last_plt_gtime) )
        hvm_set_guest_time(v, pt->last_plt_gtime);

    cb = pt->cb;
    cb_priv = pt->priv;

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);

    if ( cb != NULL )
        cb(v, cb_priv);
}

void pt_reset(struct vcpu *v)
{
    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
    struct periodic_time *pt;

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    list_for_each_entry ( pt, head, list )
    {
        pt->pending_intr_nr = 0;
        pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
        pt->scheduled = NOW() + pt->period;
        set_timer(&pt->timer, pt->scheduled);
    }

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
}

void pt_migrate(struct vcpu *v)
{
    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
    struct periodic_time *pt;

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    list_for_each_entry ( pt, head, list )
        migrate_timer(&pt->timer, v->processor);

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
}

void create_periodic_time(
    struct vcpu *v, struct periodic_time *pt, uint64_t period,
    uint8_t irq, char one_shot, time_cb *cb, void *data)
{
    destroy_periodic_time(pt);

    spin_lock(&v->arch.hvm_vcpu.tm_lock);

    pt->enabled = 1;
    pt->pending_intr_nr = 0;

    /* Periodic timer must be at least 0.9ms. */
    if ( (period < 900000) && !one_shot )
    {
        gdprintk(XENLOG_WARNING,
                 "HVM_PlatformTime: program too small period %"PRIu64"\n",
                 period);
        period = 900000;
    }

    pt->period = period;
    pt->vcpu = v;
    pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
    pt->irq = irq;
    pt->period_cycles = (u64)period * cpu_khz / 1000000L;
    pt->one_shot = one_shot;
    pt->scheduled = NOW() + period;
    /*
     * Offset LAPIC ticks from other timer ticks. Otherwise guests which use
     * LAPIC ticks for process accounting can see long sequences of process
     * ticks incorrectly accounted to interrupt processing.
     */
    if ( is_lvtt(v, irq) )
        pt->scheduled += period >> 1;
    pt->cb = cb;
    pt->priv = data;

    list_add(&pt->list, &v->arch.hvm_vcpu.tm_list);

    init_timer(&pt->timer, pt_timer_fn, pt, v->processor);
    set_timer(&pt->timer, pt->scheduled);

    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
}

void destroy_periodic_time(struct periodic_time *pt)
{
    if ( !pt->enabled )
        return;

    pt_lock(pt);
    pt->enabled = 0;
    list_del(&pt->list);
    pt_unlock(pt);

    /*
     * pt_timer_fn() can run until this kill_timer() returns. We must do this
     * outside pt_lock() otherwise we can deadlock with pt_timer_fn().
     */
    kill_timer(&pt->timer);
}

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

I'm going to apply Haitao's no_missed_ticks_fix.patch. If you think fixes
are needed apart from that, please provide a unified diff.

 -- Keir

On 30/10/07 21:16, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Keir,
> 
> Here are my comments on your change.
> I've attached an updated vpt.c with the changes.
> 
> 1. For the no_missed_tick_accounting method, we still need the update
>     to pt->scheduled taking into account the time that has elapsed when
>     missed ticks are calculated. missed_ticks (pt_process_missed_ticks)
>     is called from pt_restore_timer() and pt_timer_fn() and, thus, its
>     easiest to put the check for no_missed_tick_accounting method in
>     missed_ticks() itself.
> 
> 2. In pt_timer_fn you don't want to increment pending_intr_nr beyond 1
>     for no_missed_tick_accounting option.
> 
> thanks,
> Dave
> 
> 
> Keir Fraser wrote:
> 
>> 
>> Applied as c/s 16274. Please take a look and make sure the mode works
>> as you expect.
>> 
>>  -- Keir
>> 
>> On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:
>> 
>>     Hi, Keir,
>> 
>>     This patch adds a new timer mode, in which no missed ticks is
>>     calculated. This can be used with latest x86_64 linux guest, since
>>     it can pick up missed ticks themselves.
>> 
>>      <<no_missed_ticks.patch>>
>> 
>>     Best Regards
>>     Haitao Shan
>> 
>>     ------------------------------------------------------------------------
>>     _______________________________________________
>>     Xen-devel mailing list
>>     Xen-devel@lists.xensource.com
>>     http://lists.xensource.com/xen-devel
>> 
>> 
> 
> *** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400
> --- vpt.c 2007-10-30 15:30:57.000000000 -0400
> ***************
> *** 57,73 ****
>           return;
>   
>       missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> ! 
> !     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
> !  if ( missed_ticks > 1000 )
> !      {
> !   /* TODO: Adjust guest time together */
> !   pt->pending_intr_nr++;
> !      }
> !  else
> !      {
> !   pt->pending_intr_nr += missed_ticks;
> !      }
>       }
>   
>       pt->scheduled += missed_ticks * pt->period;
> --- 57,70 ----
>           return;
>   
>       missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> !     if ( missed_ticks > 1000 )
> !     {
> !         /* TODO: Adjust guest time together */
> !         pt->pending_intr_nr++;
> !     }
> !     else
> !     {
> !         pt->pending_intr_nr += missed_ticks;
>       }
>   
>       pt->scheduled += missed_ticks * pt->period;
> ***************
> *** 120,126 ****
>   
>       list_for_each_entry ( pt, head, list )
>       {
> !  pt_process_missed_ticks(pt);
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> --- 117,124 ----
>   
>       list_for_each_entry ( pt, head, list )
>       {
> !         if ( !mode_is(v->domain, no_missed_tick_accounting) )
> !             pt_process_missed_ticks(pt);
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> ***************
> *** 135,151 ****
>   
>       pt_lock(pt);
>   
> !     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
> !  if(!pt->pending_intr_nr)
> !      pt->pending_intr_nr++;
> !     }
> !     else
> !  pt->pending_intr_nr++;
>   
>       if ( !pt->one_shot )
>       {
>           pt->scheduled += pt->period;
> !  pt_process_missed_ticks(pt);
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> --- 133,152 ----
>   
>       pt_lock(pt);
>   
> !     pt->pending_intr_nr++;
>   
>       if ( !pt->one_shot )
>       {
>           pt->scheduled += pt->period;
> !         if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
> !         {
> !             pt_process_missed_ticks(pt);
> !         }
> !         else if ( (NOW() - pt->scheduled) >= 0 )
> !         {
> !             pt->pending_intr_nr++;
> !             pt->scheduled = NOW() + pt->period;
> !         }
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> /*
>  * vpt.c: Virtual Platform Timer
>  *
>  * Copyright (c) 2006, Xiaowei Yang, Intel Corporation.
>  *
>  * This program is free software; you can redistribute it and/or modify it
>  * under the terms and conditions of the GNU General Public License,
>  * version 2, as published by the Free Software Foundation.
>  *
>  * This program is distributed in the hope it will be useful, but WITHOUT
>  * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>  * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>  * more details.
>  *
>  * You should have received a copy of the GNU General Public License along
> with
>  * this program; if not, write to the Free Software Foundation, Inc., 59
> Temple
>  * Place - Suite 330, Boston, MA 02111-1307 USA.
>  *
>  */
> 
> #include <xen/time.h>
> #include <asm/hvm/support.h>
> #include <asm/hvm/vpt.h>
> #include <asm/event.h>
> 
> #define mode_is(d, name) \
>     ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name)
> 
> static void pt_lock(struct periodic_time *pt)
> {
>     struct vcpu *v;
> 
>     for ( ; ; )
>     {
>         v = pt->vcpu;
>         spin_lock(&v->arch.hvm_vcpu.tm_lock);
>         if ( likely(pt->vcpu == v) )
>             break;
>         spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>     }
> }
> 
> static void pt_unlock(struct periodic_time *pt)
> {
>     spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
> }
> 
> static void pt_process_missed_ticks(struct periodic_time *pt)
> {
>     s_time_t missed_ticks;
> 
>     if ( pt->one_shot )
>         return;
> 
>     missed_ticks = NOW() - pt->scheduled;
>     if ( missed_ticks <= 0 )
>         return;
> 
>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> 
>     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
> if ( missed_ticks > 1000 )
>    {
> /* TODO: Adjust guest time together */
> pt->pending_intr_nr++;
>    }
> else
>    {
> pt->pending_intr_nr += missed_ticks;
>    }
>     }
> 
>     pt->scheduled += missed_ticks * pt->period;
> }
> 
> static void pt_freeze_time(struct vcpu *v)
> {
>     if ( !mode_is(v->domain, delay_for_missed_ticks) )
>         return;
> 
>     v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v);
> }
> 
> static void pt_thaw_time(struct vcpu *v)
> {
>     if ( !mode_is(v->domain, delay_for_missed_ticks) )
>         return;
> 
>     if ( v->arch.hvm_vcpu.guest_time == 0 )
>         return;
> 
>     hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time);
>     v->arch.hvm_vcpu.guest_time = 0;
> }
> 
> void pt_save_timer(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     if ( test_bit(_VPF_blocked, &v->pause_flags) )
>         return;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>         stop_timer(&pt->timer);
> 
>     pt_freeze_time(v);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void pt_restore_timer(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>     {
> pt_process_missed_ticks(pt);
>         set_timer(&pt->timer, pt->scheduled);
>     }
> 
>     pt_thaw_time(v);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> static void pt_timer_fn(void *data)
> {
>     struct periodic_time *pt = data;
> 
>     pt_lock(pt);
> 
>     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
> if(!pt->pending_intr_nr)
>    pt->pending_intr_nr++;
>     }
>     else
> pt->pending_intr_nr++;
> 
>     if ( !pt->one_shot )
>     {
>         pt->scheduled += pt->period;
> pt_process_missed_ticks(pt);
>         set_timer(&pt->timer, pt->scheduled);
>     }
> 
>     vcpu_kick(pt->vcpu);
> 
>     pt_unlock(pt);
> }
> 
> void pt_update_irq(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
>     uint64_t max_lag = -1ULL;
>     int irq = -1;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>     {
>         if ( !is_isa_irq_masked(v, pt->irq) && pt->pending_intr_nr &&
>              ((pt->last_plt_gtime + pt->period_cycles) < max_lag) )
>         {
>             max_lag = pt->last_plt_gtime + pt->period_cycles;
>             irq = pt->irq;
>         }
>     }
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> 
>     if ( is_lvtt(v, irq) )
>     {
>         vlapic_set_irq(vcpu_vlapic(v), irq, 0);
>     }
>     else if ( irq >= 0 )
>     {
>         hvm_isa_irq_deassert(v->domain, irq);
>         hvm_isa_irq_assert(v->domain, irq);
>     }
> }
> 
> static struct periodic_time *is_pt_irq(
>     struct vcpu *v, struct hvm_intack intack)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
>     struct RTCState *rtc = &v->domain->arch.hvm_domain.pl_time.vrtc;
>     int vector;
> 
>     list_for_each_entry ( pt, head, list )
>     {
>         if ( !pt->pending_intr_nr )
>             continue;
> 
>         if ( is_lvtt(v, pt->irq) )
>         {
>             if ( pt->irq != intack.vector )
>                 continue;
>             return pt;
>         }
> 
>         vector = get_isa_irq_vector(v, pt->irq, intack.source);
> 
>         /* RTC irq need special care */
>         if ( (intack.vector != vector) ||
>              ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) )
>             continue;
> 
>         return pt;
>     }
> 
>     return NULL;
> }
> 
> void pt_intr_post(struct vcpu *v, struct hvm_intack intack)
> {
>     struct periodic_time *pt;
>     time_cb *cb;
>     void *cb_priv;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     pt = is_pt_irq(v, intack);
>     if ( pt == NULL )
>     {
>         spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>         return;
>     }
> 
>     if ( pt->one_shot )
>     {
>         pt->enabled = 0;
>         list_del(&pt->list);
>     }
>     else
>     {
>         pt->pending_intr_nr--;
>         if ( mode_is(v->domain, no_missed_tick_accounting) )
>             pt->last_plt_gtime = hvm_get_guest_time(v);
>         else
>             pt->last_plt_gtime += pt->period_cycles;
>     }
> 
>     if ( mode_is(v->domain, delay_for_missed_ticks) &&
>          (hvm_get_guest_time(v) < pt->last_plt_gtime) )
>         hvm_set_guest_time(v, pt->last_plt_gtime);
> 
>     cb = pt->cb;
>     cb_priv = pt->priv;
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> 
>     if ( cb != NULL )
>         cb(v, cb_priv);
> }
> 
> void pt_reset(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>     {
>         pt->pending_intr_nr = 0;
>         pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>         pt->scheduled = NOW() + pt->period;
>         set_timer(&pt->timer, pt->scheduled);
>     }
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void pt_migrate(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>         migrate_timer(&pt->timer, v->processor);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void create_periodic_time(
>     struct vcpu *v, struct periodic_time *pt, uint64_t period,
>     uint8_t irq, char one_shot, time_cb *cb, void *data)
> {
>     destroy_periodic_time(pt);
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     pt->enabled = 1;
>     pt->pending_intr_nr = 0;
> 
>     /* Periodic timer must be at least 0.9ms. */
>     if ( (period < 900000) && !one_shot )
>     {
>         gdprintk(XENLOG_WARNING,
>                  "HVM_PlatformTime: program too small period %"PRIu64"\n",
>                  period);
>         period = 900000;
>     }
> 
>     pt->period = period;
>     pt->vcpu = v;
>     pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>     pt->irq = irq;
>     pt->period_cycles = (u64)period * cpu_khz / 1000000L;
>     pt->one_shot = one_shot;
>     pt->scheduled = NOW() + period;
>     /*
>      * Offset LAPIC ticks from other timer ticks. Otherwise guests which use
>      * LAPIC ticks for process accounting can see long sequences of process
>      * ticks incorrectly accounted to interrupt processing.
>      */
>     if ( is_lvtt(v, irq) )
>         pt->scheduled += period >> 1;
>     pt->cb = cb;
>     pt->priv = data;
> 
>     list_add(&pt->list, &v->arch.hvm_vcpu.tm_list);
> 
>     init_timer(&pt->timer, pt_timer_fn, pt, v->processor);
>     set_timer(&pt->timer, pt->scheduled);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void destroy_periodic_time(struct periodic_time *pt)
> {
>     if ( !pt->enabled )
>         return;
> 
>     pt_lock(pt);
>     pt->enabled = 0;
>     list_del(&pt->list);
>     pt_unlock(pt);
> 
>     /*
>      * pt_timer_fn() can run until this kill_timer() returns. We must do this
>      * outside pt_lock() otherwise we can deadlock with pt_timer_fn().
>      */
>     kill_timer(&pt->timer);
> }

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 15323 bytes --]

Hi Keir, Haitao, Eddie:

Here is the patch for timekeeping.
There are three components.

1. Increase missed ticks threshold to 100 seconds. Per the comment in 
the patch,
    I have seen scheduling delays of 5 seconds in a test lasting 30 minutes.
    The result is an immediate error of 5 seconds in guest time that
    persists.

2.  In pt_timer_fn for no_missed_tick_accounting option, only
    increment pt->pending_intr_nr if its found to be zero.
    This deals with the case where the guest has interrupts disabled
    for longer than a tick period.

3.  Call pt_process_missed_ticks() unconditionally and place the
    test for no_missed_tick_accounting inside pt_process_missed_ticks().
    This returns the calculation for the next timer expiration
    to the original method. The original method is important
    as it sets up a theoretical time space at t=0 where expirations
    occur only at n*period, where n is any integer. This, in turn, removes
    rounding errors.

Accuracy tests:

A.  32bit Linux guest (Item '1' above relevant)
    Test: 2 64bit Linux guests and 1 32bit Linux guest stacked.
          8 vcpus for each guest.
            2 physical processors on the node.
          All vcpus heavily loaded with work.
          Test duration: 3 hours.

        Result: clock error < .02%.
        (Before this patch guest time was 16 seconds slow in a 3.5 
minute test.)

B.  64bit Linux guest (Items '2' and '3' above relevant)
    Test: 2 64bit Linux guests.
          8 vcpus for each guest.
          2 physical processors on the node.
          All vcpus heavily loaded with work.
          Test duration: 3 hours.

        Result: clock error  .02%.
        (Before this patch clock error was 1.3%. The contributions of
    items '2' and '3' above to the improvement in accuracy are .4%
    and .9% respectively.

Of course, these tests are without ntpd running in the guest.
Ntpd requires a local clock error of < .05%.
With the .02% accuracy, one can used ntpd to synchronize the clock.


thanks,
Dave


Keir Fraser wrote:

>I'm going to apply Haitao's no_missed_ticks_fix.patch. If you think fixes
>are needed apart from that, please provide a unified diff.
>
> -- Keir
>
>On 30/10/07 21:16, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Keir,
>>
>>Here are my comments on your change.
>>I've attached an updated vpt.c with the changes.
>>
>>1. For the no_missed_tick_accounting method, we still need the update
>>    to pt->scheduled taking into account the time that has elapsed when
>>    missed ticks are calculated. missed_ticks (pt_process_missed_ticks)
>>    is called from pt_restore_timer() and pt_timer_fn() and, thus, its
>>    easiest to put the check for no_missed_tick_accounting method in
>>    missed_ticks() itself.
>>
>>2. In pt_timer_fn you don't want to increment pending_intr_nr beyond 1
>>    for no_missed_tick_accounting option.
>>
>>thanks,
>>Dave
>>
>>
>>Keir Fraser wrote:
>>
>>    
>>
>>>Applied as c/s 16274. Please take a look and make sure the mode works
>>>as you expect.
>>>
>>> -- Keir
>>>
>>>On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:
>>>
>>>    Hi, Keir,
>>>
>>>    This patch adds a new timer mode, in which no missed ticks is
>>>    calculated. This can be used with latest x86_64 linux guest, since
>>>    it can pick up missed ticks themselves.
>>>
>>>     <<no_missed_ticks.patch>>
>>>
>>>    Best Regards
>>>    Haitao Shan
>>>
>>>    ------------------------------------------------------------------------
>>>    _______________________________________________
>>>    Xen-devel mailing list
>>>    Xen-devel@lists.xensource.com
>>>    http://lists.xensource.com/xen-devel
>>>
>>>
>>>      
>>>
>>*** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400
>>--- vpt.c 2007-10-30 15:30:57.000000000 -0400
>>***************
>>*** 57,73 ****
>>          return;
>>  
>>      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>! 
>>!     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
>>!  if ( missed_ticks > 1000 )
>>!      {
>>!   /* TODO: Adjust guest time together */
>>!   pt->pending_intr_nr++;
>>!      }
>>!  else
>>!      {
>>!   pt->pending_intr_nr += missed_ticks;
>>!      }
>>      }
>>  
>>      pt->scheduled += missed_ticks * pt->period;
>>--- 57,70 ----
>>          return;
>>  
>>      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>!     if ( missed_ticks > 1000 )
>>!     {
>>!         /* TODO: Adjust guest time together */
>>!         pt->pending_intr_nr++;
>>!     }
>>!     else
>>!     {
>>!         pt->pending_intr_nr += missed_ticks;
>>      }
>>  
>>      pt->scheduled += missed_ticks * pt->period;
>>***************
>>*** 120,126 ****
>>  
>>      list_for_each_entry ( pt, head, list )
>>      {
>>!  pt_process_missed_ticks(pt);
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>--- 117,124 ----
>>  
>>      list_for_each_entry ( pt, head, list )
>>      {
>>!         if ( !mode_is(v->domain, no_missed_tick_accounting) )
>>!             pt_process_missed_ticks(pt);
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>***************
>>*** 135,151 ****
>>  
>>      pt_lock(pt);
>>  
>>!     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
>>!  if(!pt->pending_intr_nr)
>>!      pt->pending_intr_nr++;
>>!     }
>>!     else
>>!  pt->pending_intr_nr++;
>>  
>>      if ( !pt->one_shot )
>>      {
>>          pt->scheduled += pt->period;
>>!  pt_process_missed_ticks(pt);
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>--- 133,152 ----
>>  
>>      pt_lock(pt);
>>  
>>!     pt->pending_intr_nr++;
>>  
>>      if ( !pt->one_shot )
>>      {
>>          pt->scheduled += pt->period;
>>!         if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
>>!         {
>>!             pt_process_missed_ticks(pt);
>>!         }
>>!         else if ( (NOW() - pt->scheduled) >= 0 )
>>!         {
>>!             pt->pending_intr_nr++;
>>!             pt->scheduled = NOW() + pt->period;
>>!         }
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>/*
>> * vpt.c: Virtual Platform Timer
>> *
>> * Copyright (c) 2006, Xiaowei Yang, Intel Corporation.
>> *
>> * This program is free software; you can redistribute it and/or modify it
>> * under the terms and conditions of the GNU General Public License,
>> * version 2, as published by the Free Software Foundation.
>> *
>> * This program is distributed in the hope it will be useful, but WITHOUT
>> * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> * more details.
>> *
>> * You should have received a copy of the GNU General Public License along
>>with
>> * this program; if not, write to the Free Software Foundation, Inc., 59
>>Temple
>> * Place - Suite 330, Boston, MA 02111-1307 USA.
>> *
>> */
>>
>>#include <xen/time.h>
>>#include <asm/hvm/support.h>
>>#include <asm/hvm/vpt.h>
>>#include <asm/event.h>
>>
>>#define mode_is(d, name) \
>>    ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name)
>>
>>static void pt_lock(struct periodic_time *pt)
>>{
>>    struct vcpu *v;
>>
>>    for ( ; ; )
>>    {
>>        v = pt->vcpu;
>>        spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>        if ( likely(pt->vcpu == v) )
>>            break;
>>        spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>    }
>>}
>>
>>static void pt_unlock(struct periodic_time *pt)
>>{
>>    spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
>>}
>>
>>static void pt_process_missed_ticks(struct periodic_time *pt)
>>{
>>    s_time_t missed_ticks;
>>
>>    if ( pt->one_shot )
>>        return;
>>
>>    missed_ticks = NOW() - pt->scheduled;
>>    if ( missed_ticks <= 0 )
>>        return;
>>
>>    missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>
>>    if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
>>if ( missed_ticks > 1000 )
>>   {
>>/* TODO: Adjust guest time together */
>>pt->pending_intr_nr++;
>>   }
>>else
>>   {
>>pt->pending_intr_nr += missed_ticks;
>>   }
>>    }
>>
>>    pt->scheduled += missed_ticks * pt->period;
>>}
>>
>>static void pt_freeze_time(struct vcpu *v)
>>{
>>    if ( !mode_is(v->domain, delay_for_missed_ticks) )
>>        return;
>>
>>    v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v);
>>}
>>
>>static void pt_thaw_time(struct vcpu *v)
>>{
>>    if ( !mode_is(v->domain, delay_for_missed_ticks) )
>>        return;
>>
>>    if ( v->arch.hvm_vcpu.guest_time == 0 )
>>        return;
>>
>>    hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time);
>>    v->arch.hvm_vcpu.guest_time = 0;
>>}
>>
>>void pt_save_timer(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    if ( test_bit(_VPF_blocked, &v->pause_flags) )
>>        return;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>        stop_timer(&pt->timer);
>>
>>    pt_freeze_time(v);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void pt_restore_timer(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>pt_process_missed_ticks(pt);
>>        set_timer(&pt->timer, pt->scheduled);
>>    }
>>
>>    pt_thaw_time(v);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>static void pt_timer_fn(void *data)
>>{
>>    struct periodic_time *pt = data;
>>
>>    pt_lock(pt);
>>
>>    if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
>>if(!pt->pending_intr_nr)
>>   pt->pending_intr_nr++;
>>    }
>>    else
>>pt->pending_intr_nr++;
>>
>>    if ( !pt->one_shot )
>>    {
>>        pt->scheduled += pt->period;
>>pt_process_missed_ticks(pt);
>>        set_timer(&pt->timer, pt->scheduled);
>>    }
>>
>>    vcpu_kick(pt->vcpu);
>>
>>    pt_unlock(pt);
>>}
>>
>>void pt_update_irq(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>    uint64_t max_lag = -1ULL;
>>    int irq = -1;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>        if ( !is_isa_irq_masked(v, pt->irq) && pt->pending_intr_nr &&
>>             ((pt->last_plt_gtime + pt->period_cycles) < max_lag) )
>>        {
>>            max_lag = pt->last_plt_gtime + pt->period_cycles;
>>            irq = pt->irq;
>>        }
>>    }
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    if ( is_lvtt(v, irq) )
>>    {
>>        vlapic_set_irq(vcpu_vlapic(v), irq, 0);
>>    }
>>    else if ( irq >= 0 )
>>    {
>>        hvm_isa_irq_deassert(v->domain, irq);
>>        hvm_isa_irq_assert(v->domain, irq);
>>    }
>>}
>>
>>static struct periodic_time *is_pt_irq(
>>    struct vcpu *v, struct hvm_intack intack)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>    struct RTCState *rtc = &v->domain->arch.hvm_domain.pl_time.vrtc;
>>    int vector;
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>        if ( !pt->pending_intr_nr )
>>            continue;
>>
>>        if ( is_lvtt(v, pt->irq) )
>>        {
>>            if ( pt->irq != intack.vector )
>>                continue;
>>            return pt;
>>        }
>>
>>        vector = get_isa_irq_vector(v, pt->irq, intack.source);
>>
>>        /* RTC irq need special care */
>>        if ( (intack.vector != vector) ||
>>             ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) )
>>            continue;
>>
>>        return pt;
>>    }
>>
>>    return NULL;
>>}
>>
>>void pt_intr_post(struct vcpu *v, struct hvm_intack intack)
>>{
>>    struct periodic_time *pt;
>>    time_cb *cb;
>>    void *cb_priv;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    pt = is_pt_irq(v, intack);
>>    if ( pt == NULL )
>>    {
>>        spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>        return;
>>    }
>>
>>    if ( pt->one_shot )
>>    {
>>        pt->enabled = 0;
>>        list_del(&pt->list);
>>    }
>>    else
>>    {
>>        pt->pending_intr_nr--;
>>        if ( mode_is(v->domain, no_missed_tick_accounting) )
>>            pt->last_plt_gtime = hvm_get_guest_time(v);
>>        else
>>            pt->last_plt_gtime += pt->period_cycles;
>>    }
>>
>>    if ( mode_is(v->domain, delay_for_missed_ticks) &&
>>         (hvm_get_guest_time(v) < pt->last_plt_gtime) )
>>        hvm_set_guest_time(v, pt->last_plt_gtime);
>>
>>    cb = pt->cb;
>>    cb_priv = pt->priv;
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    if ( cb != NULL )
>>        cb(v, cb_priv);
>>}
>>
>>void pt_reset(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>        pt->pending_intr_nr = 0;
>>        pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>>        pt->scheduled = NOW() + pt->period;
>>        set_timer(&pt->timer, pt->scheduled);
>>    }
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void pt_migrate(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>        migrate_timer(&pt->timer, v->processor);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void create_periodic_time(
>>    struct vcpu *v, struct periodic_time *pt, uint64_t period,
>>    uint8_t irq, char one_shot, time_cb *cb, void *data)
>>{
>>    destroy_periodic_time(pt);
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    pt->enabled = 1;
>>    pt->pending_intr_nr = 0;
>>
>>    /* Periodic timer must be at least 0.9ms. */
>>    if ( (period < 900000) && !one_shot )
>>    {
>>        gdprintk(XENLOG_WARNING,
>>                 "HVM_PlatformTime: program too small period %"PRIu64"\n",
>>                 period);
>>        period = 900000;
>>    }
>>
>>    pt->period = period;
>>    pt->vcpu = v;
>>    pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>>    pt->irq = irq;
>>    pt->period_cycles = (u64)period * cpu_khz / 1000000L;
>>    pt->one_shot = one_shot;
>>    pt->scheduled = NOW() + period;
>>    /*
>>     * Offset LAPIC ticks from other timer ticks. Otherwise guests which use
>>     * LAPIC ticks for process accounting can see long sequences of process
>>     * ticks incorrectly accounted to interrupt processing.
>>     */
>>    if ( is_lvtt(v, irq) )
>>        pt->scheduled += period >> 1;
>>    pt->cb = cb;
>>    pt->priv = data;
>>
>>    list_add(&pt->list, &v->arch.hvm_vcpu.tm_list);
>>
>>    init_timer(&pt->timer, pt_timer_fn, pt, v->processor);
>>    set_timer(&pt->timer, pt->scheduled);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void destroy_periodic_time(struct periodic_time *pt)
>>{
>>    if ( !pt->enabled )
>>        return;
>>
>>    pt_lock(pt);
>>    pt->enabled = 0;
>>    list_del(&pt->list);
>>    pt_unlock(pt);
>>
>>    /*
>>     * pt_timer_fn() can run until this kill_timer() returns. We must do this
>>     * outside pt_lock() otherwise we can deadlock with pt_timer_fn().
>>     */
>>    kill_timer(&pt->timer);
>>}
>>    
>>
>
>
>  
>


[-- Attachment #2: vpt.xen.un.patch --]
[-- Type: text/x-patch, Size: 2113 bytes --]

diff -r 2717128cbdd1 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Wed Oct 31 10:07:42 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Thu Nov 01 16:19:06 2007 -0400
@@ -57,14 +57,23 @@ static void pt_process_missed_ticks(stru
         return;
 
     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
-    if ( missed_ticks > 1000 )
-    {
-        /* TODO: Adjust guest time together */
-        pt->pending_intr_nr++;
-    }
-    else
-    {
-        pt->pending_intr_nr += missed_ticks;
+
+    if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
+	/* When many guests are stacked on a node and the total number of vcpus
+	 * running loads is 10 timed the number of physical processors,
+	 * scheduling delays of 5 seconds are common. So set the threshold at
+	 * 100 seconds to be safe. It would be reasonable to remove this
+	 * check against threshold completely.
+	 */
+	if ( missed_ticks > 100000 )
+	    {
+		/* TODO: Adjust guest time together */
+		pt->pending_intr_nr++;
+	    }
+	else
+	    {
+		pt->pending_intr_nr += missed_ticks;
+	    }
     }
 
     pt->scheduled += missed_ticks * pt->period;
@@ -117,15 +126,7 @@ void pt_restore_timer(struct vcpu *v)
 
     list_for_each_entry ( pt, head, list )
     {
-        if ( !mode_is(v->domain, no_missed_tick_accounting) )
-        {
-            pt_process_missed_ticks(pt);
-        }
-        else if ( (NOW() - pt->scheduled) >= 0 )
-        {
-            pt->pending_intr_nr++;
-            pt->scheduled = NOW() + pt->period;
-        }
+	pt_process_missed_ticks(pt);
         set_timer(&pt->timer, pt->scheduled);
     }
 
@@ -140,13 +141,14 @@ static void pt_timer_fn(void *data)
 
     pt_lock(pt);
 
-    pt->pending_intr_nr++;
+    if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) || !pt->pending_intr_nr ) {
+	pt->pending_intr_nr++;
+    }
 
     if ( !pt->one_shot )
     {
         pt->scheduled += pt->period;
-        if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
-            pt_process_missed_ticks(pt);
+	pt_process_missed_ticks(pt);
         set_timer(&pt->timer, pt->scheduled);
     }
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

In the previous mail this

       (Before this patch clock error was 1.3%. The contributions of
   items '2' and '3' above to the improvement in accuracy are .4%
   and .9% respectively.

should be

       (Before this patch clock error was 1.3%. The contributions of
   items '2' and '3' above to the improvement in accuracy are .9%
   and .4% respectively.

thanks,
Dave

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 1/11/07 21:14, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> 1. Increase missed ticks threshold to 100 seconds. Per the comment in
> the patch,

I assume that's the magic number 100000 in the patch? I think removing the
check entirely would be better than an arbitrarily high threshold that we're
baiscally hoping will never trigger.

> 3.  Call pt_process_missed_ticks() unconditionally and place the
>     test for no_missed_tick_accounting inside pt_process_missed_ticks().
>     This returns the calculation for the next timer expiration
>     to the original method. The original method is important
>     as it sets up a theoretical time space at t=0 where expirations
>     occur only at n*period, where n is any integer. This, in turn, removes
>     rounding errors.

Why do 'rounding errors' matter? I thought that no_missed_tick_accounting
was for guests which sampled the TSC on tick interrupt and used that to
determine elapsed wall time, in which case why would it matter when exactly
the tick interrupt is delivered?

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir,

I agree with removing the check.

Sorry, I was a bit vague with the rounding term.

My experiments show that delivering clock interrupts
in a constant N*period time space results in a .4% improvement
in accuracy over delivering interrupts at context switch (in) time
and resetting the N*period time space.
Lets call the former technique method S (synch) and the later AS
(async).

Now I will offer an explanation without proof.
If you would like proof, I can instrument the hypervisor to see
if this explanation is correct.

Let D be the time that the clock vcpu is descheduled and P be
the clock period.  When D < P, I think there is an issue.

The reason is that Linux's offset calculation, which affects the
last clock interrupt tsc that is recorded, doesn't kick in if the
time since the last interrupt is less than P. In this case it sets
offset to zero. Linux records the tsc of the current (last) clock interrupt
as (current tsc - offset).

Interrupt delivery method AS delivers a clock interrupt
at context switch (in) time. When D < P, Linux records the
interrupt delivery time as current tsc and bumps jiffies.
This results in a gain of time equal to P-D over wall time.

For method S this never happens because the interrupt isn't
delivered until the next boundary.

regards,
Dave




Keir Fraser wrote:

>On 1/11/07 21:14, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>1. Increase missed ticks threshold to 100 seconds. Per the comment in
>>the patch,
>>    
>>
>
>I assume that's the magic number 100000 in the patch? I think removing the
>check entirely would be better than an arbitrarily high threshold that we're
>baiscally hoping will never trigger.
>  
>
>  
>
>>3.  Call pt_process_missed_ticks() unconditionally and place the
>>    test for no_missed_tick_accounting inside pt_process_missed_ticks().
>>    This returns the calculation for the next timer expiration
>>    to the original method. The original method is important
>>    as it sets up a theoretical time space at t=0 where expirations
>>    occur only at n*period, where n is any integer. This, in turn, removes
>>    rounding errors.
>>    
>>
>
>Why do 'rounding errors' matter? I thought that no_missed_tick_accounting
>was for guests which sampled the TSC on tick interrupt and used that to
>determine elapsed wall time, in which case why would it matter when exactly
>the tick interrupt is delivered?
>  
>


> -- Keir
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

Thanks for the explanation. I think you are mis-describing the current
behaviour of vpt.c though (If I understand it correctly myself, which I may
not!). As I understand it, we do *not* unconditionally deliver a tick and
reset the time space when a vcpu is scheduled onto a cpu. We only do that if
(NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has
passed. Otherwise we wait to deliver the next tick exactly one period after
the previous tick was delivered. The remainder of your explanation seems to
be predicated on the (impossible?) case that we deliver a tick less than one
period after the previous one.

Am I confused? :-) Also, what version of Linux are we talking about here,
and what periodic timer, and x86/64 or i386? There are lots of different
Linux timer-handling logics out there in the wild!

 -- Keir

On 2/11/07 15:51, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Let D be the time that the clock vcpu is descheduled and P be
> the clock period.  When D < P, I think there is an issue.
> 
> The reason is that Linux's offset calculation, which affects the
> last clock interrupt tsc that is recorded, doesn't kick in if the
> time since the last interrupt is less than P. In this case it sets
> offset to zero. Linux records the tsc of the current (last) clock interrupt
> as (current tsc - offset).
> 
> Interrupt delivery method AS delivers a clock interrupt
> at context switch (in) time. When D < P, Linux records the
> interrupt delivery time as current tsc and bumps jiffies.
> This results in a gain of time equal to P-D over wall time.
> 
> For method S this never happens because the interrupt isn't
> delivered until the next boundary.

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

By the way I checked in the other bits we agree on as changeset 16312. It
now looks quite sane to me.

 -- Keir

On 2/11/07 16:14, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> Thanks for the explanation. I think you are mis-describing the current
> behaviour of vpt.c though (If I understand it correctly myself, which I may
> not!). As I understand it, we do *not* unconditionally deliver a tick and
> reset the time space when a vcpu is scheduled onto a cpu. We only do that if
> (NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has
> passed. Otherwise we wait to deliver the next tick exactly one period after
> the previous tick was delivered. The remainder of your explanation seems to
> be predicated on the (impossible?) case that we deliver a tick less than one
> period after the previous one.
> 
> Am I confused? :-) Also, what version of Linux are we talking about here,
> and what periodic timer, and x86/64 or i386? There are lots of different
> Linux timer-handling logics out there in the wild!
> 
>  -- Keir
> 
> On 2/11/07 15:51, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
> 
>> Let D be the time that the clock vcpu is descheduled and P be
>> the clock period.  When D < P, I think there is an issue.
>> 
>> The reason is that Linux's offset calculation, which affects the
>> last clock interrupt tsc that is recorded, doesn't kick in if the
>> time since the last interrupt is less than P. In this case it sets
>> offset to zero. Linux records the tsc of the current (last) clock interrupt
>> as (current tsc - offset).
>> 
>> Interrupt delivery method AS delivers a clock interrupt
>> at context switch (in) time. When D < P, Linux records the
>> interrupt delivery time as current tsc and bumps jiffies.
>> This results in a gain of time equal to P-D over wall time.
>> 
>> For method S this never happens because the interrupt isn't
>> delivered until the next boundary.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

oops, you're right. I missed the  ( (NOW() - pt->scheduled) >= 0 ).

This means I will have to come up with another explanation.

-Dave


Keir Fraser wrote:

>Thanks for the explanation. I think you are mis-describing the current
>behaviour of vpt.c though (If I understand it correctly myself, which I may
>not!). As I understand it, we do *not* unconditionally deliver a tick and
>reset the time space when a vcpu is scheduled onto a cpu. We only do that if
>(NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has
>passed. Otherwise we wait to deliver the next tick exactly one period after
>the previous tick was delivered. The remainder of your explanation seems to
>be predicated on the (impossible?) case that we deliver a tick less than one
>period after the previous one.
>
>Am I confused? :-) Also, what version of Linux are we talking about here,
>and what periodic timer, and x86/64 or i386? There are lots of different
>Linux timer-handling logics out there in the wild!
>
> -- Keir
>
>On 2/11/07 15:51, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Let D be the time that the clock vcpu is descheduled and P be
>>the clock period.  When D < P, I think there is an issue.
>>
>>The reason is that Linux's offset calculation, which affects the
>>last clock interrupt tsc that is recorded, doesn't kick in if the
>>time since the last interrupt is less than P. In this case it sets
>>offset to zero. Linux records the tsc of the current (last) clock interrupt
>>as (current tsc - offset).
>>
>>Interrupt delivery method AS delivers a clock interrupt
>>at context switch (in) time. When D < P, Linux records the
>>interrupt delivery time as current tsc and bumps jiffies.
>>This results in a gain of time equal to P-D over wall time.
>>
>>For method S this never happens because the interrupt isn't
>>delivered until the next boundary.
>>    
>>
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 4157 bytes --]

Hi Keir,

Thanks for applying the fixes in the last submit.
In moving the test for no_missed_tick_accounting into 
pt_process_missed_ticks()
you forgot to add the scheduling part.

This patch adds the scheduling.
It also defines two options for scheduling, PT_SCHEDULE_SYNC and 
PT_SCHEDULE_ASYNC.
The default is PT_SCHEDULE_SYNC.

I am not providing any means for selecting between these two options.
My purpose is to have something to discuss and you can do as you like with
this patch or tell me what you want.

We have been discussing the difference between SYNC and ASYNC scheduling.
I found the reason the code checked in as of 10.31 had an error of .23%.
In the following fragment from pt_restore_timer()

        if ( !mode_is(v->domain, no_missed_tick_accounting) )
        {
            pt_process_missed_ticks(pt);
        }
        else if ( (NOW() - pt->scheduled) >= 0 )
        {
            pt->pending_intr_nr++;
            pt->scheduled = NOW() + pt->period;
        }
        set_timer(&pt->timer, pt->scheduled);

the pt->pending_intr_nr++; line is the problem because if the value of
pt->pending_intr_nr is non-zero before the increment, then you get extra
timer injections. We can either test for zero, set unconditionally to 1,
or leave the line out altogether and let the timeout take care of it.
I have chosen the later in the patch.

The result with this fixed, and with the patch submitted here
is as follows with the usual test I have been describing.

For both SYNC and ASYNC methods, with a sles9sp3-64 and
rh4u4-64 guest running at the same time, the errors were less than
.028%.  I have run some tests with single guests and big loads
and found sles to be in error by .1% with the ASYNC method.
All of the SYNC tests have been under .03%.

But my tests are short, usually 10 to 20 minutes. And there is quite a bit
of variability. Looking at the timer code for the two guests,
they look identical to me.

In my review of the Linux code, I see no reason why the interrupt
deliveries need to be "SYNC". This has been your intuition all along.

So, I leave the choice up to you.

thanks,
Dave


Dave Winchell wrote:

> oops, you're right. I missed the  ( (NOW() - pt->scheduled) >= 0 ).
>
> This means I will have to come up with another explanation.
>
> -Dave
>
>
> Keir Fraser wrote:
>
>> Thanks for the explanation. I think you are mis-describing the current
>> behaviour of vpt.c though (If I understand it correctly myself, which 
>> I may
>> not!). As I understand it, we do *not* unconditionally deliver a tick 
>> and
>> reset the time space when a vcpu is scheduled onto a cpu. We only do 
>> that if
>> (NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has
>> passed. Otherwise we wait to deliver the next tick exactly one period 
>> after
>> the previous tick was delivered. The remainder of your explanation 
>> seems to
>> be predicated on the (impossible?) case that we deliver a tick less 
>> than one
>> period after the previous one.
>>
>> Am I confused? :-) Also, what version of Linux are we talking about 
>> here,
>> and what periodic timer, and x86/64 or i386? There are lots of different
>> Linux timer-handling logics out there in the wild!
>>
>> -- Keir
>>
>> On 2/11/07 15:51, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>
>>  
>>
>>> Let D be the time that the clock vcpu is descheduled and P be
>>> the clock period.  When D < P, I think there is an issue.
>>>
>>> The reason is that Linux's offset calculation, which affects the
>>> last clock interrupt tsc that is recorded, doesn't kick in if the
>>> time since the last interrupt is less than P. In this case it sets
>>> offset to zero. Linux records the tsc of the current (last) clock 
>>> interrupt
>>> as (current tsc - offset).
>>>
>>> Interrupt delivery method AS delivers a clock interrupt
>>> at context switch (in) time. When D < P, Linux records the
>>> interrupt delivery time as current tsc and bumps jiffies.
>>> This results in a gain of time equal to P-D over wall time.
>>>
>>> For method S this never happens because the interrupt isn't
>>> delivered until the next boundary.
>>>   
>>
>>
>>
>>  
>>
>


[-- Attachment #2: vpt.patch.11.03 --]
[-- Type: text/plain, Size: 1187 bytes --]

diff -r 650cadd1b283 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Fri Nov 02 16:38:11 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Sat Nov 03 16:20:04 2007 -0400
@@ -45,12 +45,20 @@ static void pt_unlock(struct periodic_ti
     spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
 }
 
+#define PT_SCHEDULE_SYNC 0
+#define PT_SCHEDULE_ASYNC 1
+int pt_missed_option = PT_SCHEDULE_SYNC;
 static void pt_process_missed_ticks(struct periodic_time *pt)
 {
     s_time_t missed_ticks;
 
-    if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
-        return;
+    if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting)  && (pt_missed_option == PT_SCHEDULE_ASYNC)) {
+	if ( pt->one_shot )
+	    return;
+	if ( (NOW() - pt->scheduled) >= 0 )
+	    pt->scheduled = NOW() + pt->period;
+	return;
+    }
 
     if ( pt->one_shot )
         return;
@@ -60,7 +68,8 @@ static void pt_process_missed_ticks(stru
         return;
 
     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
-    pt->pending_intr_nr += missed_ticks;
+    if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
+	pt->pending_intr_nr += missed_ticks;
     pt->scheduled += missed_ticks * pt->period;
 }
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On 3/11/07 21:17, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Thanks for applying the fixes in the last submit.
> In moving the test for no_missed_tick_accounting into
> pt_process_missed_ticks()
> you forgot to add the scheduling part.

Actually it was deliberate, but clearly it was one code simplification too
far: thanks for spotting it! I'll go the async route, but we do need to set
pending_intr_nr to 1. We can't leave that out -- the point of the async
route is to send a tick to the vcpu immediately, since it hasn't had one for
more than a tick period. If we wait for the timeout to do that then we have
to wait a whole extra period after the vcpu is re-scheduled.

Attached is my proposed patch. I think it's quite neat. :-)

 -- Keir


[-- Attachment #2: timer.patch --]
[-- Type: application/octet-stream, Size: 1200 bytes --]

diff -r 650cadd1b283 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Fri Nov 02 16:38:11 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Sat Nov 03 22:22:23 2007 +0000
@@ -47,21 +47,26 @@ static void pt_unlock(struct periodic_ti
 
 static void pt_process_missed_ticks(struct periodic_time *pt)
 {
-    s_time_t missed_ticks;
+    s_time_t missed_ticks, now = NOW();
+
+    if ( pt->one_shot )
+        return;
+
+    missed_ticks = now - pt->scheduled;
+    if ( missed_ticks <= 0 )
+        return;
 
     if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
-        return;
-
-    if ( pt->one_shot )
-        return;
-
-    missed_ticks = NOW() - pt->scheduled;
-    if ( missed_ticks <= 0 )
-        return;
-
-    missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
-    pt->pending_intr_nr += missed_ticks;
-    pt->scheduled += missed_ticks * pt->period;
+    {
+        pt->pending_intr = 1;
+        pt->scheduled = now + pt->scheduled;
+    }
+    else
+    {
+        missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
+        pt->pending_intr_nr += missed_ticks;
+        pt->scheduled += missed_ticks * pt->period;
+    }
 }
 
 static void pt_freeze_time(struct vcpu *v)

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir Fraser wrote:

>On 3/11/07 21:17, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Thanks for applying the fixes in the last submit.
>>In moving the test for no_missed_tick_accounting into
>>pt_process_missed_ticks()
>>you forgot to add the scheduling part.
>>    
>>
>
>Actually it was deliberate, but clearly it was one code simplification too
>far: thanks for spotting it! I'll go the async route, but we do need to set
>pending_intr_nr to 1. We can't leave that out -- the point of the async
>route is to send a tick to the vcpu immediately, since it hasn't had one for
>more than a tick period. If we wait for the timeout to do that then we have
>to wait a whole extra period after the vcpu is re-scheduled.
>
>Attached is my proposed patch. I think it's quite neat. :-)
>  
>
It looks good to me.

thanks,
Dave

> -- Keir
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

Keir,

Attached is a fix for pt_process_missed_ticks().
Without this fix, systems run very erratically and some
guests panic on boot complaining that timer interrupts
are not working. As you can imagine.

Also, I have some longer term measurements of the
accuracy of the sync and async methods.
The hardware is an eight cpu AMD box. Two eight
vcpu guests, rh4u4-64 and sles9sp3-64.
All vcpus running loads.

Method   Test duration          Clock errors

SYNC       56000 sec         6.4, 6.7 sec   (.012%)
ASYNC      52000 sec         13, 19 sec     (.036%)

More testing should be done to validate the significance
of this difference.

regards,
Dave


Dave Winchell wrote:

>
> Keir Fraser wrote:
>
>> On 3/11/07 21:17, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>
>>  
>>
>>> Thanks for applying the fixes in the last submit.
>>> In moving the test for no_missed_tick_accounting into
>>> pt_process_missed_ticks()
>>> you forgot to add the scheduling part.
>>>   
>>
>>
>> Actually it was deliberate, but clearly it was one code 
>> simplification too
>> far: thanks for spotting it! I'll go the async route, but we do need 
>> to set
>> pending_intr_nr to 1. We can't leave that out -- the point of the async
>> route is to send a tick to the vcpu immediately, since it hasn't had 
>> one for
>> more than a tick period. If we wait for the timeout to do that then 
>> we have
>> to wait a whole extra period after the vcpu is re-scheduled.
>>
>> Attached is my proposed patch. I think it's quite neat. :-)
>>  
>>
> It looks good to me.
>
> thanks,
> Dave
>
>> -- Keir
>>
>>  
>>
>


[-- Attachment #2: vpt.patch.11.07 --]
[-- Type: text/plain, Size: 442 bytes --]

diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Mon Nov 05 13:23:55 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Wed Nov 07 08:55:45 2007 -0500
@@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru
     if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
     {
         pt->pending_intr_nr = 1;
-        pt->scheduled = now + pt->scheduled;
+        pt->scheduled = now + pt->period;
     }
     else
     {

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

Oh dear, that was a silly typo on my part. Thanks for tracking it down!

Well, I'm surprised that SYNC vs ASYNC differs, if the guest is really
tracking time via the TSC value. But it sounds like the actual time-tracking
algorithm in the guest must be a bit more complicated than that? I'd be very
happy to help with any further analysis.

 -- Keir

On 7/11/07 14:39, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Keir,
> 
> Attached is a fix for pt_process_missed_ticks().
> Without this fix, systems run very erratically and some
> guests panic on boot complaining that timer interrupts
> are not working. As you can imagine.
> 
> Also, I have some longer term measurements of the
> accuracy of the sync and async methods.
> The hardware is an eight cpu AMD box. Two eight
> vcpu guests, rh4u4-64 and sles9sp3-64.
> All vcpus running loads.
> 
> Method   Test duration          Clock errors
> 
> SYNC       56000 sec         6.4, 6.7 sec   (.012%)
> ASYNC      52000 sec         13, 19 sec     (.036%)
> 
> More testing should be done to validate the significance
> of this difference.
> 
> regards,
> Dave
> 
> 
> Dave Winchell wrote:
> 
>> 
>> Keir Fraser wrote:
>> 
>>> On 3/11/07 21:17, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>> 
>>>  
>>> 
>>>> Thanks for applying the fixes in the last submit.
>>>> In moving the test for no_missed_tick_accounting into
>>>> pt_process_missed_ticks()
>>>> you forgot to add the scheduling part.
>>>>   
>>> 
>>> 
>>> Actually it was deliberate, but clearly it was one code
>>> simplification too
>>> far: thanks for spotting it! I'll go the async route, but we do need
>>> to set
>>> pending_intr_nr to 1. We can't leave that out -- the point of the async
>>> route is to send a tick to the vcpu immediately, since it hasn't had
>>> one for
>>> more than a tick period. If we wait for the timeout to do that then
>>> we have
>>> to wait a whole extra period after the vcpu is re-scheduled.
>>> 
>>> Attached is my proposed patch. I think it's quite neat. :-)
>>>  
>>> 
>> It looks good to me.
>> 
>> thanks,
>> Dave
>> 
>>> -- Keir
>>> 
>>>  
>>> 
>> 
> 
> diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c
> --- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000
> +++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500
> @@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru
>      if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
>      {
>          pt->pending_intr_nr = 1;
> -        pt->scheduled = now + pt->scheduled;
> +        pt->scheduled = now + pt->period;
>      }
>      else
>      {

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Keir,

You can help me analyze the Linux code.
Here is the code from sles (linux-2.6.5-7.244) x86_64.


static irqreturn_t timer_interrupt(int irq, void *dev_id, struct pt_regs 
*regs)

        spin_lock(&i8253_lock);
        outb_p(0x00, 0x43);
        delay = inb_p(0x40);
        delay |= inb(0x40) << 8;
        spin_unlock(&i8253_lock);
        delay = LATCH - 1 - delay;

     rdtscll_sync(&tsc);
       
        offset = (((tsc - vxtime.last_tsc) *
               vxtime.tsc_quot) >> 32) - (USEC_PER_SEC / HZ);

        if (offset < 0)
            offset = 0;

        if (offset > (USEC_PER_SEC / HZ)) {
            lost = offset / (USEC_PER_SEC / HZ);
            offset %= (USEC_PER_SEC / HZ);
        }

        monotonic_base += (tsc - vxtime.last_tsc)*1000000/cpu_khz ;

        vxtime.last_tsc = tsc - vxtime.quot * delay / vxtime.tsc_quot;

        if ((((tsc - vxtime.last_tsc) *
              vxtime.tsc_quot) >> 32) < offset)
            vxtime.last_tsc = tsc -
                (((long) offset << 32) / vxtime.tsc_quot) - 1;
   

    if (lost > 0) {
        if (report_lost_ticks) {
            printk(KERN_WARNING "time.c: Lost %ld timer "
                   "tick(s)! ", lost);
            print_symbol("rip %s)\n", regs->rip);
        }
        jiffies += lost;
    }
    do_timer(regs);


Now as 'offset' gets modified it represents the remainder in the
lost calculation. Our data indicates that when offset is close to zero,
timekeeping is more accurate. So this leads us to this line of code:

        if ((((tsc - vxtime.last_tsc) *
              vxtime.tsc_quot) >> 32) < offset)
            vxtime.last_tsc = tsc -
                (((long) offset << 32) / vxtime.tsc_quot) - 1;

Thus the problem seems to be the way the code switches from adjusting 
last_tsc
by the delay which is the normal mode when interrupts are timely or SYNCed
to handling offsets larger than the delay.

I'm concerned about the way delay is blown off in the final calculation

            vxtime.last_tsc = tsc -
                (((long) offset << 32) / vxtime.tsc_quot) - 1;


If an interrupt is right on time, and the delay is the same as last time,
then the offset should be zero in my mind. In the code above, offset will
equal delay for this example.

What do you think?

thanks,
Dave



Keir Fraser wrote:

>Oh dear, that was a silly typo on my part. Thanks for tracking it down!
>
>Well, I'm surprised that SYNC vs ASYNC differs, if the guest is really
>tracking time via the TSC value. But it sounds like the actual time-tracking
>algorithm in the guest must be a bit more complicated than that? I'd be very
>happy to help with any further analysis.
>
> -- Keir
>
>On 7/11/07 14:39, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Keir,
>>
>>Attached is a fix for pt_process_missed_ticks().
>>Without this fix, systems run very erratically and some
>>guests panic on boot complaining that timer interrupts
>>are not working. As you can imagine.
>>
>>Also, I have some longer term measurements of the
>>accuracy of the sync and async methods.
>>The hardware is an eight cpu AMD box. Two eight
>>vcpu guests, rh4u4-64 and sles9sp3-64.
>>All vcpus running loads.
>>
>>Method   Test duration          Clock errors
>>
>>SYNC       56000 sec         6.4, 6.7 sec   (.012%)
>>ASYNC      52000 sec         13, 19 sec     (.036%)
>>
>>More testing should be done to validate the significance
>>of this difference.
>>
>>regards,
>>Dave
>>
>>
>>Dave Winchell wrote:
>>
>>    
>>
>>>Keir Fraser wrote:
>>>
>>>      
>>>
>>>>On 3/11/07 21:17, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>>>
>>>> 
>>>>
>>>>        
>>>>
>>>>>Thanks for applying the fixes in the last submit.
>>>>>In moving the test for no_missed_tick_accounting into
>>>>>pt_process_missed_ticks()
>>>>>you forgot to add the scheduling part.
>>>>>  
>>>>>          
>>>>>
>>>>Actually it was deliberate, but clearly it was one code
>>>>simplification too
>>>>far: thanks for spotting it! I'll go the async route, but we do need
>>>>to set
>>>>pending_intr_nr to 1. We can't leave that out -- the point of the async
>>>>route is to send a tick to the vcpu immediately, since it hasn't had
>>>>one for
>>>>more than a tick period. If we wait for the timeout to do that then
>>>>we have
>>>>to wait a whole extra period after the vcpu is re-scheduled.
>>>>
>>>>Attached is my proposed patch. I think it's quite neat. :-)
>>>> 
>>>>
>>>>        
>>>>
>>>It looks good to me.
>>>
>>>thanks,
>>>Dave
>>>
>>>      
>>>
>>>>-- Keir
>>>>
>>>> 
>>>>
>>>>        
>>>>
>>diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c
>>--- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000
>>+++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500
>>@@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru
>>     if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
>>     {
>>         pt->pending_intr_nr = 1;
>>-        pt->scheduled = now + pt->scheduled;
>>+        pt->scheduled = now + pt->period;
>>     }
>>     else
>>     {
>>    
>>
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 7/11/07 16:23, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> I'm concerned about the way delay is blown off in the final calculation
> 
>             vxtime.last_tsc = tsc -
>                 (((long) offset << 32) / vxtime.tsc_quot) - 1;
> 
> If an interrupt is right on time, and the delay is the same as last time,
> then the offset should be zero in my mind. In the code above, offset will
> equal delay for this example.
> 
> What do you think?

I don't think this code likes the ASYNC method *at all*. The reason is that
it estimates how late the tick interrupt is by reading the PIT counter. It
knows the interrupt should have been triggered when the counter wrapped at
zero. But of course, if we are delivering ticks in ASYNC mode then quickly
the time we deliver an interrupt has *nothing* to do with the current PIT
counter value! Hence the lines that effectively fold max(offset, delay) into
last_tsc are just not going to work properly, because delay is a uniform
random variable, and hence we probably lose time.

Let me see if I can come up with a patch that gets the best of ASYNC and
SYNC in one handy mode...

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 7/11/07 17:10, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> I don't think this code likes the ASYNC method *at all*. The reason is that
> it estimates how late the tick interrupt is by reading the PIT counter. It
> knows the interrupt should have been triggered when the counter wrapped at
> zero. But of course, if we are delivering ticks in ASYNC mode then quickly
> the time we deliver an interrupt has *nothing* to do with the current PIT
> counter value! Hence the lines that effectively fold max(offset, delay) into
> last_tsc are just not going to work properly, because delay is a uniform
> random variable, and hence we probably lose time.

Actually, no, I don't understand why this code is less accurate with ASYNC
mode. My comments on 'delay' above are not true I think. Both 'delay' and
'offset' are accurate estimates of the time elapsed since the last actual
tick threshold, when the PIT counter rolled over. And hence the logic that
is being applied *should* work...

I wonder if something silly like the following is happening: Because we send
unsynchronised tick interrupts, they can actually be delivered when the PIT
is just about to roll over (but hasn't yet). If that happens we could get a
worst-case disagreement between offset (derived from TSC) and delay (derived
from PIT). Let's say that when we read the TSC it looks like we have just
passed a tick rollover. In that case we will count some whole number of lost
ticks and offset will end up being a very small number (very little 'slop'
after a whole number of ticks is accounted). However, when we read the PIT
it looks like we are just about to roll over (but haven't yet) and so delay
will be a very large number: about as large as it can be. In this case we
will subtract a lot from last_tsc, because it looks like we've had nearly a
full tick of delay. But actually the amount of delay that is getting
subtracted from last_tsc has already been accounted for elsewhere as a full
tick!

If this analysis is the truth of the matter then we would actually expect
the ASYNC mode to cause the guest timeofday to run *fast* rather than slow.

Anyhow, it's bound to be something silly like this. :-)

The SYNC/ASYNC method I was going to propose by the way was going to be, in
pt_process_missed_ticks():
    pt->scheduled += missed_ticks * pt->period;
    if ( mode_is(no_missed_tick_accounting) )
        pt->pending_intr_nr = 1;
    else
        pt->pending_intr_nr += missed_ticks;

So, you can see we send an interrupt immediately (and ASYNC) if any ticks
have been missed, but then successive ticks are delivered 'on the beat'. A
possible middleground? Or perhaps we should just go with SYNC after all...

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> So, you can see we send an interrupt immediately (and ASYNC) if any ticks
> have been missed, but then successive ticks are delivered 'on the beat'. A
> possible middleground? Or perhaps we should just go with SYNC after all...

How do these Linux x64 guests fare with the original and default timer mode,
by the way? I would expect that time should be accounted pretty accurately
in that mode, albeit with more interrupts than you'd like. Or is the lack of
synchronisation of TSCs across VCPUs causing issues that you're trying to
avoid?

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Keir,

My feeling is that we should go full SYNC. Yes, in theory the
guests should be able to handle ASYNC, but in reality it appears that
some do not. Since it is easy for us to give them SYNC,
lets just do it and not stress them out.

In terms of sending the one ASYNC interrupt, I would say no.
Lets simply give it to him on the next boundary. If we give
it to him ASYNC he will have a big offset to deal with, exactly what
gives him trouble.

 > How do these Linux x64 guests fare with the original and default 
timer mode,
 > by the way? I would expect that time should be accounted pretty 
accurately
 > in that mode, albeit with more interrupts than you'd like.

For default mode as checked into unstable is now,
64 bit guests should run quite fast as missed is calculated and then a bunch
of additional interrupts are delivered. On the other hand
32bit guests very well in default mode.

For the original code, before we put in the constant tsc offset business,
64bit guests run poorly and 32bit quests very well time-wise.

 > Or is the lack of
 > synchronization of TSCs across VCPUs causing issues that you're trying to
 > avoid?

This does cause issues, but its not the only contributor to poor timing.
Having TSCs synchronized across vcpus will help some of the time going
backwards problems we have seen, I think.

Regards,
Dave

Keir Fraser wrote:

>On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
>
>  
>
>>So, you can see we send an interrupt immediately (and ASYNC) if any ticks
>>have been missed, but then successive ticks are delivered 'on the beat'. A
>>possible middleground? Or perhaps we should just go with SYNC after all...
>>    
>>
>
>How do these Linux x64 guests fare with the original and default timer mode,
>by the way? I would expect that time should be accounted pretty accurately
>in that mode, albeit with more interrupts than you'd like. Or is the lack of
>synchronisation of TSCs across VCPUs causing issues that you're trying to
>avoid?
>
> -- Keir
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 7/11/07 19:38, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> My feeling is that we should go full SYNC. Yes, in theory the
> guests should be able to handle ASYNC, but in reality it appears that
> some do not. Since it is easy for us to give them SYNC,
> lets just do it and not stress them out.

One problem with pure SYNC is there's a fair chance you won't deliver any
ticks at all for a long time, if the guest only runs in short bursts (e.g.,
I/O bound) and happens not to be running on any tick boundary. I'm not sure
how much that matters. It could cause time goes backwards if the time
extrapolation via the TSC is not perfectly accurate, or cause problems if
there are any assumptions that TSC delta since last tick fits in 32 bits
(less likely in x64 code I suppose). Anyway, my point is that only testing
VCPUs under full load may cause us to optimise in ways that have nasty
unexpected effects for other workloads.

> For default mode as checked into unstable is now,
> 64 bit guests should run quite fast as missed is calculated and then a bunch
> of additional interrupts are delivered. On the other hand
> 32bit guests very well in default mode.
> 
> For the original code, before we put in the constant tsc offset business,
> 64bit guests run poorly and 32bit quests very well time-wise.

The default mode hasn't changed. Are you under the impression that
missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 guests
run badly with that because they treat every one of the missed ticks they
receive as a full tick.

 -- Keir

>> Or is the lack of
>> synchronization of TSCs across VCPUs causing issues that you're trying to
>> avoid?
> 
> This does cause issues, but its not the only contributor to poor timing.
> Having TSCs synchronized across vcpus will help some of the time going
> backwards problems we have seen, I think.
> 
> Regards,
> Dave
> 
> Keir Fraser wrote:
> 
>> On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
>> 
>>  
>> 
>>> So, you can see we send an interrupt immediately (and ASYNC) if any ticks
>>> have been missed, but then successive ticks are delivered 'on the beat'. A
>>> possible middleground? Or perhaps we should just go with SYNC after all...
>>>    
>>> 
>> 
>> How do these Linux x64 guests fare with the original and default timer mode,
>> by the way? I would expect that time should be accounted pretty accurately
>> in that mode, albeit with more interrupts than you'd like. Or is the lack of
>> synchronisation of TSCs across VCPUs causing issues that you're trying to
>> avoid?
>> 
>> -- Keir
>> 
>> 
>>  
>> 
> 

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Keir,

I've added comments below.
See my next mail on some interesting performance numbers.

thanks,
Dave

Keir Fraser wrote:

>On 7/11/07 19:38, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>My feeling is that we should go full SYNC. Yes, in theory the
>>guests should be able to handle ASYNC, but in reality it appears that
>>some do not. Since it is easy for us to give them SYNC,
>>lets just do it and not stress them out.
>>    
>>
>
>One problem with pure SYNC is there's a fair chance you won't deliver any
>ticks at all for a long time, if the guest only runs in short bursts (e.g.,
>I/O bound) and happens not to be running on any tick boundary. I'm not sure
>how much that matters. It could cause time goes backwards if the time
>extrapolation via the TSC is not perfectly accurate, or cause problems if
>there are any assumptions that TSC delta since last tick fits in 32 bits
>(less likely in x64 code I suppose). Anyway, my point is that only testing
>VCPUs under full load may cause us to optimise in ways that have nasty
>unexpected effects for other workloads.
>  
>
I agree that this could be a problem. I have an idea that could give us full
SYNC and eliminate the long periods without clock interrupts.
In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1.
In pt_save_timer():

    list_for_each_entry ( pt, head, list )
        if(!pt->run_timer)
             stop_timer(&pt->timer);

And in pt_timer_fn():

    pt->run_timer = 0;

So, for a guest that misses a tick, we will interrupt him once from the
descheduled state and then leave him alone in the descheduled state.

>  
>
>>For default mode as checked into unstable is now,
>>64 bit guests should run quite fast as missed is calculated and then a bunch
>>of additional interrupts are delivered. On the other hand
>>32bit guests very well in default mode.
>>
>>For the original code, before we put in the constant tsc offset business,
>>64bit guests run poorly and 32bit quests very well time-wise.
>>    
>>
>
>The default mode hasn't changed. Are you under the impression that
>missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 guests
>run badly with that because they treat every one of the missed ticks they
>receive as a full tick.
>  
>
Sorry, I was confused.
However, the default mode will still run poorly for 64 bit guests because
of the pending_nr's accumulated while the guest has interrupts disabled.
As I recall, the effect is quite large, on the order of 10% error.
I'll get you a number later today.

> -- Keir
>
>  
>
>>>Or is the lack of
>>>synchronization of TSCs across VCPUs causing issues that you're trying to
>>>avoid?
>>>      
>>>
>>This does cause issues, but its not the only contributor to poor timing.
>>Having TSCs synchronized across vcpus will help some of the time going
>>backwards problems we have seen, I think.
>>
>>Regards,
>>Dave
>>
>>Keir Fraser wrote:
>>
>>    
>>
>>>On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
>>>
>>> 
>>>
>>>      
>>>
>>>>So, you can see we send an interrupt immediately (and ASYNC) if any ticks
>>>>have been missed, but then successive ticks are delivered 'on the beat'. A
>>>>possible middleground? Or perhaps we should just go with SYNC after all...
>>>>   
>>>>
>>>>        
>>>>
>>>How do these Linux x64 guests fare with the original and default timer mode,
>>>by the way? I would expect that time should be accounted pretty accurately
>>>in that mode, albeit with more interrupts than you'd like. Or is the lack of
>>>synchronisation of TSCs across VCPUs causing issues that you're trying to
>>>avoid?
>>>
>>>-- Keir
>>>
>>>
>>> 
>>>
>>>      
>>>
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 8/11/07 14:43, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> I agree that this could be a problem. I have an idea that could give us full
> SYNC and eliminate the long periods without clock interrupts.
> In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1.
> In pt_save_timer():
> 
>     list_for_each_entry ( pt, head, list )
>         if(!pt->run_timer)
>              stop_timer(&pt->timer);
> 
> And in pt_timer_fn():
> 
>     pt->run_timer = 0;
> 
> So, for a guest that misses a tick, we will interrupt him once from the
> descheduled state and then leave him alone in the descheduled state.

Well, I'd rather not complicate the code if it's avoidable. I checked in a
SYNC/ASYNC combo and code simplification as changeset 16341, and it'd be
interesting to know how that fares against your suggested scheme. I suppose
as long as we're better than ntpd's tolerance it doesn't actually matter all
that much.

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

ok, I'll try your new code and let you know what
I find.

-Dave

Keir Fraser wrote:

>On 8/11/07 14:43, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>I agree that this could be a problem. I have an idea that could give us full
>>SYNC and eliminate the long periods without clock interrupts.
>>In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1.
>>In pt_save_timer():
>>
>>    list_for_each_entry ( pt, head, list )
>>        if(!pt->run_timer)
>>             stop_timer(&pt->timer);
>>
>>And in pt_timer_fn():
>>
>>    pt->run_timer = 0;
>>
>>So, for a guest that misses a tick, we will interrupt him once from the
>>descheduled state and then leave him alone in the descheduled state.
>>    
>>
>
>Well, I'd rather not complicate the code if it's avoidable. I checked in a
>SYNC/ASYNC combo and code simplification as changeset 16341, and it'd be
>interesting to know how that fares against your suggested scheme. I suppose
>as long as we're better than ntpd's tolerance it doesn't actually matter all
>that much.
>
> -- Keir
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 2929 bytes --]

Keir,

Here are the results of the testing I've done recently.
There are three protocols tested. I'll call them ASYNC for the code
as it stood just before your last check-in, MIXED (for ASYNC+SYNC) for
the code you recently checked in, and SYNC for the method I proposed
in the last mail.



Date     Duration       Protocol                 sles, rhat error

11/07  23 hrs 40 min     ASYNC         -4.96 sec, +4.42 sec    -.006%,  
+.005%
11/08         28 min     MIXED          -.75 sec,  -.67 sec    -.045%,  
-.040%
11/08   2 hrs 21 min      SYNC          -.80 sec,  -.34 sec,   -.009%,  
-.004%
11/08   1 hr  25 min      SYNC          -.24 sec,  -.26 sec,   -.005%,  
-.005%
11/08  15 hrs 39 min     MIXED        -19.   sec,-17.4  sec,   -.034%,  
-.031%
11/09   3 hrs 19 min     ASYNC          -.13 sec, +1.44 sec,   -.001%,  
+.012%


Each protocol was run a couple of times to gage the repeatability.

Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
I ran another ASYNC test. I think there may have been something
wrong with the code I used a couple of days ago for ASYNC. It may have been
missing the immediate delivery of interrupt after context switch in.

My results indicate that either SYNC or ASYNC give acceptable accuracy,
each running consistently around or under .01%. MIXED has a fairly high 
error of
greater than .03%. Probably too close to .05% ntp threshold for comfort.
I don't have an overnight run with SYNC. I plan to leave SYNC running
over the weekend. If you'd rather I can leave MIXED running instead.

It may be too early to pick the protocol and I can run more overnight tests
next week.

I have attached a patch based on what's checked-in now that shows what I
tested for SYNC and ASYNC. For MIXED I used the code that is checked-in now.
The patch looks more complicated than the change really is as
it backs out the MIXED code you put in.

Regards,
Dave


Keir Fraser wrote:

>On 8/11/07 14:43, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>I agree that this could be a problem. I have an idea that could give us full
>>SYNC and eliminate the long periods without clock interrupts.
>>In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1.
>>In pt_save_timer():
>>
>>    list_for_each_entry ( pt, head, list )
>>        if(!pt->run_timer)
>>             stop_timer(&pt->timer);
>>
>>And in pt_timer_fn():
>>
>>    pt->run_timer = 0;
>>
>>So, for a guest that misses a tick, we will interrupt him once from the
>>descheduled state and then leave him alone in the descheduled state.
>>    
>>
>
>Well, I'd rather not complicate the code if it's avoidable. I checked in a
>SYNC/ASYNC combo and code simplification as changeset 16341, and it'd be
>interesting to know how that fares against your suggested scheme. I suppose
>as long as we're better than ntpd's tolerance it doesn't actually matter all
>that much.
>
> -- Keir
>
>
>  
>


[-- Attachment #2: vpt.11.09.patch --]
[-- Type: text/x-patch, Size: 3281 bytes --]

diff -r c0bdfda5183d xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Thu Nov 08 14:50:01 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Fri Nov 09 09:08:59 2007 -0500
@@ -45,6 +45,26 @@ static void pt_unlock(struct periodic_ti
     spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
 }
 
+#include <xen/keyhandler.h>
+#define PT_MISSED_OPTION_SYNC 0
+#define PT_MISSED_OPTION_ASYNC 1
+int pt_missed_option = 0;
+static void pt_missed_option_fn(unsigned char key)
+{
+    pt_missed_option = pt_missed_option ^ 1;
+    if(pt_missed_option == PT_MISSED_OPTION_SYNC)
+	printk("pt_missed_option = 0 (SYNC)\n");
+    else
+	printk("pt_missed_option = 1 (ASYNC)\n");
+}
+static int __init pt_missed_option_init(void)
+{
+    register_keyhandler('D', pt_missed_option_fn, "pt_missed_option");
+    return 0;
+}
+
+__initcall(pt_missed_option_init);
+
 static void pt_process_missed_ticks(struct periodic_time *pt)
 {
     s_time_t missed_ticks, now = NOW();
@@ -56,9 +76,20 @@ static void pt_process_missed_ticks(stru
     if ( missed_ticks <= 0 )
         return;
 
-    missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
-    pt->pending_intr_nr += missed_ticks;
-    pt->scheduled += missed_ticks * pt->period;
+    if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting)  && (pt_missed_option == PT_MISSED_OPTION_ASYNC) )
+    {
+        pt->pending_intr_nr = 1;
+        pt->scheduled = now + pt->period;
+    }
+    else
+    {
+        missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
+	if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
+	    pt->pending_intr_nr += missed_ticks;
+	else
+	    pt->run_timer = 1;
+        pt->scheduled += missed_ticks * pt->period;
+    }
 }
 
 static void pt_freeze_time(struct vcpu *v)
@@ -91,8 +122,10 @@ void pt_save_timer(struct vcpu *v)
 
     spin_lock(&v->arch.hvm_vcpu.tm_lock);
 
-    list_for_each_entry ( pt, head, list )
-        stop_timer(&pt->timer);
+    list_for_each_entry ( pt, head, list ) {
+	if(!pt->run_timer)
+	    stop_timer(&pt->timer);
+    }
 
     pt_freeze_time(v);
 
@@ -123,7 +156,12 @@ static void pt_timer_fn(void *data)
 
     pt_lock(pt);
 
-    pt->pending_intr_nr++;
+    if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
+        pt->pending_intr_nr = 1;
+	pt->run_timer = 0;
+    }
+    else
+        pt->pending_intr_nr++;
 
     if ( !pt->one_shot )
     {
@@ -224,16 +262,11 @@ void pt_intr_post(struct vcpu *v, struct
     }
     else
     {
+        pt->pending_intr_nr--;
         if ( mode_is(v->domain, no_missed_tick_accounting) )
-        {
             pt->last_plt_gtime = hvm_get_guest_time(v);
-            pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */
-        }
         else
-        {
             pt->last_plt_gtime += pt->period_cycles;
-            pt->pending_intr_nr--;
-        }
     }
 
     if ( mode_is(v->domain, delay_for_missed_ticks) &&
diff -r c0bdfda5183d xen/include/asm-x86/hvm/vpt.h
--- a/xen/include/asm-x86/hvm/vpt.h	Thu Nov 08 14:50:01 2007 +0000
+++ b/xen/include/asm-x86/hvm/vpt.h	Fri Nov 09 08:53:14 2007 -0500
@@ -84,6 +84,7 @@ struct periodic_time {
     struct timer timer;         /* ac_timer */
     time_cb *cb;
     void *priv;                 /* point back to platform time source */
+    u32 run_timer;
 };
 
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 9/11/07 19:22, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
> I ran another ASYNC test. I think there may have been something
> wrong with the code I used a couple of days ago for ASYNC. It may have been
> missing the immediate delivery of interrupt after context switch in.
> 
> My results indicate that either SYNC or ASYNC give acceptable accuracy,
> each running consistently around or under .01%. MIXED has a fairly high
> error of
> greater than .03%. Probably too close to .05% ntp threshold for comfort.
> I don't have an overnight run with SYNC. I plan to leave SYNC running
> over the weekend. If you'd rather I can leave MIXED running instead.
> 
> It may be too early to pick the protocol and I can run more overnight tests
> next week.

I'm a bit worried about any unwanted side effects of the SYNC+run_timer
approach -- e.g., whether timer wakeups will cause higher system-wide CPU
contention. I find it easier to think through the implications of ASYNC. I'm
surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
delivers more timer interrupts than the other approaches, and each interrupt
event causes a small accumulated error?

Overall I would consider MIXED and ASYNC as favourites and if the latter is
actually more accurate then I can simply revert the changeset that
implemented MIXED.

Perhaps rather than running more of the same workloads you could try idle
VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
don't have any data on workloads that aren't CPU bound, so that's really an
obvious place to put any further effort imo.

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir,

I'll run the I/O and idle tests you suggested.
Since the tests will be run in the evenings, they won't be
completed until the end of the week.

Below is the table of tests updated with the weekend SYNC with cpu load 
test,
where the error was less than .01% for both linux guests.

 > I'm a bit worried about any unwanted side effects of the SYNC+run_timer
 > approach -- e.g., whether timer wakeups will cause higher system-wide CPU
 > contention.


I agree. If the SYNC model turns out to be desirable from an accuracy
standpoint, then the significance of the additional wakeups should
characterized.

Regards,
Dave


Date     Duration       Protocol                 sles, rhat 
error                      load

11/07  23 hrs 40 min     ASYNC         -4.96 sec, +4.42 sec    -.006%,  
+.005%         cpu
11/09   3 hrs 19 min     ASYNC          -.13 sec, +1.44 sec,   -.001%,  
+.012%         cpu

11/08   2 hrs 21 min      SYNC          -.80 sec,  -.34 sec,   -.009%,  
-.004%         cpu
11/08   1 hr  25 min      SYNC          -.24 sec,  -.26 sec,   -.005%,  
-.005%         cpu
11/12  65 hrs 40 min      SYNC        -18    sec, -8    sec,   -.008%,  
-.003%         cpu

11/08         28 min     MIXED          -.75 sec,  -.67 sec    -.045%,  
-.040%         cpu
11/08  15 hrs 39 min     MIXED        -19.   sec,-17.4  sec,   -.034%,  
-.031%         cpu


Keir Fraser wrote:

>On 9/11/07 19:22, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
>>I ran another ASYNC test. I think there may have been something
>>wrong with the code I used a couple of days ago for ASYNC. It may have been
>>missing the immediate delivery of interrupt after context switch in.
>>
>>My results indicate that either SYNC or ASYNC give acceptable accuracy,
>>each running consistently around or under .01%. MIXED has a fairly high
>>error of
>>greater than .03%. Probably too close to .05% ntp threshold for comfort.
>>I don't have an overnight run with SYNC. I plan to leave SYNC running
>>over the weekend. If you'd rather I can leave MIXED running instead.
>>
>>It may be too early to pick the protocol and I can run more overnight tests
>>next week.
>>    
>>
>
>I'm a bit worried about any unwanted side effects of the SYNC+run_timer
>approach -- e.g., whether timer wakeups will cause higher system-wide CPU
>contention. I find it easier to think through the implications of ASYNC. I'm
>surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
>delivers more timer interrupts than the other approaches, and each interrupt
>event causes a small accumulated error?
>
>Overall I would consider MIXED and ASYNC as favourites and if the latter is
>actually more accurate then I can simply revert the changeset that
>implemented MIXED.
>
>Perhaps rather than running more of the same workloads you could try idle
>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
>don't have any data on workloads that aren't CPU bound, so that's really an
>obvious place to put any further effort imo.
>
> -- Keir
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir,

The accuracy data I've collected for i/o loads for the
various time protocols follows. In addition, the data
for cpu loads is shown.

The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
Two guests, red hat and sles 64 bit, 8 vcpu each.
The cpu load is usex -e36 on each guest.
(usex is available at http://people.redhat.com/anderson/usex.)
i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.

The loads labeled i/o-32 are 32 instances of dd.
Also, these are run on 4 cpu AMD box.
In addition, there is an idle rh-32bit guest.
All three guests are 8vcpu.

The loads labeled i/o-4/32 are the same as i/o-32
except that the redhat-64 guest has 4 instances of dd.

Date Duration Protocol sles, rhat error load

11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu

11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu

11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu


11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8

11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8

11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8



11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32

11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32


Overhead measurements:

Progress in terms of number of passes through a fixed system workload
on an 8 vcpu red hat with an 8 vcpu sles idle.
The workload was usex -b48.


ASYNC 167 min 145 passes .868 passes/min
SYNC 167 min 144 passes .862 passes/min
SYNC 1065 min 919 passes .863 passes/min
MIXED 221 min 196 passes .887 passes/min


Conclusions:

The only protocol which meets the .05% accuracy requirement for ntp 
tracking under the loads
above is the SYNC protocol. The worst case accuracies for SYNC, MIXED, 
and ASYNC
are .022%, .12%, and .14%, respectively.

We could reduce the cost of the SYNC method by only scheduling the extra 
wakeups if a certain number
of ticks are missed.

Regards,
Dave

Keir Fraser wrote:

>On 9/11/07 19:22, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
>>I ran another ASYNC test. I think there may have been something
>>wrong with the code I used a couple of days ago for ASYNC. It may have been
>>missing the immediate delivery of interrupt after context switch in.
>>
>>My results indicate that either SYNC or ASYNC give acceptable accuracy,
>>each running consistently around or under .01%. MIXED has a fairly high
>>error of
>>greater than .03%. Probably too close to .05% ntp threshold for comfort.
>>I don't have an overnight run with SYNC. I plan to leave SYNC running
>>over the weekend. If you'd rather I can leave MIXED running instead.
>>
>>It may be too early to pick the protocol and I can run more overnight tests
>>next week.
>>    
>>
>
>I'm a bit worried about any unwanted side effects of the SYNC+run_timer
>approach -- e.g., whether timer wakeups will cause higher system-wide CPU
>contention. I find it easier to think through the implications of ASYNC. I'm
>surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
>delivers more timer interrupts than the other approaches, and each interrupt
>event causes a small accumulated error?
>
>Overall I would consider MIXED and ASYNC as favourites and if the latter is
>actually more accurate then I can simply revert the changeset that
>implemented MIXED.
>
>Perhaps rather than running more of the same workloads you could try idle
>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
>don't have any data on workloads that aren't CPU bound, so that's really an
>obvious place to put any further effort imo.
>
> -- Keir
>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

Please take a look at xen-unstable changeset 16545.

 -- Keir

On 26/11/07 20:57, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Keir,
> 
> The accuracy data I've collected for i/o loads for the
> various time protocols follows. In addition, the data
> for cpu loads is shown.
> 
> The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> Two guests, red hat and sles 64 bit, 8 vcpu each.
> The cpu load is usex -e36 on each guest.
> (usex is available at http://people.redhat.com/anderson/usex.)
> i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> 
> The loads labeled i/o-32 are 32 instances of dd.
> Also, these are run on 4 cpu AMD box.
> In addition, there is an idle rh-32bit guest.
> All three guests are 8vcpu.
> 
> The loads labeled i/o-4/32 are the same as i/o-32
> except that the redhat-64 guest has 4 instances of dd.
> 
> Date Duration Protocol sles, rhat error load
> 
> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
> 
> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> 
> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
> 
> 
> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
> 
> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> 
> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
> 
> 
> 
> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> 
> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
> 
> 
> Overhead measurements:
> 
> Progress in terms of number of passes through a fixed system workload
> on an 8 vcpu red hat with an 8 vcpu sles idle.
> The workload was usex -b48.
> 
> 
> ASYNC 167 min 145 passes .868 passes/min
> SYNC 167 min 144 passes .862 passes/min
> SYNC 1065 min 919 passes .863 passes/min
> MIXED 221 min 196 passes .887 passes/min
> 
> 
> Conclusions:
> 
> The only protocol which meets the .05% accuracy requirement for ntp
> tracking under the loads
> above is the SYNC protocol. The worst case accuracies for SYNC, MIXED,
> and ASYNC
> are .022%, .12%, and .14%, respectively.
> 
> We could reduce the cost of the SYNC method by only scheduling the extra
> wakeups if a certain number
> of ticks are missed.
> 
> Regards,
> Dave
> 
> Keir Fraser wrote:
> 
>> On 9/11/07 19:22, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>> 
>>  
>> 
>>> Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
>>> I ran another ASYNC test. I think there may have been something
>>> wrong with the code I used a couple of days ago for ASYNC. It may have been
>>> missing the immediate delivery of interrupt after context switch in.
>>> 
>>> My results indicate that either SYNC or ASYNC give acceptable accuracy,
>>> each running consistently around or under .01%. MIXED has a fairly high
>>> error of
>>> greater than .03%. Probably too close to .05% ntp threshold for comfort.
>>> I don't have an overnight run with SYNC. I plan to leave SYNC running
>>> over the weekend. If you'd rather I can leave MIXED running instead.
>>> 
>>> It may be too early to pick the protocol and I can run more overnight tests
>>> next week.
>>>    
>>> 
>> 
>> I'm a bit worried about any unwanted side effects of the SYNC+run_timer
>> approach -- e.g., whether timer wakeups will cause higher system-wide CPU
>> contention. I find it easier to think through the implications of ASYNC. I'm
>> surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
>> delivers more timer interrupts than the other approaches, and each interrupt
>> event causes a small accumulated error?
>> 
>> Overall I would consider MIXED and ASYNC as favourites and if the latter is
>> actually more accurate then I can simply revert the changeset that
>> implemented MIXED.
>> 
>> Perhaps rather than running more of the same workloads you could try idle
>> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
>> don't have any data on workloads that aren't CPU bound, so that's really an
>> obvious place to put any further effort imo.
>> 
>> -- Keir
>> 
>> 
>>  
>> 
> 

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Anyone make measurements on the final patch?

I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no load.  This was xen-unstable tip today with no options specified.  32-bit was about 0.01%.

I think I missed something... how do I run the various accounting choices and which ones are known to be appropriate for which kernels?

Thanks,
Dan

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
> Sent: Thursday, December 06, 2007 4:57 AM
> To: Dave Winchell
> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
> Yunhong
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Please take a look at xen-unstable changeset 16545.
> 
>  -- Keir
> 
> On 26/11/07 20:57, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
> 
> > Keir,
> >
> > The accuracy data I've collected for i/o loads for the
> > various time protocols follows. In addition, the data
> > for cpu loads is shown.
> >
> > The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> > Two guests, red hat and sles 64 bit, 8 vcpu each.
> > The cpu load is usex -e36 on each guest.
> > (usex is available at http://people.redhat.com/anderson/usex.)
> > i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >
> > The loads labeled i/o-32 are 32 instances of dd.
> > Also, these are run on 4 cpu AMD box.
> > In addition, there is an idle rh-32bit guest.
> > All three guests are 8vcpu.
> >
> > The loads labeled i/o-4/32 are the same as i/o-32
> > except that the redhat-64 guest has 4 instances of dd.
> >
> > Date Duration Protocol sles, rhat error load
> >
> > 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
> > 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
> >
> > 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> > 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> > 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >
> > 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> > 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
> >
> >
> > 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
> > 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
> >
> > 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
> > 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >
> > 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
> > 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
> >
> >
> >
> > 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> > 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
> > 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >
> > 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
> > 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
> >
> >
> > Overhead measurements:
> >
> > Progress in terms of number of passes through a fixed 
> system workload
> > on an 8 vcpu red hat with an 8 vcpu sles idle.
> > The workload was usex -b48.
> >
> >
> > ASYNC 167 min 145 passes .868 passes/min
> > SYNC 167 min 144 passes .862 passes/min
> > SYNC 1065 min 919 passes .863 passes/min
> > MIXED 221 min 196 passes .887 passes/min
> >
> >
> > Conclusions:
> >
> > The only protocol which meets the .05% accuracy requirement for ntp
> > tracking under the loads
> > above is the SYNC protocol. The worst case accuracies for 
> SYNC, MIXED,
> > and ASYNC
> > are .022%, .12%, and .14%, respectively.
> >
> > We could reduce the cost of the SYNC method by only 
> scheduling the extra
> > wakeups if a certain number
> > of ticks are missed.
> >
> > Regards,
> > Dave
> >
> > Keir Fraser wrote:
> >
> >> On 9/11/07 19:22, "Dave Winchell" 
> <dwinchell@virtualiron.com> wrote:
> >>
> >>
> >>
> >>> Since I had a high error (~.03%) for the ASYNC method a 
> couple of days ago,
> >>> I ran another ASYNC test. I think there may have been something
> >>> wrong with the code I used a couple of days ago for 
> ASYNC. It may have been
> >>> missing the immediate delivery of interrupt after context 
> switch in.
> >>>
> >>> My results indicate that either SYNC or ASYNC give 
> acceptable accuracy,
> >>> each running consistently around or under .01%. MIXED has 
> a fairly high
> >>> error of
> >>> greater than .03%. Probably too close to .05% ntp 
> threshold for comfort.
> >>> I don't have an overnight run with SYNC. I plan to leave 
> SYNC running
> >>> over the weekend. If you'd rather I can leave MIXED 
> running instead.
> >>>
> >>> It may be too early to pick the protocol and I can run 
> more overnight tests
> >>> next week.
> >>>
> >>>
> >>
> >> I'm a bit worried about any unwanted side effects of the 
> SYNC+run_timer
> >> approach -- e.g., whether timer wakeups will cause higher 
> system-wide CPU
> >> contention. I find it easier to think through the 
> implications of ASYNC. I'm
> >> surprised that MIXED loses time, and is less accurate than 
> ASYNC. Perhaps it
> >> delivers more timer interrupts than the other approaches, 
> and each interrupt
> >> event causes a small accumulated error?
> >>
> >> Overall I would consider MIXED and ASYNC as favourites and 
> if the latter is
> >> actually more accurate then I can simply revert the changeset that
> >> implemented MIXED.
> >>
> >> Perhaps rather than running more of the same workloads you 
> could try idle
> >> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads 
> to /dev/null)? We
> >> don't have any data on workloads that aren't CPU bound, so 
> that's really an
> >> obvious place to put any further effort imo.
> >>
> >> -- Keir
> >>
> >>
> >>
> >>
> >
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Dan,

I did some testing with the constant tsc offset SYNC method (now called 
no_missed_ticks_pending)
and found the error to be very high, much larger than 1 %, as I recall.
I have not had a chance to submit a correction. I will try to do it later
this week or the first week in January. My version of constant tsc 
offset SYNC method
produces .02 % error, so I just need to port that into the current code.

The error you got for both of those kernels is what I would expect
for the default mode, delay_for_missed_ticks.

I'll let Keir answer on how to set the time mode.

Regards,
Dave

Dan Magenheimer wrote:

>Anyone make measurements on the final patch?
>
>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no load.  This was xen-unstable tip today with no options specified.  32-bit was about 0.01%.
>
>I think I missed something... how do I run the various accounting choices and which ones are known to be appropriate for which kernels?
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
>>Sent: Thursday, December 06, 2007 4:57 AM
>>To: Dave Winchell
>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>Yunhong
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Please take a look at xen-unstable changeset 16545.
>>
>> -- Keir
>>
>>On 26/11/07 20:57, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>
>>    
>>
>>>Keir,
>>>
>>>The accuracy data I've collected for i/o loads for the
>>>various time protocols follows. In addition, the data
>>>for cpu loads is shown.
>>>
>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>The cpu load is usex -e36 on each guest.
>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>
>>>The loads labeled i/o-32 are 32 instances of dd.
>>>Also, these are run on 4 cpu AMD box.
>>>In addition, there is an idle rh-32bit guest.
>>>All three guests are 8vcpu.
>>>
>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>except that the redhat-64 guest has 4 instances of dd.
>>>
>>>Date Duration Protocol sles, rhat error load
>>>
>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>
>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>
>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>
>>>
>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>
>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>
>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>
>>>
>>>
>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>
>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>
>>>
>>>Overhead measurements:
>>>
>>>Progress in terms of number of passes through a fixed 
>>>      
>>>
>>system workload
>>    
>>
>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>The workload was usex -b48.
>>>
>>>
>>>ASYNC 167 min 145 passes .868 passes/min
>>>SYNC 167 min 144 passes .862 passes/min
>>>SYNC 1065 min 919 passes .863 passes/min
>>>MIXED 221 min 196 passes .887 passes/min
>>>
>>>
>>>Conclusions:
>>>
>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>tracking under the loads
>>>above is the SYNC protocol. The worst case accuracies for 
>>>      
>>>
>>SYNC, MIXED,
>>    
>>
>>>and ASYNC
>>>are .022%, .12%, and .14%, respectively.
>>>
>>>We could reduce the cost of the SYNC method by only 
>>>      
>>>
>>scheduling the extra
>>    
>>
>>>wakeups if a certain number
>>>of ticks are missed.
>>>
>>>Regards,
>>>Dave
>>>
>>>Keir Fraser wrote:
>>>
>>>      
>>>
>>>>On 9/11/07 19:22, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Since I had a high error (~.03%) for the ASYNC method a 
>>>>>          
>>>>>
>>couple of days ago,
>>    
>>
>>>>>I ran another ASYNC test. I think there may have been something
>>>>>wrong with the code I used a couple of days ago for 
>>>>>          
>>>>>
>>ASYNC. It may have been
>>    
>>
>>>>>missing the immediate delivery of interrupt after context 
>>>>>          
>>>>>
>>switch in.
>>    
>>
>>>>>My results indicate that either SYNC or ASYNC give 
>>>>>          
>>>>>
>>acceptable accuracy,
>>    
>>
>>>>>each running consistently around or under .01%. MIXED has 
>>>>>          
>>>>>
>>a fairly high
>>    
>>
>>>>>error of
>>>>>greater than .03%. Probably too close to .05% ntp 
>>>>>          
>>>>>
>>threshold for comfort.
>>    
>>
>>>>>I don't have an overnight run with SYNC. I plan to leave 
>>>>>          
>>>>>
>>SYNC running
>>    
>>
>>>>>over the weekend. If you'd rather I can leave MIXED 
>>>>>          
>>>>>
>>running instead.
>>    
>>
>>>>>It may be too early to pick the protocol and I can run 
>>>>>          
>>>>>
>>more overnight tests
>>    
>>
>>>>>next week.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>I'm a bit worried about any unwanted side effects of the 
>>>>        
>>>>
>>SYNC+run_timer
>>    
>>
>>>>approach -- e.g., whether timer wakeups will cause higher 
>>>>        
>>>>
>>system-wide CPU
>>    
>>
>>>>contention. I find it easier to think through the 
>>>>        
>>>>
>>implications of ASYNC. I'm
>>    
>>
>>>>surprised that MIXED loses time, and is less accurate than 
>>>>        
>>>>
>>ASYNC. Perhaps it
>>    
>>
>>>>delivers more timer interrupts than the other approaches, 
>>>>        
>>>>
>>and each interrupt
>>    
>>
>>>>event causes a small accumulated error?
>>>>
>>>>Overall I would consider MIXED and ASYNC as favourites and 
>>>>        
>>>>
>>if the latter is
>>    
>>
>>>>actually more accurate then I can simply revert the changeset that
>>>>implemented MIXED.
>>>>
>>>>Perhaps rather than running more of the same workloads you 
>>>>        
>>>>
>>could try idle
>>    
>>
>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads 
>>>>        
>>>>
>>to /dev/null)? We
>>    
>>
>>>>don't have any data on workloads that aren't CPU bound, so 
>>>>        
>>>>
>>that's really an
>>    
>>
>>>>obvious place to put any further effort imo.
>>>>
>>>>-- Keir
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>>    
>>
>
>  
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Hi Dave --

Did you get your correction ported?  If so, it would be nice to see this get into 3.1.3.

Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending)
on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far
is 0.012%.  But I haven't tried any exotic loads, just LTP.

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Wednesday, December 19, 2007 12:33 PM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
> Eddie; Jiang, Yunhong; Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan,
> 
> I did some testing with the constant tsc offset SYNC method 
> (now called
> no_missed_ticks_pending)
> and found the error to be very high, much larger than 1 %, as 
> I recall.
> I have not had a chance to submit a correction. I will try to 
> do it later
> this week or the first week in January. My version of constant tsc
> offset SYNC method
> produces .02 % error, so I just need to port that into the 
> current code.
> 
> The error you got for both of those kernels is what I would expect
> for the default mode, delay_for_missed_ticks.
> 
> I'll let Keir answer on how to set the time mode.
> 
> Regards,
> Dave
> 
> Dan Magenheimer wrote:
> 
> >Anyone make measurements on the final patch?
> >
> >I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
> about 0.2% with no load.  This was xen-unstable tip today 
> with no options specified.  32-bit was about 0.01%.
> >
> >I think I missed something... how do I run the various 
> accounting choices and which ones are known to be appropriate 
> for which kernels?
> >
> >Thanks,
> >Dan
> >
> >
> >
> >>-----Original Message-----
> >>From: xen-devel-bounces@lists.xensource.com
> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
> Keir Fraser
> >>Sent: Thursday, December 06, 2007 4:57 AM
> >>To: Dave Winchell
> >>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
> >>Yunhong
> >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>disables pending
> >>missed ticks
> >>
> >>
> >>Please take a look at xen-unstable changeset 16545.
> >>
> >> -- Keir
> >>
> >>On 26/11/07 20:57, "Dave Winchell" 
> <dwinchell@virtualiron.com> wrote:
> >>
> >>
> >>
> >>>Keir,
> >>>
> >>>The accuracy data I've collected for i/o loads for the
> >>>various time protocols follows. In addition, the data
> >>>for cpu loads is shown.
> >>>
> >>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> >>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >>>The cpu load is usex -e36 on each guest.
> >>>(usex is available at http://people.redhat.com/anderson/usex.)
> >>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >>>
> >>>The loads labeled i/o-32 are 32 instances of dd.
> >>>Also, these are run on 4 cpu AMD box.
> >>>In addition, there is an idle rh-32bit guest.
> >>>All three guests are 8vcpu.
> >>>
> >>>The loads labeled i/o-4/32 are the same as i/o-32
> >>>except that the redhat-64 guest has 4 instances of dd.
> >>>
> >>>Date Duration Protocol sles, rhat error load
> >>>
> >>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
> >>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
> >>>
> >>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> >>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> >>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >>>
> >>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> >>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
> >>>
> >>>
> >>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
> >>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
> >>>
> >>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
> >>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >>>
> >>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
> >>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
> >>>
> >>>
> >>>
> >>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> >>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
> >>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >>>
> >>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
> >>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
> >>>
> >>>
> >>>Overhead measurements:
> >>>
> >>>Progress in terms of number of passes through a fixed
> >>>
> >>>
> >>system workload
> >>
> >>
> >>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>The workload was usex -b48.
> >>>
> >>>
> >>>ASYNC 167 min 145 passes .868 passes/min
> >>>SYNC 167 min 144 passes .862 passes/min
> >>>SYNC 1065 min 919 passes .863 passes/min
> >>>MIXED 221 min 196 passes .887 passes/min
> >>>
> >>>
> >>>Conclusions:
> >>>
> >>>The only protocol which meets the .05% accuracy requirement for ntp
> >>>tracking under the loads
> >>>above is the SYNC protocol. The worst case accuracies for
> >>>
> >>>
> >>SYNC, MIXED,
> >>
> >>
> >>>and ASYNC
> >>>are .022%, .12%, and .14%, respectively.
> >>>
> >>>We could reduce the cost of the SYNC method by only
> >>>
> >>>
> >>scheduling the extra
> >>
> >>
> >>>wakeups if a certain number
> >>>of ticks are missed.
> >>>
> >>>Regards,
> >>>Dave
> >>>
> >>>Keir Fraser wrote:
> >>>
> >>>
> >>>
> >>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>
> >>>>
> >><dwinchell@virtualiron.com> wrote:
> >>
> >>
> >>>>
> >>>>
> >>>>
> >>>>>Since I had a high error (~.03%) for the ASYNC method a
> >>>>>
> >>>>>
> >>couple of days ago,
> >>
> >>
> >>>>>I ran another ASYNC test. I think there may have been something
> >>>>>wrong with the code I used a couple of days ago for
> >>>>>
> >>>>>
> >>ASYNC. It may have been
> >>
> >>
> >>>>>missing the immediate delivery of interrupt after context
> >>>>>
> >>>>>
> >>switch in.
> >>
> >>
> >>>>>My results indicate that either SYNC or ASYNC give
> >>>>>
> >>>>>
> >>acceptable accuracy,
> >>
> >>
> >>>>>each running consistently around or under .01%. MIXED has
> >>>>>
> >>>>>
> >>a fairly high
> >>
> >>
> >>>>>error of
> >>>>>greater than .03%. Probably too close to .05% ntp
> >>>>>
> >>>>>
> >>threshold for comfort.
> >>
> >>
> >>>>>I don't have an overnight run with SYNC. I plan to leave
> >>>>>
> >>>>>
> >>SYNC running
> >>
> >>
> >>>>>over the weekend. If you'd rather I can leave MIXED
> >>>>>
> >>>>>
> >>running instead.
> >>
> >>
> >>>>>It may be too early to pick the protocol and I can run
> >>>>>
> >>>>>
> >>more overnight tests
> >>
> >>
> >>>>>next week.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>I'm a bit worried about any unwanted side effects of the
> >>>>
> >>>>
> >>SYNC+run_timer
> >>
> >>
> >>>>approach -- e.g., whether timer wakeups will cause higher
> >>>>
> >>>>
> >>system-wide CPU
> >>
> >>
> >>>>contention. I find it easier to think through the
> >>>>
> >>>>
> >>implications of ASYNC. I'm
> >>
> >>
> >>>>surprised that MIXED loses time, and is less accurate than
> >>>>
> >>>>
> >>ASYNC. Perhaps it
> >>
> >>
> >>>>delivers more timer interrupts than the other approaches,
> >>>>
> >>>>
> >>and each interrupt
> >>
> >>
> >>>>event causes a small accumulated error?
> >>>>
> >>>>Overall I would consider MIXED and ASYNC as favourites and
> >>>>
> >>>>
> >>if the latter is
> >>
> >>
> >>>>actually more accurate then I can simply revert the changeset that
> >>>>implemented MIXED.
> >>>>
> >>>>Perhaps rather than running more of the same workloads you
> >>>>
> >>>>
> >>could try idle
> >>
> >>
> >>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
> >>>>
> >>>>
> >>to /dev/null)? We
> >>
> >>
> >>>>don't have any data on workloads that aren't CPU bound, so
> >>>>
> >>>>
> >>that's really an
> >>
> >>
> >>>>obvious place to put any further effort imo.
> >>>>
> >>>>-- Keir
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xensource.com
> >>http://lists.xensource.com/xen-devel
> >>
> >>
> >>
> >>
> >
> >
> >
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

I had to put that port aside with other pressing issues here.
I will try to get the port done in the next couple of days.
Is that soon enough for 3.1.3?

To see the error with timer_mode=2(=SYNC=no missed ticks pending)
you probably need to over commit the physical processors with
virtual processors, say 2:1, and then load up all the virtual processors.

Regards,
Dave

Dan Magenheimer wrote:

>Hi Dave --
>
>Did you get your correction ported?  If so, it would be nice to see this get into 3.1.3.
>
>Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending)
>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far
>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, December 19, 2007 12:33 PM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>Eddie; Jiang, Yunhong; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I did some testing with the constant tsc offset SYNC method 
>>(now called
>>no_missed_ticks_pending)
>>and found the error to be very high, much larger than 1 %, as 
>>I recall.
>>I have not had a chance to submit a correction. I will try to 
>>do it later
>>this week or the first week in January. My version of constant tsc
>>offset SYNC method
>>produces .02 % error, so I just need to port that into the 
>>current code.
>>
>>The error you got for both of those kernels is what I would expect
>>for the default mode, delay_for_missed_ticks.
>>
>>I'll let Keir answer on how to set the time mode.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Anyone make measurements on the final patch?
>>>
>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
>>>      
>>>
>>about 0.2% with no load.  This was xen-unstable tip today 
>>with no options specified.  32-bit was about 0.01%.
>>    
>>
>>>I think I missed something... how do I run the various 
>>>      
>>>
>>accounting choices and which ones are known to be appropriate 
>>for which kernels?
>>    
>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
>>>>        
>>>>
>>Keir Fraser
>>    
>>
>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>To: Dave Winchell
>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>>>Yunhong
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Please take a look at xen-unstable changeset 16545.
>>>>
>>>>-- Keir
>>>>
>>>>On 26/11/07 20:57, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Keir,
>>>>>
>>>>>The accuracy data I've collected for i/o loads for the
>>>>>various time protocols follows. In addition, the data
>>>>>for cpu loads is shown.
>>>>>
>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>The cpu load is usex -e36 on each guest.
>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>
>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>Also, these are run on 4 cpu AMD box.
>>>>>In addition, there is an idle rh-32bit guest.
>>>>>All three guests are 8vcpu.
>>>>>
>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>
>>>>>Date Duration Protocol sles, rhat error load
>>>>>
>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>>>
>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>
>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>>>
>>>>>
>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>>>
>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>
>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>>>
>>>>>
>>>>>
>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>
>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>>>
>>>>>
>>>>>Overhead measurements:
>>>>>
>>>>>Progress in terms of number of passes through a fixed
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>system workload
>>>>
>>>>
>>>>        
>>>>
>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>The workload was usex -b48.
>>>>>
>>>>>
>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>
>>>>>
>>>>>Conclusions:
>>>>>
>>>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>>>tracking under the loads
>>>>>above is the SYNC protocol. The worst case accuracies for
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>SYNC, MIXED,
>>>>
>>>>
>>>>        
>>>>
>>>>>and ASYNC
>>>>>are .022%, .12%, and .14%, respectively.
>>>>>
>>>>>We could reduce the cost of the SYNC method by only
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>scheduling the extra
>>>>
>>>>
>>>>        
>>>>
>>>>>wakeups if a certain number
>>>>>of ticks are missed.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>Keir Fraser wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>
>>>>
>>>>        
>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>couple of days ago,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I ran another ASYNC test. I think there may have been something
>>>>>>>wrong with the code I used a couple of days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ASYNC. It may have been
>>>>
>>>>
>>>>        
>>>>
>>>>>>>missing the immediate delivery of interrupt after context
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>switch in.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>acceptable accuracy,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>each running consistently around or under .01%. MIXED has
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>a fairly high
>>>>
>>>>
>>>>        
>>>>
>>>>>>>error of
>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>threshold for comfort.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I don't have an overnight run with SYNC. I plan to leave
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>SYNC running
>>>>
>>>>
>>>>        
>>>>
>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>running instead.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>It may be too early to pick the protocol and I can run
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>more overnight tests
>>>>
>>>>
>>>>        
>>>>
>>>>>>>next week.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>I'm a bit worried about any unwanted side effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>SYNC+run_timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>approach -- e.g., whether timer wakeups will cause higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>system-wide CPU
>>>>
>>>>
>>>>        
>>>>
>>>>>>contention. I find it easier to think through the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>implications of ASYNC. I'm
>>>>
>>>>
>>>>        
>>>>
>>>>>>surprised that MIXED loses time, and is less accurate than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>ASYNC. Perhaps it
>>>>
>>>>
>>>>        
>>>>
>>>>>>delivers more timer interrupts than the other approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>and each interrupt
>>>>
>>>>
>>>>        
>>>>
>>>>>>event causes a small accumulated error?
>>>>>>
>>>>>>Overall I would consider MIXED and ASYNC as favourites and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>if the latter is
>>>>
>>>>
>>>>        
>>>>
>>>>>>actually more accurate then I can simply revert the changeset that
>>>>>>implemented MIXED.
>>>>>>
>>>>>>Perhaps rather than running more of the same workloads you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>could try idle
>>>>
>>>>
>>>>        
>>>>
>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>to /dev/null)? We
>>>>
>>>>
>>>>        
>>>>
>>>>>>don't have any data on workloads that aren't CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>that's really an
>>>>
>>>>
>>>>        
>>>>
>>>>>>obvious place to put any further effort imo.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 9851 bytes --]

Hi Dan and Keir,

Attached is a patch that fixes some issues with the SYNC policy 
(no_missed_ticks_pending).
I have not tried to make the change the minimal one, but, rather, just 
ported into
the new code what I know to work well. The error for 
no_missed_ticks_pending goes from
over 3% to .03% with this change according to my testing.

Regards,
Dave

Dan Magenheimer wrote:

>Hi Dave --
>
>Did you get your correction ported?  If so, it would be nice to see this get into 3.1.3.
>
>Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending)
>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far
>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, December 19, 2007 12:33 PM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>Eddie; Jiang, Yunhong; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I did some testing with the constant tsc offset SYNC method 
>>(now called
>>no_missed_ticks_pending)
>>and found the error to be very high, much larger than 1 %, as 
>>I recall.
>>I have not had a chance to submit a correction. I will try to 
>>do it later
>>this week or the first week in January. My version of constant tsc
>>offset SYNC method
>>produces .02 % error, so I just need to port that into the 
>>current code.
>>
>>The error you got for both of those kernels is what I would expect
>>for the default mode, delay_for_missed_ticks.
>>
>>I'll let Keir answer on how to set the time mode.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Anyone make measurements on the final patch?
>>>
>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
>>>      
>>>
>>about 0.2% with no load.  This was xen-unstable tip today 
>>with no options specified.  32-bit was about 0.01%.
>>    
>>
>>>I think I missed something... how do I run the various 
>>>      
>>>
>>accounting choices and which ones are known to be appropriate 
>>for which kernels?
>>    
>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
>>>>        
>>>>
>>Keir Fraser
>>    
>>
>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>To: Dave Winchell
>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>>>Yunhong
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Please take a look at xen-unstable changeset 16545.
>>>>
>>>>-- Keir
>>>>
>>>>On 26/11/07 20:57, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Keir,
>>>>>
>>>>>The accuracy data I've collected for i/o loads for the
>>>>>various time protocols follows. In addition, the data
>>>>>for cpu loads is shown.
>>>>>
>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>The cpu load is usex -e36 on each guest.
>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>
>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>Also, these are run on 4 cpu AMD box.
>>>>>In addition, there is an idle rh-32bit guest.
>>>>>All three guests are 8vcpu.
>>>>>
>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>
>>>>>Date Duration Protocol sles, rhat error load
>>>>>
>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>>>
>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>
>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>>>
>>>>>
>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>>>
>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>
>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>>>
>>>>>
>>>>>
>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>
>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>>>
>>>>>
>>>>>Overhead measurements:
>>>>>
>>>>>Progress in terms of number of passes through a fixed
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>system workload
>>>>
>>>>
>>>>        
>>>>
>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>The workload was usex -b48.
>>>>>
>>>>>
>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>
>>>>>
>>>>>Conclusions:
>>>>>
>>>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>>>tracking under the loads
>>>>>above is the SYNC protocol. The worst case accuracies for
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>SYNC, MIXED,
>>>>
>>>>
>>>>        
>>>>
>>>>>and ASYNC
>>>>>are .022%, .12%, and .14%, respectively.
>>>>>
>>>>>We could reduce the cost of the SYNC method by only
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>scheduling the extra
>>>>
>>>>
>>>>        
>>>>
>>>>>wakeups if a certain number
>>>>>of ticks are missed.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>Keir Fraser wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>
>>>>
>>>>        
>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>couple of days ago,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I ran another ASYNC test. I think there may have been something
>>>>>>>wrong with the code I used a couple of days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ASYNC. It may have been
>>>>
>>>>
>>>>        
>>>>
>>>>>>>missing the immediate delivery of interrupt after context
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>switch in.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>acceptable accuracy,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>each running consistently around or under .01%. MIXED has
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>a fairly high
>>>>
>>>>
>>>>        
>>>>
>>>>>>>error of
>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>threshold for comfort.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I don't have an overnight run with SYNC. I plan to leave
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>SYNC running
>>>>
>>>>
>>>>        
>>>>
>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>running instead.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>It may be too early to pick the protocol and I can run
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>more overnight tests
>>>>
>>>>
>>>>        
>>>>
>>>>>>>next week.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>I'm a bit worried about any unwanted side effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>SYNC+run_timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>approach -- e.g., whether timer wakeups will cause higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>system-wide CPU
>>>>
>>>>
>>>>        
>>>>
>>>>>>contention. I find it easier to think through the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>implications of ASYNC. I'm
>>>>
>>>>
>>>>        
>>>>
>>>>>>surprised that MIXED loses time, and is less accurate than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>ASYNC. Perhaps it
>>>>
>>>>
>>>>        
>>>>
>>>>>>delivers more timer interrupts than the other approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>and each interrupt
>>>>
>>>>
>>>>        
>>>>
>>>>>>event causes a small accumulated error?
>>>>>>
>>>>>>Overall I would consider MIXED and ASYNC as favourites and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>if the latter is
>>>>
>>>>
>>>>        
>>>>
>>>>>>actually more accurate then I can simply revert the changeset that
>>>>>>implemented MIXED.
>>>>>>
>>>>>>Perhaps rather than running more of the same workloads you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>could try idle
>>>>
>>>>
>>>>        
>>>>
>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>to /dev/null)? We
>>>>
>>>>
>>>>        
>>>>
>>>>>>don't have any data on workloads that aren't CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>that's really an
>>>>
>>>>
>>>>        
>>>>
>>>>>>obvious place to put any further effort imo.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>


[-- Attachment #2: p.1.04.sync.time --]
[-- Type: text/plain, Size: 1445 bytes --]

diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Thu Dec 06 15:36:07 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Fri Jan 04 17:58:16 2008 -0500
@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
 
     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
-        pt->do_not_freeze = !pt->pending_intr_nr;
+        pt->do_not_freeze = 1;
     else
         pt->pending_intr_nr += missed_ticks;
     pt->scheduled += missed_ticks * pt->period;
@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
 
     pt_lock(pt);
 
-    pt->pending_intr_nr++;
+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
+        pt->pending_intr_nr = 1;
+	pt->do_not_freeze = 0;
+    }
+    else
+	pt->pending_intr_nr++;
 
     if ( !pt->one_shot )
     {
@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
         return;
     }
 
-    pt->do_not_freeze = 0;
-
     if ( pt->one_shot )
     {
         pt->enabled = 0;
@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
             pt->last_plt_gtime = hvm_get_guest_time(v);
             pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */
         }
+	else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
+	    pt->pending_intr_nr--;
+	    pt->last_plt_gtime = hvm_get_guest_time(v);
+	}
         else
         {
             pt->last_plt_gtime += pt->period_cycles;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

Applied as c/s 16690, although the checked-in patch is smaller. I think the
only important fix is to pt_intr_post() and the only bit of the patch I
totally omitted was the change to pt_process_missed_ticks(). I don't think
that change can be important, but let's see what happens to the error
percentage...

 -- Keir

On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Hi Dan and Keir,
> 
> Attached is a patch that fixes some issues with the SYNC policy
> (no_missed_ticks_pending).
> I have not tried to make the change the minimal one, but, rather, just
> ported into
> the new code what I know to work well. The error for
> no_missed_ticks_pending goes from
> over 3% to .03% with this change according to my testing.
> 
> Regards,
> Dave
> 
> Dan Magenheimer wrote:
> 
>> Hi Dave --
>> 
>> Did you get your correction ported?  If so, it would be nice to see this get
>> into 3.1.3.
>> 
>> Note that I just did some very limited testing with timer_mode=2(=SYNC=no
>> missed ticks pending)
>> on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've
>> seen so far
>> is 0.012%.  But I haven't tried any exotic loads, just LTP.
>> 
>> Thanks,
>> Dan
>> 
>>  
>> 
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Wednesday, December 19, 2007 12:33 PM
>>> To: dan.magenheimer@oracle.com
>>> Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>> Eddie; Jiang, Yunhong; Dave Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>> disables pending
>>> missed ticks
>>> 
>>> 
>>> Dan,
>>> 
>>> I did some testing with the constant tsc offset SYNC method
>>> (now called
>>> no_missed_ticks_pending)
>>> and found the error to be very high, much larger than 1 %, as
>>> I recall.
>>> I have not had a chance to submit a correction. I will try to
>>> do it later
>>> this week or the first week in January. My version of constant tsc
>>> offset SYNC method
>>> produces .02 % error, so I just need to port that into the
>>> current code.
>>> 
>>> The error you got for both of those kernels is what I would expect
>>> for the default mode, delay_for_missed_ticks.
>>> 
>>> I'll let Keir answer on how to set the time mode.
>>> 
>>> Regards,
>>> Dave
>>> 
>>> Dan Magenheimer wrote:
>>> 
>>>    
>>> 
>>>> Anyone make measurements on the final patch?
>>>> 
>>>> I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>      
>>>> 
>>> about 0.2% with no load.  This was xen-unstable tip today
>>> with no options specified.  32-bit was about 0.01%.
>>>    
>>> 
>>>> I think I missed something... how do I run the various
>>>>      
>>>> 
>>> accounting choices and which ones are known to be appropriate
>>> for which kernels?
>>>    
>>> 
>>>> Thanks,
>>>> Dan
>>>> 
>>>> 
>>>> 
>>>>      
>>>> 
>>>>> -----Original Message-----
>>>>> From: xen-devel-bounces@lists.xensource.com
>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>        
>>>>> 
>>> Keir Fraser
>>>    
>>> 
>>>>> Sent: Thursday, December 06, 2007 4:57 AM
>>>>> To: Dave Winchell
>>>>> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>>>> Yunhong
>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>> disables pending
>>>>> missed ticks
>>>>> 
>>>>> 
>>>>> Please take a look at xen-unstable changeset 16545.
>>>>> 
>>>>> -- Keir
>>>>> 
>>>>> On 26/11/07 20:57, "Dave Winchell"
>>>>>        
>>>>> 
>>> <dwinchell@virtualiron.com> wrote:
>>>    
>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> Keir,
>>>>>> 
>>>>>> The accuracy data I've collected for i/o loads for the
>>>>>> various time protocols follows. In addition, the data
>>>>>> for cpu loads is shown.
>>>>>> 
>>>>>> The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>>>> Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>> The cpu load is usex -e36 on each guest.
>>>>>> (usex is available at http://people.redhat.com/anderson/usex.)
>>>>>> i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>> 
>>>>>> The loads labeled i/o-32 are 32 instances of dd.
>>>>>> Also, these are run on 4 cpu AMD box.
>>>>>> In addition, there is an idle rh-32bit guest.
>>>>>> All three guests are 8vcpu.
>>>>>> 
>>>>>> The loads labeled i/o-4/32 are the same as i/o-32
>>>>>> except that the redhat-64 guest has 4 instances of dd.
>>>>>> 
>>>>>> Date Duration Protocol sles, rhat error load
>>>>>> 
>>>>>> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>>>> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>>>> 
>>>>>> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>>>> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>>>> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>> 
>>>>>> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>>>> 
>>>>>> 
>>>>>> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>>>> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>>>> 
>>>>>> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>>>> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>> 
>>>>>> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>>>> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>>>> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>> 
>>>>>> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>>>> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>>>> 
>>>>>> 
>>>>>> Overhead measurements:
>>>>>> 
>>>>>> Progress in terms of number of passes through a fixed
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>> system workload
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>> The workload was usex -b48.
>>>>>> 
>>>>>> 
>>>>>> ASYNC 167 min 145 passes .868 passes/min
>>>>>> SYNC 167 min 144 passes .862 passes/min
>>>>>> SYNC 1065 min 919 passes .863 passes/min
>>>>>> MIXED 221 min 196 passes .887 passes/min
>>>>>> 
>>>>>> 
>>>>>> Conclusions:
>>>>>> 
>>>>>> The only protocol which meets the .05% accuracy requirement for ntp
>>>>>> tracking under the loads
>>>>>> above is the SYNC protocol. The worst case accuracies for
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>> SYNC, MIXED,
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> and ASYNC
>>>>>> are .022%, .12%, and .14%, respectively.
>>>>>> 
>>>>>> We could reduce the cost of the SYNC method by only
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>> scheduling the extra
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> wakeups if a certain number
>>>>>> of ticks are missed.
>>>>>> 
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>> Keir Fraser wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>>>> On 9/11/07 19:22, "Dave Winchell"
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> <dwinchell@virtualiron.com> wrote:
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>>>>> Since I had a high error (~.03%) for the ASYNC method a
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> couple of days ago,
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> I ran another ASYNC test. I think there may have been something
>>>>>>>> wrong with the code I used a couple of days ago for
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> ASYNC. It may have been
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> missing the immediate delivery of interrupt after context
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> switch in.
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> My results indicate that either SYNC or ASYNC give
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> acceptable accuracy,
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> each running consistently around or under .01%. MIXED has
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> a fairly high
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> error of
>>>>>>>> greater than .03%. Probably too close to .05% ntp
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> threshold for comfort.
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> I don't have an overnight run with SYNC. I plan to leave
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> SYNC running
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> over the weekend. If you'd rather I can leave MIXED
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> running instead.
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> It may be too early to pick the protocol and I can run
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> more overnight tests
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> next week.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>>>> I'm a bit worried about any unwanted side effects of the
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> SYNC+run_timer
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> approach -- e.g., whether timer wakeups will cause higher
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> system-wide CPU
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> contention. I find it easier to think through the
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> implications of ASYNC. I'm
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> surprised that MIXED loses time, and is less accurate than
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> ASYNC. Perhaps it
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> delivers more timer interrupts than the other approaches,
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> and each interrupt
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> event causes a small accumulated error?
>>>>>>> 
>>>>>>> Overall I would consider MIXED and ASYNC as favourites and
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> if the latter is
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> actually more accurate then I can simply revert the changeset that
>>>>>>> implemented MIXED.
>>>>>>> 
>>>>>>> Perhaps rather than running more of the same workloads you
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> could try idle
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> to /dev/null)? We
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> don't have any data on workloads that aren't CPU bound, so
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> that's really an
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> obvious place to put any further effort imo.
>>>>>>> 
>>>>>>> -- Keir
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>> 
>>>>      
>>>> 
>>>    
>>> 
>> 
>>  
>> 
> 
> diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>  
>      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>      if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> -        pt->do_not_freeze = !pt->pending_intr_nr;
> +        pt->do_not_freeze = 1;
>      else
>          pt->pending_intr_nr += missed_ticks;
>      pt->scheduled += missed_ticks * pt->period;
> @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>  
>      pt_lock(pt);
>  
> -    pt->pending_intr_nr++;
> +    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
> +        pt->pending_intr_nr = 1;
> + pt->do_not_freeze = 0;
> +    }
> +    else
> + pt->pending_intr_nr++;
>  
>      if ( !pt->one_shot )
>      {
> @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>          return;
>      }
>  
> -    pt->do_not_freeze = 0;
> -
>      if ( pt->one_shot )
>      {
>          pt->enabled = 0;
> @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>              pt->last_plt_gtime = hvm_get_guest_time(v);
>              pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */
>          }
> + else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> +     pt->pending_intr_nr--;
> +     pt->last_plt_gtime = hvm_get_guest_time(v);
> + }
>          else
>          {
>              pt->last_plt_gtime += pt->period_cycles;

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Keir,

The latest change, c/s 16690, looks fine.
I agree that the code in c/s 16690 is equivalent to
the code I submitted. Also, your version is more
concise.

The error tests confirm the equivalence. With overnight cpu loads,
the checked in version was accurate to +.048% for sles
and +.038% for red hat. My version was +.046% and +.032% in a 2 hour test.
I don't think the difference is significant.

i/o loads produced errors of +.01%.

Thanks for all your efforts on this issue.

Regards,
Dave



Keir Fraser wrote:

>Applied as c/s 16690, although the checked-in patch is smaller. I think the
>only important fix is to pt_intr_post() and the only bit of the patch I
>totally omitted was the change to pt_process_missed_ticks(). I don't think
>that change can be important, but let's see what happens to the error
>percentage...
>
> -- Keir
>
>On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Hi Dan and Keir,
>>
>>Attached is a patch that fixes some issues with the SYNC policy
>>(no_missed_ticks_pending).
>>I have not tried to make the change the minimal one, but, rather, just
>>ported into
>>the new code what I know to work well. The error for
>>no_missed_ticks_pending goes from
>>over 3% to .03% with this change according to my testing.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Hi Dave --
>>>
>>>Did you get your correction ported?  If so, it would be nice to see this get
>>>into 3.1.3.
>>>
>>>Note that I just did some very limited testing with timer_mode=2(=SYNC=no
>>>missed ticks pending)
>>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've
>>>seen so far
>>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>>>
>>>Thanks,
>>>Dan
>>>
>>> 
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>To: dan.magenheimer@oracle.com
>>>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Dan,
>>>>
>>>>I did some testing with the constant tsc offset SYNC method
>>>>(now called
>>>>no_missed_ticks_pending)
>>>>and found the error to be very high, much larger than 1 %, as
>>>>I recall.
>>>>I have not had a chance to submit a correction. I will try to
>>>>do it later
>>>>this week or the first week in January. My version of constant tsc
>>>>offset SYNC method
>>>>produces .02 % error, so I just need to port that into the
>>>>current code.
>>>>
>>>>The error you got for both of those kernels is what I would expect
>>>>for the default mode, delay_for_missed_ticks.
>>>>
>>>>I'll let Keir answer on how to set the time mode.
>>>>
>>>>Regards,
>>>>Dave
>>>>
>>>>Dan Magenheimer wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>Anyone make measurements on the final patch?
>>>>>
>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>about 0.2% with no load.  This was xen-unstable tip today
>>>>with no options specified.  32-bit was about 0.01%.
>>>>   
>>>>
>>>>        
>>>>
>>>>>I think I missed something... how do I run the various
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>accounting choices and which ones are known to be appropriate
>>>>for which kernels?
>>>>   
>>>>
>>>>        
>>>>
>>>>>Thanks,
>>>>>Dan
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>-----Original Message-----
>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>Keir Fraser
>>>>   
>>>>
>>>>        
>>>>
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>To: Dave Winchell
>>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>>>>>Yunhong
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>disables pending
>>>>>>missed ticks
>>>>>>
>>>>>>
>>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>   
>>>>
>>>>        
>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Keir,
>>>>>>>
>>>>>>>The accuracy data I've collected for i/o loads for the
>>>>>>>various time protocols follows. In addition, the data
>>>>>>>for cpu loads is shown.
>>>>>>>
>>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>>>
>>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>All three guests are 8vcpu.
>>>>>>>
>>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>>>
>>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>>>>>
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>>>
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>>>>>
>>>>>>>
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>>>>>
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>>>
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>>>
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>>>>>
>>>>>>>
>>>>>>>Overhead measurements:
>>>>>>>
>>>>>>>Progress in terms of number of passes through a fixed
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>system workload
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>The workload was usex -b48.
>>>>>>>
>>>>>>>
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>
>>>>>>>
>>>>>>>Conclusions:
>>>>>>>
>>>>>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>>>>>tracking under the loads
>>>>>>>above is the SYNC protocol. The worst case accuracies for
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>SYNC, MIXED,
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>and ASYNC
>>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>
>>>>>>>We could reduce the cost of the SYNC method by only
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>scheduling the extra
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>wakeups if a certain number
>>>>>>>of ticks are missed.
>>>>>>>
>>>>>>>Regards,
>>>>>>>Dave
>>>>>>>
>>>>>>>Keir Fraser wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>couple of days ago,
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>I ran another ASYNC test. I think there may have been something
>>>>>>>>>wrong with the code I used a couple of days ago for
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>ASYNC. It may have been
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>missing the immediate delivery of interrupt after context
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>switch in.
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>acceptable accuracy,
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>each running consistently around or under .01%. MIXED has
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>a fairly high
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>error of
>>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>threshold for comfort.
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>I don't have an overnight run with SYNC. I plan to leave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>SYNC running
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>running instead.
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>It may be too early to pick the protocol and I can run
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>more overnight tests
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>next week.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>I'm a bit worried about any unwanted side effects of the
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>SYNC+run_timer
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>approach -- e.g., whether timer wakeups will cause higher
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>system-wide CPU
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>contention. I find it easier to think through the
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>implications of ASYNC. I'm
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>surprised that MIXED loses time, and is less accurate than
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>ASYNC. Perhaps it
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>delivers more timer interrupts than the other approaches,
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>and each interrupt
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>event causes a small accumulated error?
>>>>>>>>
>>>>>>>>Overall I would consider MIXED and ASYNC as favourites and
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>if the latter is
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>actually more accurate then I can simply revert the changeset that
>>>>>>>>implemented MIXED.
>>>>>>>>
>>>>>>>>Perhaps rather than running more of the same workloads you
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>could try idle
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>to /dev/null)? We
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>don't have any data on workloads that aren't CPU bound, so
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>that's really an
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>
>>>>>>>>-- Keir
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>_______________________________________________
>>>>>>Xen-devel mailing list
>>>>>>Xen-devel@lists.xensource.com
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>   
>>>>
>>>>        
>>>>
>>> 
>>>
>>>      
>>>
>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>> 
>>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
>>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>+        pt->do_not_freeze = 1;
>>     else
>>         pt->pending_intr_nr += missed_ticks;
>>     pt->scheduled += missed_ticks * pt->period;
>>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>> 
>>     pt_lock(pt);
>> 
>>-    pt->pending_intr_nr++;
>>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
>>+        pt->pending_intr_nr = 1;
>>+ pt->do_not_freeze = 0;
>>+    }
>>+    else
>>+ pt->pending_intr_nr++;
>> 
>>     if ( !pt->one_shot )
>>     {
>>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>         return;
>>     }
>> 
>>-    pt->do_not_freeze = 0;
>>-
>>     if ( pt->one_shot )
>>     {
>>         pt->enabled = 0;
>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>             pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */
>>         }
>>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>+     pt->pending_intr_nr--;
>>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>+ }
>>         else
>>         {
>>             pt->last_plt_gtime += pt->period_cycles;
>>    
>>
>
>
>  
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Thanks very much Dave and Keir!

Keir, if its not too late already, can this patch be added for 3.1.3?

Which reminds me, when will 3.1.3 go final?

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Keir,
> 
> The latest change, c/s 16690, looks fine.
> I agree that the code in c/s 16690 is equivalent to
> the code I submitted. Also, your version is more
> concise.
> 
> The error tests confirm the equivalence. With overnight cpu loads,
> the checked in version was accurate to +.048% for sles
> and +.038% for red hat. My version was +.046% and +.032% in a 
> 2 hour test.
> I don't think the difference is significant.
> 
> i/o loads produced errors of +.01%.
> 
> Thanks for all your efforts on this issue.
> 
> Regards,
> Dave
> 
> 
> 
> Keir Fraser wrote:
> 
> >Applied as c/s 16690, although the checked-in patch is 
> smaller. I think the
> >only important fix is to pt_intr_post() and the only bit of 
> the patch I
> >totally omitted was the change to pt_process_missed_ticks(). 
> I don't think
> >that change can be important, but let's see what happens to the error
> >percentage...
> >
> > -- Keir
> >
> >On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
> >
> >
> >
> >>Hi Dan and Keir,
> >>
> >>Attached is a patch that fixes some issues with the SYNC policy
> >>(no_missed_ticks_pending).
> >>I have not tried to make the change the minimal one, but, 
> rather, just
> >>ported into
> >>the new code what I know to work well. The error for
> >>no_missed_ticks_pending goes from
> >>over 3% to .03% with this change according to my testing.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dan Magenheimer wrote:
> >>
> >>
> >>
> >>>Hi Dave --
> >>>
> >>>Did you get your correction ported?  If so, it would be 
> nice to see this get
> >>>into 3.1.3.
> >>>
> >>>Note that I just did some very limited testing with 
> timer_mode=2(=SYNC=no
> >>>missed ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the 
> worst error I've
> >>>seen so far
> >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
> >>>
> >>>Thanks,
> >>>Dan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>To: dan.magenheimer@oracle.com
> >>>>Cc: Keir Fraser; Shan, Haitao; 
> xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>disables pending
> >>>>missed ticks
> >>>>
> >>>>
> >>>>Dan,
> >>>>
> >>>>I did some testing with the constant tsc offset SYNC method
> >>>>(now called
> >>>>no_missed_ticks_pending)
> >>>>and found the error to be very high, much larger than 1 %, as
> >>>>I recall.
> >>>>I have not had a chance to submit a correction. I will try to
> >>>>do it later
> >>>>this week or the first week in January. My version of constant tsc
> >>>>offset SYNC method
> >>>>produces .02 % error, so I just need to port that into the
> >>>>current code.
> >>>>
> >>>>The error you got for both of those kernels is what I would expect
> >>>>for the default mode, delay_for_missed_ticks.
> >>>>
> >>>>I'll let Keir answer on how to set the time mode.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Anyone make measurements on the final patch?
> >>>>>
> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>about 0.2% with no load.  This was xen-unstable tip today
> >>>>with no options specified.  32-bit was about 0.01%.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>I think I missed something... how do I run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>accounting choices and which ones are known to be appropriate
> >>>>for which kernels?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Thanks,
> >>>>>Dan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>Keir Fraser
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>To: Dave Winchell
> >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, 
> Eddie; Jiang,
> >>>>>>Yunhong
> >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>>disables pending
> >>>>>>missed ticks
> >>>>>>
> >>>>>>
> >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>>
> >>>>>>-- Keir
> >>>>>>
> >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>><dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Keir,
> >>>>>>>
> >>>>>>>The accuracy data I've collected for i/o loads for the
> >>>>>>>various time protocols follows. In addition, the data
> >>>>>>>for cpu loads is shown.
> >>>>>>>
> >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
> >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >>>>>>>
> >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>All three guests are 8vcpu.
> >>>>>>>
> >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
> >>>>>>>except that the redhat-64 guest has 4 instances of dd.
> >>>>>>>
> >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>
> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, 
> +.005% cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, 
> +.012% cpu
> >>>>>>>
> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >>>>>>>
> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, 
> -.031% cpu
> >>>>>>>
> >>>>>>>
> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, 
> -.09% i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% 
> -.14% i/o-8
> >>>>>>>
> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, 
> -.022% i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >>>>>>>
> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, 
> -.029% i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, 
> -.031% i/o-8
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, 
> -.005% i/o-32
> >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >>>>>>>
> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, 
> .003% i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, 
> .01% i/o-4/32
> >>>>>>>
> >>>>>>>
> >>>>>>>Overhead measurements:
> >>>>>>>
> >>>>>>>Progress in terms of number of passes through a fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>system workload
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>The workload was usex -b48.
> >>>>>>>
> >>>>>>>
> >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>
> >>>>>>>
> >>>>>>>Conclusions:
> >>>>>>>
> >>>>>>>The only protocol which meets the .05% accuracy 
> requirement for ntp
> >>>>>>>tracking under the loads
> >>>>>>>above is the SYNC protocol. The worst case accuracies for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>SYNC, MIXED,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>and ASYNC
> >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>
> >>>>>>>We could reduce the cost of the SYNC method by only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>scheduling the extra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>wakeups if a certain number
> >>>>>>>of ticks are missed.
> >>>>>>>
> >>>>>>>Regards,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>Keir Fraser wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>couple of days ago,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I ran another ASYNC test. I think there may have 
> been something
> >>>>>>>>>wrong with the code I used a couple of days ago for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>ASYNC. It may have been
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>missing the immediate delivery of interrupt after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>switch in.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>My results indicate that either SYNC or ASYNC give
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>acceptable accuracy,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>each running consistently around or under .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>a fairly high
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>error of
> >>>>>>>>>greater than .03%. Probably too close to .05% ntp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>threshold for comfort.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>SYNC running
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>running instead.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>It may be too early to pick the protocol and I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>more overnight tests
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>next week.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I'm a bit worried about any unwanted side effects of the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>SYNC+run_timer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>approach -- e.g., whether timer wakeups will cause higher
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>system-wide CPU
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>contention. I find it easier to think through the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>implications of ASYNC. I'm
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>surprised that MIXED loses time, and is less accurate than
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>ASYNC. Perhaps it
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>delivers more timer interrupts than the other approaches,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>and each interrupt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>event causes a small accumulated error?
> >>>>>>>>
> >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>if the latter is
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>actually more accurate then I can simply revert the 
> changeset that
> >>>>>>>>implemented MIXED.
> >>>>>>>>
> >>>>>>>>Perhaps rather than running more of the same workloads you
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>could try idle
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>to /dev/null)? We
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>don't have any data on workloads that aren't CPU bound, so
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>that's really an
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>
> >>>>>>>>-- Keir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>_______________________________________________
> >>>>>>Xen-devel mailing list
> >>>>>>Xen-devel@lists.xensource.com
> >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >>
> >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>+        pt->do_not_freeze = 1;
> >>     else
> >>         pt->pending_intr_nr += missed_ticks;
> >>     pt->scheduled += missed_ticks * pt->period;
> >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>
> >>     pt_lock(pt);
> >>
> >>-    pt->pending_intr_nr++;
> >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
> >>+        pt->pending_intr_nr = 1;
> >>+ pt->do_not_freeze = 0;
> >>+    }
> >>+    else
> >>+ pt->pending_intr_nr++;
> >>
> >>     if ( !pt->one_shot )
> >>     {
> >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >>         return;
> >>     }
> >>
> >>-    pt->do_not_freeze = 0;
> >>-
> >>     if ( pt->one_shot )
> >>     {
> >>         pt->enabled = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
> >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>             pt->pending_intr_nr = 0; /* 'collapse' all 
> missed ticks */
> >>         }
> >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >>+     pt->pending_intr_nr--;
> >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>+ }
> >>         else
> >>         {
> >>             pt->last_plt_gtime += pt->period_cycles;
> >>
> >>
> >
> >
> >
> >
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

Yes, it'll be added. 3.1.3 won't be for a couple of weeks. I want to get
3.2.0 squared away first.

 -- Keir

On 9/1/08 17:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> Thanks very much Dave and Keir!
> 
> Keir, if its not too late already, can this patch be added for 3.1.3?
> 
> Which reminds me, when will 3.1.3 go final?
> 
> Thanks,
> Dan
> 
>> -----Original Message-----
>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>> Sent: Wednesday, January 09, 2008 9:53 AM
>> To: Keir Fraser
>> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
>> Winchell
>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>> disables pending
>> missed ticks
>> 
>> 
>> Hi Keir,
>> 
>> The latest change, c/s 16690, looks fine.
>> I agree that the code in c/s 16690 is equivalent to
>> the code I submitted. Also, your version is more
>> concise.
>> 
>> The error tests confirm the equivalence. With overnight cpu loads,
>> the checked in version was accurate to +.048% for sles
>> and +.038% for red hat. My version was +.046% and +.032% in a
>> 2 hour test.
>> I don't think the difference is significant.
>> 
>> i/o loads produced errors of +.01%.
>> 
>> Thanks for all your efforts on this issue.
>> 
>> Regards,
>> Dave
>> 
>> 
>> 
>> Keir Fraser wrote:
>> 
>>> Applied as c/s 16690, although the checked-in patch is
>> smaller. I think the
>>> only important fix is to pt_intr_post() and the only bit of
>> the patch I
>>> totally omitted was the change to pt_process_missed_ticks().
>> I don't think
>>> that change can be important, but let's see what happens to the error
>>> percentage...
>>> 
>>> -- Keir
>>> 
>>> On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>> 
>>> 
>>> 
>>>> Hi Dan and Keir,
>>>> 
>>>> Attached is a patch that fixes some issues with the SYNC policy
>>>> (no_missed_ticks_pending).
>>>> I have not tried to make the change the minimal one, but,
>> rather, just
>>>> ported into
>>>> the new code what I know to work well. The error for
>>>> no_missed_ticks_pending goes from
>>>> over 3% to .03% with this change according to my testing.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>> Dan Magenheimer wrote:
>>>> 
>>>> 
>>>> 
>>>>> Hi Dave --
>>>>> 
>>>>> Did you get your correction ported?  If so, it would be
>> nice to see this get
>>>>> into 3.1.3.
>>>>> 
>>>>> Note that I just did some very limited testing with
>> timer_mode=2(=SYNC=no
>>>>> missed ticks pending)
>>>>> on tip of xen-3.1-testing (64-bit Linux hv guest) and the
>> worst error I've
>>>>> seen so far
>>>>> is 0.012%.  But I haven't tried any exotic loads, just LTP.
>>>>> 
>>>>> Thanks,
>>>>> Dan
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>> Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>> To: dan.magenheimer@oracle.com
>>>>>> Cc: Keir Fraser; Shan, Haitao;
>> xen-devel@lists.xensource.com; Dong,
>>>>>> Eddie; Jiang, Yunhong; Dave Winchell
>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>> disables pending
>>>>>> missed ticks
>>>>>> 
>>>>>> 
>>>>>> Dan,
>>>>>> 
>>>>>> I did some testing with the constant tsc offset SYNC method
>>>>>> (now called
>>>>>> no_missed_ticks_pending)
>>>>>> and found the error to be very high, much larger than 1 %, as
>>>>>> I recall.
>>>>>> I have not had a chance to submit a correction. I will try to
>>>>>> do it later
>>>>>> this week or the first week in January. My version of constant tsc
>>>>>> offset SYNC method
>>>>>> produces .02 % error, so I just need to port that into the
>>>>>> current code.
>>>>>> 
>>>>>> The error you got for both of those kernels is what I would expect
>>>>>> for the default mode, delay_for_missed_ticks.
>>>>>> 
>>>>>> I'll let Keir answer on how to set the time mode.
>>>>>> 
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>> Dan Magenheimer wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Anyone make measurements on the final patch?
>>>>>>> 
>>>>>>> I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> about 0.2% with no load.  This was xen-unstable tip today
>>>>>> with no options specified.  32-bit was about 0.01%.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> I think I missed something... how do I run the various
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> accounting choices and which ones are known to be appropriate
>>>>>> for which kernels?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Thanks,
>>>>>>> Dan
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: xen-devel-bounces@lists.xensource.com
>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> Keir Fraser
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>>> Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>> To: Dave Winchell
>>>>>>>> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>> Eddie; Jiang,
>>>>>>>> Yunhong
>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>> disables pending
>>>>>>>> missed ticks
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Please take a look at xen-unstable changeset 16545.
>>>>>>>> 
>>>>>>>> -- Keir
>>>>>>>> 
>>>>>>>> On 26/11/07 20:57, "Dave Winchell"
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Keir,
>>>>>>>>> 
>>>>>>>>> The accuracy data I've collected for i/o loads for the
>>>>>>>>> various time protocols follows. In addition, the data
>>>>>>>>> for cpu loads is shown.
>>>>>>>>> 
>>>>>>>>> The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>>>>>>> Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>> The cpu load is usex -e36 on each guest.
>>>>>>>>> (usex is available at http://people.redhat.com/anderson/usex.)
>>>>>>>>> i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>>>>> 
>>>>>>>>> The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>> Also, these are run on 4 cpu AMD box.
>>>>>>>>> In addition, there is an idle rh-32bit guest.
>>>>>>>>> All three guests are 8vcpu.
>>>>>>>>> 
>>>>>>>>> The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>>> except that the redhat-64 guest has 4 instances of dd.
>>>>>>>>> 
>>>>>>>>> Date Duration Protocol sles, rhat error load
>>>>>>>>> 
>>>>>>>>> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
>> +.005% cpu
>>>>>>>>> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
>> +.012% cpu
>>>>>>>>> 
>>>>>>>>> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>>>>>>> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>>>>>>> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>>>>> 
>>>>>>>>> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>>>>> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
>> -.031% cpu
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>> -.09% i/o-8
>>>>>>>>> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>> -.14% i/o-8
>>>>>>>>> 
>>>>>>>>> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>> -.022% i/o-8
>>>>>>>>> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>>>>> 
>>>>>>>>> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>> -.029% i/o-8
>>>>>>>>> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
>> -.031% i/o-8
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>>>>> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>> -.005% i/o-32
>>>>>>>>> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>>>>> 
>>>>>>>>> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>> .003% i/o-4/32
>>>>>>>>> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>> .01% i/o-4/32
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Overhead measurements:
>>>>>>>>> 
>>>>>>>>> Progress in terms of number of passes through a fixed
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> system workload
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>> The workload was usex -b48.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>> SYNC 167 min 144 passes .862 passes/min
>>>>>>>>> SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>> MIXED 221 min 196 passes .887 passes/min
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Conclusions:
>>>>>>>>> 
>>>>>>>>> The only protocol which meets the .05% accuracy
>> requirement for ntp
>>>>>>>>> tracking under the loads
>>>>>>>>> above is the SYNC protocol. The worst case accuracies for
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> SYNC, MIXED,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> and ASYNC
>>>>>>>>> are .022%, .12%, and .14%, respectively.
>>>>>>>>> 
>>>>>>>>> We could reduce the cost of the SYNC method by only
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> scheduling the extra
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> wakeups if a certain number
>>>>>>>>> of ticks are missed.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Dave
>>>>>>>>> 
>>>>>>>>> Keir Fraser wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Since I had a high error (~.03%) for the ASYNC method a
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> couple of days ago,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> I ran another ASYNC test. I think there may have
>> been something
>>>>>>>>>>> wrong with the code I used a couple of days ago for
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> ASYNC. It may have been
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> missing the immediate delivery of interrupt after context
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> switch in.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> My results indicate that either SYNC or ASYNC give
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> acceptable accuracy,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> each running consistently around or under .01%. MIXED has
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> a fairly high
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> error of
>>>>>>>>>>> greater than .03%. Probably too close to .05% ntp
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> threshold for comfort.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> I don't have an overnight run with SYNC. I plan to leave
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> SYNC running
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> over the weekend. If you'd rather I can leave MIXED
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> running instead.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> It may be too early to pick the protocol and I can run
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> more overnight tests
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> next week.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> I'm a bit worried about any unwanted side effects of the
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> SYNC+run_timer
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> approach -- e.g., whether timer wakeups will cause higher
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> system-wide CPU
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> contention. I find it easier to think through the
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> implications of ASYNC. I'm
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> surprised that MIXED loses time, and is less accurate than
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> ASYNC. Perhaps it
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> delivers more timer interrupts than the other approaches,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> and each interrupt
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> event causes a small accumulated error?
>>>>>>>>>> 
>>>>>>>>>> Overall I would consider MIXED and ASYNC as favourites and
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> if the latter is
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> actually more accurate then I can simply revert the
>> changeset that
>>>>>>>>>> implemented MIXED.
>>>>>>>>>> 
>>>>>>>>>> Perhaps rather than running more of the same workloads you
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> could try idle
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> to /dev/null)? We
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> don't have any data on workloads that aren't CPU bound, so
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> that's really an
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> obvious place to put any further effort imo.
>>>>>>>>>> 
>>>>>>>>>> -- Keir
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> Xen-devel mailing list
>>>>>>>> Xen-devel@lists.xensource.com
>>>>>>>> http://lists.xensource.com/xen-devel
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>> --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>>> +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>>> @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>>>> 
>>>>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>>>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
>>>> -        pt->do_not_freeze = !pt->pending_intr_nr;
>>>> +        pt->do_not_freeze = 1;
>>>>     else
>>>>         pt->pending_intr_nr += missed_ticks;
>>>>     pt->scheduled += missed_ticks * pt->period;
>>>> @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>> 
>>>>     pt_lock(pt);
>>>> 
>>>> -    pt->pending_intr_nr++;
>>>> +    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
>>>> +        pt->pending_intr_nr = 1;
>>>> + pt->do_not_freeze = 0;
>>>> +    }
>>>> +    else
>>>> + pt->pending_intr_nr++;
>>>> 
>>>>     if ( !pt->one_shot )
>>>>     {
>>>> @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>>>         return;
>>>>     }
>>>> 
>>>> -    pt->do_not_freeze = 0;
>>>> -
>>>>     if ( pt->one_shot )
>>>>     {
>>>>         pt->enabled = 0;
>>>> @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>>>>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>             pt->pending_intr_nr = 0; /* 'collapse' all
>> missed ticks */
>>>>         }
>>>> + else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>>> +     pt->pending_intr_nr--;
>>>> +     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>> + }
>>>>         else
>>>>         {
>>>>             pt->last_plt_gtime += pt->period_cycles;
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 17255 bytes --]

Sorry for the very late followup on this but we finally were able
to get our testing set up again on stable 3.1 bits and have
seen some very bad results on 3.1.3-rc1, on the order of 1%.

Test enviroment was a 4-socket dual core machine with 24GB of
memory running six two-vcpu 2GB domains, four hvm plus two pv.
All six guests were running LTP simultaneously.  The four hvm
guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
All four hvm guests experienced skew around -1%, even the 32-bit
guest.  Less intensive testing didn't exhibit much skew at all.

A representative graph is attached.

Dave, I wonder if some portion of your patches didn't end up in
the xen trees?

(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
14.318MHz HPET.)

Thanks,
Dan

P.S. Many thanks to Deepak and Akira for running tests.

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
> Dave Winchell
> Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Keir,
> 
> The latest change, c/s 16690, looks fine.
> I agree that the code in c/s 16690 is equivalent to
> the code I submitted. Also, your version is more
> concise.
> 
> The error tests confirm the equivalence. With overnight cpu loads,
> the checked in version was accurate to +.048% for sles
> and +.038% for red hat. My version was +.046% and +.032% in a 
> 2 hour test.
> I don't think the difference is significant.
> 
> i/o loads produced errors of +.01%.
> 
> Thanks for all your efforts on this issue.
> 
> Regards,
> Dave
> 
> 
> 
> Keir Fraser wrote:
> 
> >Applied as c/s 16690, although the checked-in patch is 
> smaller. I think the
> >only important fix is to pt_intr_post() and the only bit of 
> the patch I
> >totally omitted was the change to pt_process_missed_ticks(). 
> I don't think
> >that change can be important, but let's see what happens to the error
> >percentage...
> >
> > -- Keir
> >
> >On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
> >
> >
> >
> >>Hi Dan and Keir,
> >>
> >>Attached is a patch that fixes some issues with the SYNC policy
> >>(no_missed_ticks_pending).
> >>I have not tried to make the change the minimal one, but, 
> rather, just
> >>ported into
> >>the new code what I know to work well. The error for
> >>no_missed_ticks_pending goes from
> >>over 3% to .03% with this change according to my testing.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dan Magenheimer wrote:
> >>
> >>
> >>
> >>>Hi Dave --
> >>>
> >>>Did you get your correction ported?  If so, it would be 
> nice to see this get
> >>>into 3.1.3.
> >>>
> >>>Note that I just did some very limited testing with 
> timer_mode=2(=SYNC=no
> >>>missed ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the 
> worst error I've
> >>>seen so far
> >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
> >>>
> >>>Thanks,
> >>>Dan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>To: dan.magenheimer@oracle.com
> >>>>Cc: Keir Fraser; Shan, Haitao; 
> xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>disables pending
> >>>>missed ticks
> >>>>
> >>>>
> >>>>Dan,
> >>>>
> >>>>I did some testing with the constant tsc offset SYNC method
> >>>>(now called
> >>>>no_missed_ticks_pending)
> >>>>and found the error to be very high, much larger than 1 %, as
> >>>>I recall.
> >>>>I have not had a chance to submit a correction. I will try to
> >>>>do it later
> >>>>this week or the first week in January. My version of constant tsc
> >>>>offset SYNC method
> >>>>produces .02 % error, so I just need to port that into the
> >>>>current code.
> >>>>
> >>>>The error you got for both of those kernels is what I would expect
> >>>>for the default mode, delay_for_missed_ticks.
> >>>>
> >>>>I'll let Keir answer on how to set the time mode.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Anyone make measurements on the final patch?
> >>>>>
> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>about 0.2% with no load.  This was xen-unstable tip today
> >>>>with no options specified.  32-bit was about 0.01%.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>I think I missed something... how do I run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>accounting choices and which ones are known to be appropriate
> >>>>for which kernels?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Thanks,
> >>>>>Dan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>Keir Fraser
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>To: Dave Winchell
> >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, 
> Eddie; Jiang,
> >>>>>>Yunhong
> >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>>disables pending
> >>>>>>missed ticks
> >>>>>>
> >>>>>>
> >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>>
> >>>>>>-- Keir
> >>>>>>
> >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>><dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Keir,
> >>>>>>>
> >>>>>>>The accuracy data I've collected for i/o loads for the
> >>>>>>>various time protocols follows. In addition, the data
> >>>>>>>for cpu loads is shown.
> >>>>>>>
> >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
> >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >>>>>>>
> >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>All three guests are 8vcpu.
> >>>>>>>
> >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
> >>>>>>>except that the redhat-64 guest has 4 instances of dd.
> >>>>>>>
> >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>
> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, 
> +.005% cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, 
> +.012% cpu
> >>>>>>>
> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >>>>>>>
> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, 
> -.031% cpu
> >>>>>>>
> >>>>>>>
> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, 
> -.09% i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% 
> -.14% i/o-8
> >>>>>>>
> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, 
> -.022% i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >>>>>>>
> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, 
> -.029% i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, 
> -.031% i/o-8
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, 
> -.005% i/o-32
> >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >>>>>>>
> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, 
> .003% i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, 
> .01% i/o-4/32
> >>>>>>>
> >>>>>>>
> >>>>>>>Overhead measurements:
> >>>>>>>
> >>>>>>>Progress in terms of number of passes through a fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>system workload
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>The workload was usex -b48.
> >>>>>>>
> >>>>>>>
> >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>
> >>>>>>>
> >>>>>>>Conclusions:
> >>>>>>>
> >>>>>>>The only protocol which meets the .05% accuracy 
> requirement for ntp
> >>>>>>>tracking under the loads
> >>>>>>>above is the SYNC protocol. The worst case accuracies for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>SYNC, MIXED,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>and ASYNC
> >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>
> >>>>>>>We could reduce the cost of the SYNC method by only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>scheduling the extra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>wakeups if a certain number
> >>>>>>>of ticks are missed.
> >>>>>>>
> >>>>>>>Regards,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>Keir Fraser wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>couple of days ago,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I ran another ASYNC test. I think there may have 
> been something
> >>>>>>>>>wrong with the code I used a couple of days ago for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>ASYNC. It may have been
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>missing the immediate delivery of interrupt after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>switch in.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>My results indicate that either SYNC or ASYNC give
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>acceptable accuracy,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>each running consistently around or under .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>a fairly high
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>error of
> >>>>>>>>>greater than .03%. Probably too close to .05% ntp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>threshold for comfort.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>SYNC running
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>running instead.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>It may be too early to pick the protocol and I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>more overnight tests
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>next week.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I'm a bit worried about any unwanted side effects of the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>SYNC+run_timer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>approach -- e.g., whether timer wakeups will cause higher
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>system-wide CPU
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>contention. I find it easier to think through the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>implications of ASYNC. I'm
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>surprised that MIXED loses time, and is less accurate than
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>ASYNC. Perhaps it
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>delivers more timer interrupts than the other approaches,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>and each interrupt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>event causes a small accumulated error?
> >>>>>>>>
> >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>if the latter is
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>actually more accurate then I can simply revert the 
> changeset that
> >>>>>>>>implemented MIXED.
> >>>>>>>>
> >>>>>>>>Perhaps rather than running more of the same workloads you
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>could try idle
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>to /dev/null)? We
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>don't have any data on workloads that aren't CPU bound, so
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>that's really an
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>
> >>>>>>>>-- Keir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>_______________________________________________
> >>>>>>Xen-devel mailing list
> >>>>>>Xen-devel@lists.xensource.com
> >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >>
> >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>+        pt->do_not_freeze = 1;
> >>     else
> >>         pt->pending_intr_nr += missed_ticks;
> >>     pt->scheduled += missed_ticks * pt->period;
> >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>
> >>     pt_lock(pt);
> >>
> >>-    pt->pending_intr_nr++;
> >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
> >>+        pt->pending_intr_nr = 1;
> >>+ pt->do_not_freeze = 0;
> >>+    }
> >>+    else
> >>+ pt->pending_intr_nr++;
> >>
> >>     if ( !pt->one_shot )
> >>     {
> >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >>         return;
> >>     }
> >>
> >>-    pt->do_not_freeze = 0;
> >>-
> >>     if ( pt->one_shot )
> >>     {
> >>         pt->enabled = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
> >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>             pt->pending_intr_nr = 0; /* 'collapse' all 
> missed ticks */
> >>         }
> >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >>+     pt->pending_intr_nr--;
> >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>+ }
> >>         else
> >>         {
> >>             pt->last_plt_gtime += pt->period_cycles;
> >>
> >>
> >
> >
> >
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #2: el4u5-32-hvm-ltp.png --]
[-- Type: application/octet-stream, Size: 8105 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1.1: Type: text/plain, Size: 18025 bytes --]

Hi Dan,
 
Hpet timer does have a fairly large error, as I was trying this one recently.
I don't remember what I got for error, but 1% sounds about right.
The problem is that hpet is not built on top of vpt.c, the module Keir and I did
all the recent work in, for its periodic timer needs. Try specifying clock=pit
on the linux boot line. If it still picks the hpet, which it might, let me know
and I'll tell you how to get around this.
 
Regards,
Dave
 
 
 

________________________________

From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
Sent: Fri 1/25/2008 6:50 PM
To: Dave Winchell; Keir Fraser
Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com; akira.ijuin@oracle.com
Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticks



Sorry for the very late followup on this but we finally were able
to get our testing set up again on stable 3.1 bits and have
seen some very bad results on 3.1.3-rc1, on the order of 1%.

Test enviroment was a 4-socket dual core machine with 24GB of
memory running six two-vcpu 2GB domains, four hvm plus two pv.
All six guests were running LTP simultaneously.  The four hvm
guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
All four hvm guests experienced skew around -1%, even the 32-bit
guest.  Less intensive testing didn't exhibit much skew at all.

A representative graph is attached.

Dave, I wonder if some portion of your patches didn't end up in
the xen trees?

(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
14.318MHz HPET.)

Thanks,
Dan

P.S. Many thanks to Deepak and Akira for running tests.

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> Dave Winchell
> Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> disables pending
> missed ticks
>
>
> Hi Keir,
>
> The latest change, c/s 16690, looks fine.
> I agree that the code in c/s 16690 is equivalent to
> the code I submitted. Also, your version is more
> concise.
>
> The error tests confirm the equivalence. With overnight cpu loads,
> the checked in version was accurate to +.048% for sles
> and +.038% for red hat. My version was +.046% and +.032% in a
> 2 hour test.
> I don't think the difference is significant.
>
> i/o loads produced errors of +.01%.
>
> Thanks for all your efforts on this issue.
>
> Regards,
> Dave
>
>
>
> Keir Fraser wrote:
>
> >Applied as c/s 16690, although the checked-in patch is
> smaller. I think the
> >only important fix is to pt_intr_post() and the only bit of
> the patch I
> >totally omitted was the change to pt_process_missed_ticks().
> I don't think
> >that change can be important, but let's see what happens to the error
> >percentage...
> >
> > -- Keir
> >
> >On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
> >
> >
> >
> >>Hi Dan and Keir,
> >>
> >>Attached is a patch that fixes some issues with the SYNC policy
> >>(no_missed_ticks_pending).
> >>I have not tried to make the change the minimal one, but,
> rather, just
> >>ported into
> >>the new code what I know to work well. The error for
> >>no_missed_ticks_pending goes from
> >>over 3% to .03% with this change according to my testing.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dan Magenheimer wrote:
> >>
> >>
> >>
> >>>Hi Dave --
> >>>
> >>>Did you get your correction ported?  If so, it would be
> nice to see this get
> >>>into 3.1.3.
> >>>
> >>>Note that I just did some very limited testing with
> timer_mode=2(=SYNC=no
> >>>missed ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
> worst error I've
> >>>seen so far
> >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
> >>>
> >>>Thanks,
> >>>Dan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>To: dan.magenheimer@oracle.com
> >>>>Cc: Keir Fraser; Shan, Haitao;
> xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>disables pending
> >>>>missed ticks
> >>>>
> >>>>
> >>>>Dan,
> >>>>
> >>>>I did some testing with the constant tsc offset SYNC method
> >>>>(now called
> >>>>no_missed_ticks_pending)
> >>>>and found the error to be very high, much larger than 1 %, as
> >>>>I recall.
> >>>>I have not had a chance to submit a correction. I will try to
> >>>>do it later
> >>>>this week or the first week in January. My version of constant tsc
> >>>>offset SYNC method
> >>>>produces .02 % error, so I just need to port that into the
> >>>>current code.
> >>>>
> >>>>The error you got for both of those kernels is what I would expect
> >>>>for the default mode, delay_for_missed_ticks.
> >>>>
> >>>>I'll let Keir answer on how to set the time mode.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Anyone make measurements on the final patch?
> >>>>>
> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>about 0.2% with no load.  This was xen-unstable tip today
> >>>>with no options specified.  32-bit was about 0.01%.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>I think I missed something... how do I run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>accounting choices and which ones are known to be appropriate
> >>>>for which kernels?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Thanks,
> >>>>>Dan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>Keir Fraser
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>To: Dave Winchell
> >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
> Eddie; Jiang,
> >>>>>>Yunhong
> >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>>disables pending
> >>>>>>missed ticks
> >>>>>>
> >>>>>>
> >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>>
> >>>>>>-- Keir
> >>>>>>
> >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>><dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Keir,
> >>>>>>>
> >>>>>>>The accuracy data I've collected for i/o loads for the
> >>>>>>>various time protocols follows. In addition, the data
> >>>>>>>for cpu loads is shown.
> >>>>>>>
> >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
> >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >>>>>>>
> >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>All three guests are 8vcpu.
> >>>>>>>
> >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
> >>>>>>>except that the redhat-64 guest has 4 instances of dd.
> >>>>>>>
> >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>
> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
> +.005% cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
> +.012% cpu
> >>>>>>>
> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >>>>>>>
> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
> -.031% cpu
> >>>>>>>
> >>>>>>>
> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
> -.09% i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
> -.14% i/o-8
> >>>>>>>
> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
> -.022% i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >>>>>>>
> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
> -.029% i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
> -.031% i/o-8
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
> -.005% i/o-32
> >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >>>>>>>
> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
> .003% i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
> .01% i/o-4/32
> >>>>>>>
> >>>>>>>
> >>>>>>>Overhead measurements:
> >>>>>>>
> >>>>>>>Progress in terms of number of passes through a fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>system workload
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>The workload was usex -b48.
> >>>>>>>
> >>>>>>>
> >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>
> >>>>>>>
> >>>>>>>Conclusions:
> >>>>>>>
> >>>>>>>The only protocol which meets the .05% accuracy
> requirement for ntp
> >>>>>>>tracking under the loads
> >>>>>>>above is the SYNC protocol. The worst case accuracies for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>SYNC, MIXED,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>and ASYNC
> >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>
> >>>>>>>We could reduce the cost of the SYNC method by only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>scheduling the extra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>wakeups if a certain number
> >>>>>>>of ticks are missed.
> >>>>>>>
> >>>>>>>Regards,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>Keir Fraser wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>couple of days ago,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I ran another ASYNC test. I think there may have
> been something
> >>>>>>>>>wrong with the code I used a couple of days ago for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>ASYNC. It may have been
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>missing the immediate delivery of interrupt after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>switch in.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>My results indicate that either SYNC or ASYNC give
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>acceptable accuracy,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>each running consistently around or under .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>a fairly high
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>error of
> >>>>>>>>>greater than .03%. Probably too close to .05% ntp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>threshold for comfort.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>SYNC running
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>running instead.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>It may be too early to pick the protocol and I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>more overnight tests
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>next week.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I'm a bit worried about any unwanted side effects of the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>SYNC+run_timer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>approach -- e.g., whether timer wakeups will cause higher
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>system-wide CPU
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>contention. I find it easier to think through the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>implications of ASYNC. I'm
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>surprised that MIXED loses time, and is less accurate than
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>ASYNC. Perhaps it
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>delivers more timer interrupts than the other approaches,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>and each interrupt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>event causes a small accumulated error?
> >>>>>>>>
> >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>if the latter is
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>actually more accurate then I can simply revert the
> changeset that
> >>>>>>>>implemented MIXED.
> >>>>>>>>
> >>>>>>>>Perhaps rather than running more of the same workloads you
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>could try idle
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>to /dev/null)? We
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>don't have any data on workloads that aren't CPU bound, so
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>that's really an
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>
> >>>>>>>>-- Keir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>_______________________________________________
> >>>>>>Xen-devel mailing list
> >>>>>>Xen-devel@lists.xensource.com
> >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >>
> >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>+        pt->do_not_freeze = 1;
> >>     else
> >>         pt->pending_intr_nr += missed_ticks;
> >>     pt->scheduled += missed_ticks * pt->period;
> >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>
> >>     pt_lock(pt);
> >>
> >>-    pt->pending_intr_nr++;
> >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
> >>+        pt->pending_intr_nr = 1;
> >>+ pt->do_not_freeze = 0;
> >>+    }
> >>+    else
> >>+ pt->pending_intr_nr++;
> >>
> >>     if ( !pt->one_shot )
> >>     {
> >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >>         return;
> >>     }
> >>
> >>-    pt->do_not_freeze = 0;
> >>-
> >>     if ( pt->one_shot )
> >>     {
> >>         pt->enabled = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
> >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>             pt->pending_intr_nr = 0; /* 'collapse' all
> missed ticks */
> >>         }
> >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >>+     pt->pending_intr_nr--;
> >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>+ }
> >>         else
> >>         {
> >>             pt->last_plt_gtime += pt->period_cycles;
> >>
> >>
> >
> >
> >
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>



[-- Attachment #1.2: Type: text/html, Size: 35973 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

[-- Attachment #1.1: Type: text/plain, Size: 20494 bytes --]

RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticksThanks, I hadn't realized that!  No wonder we didn't see the same improvement you saw!

> Try specifying clock=pit on the linux boot line...

I'm confused... do you mean "clocksource=pit" on the Xen command line or
"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or dom0?) command
line?  Or both places?  Since the tests take awhile, it would be nice
to get this right the first time.  Do the Xen and guest clocksources need
to be the same?

Thanks,
Dan
 
 -----Original Message-----
From: Dave Winchell [mailto:dwinchell@virtualiron.com]
Sent: Sunday, January 27, 2008 2:22 PM
To: dan.magenheimer@oracle.com; Keir Fraser
Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com; akira.ijuin@oracle.com; Dave Winchell
Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticks


  Hi Dan,

  Hpet timer does have a fairly large error, as I was trying this one recently.
  I don't remember what I got for error, but 1% sounds about right.
  The problem is that hpet is not built on top of vpt.c, the module Keir and I did
  all the recent work in, for its periodic timer needs. Try specifying clock=pit
  on the linux boot line. If it still picks the hpet, which it might, let me know
  and I'll tell you how to get around this.

  Regards,
  Dave





------------------------------------------------------------------------------
  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
  Sent: Fri 1/25/2008 6:50 PM
  To: Dave Winchell; Keir Fraser
  Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com; akira.ijuin@oracle.com
  Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticks


  Sorry for the very late followup on this but we finally were able
  to get our testing set up again on stable 3.1 bits and have
  seen some very bad results on 3.1.3-rc1, on the order of 1%.

  Test enviroment was a 4-socket dual core machine with 24GB of
  memory running six two-vcpu 2GB domains, four hvm plus two pv.
  All six guests were running LTP simultaneously.  The four hvm
  guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
  Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
  All four hvm guests experienced skew around -1%, even the 32-bit
  guest.  Less intensive testing didn't exhibit much skew at all.

  A representative graph is attached.

  Dave, I wonder if some portion of your patches didn't end up in
  the xen trees?

  (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
  14.318MHz HPET.)

  Thanks,
  Dan

  P.S. Many thanks to Deepak and Akira for running tests.

  > -----Original Message-----
  > From: xen-devel-bounces@lists.xensource.com
  > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
  > Dave Winchell
  > Sent: Wednesday, January 09, 2008 9:53 AM
  > To: Keir Fraser
  > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
  > Winchell
  > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
  > disables pending
  > missed ticks
  >
  >
  > Hi Keir,
  >
  > The latest change, c/s 16690, looks fine.
  > I agree that the code in c/s 16690 is equivalent to
  > the code I submitted. Also, your version is more
  > concise.
  >
  > The error tests confirm the equivalence. With overnight cpu loads,
  > the checked in version was accurate to +.048% for sles
  > and +.038% for red hat. My version was +.046% and +.032% in a
  > 2 hour test.
  > I don't think the difference is significant.
  >
  > i/o loads produced errors of +.01%.
  >
  > Thanks for all your efforts on this issue.
  >
  > Regards,
  > Dave
  >
  >
  >
  > Keir Fraser wrote:
  >
  > >Applied as c/s 16690, although the checked-in patch is
  > smaller. I think the
  > >only important fix is to pt_intr_post() and the only bit of
  > the patch I
  > >totally omitted was the change to pt_process_missed_ticks().
  > I don't think
  > >that change can be important, but let's see what happens to the error
  > >percentage...
  > >
  > > -- Keir
  > >
  > >On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
  > >
  > >
  > >
  > >>Hi Dan and Keir,
  > >>
  > >>Attached is a patch that fixes some issues with the SYNC policy
  > >>(no_missed_ticks_pending).
  > >>I have not tried to make the change the minimal one, but,
  > rather, just
  > >>ported into
  > >>the new code what I know to work well. The error for
  > >>no_missed_ticks_pending goes from
  > >>over 3% to .03% with this change according to my testing.
  > >>
  > >>Regards,
  > >>Dave
  > >>
  > >>Dan Magenheimer wrote:
  > >>
  > >>
  > >>
  > >>>Hi Dave --
  > >>>
  > >>>Did you get your correction ported?  If so, it would be
  > nice to see this get
  > >>>into 3.1.3.
  > >>>
  > >>>Note that I just did some very limited testing with
  > timer_mode=2(=SYNC=no
  > >>>missed ticks pending)
  > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
  > worst error I've
  > >>>seen so far
  > >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
  > >>>
  > >>>Thanks,
  > >>>Dan
  > >>>
  > >>>
  > >>>
  > >>>
  > >>>
  > >>>>-----Original Message-----
  > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
  > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
  > >>>>To: dan.magenheimer@oracle.com
  > >>>>Cc: Keir Fraser; Shan, Haitao;
  > xen-devel@lists.xensource.com; Dong,
  > >>>>Eddie; Jiang, Yunhong; Dave Winchell
  > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
  > >>>>disables pending
  > >>>>missed ticks
  > >>>>
  > >>>>
  > >>>>Dan,
  > >>>>
  > >>>>I did some testing with the constant tsc offset SYNC method
  > >>>>(now called
  > >>>>no_missed_ticks_pending)
  > >>>>and found the error to be very high, much larger than 1 %, as
  > >>>>I recall.
  > >>>>I have not had a chance to submit a correction. I will try to
  > >>>>do it later
  > >>>>this week or the first week in January. My version of constant tsc
  > >>>>offset SYNC method
  > >>>>produces .02 % error, so I just need to port that into the
  > >>>>current code.
  > >>>>
  > >>>>The error you got for both of those kernels is what I would expect
  > >>>>for the default mode, delay_for_missed_ticks.
  > >>>>
  > >>>>I'll let Keir answer on how to set the time mode.
  > >>>>
  > >>>>Regards,
  > >>>>Dave
  > >>>>
  > >>>>Dan Magenheimer wrote:
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>Anyone make measurements on the final patch?
  > >>>>>
  > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>about 0.2% with no load.  This was xen-unstable tip today
  > >>>>with no options specified.  32-bit was about 0.01%.
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>I think I missed something... how do I run the various
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>accounting choices and which ones are known to be appropriate
  > >>>>for which kernels?
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>Thanks,
  > >>>>>Dan
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>>-----Original Message-----
  > >>>>>>From: xen-devel-bounces@lists.xensource.com
  > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>Keir Fraser
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
  > >>>>>>To: Dave Winchell
  > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
  > Eddie; Jiang,
  > >>>>>>Yunhong
  > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
  > >>>>>>disables pending
  > >>>>>>missed ticks
  > >>>>>>
  > >>>>>>
  > >>>>>>Please take a look at xen-unstable changeset 16545.
  > >>>>>>
  > >>>>>>-- Keir
  > >>>>>>
  > >>>>>>On 26/11/07 20:57, "Dave Winchell"
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>><dwinchell@virtualiron.com> wrote:
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>Keir,
  > >>>>>>>
  > >>>>>>>The accuracy data I've collected for i/o loads for the
  > >>>>>>>various time protocols follows. In addition, the data
  > >>>>>>>for cpu loads is shown.
  > >>>>>>>
  > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
  > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
  > >>>>>>>The cpu load is usex -e36 on each guest.
  > >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.)
  > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
  > >>>>>>>
  > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
  > >>>>>>>Also, these are run on 4 cpu AMD box.
  > >>>>>>>In addition, there is an idle rh-32bit guest.
  > >>>>>>>All three guests are 8vcpu.
  > >>>>>>>
  > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
  > >>>>>>>except that the redhat-64 guest has 4 instances of dd.
  > >>>>>>>
  > >>>>>>>Date Duration Protocol sles, rhat error load
  > >>>>>>>
  > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
  > +.005% cpu
  > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
  > +.012% cpu
  > >>>>>>>
  > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
  > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
  > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
  > >>>>>>>
  > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
  > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
  > -.031% cpu
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
  > -.09% i/o-8
  > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
  > -.14% i/o-8
  > >>>>>>>
  > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
  > -.022% i/o-8
  > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
  > >>>>>>>
  > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
  > -.029% i/o-8
  > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
  > -.031% i/o-8
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
  > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
  > -.005% i/o-32
  > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
  > >>>>>>>
  > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
  > .003% i/o-4/32
  > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
  > .01% i/o-4/32
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>Overhead measurements:
  > >>>>>>>
  > >>>>>>>Progress in terms of number of passes through a fixed
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>system workload
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
  > >>>>>>>The workload was usex -b48.
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
  > >>>>>>>SYNC 167 min 144 passes .862 passes/min
  > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
  > >>>>>>>MIXED 221 min 196 passes .887 passes/min
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>Conclusions:
  > >>>>>>>
  > >>>>>>>The only protocol which meets the .05% accuracy
  > requirement for ntp
  > >>>>>>>tracking under the loads
  > >>>>>>>above is the SYNC protocol. The worst case accuracies for
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>SYNC, MIXED,
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>and ASYNC
  > >>>>>>>are .022%, .12%, and .14%, respectively.
  > >>>>>>>
  > >>>>>>>We could reduce the cost of the SYNC method by only
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>scheduling the extra
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>wakeups if a certain number
  > >>>>>>>of ticks are missed.
  > >>>>>>>
  > >>>>>>>Regards,
  > >>>>>>>Dave
  > >>>>>>>
  > >>>>>>>Keir Fraser wrote:
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>><dwinchell@virtualiron.com> wrote:
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>couple of days ago,
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>I ran another ASYNC test. I think there may have
  > been something
  > >>>>>>>>>wrong with the code I used a couple of days ago for
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>ASYNC. It may have been
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>missing the immediate delivery of interrupt after context
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>switch in.
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>My results indicate that either SYNC or ASYNC give
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>acceptable accuracy,
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>each running consistently around or under .01%. MIXED has
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>a fairly high
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>error of
  > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>threshold for comfort.
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>SYNC running
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>running instead.
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>It may be too early to pick the protocol and I can run
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>more overnight tests
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>next week.
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>I'm a bit worried about any unwanted side effects of the
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>SYNC+run_timer
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>system-wide CPU
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>contention. I find it easier to think through the
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>implications of ASYNC. I'm
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>surprised that MIXED loses time, and is less accurate than
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>ASYNC. Perhaps it
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>delivers more timer interrupts than the other approaches,
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>and each interrupt
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>event causes a small accumulated error?
  > >>>>>>>>
  > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>if the latter is
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>actually more accurate then I can simply revert the
  > changeset that
  > >>>>>>>>implemented MIXED.
  > >>>>>>>>
  > >>>>>>>>Perhaps rather than running more of the same workloads you
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>could try idle
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>to /dev/null)? We
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>don't have any data on workloads that aren't CPU bound, so
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>that's really an
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>obvious place to put any further effort imo.
  > >>>>>>>>
  > >>>>>>>>-- Keir
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>_______________________________________________
  > >>>>>>Xen-devel mailing list
  > >>>>>>Xen-devel@lists.xensource.com
  > >>>>>>http://lists.xensource.com/xen-devel
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>
  > >>>
  > >>>
  > >>>
  > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
  > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
  > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
  > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
  > >>
  > >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
  > >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
  > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
  > >>+        pt->do_not_freeze = 1;
  > >>     else
  > >>         pt->pending_intr_nr += missed_ticks;
  > >>     pt->scheduled += missed_ticks * pt->period;
  > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
  > >>
  > >>     pt_lock(pt);
  > >>
  > >>-    pt->pending_intr_nr++;
  > >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
  > >>+        pt->pending_intr_nr = 1;
  > >>+ pt->do_not_freeze = 0;
  > >>+    }
  > >>+    else
  > >>+ pt->pending_intr_nr++;
  > >>
  > >>     if ( !pt->one_shot )
  > >>     {
  > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
  > >>         return;
  > >>     }
  > >>
  > >>-    pt->do_not_freeze = 0;
  > >>-
  > >>     if ( pt->one_shot )
  > >>     {
  > >>         pt->enabled = 0;
  > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
  > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
  > >>             pt->pending_intr_nr = 0; /* 'collapse' all
  > missed ticks */
  > >>         }
  > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
  > >>+     pt->pending_intr_nr--;
  > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
  > >>+ }
  > >>         else
  > >>         {
  > >>             pt->last_plt_gtime += pt->period_cycles;
  > >>
  > >>
  > >
  > >
  > >
  > >
  >
  >
  > _______________________________________________
  > Xen-devel mailing list
  > Xen-devel@lists.xensource.com
  > http://lists.xensource.com/xen-devel
  >


[-- Attachment #1.2: Type: text/html, Size: 40702 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Dan,

I guess I'm a bit out of date calling for clock= usage.
Looking at linux 2.6.20.4 sources, I think you should specify
"clocksource=pit nohpet" on the linux guest bootline.

You can leave the xen and dom0 bootlines as they are.
The xen and guest clocksources do not need to be the same.
In my tests, xen is using the hpet for its timekeeping and
that appears to be the default.

When you boot the guests you should see
    time.c: Using PIT/TSC based timekeeping.
on the rh4u5-64 guest, and something similar on the others.

 > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
 > 14.318MHz HPET.)

This appears to be the xen state, which is fine.
I was wrongly assuming that this was the guest state.
You might want to look in your guest logs and see what they were picking
for a clock source.

Regards,
Dave




Dan Magenheimer wrote:

> Thanks, I hadn't realized that!  No wonder we didn't see the same 
> improvement you saw!
>  
>> Try specifying clock=pit on the linux boot line...
>  
> I'm confused... do you mean "clocksource=pit" on the Xen command line or
> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or dom0?) command
> line?  Or both places?  Since the tests take awhile, it would be nice
> to get this right the first time.  Do the Xen and guest clocksources need
> to be the same?
>  
> Thanks,
> Dan
>  
>  -----Original Message-----
> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> *Sent:* Sunday, January 27, 2008 2:22 PM
> *To:* dan.magenheimer@oracle.com; Keir Fraser
> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; 
> akira.ijuin@oracle.com; Dave Winchell
> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables 
> pending missed ticks
>
>     Hi Dan,
>      
>     Hpet timer does have a fairly large error, as I was trying this
>     one recently.
>     I don't remember what I got for error, but 1% sounds about right.
>     The problem is that hpet is not built on top of vpt.c, the module
>     Keir and I did
>     all the recent work in, for its periodic timer needs. Try
>     specifying clock=pit
>     on the linux boot line. If it still picks the hpet, which it
>     might, let me know
>     and I'll tell you how to get around this.
>      
>     Regards,
>     Dave
>      
>      
>      
>
>     ------------------------------------------------------------------------
>     *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>     *Sent:* Fri 1/25/2008 6:50 PM
>     *To:* Dave Winchell; Keir Fraser
>     *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>     akira.ijuin@oracle.com
>     *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
>     pending missed ticks
>
>     Sorry for the very late followup on this but we finally were able
>     to get our testing set up again on stable 3.1 bits and have
>     seen some very bad results on 3.1.3-rc1, on the order of 1%.
>
>     Test enviroment was a 4-socket dual core machine with 24GB of
>     memory running six two-vcpu 2GB domains, four hvm plus two pv.
>     All six guests were running LTP simultaneously.  The four hvm
>     guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
>     Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
>     All four hvm guests experienced skew around -1%, even the 32-bit
>     guest.  Less intensive testing didn't exhibit much skew at all.
>
>     A representative graph is attached.
>
>     Dave, I wonder if some portion of your patches didn't end up in
>     the xen trees?
>
>     (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>     14.318MHz HPET.)
>
>     Thanks,
>     Dan
>
>     P.S. Many thanks to Deepak and Akira for running tests.
>
>     > -----Original Message-----
>     > From: xen-devel-bounces@lists.xensource.com
>     > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>     > Dave Winchell
>     > Sent: Wednesday, January 09, 2008 9:53 AM
>     > To: Keir Fraser
>     > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
>     > Winchell
>     > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>     > disables pending
>     > missed ticks
>     >
>     >
>     > Hi Keir,
>     >
>     > The latest change, c/s 16690, looks fine.
>     > I agree that the code in c/s 16690 is equivalent to
>     > the code I submitted. Also, your version is more
>     > concise.
>     >
>     > The error tests confirm the equivalence. With overnight cpu loads,
>     > the checked in version was accurate to +.048% for sles
>     > and +.038% for red hat. My version was +.046% and +.032% in a
>     > 2 hour test.
>     > I don't think the difference is significant.
>     >
>     > i/o loads produced errors of +.01%.
>     >
>     > Thanks for all your efforts on this issue.
>     >
>     > Regards,
>     > Dave
>     >
>     >
>     >
>     > Keir Fraser wrote:
>     >
>     > >Applied as c/s 16690, although the checked-in patch is
>     > smaller. I think the
>     > >only important fix is to pt_intr_post() and the only bit of
>     > the patch I
>     > >totally omitted was the change to pt_process_missed_ticks().
>     > I don't think
>     > >that change can be important, but let's see what happens to the
>     error
>     > >percentage...
>     > >
>     > > -- Keir
>     > >
>     > >On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>     > >
>     > >
>     > >
>     > >>Hi Dan and Keir,
>     > >>
>     > >>Attached is a patch that fixes some issues with the SYNC policy
>     > >>(no_missed_ticks_pending).
>     > >>I have not tried to make the change the minimal one, but,
>     > rather, just
>     > >>ported into
>     > >>the new code what I know to work well. The error for
>     > >>no_missed_ticks_pending goes from
>     > >>over 3% to .03% with this change according to my testing.
>     > >>
>     > >>Regards,
>     > >>Dave
>     > >>
>     > >>Dan Magenheimer wrote:
>     > >>
>     > >>
>     > >>
>     > >>>Hi Dave --
>     > >>>
>     > >>>Did you get your correction ported?  If so, it would be
>     > nice to see this get
>     > >>>into 3.1.3.
>     > >>>
>     > >>>Note that I just did some very limited testing with
>     > timer_mode=2(=SYNC=no
>     > >>>missed ticks pending)
>     > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
>     > worst error I've
>     > >>>seen so far
>     > >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>     > >>>
>     > >>>Thanks,
>     > >>>Dan
>     > >>>
>     > >>>
>     > >>>
>     > >>>
>     > >>>
>     > >>>>-----Original Message-----
>     > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>     > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>     > >>>>To: dan.magenheimer@oracle.com
>     > >>>>Cc: Keir Fraser; Shan, Haitao;
>     > xen-devel@lists.xensource.com; Dong,
>     > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>     > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>     > >>>>disables pending
>     > >>>>missed ticks
>     > >>>>
>     > >>>>
>     > >>>>Dan,
>     > >>>>
>     > >>>>I did some testing with the constant tsc offset SYNC method
>     > >>>>(now called
>     > >>>>no_missed_ticks_pending)
>     > >>>>and found the error to be very high, much larger than 1 %, as
>     > >>>>I recall.
>     > >>>>I have not had a chance to submit a correction. I will try to
>     > >>>>do it later
>     > >>>>this week or the first week in January. My version of
>     constant tsc
>     > >>>>offset SYNC method
>     > >>>>produces .02 % error, so I just need to port that into the
>     > >>>>current code.
>     > >>>>
>     > >>>>The error you got for both of those kernels is what I would
>     expect
>     > >>>>for the default mode, delay_for_missed_ticks.
>     > >>>>
>     > >>>>I'll let Keir answer on how to set the time mode.
>     > >>>>
>     > >>>>Regards,
>     > >>>>Dave
>     > >>>>
>     > >>>>Dan Magenheimer wrote:
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>Anyone make measurements on the final patch?
>     > >>>>>
>     > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>about 0.2% with no load.  This was xen-unstable tip today
>     > >>>>with no options specified.  32-bit was about 0.01%.
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>I think I missed something... how do I run the various
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>accounting choices and which ones are known to be appropriate
>     > >>>>for which kernels?
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>Thanks,
>     > >>>>>Dan
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>>-----Original Message-----
>     > >>>>>>From: xen-devel-bounces@lists.xensource.com
>     > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>Keir Fraser
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>     > >>>>>>To: Dave Winchell
>     > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>     > Eddie; Jiang,
>     > >>>>>>Yunhong
>     > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>     > >>>>>>disables pending
>     > >>>>>>missed ticks
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>Please take a look at xen-unstable changeset 16545.
>     > >>>>>>
>     > >>>>>>-- Keir
>     > >>>>>>
>     > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>><dwinchell@virtualiron.com> wrote:
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>Keir,
>     > >>>>>>>
>     > >>>>>>>The accuracy data I've collected for i/o loads for the
>     > >>>>>>>various time protocols follows. In addition, the data
>     > >>>>>>>for cpu loads is shown.
>     > >>>>>>>
>     > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD
>     box.
>     > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>     > >>>>>>>The cpu load is usex -e36 on each guest.
>     > >>>>>>>(usex is available at
>     http://people.redhat.com/anderson/usex.)
>     > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>     > >>>>>>>
>     > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>     > >>>>>>>Also, these are run on 4 cpu AMD box.
>     > >>>>>>>In addition, there is an idle rh-32bit guest.
>     > >>>>>>>All three guests are 8vcpu.
>     > >>>>>>>
>     > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>     > >>>>>>>except that the redhat-64 guest has 4 instances of dd.
>     > >>>>>>>
>     > >>>>>>>Date Duration Protocol sles, rhat error load
>     > >>>>>>>
>     > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
>     > +.005% cpu
>     > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
>     > +.012% cpu
>     > >>>>>>>
>     > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>     -.004% cpu
>     > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>     > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>     > >>>>>>>
>     > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>     > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
>     > -.031% cpu
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>     > -.09% i/o-8
>     > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>     > -.14% i/o-8
>     > >>>>>>>
>     > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>     > -.022% i/o-8
>     > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>     > >>>>>>>
>     > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>     > -.029% i/o-8
>     > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
>     > -.031% i/o-8
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>     > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>     > -.005% i/o-32
>     > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>     > >>>>>>>
>     > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>     > .003% i/o-4/32
>     > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>     > .01% i/o-4/32
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>Overhead measurements:
>     > >>>>>>>
>     > >>>>>>>Progress in terms of number of passes through a fixed
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>system workload
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>     > >>>>>>>The workload was usex -b48.
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>     > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>     > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>     > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>Conclusions:
>     > >>>>>>>
>     > >>>>>>>The only protocol which meets the .05% accuracy
>     > requirement for ntp
>     > >>>>>>>tracking under the loads
>     > >>>>>>>above is the SYNC protocol. The worst case accuracies for
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>SYNC, MIXED,
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>and ASYNC
>     > >>>>>>>are .022%, .12%, and .14%, respectively.
>     > >>>>>>>
>     > >>>>>>>We could reduce the cost of the SYNC method by only
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>scheduling the extra
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>wakeups if a certain number
>     > >>>>>>>of ticks are missed.
>     > >>>>>>>
>     > >>>>>>>Regards,
>     > >>>>>>>Dave
>     > >>>>>>>
>     > >>>>>>>Keir Fraser wrote:
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>><dwinchell@virtualiron.com> wrote:
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>couple of days ago,
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>I ran another ASYNC test. I think there may have
>     > been something
>     > >>>>>>>>>wrong with the code I used a couple of days ago for
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>ASYNC. It may have been
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>missing the immediate delivery of interrupt after context
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>switch in.
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>acceptable accuracy,
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>each running consistently around or under .01%. MIXED has
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>a fairly high
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>error of
>     > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>threshold for comfort.
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>SYNC running
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>running instead.
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>It may be too early to pick the protocol and I can run
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>more overnight tests
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>next week.
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>I'm a bit worried about any unwanted side effects of the
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>SYNC+run_timer
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>system-wide CPU
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>contention. I find it easier to think through the
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>implications of ASYNC. I'm
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>surprised that MIXED loses time, and is less accurate than
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>ASYNC. Perhaps it
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>delivers more timer interrupts than the other approaches,
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>and each interrupt
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>event causes a small accumulated error?
>     > >>>>>>>>
>     > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>if the latter is
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>actually more accurate then I can simply revert the
>     > changeset that
>     > >>>>>>>>implemented MIXED.
>     > >>>>>>>>
>     > >>>>>>>>Perhaps rather than running more of the same workloads you
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>could try idle
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>to /dev/null)? We
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>don't have any data on workloads that aren't CPU bound, so
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>that's really an
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>obvious place to put any further effort imo.
>     > >>>>>>>>
>     > >>>>>>>>-- Keir
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>_______________________________________________
>     > >>>>>>Xen-devel mailing list
>     > >>>>>>Xen-devel@lists.xensource.com
>     > >>>>>>http://lists.xensource.com/xen-devel
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>
>     > >>>
>     > >>>
>     > >>>
>     > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>     > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>     > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>     > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>     > >>
>     > >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>     > >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
>     > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>     > >>+        pt->do_not_freeze = 1;
>     > >>     else
>     > >>         pt->pending_intr_nr += missed_ticks;
>     > >>     pt->scheduled += missed_ticks * pt->period;
>     > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>     > >>
>     > >>     pt_lock(pt);
>     > >>
>     > >>-    pt->pending_intr_nr++;
>     > >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
>     > >>+        pt->pending_intr_nr = 1;
>     > >>+ pt->do_not_freeze = 0;
>     > >>+    }
>     > >>+    else
>     > >>+ pt->pending_intr_nr++;
>     > >>
>     > >>     if ( !pt->one_shot )
>     > >>     {
>     > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>     > >>         return;
>     > >>     }
>     > >>
>     > >>-    pt->do_not_freeze = 0;
>     > >>-
>     > >>     if ( pt->one_shot )
>     > >>     {
>     > >>         pt->enabled = 0;
>     > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>     > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>     > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>     > missed ticks */
>     > >>         }
>     > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>     > >>+     pt->pending_intr_nr--;
>     > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>     > >>+ }
>     > >>         else
>     > >>         {
>     > >>             pt->last_plt_gtime += pt->period_cycles;
>     > >>
>     > >>
>     > >
>     > >
>     > >
>     > >
>     >
>     >
>     > _______________________________________________
>     > Xen-devel mailing list
>     > Xen-devel@lists.xensource.com
>     > http://lists.xensource.com/xen-devel
>     >
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 27717 bytes --]

Hi Dave (Keir, see suggestion below) --

Thanks!

Turning off vhpet certainly helps a lot (though see below).

I wonder if timekeeping with vhpet is so bad that it should be
turned off by default (in 3.1, 3.2, and unstable) until it is
fixed?  (I have a patch that defaults it off, can post it if
there is agreement on the above point.)  The whole point of an
HPET is to provide more precise timekeeping and if vhpet is
worse than vpit, it can only confuse users.  Comments?

In your testing, are you just measuring % skew over a long
period of time?   We are graphing the skew continuously and
seeing periodic behavior that is unsettling, even with pit.
See attached.  Though your algorithm recovers, the "cliffs"
could still cause real user problems.  I wonder if there is
anything that can be done to make the "recovery" more
responsive?

We are looking into what part(s) of LTP is causing the cliffs.

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Monday, January 28, 2008 8:21 AM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; xen-devel@lists.xensource.com; 
> deepak.patel@oracle.com;
> akira.ijuin@oracle.com; Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan,
> 
> I guess I'm a bit out of date calling for clock= usage.
> Looking at linux 2.6.20.4 sources, I think you should specify
> "clocksource=pit nohpet" on the linux guest bootline.
> 
> You can leave the xen and dom0 bootlines as they are.
> The xen and guest clocksources do not need to be the same.
> In my tests, xen is using the hpet for its timekeeping and
> that appears to be the default.
> 
> When you boot the guests you should see
>     time.c: Using PIT/TSC based timekeeping.
> on the rh4u5-64 guest, and something similar on the others.
> 
>  > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>  > 14.318MHz HPET.)
> 
> This appears to be the xen state, which is fine.
> I was wrongly assuming that this was the guest state.
> You might want to look in your guest logs and see what they 
> were picking
> for a clock source.
> 
> Regards,
> Dave
> 
> 
> 
> 
> Dan Magenheimer wrote:
> 
> > Thanks, I hadn't realized that!  No wonder we didn't see the same
> > improvement you saw!
> >
> >> Try specifying clock=pit on the linux boot line...
> >
> > I'm confused... do you mean "clocksource=pit" on the Xen 
> command line or
> > "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or 
> dom0?) command
> > line?  Or both places?  Since the tests take awhile, it 
> would be nice
> > to get this right the first time.  Do the Xen and guest 
> clocksources need
> > to be the same?
> >
> > Thanks,
> > Dan
> >
> >  -----Original Message-----
> > *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> > *Sent:* Sunday, January 27, 2008 2:22 PM
> > *To:* dan.magenheimer@oracle.com; Keir Fraser
> > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
> > akira.ijuin@oracle.com; Dave Winchell
> > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
> > pending missed ticks
> >
> >     Hi Dan,
> >
> >     Hpet timer does have a fairly large error, as I was trying this
> >     one recently.
> >     I don't remember what I got for error, but 1% sounds 
> about right.
> >     The problem is that hpet is not built on top of vpt.c, 
> the module
> >     Keir and I did
> >     all the recent work in, for its periodic timer needs. Try
> >     specifying clock=pit
> >     on the linux boot line. If it still picks the hpet, which it
> >     might, let me know
> >     and I'll tell you how to get around this.
> >
> >     Regards,
> >     Dave
> >
> >
> >
> >
> >     
> --------------------------------------------------------------
> ----------
> >     *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >     *Sent:* Fri 1/25/2008 6:50 PM
> >     *To:* Dave Winchell; Keir Fraser
> >     *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
> >     akira.ijuin@oracle.com
> >     *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
> that disables
> >     pending missed ticks
> >
> >     Sorry for the very late followup on this but we finally 
> were able
> >     to get our testing set up again on stable 3.1 bits and have
> >     seen some very bad results on 3.1.3-rc1, on the order of 1%.
> >
> >     Test enviroment was a 4-socket dual core machine with 24GB of
> >     memory running six two-vcpu 2GB domains, four hvm plus two pv.
> >     All six guests were running LTP simultaneously.  The four hvm
> >     guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
> >     Timer_mode was set to 2 for 64-bit guests and 0 for 
> 32-bit guests.
> >     All four hvm guests experienced skew around -1%, even the 32-bit
> >     guest.  Less intensive testing didn't exhibit much skew at all.
> >
> >     A representative graph is attached.
> >
> >     Dave, I wonder if some portion of your patches didn't end up in
> >     the xen trees?
> >
> >     (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
> >     14.318MHz HPET.)
> >
> >     Thanks,
> >     Dan
> >
> >     P.S. Many thanks to Deepak and Akira for running tests.
> >
> >     > -----Original Message-----
> >     > From: xen-devel-bounces@lists.xensource.com
> >     > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >     > Dave Winchell
> >     > Sent: Wednesday, January 09, 2008 9:53 AM
> >     > To: Keir Fraser
> >     > Cc: dan.magenheimer@oracle.com; 
> xen-devel@lists.xensource.com; Dave
> >     > Winchell
> >     > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >     > disables pending
> >     > missed ticks
> >     >
> >     >
> >     > Hi Keir,
> >     >
> >     > The latest change, c/s 16690, looks fine.
> >     > I agree that the code in c/s 16690 is equivalent to
> >     > the code I submitted. Also, your version is more
> >     > concise.
> >     >
> >     > The error tests confirm the equivalence. With 
> overnight cpu loads,
> >     > the checked in version was accurate to +.048% for sles
> >     > and +.038% for red hat. My version was +.046% and +.032% in a
> >     > 2 hour test.
> >     > I don't think the difference is significant.
> >     >
> >     > i/o loads produced errors of +.01%.
> >     >
> >     > Thanks for all your efforts on this issue.
> >     >
> >     > Regards,
> >     > Dave
> >     >
> >     >
> >     >
> >     > Keir Fraser wrote:
> >     >
> >     > >Applied as c/s 16690, although the checked-in patch is
> >     > smaller. I think the
> >     > >only important fix is to pt_intr_post() and the only bit of
> >     > the patch I
> >     > >totally omitted was the change to pt_process_missed_ticks().
> >     > I don't think
> >     > >that change can be important, but let's see what 
> happens to the
> >     error
> >     > >percentage...
> >     > >
> >     > > -- Keir
> >     > >
> >     > >On 4/1/08 23:24, "Dave Winchell" 
> <dwinchell@virtualiron.com> wrote:
> >     > >
> >     > >
> >     > >
> >     > >>Hi Dan and Keir,
> >     > >>
> >     > >>Attached is a patch that fixes some issues with the 
> SYNC policy
> >     > >>(no_missed_ticks_pending).
> >     > >>I have not tried to make the change the minimal one, but,
> >     > rather, just
> >     > >>ported into
> >     > >>the new code what I know to work well. The error for
> >     > >>no_missed_ticks_pending goes from
> >     > >>over 3% to .03% with this change according to my testing.
> >     > >>
> >     > >>Regards,
> >     > >>Dave
> >     > >>
> >     > >>Dan Magenheimer wrote:
> >     > >>
> >     > >>
> >     > >>
> >     > >>>Hi Dave --
> >     > >>>
> >     > >>>Did you get your correction ported?  If so, it would be
> >     > nice to see this get
> >     > >>>into 3.1.3.
> >     > >>>
> >     > >>>Note that I just did some very limited testing with
> >     > timer_mode=2(=SYNC=no
> >     > >>>missed ticks pending)
> >     > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
> >     > worst error I've
> >     > >>>seen so far
> >     > >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
> >     > >>>
> >     > >>>Thanks,
> >     > >>>Dan
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>>-----Original Message-----
> >     > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >     > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >     > >>>>To: dan.magenheimer@oracle.com
> >     > >>>>Cc: Keir Fraser; Shan, Haitao;
> >     > xen-devel@lists.xensource.com; Dong,
> >     > >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >     > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >     > >>>>disables pending
> >     > >>>>missed ticks
> >     > >>>>
> >     > >>>>
> >     > >>>>Dan,
> >     > >>>>
> >     > >>>>I did some testing with the constant tsc offset 
> SYNC method
> >     > >>>>(now called
> >     > >>>>no_missed_ticks_pending)
> >     > >>>>and found the error to be very high, much larger 
> than 1 %, as
> >     > >>>>I recall.
> >     > >>>>I have not had a chance to submit a correction. I 
> will try to
> >     > >>>>do it later
> >     > >>>>this week or the first week in January. My version of
> >     constant tsc
> >     > >>>>offset SYNC method
> >     > >>>>produces .02 % error, so I just need to port that into the
> >     > >>>>current code.
> >     > >>>>
> >     > >>>>The error you got for both of those kernels is 
> what I would
> >     expect
> >     > >>>>for the default mode, delay_for_missed_ticks.
> >     > >>>>
> >     > >>>>I'll let Keir answer on how to set the time mode.
> >     > >>>>
> >     > >>>>Regards,
> >     > >>>>Dave
> >     > >>>>
> >     > >>>>Dan Magenheimer wrote:
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>Anyone make measurements on the final patch?
> >     > >>>>>
> >     > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>about 0.2% with no load.  This was xen-unstable tip today
> >     > >>>>with no options specified.  32-bit was about 0.01%.
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>I think I missed something... how do I run the various
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>accounting choices and which ones are known to be 
> appropriate
> >     > >>>>for which kernels?
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>Thanks,
> >     > >>>>>Dan
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>>-----Original Message-----
> >     > >>>>>>From: xen-devel-bounces@lists.xensource.com
> >     > 
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>Keir Fraser
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >     > >>>>>>To: Dave Winchell
> >     > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
> >     > Eddie; Jiang,
> >     > >>>>>>Yunhong
> >     > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >     > >>>>>>disables pending
> >     > >>>>>>missed ticks
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>Please take a look at xen-unstable changeset 16545.
> >     > >>>>>>
> >     > >>>>>>-- Keir
> >     > >>>>>>
> >     > >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>><dwinchell@virtualiron.com> wrote:
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>Keir,
> >     > >>>>>>>
> >     > >>>>>>>The accuracy data I've collected for i/o loads for the
> >     > >>>>>>>various time protocols follows. In addition, the data
> >     > >>>>>>>for cpu loads is shown.
> >     > >>>>>>>
> >     > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 
> processor AMD
> >     box.
> >     > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >     > >>>>>>>The cpu load is usex -e36 on each guest.
> >     > >>>>>>>(usex is available at
> >     http://people.redhat.com/anderson/usex.)
> >     > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 
> of=/dev/null.
> >     > >>>>>>>
> >     > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >     > >>>>>>>Also, these are run on 4 cpu AMD box.
> >     > >>>>>>>In addition, there is an idle rh-32bit guest.
> >     > >>>>>>>All three guests are 8vcpu.
> >     > >>>>>>>
> >     > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
> >     > >>>>>>>except that the redhat-64 guest has 4 instances of dd.
> >     > >>>>>>>
> >     > >>>>>>>Date Duration Protocol sles, rhat error load
> >     > >>>>>>>
> >     > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
> >     > +.005% cpu
> >     > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
> >     > +.012% cpu
> >     > >>>>>>>
> >     > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
> >     -.004% cpu
> >     > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, 
> -.005%, -.005% cpu
> >     > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, 
> -.008%, -.003% cpu
> >     > >>>>>>>
> >     > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, 
> -.040% cpu
> >     > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
> >     > -.031% cpu
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
> >     > -.09% i/o-8
> >     > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
> >     > -.14% i/o-8
> >     > >>>>>>>
> >     > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
> >     > -.022% i/o-8
> >     > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, 
> -.017%, -.018% i/o-8
> >     > >>>>>>>
> >     > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
> >     > -.029% i/o-8
> >     > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
> >     > -.031% i/o-8
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, 
> -.04% i/o-32
> >     > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
> >     > -.005% i/o-32
> >     > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, 
> -.11% i/o-32
> >     > >>>>>>>
> >     > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
> >     > .003% i/o-4/32
> >     > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
> >     > .01% i/o-4/32
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>Overhead measurements:
> >     > >>>>>>>
> >     > >>>>>>>Progress in terms of number of passes through a fixed
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>system workload
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >     > >>>>>>>The workload was usex -b48.
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >     > >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >     > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >     > >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>Conclusions:
> >     > >>>>>>>
> >     > >>>>>>>The only protocol which meets the .05% accuracy
> >     > requirement for ntp
> >     > >>>>>>>tracking under the loads
> >     > >>>>>>>above is the SYNC protocol. The worst case 
> accuracies for
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>SYNC, MIXED,
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>and ASYNC
> >     > >>>>>>>are .022%, .12%, and .14%, respectively.
> >     > >>>>>>>
> >     > >>>>>>>We could reduce the cost of the SYNC method by only
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>scheduling the extra
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>wakeups if a certain number
> >     > >>>>>>>of ticks are missed.
> >     > >>>>>>>
> >     > >>>>>>>Regards,
> >     > >>>>>>>Dave
> >     > >>>>>>>
> >     > >>>>>>>Keir Fraser wrote:
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>><dwinchell@virtualiron.com> wrote:
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>>Since I had a high error (~.03%) for the 
> ASYNC method a
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>couple of days ago,
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>I ran another ASYNC test. I think there may have
> >     > been something
> >     > >>>>>>>>>wrong with the code I used a couple of days ago for
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>ASYNC. It may have been
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>missing the immediate delivery of interrupt 
> after context
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>switch in.
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>My results indicate that either SYNC or ASYNC give
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>acceptable accuracy,
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>each running consistently around or under 
> .01%. MIXED has
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>a fairly high
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>error of
> >     > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>threshold for comfort.
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>I don't have an overnight run with SYNC. I 
> plan to leave
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>SYNC running
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>running instead.
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>It may be too early to pick the protocol and 
> I can run
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>more overnight tests
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>next week.
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>I'm a bit worried about any unwanted side 
> effects of the
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>SYNC+run_timer
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>approach -- e.g., whether timer wakeups will 
> cause higher
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>system-wide CPU
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>contention. I find it easier to think through the
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>implications of ASYNC. I'm
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>surprised that MIXED loses time, and is less 
> accurate than
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>ASYNC. Perhaps it
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>delivers more timer interrupts than the other 
> approaches,
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>and each interrupt
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>event causes a small accumulated error?
> >     > >>>>>>>>
> >     > >>>>>>>>Overall I would consider MIXED and ASYNC as 
> favourites and
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>if the latter is
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>actually more accurate then I can simply revert the
> >     > changeset that
> >     > >>>>>>>>implemented MIXED.
> >     > >>>>>>>>
> >     > >>>>>>>>Perhaps rather than running more of the same 
> workloads you
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>could try idle
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated 
> large disc reads
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>to /dev/null)? We
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>don't have any data on workloads that aren't 
> CPU bound, so
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>that's really an
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>obvious place to put any further effort imo.
> >     > >>>>>>>>
> >     > >>>>>>>>-- Keir
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>_______________________________________________
> >     > >>>>>>Xen-devel mailing list
> >     > >>>>>>Xen-devel@lists.xensource.com
> >     > >>>>>>http://lists.xensource.com/xen-devel
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >     > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >     > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >     > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >     > >>
> >     > >>     missed_ticks = missed_ticks / (s_time_t) 
> pt->period + 1;
> >     > >>     if ( mode_is(pt->vcpu->domain, 
> no_missed_ticks_pending) )
> >     > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >     > >>+        pt->do_not_freeze = 1;
> >     > >>     else
> >     > >>         pt->pending_intr_nr += missed_ticks;
> >     > >>     pt->scheduled += missed_ticks * pt->period;
> >     > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >     > >>
> >     > >>     pt_lock(pt);
> >     > >>
> >     > >>-    pt->pending_intr_nr++;
> >     > >>+    if ( mode_is(pt->vcpu->domain, 
> no_missed_ticks_pending) ) {
> >     > >>+        pt->pending_intr_nr = 1;
> >     > >>+ pt->do_not_freeze = 0;
> >     > >>+    }
> >     > >>+    else
> >     > >>+ pt->pending_intr_nr++;
> >     > >>
> >     > >>     if ( !pt->one_shot )
> >     > >>     {
> >     > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >     > >>         return;
> >     > >>     }
> >     > >>
> >     > >>-    pt->do_not_freeze = 0;
> >     > >>-
> >     > >>     if ( pt->one_shot )
> >     > >>     {
> >     > >>         pt->enabled = 0;
> >     > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu 
> *v, struct
> >     > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >     > >>             pt->pending_intr_nr = 0; /* 'collapse' all
> >     > missed ticks */
> >     > >>         }
> >     > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >     > >>+     pt->pending_intr_nr--;
> >     > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >     > >>+ }
> >     > >>         else
> >     > >>         {
> >     > >>             pt->last_plt_gtime += pt->period_cycles;
> >     > >>
> >     > >>
> >     > >
> >     > >
> >     > >
> >     > >
> >     >
> >     >
> >     > _______________________________________________
> >     > Xen-devel mailing list
> >     > Xen-devel@lists.xensource.com
> >     > http://lists.xensource.com/xen-devel
> >     >
> >
> 
>

[-- Attachment #2: el5u1-64-hvm-const-ltp.png --]
[-- Type: application/octet-stream, Size: 9273 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

I agree with you on disabling and the need for fixing hpet.
If no one else wants to fix it, I could give it a try.
I think we could make the hpet work better than pit/tsc.

While I monitor the skew over time, I'm not as methodical as you.
Its possible that I missed this behaviour.

I've been thinking about what you've seen and have no clue as to the cause.
Let me ask you a few questions:

Is the graph for RHEL5u1-64? (I've never tested this one.)
What was the behaviour of the other guests running?
If they had spikes, were they at the same wall time?
Were the other guests running ltp as well?

How are you measuring skew?

Are you running ntpd?

Anything that you can discover that would be in sync with
the spikes would be very helpful!

The code that I test with is our product code, which is based
on 3.1. So it is possible that something in 3.2 other than vpt.c
is the cause. I can test with 3.2, if necessary.

thanks,
Dave



Dan Magenheimer wrote:

>Hi Dave (Keir, see suggestion below) --
>
>Thanks!
>
>Turning off vhpet certainly helps a lot (though see below).
>
>I wonder if timekeeping with vhpet is so bad that it should be
>turned off by default (in 3.1, 3.2, and unstable) until it is
>fixed?  (I have a patch that defaults it off, can post it if
>there is agreement on the above point.)  The whole point of an
>HPET is to provide more precise timekeeping and if vhpet is
>worse than vpit, it can only confuse users.  Comments?
>  
>
>In your testing, are you just measuring % skew over a long
>period of time? 
>
>  We are graphing the skew continuously and
>seeing periodic behavior that is unsettling, even with pit.
>See attached.  Though your algorithm recovers, the "cliffs"
>could still cause real user problems.  I wonder if there is
>anything that can be done to make the "recovery" more
>responsive?
>
>We are looking into what part(s) of LTP is causing the cliffs.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Monday, January 28, 2008 8:21 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; xen-devel@lists.xensource.com; 
>>deepak.patel@oracle.com;
>>akira.ijuin@oracle.com; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I guess I'm a bit out of date calling for clock= usage.
>>Looking at linux 2.6.20.4 sources, I think you should specify
>>"clocksource=pit nohpet" on the linux guest bootline.
>>
>>You can leave the xen and dom0 bootlines as they are.
>>The xen and guest clocksources do not need to be the same.
>>In my tests, xen is using the hpet for its timekeeping and
>>that appears to be the default.
>>
>>When you boot the guests you should see
>>    time.c: Using PIT/TSC based timekeeping.
>>on the rh4u5-64 guest, and something similar on the others.
>>
>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>> > 14.318MHz HPET.)
>>
>>This appears to be the xen state, which is fine.
>>I was wrongly assuming that this was the guest state.
>>You might want to look in your guest logs and see what they 
>>were picking
>>for a clock source.
>>
>>Regards,
>>Dave
>>
>>
>>
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Thanks, I hadn't realized that!  No wonder we didn't see the same
>>>improvement you saw!
>>>
>>>      
>>>
>>>>Try specifying clock=pit on the linux boot line...
>>>>        
>>>>
>>>I'm confused... do you mean "clocksource=pit" on the Xen 
>>>      
>>>
>>command line or
>>    
>>
>>>"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or 
>>>      
>>>
>>dom0?) command
>>    
>>
>>>line?  Or both places?  Since the tests take awhile, it 
>>>      
>>>
>>would be nice
>>    
>>
>>>to get this right the first time.  Do the Xen and guest 
>>>      
>>>
>>clocksources need
>>    
>>
>>>to be the same?
>>>
>>>Thanks,
>>>Dan
>>>
>>> -----Original Message-----
>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>*Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>akira.ijuin@oracle.com; Dave Winchell
>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
>>>pending missed ticks
>>>
>>>    Hi Dan,
>>>
>>>    Hpet timer does have a fairly large error, as I was trying this
>>>    one recently.
>>>    I don't remember what I got for error, but 1% sounds 
>>>      
>>>
>>about right.
>>    
>>
>>>    The problem is that hpet is not built on top of vpt.c, 
>>>      
>>>
>>the module
>>    
>>
>>>    Keir and I did
>>>    all the recent work in, for its periodic timer needs. Try
>>>    specifying clock=pit
>>>    on the linux boot line. If it still picks the hpet, which it
>>>    might, let me know
>>>    and I'll tell you how to get around this.
>>>
>>>    Regards,
>>>    Dave
>>>
>>>
>>>
>>>
>>>    
>>>      
>>>
>>--------------------------------------------------------------
>>----------
>>    
>>
>>>    *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>    *Sent:* Fri 1/25/2008 6:50 PM
>>>    *To:* Dave Winchell; Keir Fraser
>>>    *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>    akira.ijuin@oracle.com
>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
>>>      
>>>
>>that disables
>>    
>>
>>>    pending missed ticks
>>>
>>>    Sorry for the very late followup on this but we finally 
>>>      
>>>
>>were able
>>    
>>
>>>    to get our testing set up again on stable 3.1 bits and have
>>>    seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>
>>>    Test enviroment was a 4-socket dual core machine with 24GB of
>>>    memory running six two-vcpu 2GB domains, four hvm plus two pv.
>>>    All six guests were running LTP simultaneously.  The four hvm
>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
>>>    Timer_mode was set to 2 for 64-bit guests and 0 for 
>>>      
>>>
>>32-bit guests.
>>    
>>
>>>    All four hvm guests experienced skew around -1%, even the 32-bit
>>>    guest.  Less intensive testing didn't exhibit much skew at all.
>>>
>>>    A representative graph is attached.
>>>
>>>    Dave, I wonder if some portion of your patches didn't end up in
>>>    the xen trees?
>>>
>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>    14.318MHz HPET.)
>>>
>>>    Thanks,
>>>    Dan
>>>
>>>    P.S. Many thanks to Deepak and Akira for running tests.
>>>
>>>    > -----Original Message-----
>>>    > From: xen-devel-bounces@lists.xensource.com
>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>    > Dave Winchell
>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
>>>    > To: Keir Fraser
>>>    > Cc: dan.magenheimer@oracle.com; 
>>>      
>>>
>>xen-devel@lists.xensource.com; Dave
>>    
>>
>>>    > Winchell
>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>    > disables pending
>>>    > missed ticks
>>>    >
>>>    >
>>>    > Hi Keir,
>>>    >
>>>    > The latest change, c/s 16690, looks fine.
>>>    > I agree that the code in c/s 16690 is equivalent to
>>>    > the code I submitted. Also, your version is more
>>>    > concise.
>>>    >
>>>    > The error tests confirm the equivalence. With 
>>>      
>>>
>>overnight cpu loads,
>>    
>>
>>>    > the checked in version was accurate to +.048% for sles
>>>    > and +.038% for red hat. My version was +.046% and +.032% in a
>>>    > 2 hour test.
>>>    > I don't think the difference is significant.
>>>    >
>>>    > i/o loads produced errors of +.01%.
>>>    >
>>>    > Thanks for all your efforts on this issue.
>>>    >
>>>    > Regards,
>>>    > Dave
>>>    >
>>>    >
>>>    >
>>>    > Keir Fraser wrote:
>>>    >
>>>    > >Applied as c/s 16690, although the checked-in patch is
>>>    > smaller. I think the
>>>    > >only important fix is to pt_intr_post() and the only bit of
>>>    > the patch I
>>>    > >totally omitted was the change to pt_process_missed_ticks().
>>>    > I don't think
>>>    > >that change can be important, but let's see what 
>>>      
>>>
>>happens to the
>>    
>>
>>>    error
>>>    > >percentage...
>>>    > >
>>>    > > -- Keir
>>>    > >
>>>    > >On 4/1/08 23:24, "Dave Winchell" 
>>>      
>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>    > >
>>>    > >
>>>    > >
>>>    > >>Hi Dan and Keir,
>>>    > >>
>>>    > >>Attached is a patch that fixes some issues with the 
>>>      
>>>
>>SYNC policy
>>    
>>
>>>    > >>(no_missed_ticks_pending).
>>>    > >>I have not tried to make the change the minimal one, but,
>>>    > rather, just
>>>    > >>ported into
>>>    > >>the new code what I know to work well. The error for
>>>    > >>no_missed_ticks_pending goes from
>>>    > >>over 3% to .03% with this change according to my testing.
>>>    > >>
>>>    > >>Regards,
>>>    > >>Dave
>>>    > >>
>>>    > >>Dan Magenheimer wrote:
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>>Hi Dave --
>>>    > >>>
>>>    > >>>Did you get your correction ported?  If so, it would be
>>>    > nice to see this get
>>>    > >>>into 3.1.3.
>>>    > >>>
>>>    > >>>Note that I just did some very limited testing with
>>>    > timer_mode=2(=SYNC=no
>>>    > >>>missed ticks pending)
>>>    > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
>>>    > worst error I've
>>>    > >>>seen so far
>>>    > >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>>>    > >>>
>>>    > >>>Thanks,
>>>    > >>>Dan
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>>-----Original Message-----
>>>    > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>    > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>    > >>>>To: dan.magenheimer@oracle.com
>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>    > xen-devel@lists.xensource.com; Dong,
>>>    > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>    > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>    > >>>>disables pending
>>>    > >>>>missed ticks
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>Dan,
>>>    > >>>>
>>>    > >>>>I did some testing with the constant tsc offset 
>>>      
>>>
>>SYNC method
>>    
>>
>>>    > >>>>(now called
>>>    > >>>>no_missed_ticks_pending)
>>>    > >>>>and found the error to be very high, much larger 
>>>      
>>>
>>than 1 %, as
>>    
>>
>>>    > >>>>I recall.
>>>    > >>>>I have not had a chance to submit a correction. I 
>>>      
>>>
>>will try to
>>    
>>
>>>    > >>>>do it later
>>>    > >>>>this week or the first week in January. My version of
>>>    constant tsc
>>>    > >>>>offset SYNC method
>>>    > >>>>produces .02 % error, so I just need to port that into the
>>>    > >>>>current code.
>>>    > >>>>
>>>    > >>>>The error you got for both of those kernels is 
>>>      
>>>
>>what I would
>>    
>>
>>>    expect
>>>    > >>>>for the default mode, delay_for_missed_ticks.
>>>    > >>>>
>>>    > >>>>I'll let Keir answer on how to set the time mode.
>>>    > >>>>
>>>    > >>>>Regards,
>>>    > >>>>Dave
>>>    > >>>>
>>>    > >>>>Dan Magenheimer wrote:
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>Anyone make measurements on the final patch?
>>>    > >>>>>
>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>about 0.2% with no load.  This was xen-unstable tip today
>>>    > >>>>with no options specified.  32-bit was about 0.01%.
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>I think I missed something... how do I run the various
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>accounting choices and which ones are known to be 
>>>      
>>>
>>appropriate
>>    
>>
>>>    > >>>>for which kernels?
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>Thanks,
>>>    > >>>>>Dan
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>>-----Original Message-----
>>>    > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>    > 
>>>      
>>>
>>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>                
>>>>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>Keir Fraser
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>    > >>>>>>To: Dave Winchell
>>>    > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>>    > Eddie; Jiang,
>>>    > >>>>>>Yunhong
>>>    > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>    > >>>>>>disables pending
>>>    > >>>>>>missed ticks
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>    > >>>>>>
>>>    > >>>>>>-- Keir
>>>    > >>>>>>
>>>    > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>><dwinchell@virtualiron.com> wrote:
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>Keir,
>>>    > >>>>>>>
>>>    > >>>>>>>The accuracy data I've collected for i/o loads for the
>>>    > >>>>>>>various time protocols follows. In addition, the data
>>>    > >>>>>>>for cpu loads is shown.
>>>    > >>>>>>>
>>>    > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 
>>>      
>>>
>>processor AMD
>>    
>>
>>>    box.
>>>    > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>    > >>>>>>>The cpu load is usex -e36 on each guest.
>>>    > >>>>>>>(usex is available at
>>>    http://people.redhat.com/anderson/usex.)
>>>    > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 
>>>      
>>>
>>of=/dev/null.
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>    > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>    > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>    > >>>>>>>All three guests are 8vcpu.
>>>    > >>>>>>>
>>>    > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>    > >>>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>    > >>>>>>>
>>>    > >>>>>>>Date Duration Protocol sles, rhat error load
>>>    > >>>>>>>
>>>    > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
>>>    > +.005% cpu
>>>    > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
>>>    > +.012% cpu
>>>    > >>>>>>>
>>>    > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>    -.004% cpu
>>>    > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, 
>>>      
>>>
>>-.005%, -.005% cpu
>>    
>>
>>>    > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, 
>>>      
>>>
>>-.008%, -.003% cpu
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, 
>>>      
>>>
>>-.040% cpu
>>    
>>
>>>    > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
>>>    > -.031% cpu
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>    > -.09% i/o-8
>>>    > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>    > -.14% i/o-8
>>>    > >>>>>>>
>>>    > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>    > -.022% i/o-8
>>>    > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, 
>>>      
>>>
>>-.017%, -.018% i/o-8
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>    > -.029% i/o-8
>>>    > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
>>>    > -.031% i/o-8
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, 
>>>      
>>>
>>-.04% i/o-32
>>    
>>
>>>    > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>    > -.005% i/o-32
>>>    > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, 
>>>      
>>>
>>-.11% i/o-32
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>    > .003% i/o-4/32
>>>    > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>    > .01% i/o-4/32
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>Overhead measurements:
>>>    > >>>>>>>
>>>    > >>>>>>>Progress in terms of number of passes through a fixed
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>system workload
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>    > >>>>>>>The workload was usex -b48.
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>    > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>    > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>    > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>Conclusions:
>>>    > >>>>>>>
>>>    > >>>>>>>The only protocol which meets the .05% accuracy
>>>    > requirement for ntp
>>>    > >>>>>>>tracking under the loads
>>>    > >>>>>>>above is the SYNC protocol. The worst case 
>>>      
>>>
>>accuracies for
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>SYNC, MIXED,
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>and ASYNC
>>>    > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>    > >>>>>>>
>>>    > >>>>>>>We could reduce the cost of the SYNC method by only
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>scheduling the extra
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>wakeups if a certain number
>>>    > >>>>>>>of ticks are missed.
>>>    > >>>>>>>
>>>    > >>>>>>>Regards,
>>>    > >>>>>>>Dave
>>>    > >>>>>>>
>>>    > >>>>>>>Keir Fraser wrote:
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>><dwinchell@virtualiron.com> wrote:
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>>Since I had a high error (~.03%) for the 
>>>      
>>>
>>ASYNC method a
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>couple of days ago,
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>    > been something
>>>    > >>>>>>>>>wrong with the code I used a couple of days ago for
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>ASYNC. It may have been
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>missing the immediate delivery of interrupt 
>>>      
>>>
>>after context
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>switch in.
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>acceptable accuracy,
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>each running consistently around or under 
>>>      
>>>
>>.01%. MIXED has
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>a fairly high
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>error of
>>>    > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>threshold for comfort.
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>I don't have an overnight run with SYNC. I 
>>>      
>>>
>>plan to leave
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>SYNC running
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>running instead.
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>It may be too early to pick the protocol and 
>>>      
>>>
>>I can run
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>more overnight tests
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>next week.
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>I'm a bit worried about any unwanted side 
>>>      
>>>
>>effects of the
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>SYNC+run_timer
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>approach -- e.g., whether timer wakeups will 
>>>      
>>>
>>cause higher
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>system-wide CPU
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>contention. I find it easier to think through the
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>implications of ASYNC. I'm
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>surprised that MIXED loses time, and is less 
>>>      
>>>
>>accurate than
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>ASYNC. Perhaps it
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>delivers more timer interrupts than the other 
>>>      
>>>
>>approaches,
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>and each interrupt
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>event causes a small accumulated error?
>>>    > >>>>>>>>
>>>    > >>>>>>>>Overall I would consider MIXED and ASYNC as 
>>>      
>>>
>>favourites and
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>if the latter is
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>actually more accurate then I can simply revert the
>>>    > changeset that
>>>    > >>>>>>>>implemented MIXED.
>>>    > >>>>>>>>
>>>    > >>>>>>>>Perhaps rather than running more of the same 
>>>      
>>>
>>workloads you
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>could try idle
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated 
>>>      
>>>
>>large disc reads
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>to /dev/null)? We
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>don't have any data on workloads that aren't 
>>>      
>>>
>>CPU bound, so
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>that's really an
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>obvious place to put any further effort imo.
>>>    > >>>>>>>>
>>>    > >>>>>>>>-- Keir
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>_______________________________________________
>>>    > >>>>>>Xen-devel mailing list
>>>    > >>>>>>Xen-devel@lists.xensource.com
>>>    > >>>>>>http://lists.xensource.com/xen-devel
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>>    > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>>>    > >>
>>>    > >>     missed_ticks = missed_ticks / (s_time_t) 
>>>      
>>>
>>pt->period + 1;
>>    
>>
>>>    > >>     if ( mode_is(pt->vcpu->domain, 
>>>      
>>>
>>no_missed_ticks_pending) )
>>    
>>
>>>    > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>    > >>+        pt->do_not_freeze = 1;
>>>    > >>     else
>>>    > >>         pt->pending_intr_nr += missed_ticks;
>>>    > >>     pt->scheduled += missed_ticks * pt->period;
>>>    > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>    > >>
>>>    > >>     pt_lock(pt);
>>>    > >>
>>>    > >>-    pt->pending_intr_nr++;
>>>    > >>+    if ( mode_is(pt->vcpu->domain, 
>>>      
>>>
>>no_missed_ticks_pending) ) {
>>    
>>
>>>    > >>+        pt->pending_intr_nr = 1;
>>>    > >>+ pt->do_not_freeze = 0;
>>>    > >>+    }
>>>    > >>+    else
>>>    > >>+ pt->pending_intr_nr++;
>>>    > >>
>>>    > >>     if ( !pt->one_shot )
>>>    > >>     {
>>>    > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>>    > >>         return;
>>>    > >>     }
>>>    > >>
>>>    > >>-    pt->do_not_freeze = 0;
>>>    > >>-
>>>    > >>     if ( pt->one_shot )
>>>    > >>     {
>>>    > >>         pt->enabled = 0;
>>>    > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu 
>>>      
>>>
>>*v, struct
>>    
>>
>>>    > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>    > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>    > missed ticks */
>>>    > >>         }
>>>    > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>>    > >>+     pt->pending_intr_nr--;
>>>    > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>    > >>+ }
>>>    > >>         else
>>>    > >>         {
>>>    > >>             pt->last_plt_gtime += pt->period_cycles;
>>>    > >>
>>>    > >>
>>>    > >
>>>    > >
>>>    > >
>>>    > >
>>>    >
>>>    >
>>>    > _______________________________________________
>>>    > Xen-devel mailing list
>>>    > Xen-devel@lists.xensource.com
>>>    > http://lists.xensource.com/xen-devel
>>>    >
>>>
>>>      
>>>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Deepak Patel]

[-- Attachment #1: Type: text/plain, Size: 29297 bytes --]


>
> Is the graph for RHEL5u1-64? (I've never tested this one.)

I do not know which graph was attached with this. But I saw this 
behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I was 
running ltp tests continuously.

> What was the behaviour of the other guests running?

All pvm guests are fine. But behavior of most of the hvm guests were as 
described.

> If they had spikes, were they at the same wall time?

No. They are not at the same wall time.

> Were the other guests running ltp as well?
>
Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp continuously.

> How are you measuring skew?

I was collecting output of "ntpdate -q  <timeserver> every 300 seconds 
(5 minutes) and have created graph based on that.

>
> Are you running ntpd?
>
Yes. ntp was running on all the guests.

I am investigating what causes this spikes and let everyone know what 
are my findings.

Thanks,
Deepak

> Anything that you can discover that would be in sync with
> the spikes would be very helpful!
>
> The code that I test with is our product code, which is based
> on 3.1. So it is possible that something in 3.2 other than vpt.c
> is the cause. I can test with 3.2, if necessary.
>
> thanks,
> Dave
>
>
>
> Dan Magenheimer wrote:
>
>> Hi Dave (Keir, see suggestion below) --
>>
>> Thanks!
>>
>> Turning off vhpet certainly helps a lot (though see below).
>>
>> I wonder if timekeeping with vhpet is so bad that it should be
>> turned off by default (in 3.1, 3.2, and unstable) until it is
>> fixed?  (I have a patch that defaults it off, can post it if
>> there is agreement on the above point.)  The whole point of an
>> HPET is to provide more precise timekeeping and if vhpet is
>> worse than vpit, it can only confuse users.  Comments?
>>  
>>
>> In your testing, are you just measuring % skew over a long
>> period of time?
>>  We are graphing the skew continuously and
>> seeing periodic behavior that is unsettling, even with pit.
>> See attached.  Though your algorithm recovers, the "cliffs"
>> could still cause real user problems.  I wonder if there is
>> anything that can be done to make the "recovery" more
>> responsive?
>>
>> We are looking into what part(s) of LTP is causing the cliffs.
>>
>> Thanks,
>> Dan
>>
>>  
>>
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Monday, January 28, 2008 8:21 AM
>>> To: dan.magenheimer@oracle.com
>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; 
>>> deepak.patel@oracle.com;
>>> akira.ijuin@oracle.com; Dave Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending
>>> missed ticks
>>>
>>>
>>> Dan,
>>>
>>> I guess I'm a bit out of date calling for clock= usage.
>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>
>>> You can leave the xen and dom0 bootlines as they are.
>>> The xen and guest clocksources do not need to be the same.
>>> In my tests, xen is using the hpet for its timekeeping and
>>> that appears to be the default.
>>>
>>> When you boot the guests you should see
>>>    time.c: Using PIT/TSC based timekeeping.
>>> on the rh4u5-64 guest, and something similar on the others.
>>>
>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>> > 14.318MHz HPET.)
>>>
>>> This appears to be the xen state, which is fine.
>>> I was wrongly assuming that this was the guest state.
>>> You might want to look in your guest logs and see what they were 
>>> picking
>>> for a clock source.
>>>
>>> Regards,
>>> Dave
>>>
>>>
>>>
>>>
>>> Dan Magenheimer wrote:
>>>
>>>   
>>>
>>>> Thanks, I hadn't realized that!  No wonder we didn't see the same
>>>> improvement you saw!
>>>>
>>>>     
>>>>
>>>>> Try specifying clock=pit on the linux boot line...
>>>>>       
>>>>
>>>> I'm confused... do you mean "clocksource=pit" on the Xen     
>>>
>>> command line or
>>>   
>>>
>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or     
>>>
>>> dom0?) command
>>>   
>>>
>>>> line?  Or both places?  Since the tests take awhile, it     
>>>
>>> would be nice
>>>   
>>>
>>>> to get this right the first time.  Do the Xen and guest     
>>>
>>> clocksources need
>>>   
>>>
>>>> to be the same?
>>>>
>>>> Thanks,
>>>> Dan
>>>>
>>>> -----Original Message-----
>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>> akira.ijuin@oracle.com; Dave Winchell
>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
>>>> pending missed ticks
>>>>
>>>>    Hi Dan,
>>>>
>>>>    Hpet timer does have a fairly large error, as I was trying this
>>>>    one recently.
>>>>    I don't remember what I got for error, but 1% sounds     
>>>
>>> about right.
>>>   
>>>
>>>>    The problem is that hpet is not built on top of vpt.c,     
>>>
>>> the module
>>>   
>>>
>>>>    Keir and I did
>>>>    all the recent work in, for its periodic timer needs. Try
>>>>    specifying clock=pit
>>>>    on the linux boot line. If it still picks the hpet, which it
>>>>    might, let me know
>>>>    and I'll tell you how to get around this.
>>>>
>>>>    Regards,
>>>>    Dave
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>
>>> --------------------------------------------------------------
>>> ----------
>>>   
>>>
>>>>    *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>    *Sent:* Fri 1/25/2008 6:50 PM
>>>>    *To:* Dave Winchell; Keir Fraser
>>>>    *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>    akira.ijuin@oracle.com
>>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode     
>>>
>>> that disables
>>>   
>>>
>>>>    pending missed ticks
>>>>
>>>>    Sorry for the very late followup on this but we finally     
>>>
>>> were able
>>>   
>>>
>>>>    to get our testing set up again on stable 3.1 bits and have
>>>>    seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>
>>>>    Test enviroment was a 4-socket dual core machine with 24GB of
>>>>    memory running six two-vcpu 2GB domains, four hvm plus two pv.
>>>>    All six guests were running LTP simultaneously.  The four hvm
>>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
>>>>    Timer_mode was set to 2 for 64-bit guests and 0 for     
>>>
>>> 32-bit guests.
>>>   
>>>
>>>>    All four hvm guests experienced skew around -1%, even the 32-bit
>>>>    guest.  Less intensive testing didn't exhibit much skew at all.
>>>>
>>>>    A representative graph is attached.
>>>>
>>>>    Dave, I wonder if some portion of your patches didn't end up in
>>>>    the xen trees?
>>>>
>>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>    14.318MHz HPET.)
>>>>
>>>>    Thanks,
>>>>    Dan
>>>>
>>>>    P.S. Many thanks to Deepak and Akira for running tests.
>>>>
>>>>    > -----Original Message-----
>>>>    > From: xen-devel-bounces@lists.xensource.com
>>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>    > Dave Winchell
>>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>    > To: Keir Fraser
>>>>    > Cc: dan.magenheimer@oracle.com;     
>>>
>>> xen-devel@lists.xensource.com; Dave
>>>   
>>>
>>>>    > Winchell
>>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>    > disables pending
>>>>    > missed ticks
>>>>    >
>>>>    >
>>>>    > Hi Keir,
>>>>    >
>>>>    > The latest change, c/s 16690, looks fine.
>>>>    > I agree that the code in c/s 16690 is equivalent to
>>>>    > the code I submitted. Also, your version is more
>>>>    > concise.
>>>>    >
>>>>    > The error tests confirm the equivalence. With     
>>>
>>> overnight cpu loads,
>>>   
>>>
>>>>    > the checked in version was accurate to +.048% for sles
>>>>    > and +.038% for red hat. My version was +.046% and +.032% in a
>>>>    > 2 hour test.
>>>>    > I don't think the difference is significant.
>>>>    >
>>>>    > i/o loads produced errors of +.01%.
>>>>    >
>>>>    > Thanks for all your efforts on this issue.
>>>>    >
>>>>    > Regards,
>>>>    > Dave
>>>>    >
>>>>    >
>>>>    >
>>>>    > Keir Fraser wrote:
>>>>    >
>>>>    > >Applied as c/s 16690, although the checked-in patch is
>>>>    > smaller. I think the
>>>>    > >only important fix is to pt_intr_post() and the only bit of
>>>>    > the patch I
>>>>    > >totally omitted was the change to pt_process_missed_ticks().
>>>>    > I don't think
>>>>    > >that change can be important, but let's see what     
>>>
>>> happens to the
>>>   
>>>
>>>>    error
>>>>    > >percentage...
>>>>    > >
>>>>    > > -- Keir
>>>>    > >
>>>>    > >On 4/1/08 23:24, "Dave Winchell"     
>>>
>>> <dwinchell@virtualiron.com> wrote:
>>>   
>>>
>>>>    > >
>>>>    > >
>>>>    > >
>>>>    > >>Hi Dan and Keir,
>>>>    > >>
>>>>    > >>Attached is a patch that fixes some issues with the     
>>>
>>> SYNC policy
>>>   
>>>
>>>>    > >>(no_missed_ticks_pending).
>>>>    > >>I have not tried to make the change the minimal one, but,
>>>>    > rather, just
>>>>    > >>ported into
>>>>    > >>the new code what I know to work well. The error for
>>>>    > >>no_missed_ticks_pending goes from
>>>>    > >>over 3% to .03% with this change according to my testing.
>>>>    > >>
>>>>    > >>Regards,
>>>>    > >>Dave
>>>>    > >>
>>>>    > >>Dan Magenheimer wrote:
>>>>    > >>
>>>>    > >>
>>>>    > >>
>>>>    > >>>Hi Dave --
>>>>    > >>>
>>>>    > >>>Did you get your correction ported?  If so, it would be
>>>>    > nice to see this get
>>>>    > >>>into 3.1.3.
>>>>    > >>>
>>>>    > >>>Note that I just did some very limited testing with
>>>>    > timer_mode=2(=SYNC=no
>>>>    > >>>missed ticks pending)
>>>>    > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
>>>>    > worst error I've
>>>>    > >>>seen so far
>>>>    > >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>>>>    > >>>
>>>>    > >>>Thanks,
>>>>    > >>>Dan
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>>-----Original Message-----
>>>>    > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>    > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>    > >>>>To: dan.magenheimer@oracle.com
>>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>    > xen-devel@lists.xensource.com; Dong,
>>>>    > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>    > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>    > >>>>disables pending
>>>>    > >>>>missed ticks
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>Dan,
>>>>    > >>>>
>>>>    > >>>>I did some testing with the constant tsc offset     
>>>
>>> SYNC method
>>>   
>>>
>>>>    > >>>>(now called
>>>>    > >>>>no_missed_ticks_pending)
>>>>    > >>>>and found the error to be very high, much larger     
>>>
>>> than 1 %, as
>>>   
>>>
>>>>    > >>>>I recall.
>>>>    > >>>>I have not had a chance to submit a correction. I     
>>>
>>> will try to
>>>   
>>>
>>>>    > >>>>do it later
>>>>    > >>>>this week or the first week in January. My version of
>>>>    constant tsc
>>>>    > >>>>offset SYNC method
>>>>    > >>>>produces .02 % error, so I just need to port that into the
>>>>    > >>>>current code.
>>>>    > >>>>
>>>>    > >>>>The error you got for both of those kernels is     
>>>
>>> what I would
>>>   
>>>
>>>>    expect
>>>>    > >>>>for the default mode, delay_for_missed_ticks.
>>>>    > >>>>
>>>>    > >>>>I'll let Keir answer on how to set the time mode.
>>>>    > >>>>
>>>>    > >>>>Regards,
>>>>    > >>>>Dave
>>>>    > >>>>
>>>>    > >>>>Dan Magenheimer wrote:
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>Anyone make measurements on the final patch?
>>>>    > >>>>>
>>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>about 0.2% with no load.  This was xen-unstable tip today
>>>>    > >>>>with no options specified.  32-bit was about 0.01%.
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>I think I missed something... how do I run the various
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>accounting choices and which ones are known to be     
>>>
>>> appropriate
>>>   
>>>
>>>>    > >>>>for which kernels?
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>Thanks,
>>>>    > >>>>>Dan
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>>-----Original Message-----
>>>>    > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>    >     
>>>>
>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>               
>>>>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>Keir Fraser
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>    > >>>>>>To: Dave Winchell
>>>>    > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>>>    > Eddie; Jiang,
>>>>    > >>>>>>Yunhong
>>>>    > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>    > >>>>>>disables pending
>>>>    > >>>>>>missed ticks
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>>    > >>>>>>
>>>>    > >>>>>>-- Keir
>>>>    > >>>>>>
>>>>    > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>><dwinchell@virtualiron.com> wrote:
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>Keir,
>>>>    > >>>>>>>
>>>>    > >>>>>>>The accuracy data I've collected for i/o loads for the
>>>>    > >>>>>>>various time protocols follows. In addition, the data
>>>>    > >>>>>>>for cpu loads is shown.
>>>>    > >>>>>>>
>>>>    > >>>>>>>The loads labeled cpu and i/o-8 are on an 8     
>>>
>>> processor AMD
>>>   
>>>
>>>>    box.
>>>>    > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>    > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>    > >>>>>>>(usex is available at
>>>>    http://people.redhat.com/anderson/usex.)
>>>>    > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6     
>>>
>>> of=/dev/null.
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>    > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>    > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>    > >>>>>>>All three guests are 8vcpu.
>>>>    > >>>>>>>
>>>>    > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>    > >>>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>    > >>>>>>>
>>>>    > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
>>>>    > +.005% cpu
>>>>    > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
>>>>    > +.012% cpu
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>    -.004% cpu
>>>>    > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,     
>>>
>>> -.005%, -.005% cpu
>>>   
>>>
>>>>    > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,     
>>>
>>> -.008%, -.003% cpu
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,     
>>>
>>> -.040% cpu
>>>   
>>>
>>>>    > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
>>>>    > -.031% cpu
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>    > -.09% i/o-8
>>>>    > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>    > -.14% i/o-8
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>    > -.022% i/o-8
>>>>    > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,     
>>>
>>> -.017%, -.018% i/o-8
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>    > -.029% i/o-8
>>>>    > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
>>>>    > -.031% i/o-8
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,     
>>>
>>> -.04% i/o-32
>>>   
>>>
>>>>    > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>    > -.005% i/o-32
>>>>    > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,     
>>>
>>> -.11% i/o-32
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>    > .003% i/o-4/32
>>>>    > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>    > .01% i/o-4/32
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>Overhead measurements:
>>>>    > >>>>>>>
>>>>    > >>>>>>>Progress in terms of number of passes through a fixed
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>system workload
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>    > >>>>>>>The workload was usex -b48.
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>    > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>    > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>    > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>Conclusions:
>>>>    > >>>>>>>
>>>>    > >>>>>>>The only protocol which meets the .05% accuracy
>>>>    > requirement for ntp
>>>>    > >>>>>>>tracking under the loads
>>>>    > >>>>>>>above is the SYNC protocol. The worst case     
>>>
>>> accuracies for
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>SYNC, MIXED,
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>and ASYNC
>>>>    > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>    > >>>>>>>
>>>>    > >>>>>>>We could reduce the cost of the SYNC method by only
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>scheduling the extra
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>wakeups if a certain number
>>>>    > >>>>>>>of ticks are missed.
>>>>    > >>>>>>>
>>>>    > >>>>>>>Regards,
>>>>    > >>>>>>>Dave
>>>>    > >>>>>>>
>>>>    > >>>>>>>Keir Fraser wrote:
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>>Since I had a high error (~.03%) for the     
>>>
>>> ASYNC method a
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>couple of days ago,
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>>    > been something
>>>>    > >>>>>>>>>wrong with the code I used a couple of days ago for
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>ASYNC. It may have been
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>missing the immediate delivery of interrupt     
>>>
>>> after context
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>switch in.
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>acceptable accuracy,
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>each running consistently around or under     
>>>
>>> .01%. MIXED has
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>a fairly high
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>error of
>>>>    > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>threshold for comfort.
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>I don't have an overnight run with SYNC. I     
>>>
>>> plan to leave
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>SYNC running
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>running instead.
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>It may be too early to pick the protocol and     
>>>
>>> I can run
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>more overnight tests
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>next week.
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>I'm a bit worried about any unwanted side     
>>>
>>> effects of the
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>SYNC+run_timer
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>approach -- e.g., whether timer wakeups will     
>>>
>>> cause higher
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>system-wide CPU
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>contention. I find it easier to think through the
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>implications of ASYNC. I'm
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>surprised that MIXED loses time, and is less     
>>>
>>> accurate than
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>ASYNC. Perhaps it
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>delivers more timer interrupts than the other     
>>>
>>> approaches,
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>and each interrupt
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>event causes a small accumulated error?
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>Overall I would consider MIXED and ASYNC as     
>>>
>>> favourites and
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>if the latter is
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>actually more accurate then I can simply revert the
>>>>    > changeset that
>>>>    > >>>>>>>>implemented MIXED.
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>Perhaps rather than running more of the same     
>>>
>>> workloads you
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>could try idle
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated     
>>>
>>> large disc reads
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>to /dev/null)? We
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>don't have any data on workloads that aren't     
>>>
>>> CPU bound, so
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>that's really an
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>obvious place to put any further effort imo.
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>-- Keir
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>_______________________________________________
>>>>    > >>>>>>Xen-devel mailing list
>>>>    > >>>>>>Xen-devel@lists.xensource.com
>>>>    > >>>>>>http://lists.xensource.com/xen-devel
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>>>    > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>>>>    > >>
>>>>    > >>     missed_ticks = missed_ticks / (s_time_t)     
>>>
>>> pt->period + 1;
>>>   
>>>
>>>>    > >>     if ( mode_is(pt->vcpu->domain,     
>>>
>>> no_missed_ticks_pending) )
>>>   
>>>
>>>>    > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>    > >>+        pt->do_not_freeze = 1;
>>>>    > >>     else
>>>>    > >>         pt->pending_intr_nr += missed_ticks;
>>>>    > >>     pt->scheduled += missed_ticks * pt->period;
>>>>    > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>>    > >>
>>>>    > >>     pt_lock(pt);
>>>>    > >>
>>>>    > >>-    pt->pending_intr_nr++;
>>>>    > >>+    if ( mode_is(pt->vcpu->domain,     
>>>
>>> no_missed_ticks_pending) ) {
>>>   
>>>
>>>>    > >>+        pt->pending_intr_nr = 1;
>>>>    > >>+ pt->do_not_freeze = 0;
>>>>    > >>+    }
>>>>    > >>+    else
>>>>    > >>+ pt->pending_intr_nr++;
>>>>    > >>
>>>>    > >>     if ( !pt->one_shot )
>>>>    > >>     {
>>>>    > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>>>    > >>         return;
>>>>    > >>     }
>>>>    > >>
>>>>    > >>-    pt->do_not_freeze = 0;
>>>>    > >>-
>>>>    > >>     if ( pt->one_shot )
>>>>    > >>     {
>>>>    > >>         pt->enabled = 0;
>>>>    > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu     
>>>
>>> *v, struct
>>>   
>>>
>>>>    > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>    > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>>    > missed ticks */
>>>>    > >>         }
>>>>    > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>>>    > >>+     pt->pending_intr_nr--;
>>>>    > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>    > >>+ }
>>>>    > >>         else
>>>>    > >>         {
>>>>    > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>    > >>
>>>>    > >>
>>>>    > >
>>>>    > >
>>>>    > >
>>>>    > >
>>>>    >
>>>>    >
>>>>    > _______________________________________________
>>>>    > Xen-devel mailing list
>>>>    > Xen-devel@lists.xensource.com
>>>>    > http://lists.xensource.com/xen-devel
>>>>    >
>>>>
>>>>     
>>>
>

[-- Attachment #2: deepak.patel.vcf --]
[-- Type: text/x-vcard, Size: 105 bytes --]

begin:vcard
fn:Deepak Patel
n:Patel;Deepak
email;internet:deepak.patel@oracle.com
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Dan, Deeepak,

It may be that the underlying clock error is too great for ntp
to handle. It would be useful if you did not run ntpd
and, instead did ntpdate -b <timeserver> at the start of the test
for each guest. Then capture the data as you have been doing.
If the drift is greater than .05%, then we need to address that.

Another option is, when running ntpd, to enable loop statistics in 
/etc/ntp.conf
by adding this to the file:

statistics loopstats
statsdir /var/lib/ntp/

Then you will see loop data in that directory.
Correlating the data in the loopstats files with the
peaks in skew would be interesting. You will see entries of the form

54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10

Where the second to last column is the Allan Deviation. When that
gets over 1000, ntpd is working pretty hard. However, I have not seen ntpd
completely loose it like you have.

I'm on vacation until Monday, and won't be reading
email.

Thanks for all your work on this!

-Dave

Deepak Patel wrote:

>
>>
>> Is the graph for RHEL5u1-64? (I've never tested this one.)
>
>
> I do not know which graph was attached with this. But I saw this 
> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I 
> was running ltp tests continuously.
>
>> What was the behaviour of the other guests running?
>
>
> All pvm guests are fine. But behavior of most of the hvm guests were 
> as described.
>
>> If they had spikes, were they at the same wall time?
>
>
> No. They are not at the same wall time.
>
>> Were the other guests running ltp as well?
>>
> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp 
> continuously.
>
>> How are you measuring skew?
>
>
> I was collecting output of "ntpdate -q  <timeserver> every 300 seconds 
> (5 minutes) and have created graph based on that.
>
>>
>> Are you running ntpd?
>>
> Yes. ntp was running on all the guests.
>
> I am investigating what causes this spikes and let everyone know what 
> are my findings.
>
> Thanks,
> Deepak
>
>> Anything that you can discover that would be in sync with
>> the spikes would be very helpful!
>>
>> The code that I test with is our product code, which is based
>> on 3.1. So it is possible that something in 3.2 other than vpt.c
>> is the cause. I can test with 3.2, if necessary.
>>
>> thanks,
>> Dave
>>
>>
>>
>> Dan Magenheimer wrote:
>>
>>> Hi Dave (Keir, see suggestion below) --
>>>
>>> Thanks!
>>>
>>> Turning off vhpet certainly helps a lot (though see below).
>>>
>>> I wonder if timekeeping with vhpet is so bad that it should be
>>> turned off by default (in 3.1, 3.2, and unstable) until it is
>>> fixed?  (I have a patch that defaults it off, can post it if
>>> there is agreement on the above point.)  The whole point of an
>>> HPET is to provide more precise timekeeping and if vhpet is
>>> worse than vpit, it can only confuse users.  Comments?
>>>  
>>>
>>> In your testing, are you just measuring % skew over a long
>>> period of time?
>>>  We are graphing the skew continuously and
>>> seeing periodic behavior that is unsettling, even with pit.
>>> See attached.  Though your algorithm recovers, the "cliffs"
>>> could still cause real user problems.  I wonder if there is
>>> anything that can be done to make the "recovery" more
>>> responsive?
>>>
>>> We are looking into what part(s) of LTP is causing the cliffs.
>>>
>>> Thanks,
>>> Dan
>>>
>>>  
>>>
>>>> -----Original Message-----
>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>> To: dan.magenheimer@oracle.com
>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; 
>>>> deepak.patel@oracle.com;
>>>> akira.ijuin@oracle.com; Dave Winchell
>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables 
>>>> pending
>>>> missed ticks
>>>>
>>>>
>>>> Dan,
>>>>
>>>> I guess I'm a bit out of date calling for clock= usage.
>>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>>
>>>> You can leave the xen and dom0 bootlines as they are.
>>>> The xen and guest clocksources do not need to be the same.
>>>> In my tests, xen is using the hpet for its timekeeping and
>>>> that appears to be the default.
>>>>
>>>> When you boot the guests you should see
>>>>    time.c: Using PIT/TSC based timekeeping.
>>>> on the rh4u5-64 guest, and something similar on the others.
>>>>
>>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>> > 14.318MHz HPET.)
>>>>
>>>> This appears to be the xen state, which is fine.
>>>> I was wrongly assuming that this was the guest state.
>>>> You might want to look in your guest logs and see what they were 
>>>> picking
>>>> for a clock source.
>>>>
>>>> Regards,
>>>> Dave
>>>>
>>>>
>>>>
>>>>
>>>> Dan Magenheimer wrote:
>>>>
>>>>  
>>>>
>>>>> Thanks, I hadn't realized that!  No wonder we didn't see the same
>>>>> improvement you saw!
>>>>>
>>>>>    
>>>>>
>>>>>> Try specifying clock=pit on the linux boot line...
>>>>>>       
>>>>>
>>>>>
>>>>> I'm confused... do you mean "clocksource=pit" on the Xen     
>>>>
>>>>
>>>> command line or
>>>>  
>>>>
>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or     
>>>>
>>>>
>>>> dom0?) command
>>>>  
>>>>
>>>>> line?  Or both places?  Since the tests take awhile, it     
>>>>
>>>>
>>>> would be nice
>>>>  
>>>>
>>>>> to get this right the first time.  Do the Xen and guest     
>>>>
>>>>
>>>> clocksources need
>>>>  
>>>>
>>>>> to be the same?
>>>>>
>>>>> Thanks,
>>>>> Dan
>>>>>
>>>>> -----Original Message-----
>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>> pending missed ticks
>>>>>
>>>>>    Hi Dan,
>>>>>
>>>>>    Hpet timer does have a fairly large error, as I was trying this
>>>>>    one recently.
>>>>>    I don't remember what I got for error, but 1% sounds     
>>>>
>>>>
>>>> about right.
>>>>  
>>>>
>>>>>    The problem is that hpet is not built on top of vpt.c,     
>>>>
>>>>
>>>> the module
>>>>  
>>>>
>>>>>    Keir and I did
>>>>>    all the recent work in, for its periodic timer needs. Try
>>>>>    specifying clock=pit
>>>>>    on the linux boot line. If it still picks the hpet, which it
>>>>>    might, let me know
>>>>>    and I'll tell you how to get around this.
>>>>>
>>>>>    Regards,
>>>>>    Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>        
>>>>
>>>>
>>>> --------------------------------------------------------------
>>>> ----------
>>>>  
>>>>
>>>>>    *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>    *Sent:* Fri 1/25/2008 6:50 PM
>>>>>    *To:* Dave Winchell; Keir Fraser
>>>>>    *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>    akira.ijuin@oracle.com
>>>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode     
>>>>
>>>>
>>>> that disables
>>>>  
>>>>
>>>>>    pending missed ticks
>>>>>
>>>>>    Sorry for the very late followup on this but we finally     
>>>>
>>>>
>>>> were able
>>>>  
>>>>
>>>>>    to get our testing set up again on stable 3.1 bits and have
>>>>>    seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>>
>>>>>    Test enviroment was a 4-socket dual core machine with 24GB of
>>>>>    memory running six two-vcpu 2GB domains, four hvm plus two pv.
>>>>>    All six guests were running LTP simultaneously.  The four hvm
>>>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
>>>>>    Timer_mode was set to 2 for 64-bit guests and 0 for     
>>>>
>>>>
>>>> 32-bit guests.
>>>>  
>>>>
>>>>>    All four hvm guests experienced skew around -1%, even the 32-bit
>>>>>    guest.  Less intensive testing didn't exhibit much skew at all.
>>>>>
>>>>>    A representative graph is attached.
>>>>>
>>>>>    Dave, I wonder if some portion of your patches didn't end up in
>>>>>    the xen trees?
>>>>>
>>>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>    14.318MHz HPET.)
>>>>>
>>>>>    Thanks,
>>>>>    Dan
>>>>>
>>>>>    P.S. Many thanks to Deepak and Akira for running tests.
>>>>>
>>>>>    > -----Original Message-----
>>>>>    > From: xen-devel-bounces@lists.xensource.com
>>>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>    > Dave Winchell
>>>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>    > To: Keir Fraser
>>>>>    > Cc: dan.magenheimer@oracle.com;     
>>>>
>>>>
>>>> xen-devel@lists.xensource.com; Dave
>>>>  
>>>>
>>>>>    > Winchell
>>>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>    > disables pending
>>>>>    > missed ticks
>>>>>    >
>>>>>    >
>>>>>    > Hi Keir,
>>>>>    >
>>>>>    > The latest change, c/s 16690, looks fine.
>>>>>    > I agree that the code in c/s 16690 is equivalent to
>>>>>    > the code I submitted. Also, your version is more
>>>>>    > concise.
>>>>>    >
>>>>>    > The error tests confirm the equivalence. With     
>>>>
>>>>
>>>> overnight cpu loads,
>>>>  
>>>>
>>>>>    > the checked in version was accurate to +.048% for sles
>>>>>    > and +.038% for red hat. My version was +.046% and +.032% in a
>>>>>    > 2 hour test.
>>>>>    > I don't think the difference is significant.
>>>>>    >
>>>>>    > i/o loads produced errors of +.01%.
>>>>>    >
>>>>>    > Thanks for all your efforts on this issue.
>>>>>    >
>>>>>    > Regards,
>>>>>    > Dave
>>>>>    >
>>>>>    >
>>>>>    >
>>>>>    > Keir Fraser wrote:
>>>>>    >
>>>>>    > >Applied as c/s 16690, although the checked-in patch is
>>>>>    > smaller. I think the
>>>>>    > >only important fix is to pt_intr_post() and the only bit of
>>>>>    > the patch I
>>>>>    > >totally omitted was the change to pt_process_missed_ticks().
>>>>>    > I don't think
>>>>>    > >that change can be important, but let's see what     
>>>>
>>>>
>>>> happens to the
>>>>  
>>>>
>>>>>    error
>>>>>    > >percentage...
>>>>>    > >
>>>>>    > > -- Keir
>>>>>    > >
>>>>>    > >On 4/1/08 23:24, "Dave Winchell"     
>>>>
>>>>
>>>> <dwinchell@virtualiron.com> wrote:
>>>>  
>>>>
>>>>>    > >
>>>>>    > >
>>>>>    > >
>>>>>    > >>Hi Dan and Keir,
>>>>>    > >>
>>>>>    > >>Attached is a patch that fixes some issues with the     
>>>>
>>>>
>>>> SYNC policy
>>>>  
>>>>
>>>>>    > >>(no_missed_ticks_pending).
>>>>>    > >>I have not tried to make the change the minimal one, but,
>>>>>    > rather, just
>>>>>    > >>ported into
>>>>>    > >>the new code what I know to work well. The error for
>>>>>    > >>no_missed_ticks_pending goes from
>>>>>    > >>over 3% to .03% with this change according to my testing.
>>>>>    > >>
>>>>>    > >>Regards,
>>>>>    > >>Dave
>>>>>    > >>
>>>>>    > >>Dan Magenheimer wrote:
>>>>>    > >>
>>>>>    > >>
>>>>>    > >>
>>>>>    > >>>Hi Dave --
>>>>>    > >>>
>>>>>    > >>>Did you get your correction ported?  If so, it would be
>>>>>    > nice to see this get
>>>>>    > >>>into 3.1.3.
>>>>>    > >>>
>>>>>    > >>>Note that I just did some very limited testing with
>>>>>    > timer_mode=2(=SYNC=no
>>>>>    > >>>missed ticks pending)
>>>>>    > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
>>>>>    > worst error I've
>>>>>    > >>>seen so far
>>>>>    > >>>is 0.012%.  But I haven't tried any exotic loads, just LTP.
>>>>>    > >>>
>>>>>    > >>>Thanks,
>>>>>    > >>>Dan
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>>-----Original Message-----
>>>>>    > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>    > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>    > >>>>To: dan.magenheimer@oracle.com
>>>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>    > xen-devel@lists.xensource.com; Dong,
>>>>>    > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>    > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>    > >>>>disables pending
>>>>>    > >>>>missed ticks
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>Dan,
>>>>>    > >>>>
>>>>>    > >>>>I did some testing with the constant tsc offset     
>>>>
>>>>
>>>> SYNC method
>>>>  
>>>>
>>>>>    > >>>>(now called
>>>>>    > >>>>no_missed_ticks_pending)
>>>>>    > >>>>and found the error to be very high, much larger     
>>>>
>>>>
>>>> than 1 %, as
>>>>  
>>>>
>>>>>    > >>>>I recall.
>>>>>    > >>>>I have not had a chance to submit a correction. I     
>>>>
>>>>
>>>> will try to
>>>>  
>>>>
>>>>>    > >>>>do it later
>>>>>    > >>>>this week or the first week in January. My version of
>>>>>    constant tsc
>>>>>    > >>>>offset SYNC method
>>>>>    > >>>>produces .02 % error, so I just need to port that into the
>>>>>    > >>>>current code.
>>>>>    > >>>>
>>>>>    > >>>>The error you got for both of those kernels is     
>>>>
>>>>
>>>> what I would
>>>>  
>>>>
>>>>>    expect
>>>>>    > >>>>for the default mode, delay_for_missed_ticks.
>>>>>    > >>>>
>>>>>    > >>>>I'll let Keir answer on how to set the time mode.
>>>>>    > >>>>
>>>>>    > >>>>Regards,
>>>>>    > >>>>Dave
>>>>>    > >>>>
>>>>>    > >>>>Dan Magenheimer wrote:
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>Anyone make measurements on the final patch?
>>>>>    > >>>>>
>>>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>about 0.2% with no load.  This was xen-unstable tip today
>>>>>    > >>>>with no options specified.  32-bit was about 0.01%.
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>I think I missed something... how do I run the various
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>accounting choices and which ones are known to be     
>>>>
>>>>
>>>> appropriate
>>>>  
>>>>
>>>>>    > >>>>for which kernels?
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>Thanks,
>>>>>    > >>>>>Dan
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>>-----Original Message-----
>>>>>    > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>    >    
>>>>>
>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>               
>>>>>>>>>
>>>>>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>Keir Fraser
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>    > >>>>>>To: Dave Winchell
>>>>>    > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>>>>    > Eddie; Jiang,
>>>>>    > >>>>>>Yunhong
>>>>>    > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>    > >>>>>>disables pending
>>>>>    > >>>>>>missed ticks
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>    > >>>>>>
>>>>>    > >>>>>>-- Keir
>>>>>    > >>>>>>
>>>>>    > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>><dwinchell@virtualiron.com> wrote:
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>Keir,
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The accuracy data I've collected for i/o loads for the
>>>>>    > >>>>>>>various time protocols follows. In addition, the data
>>>>>    > >>>>>>>for cpu loads is shown.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The loads labeled cpu and i/o-8 are on an 8     
>>>>
>>>>
>>>> processor AMD
>>>>  
>>>>
>>>>>    box.
>>>>>    > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>    > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>    > >>>>>>>(usex is available at
>>>>>    http://people.redhat.com/anderson/usex.)
>>>>>    > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6     
>>>>
>>>>
>>>> of=/dev/null.
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>    > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>    > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>    > >>>>>>>All three guests are 8vcpu.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>    > >>>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
>>>>>    > +.005% cpu
>>>>>    > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
>>>>>    > +.012% cpu
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>    -.004% cpu
>>>>>    > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,     
>>>>
>>>>
>>>> -.005%, -.005% cpu
>>>>  
>>>>
>>>>>    > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,     
>>>>
>>>>
>>>> -.008%, -.003% cpu
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,     
>>>>
>>>>
>>>> -.040% cpu
>>>>  
>>>>
>>>>>    > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
>>>>>    > -.031% cpu
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>    > -.09% i/o-8
>>>>>    > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>    > -.14% i/o-8
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>    > -.022% i/o-8
>>>>>    > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,     
>>>>
>>>>
>>>> -.017%, -.018% i/o-8
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>    > -.029% i/o-8
>>>>>    > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
>>>>>    > -.031% i/o-8
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,     
>>>>
>>>>
>>>> -.04% i/o-32
>>>>  
>>>>
>>>>>    > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>    > -.005% i/o-32
>>>>>    > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,     
>>>>
>>>>
>>>> -.11% i/o-32
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>    > .003% i/o-4/32
>>>>>    > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>    > .01% i/o-4/32
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Overhead measurements:
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Progress in terms of number of passes through a fixed
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>system workload
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>    > >>>>>>>The workload was usex -b48.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>    > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>    > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>    > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Conclusions:
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>    > requirement for ntp
>>>>>    > >>>>>>>tracking under the loads
>>>>>    > >>>>>>>above is the SYNC protocol. The worst case     
>>>>
>>>>
>>>> accuracies for
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>SYNC, MIXED,
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>and ASYNC
>>>>>    > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>We could reduce the cost of the SYNC method by only
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>scheduling the extra
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>wakeups if a certain number
>>>>>    > >>>>>>>of ticks are missed.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Regards,
>>>>>    > >>>>>>>Dave
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Keir Fraser wrote:
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>>Since I had a high error (~.03%) for the     
>>>>
>>>>
>>>> ASYNC method a
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>couple of days ago,
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>>>    > been something
>>>>>    > >>>>>>>>>wrong with the code I used a couple of days ago for
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>ASYNC. It may have been
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>missing the immediate delivery of interrupt     
>>>>
>>>>
>>>> after context
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>switch in.
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>acceptable accuracy,
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>each running consistently around or under     
>>>>
>>>>
>>>> .01%. MIXED has
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>a fairly high
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>error of
>>>>>    > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>threshold for comfort.
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>I don't have an overnight run with SYNC. I     
>>>>
>>>>
>>>> plan to leave
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>SYNC running
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>running instead.
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>It may be too early to pick the protocol and     
>>>>
>>>>
>>>> I can run
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>more overnight tests
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>next week.
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>I'm a bit worried about any unwanted side     
>>>>
>>>>
>>>> effects of the
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>SYNC+run_timer
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>approach -- e.g., whether timer wakeups will     
>>>>
>>>>
>>>> cause higher
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>system-wide CPU
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>contention. I find it easier to think through the
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>implications of ASYNC. I'm
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>surprised that MIXED loses time, and is less     
>>>>
>>>>
>>>> accurate than
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>ASYNC. Perhaps it
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>delivers more timer interrupts than the other     
>>>>
>>>>
>>>> approaches,
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>and each interrupt
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>event causes a small accumulated error?
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>Overall I would consider MIXED and ASYNC as     
>>>>
>>>>
>>>> favourites and
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>if the latter is
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>actually more accurate then I can simply revert the
>>>>>    > changeset that
>>>>>    > >>>>>>>>implemented MIXED.
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>Perhaps rather than running more of the same     
>>>>
>>>>
>>>> workloads you
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>could try idle
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated     
>>>>
>>>>
>>>> large disc reads
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>to /dev/null)? We
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>don't have any data on workloads that aren't     
>>>>
>>>>
>>>> CPU bound, so
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>that's really an
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>obvious place to put any further effort imo.
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>-- Keir
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>_______________________________________________
>>>>>    > >>>>>>Xen-devel mailing list
>>>>>    > >>>>>>Xen-devel@lists.xensource.com
>>>>>    > >>>>>>http://lists.xensource.com/xen-devel
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>>>>    > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>>>>>    > >>
>>>>>    > >>     missed_ticks = missed_ticks / (s_time_t)     
>>>>
>>>>
>>>> pt->period + 1;
>>>>  
>>>>
>>>>>    > >>     if ( mode_is(pt->vcpu->domain,     
>>>>
>>>>
>>>> no_missed_ticks_pending) )
>>>>  
>>>>
>>>>>    > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>    > >>+        pt->do_not_freeze = 1;
>>>>>    > >>     else
>>>>>    > >>         pt->pending_intr_nr += missed_ticks;
>>>>>    > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>    > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>>>    > >>
>>>>>    > >>     pt_lock(pt);
>>>>>    > >>
>>>>>    > >>-    pt->pending_intr_nr++;
>>>>>    > >>+    if ( mode_is(pt->vcpu->domain,     
>>>>
>>>>
>>>> no_missed_ticks_pending) ) {
>>>>  
>>>>
>>>>>    > >>+        pt->pending_intr_nr = 1;
>>>>>    > >>+ pt->do_not_freeze = 0;
>>>>>    > >>+    }
>>>>>    > >>+    else
>>>>>    > >>+ pt->pending_intr_nr++;
>>>>>    > >>
>>>>>    > >>     if ( !pt->one_shot )
>>>>>    > >>     {
>>>>>    > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>>>>    > >>         return;
>>>>>    > >>     }
>>>>>    > >>
>>>>>    > >>-    pt->do_not_freeze = 0;
>>>>>    > >>-
>>>>>    > >>     if ( pt->one_shot )
>>>>>    > >>     {
>>>>>    > >>         pt->enabled = 0;
>>>>>    > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu     
>>>>
>>>>
>>>> *v, struct
>>>>  
>>>>
>>>>>    > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>    > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>>>    > missed ticks */
>>>>>    > >>         }
>>>>>    > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>>>>    > >>+     pt->pending_intr_nr--;
>>>>>    > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>    > >>+ }
>>>>>    > >>         else
>>>>>    > >>         {
>>>>>    > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>>    > >>
>>>>>    > >>
>>>>>    > >
>>>>>    > >
>>>>>    > >
>>>>>    > >
>>>>>    >
>>>>>    >
>>>>>    > _______________________________________________
>>>>>    > Xen-devel mailing list
>>>>>    > Xen-devel@lists.xensource.com
>>>>>    > http://lists.xensource.com/xen-devel
>>>>>    >
>>>>>
>>>>>     
>>>>
>>>>
>>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 35825 bytes --]

OK, Deepak repeated the test without ntpd and using ntpdate -b before
the test.

The attached graph shows his results: el5u1-64 (best=~0.07%),
el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).

We will continue to look at LTP to try to isolate.

Thanks,
Dan

P.S. elXuY is essentially RHEL XuY with some patches.

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Wednesday, January 30, 2008 2:45 PM
> To: Deepak Patel
> Cc: dan.magenheimer@oracle.com; Keir Fraser;
> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan, Deeepak,
> 
> It may be that the underlying clock error is too great for ntp
> to handle. It would be useful if you did not run ntpd
> and, instead did ntpdate -b <timeserver> at the start of the test
> for each guest. Then capture the data as you have been doing.
> If the drift is greater than .05%, then we need to address that.
> 
> Another option is, when running ntpd, to enable loop statistics in
> /etc/ntp.conf
> by adding this to the file:
> 
> statistics loopstats
> statsdir /var/lib/ntp/
> 
> Then you will see loop data in that directory.
> Correlating the data in the loopstats files with the
> peaks in skew would be interesting. You will see entries of the form
> 
> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
> 
> Where the second to last column is the Allan Deviation. When that
> gets over 1000, ntpd is working pretty hard. However, I have 
> not seen ntpd
> completely loose it like you have.
> 
> I'm on vacation until Monday, and won't be reading
> email.
> 
> Thanks for all your work on this!
> 
> -Dave
> 
> Deepak Patel wrote:
> 
> >
> >>
> >> Is the graph for RHEL5u1-64? (I've never tested this one.)
> >
> >
> > I do not know which graph was attached with this. But I saw this
> > behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
> > was running ltp tests continuously.
> >
> >> What was the behaviour of the other guests running?
> >
> >
> > All pvm guests are fine. But behavior of most of the hvm guests were
> > as described.
> >
> >> If they had spikes, were they at the same wall time?
> >
> >
> > No. They are not at the same wall time.
> >
> >> Were the other guests running ltp as well?
> >>
> > Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
> > continuously.
> >
> >> How are you measuring skew?
> >
> >
> > I was collecting output of "ntpdate -q  <timeserver> every 
> 300 seconds
> > (5 minutes) and have created graph based on that.
> >
> >>
> >> Are you running ntpd?
> >>
> > Yes. ntp was running on all the guests.
> >
> > I am investigating what causes this spikes and let everyone 
> know what
> > are my findings.
> >
> > Thanks,
> > Deepak
> >
> >> Anything that you can discover that would be in sync with
> >> the spikes would be very helpful!
> >>
> >> The code that I test with is our product code, which is based
> >> on 3.1. So it is possible that something in 3.2 other than vpt.c
> >> is the cause. I can test with 3.2, if necessary.
> >>
> >> thanks,
> >> Dave
> >>
> >>
> >>
> >> Dan Magenheimer wrote:
> >>
> >>> Hi Dave (Keir, see suggestion below) --
> >>>
> >>> Thanks!
> >>>
> >>> Turning off vhpet certainly helps a lot (though see below).
> >>>
> >>> I wonder if timekeeping with vhpet is so bad that it should be
> >>> turned off by default (in 3.1, 3.2, and unstable) until it is
> >>> fixed?  (I have a patch that defaults it off, can post it if
> >>> there is agreement on the above point.)  The whole point of an
> >>> HPET is to provide more precise timekeeping and if vhpet is
> >>> worse than vpit, it can only confuse users.  Comments?
> >>>
> >>>
> >>> In your testing, are you just measuring % skew over a long
> >>> period of time?
> >>>  We are graphing the skew continuously and
> >>> seeing periodic behavior that is unsettling, even with pit.
> >>> See attached.  Though your algorithm recovers, the "cliffs"
> >>> could still cause real user problems.  I wonder if there is
> >>> anything that can be done to make the "recovery" more
> >>> responsive?
> >>>
> >>> We are looking into what part(s) of LTP is causing the cliffs.
> >>>
> >>> Thanks,
> >>> Dan
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>> Sent: Monday, January 28, 2008 8:21 AM
> >>>> To: dan.magenheimer@oracle.com
> >>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
> >>>> deepak.patel@oracle.com;
> >>>> akira.ijuin@oracle.com; Dave Winchell
> >>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
> >>>> pending
> >>>> missed ticks
> >>>>
> >>>>
> >>>> Dan,
> >>>>
> >>>> I guess I'm a bit out of date calling for clock= usage.
> >>>> Looking at linux 2.6.20.4 sources, I think you should specify
> >>>> "clocksource=pit nohpet" on the linux guest bootline.
> >>>>
> >>>> You can leave the xen and dom0 bootlines as they are.
> >>>> The xen and guest clocksources do not need to be the same.
> >>>> In my tests, xen is using the hpet for its timekeeping and
> >>>> that appears to be the default.
> >>>>
> >>>> When you boot the guests you should see
> >>>>    time.c: Using PIT/TSC based timekeeping.
> >>>> on the rh4u5-64 guest, and something similar on the others.
> >>>>
> >>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
> >>>> > 14.318MHz HPET.)
> >>>>
> >>>> This appears to be the xen state, which is fine.
> >>>> I was wrongly assuming that this was the guest state.
> >>>> You might want to look in your guest logs and see what they were
> >>>> picking
> >>>> for a clock source.
> >>>>
> >>>> Regards,
> >>>> Dave
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>> Thanks, I hadn't realized that!  No wonder we didn't 
> see the same
> >>>>> improvement you saw!
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Try specifying clock=pit on the linux boot line...
> >>>>>>
> >>>>>
> >>>>>
> >>>>> I'm confused... do you mean "clocksource=pit" on the Xen
> >>>>
> >>>>
> >>>> command line or
> >>>>
> >>>>
> >>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
> >>>>
> >>>>
> >>>> dom0?) command
> >>>>
> >>>>
> >>>>> line?  Or both places?  Since the tests take awhile, it
> >>>>
> >>>>
> >>>> would be nice
> >>>>
> >>>>
> >>>>> to get this right the first time.  Do the Xen and guest
> >>>>
> >>>>
> >>>> clocksources need
> >>>>
> >>>>
> >>>>> to be the same?
> >>>>>
> >>>>> Thanks,
> >>>>> Dan
> >>>>>
> >>>>> -----Original Message-----
> >>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
> >>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
> >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
> >>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
> that disables
> >>>>> pending missed ticks
> >>>>>
> >>>>>    Hi Dan,
> >>>>>
> >>>>>    Hpet timer does have a fairly large error, as I was 
> trying this
> >>>>>    one recently.
> >>>>>    I don't remember what I got for error, but 1% sounds
> >>>>
> >>>>
> >>>> about right.
> >>>>
> >>>>
> >>>>>    The problem is that hpet is not built on top of vpt.c,
> >>>>
> >>>>
> >>>> the module
> >>>>
> >>>>
> >>>>>    Keir and I did
> >>>>>    all the recent work in, for its periodic timer needs. Try
> >>>>>    specifying clock=pit
> >>>>>    on the linux boot line. If it still picks the hpet, which it
> >>>>>    might, let me know
> >>>>>    and I'll tell you how to get around this.
> >>>>>
> >>>>>    Regards,
> >>>>>    Dave
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --------------------------------------------------------------
> >>>> ----------
> >>>>
> >>>>
> >>>>>    *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >>>>>    *Sent:* Fri 1/25/2008 6:50 PM
> >>>>>    *To:* Dave Winchell; Keir Fraser
> >>>>>    *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
> >>>>>    akira.ijuin@oracle.com
> >>>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>
> >>>>
> >>>> that disables
> >>>>
> >>>>
> >>>>>    pending missed ticks
> >>>>>
> >>>>>    Sorry for the very late followup on this but we finally
> >>>>
> >>>>
> >>>> were able
> >>>>
> >>>>
> >>>>>    to get our testing set up again on stable 3.1 bits and have
> >>>>>    seen some very bad results on 3.1.3-rc1, on the order of 1%.
> >>>>>
> >>>>>    Test enviroment was a 4-socket dual core machine with 24GB of
> >>>>>    memory running six two-vcpu 2GB domains, four hvm 
> plus two pv.
> >>>>>    All six guests were running LTP simultaneously.  The four hvm
> >>>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and 
> RHEL4u5-64.
> >>>>>    Timer_mode was set to 2 for 64-bit guests and 0 for
> >>>>
> >>>>
> >>>> 32-bit guests.
> >>>>
> >>>>
> >>>>>    All four hvm guests experienced skew around -1%, 
> even the 32-bit
> >>>>>    guest.  Less intensive testing didn't exhibit much 
> skew at all.
> >>>>>
> >>>>>    A representative graph is attached.
> >>>>>
> >>>>>    Dave, I wonder if some portion of your patches 
> didn't end up in
> >>>>>    the xen trees?
> >>>>>
> >>>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
> >>>>>    14.318MHz HPET.)
> >>>>>
> >>>>>    Thanks,
> >>>>>    Dan
> >>>>>
> >>>>>    P.S. Many thanks to Deepak and Akira for running tests.
> >>>>>
> >>>>>    > -----Original Message-----
> >>>>>    > From: xen-devel-bounces@lists.xensource.com
> >>>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>    > Dave Winchell
> >>>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
> >>>>>    > To: Keir Fraser
> >>>>>    > Cc: dan.magenheimer@oracle.com;
> >>>>
> >>>>
> >>>> xen-devel@lists.xensource.com; Dave
> >>>>
> >>>>
> >>>>>    > Winchell
> >>>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>    > disables pending
> >>>>>    > missed ticks
> >>>>>    >
> >>>>>    >
> >>>>>    > Hi Keir,
> >>>>>    >
> >>>>>    > The latest change, c/s 16690, looks fine.
> >>>>>    > I agree that the code in c/s 16690 is equivalent to
> >>>>>    > the code I submitted. Also, your version is more
> >>>>>    > concise.
> >>>>>    >
> >>>>>    > The error tests confirm the equivalence. With
> >>>>
> >>>>
> >>>> overnight cpu loads,
> >>>>
> >>>>
> >>>>>    > the checked in version was accurate to +.048% for sles
> >>>>>    > and +.038% for red hat. My version was +.046% and 
> +.032% in a
> >>>>>    > 2 hour test.
> >>>>>    > I don't think the difference is significant.
> >>>>>    >
> >>>>>    > i/o loads produced errors of +.01%.
> >>>>>    >
> >>>>>    > Thanks for all your efforts on this issue.
> >>>>>    >
> >>>>>    > Regards,
> >>>>>    > Dave
> >>>>>    >
> >>>>>    >
> >>>>>    >
> >>>>>    > Keir Fraser wrote:
> >>>>>    >
> >>>>>    > >Applied as c/s 16690, although the checked-in patch is
> >>>>>    > smaller. I think the
> >>>>>    > >only important fix is to pt_intr_post() and the 
> only bit of
> >>>>>    > the patch I
> >>>>>    > >totally omitted was the change to 
> pt_process_missed_ticks().
> >>>>>    > I don't think
> >>>>>    > >that change can be important, but let's see what
> >>>>
> >>>>
> >>>> happens to the
> >>>>
> >>>>
> >>>>>    error
> >>>>>    > >percentage...
> >>>>>    > >
> >>>>>    > > -- Keir
> >>>>>    > >
> >>>>>    > >On 4/1/08 23:24, "Dave Winchell"
> >>>>
> >>>>
> >>>> <dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >>Hi Dan and Keir,
> >>>>>    > >>
> >>>>>    > >>Attached is a patch that fixes some issues with the
> >>>>
> >>>>
> >>>> SYNC policy
> >>>>
> >>>>
> >>>>>    > >>(no_missed_ticks_pending).
> >>>>>    > >>I have not tried to make the change the minimal one, but,
> >>>>>    > rather, just
> >>>>>    > >>ported into
> >>>>>    > >>the new code what I know to work well. The error for
> >>>>>    > >>no_missed_ticks_pending goes from
> >>>>>    > >>over 3% to .03% with this change according to my testing.
> >>>>>    > >>
> >>>>>    > >>Regards,
> >>>>>    > >>Dave
> >>>>>    > >>
> >>>>>    > >>Dan Magenheimer wrote:
> >>>>>    > >>
> >>>>>    > >>
> >>>>>    > >>
> >>>>>    > >>>Hi Dave --
> >>>>>    > >>>
> >>>>>    > >>>Did you get your correction ported?  If so, it would be
> >>>>>    > nice to see this get
> >>>>>    > >>>into 3.1.3.
> >>>>>    > >>>
> >>>>>    > >>>Note that I just did some very limited testing with
> >>>>>    > timer_mode=2(=SYNC=no
> >>>>>    > >>>missed ticks pending)
> >>>>>    > >>>on tip of xen-3.1-testing (64-bit Linux hv 
> guest) and the
> >>>>>    > worst error I've
> >>>>>    > >>>seen so far
> >>>>>    > >>>is 0.012%.  But I haven't tried any exotic 
> loads, just LTP.
> >>>>>    > >>>
> >>>>>    > >>>Thanks,
> >>>>>    > >>>Dan
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>>-----Original Message-----
> >>>>>    > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>    > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>    > >>>>To: dan.magenheimer@oracle.com
> >>>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>    > xen-devel@lists.xensource.com; Dong,
> >>>>>    > >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>    > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>    > >>>>disables pending
> >>>>>    > >>>>missed ticks
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>Dan,
> >>>>>    > >>>>
> >>>>>    > >>>>I did some testing with the constant tsc offset
> >>>>
> >>>>
> >>>> SYNC method
> >>>>
> >>>>
> >>>>>    > >>>>(now called
> >>>>>    > >>>>no_missed_ticks_pending)
> >>>>>    > >>>>and found the error to be very high, much larger
> >>>>
> >>>>
> >>>> than 1 %, as
> >>>>
> >>>>
> >>>>>    > >>>>I recall.
> >>>>>    > >>>>I have not had a chance to submit a correction. I
> >>>>
> >>>>
> >>>> will try to
> >>>>
> >>>>
> >>>>>    > >>>>do it later
> >>>>>    > >>>>this week or the first week in January. My version of
> >>>>>    constant tsc
> >>>>>    > >>>>offset SYNC method
> >>>>>    > >>>>produces .02 % error, so I just need to port 
> that into the
> >>>>>    > >>>>current code.
> >>>>>    > >>>>
> >>>>>    > >>>>The error you got for both of those kernels is
> >>>>
> >>>>
> >>>> what I would
> >>>>
> >>>>
> >>>>>    expect
> >>>>>    > >>>>for the default mode, delay_for_missed_ticks.
> >>>>>    > >>>>
> >>>>>    > >>>>I'll let Keir answer on how to set the time mode.
> >>>>>    > >>>>
> >>>>>    > >>>>Regards,
> >>>>>    > >>>>Dave
> >>>>>    > >>>>
> >>>>>    > >>>>Dan Magenheimer wrote:
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>Anyone make measurements on the final patch?
> >>>>>    > >>>>>
> >>>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and 
> saw a loss of
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>about 0.2% with no load.  This was 
> xen-unstable tip today
> >>>>>    > >>>>with no options specified.  32-bit was about 0.01%.
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>I think I missed something... how do I run the various
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>accounting choices and which ones are known to be
> >>>>
> >>>>
> >>>> appropriate
> >>>>
> >>>>
> >>>>>    > >>>>for which kernels?
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>Thanks,
> >>>>>    > >>>>>Dan
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>>-----Original Message-----
> >>>>>    > >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>    >
> >>>>>
> >>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>Keir Fraser
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>    > >>>>>>To: Dave Winchell
> >>>>>    > >>>>>>Cc: Shan, Haitao; 
> xen-devel@lists.xensource.com; Dong,
> >>>>>    > Eddie; Jiang,
> >>>>>    > >>>>>>Yunhong
> >>>>>    > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer 
> mode that
> >>>>>    > >>>>>>disables pending
> >>>>>    > >>>>>>missed ticks
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>-- Keir
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>><dwinchell@virtualiron.com> wrote:
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>Keir,
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The accuracy data I've collected for i/o 
> loads for the
> >>>>>    > >>>>>>>various time protocols follows. In 
> addition, the data
> >>>>>    > >>>>>>>for cpu loads is shown.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>
> >>>>
> >>>> processor AMD
> >>>>
> >>>>
> >>>>>    box.
> >>>>>    > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >>>>>    > >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>    > >>>>>>>(usex is available at
> >>>>>    http://people.redhat.com/anderson/usex.)
> >>>>>    > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>
> >>>>
> >>>> of=/dev/null.
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >>>>>    > >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>    > >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>    > >>>>>>>All three guests are 8vcpu.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
> >>>>>    > >>>>>>>except that the redhat-64 guest has 4 
> instances of dd.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 
> sec -.006%,
> >>>>>    > +.005% cpu
> >>>>>    > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 
> sec, -.001%,
> >>>>>    > +.012% cpu
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
> >>>>>    -.004% cpu
> >>>>>    > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>
> >>>>
> >>>> -.005%, -.005% cpu
> >>>>
> >>>>
> >>>>>    > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>
> >>>>
> >>>> -.008%, -.003% cpu
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>
> >>>>
> >>>> -.040% cpu
> >>>>
> >>>>
> >>>>>    > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 
> sec, -.034%,
> >>>>>    > -.031% cpu
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
> >>>>>    > -.09% i/o-8
> >>>>>    > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
> >>>>>    > -.14% i/o-8
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
> >>>>>    > -.022% i/o-8
> >>>>>    > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>
> >>>>
> >>>> -.017%, -.018% i/o-8
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
> >>>>>    > -.029% i/o-8
> >>>>>    > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 
> sec, -.023%,
> >>>>>    > -.031% i/o-8
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
> >>>>
> >>>>
> >>>> -.04% i/o-32
> >>>>
> >>>>
> >>>>>    > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
> >>>>>    > -.005% i/o-32
> >>>>>    > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
> >>>>
> >>>>
> >>>> -.11% i/o-32
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
> >>>>>    > .003% i/o-4/32
> >>>>>    > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
> >>>>>    > .01% i/o-4/32
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Overhead measurements:
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Progress in terms of number of passes 
> through a fixed
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>system workload
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>    > >>>>>>>The workload was usex -b48.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>    > >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>    > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>    > >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Conclusions:
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The only protocol which meets the .05% accuracy
> >>>>>    > requirement for ntp
> >>>>>    > >>>>>>>tracking under the loads
> >>>>>    > >>>>>>>above is the SYNC protocol. The worst case
> >>>>
> >>>>
> >>>> accuracies for
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>SYNC, MIXED,
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>and ASYNC
> >>>>>    > >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>We could reduce the cost of the SYNC method by only
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>scheduling the extra
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>wakeups if a certain number
> >>>>>    > >>>>>>>of ticks are missed.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Regards,
> >>>>>    > >>>>>>>Dave
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Keir Fraser wrote:
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>>Since I had a high error (~.03%) for the
> >>>>
> >>>>
> >>>> ASYNC method a
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>couple of days ago,
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>I ran another ASYNC test. I think there may have
> >>>>>    > been something
> >>>>>    > >>>>>>>>>wrong with the code I used a couple of 
> days ago for
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>ASYNC. It may have been
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>missing the immediate delivery of interrupt
> >>>>
> >>>>
> >>>> after context
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>switch in.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>My results indicate that either SYNC or ASYNC give
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>acceptable accuracy,
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>each running consistently around or under
> >>>>
> >>>>
> >>>> .01%. MIXED has
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>a fairly high
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>error of
> >>>>>    > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>threshold for comfort.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>I don't have an overnight run with SYNC. I
> >>>>
> >>>>
> >>>> plan to leave
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>SYNC running
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>over the weekend. If you'd rather I can 
> leave MIXED
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>running instead.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>It may be too early to pick the protocol and
> >>>>
> >>>>
> >>>> I can run
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>more overnight tests
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>next week.
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>I'm a bit worried about any unwanted side
> >>>>
> >>>>
> >>>> effects of the
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>SYNC+run_timer
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>
> >>>>
> >>>> cause higher
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>system-wide CPU
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>contention. I find it easier to think through the
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>implications of ASYNC. I'm
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>surprised that MIXED loses time, and is less
> >>>>
> >>>>
> >>>> accurate than
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>ASYNC. Perhaps it
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>delivers more timer interrupts than the other
> >>>>
> >>>>
> >>>> approaches,
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>and each interrupt
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>event causes a small accumulated error?
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>
> >>>>
> >>>> favourites and
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>if the latter is
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>actually more accurate then I can simply revert the
> >>>>>    > changeset that
> >>>>>    > >>>>>>>>implemented MIXED.
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>Perhaps rather than running more of the same
> >>>>
> >>>>
> >>>> workloads you
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>could try idle
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>
> >>>>
> >>>> large disc reads
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>to /dev/null)? We
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>don't have any data on workloads that aren't
> >>>>
> >>>>
> >>>> CPU bound, so
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>that's really an
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>obvious place to put any further effort imo.
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>-- Keir
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>_______________________________________________
> >>>>>    > >>>>>>Xen-devel mailing list
> >>>>>    > >>>>>>Xen-devel@lists.xensource.com
> >>>>>    > >>>>>>http://lists.xensource.com/xen-devel
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 
> 2007 +0000
> >>>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 
> 2008 -0500
> >>>>>    > >>@@ -58,7 +58,7 @@ static void 
> pt_process_missed_ticks(stru
> >>>>>    > >>
> >>>>>    > >>     missed_ticks = missed_ticks / (s_time_t)
> >>>>
> >>>>
> >>>> pt->period + 1;
> >>>>
> >>>>
> >>>>>    > >>     if ( mode_is(pt->vcpu->domain,
> >>>>
> >>>>
> >>>> no_missed_ticks_pending) )
> >>>>
> >>>>
> >>>>>    > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>    > >>+        pt->do_not_freeze = 1;
> >>>>>    > >>     else
> >>>>>    > >>         pt->pending_intr_nr += missed_ticks;
> >>>>>    > >>     pt->scheduled += missed_ticks * pt->period;
> >>>>>    > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>>>>    > >>
> >>>>>    > >>     pt_lock(pt);
> >>>>>    > >>
> >>>>>    > >>-    pt->pending_intr_nr++;
> >>>>>    > >>+    if ( mode_is(pt->vcpu->domain,
> >>>>
> >>>>
> >>>> no_missed_ticks_pending) ) {
> >>>>
> >>>>
> >>>>>    > >>+        pt->pending_intr_nr = 1;
> >>>>>    > >>+ pt->do_not_freeze = 0;
> >>>>>    > >>+    }
> >>>>>    > >>+    else
> >>>>>    > >>+ pt->pending_intr_nr++;
> >>>>>    > >>
> >>>>>    > >>     if ( !pt->one_shot )
> >>>>>    > >>     {
> >>>>>    > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct 
> vcpu *v, struct
> >>>>>    > >>         return;
> >>>>>    > >>     }
> >>>>>    > >>
> >>>>>    > >>-    pt->do_not_freeze = 0;
> >>>>>    > >>-
> >>>>>    > >>     if ( pt->one_shot )
> >>>>>    > >>     {
> >>>>>    > >>         pt->enabled = 0;
> >>>>>    > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
> >>>>
> >>>>
> >>>> *v, struct
> >>>>
> >>>>
> >>>>>    > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>    > >>             pt->pending_intr_nr = 0; /* 'collapse' all
> >>>>>    > missed ticks */
> >>>>>    > >>         }
> >>>>>    > >>+ else if ( mode_is(v->domain, 
> no_missed_ticks_pending) ) {
> >>>>>    > >>+     pt->pending_intr_nr--;
> >>>>>    > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>    > >>+ }
> >>>>>    > >>         else
> >>>>>    > >>         {
> >>>>>    > >>             pt->last_plt_gtime += pt->period_cycles;
> >>>>>    > >>
> >>>>>    > >>
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >
> >>>>>    >
> >>>>>    >
> >>>>>    > _______________________________________________
> >>>>>    > Xen-devel mailing list
> >>>>>    > Xen-devel@lists.xensource.com
> >>>>>    > http://lists.xensource.com/xen-devel
> >>>>>    >
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>
> 
>

[-- Attachment #2: hvm-compare.png --]
[-- Type: application/octet-stream, Size: 8753 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan, Deepak:

Thanks for the data. Those drifts are severe - no wonder ntp couldn't
keep then in synch. I'll try to reproduce that behaviour here, with my 
code base.
If I can't reproduce it, I'll try 3.2.

If you can isolate what ltp is doing during the cliffs, that would be very
helpful.

thanks,
Dave




Dan Magenheimer wrote:

>OK, Deepak repeated the test without ntpd and using ntpdate -b before
>the test.
>
>The attached graph shows his results: el5u1-64 (best=~0.07%),
>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>
>We will continue to look at LTP to try to isolate.
>
>Thanks,
>Dan
>
>P.S. elXuY is essentially RHEL XuY with some patches.
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, January 30, 2008 2:45 PM
>>To: Deepak Patel
>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan, Deeepak,
>>
>>It may be that the underlying clock error is too great for ntp
>>to handle. It would be useful if you did not run ntpd
>>and, instead did ntpdate -b <timeserver> at the start of the test
>>for each guest. Then capture the data as you have been doing.
>>If the drift is greater than .05%, then we need to address that.
>>
>>Another option is, when running ntpd, to enable loop statistics in
>>/etc/ntp.conf
>>by adding this to the file:
>>
>>statistics loopstats
>>statsdir /var/lib/ntp/
>>
>>Then you will see loop data in that directory.
>>Correlating the data in the loopstats files with the
>>peaks in skew would be interesting. You will see entries of the form
>>
>>54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
>>
>>Where the second to last column is the Allan Deviation. When that
>>gets over 1000, ntpd is working pretty hard. However, I have 
>>not seen ntpd
>>completely loose it like you have.
>>
>>I'm on vacation until Monday, and won't be reading
>>email.
>>
>>Thanks for all your work on this!
>>
>>-Dave
>>
>>Deepak Patel wrote:
>>
>>    
>>
>>>>Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>        
>>>>
>>>I do not know which graph was attached with this. But I saw this
>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
>>>was running ltp tests continuously.
>>>
>>>      
>>>
>>>>What was the behaviour of the other guests running?
>>>>        
>>>>
>>>All pvm guests are fine. But behavior of most of the hvm guests were
>>>as described.
>>>
>>>      
>>>
>>>>If they had spikes, were they at the same wall time?
>>>>        
>>>>
>>>No. They are not at the same wall time.
>>>
>>>      
>>>
>>>>Were the other guests running ltp as well?
>>>>
>>>>        
>>>>
>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>continuously.
>>>
>>>      
>>>
>>>>How are you measuring skew?
>>>>        
>>>>
>>>I was collecting output of "ntpdate -q  <timeserver> every 
>>>      
>>>
>>300 seconds
>>    
>>
>>>(5 minutes) and have created graph based on that.
>>>
>>>      
>>>
>>>>Are you running ntpd?
>>>>
>>>>        
>>>>
>>>Yes. ntp was running on all the guests.
>>>
>>>I am investigating what causes this spikes and let everyone 
>>>      
>>>
>>know what
>>    
>>
>>>are my findings.
>>>
>>>Thanks,
>>>Deepak
>>>
>>>      
>>>
>>>>Anything that you can discover that would be in sync with
>>>>the spikes would be very helpful!
>>>>
>>>>The code that I test with is our product code, which is based
>>>>on 3.1. So it is possible that something in 3.2 other than vpt.c
>>>>is the cause. I can test with 3.2, if necessary.
>>>>
>>>>thanks,
>>>>Dave
>>>>
>>>>
>>>>
>>>>Dan Magenheimer wrote:
>>>>
>>>>        
>>>>
>>>>>Hi Dave (Keir, see suggestion below) --
>>>>>
>>>>>Thanks!
>>>>>
>>>>>Turning off vhpet certainly helps a lot (though see below).
>>>>>
>>>>>I wonder if timekeeping with vhpet is so bad that it should be
>>>>>turned off by default (in 3.1, 3.2, and unstable) until it is
>>>>>fixed?  (I have a patch that defaults it off, can post it if
>>>>>there is agreement on the above point.)  The whole point of an
>>>>>HPET is to provide more precise timekeeping and if vhpet is
>>>>>worse than vpit, it can only confuse users.  Comments?
>>>>>
>>>>>
>>>>>In your testing, are you just measuring % skew over a long
>>>>>period of time?
>>>>> We are graphing the skew continuously and
>>>>>seeing periodic behavior that is unsettling, even with pit.
>>>>>See attached.  Though your algorithm recovers, the "cliffs"
>>>>>could still cause real user problems.  I wonder if there is
>>>>>anything that can be done to make the "recovery" more
>>>>>responsive?
>>>>>
>>>>>We are looking into what part(s) of LTP is causing the cliffs.
>>>>>
>>>>>Thanks,
>>>>>Dan
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>Sent: Monday, January 28, 2008 8:21 AM
>>>>>>To: dan.magenheimer@oracle.com
>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>deepak.patel@oracle.com;
>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>pending
>>>>>>missed ticks
>>>>>>
>>>>>>
>>>>>>Dan,
>>>>>>
>>>>>>I guess I'm a bit out of date calling for clock= usage.
>>>>>>Looking at linux 2.6.20.4 sources, I think you should specify
>>>>>>"clocksource=pit nohpet" on the linux guest bootline.
>>>>>>
>>>>>>You can leave the xen and dom0 bootlines as they are.
>>>>>>The xen and guest clocksources do not need to be the same.
>>>>>>In my tests, xen is using the hpet for its timekeeping and
>>>>>>that appears to be the default.
>>>>>>
>>>>>>When you boot the guests you should see
>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>on the rh4u5-64 guest, and something similar on the others.
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>14.318MHz HPET.)
>>>>>>>              
>>>>>>>
>>>>>>This appears to be the xen state, which is fine.
>>>>>>I was wrongly assuming that this was the guest state.
>>>>>>You might want to look in your guest logs and see what they were
>>>>>>picking
>>>>>>for a clock source.
>>>>>>
>>>>>>Regards,
>>>>>>Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>Dan Magenheimer wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Thanks, I hadn't realized that!  No wonder we didn't 
>>>>>>>              
>>>>>>>
>>see the same
>>    
>>
>>>>>>>improvement you saw!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>Try specifying clock=pit on the linux boot line...
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>              
>>>>>>>
>>>>>>command line or
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>              
>>>>>>>
>>>>>>dom0?) command
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>line?  Or both places?  Since the tests take awhile, it
>>>>>>>              
>>>>>>>
>>>>>>would be nice
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>to get this right the first time.  Do the Xen and guest
>>>>>>>              
>>>>>>>
>>>>>>clocksources need
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>to be the same?
>>>>>>>
>>>>>>>Thanks,
>>>>>>>Dan
>>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>*Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
>>>>>>>              
>>>>>>>
>>that disables
>>    
>>
>>>>>>>pending missed ticks
>>>>>>>
>>>>>>>   Hi Dan,
>>>>>>>
>>>>>>>   Hpet timer does have a fairly large error, as I was 
>>>>>>>              
>>>>>>>
>>trying this
>>    
>>
>>>>>>>   one recently.
>>>>>>>   I don't remember what I got for error, but 1% sounds
>>>>>>>              
>>>>>>>
>>>>>>about right.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   The problem is that hpet is not built on top of vpt.c,
>>>>>>>              
>>>>>>>
>>>>>>the module
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   Keir and I did
>>>>>>>   all the recent work in, for its periodic timer needs. Try
>>>>>>>   specifying clock=pit
>>>>>>>   on the linux boot line. If it still picks the hpet, which it
>>>>>>>   might, let me know
>>>>>>>   and I'll tell you how to get around this.
>>>>>>>
>>>>>>>   Regards,
>>>>>>>   Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>--------------------------------------------------------------
>>>>>>----------
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>   *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>   akira.ijuin@oracle.com
>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>              
>>>>>>>
>>>>>>that disables
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   pending missed ticks
>>>>>>>
>>>>>>>   Sorry for the very late followup on this but we finally
>>>>>>>              
>>>>>>>
>>>>>>were able
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   to get our testing set up again on stable 3.1 bits and have
>>>>>>>   seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>>>>
>>>>>>>   Test enviroment was a 4-socket dual core machine with 24GB of
>>>>>>>   memory running six two-vcpu 2GB domains, four hvm 
>>>>>>>              
>>>>>>>
>>plus two pv.
>>    
>>
>>>>>>>   All six guests were running LTP simultaneously.  The four hvm
>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and 
>>>>>>>              
>>>>>>>
>>RHEL4u5-64.
>>    
>>
>>>>>>>   Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>              
>>>>>>>
>>>>>>32-bit guests.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   All four hvm guests experienced skew around -1%, 
>>>>>>>              
>>>>>>>
>>even the 32-bit
>>    
>>
>>>>>>>   guest.  Less intensive testing didn't exhibit much 
>>>>>>>              
>>>>>>>
>>skew at all.
>>    
>>
>>>>>>>   A representative graph is attached.
>>>>>>>
>>>>>>>   Dave, I wonder if some portion of your patches 
>>>>>>>              
>>>>>>>
>>didn't end up in
>>    
>>
>>>>>>>   the xen trees?
>>>>>>>
>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>   14.318MHz HPET.)
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>   Dan
>>>>>>>
>>>>>>>   P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>
>>>>>>>   > -----Original Message-----
>>>>>>>   > From: xen-devel-bounces@lists.xensource.com
>>>>>>>   > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>   > Dave Winchell
>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>   > To: Keir Fraser
>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>              
>>>>>>>
>>>>>>xen-devel@lists.xensource.com; Dave
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > Winchell
>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>   > disables pending
>>>>>>>   > missed ticks
>>>>>>>   >
>>>>>>>   >
>>>>>>>   > Hi Keir,
>>>>>>>   >
>>>>>>>   > The latest change, c/s 16690, looks fine.
>>>>>>>   > I agree that the code in c/s 16690 is equivalent to
>>>>>>>   > the code I submitted. Also, your version is more
>>>>>>>   > concise.
>>>>>>>   >
>>>>>>>   > The error tests confirm the equivalence. With
>>>>>>>              
>>>>>>>
>>>>>>overnight cpu loads,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > the checked in version was accurate to +.048% for sles
>>>>>>>   > and +.038% for red hat. My version was +.046% and 
>>>>>>>              
>>>>>>>
>>+.032% in a
>>    
>>
>>>>>>>   > 2 hour test.
>>>>>>>   > I don't think the difference is significant.
>>>>>>>   >
>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>   >
>>>>>>>   > Thanks for all your efforts on this issue.
>>>>>>>   >
>>>>>>>   > Regards,
>>>>>>>   > Dave
>>>>>>>   >
>>>>>>>   >
>>>>>>>   >
>>>>>>>   > Keir Fraser wrote:
>>>>>>>   >
>>>>>>>   > >Applied as c/s 16690, although the checked-in patch is
>>>>>>>   > smaller. I think the
>>>>>>>   > >only important fix is to pt_intr_post() and the 
>>>>>>>              
>>>>>>>
>>only bit of
>>    
>>
>>>>>>>   > the patch I
>>>>>>>   > >totally omitted was the change to 
>>>>>>>              
>>>>>>>
>>pt_process_missed_ticks().
>>    
>>
>>>>>>>   > I don't think
>>>>>>>   > >that change can be important, but let's see what
>>>>>>>              
>>>>>>>
>>>>>>happens to the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   error
>>>>>>>   > >percentage...
>>>>>>>   > >
>>>>>>>   > > -- Keir
>>>>>>>   > >
>>>>>>>   > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>              
>>>>>>>
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>   > >>
>>>>>>>   > >>Attached is a patch that fixes some issues with the
>>>>>>>              
>>>>>>>
>>>>>>SYNC policy
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>   > >>I have not tried to make the change the minimal one, but,
>>>>>>>   > rather, just
>>>>>>>   > >>ported into
>>>>>>>   > >>the new code what I know to work well. The error for
>>>>>>>   > >>no_missed_ticks_pending goes from
>>>>>>>   > >>over 3% to .03% with this change according to my testing.
>>>>>>>   > >>
>>>>>>>   > >>Regards,
>>>>>>>   > >>Dave
>>>>>>>   > >>
>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>   > >>
>>>>>>>   > >>
>>>>>>>   > >>
>>>>>>>   > >>>Hi Dave --
>>>>>>>   > >>>
>>>>>>>   > >>>Did you get your correction ported?  If so, it would be
>>>>>>>   > nice to see this get
>>>>>>>   > >>>into 3.1.3.
>>>>>>>   > >>>
>>>>>>>   > >>>Note that I just did some very limited testing with
>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>   > >>>missed ticks pending)
>>>>>>>   > >>>on tip of xen-3.1-testing (64-bit Linux hv 
>>>>>>>              
>>>>>>>
>>guest) and the
>>    
>>
>>>>>>>   > worst error I've
>>>>>>>   > >>>seen so far
>>>>>>>   > >>>is 0.012%.  But I haven't tried any exotic 
>>>>>>>              
>>>>>>>
>>loads, just LTP.
>>    
>>
>>>>>>>   > >>>
>>>>>>>   > >>>Thanks,
>>>>>>>   > >>>Dan
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>>-----Original Message-----
>>>>>>>   > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>   > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>   > >>>>To: dan.magenheimer@oracle.com
>>>>>>>   > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>   > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>   > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>   > >>>>disables pending
>>>>>>>   > >>>>missed ticks
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>Dan,
>>>>>>>   > >>>>
>>>>>>>   > >>>>I did some testing with the constant tsc offset
>>>>>>>              
>>>>>>>
>>>>>>SYNC method
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>(now called
>>>>>>>   > >>>>no_missed_ticks_pending)
>>>>>>>   > >>>>and found the error to be very high, much larger
>>>>>>>              
>>>>>>>
>>>>>>than 1 %, as
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>I recall.
>>>>>>>   > >>>>I have not had a chance to submit a correction. I
>>>>>>>              
>>>>>>>
>>>>>>will try to
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>do it later
>>>>>>>   > >>>>this week or the first week in January. My version of
>>>>>>>   constant tsc
>>>>>>>   > >>>>offset SYNC method
>>>>>>>   > >>>>produces .02 % error, so I just need to port 
>>>>>>>              
>>>>>>>
>>that into the
>>    
>>
>>>>>>>   > >>>>current code.
>>>>>>>   > >>>>
>>>>>>>   > >>>>The error you got for both of those kernels is
>>>>>>>              
>>>>>>>
>>>>>>what I would
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   expect
>>>>>>>   > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>   > >>>>
>>>>>>>   > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>   > >>>>
>>>>>>>   > >>>>Regards,
>>>>>>>   > >>>>Dave
>>>>>>>   > >>>>
>>>>>>>   > >>>>Dan Magenheimer wrote:
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>Anyone make measurements on the final patch?
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and 
>>>>>>>              
>>>>>>>
>>saw a loss of
>>    
>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>about 0.2% with no load.  This was 
>>>>>>>              
>>>>>>>
>>xen-unstable tip today
>>    
>>
>>>>>>>   > >>>>with no options specified.  32-bit was about 0.01%.
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>I think I missed something... how do I run the various
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>accounting choices and which ones are known to be
>>>>>>>              
>>>>>>>
>>>>>>appropriate
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>for which kernels?
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>Thanks,
>>>>>>>   > >>>>>Dan
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>>-----Original Message-----
>>>>>>>   > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>   >
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>Keir Fraser
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>   > >>>>>>To: Dave Winchell
>>>>>>>   > >>>>>>Cc: Shan, Haitao; 
>>>>>>>              
>>>>>>>
>>xen-devel@lists.xensource.com; Dong,
>>    
>>
>>>>>>>   > Eddie; Jiang,
>>>>>>>   > >>>>>>Yunhong
>>>>>>>   > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer 
>>>>>>>              
>>>>>>>
>>mode that
>>    
>>
>>>>>>>   > >>>>>>disables pending
>>>>>>>   > >>>>>>missed ticks
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>-- Keir
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>Keir,
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The accuracy data I've collected for i/o 
>>>>>>>              
>>>>>>>
>>loads for the
>>    
>>
>>>>>>>   > >>>>>>>various time protocols follows. In 
>>>>>>>              
>>>>>>>
>>addition, the data
>>    
>>
>>>>>>>   > >>>>>>>for cpu loads is shown.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>              
>>>>>>>
>>>>>>processor AMD
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   box.
>>>>>>>   > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>   > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>   > >>>>>>>(usex is available at
>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>   > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>              
>>>>>>>
>>>>>>of=/dev/null.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>   > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>   > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>   > >>>>>>>All three guests are 8vcpu.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>   > >>>>>>>except that the redhat-64 guest has 4 
>>>>>>>              
>>>>>>>
>>instances of dd.
>>    
>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 
>>>>>>>              
>>>>>>>
>>sec -.006%,
>>    
>>
>>>>>>>   > +.005% cpu
>>>>>>>   > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 
>>>>>>>              
>>>>>>>
>>sec, -.001%,
>>    
>>
>>>>>>>   > +.012% cpu
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>   -.004% cpu
>>>>>>>   > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>              
>>>>>>>
>>>>>>-.005%, -.005% cpu
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>              
>>>>>>>
>>>>>>-.008%, -.003% cpu
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>              
>>>>>>>
>>>>>>-.040% cpu
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 
>>>>>>>              
>>>>>>>
>>sec, -.034%,
>>    
>>
>>>>>>>   > -.031% cpu
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>   > -.09% i/o-8
>>>>>>>   > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>   > -.14% i/o-8
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>   > -.022% i/o-8
>>>>>>>   > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>              
>>>>>>>
>>>>>>-.017%, -.018% i/o-8
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>   > -.029% i/o-8
>>>>>>>   > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 
>>>>>>>              
>>>>>>>
>>sec, -.023%,
>>    
>>
>>>>>>>   > -.031% i/o-8
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>              
>>>>>>>
>>>>>>-.04% i/o-32
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>   > -.005% i/o-32
>>>>>>>   > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>              
>>>>>>>
>>>>>>-.11% i/o-32
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>   > .003% i/o-4/32
>>>>>>>   > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>   > .01% i/o-4/32
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Overhead measurements:
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Progress in terms of number of passes 
>>>>>>>              
>>>>>>>
>>through a fixed
>>    
>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>system workload
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>   > >>>>>>>The workload was usex -b48.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>   > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>   > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>   > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Conclusions:
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>   > requirement for ntp
>>>>>>>   > >>>>>>>tracking under the loads
>>>>>>>   > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>              
>>>>>>>
>>>>>>accuracies for
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>and ASYNC
>>>>>>>   > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>We could reduce the cost of the SYNC method by only
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>scheduling the extra
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>wakeups if a certain number
>>>>>>>   > >>>>>>>of ticks are missed.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Regards,
>>>>>>>   > >>>>>>>Dave
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Keir Fraser wrote:
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>              
>>>>>>>
>>>>>>ASYNC method a
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>couple of days ago,
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>>>>>   > been something
>>>>>>>   > >>>>>>>>>wrong with the code I used a couple of 
>>>>>>>              
>>>>>>>
>>days ago for
>>    
>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>ASYNC. It may have been
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>              
>>>>>>>
>>>>>>after context
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>switch in.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>acceptable accuracy,
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>each running consistently around or under
>>>>>>>              
>>>>>>>
>>>>>>.01%. MIXED has
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>a fairly high
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>error of
>>>>>>>   > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>threshold for comfort.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>              
>>>>>>>
>>>>>>plan to leave
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>SYNC running
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>over the weekend. If you'd rather I can 
>>>>>>>              
>>>>>>>
>>leave MIXED
>>    
>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>running instead.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>              
>>>>>>>
>>>>>>I can run
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>more overnight tests
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>next week.
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>              
>>>>>>>
>>>>>>effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>SYNC+run_timer
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>              
>>>>>>>
>>>>>>cause higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>system-wide CPU
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>contention. I find it easier to think through the
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>implications of ASYNC. I'm
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>              
>>>>>>>
>>>>>>accurate than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>ASYNC. Perhaps it
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>              
>>>>>>>
>>>>>>approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>and each interrupt
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>event causes a small accumulated error?
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>              
>>>>>>>
>>>>>>favourites and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>if the latter is
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>actually more accurate then I can simply revert the
>>>>>>>   > changeset that
>>>>>>>   > >>>>>>>>implemented MIXED.
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>              
>>>>>>>
>>>>>>workloads you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>could try idle
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>              
>>>>>>>
>>>>>>large disc reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>to /dev/null)? We
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>              
>>>>>>>
>>>>>>CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>that's really an
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>-- Keir
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>_______________________________________________
>>>>>>>   > >>>>>>Xen-devel mailing list
>>>>>>>   > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>   > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 
>>>>>>>              
>>>>>>>
>>2007 +0000
>>    
>>
>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 
>>>>>>>              
>>>>>>>
>>2008 -0500
>>    
>>
>>>>>>>   > >>@@ -58,7 +58,7 @@ static void 
>>>>>>>              
>>>>>>>
>>pt_process_missed_ticks(stru
>>    
>>
>>>>>>>   > >>
>>>>>>>   > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>              
>>>>>>>
>>>>>>pt->period + 1;
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>              
>>>>>>>
>>>>>>no_missed_ticks_pending) )
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>   > >>+        pt->do_not_freeze = 1;
>>>>>>>   > >>     else
>>>>>>>   > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>   > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>   > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>>>>>   > >>
>>>>>>>   > >>     pt_lock(pt);
>>>>>>>   > >>
>>>>>>>   > >>-    pt->pending_intr_nr++;
>>>>>>>   > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>              
>>>>>>>
>>>>>>no_missed_ticks_pending) ) {
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>+        pt->pending_intr_nr = 1;
>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>   > >>+    }
>>>>>>>   > >>+    else
>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>   > >>
>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>   > >>     {
>>>>>>>   > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct 
>>>>>>>              
>>>>>>>
>>vcpu *v, struct
>>    
>>
>>>>>>>   > >>         return;
>>>>>>>   > >>     }
>>>>>>>   > >>
>>>>>>>   > >>-    pt->do_not_freeze = 0;
>>>>>>>   > >>-
>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>   > >>     {
>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>   > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>              
>>>>>>>
>>>>>>*v, struct
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>   > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>>>>>   > missed ticks */
>>>>>>>   > >>         }
>>>>>>>   > >>+ else if ( mode_is(v->domain, 
>>>>>>>              
>>>>>>>
>>no_missed_ticks_pending) ) {
>>    
>>
>>>>>>>   > >>+     pt->pending_intr_nr--;
>>>>>>>   > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>   > >>+ }
>>>>>>>   > >>         else
>>>>>>>   > >>         {
>>>>>>>   > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>>>>   > >>
>>>>>>>   > >>
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   >
>>>>>>>   >
>>>>>>>   > _______________________________________________
>>>>>>>   > Xen-devel mailing list
>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>   >
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>            
>>>>>>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

Sorry it took me so long, but I finally ran an ltp test today.
Its on rh4u4-64. I'm using the defaults for ltp and using a script
called runltp. I had a usex load on rh4u5-64. No ntpd.
virtual processors / physical processors = 2.

The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
for -.005% and .008%.

I'm running a 3.1 based hypervisor with some clock related tweaks that
I haven't submitted, because I'm still characterizing them.

I'm stopping the usex load on 4u5-64 now and replacing it with ltp
and will leave the two guests running ltp over the weekend.

Regards,
Dave


Dave Winchell wrote:

> Hi Dan, Deepak:
>
> Thanks for the data. Those drifts are severe - no wonder ntp couldn't
> keep then in synch. I'll try to reproduce that behaviour here, with my 
> code base.
> If I can't reproduce it, I'll try 3.2.
>
> If you can isolate what ltp is doing during the cliffs, that would be 
> very
> helpful.
>
> thanks,
> Dave
>
>
>
>
> Dan Magenheimer wrote:
>
>> OK, Deepak repeated the test without ntpd and using ntpdate -b before
>> the test.
>>
>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>
>> We will continue to look at LTP to try to isolate.
>>
>> Thanks,
>> Dan
>>
>> P.S. elXuY is essentially RHEL XuY with some patches.
>>
>>  
>>
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>> To: Deepak Patel
>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending
>>> missed ticks
>>>
>>>
>>> Dan, Deeepak,
>>>
>>> It may be that the underlying clock error is too great for ntp
>>> to handle. It would be useful if you did not run ntpd
>>> and, instead did ntpdate -b <timeserver> at the start of the test
>>> for each guest. Then capture the data as you have been doing.
>>> If the drift is greater than .05%, then we need to address that.
>>>
>>> Another option is, when running ntpd, to enable loop statistics in
>>> /etc/ntp.conf
>>> by adding this to the file:
>>>
>>> statistics loopstats
>>> statsdir /var/lib/ntp/
>>>
>>> Then you will see loop data in that directory.
>>> Correlating the data in the loopstats files with the
>>> peaks in skew would be interesting. You will see entries of the form
>>>
>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
>>>
>>> Where the second to last column is the Allan Deviation. When that
>>> gets over 1000, ntpd is working pretty hard. However, I have not 
>>> seen ntpd
>>> completely loose it like you have.
>>>
>>> I'm on vacation until Monday, and won't be reading
>>> email.
>>>
>>> Thanks for all your work on this!
>>>
>>> -Dave
>>>
>>> Deepak Patel wrote:
>>>
>>>   
>>>
>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>>       
>>>>
>>>> I do not know which graph was attached with this. But I saw this
>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
>>>> was running ltp tests continuously.
>>>>
>>>>     
>>>>
>>>>> What was the behaviour of the other guests running?
>>>>>       
>>>>
>>>> All pvm guests are fine. But behavior of most of the hvm guests were
>>>> as described.
>>>>
>>>>     
>>>>
>>>>> If they had spikes, were they at the same wall time?
>>>>>       
>>>>
>>>> No. They are not at the same wall time.
>>>>
>>>>     
>>>>
>>>>> Were the other guests running ltp as well?
>>>>>
>>>>>       
>>>>
>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>> continuously.
>>>>
>>>>     
>>>>
>>>>> How are you measuring skew?
>>>>>       
>>>>
>>>> I was collecting output of "ntpdate -q  <timeserver> every     
>>>
>>> 300 seconds
>>>   
>>>
>>>> (5 minutes) and have created graph based on that.
>>>>
>>>>     
>>>>
>>>>> Are you running ntpd?
>>>>>
>>>>>       
>>>>
>>>> Yes. ntp was running on all the guests.
>>>>
>>>> I am investigating what causes this spikes and let everyone     
>>>
>>> know what
>>>   
>>>
>>>> are my findings.
>>>>
>>>> Thanks,
>>>> Deepak
>>>>
>>>>     
>>>>
>>>>> Anything that you can discover that would be in sync with
>>>>> the spikes would be very helpful!
>>>>>
>>>>> The code that I test with is our product code, which is based
>>>>> on 3.1. So it is possible that something in 3.2 other than vpt.c
>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>
>>>>> thanks,
>>>>> Dave
>>>>>
>>>>>
>>>>>
>>>>> Dan Magenheimer wrote:
>>>>>
>>>>>       
>>>>>
>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Turning off vhpet certainly helps a lot (though see below).
>>>>>>
>>>>>> I wonder if timekeeping with vhpet is so bad that it should be
>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is
>>>>>> fixed?  (I have a patch that defaults it off, can post it if
>>>>>> there is agreement on the above point.)  The whole point of an
>>>>>> HPET is to provide more precise timekeeping and if vhpet is
>>>>>> worse than vpit, it can only confuse users.  Comments?
>>>>>>
>>>>>>
>>>>>> In your testing, are you just measuring % skew over a long
>>>>>> period of time?
>>>>>> We are graphing the skew continuously and
>>>>>> seeing periodic behavior that is unsettling, even with pit.
>>>>>> See attached.  Though your algorithm recovers, the "cliffs"
>>>>>> could still cause real user problems.  I wonder if there is
>>>>>> anything that can be done to make the "recovery" more
>>>>>> responsive?
>>>>>>
>>>>>> We are looking into what part(s) of LTP is causing the cliffs.
>>>>>>
>>>>>> Thanks,
>>>>>> Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>> deepak.patel@oracle.com;
>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>> pending
>>>>>>> missed ticks
>>>>>>>
>>>>>>>
>>>>>>> Dan,
>>>>>>>
>>>>>>> I guess I'm a bit out of date calling for clock= usage.
>>>>>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>>>>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>
>>>>>>> You can leave the xen and dom0 bootlines as they are.
>>>>>>> The xen and guest clocksources do not need to be the same.
>>>>>>> In my tests, xen is using the hpet for its timekeeping and
>>>>>>> that appears to be the default.
>>>>>>>
>>>>>>> When you boot the guests you should see
>>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>> on the rh4u5-64 guest, and something similar on the others.
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>> 14.318MHz HPET.)
>>>>>>>>             
>>>>>>>
>>>>>>> This appears to be the xen state, which is fine.
>>>>>>> I was wrongly assuming that this was the guest state.
>>>>>>> You might want to look in your guest logs and see what they were
>>>>>>> picking
>>>>>>> for a clock source.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dan Magenheimer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> Thanks, I hadn't realized that!  No wonder we didn't             
>>>>>>>
>>> see the same
>>>   
>>>
>>>>>>>> improvement you saw!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>> Try specifying clock=pit on the linux boot line...
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>
>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>             
>>>>>>>
>>>>>>> command line or
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>>             
>>>>>>>
>>>>>>> dom0?) command
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> line?  Or both places?  Since the tests take awhile, it
>>>>>>>>             
>>>>>>>
>>>>>>> would be nice
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> to get this right the first time.  Do the Xen and guest
>>>>>>>>             
>>>>>>>
>>>>>>> clocksources need
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> to be the same?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Dan
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode             
>>>>>>>
>>> that disables
>>>   
>>>
>>>>>>>> pending missed ticks
>>>>>>>>
>>>>>>>>   Hi Dan,
>>>>>>>>
>>>>>>>>   Hpet timer does have a fairly large error, as I was             
>>>>>>>
>>> trying this
>>>   
>>>
>>>>>>>>   one recently.
>>>>>>>>   I don't remember what I got for error, but 1% sounds
>>>>>>>>             
>>>>>>>
>>>>>>> about right.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   The problem is that hpet is not built on top of vpt.c,
>>>>>>>>             
>>>>>>>
>>>>>>> the module
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   Keir and I did
>>>>>>>>   all the recent work in, for its periodic timer needs. Try
>>>>>>>>   specifying clock=pit
>>>>>>>>   on the linux boot line. If it still picks the hpet, which it
>>>>>>>>   might, let me know
>>>>>>>>   and I'll tell you how to get around this.
>>>>>>>>
>>>>>>>>   Regards,
>>>>>>>>   Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>>>> --------------------------------------------------------------
>>>>>>> ----------
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>>   *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>>   akira.ijuin@oracle.com
>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>             
>>>>>>>
>>>>>>> that disables
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   pending missed ticks
>>>>>>>>
>>>>>>>>   Sorry for the very late followup on this but we finally
>>>>>>>>             
>>>>>>>
>>>>>>> were able
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   to get our testing set up again on stable 3.1 bits and have
>>>>>>>>   seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>>>>>
>>>>>>>>   Test enviroment was a 4-socket dual core machine with 24GB of
>>>>>>>>   memory running six two-vcpu 2GB domains, four hvm             
>>>>>>>
>>> plus two pv.
>>>   
>>>
>>>>>>>>   All six guests were running LTP simultaneously.  The four hvm
>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and             
>>>>>>>
>>> RHEL4u5-64.
>>>   
>>>
>>>>>>>>   Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>             
>>>>>>>
>>>>>>> 32-bit guests.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   All four hvm guests experienced skew around -1%,             
>>>>>>>
>>> even the 32-bit
>>>   
>>>
>>>>>>>>   guest.  Less intensive testing didn't exhibit much             
>>>>>>>
>>> skew at all.
>>>   
>>>
>>>>>>>>   A representative graph is attached.
>>>>>>>>
>>>>>>>>   Dave, I wonder if some portion of your patches             
>>>>>>>
>>> didn't end up in
>>>   
>>>
>>>>>>>>   the xen trees?
>>>>>>>>
>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>   14.318MHz HPET.)
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>   Dan
>>>>>>>>
>>>>>>>>   P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>>
>>>>>>>>   > -----Original Message-----
>>>>>>>>   > From: xen-devel-bounces@lists.xensource.com
>>>>>>>>   > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>   > Dave Winchell
>>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>   > To: Keir Fraser
>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>>             
>>>>>>>
>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > Winchell
>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>   > disables pending
>>>>>>>>   > missed ticks
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   > Hi Keir,
>>>>>>>>   >
>>>>>>>>   > The latest change, c/s 16690, looks fine.
>>>>>>>>   > I agree that the code in c/s 16690 is equivalent to
>>>>>>>>   > the code I submitted. Also, your version is more
>>>>>>>>   > concise.
>>>>>>>>   >
>>>>>>>>   > The error tests confirm the equivalence. With
>>>>>>>>             
>>>>>>>
>>>>>>> overnight cpu loads,
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > the checked in version was accurate to +.048% for sles
>>>>>>>>   > and +.038% for red hat. My version was +.046% and             
>>>>>>>
>>> +.032% in a
>>>   
>>>
>>>>>>>>   > 2 hour test.
>>>>>>>>   > I don't think the difference is significant.
>>>>>>>>   >
>>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>>   >
>>>>>>>>   > Thanks for all your efforts on this issue.
>>>>>>>>   >
>>>>>>>>   > Regards,
>>>>>>>>   > Dave
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   > Keir Fraser wrote:
>>>>>>>>   >
>>>>>>>>   > >Applied as c/s 16690, although the checked-in patch is
>>>>>>>>   > smaller. I think the
>>>>>>>>   > >only important fix is to pt_intr_post() and the             
>>>>>>>
>>> only bit of
>>>   
>>>
>>>>>>>>   > the patch I
>>>>>>>>   > >totally omitted was the change to             
>>>>>>>
>>> pt_process_missed_ticks().
>>>   
>>>
>>>>>>>>   > I don't think
>>>>>>>>   > >that change can be important, but let's see what
>>>>>>>>             
>>>>>>>
>>>>>>> happens to the
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   error
>>>>>>>>   > >percentage...
>>>>>>>>   > >
>>>>>>>>   > > -- Keir
>>>>>>>>   > >
>>>>>>>>   > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>             
>>>>>>>
>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>>   > >>
>>>>>>>>   > >>Attached is a patch that fixes some issues with the
>>>>>>>>             
>>>>>>>
>>>>>>> SYNC policy
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>>   > >>I have not tried to make the change the minimal one, but,
>>>>>>>>   > rather, just
>>>>>>>>   > >>ported into
>>>>>>>>   > >>the new code what I know to work well. The error for
>>>>>>>>   > >>no_missed_ticks_pending goes from
>>>>>>>>   > >>over 3% to .03% with this change according to my testing.
>>>>>>>>   > >>
>>>>>>>>   > >>Regards,
>>>>>>>>   > >>Dave
>>>>>>>>   > >>
>>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>>   > >>
>>>>>>>>   > >>
>>>>>>>>   > >>
>>>>>>>>   > >>>Hi Dave --
>>>>>>>>   > >>>
>>>>>>>>   > >>>Did you get your correction ported?  If so, it would be
>>>>>>>>   > nice to see this get
>>>>>>>>   > >>>into 3.1.3.
>>>>>>>>   > >>>
>>>>>>>>   > >>>Note that I just did some very limited testing with
>>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>>   > >>>missed ticks pending)
>>>>>>>>   > >>>on tip of xen-3.1-testing (64-bit Linux hv             
>>>>>>>
>>> guest) and the
>>>   
>>>
>>>>>>>>   > worst error I've
>>>>>>>>   > >>>seen so far
>>>>>>>>   > >>>is 0.012%.  But I haven't tried any exotic             
>>>>>>>
>>> loads, just LTP.
>>>   
>>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>Thanks,
>>>>>>>>   > >>>Dan
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>>-----Original Message-----
>>>>>>>>   > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>   > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>   > >>>>To: dan.magenheimer@oracle.com
>>>>>>>>   > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>>   > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>   > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>   > >>>>disables pending
>>>>>>>>   > >>>>missed ticks
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>Dan,
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>I did some testing with the constant tsc offset
>>>>>>>>             
>>>>>>>
>>>>>>> SYNC method
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>(now called
>>>>>>>>   > >>>>no_missed_ticks_pending)
>>>>>>>>   > >>>>and found the error to be very high, much larger
>>>>>>>>             
>>>>>>>
>>>>>>> than 1 %, as
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>I recall.
>>>>>>>>   > >>>>I have not had a chance to submit a correction. I
>>>>>>>>             
>>>>>>>
>>>>>>> will try to
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>do it later
>>>>>>>>   > >>>>this week or the first week in January. My version of
>>>>>>>>   constant tsc
>>>>>>>>   > >>>>offset SYNC method
>>>>>>>>   > >>>>produces .02 % error, so I just need to port             
>>>>>>>
>>> that into the
>>>   
>>>
>>>>>>>>   > >>>>current code.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>The error you got for both of those kernels is
>>>>>>>>             
>>>>>>>
>>>>>>> what I would
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   expect
>>>>>>>>   > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>Regards,
>>>>>>>>   > >>>>Dave
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>Dan Magenheimer wrote:
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>Anyone make measurements on the final patch?
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and             
>>>>>>>
>>> saw a loss of
>>>   
>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>about 0.2% with no load.  This was             
>>>>>>>
>>> xen-unstable tip today
>>>   
>>>
>>>>>>>>   > >>>>with no options specified.  32-bit was about 0.01%.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>I think I missed something... how do I run the various
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>accounting choices and which ones are known to be
>>>>>>>>             
>>>>>>>
>>>>>>> appropriate
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>for which kernels?
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>Thanks,
>>>>>>>>   > >>>>>Dan
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>>-----Original Message-----
>>>>>>>>   > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>   >
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>Keir Fraser
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>   > >>>>>>To: Dave Winchell
>>>>>>>>   > >>>>>>Cc: Shan, Haitao;             
>>>>>>>
>>> xen-devel@lists.xensource.com; Dong,
>>>   
>>>
>>>>>>>>   > Eddie; Jiang,
>>>>>>>>   > >>>>>>Yunhong
>>>>>>>>   > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer 
>>>>>>>>             
>>>>>>>
>>> mode that
>>>   
>>>
>>>>>>>>   > >>>>>>disables pending
>>>>>>>>   > >>>>>>missed ticks
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>-- Keir
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>Keir,
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The accuracy data I've collected for i/o             
>>>>>>>
>>> loads for the
>>>   
>>>
>>>>>>>>   > >>>>>>>various time protocols follows. In             
>>>>>>>
>>> addition, the data
>>>   
>>>
>>>>>>>>   > >>>>>>>for cpu loads is shown.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>             
>>>>>>>
>>>>>>> processor AMD
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   box.
>>>>>>>>   > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>   > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>   > >>>>>>>(usex is available at
>>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>>   > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>             
>>>>>>>
>>>>>>> of=/dev/null.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>   > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>   > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>   > >>>>>>>All three guests are 8vcpu.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>>   > >>>>>>>except that the redhat-64 guest has 4             
>>>>>>>
>>> instances of dd.
>>>   
>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42             
>>>>>>>
>>> sec -.006%,
>>>   
>>>
>>>>>>>>   > +.005% cpu
>>>>>>>>   > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44             
>>>>>>>
>>> sec, -.001%,
>>>   
>>>
>>>>>>>>   > +.012% cpu
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>>   -.004% cpu
>>>>>>>>   > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>             
>>>>>>>
>>>>>>> -.005%, -.005% cpu
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>             
>>>>>>>
>>>>>>> -.008%, -.003% cpu
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>             
>>>>>>>
>>>>>>> -.040% cpu
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4             
>>>>>>>
>>> sec, -.034%,
>>>   
>>>
>>>>>>>>   > -.031% cpu
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>>   > -.09% i/o-8
>>>>>>>>   > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>>   > -.14% i/o-8
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>>   > -.022% i/o-8
>>>>>>>>   > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>             
>>>>>>>
>>>>>>> -.017%, -.018% i/o-8
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>>   > -.029% i/o-8
>>>>>>>>   > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1             
>>>>>>>
>>> sec, -.023%,
>>>   
>>>
>>>>>>>>   > -.031% i/o-8
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>             
>>>>>>>
>>>>>>> -.04% i/o-32
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>>   > -.005% i/o-32
>>>>>>>>   > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>             
>>>>>>>
>>>>>>> -.11% i/o-32
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>>   > .003% i/o-4/32
>>>>>>>>   > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>>   > .01% i/o-4/32
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Overhead measurements:
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Progress in terms of number of passes             
>>>>>>>
>>> through a fixed
>>>   
>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>system workload
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>   > >>>>>>>The workload was usex -b48.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>   > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>   > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>   > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Conclusions:
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>   > requirement for ntp
>>>>>>>>   > >>>>>>>tracking under the loads
>>>>>>>>   > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>             
>>>>>>>
>>>>>>> accuracies for
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>and ASYNC
>>>>>>>>   > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>We could reduce the cost of the SYNC method by only
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>scheduling the extra
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>wakeups if a certain number
>>>>>>>>   > >>>>>>>of ticks are missed.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Regards,
>>>>>>>>   > >>>>>>>Dave
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Keir Fraser wrote:
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>             
>>>>>>>
>>>>>>> ASYNC method a
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>couple of days ago,
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>>>>>>   > been something
>>>>>>>>   > >>>>>>>>>wrong with the code I used a couple of             
>>>>>>>
>>> days ago for
>>>   
>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>ASYNC. It may have been
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>             
>>>>>>>
>>>>>>> after context
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>switch in.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>acceptable accuracy,
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>each running consistently around or under
>>>>>>>>             
>>>>>>>
>>>>>>> .01%. MIXED has
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>a fairly high
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>error of
>>>>>>>>   > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>threshold for comfort.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>             
>>>>>>>
>>>>>>> plan to leave
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>SYNC running
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>over the weekend. If you'd rather I can             
>>>>>>>
>>> leave MIXED
>>>   
>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>running instead.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>             
>>>>>>>
>>>>>>> I can run
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>more overnight tests
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>next week.
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>             
>>>>>>>
>>>>>>> effects of the
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>SYNC+run_timer
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>             
>>>>>>>
>>>>>>> cause higher
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>system-wide CPU
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>contention. I find it easier to think through the
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>implications of ASYNC. I'm
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>             
>>>>>>>
>>>>>>> accurate than
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>ASYNC. Perhaps it
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>>             
>>>>>>>
>>>>>>> approaches,
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>and each interrupt
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>event causes a small accumulated error?
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>             
>>>>>>>
>>>>>>> favourites and
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>if the latter is
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>actually more accurate then I can simply revert the
>>>>>>>>   > changeset that
>>>>>>>>   > >>>>>>>>implemented MIXED.
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>>             
>>>>>>>
>>>>>>> workloads you
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>could try idle
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>             
>>>>>>>
>>>>>>> large disc reads
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>to /dev/null)? We
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>>             
>>>>>>>
>>>>>>> CPU bound, so
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>that's really an
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>-- Keir
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>_______________________________________________
>>>>>>>>   > >>>>>>Xen-devel mailing list
>>>>>>>>   > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>>   > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 
>>>>>>>>             
>>>>>>>
>>> 2007 +0000
>>>   
>>>
>>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 
>>>>>>>>             
>>>>>>>
>>> 2008 -0500
>>>   
>>>
>>>>>>>>   > >>@@ -58,7 +58,7 @@ static void             
>>>>>>>
>>> pt_process_missed_ticks(stru
>>>   
>>>
>>>>>>>>   > >>
>>>>>>>>   > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>             
>>>>>>>
>>>>>>> pt->period + 1;
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>>             
>>>>>>>
>>>>>>> no_missed_ticks_pending) )
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>   > >>+        pt->do_not_freeze = 1;
>>>>>>>>   > >>     else
>>>>>>>>   > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>>   > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>>   > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>>>>>>   > >>
>>>>>>>>   > >>     pt_lock(pt);
>>>>>>>>   > >>
>>>>>>>>   > >>-    pt->pending_intr_nr++;
>>>>>>>>   > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>             
>>>>>>>
>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>+        pt->pending_intr_nr = 1;
>>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>>   > >>+    }
>>>>>>>>   > >>+    else
>>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>>   > >>
>>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>>   > >>     {
>>>>>>>>   > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct             
>>>>>>>
>>> vcpu *v, struct
>>>   
>>>
>>>>>>>>   > >>         return;
>>>>>>>>   > >>     }
>>>>>>>>   > >>
>>>>>>>>   > >>-    pt->do_not_freeze = 0;
>>>>>>>>   > >>-
>>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>>   > >>     {
>>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>>   > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>             
>>>>>>>
>>>>>>> *v, struct
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>   > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>>>>>>   > missed ticks */
>>>>>>>>   > >>         }
>>>>>>>>   > >>+ else if ( mode_is(v->domain,             
>>>>>>>
>>> no_missed_ticks_pending) ) {
>>>   
>>>
>>>>>>>>   > >>+     pt->pending_intr_nr--;
>>>>>>>>   > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>   > >>+ }
>>>>>>>>   > >>         else
>>>>>>>>   > >>         {
>>>>>>>>   > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>   > >>
>>>>>>>>   > >>
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   > _______________________________________________
>>>>>>>>   > Xen-devel mailing list
>>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>>   >
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>>>>           
>>>>>>
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

[-- Attachment #1: Type: text/plain, Size: 42502 bytes --]

Hi Dan,

Over the weekend the drift was +18 seconds for each guest (no ntp).
The duration was 3900 minutes, so the error for each was +.0077%.
Looking back through the data, it appears to drift linearly at
this rate. I've attached a plot for rh4u5-64.

This accuracy is better than what I've seen before (.03-.05%).
This may be due to the different load (ltp vs usex) or to one of the
changes I've made recently. I'll do some experimentation to see if there is
a fix I should propose.

This still doesn't address the radical drift you saw.
The next step for me is to run 3.2 and see if I can reproduce it.

Regards,
Dave





Dave Winchell wrote:

> Hi Dan,
>
> Sorry it took me so long, but I finally ran an ltp test today.
> Its on rh4u4-64. I'm using the defaults for ltp and using a script
> called runltp. I had a usex load on rh4u5-64. No ntpd.
> virtual processors / physical processors = 2.
>
> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
> for -.005% and .008%.
>
> I'm running a 3.1 based hypervisor with some clock related tweaks that
> I haven't submitted, because I'm still characterizing them.
>
> I'm stopping the usex load on 4u5-64 now and replacing it with ltp
> and will leave the two guests running ltp over the weekend.
>
> Regards,
> Dave
>
>
> Dave Winchell wrote:
>
>> Hi Dan, Deepak:
>>
>> Thanks for the data. Those drifts are severe - no wonder ntp couldn't
>> keep then in synch. I'll try to reproduce that behaviour here, with 
>> my code base.
>> If I can't reproduce it, I'll try 3.2.
>>
>> If you can isolate what ltp is doing during the cliffs, that would be 
>> very
>> helpful.
>>
>> thanks,
>> Dave
>>
>>
>>
>>
>> Dan Magenheimer wrote:
>>
>>> OK, Deepak repeated the test without ntpd and using ntpdate -b before
>>> the test.
>>>
>>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>
>>> We will continue to look at LTP to try to isolate.
>>>
>>> Thanks,
>>> Dan
>>>
>>> P.S. elXuY is essentially RHEL XuY with some patches.
>>>
>>>  
>>>
>>>> -----Original Message-----
>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>>> To: Deepak Patel
>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables 
>>>> pending
>>>> missed ticks
>>>>
>>>>
>>>> Dan, Deeepak,
>>>>
>>>> It may be that the underlying clock error is too great for ntp
>>>> to handle. It would be useful if you did not run ntpd
>>>> and, instead did ntpdate -b <timeserver> at the start of the test
>>>> for each guest. Then capture the data as you have been doing.
>>>> If the drift is greater than .05%, then we need to address that.
>>>>
>>>> Another option is, when running ntpd, to enable loop statistics in
>>>> /etc/ntp.conf
>>>> by adding this to the file:
>>>>
>>>> statistics loopstats
>>>> statsdir /var/lib/ntp/
>>>>
>>>> Then you will see loop data in that directory.
>>>> Correlating the data in the loopstats files with the
>>>> peaks in skew would be interesting. You will see entries of the form
>>>>
>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
>>>>
>>>> Where the second to last column is the Allan Deviation. When that
>>>> gets over 1000, ntpd is working pretty hard. However, I have not 
>>>> seen ntpd
>>>> completely loose it like you have.
>>>>
>>>> I'm on vacation until Monday, and won't be reading
>>>> email.
>>>>
>>>> Thanks for all your work on this!
>>>>
>>>> -Dave
>>>>
>>>> Deepak Patel wrote:
>>>>
>>>>  
>>>>
>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>>>       
>>>>>
>>>>>
>>>>> I do not know which graph was attached with this. But I saw this
>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
>>>>> was running ltp tests continuously.
>>>>>
>>>>>    
>>>>>
>>>>>> What was the behaviour of the other guests running?
>>>>>>       
>>>>>
>>>>>
>>>>> All pvm guests are fine. But behavior of most of the hvm guests were
>>>>> as described.
>>>>>
>>>>>    
>>>>>
>>>>>> If they had spikes, were they at the same wall time?
>>>>>>       
>>>>>
>>>>>
>>>>> No. They are not at the same wall time.
>>>>>
>>>>>    
>>>>>
>>>>>> Were the other guests running ltp as well?
>>>>>>
>>>>>>       
>>>>>
>>>>>
>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>>> continuously.
>>>>>
>>>>>    
>>>>>
>>>>>> How are you measuring skew?
>>>>>>       
>>>>>
>>>>>
>>>>> I was collecting output of "ntpdate -q  <timeserver> every     
>>>>
>>>>
>>>> 300 seconds
>>>>  
>>>>
>>>>> (5 minutes) and have created graph based on that.
>>>>>
>>>>>    
>>>>>
>>>>>> Are you running ntpd?
>>>>>>
>>>>>>       
>>>>>
>>>>>
>>>>> Yes. ntp was running on all the guests.
>>>>>
>>>>> I am investigating what causes this spikes and let everyone     
>>>>
>>>>
>>>> know what
>>>>  
>>>>
>>>>> are my findings.
>>>>>
>>>>> Thanks,
>>>>> Deepak
>>>>>
>>>>>    
>>>>>
>>>>>> Anything that you can discover that would be in sync with
>>>>>> the spikes would be very helpful!
>>>>>>
>>>>>> The code that I test with is our product code, which is based
>>>>>> on 3.1. So it is possible that something in 3.2 other than vpt.c
>>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>>
>>>>>> thanks,
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dan Magenheimer wrote:
>>>>>>
>>>>>>      
>>>>>>
>>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Turning off vhpet certainly helps a lot (though see below).
>>>>>>>
>>>>>>> I wonder if timekeeping with vhpet is so bad that it should be
>>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is
>>>>>>> fixed?  (I have a patch that defaults it off, can post it if
>>>>>>> there is agreement on the above point.)  The whole point of an
>>>>>>> HPET is to provide more precise timekeeping and if vhpet is
>>>>>>> worse than vpit, it can only confuse users.  Comments?
>>>>>>>
>>>>>>>
>>>>>>> In your testing, are you just measuring % skew over a long
>>>>>>> period of time?
>>>>>>> We are graphing the skew continuously and
>>>>>>> seeing periodic behavior that is unsettling, even with pit.
>>>>>>> See attached.  Though your algorithm recovers, the "cliffs"
>>>>>>> could still cause real user problems.  I wonder if there is
>>>>>>> anything that can be done to make the "recovery" more
>>>>>>> responsive?
>>>>>>>
>>>>>>> We are looking into what part(s) of LTP is causing the cliffs.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>> deepak.patel@oracle.com;
>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>>> pending
>>>>>>>> missed ticks
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan,
>>>>>>>>
>>>>>>>> I guess I'm a bit out of date calling for clock= usage.
>>>>>>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>>
>>>>>>>> You can leave the xen and dom0 bootlines as they are.
>>>>>>>> The xen and guest clocksources do not need to be the same.
>>>>>>>> In my tests, xen is using the hpet for its timekeeping and
>>>>>>>> that appears to be the default.
>>>>>>>>
>>>>>>>> When you boot the guests you should see
>>>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>>> on the rh4u5-64 guest, and something similar on the others.
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> This appears to be the xen state, which is fine.
>>>>>>>> I was wrongly assuming that this was the guest state.
>>>>>>>> You might want to look in your guest logs and see what they were
>>>>>>>> picking
>>>>>>>> for a clock source.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> Thanks, I hadn't realized that!  No wonder we didn't             
>>>>>>>>
>>>>>>>>
>>>> see the same
>>>>  
>>>>
>>>>>>>>> improvement you saw!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>> Try specifying clock=pit on the linux boot line...
>>>>>>>>>>
>>>>>>>>>>               
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> command line or
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> dom0?) command
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> line?  Or both places?  Since the tests take awhile, it
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> would be nice
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> to get this right the first time.  Do the Xen and guest
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> clocksources need
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> to be the same?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode             
>>>>>>>>
>>>>>>>>
>>>> that disables
>>>>  
>>>>
>>>>>>>>> pending missed ticks
>>>>>>>>>
>>>>>>>>>   Hi Dan,
>>>>>>>>>
>>>>>>>>>   Hpet timer does have a fairly large error, as I was             
>>>>>>>>
>>>>>>>>
>>>> trying this
>>>>  
>>>>
>>>>>>>>>   one recently.
>>>>>>>>>   I don't remember what I got for error, but 1% sounds
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> about right.
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   The problem is that hpet is not built on top of vpt.c,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> the module
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   Keir and I did
>>>>>>>>>   all the recent work in, for its periodic timer needs. Try
>>>>>>>>>   specifying clock=pit
>>>>>>>>>   on the linux boot line. If it still picks the hpet, which it
>>>>>>>>>   might, let me know
>>>>>>>>>   and I'll tell you how to get around this.
>>>>>>>>>
>>>>>>>>>   Regards,
>>>>>>>>>   Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------
>>>>>>>> ----------
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>>>   *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>>>   akira.ijuin@oracle.com
>>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> that disables
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   pending missed ticks
>>>>>>>>>
>>>>>>>>>   Sorry for the very late followup on this but we finally
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> were able
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   to get our testing set up again on stable 3.1 bits and have
>>>>>>>>>   seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>>>>>>
>>>>>>>>>   Test enviroment was a 4-socket dual core machine with 24GB of
>>>>>>>>>   memory running six two-vcpu 2GB domains, four hvm             
>>>>>>>>
>>>>>>>>
>>>> plus two pv.
>>>>  
>>>>
>>>>>>>>>   All six guests were running LTP simultaneously.  The four hvm
>>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and             
>>>>>>>>
>>>>>>>>
>>>> RHEL4u5-64.
>>>>  
>>>>
>>>>>>>>>   Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> 32-bit guests.
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   All four hvm guests experienced skew around -1%,             
>>>>>>>>
>>>>>>>>
>>>> even the 32-bit
>>>>  
>>>>
>>>>>>>>>   guest.  Less intensive testing didn't exhibit much             
>>>>>>>>
>>>>>>>>
>>>> skew at all.
>>>>  
>>>>
>>>>>>>>>   A representative graph is attached.
>>>>>>>>>
>>>>>>>>>   Dave, I wonder if some portion of your patches             
>>>>>>>>
>>>>>>>>
>>>> didn't end up in
>>>>  
>>>>
>>>>>>>>>   the xen trees?
>>>>>>>>>
>>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>   14.318MHz HPET.)
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>   Dan
>>>>>>>>>
>>>>>>>>>   P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>>>
>>>>>>>>>   > -----Original Message-----
>>>>>>>>>   > From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>   > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>   > Dave Winchell
>>>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>   > To: Keir Fraser
>>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > Winchell
>>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>   > disables pending
>>>>>>>>>   > missed ticks
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   > Hi Keir,
>>>>>>>>>   >
>>>>>>>>>   > The latest change, c/s 16690, looks fine.
>>>>>>>>>   > I agree that the code in c/s 16690 is equivalent to
>>>>>>>>>   > the code I submitted. Also, your version is more
>>>>>>>>>   > concise.
>>>>>>>>>   >
>>>>>>>>>   > The error tests confirm the equivalence. With
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> overnight cpu loads,
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > the checked in version was accurate to +.048% for sles
>>>>>>>>>   > and +.038% for red hat. My version was +.046% and             
>>>>>>>>
>>>>>>>>
>>>> +.032% in a
>>>>  
>>>>
>>>>>>>>>   > 2 hour test.
>>>>>>>>>   > I don't think the difference is significant.
>>>>>>>>>   >
>>>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>>>   >
>>>>>>>>>   > Thanks for all your efforts on this issue.
>>>>>>>>>   >
>>>>>>>>>   > Regards,
>>>>>>>>>   > Dave
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   > Keir Fraser wrote:
>>>>>>>>>   >
>>>>>>>>>   > >Applied as c/s 16690, although the checked-in patch is
>>>>>>>>>   > smaller. I think the
>>>>>>>>>   > >only important fix is to pt_intr_post() and the             
>>>>>>>>
>>>>>>>>
>>>> only bit of
>>>>  
>>>>
>>>>>>>>>   > the patch I
>>>>>>>>>   > >totally omitted was the change to             
>>>>>>>>
>>>>>>>>
>>>> pt_process_missed_ticks().
>>>>  
>>>>
>>>>>>>>>   > I don't think
>>>>>>>>>   > >that change can be important, but let's see what
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> happens to the
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   error
>>>>>>>>>   > >percentage...
>>>>>>>>>   > >
>>>>>>>>>   > > -- Keir
>>>>>>>>>   > >
>>>>>>>>>   > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>>>   > >>
>>>>>>>>>   > >>Attached is a patch that fixes some issues with the
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> SYNC policy
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>>>   > >>I have not tried to make the change the minimal one, but,
>>>>>>>>>   > rather, just
>>>>>>>>>   > >>ported into
>>>>>>>>>   > >>the new code what I know to work well. The error for
>>>>>>>>>   > >>no_missed_ticks_pending goes from
>>>>>>>>>   > >>over 3% to .03% with this change according to my testing.
>>>>>>>>>   > >>
>>>>>>>>>   > >>Regards,
>>>>>>>>>   > >>Dave
>>>>>>>>>   > >>
>>>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>>>   > >>
>>>>>>>>>   > >>
>>>>>>>>>   > >>
>>>>>>>>>   > >>>Hi Dave --
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>Did you get your correction ported?  If so, it would be
>>>>>>>>>   > nice to see this get
>>>>>>>>>   > >>>into 3.1.3.
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>Note that I just did some very limited testing with
>>>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>>>   > >>>missed ticks pending)
>>>>>>>>>   > >>>on tip of xen-3.1-testing (64-bit Linux hv             
>>>>>>>>
>>>>>>>>
>>>> guest) and the
>>>>  
>>>>
>>>>>>>>>   > worst error I've
>>>>>>>>>   > >>>seen so far
>>>>>>>>>   > >>>is 0.012%.  But I haven't tried any exotic             
>>>>>>>>
>>>>>>>>
>>>> loads, just LTP.
>>>>  
>>>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>Thanks,
>>>>>>>>>   > >>>Dan
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>>-----Original Message-----
>>>>>>>>>   > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>   > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>   > >>>>To: dan.magenheimer@oracle.com
>>>>>>>>>   > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>>>   > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>   > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>   > >>>>disables pending
>>>>>>>>>   > >>>>missed ticks
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>Dan,
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>I did some testing with the constant tsc offset
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> SYNC method
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>(now called
>>>>>>>>>   > >>>>no_missed_ticks_pending)
>>>>>>>>>   > >>>>and found the error to be very high, much larger
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> than 1 %, as
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>I recall.
>>>>>>>>>   > >>>>I have not had a chance to submit a correction. I
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> will try to
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>do it later
>>>>>>>>>   > >>>>this week or the first week in January. My version of
>>>>>>>>>   constant tsc
>>>>>>>>>   > >>>>offset SYNC method
>>>>>>>>>   > >>>>produces .02 % error, so I just need to port             
>>>>>>>>
>>>>>>>>
>>>> that into the
>>>>  
>>>>
>>>>>>>>>   > >>>>current code.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>The error you got for both of those kernels is
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> what I would
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   expect
>>>>>>>>>   > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>Regards,
>>>>>>>>>   > >>>>Dave
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>Dan Magenheimer wrote:
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>Anyone make measurements on the final patch?
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and             
>>>>>>>>
>>>>>>>>
>>>> saw a loss of
>>>>  
>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>about 0.2% with no load.  This was             
>>>>>>>>
>>>>>>>>
>>>> xen-unstable tip today
>>>>  
>>>>
>>>>>>>>>   > >>>>with no options specified.  32-bit was about 0.01%.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>I think I missed something... how do I run the various
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>accounting choices and which ones are known to be
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> appropriate
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>for which kernels?
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>Thanks,
>>>>>>>>>   > >>>>>Dan
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>>-----Original Message-----
>>>>>>>>>   > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>Keir Fraser
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>   > >>>>>>To: Dave Winchell
>>>>>>>>>   > >>>>>>Cc: Shan, Haitao;             
>>>>>>>>
>>>>>>>>
>>>> xen-devel@lists.xensource.com; Dong,
>>>>  
>>>>
>>>>>>>>>   > Eddie; Jiang,
>>>>>>>>>   > >>>>>>Yunhong
>>>>>>>>>   > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer 
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>> mode that
>>>>  
>>>>
>>>>>>>>>   > >>>>>>disables pending
>>>>>>>>>   > >>>>>>missed ticks
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>-- Keir
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>Keir,
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The accuracy data I've collected for i/o             
>>>>>>>>
>>>>>>>>
>>>> loads for the
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>various time protocols follows. In             
>>>>>>>>
>>>>>>>>
>>>> addition, the data
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>for cpu loads is shown.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> processor AMD
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   box.
>>>>>>>>>   > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>>   > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>   > >>>>>>>(usex is available at
>>>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>>>   > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> of=/dev/null.
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>>   > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>   > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>   > >>>>>>>All three guests are 8vcpu.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>>>   > >>>>>>>except that the redhat-64 guest has 4             
>>>>>>>>
>>>>>>>>
>>>> instances of dd.
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42             
>>>>>>>>
>>>>>>>>
>>>> sec -.006%,
>>>>  
>>>>
>>>>>>>>>   > +.005% cpu
>>>>>>>>>   > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44             
>>>>>>>>
>>>>>>>>
>>>> sec, -.001%,
>>>>  
>>>>
>>>>>>>>>   > +.012% cpu
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>>>   -.004% cpu
>>>>>>>>>   > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.005%, -.005% cpu
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.008%, -.003% cpu
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.040% cpu
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4             
>>>>>>>>
>>>>>>>>
>>>> sec, -.034%,
>>>>  
>>>>
>>>>>>>>>   > -.031% cpu
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>>>   > -.09% i/o-8
>>>>>>>>>   > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>>>   > -.14% i/o-8
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>>>   > -.022% i/o-8
>>>>>>>>>   > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.017%, -.018% i/o-8
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>>>   > -.029% i/o-8
>>>>>>>>>   > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1             
>>>>>>>>
>>>>>>>>
>>>> sec, -.023%,
>>>>  
>>>>
>>>>>>>>>   > -.031% i/o-8
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.04% i/o-32
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>>>   > -.005% i/o-32
>>>>>>>>>   > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.11% i/o-32
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>>>   > .003% i/o-4/32
>>>>>>>>>   > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>>>   > .01% i/o-4/32
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Overhead measurements:
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Progress in terms of number of passes             
>>>>>>>>
>>>>>>>>
>>>> through a fixed
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>system workload
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>   > >>>>>>>The workload was usex -b48.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>   > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>   > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>   > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Conclusions:
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>   > requirement for ntp
>>>>>>>>>   > >>>>>>>tracking under the loads
>>>>>>>>>   > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> accuracies for
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>and ASYNC
>>>>>>>>>   > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>We could reduce the cost of the SYNC method by only
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>scheduling the extra
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>wakeups if a certain number
>>>>>>>>>   > >>>>>>>of ticks are missed.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Regards,
>>>>>>>>>   > >>>>>>>Dave
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Keir Fraser wrote:
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> ASYNC method a
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>couple of days ago,
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>>>>>>>   > been something
>>>>>>>>>   > >>>>>>>>>wrong with the code I used a couple of             
>>>>>>>>
>>>>>>>>
>>>> days ago for
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>ASYNC. It may have been
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> after context
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>switch in.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>acceptable accuracy,
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>each running consistently around or under
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> .01%. MIXED has
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>a fairly high
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>error of
>>>>>>>>>   > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>threshold for comfort.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> plan to leave
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>SYNC running
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>over the weekend. If you'd rather I can             
>>>>>>>>
>>>>>>>>
>>>> leave MIXED
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>running instead.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> I can run
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>more overnight tests
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>>next week.
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> effects of the
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>SYNC+run_timer
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> cause higher
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>system-wide CPU
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>contention. I find it easier to think through the
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>implications of ASYNC. I'm
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> accurate than
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>ASYNC. Perhaps it
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> approaches,
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>and each interrupt
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>event causes a small accumulated error?
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> favourites and
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>if the latter is
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>actually more accurate then I can simply revert the
>>>>>>>>>   > changeset that
>>>>>>>>>   > >>>>>>>>implemented MIXED.
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> workloads you
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>could try idle
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> large disc reads
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>to /dev/null)? We
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> CPU bound, so
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>that's really an
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>-- Keir
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>_______________________________________________
>>>>>>>>>   > >>>>>>Xen-devel mailing list
>>>>>>>>>   > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>   > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>> 2007 +0000
>>>>  
>>>>
>>>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>> 2008 -0500
>>>>  
>>>>
>>>>>>>>>   > >>@@ -58,7 +58,7 @@ static void             
>>>>>>>>
>>>>>>>>
>>>> pt_process_missed_ticks(stru
>>>>  
>>>>
>>>>>>>>>   > >>
>>>>>>>>>   > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> pt->period + 1;
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> no_missed_ticks_pending) )
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>   > >>+        pt->do_not_freeze = 1;
>>>>>>>>>   > >>     else
>>>>>>>>>   > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>>>   > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>   > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>>>>>>>   > >>
>>>>>>>>>   > >>     pt_lock(pt);
>>>>>>>>>   > >>
>>>>>>>>>   > >>-    pt->pending_intr_nr++;
>>>>>>>>>   > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>+        pt->pending_intr_nr = 1;
>>>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>>>   > >>+    }
>>>>>>>>>   > >>+    else
>>>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>>>   > >>
>>>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>>>   > >>     {
>>>>>>>>>   > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct             
>>>>>>>>
>>>>>>>>
>>>> vcpu *v, struct
>>>>  
>>>>
>>>>>>>>>   > >>         return;
>>>>>>>>>   > >>     }
>>>>>>>>>   > >>
>>>>>>>>>   > >>-    pt->do_not_freeze = 0;
>>>>>>>>>   > >>-
>>>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>>>   > >>     {
>>>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>>>   > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> *v, struct
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>   > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>>>>>>>   > missed ticks */
>>>>>>>>>   > >>         }
>>>>>>>>>   > >>+ else if ( mode_is(v->domain,             
>>>>>>>>
>>>>>>>>
>>>> no_missed_ticks_pending) ) {
>>>>  
>>>>
>>>>>>>>>   > >>+     pt->pending_intr_nr--;
>>>>>>>>>   > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>   > >>+ }
>>>>>>>>>   > >>         else
>>>>>>>>>   > >>         {
>>>>>>>>>   > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>>   > >>
>>>>>>>>>   > >>
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   > _______________________________________________
>>>>>>>>>   > Xen-devel mailing list
>>>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>
>>>>>>>
>>
>


[-- Attachment #2: rh4u5.64.2.11.png --]
[-- Type: image/png, Size: 3951 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

I ran the ltp tests with 3.2 and found the errors
for a 16 hour run to be:

rh4u564 -9.9 sec (-.017%)
rh4u464 -7.3 sec (-.013%)

There were no cliffs and the drift was linear.

I think the problem you had may be due to the use of the
pm timer. If you still have the boot log, it would tell you.

When I first tried a guest on 3.2 with "clocksource=pit nohpet"
I noticed that it picked the pm timer. Adding "nopmtimer", the
guest will pick the pit.

The reason I didn't have the problem with our 3.1 base is that
I had disabled the hpet and the pmtimer by not advertising them
in the acpi tables. I did this so long ago, I forgot that I had to
disable pmtimer as well as hpet.

So, can you re-run your test with "clocksource=pit nohpet nopmtimer"?
You should see this in the boot messages:

time.c: Using PIT/TSC based timekeeping.

Thanks,
Dave


Dave Winchell wrote:

> Hi Dan,
>
> Over the weekend the drift was +18 seconds for each guest (no ntp).
> The duration was 3900 minutes, so the error for each was +.0077%.
> Looking back through the data, it appears to drift linearly at
> this rate. I've attached a plot for rh4u5-64.
>
> This accuracy is better than what I've seen before (.03-.05%).
> This may be due to the different load (ltp vs usex) or to one of the
> changes I've made recently. I'll do some experimentation to see if 
> there is
> a fix I should propose.
>
> This still doesn't address the radical drift you saw.
> The next step for me is to run 3.2 and see if I can reproduce it.
>
> Regards,
> Dave
>
>
>
>
>
> Dave Winchell wrote:
>
>> Hi Dan,
>>
>> Sorry it took me so long, but I finally ran an ltp test today.
>> Its on rh4u4-64. I'm using the defaults for ltp and using a script
>> called runltp. I had a usex load on rh4u5-64. No ntpd.
>> virtual processors / physical processors = 2.
>>
>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
>> for -.005% and .008%.
>>
>> I'm running a 3.1 based hypervisor with some clock related tweaks that
>> I haven't submitted, because I'm still characterizing them.
>>
>> I'm stopping the usex load on 4u5-64 now and replacing it with ltp
>> and will leave the two guests running ltp over the weekend.
>>
>> Regards,
>> Dave
>>
>>
>> Dave Winchell wrote:
>>
>>> Hi Dan, Deepak:
>>>
>>> Thanks for the data. Those drifts are severe - no wonder ntp couldn't
>>> keep then in synch. I'll try to reproduce that behaviour here, with 
>>> my code base.
>>> If I can't reproduce it, I'll try 3.2.
>>>
>>> If you can isolate what ltp is doing during the cliffs, that would 
>>> be very
>>> helpful.
>>>
>>> thanks,
>>> Dave
>>>
>>>
>>>
>>>
>>> Dan Magenheimer wrote:
>>>
>>>> OK, Deepak repeated the test without ntpd and using ntpdate -b before
>>>> the test.
>>>>
>>>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>
>>>> We will continue to look at LTP to try to isolate.
>>>>
>>>> Thanks,
>>>> Dan
>>>>
>>>> P.S. elXuY is essentially RHEL XuY with some patches.
>>>>
>>>>  
>>>>
>>>>> -----Original Message-----
>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>>>> To: Deepak Patel
>>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables 
>>>>> pending
>>>>> missed ticks
>>>>>
>>>>>
>>>>> Dan, Deeepak,
>>>>>
>>>>> It may be that the underlying clock error is too great for ntp
>>>>> to handle. It would be useful if you did not run ntpd
>>>>> and, instead did ntpdate -b <timeserver> at the start of the test
>>>>> for each guest. Then capture the data as you have been doing.
>>>>> If the drift is greater than .05%, then we need to address that.
>>>>>
>>>>> Another option is, when running ntpd, to enable loop statistics in
>>>>> /etc/ntp.conf
>>>>> by adding this to the file:
>>>>>
>>>>> statistics loopstats
>>>>> statsdir /var/lib/ntp/
>>>>>
>>>>> Then you will see loop data in that directory.
>>>>> Correlating the data in the loopstats files with the
>>>>> peaks in skew would be interesting. You will see entries of the form
>>>>>
>>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
>>>>>
>>>>> Where the second to last column is the Allan Deviation. When that
>>>>> gets over 1000, ntpd is working pretty hard. However, I have not 
>>>>> seen ntpd
>>>>> completely loose it like you have.
>>>>>
>>>>> I'm on vacation until Monday, and won't be reading
>>>>> email.
>>>>>
>>>>> Thanks for all your work on this!
>>>>>
>>>>> -Dave
>>>>>
>>>>> Deepak Patel wrote:
>>>>>
>>>>>  
>>>>>
>>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> I do not know which graph was attached with this. But I saw this
>>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
>>>>>> was running ltp tests continuously.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> What was the behaviour of the other guests running?
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> All pvm guests are fine. But behavior of most of the hvm guests were
>>>>>> as described.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> If they had spikes, were they at the same wall time?
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> No. They are not at the same wall time.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> Were the other guests running ltp as well?
>>>>>>>
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>>>> continuously.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> How are you measuring skew?
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> I was collecting output of "ntpdate -q  <timeserver> every     
>>>>>
>>>>>
>>>>>
>>>>> 300 seconds
>>>>>  
>>>>>
>>>>>> (5 minutes) and have created graph based on that.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> Are you running ntpd?
>>>>>>>
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes. ntp was running on all the guests.
>>>>>>
>>>>>> I am investigating what causes this spikes and let everyone     
>>>>>
>>>>>
>>>>>
>>>>> know what
>>>>>  
>>>>>
>>>>>> are my findings.
>>>>>>
>>>>>> Thanks,
>>>>>> Deepak
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> Anything that you can discover that would be in sync with
>>>>>>> the spikes would be very helpful!
>>>>>>>
>>>>>>> The code that I test with is our product code, which is based
>>>>>>> on 3.1. So it is possible that something in 3.2 other than vpt.c
>>>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>>>
>>>>>>> thanks,
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dan Magenheimer wrote:
>>>>>>>
>>>>>>>     
>>>>>>>
>>>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Turning off vhpet certainly helps a lot (though see below).
>>>>>>>>
>>>>>>>> I wonder if timekeeping with vhpet is so bad that it should be
>>>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is
>>>>>>>> fixed?  (I have a patch that defaults it off, can post it if
>>>>>>>> there is agreement on the above point.)  The whole point of an
>>>>>>>> HPET is to provide more precise timekeeping and if vhpet is
>>>>>>>> worse than vpit, it can only confuse users.  Comments?
>>>>>>>>
>>>>>>>>
>>>>>>>> In your testing, are you just measuring % skew over a long
>>>>>>>> period of time?
>>>>>>>> We are graphing the skew continuously and
>>>>>>>> seeing periodic behavior that is unsettling, even with pit.
>>>>>>>> See attached.  Though your algorithm recovers, the "cliffs"
>>>>>>>> could still cause real user problems.  I wonder if there is
>>>>>>>> anything that can be done to make the "recovery" more
>>>>>>>> responsive?
>>>>>>>>
>>>>>>>> We are looking into what part(s) of LTP is causing the cliffs.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Dan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>       
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>>> deepak.patel@oracle.com;
>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>>>> pending
>>>>>>>>> missed ticks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dan,
>>>>>>>>>
>>>>>>>>> I guess I'm a bit out of date calling for clock= usage.
>>>>>>>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>>>
>>>>>>>>> You can leave the xen and dom0 bootlines as they are.
>>>>>>>>> The xen and guest clocksources do not need to be the same.
>>>>>>>>> In my tests, xen is using the hpet for its timekeeping and
>>>>>>>>> that appears to be the default.
>>>>>>>>>
>>>>>>>>> When you boot the guests you should see
>>>>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>>>> on the rh4u5-64 guest, and something similar on the others.
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This appears to be the xen state, which is fine.
>>>>>>>>> I was wrongly assuming that this was the guest state.
>>>>>>>>> You might want to look in your guest logs and see what they were
>>>>>>>>> picking
>>>>>>>>> for a clock source.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> Thanks, I hadn't realized that!  No wonder we didn't             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> see the same
>>>>>  
>>>>>
>>>>>>>>>> improvement you saw!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>           
>>>>>>>>>>
>>>>>>>>>>> Try specifying clock=pit on the linux boot line...
>>>>>>>>>>>
>>>>>>>>>>>               
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> command line or
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> dom0?) command
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> line?  Or both places?  Since the tests take awhile, it
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> would be nice
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> to get this right the first time.  Do the Xen and guest
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> clocksources need
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> to be the same?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Dan
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> that disables
>>>>>  
>>>>>
>>>>>>>>>> pending missed ticks
>>>>>>>>>>
>>>>>>>>>>   Hi Dan,
>>>>>>>>>>
>>>>>>>>>>   Hpet timer does have a fairly large error, as I 
>>>>>>>>>> was             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> trying this
>>>>>  
>>>>>
>>>>>>>>>>   one recently.
>>>>>>>>>>   I don't remember what I got for error, but 1% sounds
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> about right.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   The problem is that hpet is not built on top of vpt.c,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the module
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   Keir and I did
>>>>>>>>>>   all the recent work in, for its periodic timer needs. Try
>>>>>>>>>>   specifying clock=pit
>>>>>>>>>>   on the linux boot line. If it still picks the hpet, which it
>>>>>>>>>>   might, let me know
>>>>>>>>>>   and I'll tell you how to get around this.
>>>>>>>>>>
>>>>>>>>>>   Regards,
>>>>>>>>>>   Dave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> ----------
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>>>>   *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>>>>>>>>   akira.ijuin@oracle.com
>>>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> that disables
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   pending missed ticks
>>>>>>>>>>
>>>>>>>>>>   Sorry for the very late followup on this but we finally
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> were able
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   to get our testing set up again on stable 3.1 bits and have
>>>>>>>>>>   seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>>>>>>>
>>>>>>>>>>   Test enviroment was a 4-socket dual core machine with 24GB of
>>>>>>>>>>   memory running six two-vcpu 2GB domains, four hvm             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> plus two pv.
>>>>>  
>>>>>
>>>>>>>>>>   All six guests were running LTP simultaneously.  The four hvm
>>>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> RHEL4u5-64.
>>>>>  
>>>>>
>>>>>>>>>>   Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 32-bit guests.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   All four hvm guests experienced skew around -1%,             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> even the 32-bit
>>>>>  
>>>>>
>>>>>>>>>>   guest.  Less intensive testing didn't exhibit much             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> skew at all.
>>>>>  
>>>>>
>>>>>>>>>>   A representative graph is attached.
>>>>>>>>>>
>>>>>>>>>>   Dave, I wonder if some portion of your patches             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> didn't end up in
>>>>>  
>>>>>
>>>>>>>>>>   the xen trees?
>>>>>>>>>>
>>>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>>   14.318MHz HPET.)
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>   Dan
>>>>>>>>>>
>>>>>>>>>>   P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>>>>
>>>>>>>>>>   > -----Original Message-----
>>>>>>>>>>   > From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>   > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>   > Dave Winchell
>>>>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>   > To: Keir Fraser
>>>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > Winchell
>>>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>   > disables pending
>>>>>>>>>>   > missed ticks
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   > Hi Keir,
>>>>>>>>>>   >
>>>>>>>>>>   > The latest change, c/s 16690, looks fine.
>>>>>>>>>>   > I agree that the code in c/s 16690 is equivalent to
>>>>>>>>>>   > the code I submitted. Also, your version is more
>>>>>>>>>>   > concise.
>>>>>>>>>>   >
>>>>>>>>>>   > The error tests confirm the equivalence. With
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> overnight cpu loads,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > the checked in version was accurate to +.048% for sles
>>>>>>>>>>   > and +.038% for red hat. My version was +.046% 
>>>>>>>>>> and             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> +.032% in a
>>>>>  
>>>>>
>>>>>>>>>>   > 2 hour test.
>>>>>>>>>>   > I don't think the difference is significant.
>>>>>>>>>>   >
>>>>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>>>>   >
>>>>>>>>>>   > Thanks for all your efforts on this issue.
>>>>>>>>>>   >
>>>>>>>>>>   > Regards,
>>>>>>>>>>   > Dave
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   > Keir Fraser wrote:
>>>>>>>>>>   >
>>>>>>>>>>   > >Applied as c/s 16690, although the checked-in patch is
>>>>>>>>>>   > smaller. I think the
>>>>>>>>>>   > >only important fix is to pt_intr_post() and the             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> only bit of
>>>>>  
>>>>>
>>>>>>>>>>   > the patch I
>>>>>>>>>>   > >totally omitted was the change to             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> pt_process_missed_ticks().
>>>>>  
>>>>>
>>>>>>>>>>   > I don't think
>>>>>>>>>>   > >that change can be important, but let's see what
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> happens to the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   error
>>>>>>>>>>   > >percentage...
>>>>>>>>>>   > >
>>>>>>>>>>   > > -- Keir
>>>>>>>>>>   > >
>>>>>>>>>>   > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>Attached is a patch that fixes some issues with the
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SYNC policy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>>>>   > >>I have not tried to make the change the minimal one, but,
>>>>>>>>>>   > rather, just
>>>>>>>>>>   > >>ported into
>>>>>>>>>>   > >>the new code what I know to work well. The error for
>>>>>>>>>>   > >>no_missed_ticks_pending goes from
>>>>>>>>>>   > >>over 3% to .03% with this change according to my testing.
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>Regards,
>>>>>>>>>>   > >>Dave
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>>Hi Dave --
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>Did you get your correction ported?  If so, it would be
>>>>>>>>>>   > nice to see this get
>>>>>>>>>>   > >>>into 3.1.3.
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>Note that I just did some very limited testing with
>>>>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>>>>   > >>>missed ticks pending)
>>>>>>>>>>   > >>>on tip of xen-3.1-testing (64-bit Linux hv             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> guest) and the
>>>>>  
>>>>>
>>>>>>>>>>   > worst error I've
>>>>>>>>>>   > >>>seen so far
>>>>>>>>>>   > >>>is 0.012%.  But I haven't tried any exotic             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> loads, just LTP.
>>>>>  
>>>>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>Thanks,
>>>>>>>>>>   > >>>Dan
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>>-----Original Message-----
>>>>>>>>>>   > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>   > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>   > >>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>   > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>   > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>   > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>   > >>>>disables pending
>>>>>>>>>>   > >>>>missed ticks
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>Dan,
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>I did some testing with the constant tsc offset
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SYNC method
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>(now called
>>>>>>>>>>   > >>>>no_missed_ticks_pending)
>>>>>>>>>>   > >>>>and found the error to be very high, much larger
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> than 1 %, as
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>I recall.
>>>>>>>>>>   > >>>>I have not had a chance to submit a correction. I
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> will try to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>do it later
>>>>>>>>>>   > >>>>this week or the first week in January. My version of
>>>>>>>>>>   constant tsc
>>>>>>>>>>   > >>>>offset SYNC method
>>>>>>>>>>   > >>>>produces .02 % error, so I just need to port             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> that into the
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>current code.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>The error you got for both of those kernels is
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> what I would
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   expect
>>>>>>>>>>   > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>Regards,
>>>>>>>>>>   > >>>>Dave
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>Dan Magenheimer wrote:
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>Anyone make measurements on the final patch?
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> saw a loss of
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>about 0.2% with no load.  This was             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> xen-unstable tip today
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>with no options specified.  32-bit was about 0.01%.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>I think I missed something... how do I run the various
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>accounting choices and which ones are known to be
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> appropriate
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>for which kernels?
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>Thanks,
>>>>>>>>>>   > >>>>>Dan
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>>-----Original Message-----
>>>>>>>>>>   > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>   >
>>>>>>>>>>
>>>>>>>>>>           
>>>>>>>>>>
>>>>>>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                     
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>Keir Fraser
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>   > >>>>>>To: Dave Winchell
>>>>>>>>>>   > >>>>>>Cc: Shan, Haitao;             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> xen-devel@lists.xensource.com; Dong,
>>>>>  
>>>>>
>>>>>>>>>>   > Eddie; Jiang,
>>>>>>>>>>   > >>>>>>Yunhong
>>>>>>>>>>   > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer 
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> mode that
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>disables pending
>>>>>>>>>>   > >>>>>>missed ticks
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>-- Keir
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>Keir,
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The accuracy data I've collected for i/o             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> loads for the
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>various time protocols follows. In             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> addition, the data
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>for cpu loads is shown.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> processor AMD
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   box.
>>>>>>>>>>   > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>>>   > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>   > >>>>>>>(usex is available at
>>>>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>>>>   > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> of=/dev/null.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>>>   > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>   > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>   > >>>>>>>All three guests are 8vcpu.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>>>>   > >>>>>>>except that the redhat-64 guest has 4             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> instances of dd.
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, 
>>>>>>>>>> +4.42             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec -.006%,
>>>>>  
>>>>>
>>>>>>>>>>   > +.005% cpu
>>>>>>>>>>   > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec, -.001%,
>>>>>  
>>>>>
>>>>>>>>>>   > +.012% cpu
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>>>>   -.004% cpu
>>>>>>>>>>   > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.005%, -.005% cpu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.008%, -.003% cpu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.040% cpu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec, -.034%,
>>>>>  
>>>>>
>>>>>>>>>>   > -.031% cpu
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>>>>   > -.09% i/o-8
>>>>>>>>>>   > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>>>>   > -.14% i/o-8
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>>>>   > -.022% i/o-8
>>>>>>>>>>   > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.017%, -.018% i/o-8
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>>>>   > -.029% i/o-8
>>>>>>>>>>   > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec, -.023%,
>>>>>  
>>>>>
>>>>>>>>>>   > -.031% i/o-8
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.04% i/o-32
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>>>>   > -.005% i/o-32
>>>>>>>>>>   > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.11% i/o-32
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>>>>   > .003% i/o-4/32
>>>>>>>>>>   > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>>>>   > .01% i/o-4/32
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Overhead measurements:
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Progress in terms of number of passes             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> through a fixed
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>system workload
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>   > >>>>>>>The workload was usex -b48.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>   > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>   > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>   > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Conclusions:
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>   > requirement for ntp
>>>>>>>>>>   > >>>>>>>tracking under the loads
>>>>>>>>>>   > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> accuracies for
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>and ASYNC
>>>>>>>>>>   > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>We could reduce the cost of the SYNC method by only
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>scheduling the extra
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>wakeups if a certain number
>>>>>>>>>>   > >>>>>>>of ticks are missed.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Regards,
>>>>>>>>>>   > >>>>>>>Dave
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Keir Fraser wrote:
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ASYNC method a
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>couple of days ago,
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>I ran another ASYNC test. I think there may have
>>>>>>>>>>   > been something
>>>>>>>>>>   > >>>>>>>>>wrong with the code I used a couple of             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> days ago for
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>ASYNC. It may have been
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> after context
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>switch in.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>acceptable accuracy,
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>each running consistently around or under
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> .01%. MIXED has
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>a fairly high
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>error of
>>>>>>>>>>   > >>>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>threshold for comfort.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> plan to leave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>SYNC running
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>over the weekend. If you'd rather I can             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> leave MIXED
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>running instead.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I can run
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>more overnight tests
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>>next week.
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>>   > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> effects of the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>SYNC+run_timer
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cause higher
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>system-wide CPU
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>contention. I find it easier to think through the
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>implications of ASYNC. I'm
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> accurate than
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>ASYNC. Perhaps it
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> approaches,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>and each interrupt
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>event causes a small accumulated error?
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> favourites and
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>if the latter is
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>actually more accurate then I can simply revert the
>>>>>>>>>>   > changeset that
>>>>>>>>>>   > >>>>>>>>implemented MIXED.
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> workloads you
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>could try idle
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> large disc reads
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>to /dev/null)? We
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> CPU bound, so
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>that's really an
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>-- Keir
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>_______________________________________________
>>>>>>>>>>   > >>>>>>Xen-devel mailing list
>>>>>>>>>>   > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>   > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> 2007 +0000
>>>>>  
>>>>>
>>>>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> 2008 -0500
>>>>>  
>>>>>
>>>>>>>>>>   > >>@@ -58,7 +58,7 @@ static void             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> pt_process_missed_ticks(stru
>>>>>  
>>>>>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> pt->period + 1;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> no_missed_ticks_pending) )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>   > >>+        pt->do_not_freeze = 1;
>>>>>>>>>>   > >>     else
>>>>>>>>>>   > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>   > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>   > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>     pt_lock(pt);
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>-    pt->pending_intr_nr++;
>>>>>>>>>>   > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>+        pt->pending_intr_nr = 1;
>>>>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>>>>   > >>+    }
>>>>>>>>>>   > >>+    else
>>>>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>>>>   > >>     {
>>>>>>>>>>   > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> vcpu *v, struct
>>>>>  
>>>>>
>>>>>>>>>>   > >>         return;
>>>>>>>>>>   > >>     }
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>-    pt->do_not_freeze = 0;
>>>>>>>>>>   > >>-
>>>>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>>>>   > >>     {
>>>>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>>>>   > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *v, struct
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>   > >>             pt->pending_intr_nr = 0; /* 'collapse' all
>>>>>>>>>>   > missed ticks */
>>>>>>>>>>   > >>         }
>>>>>>>>>>   > >>+ else if ( mode_is(v->domain,             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> no_missed_ticks_pending) ) {
>>>>>  
>>>>>
>>>>>>>>>>   > >>+     pt->pending_intr_nr--;
>>>>>>>>>>   > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>   > >>+ }
>>>>>>>>>>   > >>         else
>>>>>>>>>>   > >>         {
>>>>>>>>>>   > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   > _______________________________________________
>>>>>>>>>>   > Xen-devel mailing list
>>>>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>>>>   >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>           
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>
>>
>
>
> ------------------------------------------------------------------------
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Hi Dave --

Thanks for continuing to run tests!

Hmmm... I thought I had noticed that even though Linux will acknowledge
the existence of the pmtimer, it still prints:

time.c: Using PIT/TSC based timekeeping.

I will check again, but assuming the clocksource for our tests is
indeed pit, the huge difference in the results (yours vs ours) is
baffling. I wonder if the difference may be the underlying hardware.
Maybe we will try to ensure we can duplicate the results on a different
box.

So your testing was with stock 3.2.0 xen bits (what cset?) without
any of your [quote from below] "clock related tweaks that I haven't
submitted, because I'm still characterizing them"?

Could you also send detail on the rhel4u4-64 kernel you
are testing with, just to ensure we are not comparing apples
and oranges?  (Perhaps there's some way we can even share the
identical disk image and vm.cfg file?)

And if our problem is indeed the pmtimer, I will need to submit
another patch to Keir to add an hvm pmtimer platform variable.
(Hmmm... I don't think he's even accepted the hpet variable patch
yet.  I'll have to check.)

Thanks,
Dan


> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Thursday, February 14, 2008 9:00 AM
> To: dan.magenheimer@oracle.com
> Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
> Patel
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Dan,
> 
> I ran the ltp tests with 3.2 and found the errors
> for a 16 hour run to be:
> 
> rh4u564 -9.9 sec (-.017%)
> rh4u464 -7.3 sec (-.013%)
> 
> There were no cliffs and the drift was linear.
> 
> I think the problem you had may be due to the use of the
> pm timer. If you still have the boot log, it would tell you.
> 
> When I first tried a guest on 3.2 with "clocksource=pit nohpet"
> I noticed that it picked the pm timer. Adding "nopmtimer", the
> guest will pick the pit.
> 
> The reason I didn't have the problem with our 3.1 base is that
> I had disabled the hpet and the pmtimer by not advertising them
> in the acpi tables. I did this so long ago, I forgot that I had to
> disable pmtimer as well as hpet.
> 
> So, can you re-run your test with "clocksource=pit nohpet nopmtimer"?
> You should see this in the boot messages:
> 
> time.c: Using PIT/TSC based timekeeping.
> 
> Thanks,
> Dave
> 
> 
> Dave Winchell wrote:
> 
> > Hi Dan,
> >
> > Over the weekend the drift was +18 seconds for each guest (no ntp).
> > The duration was 3900 minutes, so the error for each was +.0077%.
> > Looking back through the data, it appears to drift linearly at
> > this rate. I've attached a plot for rh4u5-64.
> >
> > This accuracy is better than what I've seen before (.03-.05%).
> > This may be due to the different load (ltp vs usex) or to one of the
> > changes I've made recently. I'll do some experimentation to see if
> > there is
> > a fix I should propose.
> >
> > This still doesn't address the radical drift you saw.
> > The next step for me is to run 3.2 and see if I can reproduce it.
> >
> > Regards,
> > Dave
> >
> >
> >
> >
> >
> > Dave Winchell wrote:
> >
> >> Hi Dan,
> >>
> >> Sorry it took me so long, but I finally ran an ltp test today.
> >> Its on rh4u4-64. I'm using the defaults for ltp and using a script
> >> called runltp. I had a usex load on rh4u5-64. No ntpd.
> >> virtual processors / physical processors = 2.
> >>
> >> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
> >> for -.005% and .008%.
> >>
> >> I'm running a 3.1 based hypervisor with some clock related 
> tweaks that
> >> I haven't submitted, because I'm still characterizing them.
> >>
> >> I'm stopping the usex load on 4u5-64 now and replacing it with ltp
> >> and will leave the two guests running ltp over the weekend.
> >>
> >> Regards,
> >> Dave
> >>
> >>
> >> Dave Winchell wrote:
> >>
> >>> Hi Dan, Deepak:
> >>>
> >>> Thanks for the data. Those drifts are severe - no wonder 
> ntp couldn't
> >>> keep then in synch. I'll try to reproduce that behaviour 
> here, with
> >>> my code base.
> >>> If I can't reproduce it, I'll try 3.2.
> >>>
> >>> If you can isolate what ltp is doing during the cliffs, that would
> >>> be very
> >>> helpful.
> >>>
> >>> thanks,
> >>> Dave
> >>>
> >>>
> >>>
> >>>
> >>> Dan Magenheimer wrote:
> >>>
> >>>> OK, Deepak repeated the test without ntpd and using 
> ntpdate -b before
> >>>> the test.
> >>>>
> >>>> The attached graph shows his results: el5u1-64 (best=~0.07%),
> >>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
> >>>>
> >>>> We will continue to look at LTP to try to isolate.
> >>>>
> >>>> Thanks,
> >>>> Dan
> >>>>
> >>>> P.S. elXuY is essentially RHEL XuY with some patches.
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>> Sent: Wednesday, January 30, 2008 2:45 PM
> >>>>> To: Deepak Patel
> >>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
> >>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; 
> Dave Winchell
> >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
> >>>>> pending
> >>>>> missed ticks
> >>>>>
> >>>>>
> >>>>> Dan, Deeepak,
> >>>>>
> >>>>> It may be that the underlying clock error is too great for ntp
> >>>>> to handle. It would be useful if you did not run ntpd
> >>>>> and, instead did ntpdate -b <timeserver> at the start 
> of the test
> >>>>> for each guest. Then capture the data as you have been doing.
> >>>>> If the drift is greater than .05%, then we need to address that.
> >>>>>
> >>>>> Another option is, when running ntpd, to enable loop 
> statistics in
> >>>>> /etc/ntp.conf
> >>>>> by adding this to the file:
> >>>>>
> >>>>> statistics loopstats
> >>>>> statsdir /var/lib/ntp/
> >>>>>
> >>>>> Then you will see loop data in that directory.
> >>>>> Correlating the data in the loopstats files with the
> >>>>> peaks in skew would be interesting. You will see 
> entries of the form
> >>>>>
> >>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 
> 239.735511 10
> >>>>>
> >>>>> Where the second to last column is the Allan Deviation. 
> When that
> >>>>> gets over 1000, ntpd is working pretty hard. However, I have not
> >>>>> seen ntpd
> >>>>> completely loose it like you have.
> >>>>>
> >>>>> I'm on vacation until Monday, and won't be reading
> >>>>> email.
> >>>>>
> >>>>> Thanks for all your work on this!
> >>>>>
> >>>>> -Dave
> >>>>>
> >>>>> Deepak Patel wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.)
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I do not know which graph was attached with this. But 
> I saw this
> >>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm 
> guests when I
> >>>>>> was running ltp tests continuously.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> What was the behaviour of the other guests running?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> All pvm guests are fine. But behavior of most of the 
> hvm guests were
> >>>>>> as described.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> If they had spikes, were they at the same wall time?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> No. They are not at the same wall time.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Were the other guests running ltp as well?
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
> >>>>>> continuously.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> How are you measuring skew?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I was collecting output of "ntpdate -q  <timeserver> every
> >>>>>
> >>>>>
> >>>>>
> >>>>> 300 seconds
> >>>>>
> >>>>>
> >>>>>> (5 minutes) and have created graph based on that.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Are you running ntpd?
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Yes. ntp was running on all the guests.
> >>>>>>
> >>>>>> I am investigating what causes this spikes and let everyone
> >>>>>
> >>>>>
> >>>>>
> >>>>> know what
> >>>>>
> >>>>>
> >>>>>> are my findings.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Deepak
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Anything that you can discover that would be in sync with
> >>>>>>> the spikes would be very helpful!
> >>>>>>>
> >>>>>>> The code that I test with is our product code, which is based
> >>>>>>> on 3.1. So it is possible that something in 3.2 other 
> than vpt.c
> >>>>>>> is the cause. I can test with 3.2, if necessary.
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> Dave
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Dan Magenheimer wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hi Dave (Keir, see suggestion below) --
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Turning off vhpet certainly helps a lot (though see below).
> >>>>>>>>
> >>>>>>>> I wonder if timekeeping with vhpet is so bad that it 
> should be
> >>>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is
> >>>>>>>> fixed?  (I have a patch that defaults it off, can post it if
> >>>>>>>> there is agreement on the above point.)  The whole 
> point of an
> >>>>>>>> HPET is to provide more precise timekeeping and if vhpet is
> >>>>>>>> worse than vpit, it can only confuse users.  Comments?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In your testing, are you just measuring % skew over a long
> >>>>>>>> period of time?
> >>>>>>>> We are graphing the skew continuously and
> >>>>>>>> seeing periodic behavior that is unsettling, even with pit.
> >>>>>>>> See attached.  Though your algorithm recovers, the "cliffs"
> >>>>>>>> could still cause real user problems.  I wonder if there is
> >>>>>>>> anything that can be done to make the "recovery" more
> >>>>>>>> responsive?
> >>>>>>>>
> >>>>>>>> We are looking into what part(s) of LTP is causing 
> the cliffs.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Dan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
> >>>>>>>>> To: dan.magenheimer@oracle.com
> >>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
> >>>>>>>>> deepak.patel@oracle.com;
> >>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode 
> that disables
> >>>>>>>>> pending
> >>>>>>>>> missed ticks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Dan,
> >>>>>>>>>
> >>>>>>>>> I guess I'm a bit out of date calling for clock= usage.
> >>>>>>>>> Looking at linux 2.6.20.4 sources, I think you 
> should specify
> >>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline.
> >>>>>>>>>
> >>>>>>>>> You can leave the xen and dom0 bootlines as they are.
> >>>>>>>>> The xen and guest clocksources do not need to be the same.
> >>>>>>>>> In my tests, xen is using the hpet for its timekeeping and
> >>>>>>>>> that appears to be the default.
> >>>>>>>>>
> >>>>>>>>> When you boot the guests you should see
> >>>>>>>>>   time.c: Using PIT/TSC based timekeeping.
> >>>>>>>>> on the rh4u5-64 guest, and something similar on the others.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
> >>>>>>>>>> 14.318MHz HPET.)
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> This appears to be the xen state, which is fine.
> >>>>>>>>> I was wrongly assuming that this was the guest state.
> >>>>>>>>> You might want to look in your guest logs and see 
> what they were
> >>>>>>>>> picking
> >>>>>>>>> for a clock source.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Thanks, I hadn't realized that!  No wonder we didn't
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> see the same
> >>>>>
> >>>>>
> >>>>>>>>>> improvement you saw!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Try specifying clock=pit on the linux boot line...
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> command line or
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> dom0?) command
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> line?  Or both places?  Since the tests take awhile, it
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> would be nice
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> to get this right the first time.  Do the Xen and guest
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> clocksources need
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> to be the same?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Dan
> >>>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
> >>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; 
> deepak.patel@oracle.com;
> >>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> that disables
> >>>>>
> >>>>>
> >>>>>>>>>> pending missed ticks
> >>>>>>>>>>
> >>>>>>>>>>   Hi Dan,
> >>>>>>>>>>
> >>>>>>>>>>   Hpet timer does have a fairly large error, as I
> >>>>>>>>>> was
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> trying this
> >>>>>
> >>>>>
> >>>>>>>>>>   one recently.
> >>>>>>>>>>   I don't remember what I got for error, but 1% sounds
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> about right.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   The problem is that hpet is not built on top of vpt.c,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> the module
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   Keir and I did
> >>>>>>>>>>   all the recent work in, for its periodic timer needs. Try
> >>>>>>>>>>   specifying clock=pit
> >>>>>>>>>>   on the linux boot line. If it still picks the 
> hpet, which it
> >>>>>>>>>>   might, let me know
> >>>>>>>>>>   and I'll tell you how to get around this.
> >>>>>>>>>>
> >>>>>>>>>>   Regards,
> >>>>>>>>>>   Dave
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 
> --------------------------------------------------------------
> >>>>>>>>> ----------
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   *From:* Dan Magenheimer 
> [mailto:dan.magenheimer@oracle.com]
> >>>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
> >>>>>>>>>>   *To:* Dave Winchell; Keir Fraser
> >>>>>>>>>>   *Cc:* xen-devel@lists.xensource.com; 
> deepak.patel@oracle.com;
> >>>>>>>>>>   akira.ijuin@oracle.com
> >>>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> that disables
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   pending missed ticks
> >>>>>>>>>>
> >>>>>>>>>>   Sorry for the very late followup on this but we finally
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> were able
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   to get our testing set up again on stable 3.1 
> bits and have
> >>>>>>>>>>   seen some very bad results on 3.1.3-rc1, on the 
> order of 1%.
> >>>>>>>>>>
> >>>>>>>>>>   Test enviroment was a 4-socket dual core machine 
> with 24GB of
> >>>>>>>>>>   memory running six two-vcpu 2GB domains, four hvm
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> plus two pv.
> >>>>>
> >>>>>
> >>>>>>>>>>   All six guests were running LTP simultaneously.  
> The four hvm
> >>>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> RHEL4u5-64.
> >>>>>
> >>>>>
> >>>>>>>>>>   Timer_mode was set to 2 for 64-bit guests and 0 for
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 32-bit guests.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   All four hvm guests experienced skew around -1%,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> even the 32-bit
> >>>>>
> >>>>>
> >>>>>>>>>>   guest.  Less intensive testing didn't exhibit much
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> skew at all.
> >>>>>
> >>>>>
> >>>>>>>>>>   A representative graph is attached.
> >>>>>>>>>>
> >>>>>>>>>>   Dave, I wonder if some portion of your patches
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> didn't end up in
> >>>>>
> >>>>>
> >>>>>>>>>>   the xen trees?
> >>>>>>>>>>
> >>>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04, 
> Platform timer
> >>>>>>>>>>   14.318MHz HPET.)
> >>>>>>>>>>
> >>>>>>>>>>   Thanks,
> >>>>>>>>>>   Dan
> >>>>>>>>>>
> >>>>>>>>>>   P.S. Many thanks to Deepak and Akira for running tests.
> >>>>>>>>>>
> >>>>>>>>>>   > -----Original Message-----
> >>>>>>>>>>   > From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>   > 
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>>>>   > Dave Winchell
> >>>>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53 AM
> >>>>>>>>>>   > To: Keir Fraser
> >>>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> xen-devel@lists.xensource.com; Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > Winchell
> >>>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>>>>>>   > disables pending
> >>>>>>>>>>   > missed ticks
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   > Hi Keir,
> >>>>>>>>>>   >
> >>>>>>>>>>   > The latest change, c/s 16690, looks fine.
> >>>>>>>>>>   > I agree that the code in c/s 16690 is equivalent to
> >>>>>>>>>>   > the code I submitted. Also, your version is more
> >>>>>>>>>>   > concise.
> >>>>>>>>>>   >
> >>>>>>>>>>   > The error tests confirm the equivalence. With
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> overnight cpu loads,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > the checked in version was accurate to +.048% for sles
> >>>>>>>>>>   > and +.038% for red hat. My version was +.046%
> >>>>>>>>>> and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> +.032% in a
> >>>>>
> >>>>>
> >>>>>>>>>>   > 2 hour test.
> >>>>>>>>>>   > I don't think the difference is significant.
> >>>>>>>>>>   >
> >>>>>>>>>>   > i/o loads produced errors of +.01%.
> >>>>>>>>>>   >
> >>>>>>>>>>   > Thanks for all your efforts on this issue.
> >>>>>>>>>>   >
> >>>>>>>>>>   > Regards,
> >>>>>>>>>>   > Dave
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   > Keir Fraser wrote:
> >>>>>>>>>>   >
> >>>>>>>>>>   > >Applied as c/s 16690, although the checked-in patch is
> >>>>>>>>>>   > smaller. I think the
> >>>>>>>>>>   > >only important fix is to pt_intr_post() and the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> only bit of
> >>>>>
> >>>>>
> >>>>>>>>>>   > the patch I
> >>>>>>>>>>   > >totally omitted was the change to
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> pt_process_missed_ticks().
> >>>>>
> >>>>>
> >>>>>>>>>>   > I don't think
> >>>>>>>>>>   > >that change can be important, but let's see what
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> happens to the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   error
> >>>>>>>>>>   > >percentage...
> >>>>>>>>>>   > >
> >>>>>>>>>>   > > -- Keir
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >On 4/1/08 23:24, "Dave Winchell"
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> <dwinchell@virtualiron.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >>Hi Dan and Keir,
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>Attached is a patch that fixes some issues with the
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> SYNC policy
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>(no_missed_ticks_pending).
> >>>>>>>>>>   > >>I have not tried to make the change the 
> minimal one, but,
> >>>>>>>>>>   > rather, just
> >>>>>>>>>>   > >>ported into
> >>>>>>>>>>   > >>the new code what I know to work well. The error for
> >>>>>>>>>>   > >>no_missed_ticks_pending goes from
> >>>>>>>>>>   > >>over 3% to .03% with this change according 
> to my testing.
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>Regards,
> >>>>>>>>>>   > >>Dave
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>Dan Magenheimer wrote:
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>>Hi Dave --
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>Did you get your correction ported?  If so, 
> it would be
> >>>>>>>>>>   > nice to see this get
> >>>>>>>>>>   > >>>into 3.1.3.
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>Note that I just did some very limited testing with
> >>>>>>>>>>   > timer_mode=2(=SYNC=no
> >>>>>>>>>>   > >>>missed ticks pending)
> >>>>>>>>>>   > >>>on tip of xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> guest) and the
> >>>>>
> >>>>>
> >>>>>>>>>>   > worst error I've
> >>>>>>>>>>   > >>>seen so far
> >>>>>>>>>>   > >>>is 0.012%.  But I haven't tried any exotic
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> loads, just LTP.
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>Thanks,
> >>>>>>>>>>   > >>>Dan
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>>-----Original Message-----
> >>>>>>>>>>   > >>>>From: Dave Winchell 
> [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>   > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>   > >>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>   > >>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>>>>>>   > xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>   > >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>   > >>>>Subject: Re: [Xen-devel] [PATCH] Add a 
> timer mode that
> >>>>>>>>>>   > >>>>disables pending
> >>>>>>>>>>   > >>>>missed ticks
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>Dan,
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>I did some testing with the constant tsc offset
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> SYNC method
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>(now called
> >>>>>>>>>>   > >>>>no_missed_ticks_pending)
> >>>>>>>>>>   > >>>>and found the error to be very high, much larger
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> than 1 %, as
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>I recall.
> >>>>>>>>>>   > >>>>I have not had a chance to submit a correction. I
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> will try to
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>do it later
> >>>>>>>>>>   > >>>>this week or the first week in January. My 
> version of
> >>>>>>>>>>   constant tsc
> >>>>>>>>>>   > >>>>offset SYNC method
> >>>>>>>>>>   > >>>>produces .02 % error, so I just need to port
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> that into the
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>current code.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>The error you got for both of those kernels is
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> what I would
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   expect
> >>>>>>>>>>   > >>>>for the default mode, delay_for_missed_ticks.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>I'll let Keir answer on how to set the time mode.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>Regards,
> >>>>>>>>>>   > >>>>Dave
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>Dan Magenheimer wrote:
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>Anyone make measurements on the final patch?
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> saw a loss of
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>about 0.2% with no load.  This was
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> xen-unstable tip today
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>with no options specified.  32-bit was about 0.01%.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>I think I missed something... how do I 
> run the various
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>accounting choices and which ones are known to be
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> appropriate
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>for which kernels?
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>Thanks,
> >>>>>>>>>>   > >>>>>Dan
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>>-----Original Message-----
> >>>>>>>>>>   > >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>   >
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> 
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>Keir Fraser
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>   > >>>>>>To: Dave Winchell
> >>>>>>>>>>   > >>>>>>Cc: Shan, Haitao;
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> xen-devel@lists.xensource.com; Dong,
> >>>>>
> >>>>>
> >>>>>>>>>>   > Eddie; Jiang,
> >>>>>>>>>>   > >>>>>>Yunhong
> >>>>>>>>>>   > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> mode that
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>disables pending
> >>>>>>>>>>   > >>>>>>missed ticks
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>Please take a look at xen-unstable 
> changeset 16545.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>-- Keir
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>Keir,
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>The accuracy data I've collected for i/o
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> loads for the
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>>various time protocols follows. In
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> addition, the data
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>>for cpu loads is shown.
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> processor AMD
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   box.
> >>>>>>>>>>   > >>>>>>>Two guests, red hat and sles 64 bit, 8 
> vcpu each.
> >>>>>>>>>>   > >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>   > >>>>>>>(usex is available at
> >>>>>>>>>>   http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>   > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> of=/dev/null.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >>>>>>>>>>   > >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>   > >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>   > >>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>The loads labeled i/o-4/32 are the same 
> as i/o-32
> >>>>>>>>>>   > >>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> instances of dd.
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>> +4.42
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec -.006%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > +.005% cpu
> >>>>>>>>>>   > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec, -.001%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > +.012% cpu
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 
> sec, -.009%,
> >>>>>>>>>>   -.004% cpu
> >>>>>>>>>>   > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.005%, -.005% cpu
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.008%, -.003% cpu
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.040% cpu
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec, -.034%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > -.031% cpu
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 
> sec,-55.7 sec, -.01%,
> >>>>>>>>>>   > -.09% i/o-8
> >>>>>>>>>>   > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 
> sec,-14.0 sec, -.015%
> >>>>>>>>>>   > -.14% i/o-8
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 
> sec, -.017%,
> >>>>>>>>>>   > -.022% i/o-8
> >>>>>>>>>>   > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.017%, -.018% i/o-8
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 
> sec, -.020%,
> >>>>>>>>>>   > -.029% i/o-8
> >>>>>>>>>>   > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec, -.023%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > -.031% i/o-8
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.04% i/o-32
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 
> sec, -.011%,
> >>>>>>>>>>   > -.005% i/o-32
> >>>>>>>>>>   > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.11% i/o-32
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 
> 13. sec -.07%,
> >>>>>>>>>>   > .003% i/o-4/32
> >>>>>>>>>>   > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 
> sec, -.017%,
> >>>>>>>>>>   > .01% i/o-4/32
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>Overhead measurements:
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>Progress in terms of number of passes
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> through a fixed
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>system workload
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>>>>   > >>>>>>>The workload was usex -b48.
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>   > >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>   > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>   > >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>Conclusions:
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>The only protocol which meets the .05% accuracy
> >>>>>>>>>>   > requirement for ntp
> >>>>>>>>>>   > >>>>>>>tracking under the loads
> >>>>>>>>>>   > >>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> accuracies for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>SYNC, MIXED,
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>and ASYNC
> >>>>>>>>>>   > >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>We could reduce the cost of the SYNC 
> method by only
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>scheduling the extra
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>wakeups if a certain number
> >>>>>>>>>>   > >>>>>>>of ticks are missed.
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>Regards,
> >>>>>>>>>>   > >>>>>>>Dave
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>Keir Fraser wrote:
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>
> >>>>>>>>>>   > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>couple of days ago,
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>I ran another ASYNC test. I think 
> there may have
> >>>>>>>>>>   > been something
> >>>>>>>>>>   > >>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> days ago for
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>ASYNC. It may have been
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>switch in.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>My results indicate that either SYNC 
> or ASYNC give
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>acceptable accuracy,
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>each running consistently around or under
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>a fairly high
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>error of
> >>>>>>>>>>   > >>>>>>>>>greater than .03%. Probably too close 
> to .05% ntp
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>threshold for comfort.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>I don't have an overnight run with SYNC. I
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>SYNC running
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>over the weekend. If you'd rather I can
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> leave MIXED
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>running instead.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>It may be too early to pick the protocol and
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>more overnight tests
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>>next week.
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>I'm a bit worried about any unwanted side
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> effects of the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>SYNC+run_timer
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> cause higher
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>system-wide CPU
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>contention. I find it easier to think 
> through the
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>implications of ASYNC. I'm
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> accurate than
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>ASYNC. Perhaps it
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>delivers more timer interrupts than the other
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> approaches,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>and each interrupt
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> favourites and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>if the latter is
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>actually more accurate then I can 
> simply revert the
> >>>>>>>>>>   > changeset that
> >>>>>>>>>>   > >>>>>>>>implemented MIXED.
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> workloads you
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>could try idle
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> large disc reads
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>to /dev/null)? We
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>don't have any data on workloads that aren't
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> CPU bound, so
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>that's really an
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>-- Keir
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>>>
> >>>>>>>>>>   > >>>>>>_______________________________________________
> >>>>>>>>>>   > >>>>>>Xen-devel mailing list
> >>>>>>>>>>   > >>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>   > >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> 2007 +0000
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> 2008 -0500
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>@@ -58,7 +58,7 @@ static void
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> pt_process_missed_ticks(stru
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>     missed_ticks = missed_ticks / (s_time_t)
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> pt->period + 1;
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>     if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> no_missed_ticks_pending) )
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>   > >>+        pt->do_not_freeze = 1;
> >>>>>>>>>>   > >>     else
> >>>>>>>>>>   > >>         pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>   > >>     pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>   > >>@@ -127,7 +127,12 @@ static void 
> pt_timer_fn(void *data)
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>     pt_lock(pt);
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>-    pt->pending_intr_nr++;
> >>>>>>>>>>   > >>+    if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> no_missed_ticks_pending) ) {
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>+        pt->pending_intr_nr = 1;
> >>>>>>>>>>   > >>+ pt->do_not_freeze = 0;
> >>>>>>>>>>   > >>+    }
> >>>>>>>>>>   > >>+    else
> >>>>>>>>>>   > >>+ pt->pending_intr_nr++;
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>     if ( !pt->one_shot )
> >>>>>>>>>>   > >>     {
> >>>>>>>>>>   > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> vcpu *v, struct
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>         return;
> >>>>>>>>>>   > >>     }
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>-    pt->do_not_freeze = 0;
> >>>>>>>>>>   > >>-
> >>>>>>>>>>   > >>     if ( pt->one_shot )
> >>>>>>>>>>   > >>     {
> >>>>>>>>>>   > >>         pt->enabled = 0;
> >>>>>>>>>>   > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> *v, struct
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>             pt->last_plt_gtime = 
> hvm_get_guest_time(v);
> >>>>>>>>>>   > >>             pt->pending_intr_nr = 0; /* 
> 'collapse' all
> >>>>>>>>>>   > missed ticks */
> >>>>>>>>>>   > >>         }
> >>>>>>>>>>   > >>+ else if ( mode_is(v->domain,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> no_missed_ticks_pending) ) {
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>+     pt->pending_intr_nr--;
> >>>>>>>>>>   > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>   > >>+ }
> >>>>>>>>>>   > >>         else
> >>>>>>>>>>   > >>         {
> >>>>>>>>>>   > >>             pt->last_plt_gtime += pt->period_cycles;
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   > _______________________________________________
> >>>>>>>>>>   > Xen-devel mailing list
> >>>>>>>>>>   > Xen-devel@lists.xensource.com
> >>>>>>>>>>   > http://lists.xensource.com/xen-devel
> >>>>>>>>>>   >
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>
> >>
> >
> >
> > 
> --------------------------------------------------------------
> ----------
> >
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Dan,

Here are some boot snipets for rh4u564 on xen 3.2.


#1:

Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro 
root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp 
(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 
(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
...
Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/ 
console=ttyS0 clocksource=pit nohpet
Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11, 
65536 bytes)
Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
...
Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8 
CPUs: passed.
Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use of PM timer
Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.



#2:

Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro 
root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp 
(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 
(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
...
Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/ 
console=ttyS0 clocksource=pit nohpet nopmtimer
Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11, 
65536 bytes)
Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
...
Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8 
CPUs: passed.
Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.


As you can see, I only get the pit if I specify nopmtimer.

Dan Magenheimer wrote:

>Hi Dave --
>
>Thanks for continuing to run tests!
>
>Hmmm... I thought I had noticed that even though Linux will acknowledge
>the existence of the pmtimer, it still prints:
>
>time.c: Using PIT/TSC based timekeeping.
>
>I will check again, but assuming the clocksource for our tests is
>indeed pit, the huge difference in the results (yours vs ours) is
>baffling. I wonder if the difference may be the underlying hardware.
>Maybe we will try to ensure we can duplicate the results on a different
>box.
>  
>
>So your testing was with stock 3.2.0 xen bits (what cset?) without
>any of your [quote from below] "clock related tweaks that I haven't
>submitted, because I'm still characterizing them"?
>  
>
None of the tweaks I mentioned are in this test.
It was stock with some patches. However, none of the patches are time 
related to
my knowledge and I checked vpt.c to make sure that it is the same as 
what's in unstable.
The only difference is in pt_intr_post, where I set the timer mode.
I don't have timer mode tied into our config process yet, which
is different than official xen method.


(In pt_intr_post)
      else
      {
+       if(v->arch.paging.mode->guest_levels == 4)
+           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = 
HVMPTM_no_missed_ticks_pending;
+       else
+           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = 
HVMPTM_delay_for_missed_ticks;
          if ( mode_is(v->domain, one_missed_tick_pending) ||
               mode_is(v->domain, no_missed_ticks_pending) )
          {

>Could you also send detail on the rhel4u4-64 kernel you
>are testing with, just to ensure we are not comparing apples
>and oranges?  (Perhaps there's some way we can even share the
>identical disk image and vm.cfg file?)
>
>And if our problem is indeed the pmtimer, I will need to submit
>another patch to Keir to add an hvm pmtimer platform variable.
>(Hmmm... I don't think he's even accepted the hpet variable patch
>yet.  I'll have to check.)
>
>Thanks,
>Dan
>
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Thursday, February 14, 2008 9:00 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
>>Patel
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Hi Dan,
>>
>>I ran the ltp tests with 3.2 and found the errors
>>for a 16 hour run to be:
>>
>>rh4u564 -9.9 sec (-.017%)
>>rh4u464 -7.3 sec (-.013%)
>>
>>There were no cliffs and the drift was linear.
>>
>>I think the problem you had may be due to the use of the
>>pm timer. If you still have the boot log, it would tell you.
>>
>>When I first tried a guest on 3.2 with "clocksource=pit nohpet"
>>I noticed that it picked the pm timer. Adding "nopmtimer", the
>>guest will pick the pit.
>>
>>The reason I didn't have the problem with our 3.1 base is that
>>I had disabled the hpet and the pmtimer by not advertising them
>>in the acpi tables. I did this so long ago, I forgot that I had to
>>disable pmtimer as well as hpet.
>>
>>So, can you re-run your test with "clocksource=pit nohpet nopmtimer"?
>>You should see this in the boot messages:
>>
>>time.c: Using PIT/TSC based timekeeping.
>>
>>Thanks,
>>Dave
>>
>>
>>Dave Winchell wrote:
>>
>>    
>>
>>>Hi Dan,
>>>
>>>Over the weekend the drift was +18 seconds for each guest (no ntp).
>>>The duration was 3900 minutes, so the error for each was +.0077%.
>>>Looking back through the data, it appears to drift linearly at
>>>this rate. I've attached a plot for rh4u5-64.
>>>
>>>This accuracy is better than what I've seen before (.03-.05%).
>>>This may be due to the different load (ltp vs usex) or to one of the
>>>changes I've made recently. I'll do some experimentation to see if
>>>there is
>>>a fix I should propose.
>>>
>>>This still doesn't address the radical drift you saw.
>>>The next step for me is to run 3.2 and see if I can reproduce it.
>>>
>>>Regards,
>>>Dave
>>>
>>>
>>>
>>>
>>>
>>>Dave Winchell wrote:
>>>
>>>      
>>>
>>>>Hi Dan,
>>>>
>>>>Sorry it took me so long, but I finally ran an ltp test today.
>>>>Its on rh4u4-64. I'm using the defaults for ltp and using a script
>>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
>>>>virtual processors / physical processors = 2.
>>>>
>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
>>>>for -.005% and .008%.
>>>>
>>>>I'm running a 3.1 based hypervisor with some clock related 
>>>>        
>>>>
>>tweaks that
>>    
>>
>>>>I haven't submitted, because I'm still characterizing them.
>>>>
>>>>I'm stopping the usex load on 4u5-64 now and replacing it with ltp
>>>>and will leave the two guests running ltp over the weekend.
>>>>
>>>>Regards,
>>>>Dave
>>>>
>>>>
>>>>Dave Winchell wrote:
>>>>
>>>>        
>>>>
>>>>>Hi Dan, Deepak:
>>>>>
>>>>>Thanks for the data. Those drifts are severe - no wonder 
>>>>>          
>>>>>
>>ntp couldn't
>>    
>>
>>>>>keep then in synch. I'll try to reproduce that behaviour 
>>>>>          
>>>>>
>>here, with
>>    
>>
>>>>>my code base.
>>>>>If I can't reproduce it, I'll try 3.2.
>>>>>
>>>>>If you can isolate what ltp is doing during the cliffs, that would
>>>>>be very
>>>>>helpful.
>>>>>
>>>>>thanks,
>>>>>Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Dan Magenheimer wrote:
>>>>>
>>>>>          
>>>>>
>>>>>>OK, Deepak repeated the test without ntpd and using 
>>>>>>            
>>>>>>
>>ntpdate -b before
>>    
>>
>>>>>>the test.
>>>>>>
>>>>>>The attached graph shows his results: el5u1-64 (best=~0.07%),
>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>>>
>>>>>>We will continue to look at LTP to try to isolate.
>>>>>>
>>>>>>Thanks,
>>>>>>Dan
>>>>>>
>>>>>>P.S. elXuY is essentially RHEL XuY with some patches.
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
>>>>>>>To: Deepak Patel
>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; 
>>>>>>>              
>>>>>>>
>>Dave Winchell
>>    
>>
>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>>pending
>>>>>>>missed ticks
>>>>>>>
>>>>>>>
>>>>>>>Dan, Deeepak,
>>>>>>>
>>>>>>>It may be that the underlying clock error is too great for ntp
>>>>>>>to handle. It would be useful if you did not run ntpd
>>>>>>>and, instead did ntpdate -b <timeserver> at the start 
>>>>>>>              
>>>>>>>
>>of the test
>>    
>>
>>>>>>>for each guest. Then capture the data as you have been doing.
>>>>>>>If the drift is greater than .05%, then we need to address that.
>>>>>>>
>>>>>>>Another option is, when running ntpd, to enable loop 
>>>>>>>              
>>>>>>>
>>statistics in
>>    
>>
>>>>>>>/etc/ntp.conf
>>>>>>>by adding this to the file:
>>>>>>>
>>>>>>>statistics loopstats
>>>>>>>statsdir /var/lib/ntp/
>>>>>>>
>>>>>>>Then you will see loop data in that directory.
>>>>>>>Correlating the data in the loopstats files with the
>>>>>>>peaks in skew would be interesting. You will see 
>>>>>>>              
>>>>>>>
>>entries of the form
>>    
>>
>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776 
>>>>>>>              
>>>>>>>
>>239.735511 10
>>    
>>
>>>>>>>Where the second to last column is the Allan Deviation. 
>>>>>>>              
>>>>>>>
>>When that
>>    
>>
>>>>>>>gets over 1000, ntpd is working pretty hard. However, I have not
>>>>>>>seen ntpd
>>>>>>>completely loose it like you have.
>>>>>>>
>>>>>>>I'm on vacation until Monday, and won't be reading
>>>>>>>email.
>>>>>>>
>>>>>>>Thanks for all your work on this!
>>>>>>>
>>>>>>>-Dave
>>>>>>>
>>>>>>>Deepak Patel wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>I do not know which graph was attached with this. But 
>>>>>>>>                
>>>>>>>>
>>I saw this
>>    
>>
>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm 
>>>>>>>>                
>>>>>>>>
>>guests when I
>>    
>>
>>>>>>>>was running ltp tests continuously.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>What was the behaviour of the other guests running?
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>All pvm guests are fine. But behavior of most of the 
>>>>>>>>                
>>>>>>>>
>>hvm guests were
>>    
>>
>>>>>>>>as described.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>If they had spikes, were they at the same wall time?
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>No. They are not at the same wall time.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Were the other guests running ltp as well?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>>>>>>continuously.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>How are you measuring skew?
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>I was collecting output of "ntpdate -q  <timeserver> every
>>>>>>>>                
>>>>>>>>
>>>>>>>
>>>>>>>300 seconds
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>(5 minutes) and have created graph based on that.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Are you running ntpd?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>Yes. ntp was running on all the guests.
>>>>>>>>
>>>>>>>>I am investigating what causes this spikes and let everyone
>>>>>>>>                
>>>>>>>>
>>>>>>>
>>>>>>>know what
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>are my findings.
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>Deepak
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Anything that you can discover that would be in sync with
>>>>>>>>>the spikes would be very helpful!
>>>>>>>>>
>>>>>>>>>The code that I test with is our product code, which is based
>>>>>>>>>on 3.1. So it is possible that something in 3.2 other 
>>>>>>>>>                  
>>>>>>>>>
>>than vpt.c
>>    
>>
>>>>>>>>>is the cause. I can test with 3.2, if necessary.
>>>>>>>>>
>>>>>>>>>thanks,
>>>>>>>>>Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>Hi Dave (Keir, see suggestion below) --
>>>>>>>>>>
>>>>>>>>>>Thanks!
>>>>>>>>>>
>>>>>>>>>>Turning off vhpet certainly helps a lot (though see below).
>>>>>>>>>>
>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it 
>>>>>>>>>>                    
>>>>>>>>>>
>>should be
>>    
>>
>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) until it is
>>>>>>>>>>fixed?  (I have a patch that defaults it off, can post it if
>>>>>>>>>>there is agreement on the above point.)  The whole 
>>>>>>>>>>                    
>>>>>>>>>>
>>point of an
>>    
>>
>>>>>>>>>>HPET is to provide more precise timekeeping and if vhpet is
>>>>>>>>>>worse than vpit, it can only confuse users.  Comments?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>In your testing, are you just measuring % skew over a long
>>>>>>>>>>period of time?
>>>>>>>>>>We are graphing the skew continuously and
>>>>>>>>>>seeing periodic behavior that is unsettling, even with pit.
>>>>>>>>>>See attached.  Though your algorithm recovers, the "cliffs"
>>>>>>>>>>could still cause real user problems.  I wonder if there is
>>>>>>>>>>anything that can be done to make the "recovery" more
>>>>>>>>>>responsive?
>>>>>>>>>>
>>>>>>>>>>We are looking into what part(s) of LTP is causing 
>>>>>>>>>>                    
>>>>>>>>>>
>>the cliffs.
>>    
>>
>>>>>>>>>>Thanks,
>>>>>>>>>>Dan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>>>>>deepak.patel@oracle.com;
>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode 
>>>>>>>>>>>                      
>>>>>>>>>>>
>>that disables
>>    
>>
>>>>>>>>>>>pending
>>>>>>>>>>>missed ticks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Dan,
>>>>>>>>>>>
>>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage.
>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you 
>>>>>>>>>>>                      
>>>>>>>>>>>
>>should specify
>>    
>>
>>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>>>>>
>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are.
>>>>>>>>>>>The xen and guest clocksources do not need to be the same.
>>>>>>>>>>>In my tests, xen is using the hpet for its timekeeping and
>>>>>>>>>>>that appears to be the default.
>>>>>>>>>>>
>>>>>>>>>>>When you boot the guests you should see
>>>>>>>>>>>  time.c: Using PIT/TSC based timekeeping.
>>>>>>>>>>>on the rh4u5-64 guest, and something similar on the others.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>>>>14.318MHz HPET.)
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>This appears to be the xen state, which is fine.
>>>>>>>>>>>I was wrongly assuming that this was the guest state.
>>>>>>>>>>>You might want to look in your guest logs and see 
>>>>>>>>>>>                      
>>>>>>>>>>>
>>what they were
>>    
>>
>>>>>>>>>>>picking
>>>>>>>>>>>for a clock source.
>>>>>>>>>>>
>>>>>>>>>>>Regards,
>>>>>>>>>>>Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>Thanks, I hadn't realized that!  No wonder we didn't
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>see the same
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>improvement you saw!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>Try specifying clock=pit on the linux boot line...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>command line or
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>dom0?) command
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>line?  Or both places?  Since the tests take awhile, it
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>would be nice
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>to get this right the first time.  Do the Xen and guest
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>clocksources need
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>to be the same?
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>Dan
>>>>>>>>>>>>
>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>deepak.patel@oracle.com;
>>    
>>
>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>that disables
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>pending missed ticks
>>>>>>>>>>>>
>>>>>>>>>>>>  Hi Dan,
>>>>>>>>>>>>
>>>>>>>>>>>>  Hpet timer does have a fairly large error, as I
>>>>>>>>>>>>was
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>trying this
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  one recently.
>>>>>>>>>>>>  I don't remember what I got for error, but 1% sounds
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>about right.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  The problem is that hpet is not built on top of vpt.c,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>the module
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  Keir and I did
>>>>>>>>>>>>  all the recent work in, for its periodic timer needs. Try
>>>>>>>>>>>>  specifying clock=pit
>>>>>>>>>>>>  on the linux boot line. If it still picks the 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>hpet, which it
>>    
>>
>>>>>>>>>>>>  might, let me know
>>>>>>>>>>>>  and I'll tell you how to get around this.
>>>>>>>>>>>>
>>>>>>>>>>>>  Regards,
>>>>>>>>>>>>  Dave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>--------------------------------------------------------------
>>    
>>
>>>>>>>>>>>----------
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  *From:* Dan Magenheimer 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:dan.magenheimer@oracle.com]
>>    
>>
>>>>>>>>>>>>  *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>>>  *To:* Dave Winchell; Keir Fraser
>>>>>>>>>>>>  *Cc:* xen-devel@lists.xensource.com; 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>deepak.patel@oracle.com;
>>    
>>
>>>>>>>>>>>>  akira.ijuin@oracle.com
>>>>>>>>>>>>  *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>that disables
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  pending missed ticks
>>>>>>>>>>>>
>>>>>>>>>>>>  Sorry for the very late followup on this but we finally
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>were able
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  to get our testing set up again on stable 3.1 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>bits and have
>>    
>>
>>>>>>>>>>>>  seen some very bad results on 3.1.3-rc1, on the 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>order of 1%.
>>    
>>
>>>>>>>>>>>>  Test enviroment was a 4-socket dual core machine 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>with 24GB of
>>    
>>
>>>>>>>>>>>>  memory running six two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>plus two pv.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  All six guests were running LTP simultaneously.  
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>The four hvm
>>    
>>
>>>>>>>>>>>>  guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>RHEL4u5-64.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>32-bit guests.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  All four hvm guests experienced skew around -1%,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>even the 32-bit
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  guest.  Less intensive testing didn't exhibit much
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>skew at all.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  A representative graph is attached.
>>>>>>>>>>>>
>>>>>>>>>>>>  Dave, I wonder if some portion of your patches
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>didn't end up in
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  the xen trees?
>>>>>>>>>>>>
>>>>>>>>>>>>  (xm dmesg shows 8x Xeon 3.2GHz stepping 04, 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>Platform timer
>>    
>>
>>>>>>>>>>>>  14.318MHz HPET.)
>>>>>>>>>>>>
>>>>>>>>>>>>  Thanks,
>>>>>>>>>>>>  Dan
>>>>>>>>>>>>
>>>>>>>>>>>>  P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>>>>>>
>>>>>>>>>>>>  > -----Original Message-----
>>>>>>>>>>>>  > From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>  > 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>    
>>
>>>>>>>>>>>>  > Dave Winchell
>>>>>>>>>>>>  > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>>>  > To: Keir Fraser
>>>>>>>>>>>>  > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>xen-devel@lists.xensource.com; Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > Winchell
>>>>>>>>>>>>  > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>>>  > disables pending
>>>>>>>>>>>>  > missed ticks
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Hi Keir,
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > The latest change, c/s 16690, looks fine.
>>>>>>>>>>>>  > I agree that the code in c/s 16690 is equivalent to
>>>>>>>>>>>>  > the code I submitted. Also, your version is more
>>>>>>>>>>>>  > concise.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > The error tests confirm the equivalence. With
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>overnight cpu loads,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > the checked in version was accurate to +.048% for sles
>>>>>>>>>>>>  > and +.038% for red hat. My version was +.046%
>>>>>>>>>>>>and
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>+.032% in a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > 2 hour test.
>>>>>>>>>>>>  > I don't think the difference is significant.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > i/o loads produced errors of +.01%.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Thanks for all your efforts on this issue.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Regards,
>>>>>>>>>>>>  > Dave
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Keir Fraser wrote:
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > >Applied as c/s 16690, although the checked-in patch is
>>>>>>>>>>>>  > smaller. I think the
>>>>>>>>>>>>  > >only important fix is to pt_intr_post() and the
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>only bit of
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > the patch I
>>>>>>>>>>>>  > >totally omitted was the change to
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>pt_process_missed_ticks().
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > I don't think
>>>>>>>>>>>>  > >that change can be important, but let's see what
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>happens to the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  error
>>>>>>>>>>>>  > >percentage...
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > > -- Keir
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >>Hi Dan and Keir,
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>Attached is a patch that fixes some issues with the
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>SYNC policy
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>(no_missed_ticks_pending).
>>>>>>>>>>>>  > >>I have not tried to make the change the 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>minimal one, but,
>>    
>>
>>>>>>>>>>>>  > rather, just
>>>>>>>>>>>>  > >>ported into
>>>>>>>>>>>>  > >>the new code what I know to work well. The error for
>>>>>>>>>>>>  > >>no_missed_ticks_pending goes from
>>>>>>>>>>>>  > >>over 3% to .03% with this change according 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>to my testing.
>>    
>>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>Regards,
>>>>>>>>>>>>  > >>Dave
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>Dan Magenheimer wrote:
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>>Hi Dave --
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>Did you get your correction ported?  If so, 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>it would be
>>    
>>
>>>>>>>>>>>>  > nice to see this get
>>>>>>>>>>>>  > >>>into 3.1.3.
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>Note that I just did some very limited testing with
>>>>>>>>>>>>  > timer_mode=2(=SYNC=no
>>>>>>>>>>>>  > >>>missed ticks pending)
>>>>>>>>>>>>  > >>>on tip of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>guest) and the
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > worst error I've
>>>>>>>>>>>>  > >>>seen so far
>>>>>>>>>>>>  > >>>is 0.012%.  But I haven't tried any exotic
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>loads, just LTP.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>Thanks,
>>>>>>>>>>>>  > >>>Dan
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>>-----Original Message-----
>>>>>>>>>>>>  > >>>>From: Dave Winchell 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:dwinchell@virtualiron.com]
>>    
>>
>>>>>>>>>>>>  > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>  > >>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>  > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>  > xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>  > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>  > >>>>Subject: Re: [Xen-devel] [PATCH] Add a 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>timer mode that
>>    
>>
>>>>>>>>>>>>  > >>>>disables pending
>>>>>>>>>>>>  > >>>>missed ticks
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>Dan,
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>I did some testing with the constant tsc offset
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>SYNC method
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>(now called
>>>>>>>>>>>>  > >>>>no_missed_ticks_pending)
>>>>>>>>>>>>  > >>>>and found the error to be very high, much larger
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>than 1 %, as
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>I recall.
>>>>>>>>>>>>  > >>>>I have not had a chance to submit a correction. I
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>will try to
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>do it later
>>>>>>>>>>>>  > >>>>this week or the first week in January. My 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>version of
>>    
>>
>>>>>>>>>>>>  constant tsc
>>>>>>>>>>>>  > >>>>offset SYNC method
>>>>>>>>>>>>  > >>>>produces .02 % error, so I just need to port
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>that into the
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>current code.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>The error you got for both of those kernels is
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>what I would
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  expect
>>>>>>>>>>>>  > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>Regards,
>>>>>>>>>>>>  > >>>>Dave
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>Dan Magenheimer wrote:
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>saw a loss of
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>about 0.2% with no load.  This was
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>xen-unstable tip today
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>with no options specified.  32-bit was about 0.01%.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>I think I missed something... how do I 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>run the various
>>    
>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>accounting choices and which ones are known to be
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>appropriate
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>for which kernels?
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>Thanks,
>>>>>>>>>>>>  > >>>>>Dan
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>>-----Original Message-----
>>>>>>>>>>>>  > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>  >
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>    
>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>Keir Fraser
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>  > >>>>>>To: Dave Winchell
>>>>>>>>>>>>  > >>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>xen-devel@lists.xensource.com; Dong,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > Eddie; Jiang,
>>>>>>>>>>>>  > >>>>>>Yunhong
>>>>>>>>>>>>  > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>mode that
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>disables pending
>>>>>>>>>>>>  > >>>>>>missed ticks
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>Please take a look at xen-unstable 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>changeset 16545.
>>    
>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>-- Keir
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>Keir,
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>The accuracy data I've collected for i/o
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>loads for the
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>>various time protocols follows. In
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>addition, the data
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>>for cpu loads is shown.
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>processor AMD
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  box.
>>>>>>>>>>>>  > >>>>>>>Two guests, red hat and sles 64 bit, 8 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>vcpu each.
>>    
>>
>>>>>>>>>>>>  > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>  > >>>>>>>(usex is available at
>>>>>>>>>>>>  http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>  > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>of=/dev/null.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>>>>>  > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>  > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>  > >>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>The loads labeled i/o-4/32 are the same 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>as i/o-32
>>    
>>
>>>>>>>>>>>>  > >>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>instances of dd.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>+4.42
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec -.006%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > +.005% cpu
>>>>>>>>>>>>  > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec, -.001%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > +.012% cpu
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.009%,
>>    
>>
>>>>>>>>>>>>  -.004% cpu
>>>>>>>>>>>>  > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.005%, -.005% cpu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.008%, -.003% cpu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.040% cpu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec, -.034%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > -.031% cpu
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec,-55.7 sec, -.01%,
>>    
>>
>>>>>>>>>>>>  > -.09% i/o-8
>>>>>>>>>>>>  > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec,-14.0 sec, -.015%
>>    
>>
>>>>>>>>>>>>  > -.14% i/o-8
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.017%,
>>    
>>
>>>>>>>>>>>>  > -.022% i/o-8
>>>>>>>>>>>>  > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.017%, -.018% i/o-8
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.020%,
>>    
>>
>>>>>>>>>>>>  > -.029% i/o-8
>>>>>>>>>>>>  > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec, -.023%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > -.031% i/o-8
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.04% i/o-32
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.011%,
>>    
>>
>>>>>>>>>>>>  > -.005% i/o-32
>>>>>>>>>>>>  > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.11% i/o-32
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>13. sec -.07%,
>>    
>>
>>>>>>>>>>>>  > .003% i/o-4/32
>>>>>>>>>>>>  > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.017%,
>>    
>>
>>>>>>>>>>>>  > .01% i/o-4/32
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>Overhead measurements:
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>through a fixed
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>system workload
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>>>  > >>>>>>>The workload was usex -b48.
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>  > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>  > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>  > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>Conclusions:
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>>>  > requirement for ntp
>>>>>>>>>>>>  > >>>>>>>tracking under the loads
>>>>>>>>>>>>  > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>accuracies for
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>SYNC, MIXED,
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>and ASYNC
>>>>>>>>>>>>  > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>We could reduce the cost of the SYNC 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>method by only
>>    
>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>scheduling the extra
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>wakeups if a certain number
>>>>>>>>>>>>  > >>>>>>>of ticks are missed.
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>Regards,
>>>>>>>>>>>>  > >>>>>>>Dave
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>
>>>>>>>>>>>>  > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>ASYNC method a
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>couple of days ago,
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>I ran another ASYNC test. I think 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>there may have
>>    
>>
>>>>>>>>>>>>  > been something
>>>>>>>>>>>>  > >>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>ASYNC. It may have been
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>after context
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>switch in.
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>My results indicate that either SYNC 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>or ASYNC give
>>    
>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>acceptable accuracy,
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>each running consistently around or under
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>.01%. MIXED has
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>a fairly high
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>error of
>>>>>>>>>>>>  > >>>>>>>>>greater than .03%. Probably too close 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>to .05% ntp
>>    
>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>threshold for comfort.
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>plan to leave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>SYNC running
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>over the weekend. If you'd rather I can
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>running instead.
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>I can run
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>more overnight tests
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>>next week.
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>effects of the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>SYNC+run_timer
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>cause higher
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>system-wide CPU
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>contention. I find it easier to think 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>through the
>>    
>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>implications of ASYNC. I'm
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>accurate than
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>approaches,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>and each interrupt
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>favourites and
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>if the latter is
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>actually more accurate then I can 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>simply revert the
>>    
>>
>>>>>>>>>>>>  > changeset that
>>>>>>>>>>>>  > >>>>>>>>implemented MIXED.
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>workloads you
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>could try idle
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>large disc reads
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>to /dev/null)? We
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>CPU bound, so
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>that's really an
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>-- Keir
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>>>
>>>>>>>>>>>>  > >>>>>>_______________________________________________
>>>>>>>>>>>>  > >>>>>>Xen-devel mailing list
>>>>>>>>>>>>  > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>  > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>  > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>2007 +0000
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>2008 -0500
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>@@ -58,7 +58,7 @@ static void
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>pt_process_missed_ticks(stru
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>pt->period + 1;
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>no_missed_ticks_pending) )
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>  > >>+        pt->do_not_freeze = 1;
>>>>>>>>>>>>  > >>     else
>>>>>>>>>>>>  > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>  > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>  > >>@@ -127,7 +127,12 @@ static void 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>pt_timer_fn(void *data)
>>    
>>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>     pt_lock(pt);
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>-    pt->pending_intr_nr++;
>>>>>>>>>>>>  > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>+        pt->pending_intr_nr = 1;
>>>>>>>>>>>>  > >>+ pt->do_not_freeze = 0;
>>>>>>>>>>>>  > >>+    }
>>>>>>>>>>>>  > >>+    else
>>>>>>>>>>>>  > >>+ pt->pending_intr_nr++;
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>     if ( !pt->one_shot )
>>>>>>>>>>>>  > >>     {
>>>>>>>>>>>>  > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>vcpu *v, struct
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>         return;
>>>>>>>>>>>>  > >>     }
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>-    pt->do_not_freeze = 0;
>>>>>>>>>>>>  > >>-
>>>>>>>>>>>>  > >>     if ( pt->one_shot )
>>>>>>>>>>>>  > >>     {
>>>>>>>>>>>>  > >>         pt->enabled = 0;
>>>>>>>>>>>>  > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>*v, struct
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>             pt->last_plt_gtime = 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>hvm_get_guest_time(v);
>>    
>>
>>>>>>>>>>>>  > >>             pt->pending_intr_nr = 0; /* 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>'collapse' all
>>    
>>
>>>>>>>>>>>>  > missed ticks */
>>>>>>>>>>>>  > >>         }
>>>>>>>>>>>>  > >>+ else if ( mode_is(v->domain,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>+     pt->pending_intr_nr--;
>>>>>>>>>>>>  > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>  > >>+ }
>>>>>>>>>>>>  > >>         else
>>>>>>>>>>>>  > >>         {
>>>>>>>>>>>>  > >>             pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > _______________________________________________
>>>>>>>>>>>>  > Xen-devel mailing list
>>>>>>>>>>>>  > Xen-devel@lists.xensource.com
>>>>>>>>>>>>  > http://lists.xensource.com/xen-devel
>>>>>>>>>>>>  >
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>
>>>      
>>>
>>--------------------------------------------------------------
>>----------
>>    
>>
>>    
>>
>
>  
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Hi Dave --

No new results yet but one other question:

The problems we've seen with our testing have been with a heavily
oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
running LTP simultaneously.

Was your LTP testing oversubscribed or just a single guest?

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Thursday, February 14, 2008 10:56 AM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan,
> 
> Here are some boot snipets for rh4u564 on xen 3.2.
> 
> 
> #1:
> 
> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
> root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> ...
> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
> console=ttyS0 clocksource=pit nohpet
> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11,
> 65536 bytes)
> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
> ...
> Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8
> CPUs: passed.
> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use 
> of PM timer
> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
> 
> 
> 
> #2:
> 
> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> ...
> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
> console=ttyS0 clocksource=pit nohpet nopmtimer
> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11,
> 65536 bytes)
> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
> ...
> Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8
> CPUs: passed.
> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.
> 
> 
> As you can see, I only get the pit if I specify nopmtimer.
> 
> Dan Magenheimer wrote:
> 
> >Hi Dave --
> >
> >Thanks for continuing to run tests!
> >
> >Hmmm... I thought I had noticed that even though Linux will 
> acknowledge
> >the existence of the pmtimer, it still prints:
> >
> >time.c: Using PIT/TSC based timekeeping.
> >
> >I will check again, but assuming the clocksource for our tests is
> >indeed pit, the huge difference in the results (yours vs ours) is
> >baffling. I wonder if the difference may be the underlying hardware.
> >Maybe we will try to ensure we can duplicate the results on 
> a different
> >box.
> >
> >
> >So your testing was with stock 3.2.0 xen bits (what cset?) without
> >any of your [quote from below] "clock related tweaks that I haven't
> >submitted, because I'm still characterizing them"?
> >
> >
> None of the tweaks I mentioned are in this test.
> It was stock with some patches. However, none of the patches are time
> related to
> my knowledge and I checked vpt.c to make sure that it is the same as
> what's in unstable.
> The only difference is in pt_intr_post, where I set the timer mode.
> I don't have timer mode tied into our config process yet, which
> is different than official xen method.
> 
> 
> (In pt_intr_post)
>       else
>       {
> +       if(v->arch.paging.mode->guest_levels == 4)
> +           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
> HVMPTM_no_missed_ticks_pending;
> +       else
> +           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
> HVMPTM_delay_for_missed_ticks;
>           if ( mode_is(v->domain, one_missed_tick_pending) ||
>                mode_is(v->domain, no_missed_ticks_pending) )
>           {
> 
> >Could you also send detail on the rhel4u4-64 kernel you
> >are testing with, just to ensure we are not comparing apples
> >and oranges?  (Perhaps there's some way we can even share the
> >identical disk image and vm.cfg file?)
> >
> >And if our problem is indeed the pmtimer, I will need to submit
> >another patch to Keir to add an hvm pmtimer platform variable.
> >(Hmmm... I don't think he's even accepted the hpet variable patch
> >yet.  I'll have to check.)
> >
> >Thanks,
> >Dan
> >
> >
> >
> >
> >>-----Original Message-----
> >>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>Sent: Thursday, February 14, 2008 9:00 AM
> >>To: dan.magenheimer@oracle.com
> >>Cc: Dave Winchell; Keir Fraser; 
> xen-devel@lists.xensource.com; Deepak
> >>Patel
> >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>disables pending
> >>missed ticks
> >>
> >>
> >>Hi Dan,
> >>
> >>I ran the ltp tests with 3.2 and found the errors
> >>for a 16 hour run to be:
> >>
> >>rh4u564 -9.9 sec (-.017%)
> >>rh4u464 -7.3 sec (-.013%)
> >>
> >>There were no cliffs and the drift was linear.
> >>
> >>I think the problem you had may be due to the use of the
> >>pm timer. If you still have the boot log, it would tell you.
> >>
> >>When I first tried a guest on 3.2 with "clocksource=pit nohpet"
> >>I noticed that it picked the pm timer. Adding "nopmtimer", the
> >>guest will pick the pit.
> >>
> >>The reason I didn't have the problem with our 3.1 base is that
> >>I had disabled the hpet and the pmtimer by not advertising them
> >>in the acpi tables. I did this so long ago, I forgot that I had to
> >>disable pmtimer as well as hpet.
> >>
> >>So, can you re-run your test with "clocksource=pit nohpet 
> nopmtimer"?
> >>You should see this in the boot messages:
> >>
> >>time.c: Using PIT/TSC based timekeeping.
> >>
> >>Thanks,
> >>Dave
> >>
> >>
> >>Dave Winchell wrote:
> >>
> >>
> >>
> >>>Hi Dan,
> >>>
> >>>Over the weekend the drift was +18 seconds for each guest (no ntp).
> >>>The duration was 3900 minutes, so the error for each was +.0077%.
> >>>Looking back through the data, it appears to drift linearly at
> >>>this rate. I've attached a plot for rh4u5-64.
> >>>
> >>>This accuracy is better than what I've seen before (.03-.05%).
> >>>This may be due to the different load (ltp vs usex) or to 
> one of the
> >>>changes I've made recently. I'll do some experimentation to see if
> >>>there is
> >>>a fix I should propose.
> >>>
> >>>This still doesn't address the radical drift you saw.
> >>>The next step for me is to run 3.2 and see if I can reproduce it.
> >>>
> >>>Regards,
> >>>Dave
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>Dave Winchell wrote:
> >>>
> >>>
> >>>
> >>>>Hi Dan,
> >>>>
> >>>>Sorry it took me so long, but I finally ran an ltp test today.
> >>>>Its on rh4u4-64. I'm using the defaults for ltp and using a script
> >>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
> >>>>virtual processors / physical processors = 2.
> >>>>
> >>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
> >>>>for -.005% and .008%.
> >>>>
> >>>>I'm running a 3.1 based hypervisor with some clock related
> >>>>
> >>>>
> >>tweaks that
> >>
> >>
> >>>>I haven't submitted, because I'm still characterizing them.
> >>>>
> >>>>I'm stopping the usex load on 4u5-64 now and replacing it with ltp
> >>>>and will leave the two guests running ltp over the weekend.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>
> >>>>Dave Winchell wrote:
> >>>>
> >>>>
> >>>>
> >>>>>Hi Dan, Deepak:
> >>>>>
> >>>>>Thanks for the data. Those drifts are severe - no wonder
> >>>>>
> >>>>>
> >>ntp couldn't
> >>
> >>
> >>>>>keep then in synch. I'll try to reproduce that behaviour
> >>>>>
> >>>>>
> >>here, with
> >>
> >>
> >>>>>my code base.
> >>>>>If I can't reproduce it, I'll try 3.2.
> >>>>>
> >>>>>If you can isolate what ltp is doing during the cliffs, 
> that would
> >>>>>be very
> >>>>>helpful.
> >>>>>
> >>>>>thanks,
> >>>>>Dave
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>Dan Magenheimer wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>OK, Deepak repeated the test without ntpd and using
> >>>>>>
> >>>>>>
> >>ntpdate -b before
> >>
> >>
> >>>>>>the test.
> >>>>>>
> >>>>>>The attached graph shows his results: el5u1-64 (best=~0.07%),
> >>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
> >>>>>>
> >>>>>>We will continue to look at LTP to try to isolate.
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Dan
> >>>>>>
> >>>>>>P.S. elXuY is essentially RHEL XuY with some patches.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>-----Original Message-----
> >>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
> >>>>>>>To: Deepak Patel
> >>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
> >>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
> >>>>>>>
> >>>>>>>
> >>Dave Winchell
> >>
> >>
> >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
> >>>>>>>pending
> >>>>>>>missed ticks
> >>>>>>>
> >>>>>>>
> >>>>>>>Dan, Deeepak,
> >>>>>>>
> >>>>>>>It may be that the underlying clock error is too great for ntp
> >>>>>>>to handle. It would be useful if you did not run ntpd
> >>>>>>>and, instead did ntpdate -b <timeserver> at the start
> >>>>>>>
> >>>>>>>
> >>of the test
> >>
> >>
> >>>>>>>for each guest. Then capture the data as you have been doing.
> >>>>>>>If the drift is greater than .05%, then we need to 
> address that.
> >>>>>>>
> >>>>>>>Another option is, when running ntpd, to enable loop
> >>>>>>>
> >>>>>>>
> >>statistics in
> >>
> >>
> >>>>>>>/etc/ntp.conf
> >>>>>>>by adding this to the file:
> >>>>>>>
> >>>>>>>statistics loopstats
> >>>>>>>statsdir /var/lib/ntp/
> >>>>>>>
> >>>>>>>Then you will see loop data in that directory.
> >>>>>>>Correlating the data in the loopstats files with the
> >>>>>>>peaks in skew would be interesting. You will see
> >>>>>>>
> >>>>>>>
> >>entries of the form
> >>
> >>
> >>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776
> >>>>>>>
> >>>>>>>
> >>239.735511 10
> >>
> >>
> >>>>>>>Where the second to last column is the Allan Deviation.
> >>>>>>>
> >>>>>>>
> >>When that
> >>
> >>
> >>>>>>>gets over 1000, ntpd is working pretty hard. However, 
> I have not
> >>>>>>>seen ntpd
> >>>>>>>completely loose it like you have.
> >>>>>>>
> >>>>>>>I'm on vacation until Monday, and won't be reading
> >>>>>>>email.
> >>>>>>>
> >>>>>>>Thanks for all your work on this!
> >>>>>>>
> >>>>>>>-Dave
> >>>>>>>
> >>>>>>>Deepak Patel wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested this one.)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>I do not know which graph was attached with this. But
> >>>>>>>>
> >>>>>>>>
> >>I saw this
> >>
> >>
> >>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
> >>>>>>>>
> >>>>>>>>
> >>guests when I
> >>
> >>
> >>>>>>>>was running ltp tests continuously.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>What was the behaviour of the other guests running?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>All pvm guests are fine. But behavior of most of the
> >>>>>>>>
> >>>>>>>>
> >>hvm guests were
> >>
> >>
> >>>>>>>>as described.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>If they had spikes, were they at the same wall time?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>No. They are not at the same wall time.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Were the other guests running ltp as well?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
> >>>>>>>>continuously.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>How are you measuring skew?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>I was collecting output of "ntpdate -q  <timeserver> every
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>300 seconds
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>(5 minutes) and have created graph based on that.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Are you running ntpd?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>Yes. ntp was running on all the guests.
> >>>>>>>>
> >>>>>>>>I am investigating what causes this spikes and let everyone
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>know what
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>are my findings.
> >>>>>>>>
> >>>>>>>>Thanks,
> >>>>>>>>Deepak
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Anything that you can discover that would be in sync with
> >>>>>>>>>the spikes would be very helpful!
> >>>>>>>>>
> >>>>>>>>>The code that I test with is our product code, which is based
> >>>>>>>>>on 3.1. So it is possible that something in 3.2 other
> >>>>>>>>>
> >>>>>>>>>
> >>than vpt.c
> >>
> >>
> >>>>>>>>>is the cause. I can test with 3.2, if necessary.
> >>>>>>>>>
> >>>>>>>>>thanks,
> >>>>>>>>>Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>Hi Dave (Keir, see suggestion below) --
> >>>>>>>>>>
> >>>>>>>>>>Thanks!
> >>>>>>>>>>
> >>>>>>>>>>Turning off vhpet certainly helps a lot (though see below).
> >>>>>>>>>>
> >>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it
> >>>>>>>>>>
> >>>>>>>>>>
> >>should be
> >>
> >>
> >>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) 
> until it is
> >>>>>>>>>>fixed?  (I have a patch that defaults it off, can post it if
> >>>>>>>>>>there is agreement on the above point.)  The whole
> >>>>>>>>>>
> >>>>>>>>>>
> >>point of an
> >>
> >>
> >>>>>>>>>>HPET is to provide more precise timekeeping and if vhpet is
> >>>>>>>>>>worse than vpit, it can only confuse users.  Comments?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>In your testing, are you just measuring % skew over a long
> >>>>>>>>>>period of time?
> >>>>>>>>>>We are graphing the skew continuously and
> >>>>>>>>>>seeing periodic behavior that is unsettling, even with pit.
> >>>>>>>>>>See attached.  Though your algorithm recovers, the "cliffs"
> >>>>>>>>>>could still cause real user problems.  I wonder if there is
> >>>>>>>>>>anything that can be done to make the "recovery" more
> >>>>>>>>>>responsive?
> >>>>>>>>>>
> >>>>>>>>>>We are looking into what part(s) of LTP is causing
> >>>>>>>>>>
> >>>>>>>>>>
> >>the cliffs.
> >>
> >>
> >>>>>>>>>>Thanks,
> >>>>>>>>>>Dan
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM
> >>>>>>>>>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
> >>>>>>>>>>>deepak.patel@oracle.com;
> >>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>that disables
> >>
> >>
> >>>>>>>>>>>pending
> >>>>>>>>>>>missed ticks
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>Dan,
> >>>>>>>>>>>
> >>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage.
> >>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>should specify
> >>
> >>
> >>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline.
> >>>>>>>>>>>
> >>>>>>>>>>>You can leave the xen and dom0 bootlines as they are.
> >>>>>>>>>>>The xen and guest clocksources do not need to be the same.
> >>>>>>>>>>>In my tests, xen is using the hpet for its timekeeping and
> >>>>>>>>>>>that appears to be the default.
> >>>>>>>>>>>
> >>>>>>>>>>>When you boot the guests you should see
> >>>>>>>>>>>  time.c: Using PIT/TSC based timekeeping.
> >>>>>>>>>>>on the rh4u5-64 guest, and something similar on the others.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
> >>>>>>>>>>>>14.318MHz HPET.)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>This appears to be the xen state, which is fine.
> >>>>>>>>>>>I was wrongly assuming that this was the guest state.
> >>>>>>>>>>>You might want to look in your guest logs and see
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>what they were
> >>
> >>
> >>>>>>>>>>>picking
> >>>>>>>>>>>for a clock source.
> >>>>>>>>>>>
> >>>>>>>>>>>Regards,
> >>>>>>>>>>>Dave
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>Thanks, I hadn't realized that!  No wonder we didn't
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>see the same
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>improvement you saw!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>Try specifying clock=pit on the linux boot line...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>command line or
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the 
> guest (or
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>dom0?) command
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>line?  Or both places?  Since the tests take awhile, it
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>would be nice
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>to get this right the first time.  Do the Xen and guest
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>clocksources need
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>to be the same?
> >>>>>>>>>>>>
> >>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>Dan
> >>>>>>>>>>>>
> >>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
> >>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>deepak.patel@oracle.com;
> >>
> >>
> >>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>that disables
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>pending missed ticks
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Hi Dan,
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Hpet timer does have a fairly large error, as I
> >>>>>>>>>>>>was
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>trying this
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  one recently.
> >>>>>>>>>>>>  I don't remember what I got for error, but 1% sounds
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>about right.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  The problem is that hpet is not built on top of vpt.c,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>the module
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  Keir and I did
> >>>>>>>>>>>>  all the recent work in, for its periodic timer 
> needs. Try
> >>>>>>>>>>>>  specifying clock=pit
> >>>>>>>>>>>>  on the linux boot line. If it still picks the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>hpet, which it
> >>
> >>
> >>>>>>>>>>>>  might, let me know
> >>>>>>>>>>>>  and I'll tell you how to get around this.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Regards,
> >>>>>>>>>>>>  Dave
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>--------------------------------------------------------------
> >>
> >>
> >>>>>>>>>>>----------
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  *From:* Dan Magenheimer
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:dan.magenheimer@oracle.com]
> >>
> >>
> >>>>>>>>>>>>  *Sent:* Fri 1/25/2008 6:50 PM
> >>>>>>>>>>>>  *To:* Dave Winchell; Keir Fraser
> >>>>>>>>>>>>  *Cc:* xen-devel@lists.xensource.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>deepak.patel@oracle.com;
> >>
> >>
> >>>>>>>>>>>>  akira.ijuin@oracle.com
> >>>>>>>>>>>>  *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>that disables
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  pending missed ticks
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Sorry for the very late followup on this but we finally
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>were able
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  to get our testing set up again on stable 3.1
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>bits and have
> >>
> >>
> >>>>>>>>>>>>  seen some very bad results on 3.1.3-rc1, on the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>order of 1%.
> >>
> >>
> >>>>>>>>>>>>  Test enviroment was a 4-socket dual core machine
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>with 24GB of
> >>
> >>
> >>>>>>>>>>>>  memory running six two-vcpu 2GB domains, four hvm
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>plus two pv.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  All six guests were running LTP simultaneously.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>The four hvm
> >>
> >>
> >>>>>>>>>>>>  guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>RHEL4u5-64.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  Timer_mode was set to 2 for 64-bit guests and 0 for
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>32-bit guests.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  All four hvm guests experienced skew around -1%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>even the 32-bit
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  guest.  Less intensive testing didn't exhibit much
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>skew at all.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  A representative graph is attached.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Dave, I wonder if some portion of your patches
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>didn't end up in
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  the xen trees?
> >>>>>>>>>>>>
> >>>>>>>>>>>>  (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>Platform timer
> >>
> >>
> >>>>>>>>>>>>  14.318MHz HPET.)
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Thanks,
> >>>>>>>>>>>>  Dan
> >>>>>>>>>>>>
> >>>>>>>>>>>>  P.S. Many thanks to Deepak and Akira for running tests.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  > -----Original Message-----
> >>>>>>>>>>>>  > From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>
> >>
> >>>>>>>>>>>>  > Dave Winchell
> >>>>>>>>>>>>  > Sent: Wednesday, January 09, 2008 9:53 AM
> >>>>>>>>>>>>  > To: Keir Fraser
> >>>>>>>>>>>>  > Cc: dan.magenheimer@oracle.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>xen-devel@lists.xensource.com; Dave
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > Winchell
> >>>>>>>>>>>>  > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>>>>>>>>  > disables pending
> >>>>>>>>>>>>  > missed ticks
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Hi Keir,
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > The latest change, c/s 16690, looks fine.
> >>>>>>>>>>>>  > I agree that the code in c/s 16690 is equivalent to
> >>>>>>>>>>>>  > the code I submitted. Also, your version is more
> >>>>>>>>>>>>  > concise.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > The error tests confirm the equivalence. With
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>overnight cpu loads,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > the checked in version was accurate to +.048% for sles
> >>>>>>>>>>>>  > and +.038% for red hat. My version was +.046%
> >>>>>>>>>>>>and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>+.032% in a
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > 2 hour test.
> >>>>>>>>>>>>  > I don't think the difference is significant.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > i/o loads produced errors of +.01%.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Thanks for all your efforts on this issue.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Regards,
> >>>>>>>>>>>>  > Dave
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Keir Fraser wrote:
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > >Applied as c/s 16690, although the 
> checked-in patch is
> >>>>>>>>>>>>  > smaller. I think the
> >>>>>>>>>>>>  > >only important fix is to pt_intr_post() and the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>only bit of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > the patch I
> >>>>>>>>>>>>  > >totally omitted was the change to
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>pt_process_missed_ticks().
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > I don't think
> >>>>>>>>>>>>  > >that change can be important, but let's see what
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>happens to the
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  error
> >>>>>>>>>>>>  > >percentage...
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > > -- Keir
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >On 4/1/08 23:24, "Dave Winchell"
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >>Hi Dan and Keir,
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>Attached is a patch that fixes some issues with the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>SYNC policy
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>(no_missed_ticks_pending).
> >>>>>>>>>>>>  > >>I have not tried to make the change the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>minimal one, but,
> >>
> >>
> >>>>>>>>>>>>  > rather, just
> >>>>>>>>>>>>  > >>ported into
> >>>>>>>>>>>>  > >>the new code what I know to work well. The error for
> >>>>>>>>>>>>  > >>no_missed_ticks_pending goes from
> >>>>>>>>>>>>  > >>over 3% to .03% with this change according
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>to my testing.
> >>
> >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>Regards,
> >>>>>>>>>>>>  > >>Dave
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>Dan Magenheimer wrote:
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>>Hi Dave --
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>Did you get your correction ported?  If so,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>it would be
> >>
> >>
> >>>>>>>>>>>>  > nice to see this get
> >>>>>>>>>>>>  > >>>into 3.1.3.
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>Note that I just did some very limited testing with
> >>>>>>>>>>>>  > timer_mode=2(=SYNC=no
> >>>>>>>>>>>>  > >>>missed ticks pending)
> >>>>>>>>>>>>  > >>>on tip of xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>guest) and the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > worst error I've
> >>>>>>>>>>>>  > >>>seen so far
> >>>>>>>>>>>>  > >>>is 0.012%.  But I haven't tried any exotic
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>loads, just LTP.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>Thanks,
> >>>>>>>>>>>>  > >>>Dan
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>>-----Original Message-----
> >>>>>>>>>>>>  > >>>>From: Dave Winchell
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:dwinchell@virtualiron.com]
> >>
> >>
> >>>>>>>>>>>>  > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>>>  > >>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>>  > >>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>>>>>>>>  > xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>>>  > >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>>>  > >>>>Subject: Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>timer mode that
> >>
> >>
> >>>>>>>>>>>>  > >>>>disables pending
> >>>>>>>>>>>>  > >>>>missed ticks
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>Dan,
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>I did some testing with the constant tsc offset
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>SYNC method
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>(now called
> >>>>>>>>>>>>  > >>>>no_missed_ticks_pending)
> >>>>>>>>>>>>  > >>>>and found the error to be very high, much larger
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>than 1 %, as
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>I recall.
> >>>>>>>>>>>>  > >>>>I have not had a chance to submit a correction. I
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>will try to
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>do it later
> >>>>>>>>>>>>  > >>>>this week or the first week in January. My
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>version of
> >>
> >>
> >>>>>>>>>>>>  constant tsc
> >>>>>>>>>>>>  > >>>>offset SYNC method
> >>>>>>>>>>>>  > >>>>produces .02 % error, so I just need to port
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>that into the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>current code.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>The error you got for both of those kernels is
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>what I would
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  expect
> >>>>>>>>>>>>  > >>>>for the default mode, delay_for_missed_ticks.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>I'll let Keir answer on how to set the time mode.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>Regards,
> >>>>>>>>>>>>  > >>>>Dave
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>>Anyone make measurements on the final patch?
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>saw a loss of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>about 0.2% with no load.  This was
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>xen-unstable tip today
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>with no options specified.  32-bit was 
> about 0.01%.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>>I think I missed something... how do I
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>run the various
> >>
> >>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>accounting choices and which ones are known to be
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>appropriate
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>for which kernels?
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>>Thanks,
> >>>>>>>>>>>>  > >>>>>Dan
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>>-----Original Message-----
> >>>>>>>>>>>>  > >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>Keir Fraser
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>>>  > >>>>>>To: Dave Winchell
> >>>>>>>>>>>>  > >>>>>>Cc: Shan, Haitao;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>xen-devel@lists.xensource.com; Dong,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > Eddie; Jiang,
> >>>>>>>>>>>>  > >>>>>>Yunhong
> >>>>>>>>>>>>  > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>mode that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>disables pending
> >>>>>>>>>>>>  > >>>>>>missed ticks
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>Please take a look at xen-unstable
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>changeset 16545.
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>-- Keir
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>Keir,
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>The accuracy data I've collected for i/o
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>loads for the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>various time protocols follows. In
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>addition, the data
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>for cpu loads is shown.
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>processor AMD
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  box.
> >>>>>>>>>>>>  > >>>>>>>Two guests, red hat and sles 64 bit, 8
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>vcpu each.
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>>>  > >>>>>>>(usex is available at
> >>>>>>>>>>>>  http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>>>  > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>of=/dev/null.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>The loads labeled i/o-32 are 32 
> instances of dd.
> >>>>>>>>>>>>  > >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>>>  > >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>>>  > >>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>The loads labeled i/o-4/32 are the same
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>as i/o-32
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>instances of dd.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>>>>+4.42
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec -.006%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > +.005% cpu
> >>>>>>>>>>>>  > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec, -.001%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > +.012% cpu
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.009%,
> >>
> >>
> >>>>>>>>>>>>  -.004% cpu
> >>>>>>>>>>>>  > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.005%, -.005% cpu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.008%, -.003% cpu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.040% cpu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec, -.034%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > -.031% cpu
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec,-55.7 sec, -.01%,
> >>
> >>
> >>>>>>>>>>>>  > -.09% i/o-8
> >>>>>>>>>>>>  > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec,-14.0 sec, -.015%
> >>
> >>
> >>>>>>>>>>>>  > -.14% i/o-8
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.017%,
> >>
> >>
> >>>>>>>>>>>>  > -.022% i/o-8
> >>>>>>>>>>>>  > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.017%, -.018% i/o-8
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.020%,
> >>
> >>
> >>>>>>>>>>>>  > -.029% i/o-8
> >>>>>>>>>>>>  > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec, -.023%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > -.031% i/o-8
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.04% i/o-32
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.011%,
> >>
> >>
> >>>>>>>>>>>>  > -.005% i/o-32
> >>>>>>>>>>>>  > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.11% i/o-32
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>13. sec -.07%,
> >>
> >>
> >>>>>>>>>>>>  > .003% i/o-4/32
> >>>>>>>>>>>>  > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.017%,
> >>
> >>
> >>>>>>>>>>>>  > .01% i/o-4/32
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>Overhead measurements:
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>Progress in terms of number of passes
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>through a fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>system workload
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>>>>>>  > >>>>>>>The workload was usex -b48.
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>>>  > >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>>>  > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>>>  > >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>Conclusions:
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>The only protocol which meets the .05% accuracy
> >>>>>>>>>>>>  > requirement for ntp
> >>>>>>>>>>>>  > >>>>>>>tracking under the loads
> >>>>>>>>>>>>  > >>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>accuracies for
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>SYNC, MIXED,
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>and ASYNC
> >>>>>>>>>>>>  > >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>We could reduce the cost of the SYNC
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>method by only
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>scheduling the extra
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>wakeups if a certain number
> >>>>>>>>>>>>  > >>>>>>>of ticks are missed.
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>Regards,
> >>>>>>>>>>>>  > >>>>>>>Dave
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>Keir Fraser wrote:
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>ASYNC method a
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>couple of days ago,
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>I ran another ASYNC test. I think
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>there may have
> >>
> >>
> >>>>>>>>>>>>  > been something
> >>>>>>>>>>>>  > >>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>days ago for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>ASYNC. It may have been
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>after context
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>switch in.
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>My results indicate that either SYNC
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>or ASYNC give
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>acceptable accuracy,
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>each running consistently around or under
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>.01%. MIXED has
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>a fairly high
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>error of
> >>>>>>>>>>>>  > >>>>>>>>>greater than .03%. Probably too close
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>to .05% ntp
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>threshold for comfort.
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>I don't have an overnight run with SYNC. I
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>plan to leave
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>SYNC running
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>over the weekend. If you'd rather I can
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>leave MIXED
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>running instead.
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>It may be too early to pick the protocol and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>I can run
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>more overnight tests
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>next week.
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>I'm a bit worried about any unwanted side
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>effects of the
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>SYNC+run_timer
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>cause higher
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>system-wide CPU
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>contention. I find it easier to think
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>through the
> >>
> >>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>implications of ASYNC. I'm
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>accurate than
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>ASYNC. Perhaps it
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>delivers more timer interrupts than the other
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>approaches,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>and each interrupt
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>favourites and
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>if the latter is
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>actually more accurate then I can
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>simply revert the
> >>
> >>
> >>>>>>>>>>>>  > changeset that
> >>>>>>>>>>>>  > >>>>>>>>implemented MIXED.
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>workloads you
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>could try idle
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>large disc reads
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>to /dev/null)? We
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>don't have any data on workloads that aren't
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>CPU bound, so
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>that's really an
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>-- Keir
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>>>
> >>>>>>>>>>>>  > >>>>>>_______________________________________________
> >>>>>>>>>>>>  > >>>>>>Xen-devel mailing list
> >>>>>>>>>>>>  > >>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>>>  > >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>>>  > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>2007 +0000
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>2008 -0500
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>@@ -58,7 +58,7 @@ static void
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>pt_process_missed_ticks(stru
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>     missed_ticks = missed_ticks / (s_time_t)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>pt->period + 1;
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>     if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>no_missed_ticks_pending) )
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>>>  > >>+        pt->do_not_freeze = 1;
> >>>>>>>>>>>>  > >>     else
> >>>>>>>>>>>>  > >>         pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>>>  > >>     pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>>>  > >>@@ -127,7 +127,12 @@ static void
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>pt_timer_fn(void *data)
> >>
> >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>     pt_lock(pt);
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>-    pt->pending_intr_nr++;
> >>>>>>>>>>>>  > >>+    if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>no_missed_ticks_pending) ) {
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>+        pt->pending_intr_nr = 1;
> >>>>>>>>>>>>  > >>+ pt->do_not_freeze = 0;
> >>>>>>>>>>>>  > >>+    }
> >>>>>>>>>>>>  > >>+    else
> >>>>>>>>>>>>  > >>+ pt->pending_intr_nr++;
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>     if ( !pt->one_shot )
> >>>>>>>>>>>>  > >>     {
> >>>>>>>>>>>>  > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>vcpu *v, struct
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>         return;
> >>>>>>>>>>>>  > >>     }
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>-    pt->do_not_freeze = 0;
> >>>>>>>>>>>>  > >>-
> >>>>>>>>>>>>  > >>     if ( pt->one_shot )
> >>>>>>>>>>>>  > >>     {
> >>>>>>>>>>>>  > >>         pt->enabled = 0;
> >>>>>>>>>>>>  > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>*v, struct
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>             pt->last_plt_gtime =
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>hvm_get_guest_time(v);
> >>
> >>
> >>>>>>>>>>>>  > >>             pt->pending_intr_nr = 0; /*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>'collapse' all
> >>
> >>
> >>>>>>>>>>>>  > missed ticks */
> >>>>>>>>>>>>  > >>         }
> >>>>>>>>>>>>  > >>+ else if ( mode_is(v->domain,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>no_missed_ticks_pending) ) {
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>+     pt->pending_intr_nr--;
> >>>>>>>>>>>>  > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>>>  > >>+ }
> >>>>>>>>>>>>  > >>         else
> >>>>>>>>>>>>  > >>         {
> >>>>>>>>>>>>  > >>             pt->last_plt_gtime += 
> pt->period_cycles;
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > _______________________________________________
> >>>>>>>>>>>>  > Xen-devel mailing list
> >>>>>>>>>>>>  > Xen-devel@lists.xensource.com
> >>>>>>>>>>>>  > http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> >>>
> >>>
> >>--------------------------------------------------------------
> >>----------
> >>
> >>
> >>
> >>
> >
> >
> >
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

Mine was oversubscribed.
8 physical cpu, 2 guests, each with 8 vcpu.
I ran one instance of ltp on each guest, continuously. I hope ltp
loaded up all the vcpus. I seem to recall that it did, but I
could be wrong. If it didn't, that would be a major difference
between our tests. I'll verify this afternoon and run multiple instances,
if necessary.

Thanks,
Dave



Dan Magenheimer wrote:

>Hi Dave --
>
>No new results yet but one other question:
>
>The problems we've seen with our testing have been with a heavily
>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
>running LTP simultaneously.
>
>Was your LTP testing oversubscribed or just a single guest?
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Thursday, February 14, 2008 10:56 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
>>Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>Here are some boot snipets for rh4u564 on xen 3.2.
>>
>>
>>#1:
>>
>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>...
>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
>>console=ttyS0 clocksource=pit nohpet
>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
>>Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11,
>>65536 bytes)
>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
>>...
>>Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8
>>CPUs: passed.
>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use 
>>of PM timer
>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
>>
>>
>>
>>#2:
>>
>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>...
>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
>>console=ttyS0 clocksource=pit nohpet nopmtimer
>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
>>Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11,
>>65536 bytes)
>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
>>...
>>Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8
>>CPUs: passed.
>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.
>>
>>
>>As you can see, I only get the pit if I specify nopmtimer.
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Hi Dave --
>>>
>>>Thanks for continuing to run tests!
>>>
>>>Hmmm... I thought I had noticed that even though Linux will 
>>>      
>>>
>>acknowledge
>>    
>>
>>>the existence of the pmtimer, it still prints:
>>>
>>>time.c: Using PIT/TSC based timekeeping.
>>>
>>>I will check again, but assuming the clocksource for our tests is
>>>indeed pit, the huge difference in the results (yours vs ours) is
>>>baffling. I wonder if the difference may be the underlying hardware.
>>>Maybe we will try to ensure we can duplicate the results on 
>>>      
>>>
>>a different
>>    
>>
>>>box.
>>>
>>>
>>>So your testing was with stock 3.2.0 xen bits (what cset?) without
>>>any of your [quote from below] "clock related tweaks that I haven't
>>>submitted, because I'm still characterizing them"?
>>>
>>>
>>>      
>>>
>>None of the tweaks I mentioned are in this test.
>>It was stock with some patches. However, none of the patches are time
>>related to
>>my knowledge and I checked vpt.c to make sure that it is the same as
>>what's in unstable.
>>The only difference is in pt_intr_post, where I set the timer mode.
>>I don't have timer mode tied into our config process yet, which
>>is different than official xen method.
>>
>>
>>(In pt_intr_post)
>>      else
>>      {
>>+       if(v->arch.paging.mode->guest_levels == 4)
>>+           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
>>HVMPTM_no_missed_ticks_pending;
>>+       else
>>+           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
>>HVMPTM_delay_for_missed_ticks;
>>          if ( mode_is(v->domain, one_missed_tick_pending) ||
>>               mode_is(v->domain, no_missed_ticks_pending) )
>>          {
>>
>>    
>>
>>>Could you also send detail on the rhel4u4-64 kernel you
>>>are testing with, just to ensure we are not comparing apples
>>>and oranges?  (Perhaps there's some way we can even share the
>>>identical disk image and vm.cfg file?)
>>>
>>>And if our problem is indeed the pmtimer, I will need to submit
>>>another patch to Keir to add an hvm pmtimer platform variable.
>>>(Hmmm... I don't think he's even accepted the hpet variable patch
>>>yet.  I'll have to check.)
>>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>Sent: Thursday, February 14, 2008 9:00 AM
>>>>To: dan.magenheimer@oracle.com
>>>>Cc: Dave Winchell; Keir Fraser; 
>>>>        
>>>>
>>xen-devel@lists.xensource.com; Deepak
>>    
>>
>>>>Patel
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Hi Dan,
>>>>
>>>>I ran the ltp tests with 3.2 and found the errors
>>>>for a 16 hour run to be:
>>>>
>>>>rh4u564 -9.9 sec (-.017%)
>>>>rh4u464 -7.3 sec (-.013%)
>>>>
>>>>There were no cliffs and the drift was linear.
>>>>
>>>>I think the problem you had may be due to the use of the
>>>>pm timer. If you still have the boot log, it would tell you.
>>>>
>>>>When I first tried a guest on 3.2 with "clocksource=pit nohpet"
>>>>I noticed that it picked the pm timer. Adding "nopmtimer", the
>>>>guest will pick the pit.
>>>>
>>>>The reason I didn't have the problem with our 3.1 base is that
>>>>I had disabled the hpet and the pmtimer by not advertising them
>>>>in the acpi tables. I did this so long ago, I forgot that I had to
>>>>disable pmtimer as well as hpet.
>>>>
>>>>So, can you re-run your test with "clocksource=pit nohpet 
>>>>        
>>>>
>>nopmtimer"?
>>    
>>
>>>>You should see this in the boot messages:
>>>>
>>>>time.c: Using PIT/TSC based timekeeping.
>>>>
>>>>Thanks,
>>>>Dave
>>>>
>>>>
>>>>Dave Winchell wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>Hi Dan,
>>>>>
>>>>>Over the weekend the drift was +18 seconds for each guest (no ntp).
>>>>>The duration was 3900 minutes, so the error for each was +.0077%.
>>>>>Looking back through the data, it appears to drift linearly at
>>>>>this rate. I've attached a plot for rh4u5-64.
>>>>>
>>>>>This accuracy is better than what I've seen before (.03-.05%).
>>>>>This may be due to the different load (ltp vs usex) or to 
>>>>>          
>>>>>
>>one of the
>>    
>>
>>>>>changes I've made recently. I'll do some experimentation to see if
>>>>>there is
>>>>>a fix I should propose.
>>>>>
>>>>>This still doesn't address the radical drift you saw.
>>>>>The next step for me is to run 3.2 and see if I can reproduce it.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Dave Winchell wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Hi Dan,
>>>>>>
>>>>>>Sorry it took me so long, but I finally ran an ltp test today.
>>>>>>Its on rh4u4-64. I'm using the defaults for ltp and using a script
>>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
>>>>>>virtual processors / physical processors = 2.
>>>>>>
>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
>>>>>>for -.005% and .008%.
>>>>>>
>>>>>>I'm running a 3.1 based hypervisor with some clock related
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>tweaks that
>>>>
>>>>
>>>>        
>>>>
>>>>>>I haven't submitted, because I'm still characterizing them.
>>>>>>
>>>>>>I'm stopping the usex load on 4u5-64 now and replacing it with ltp
>>>>>>and will leave the two guests running ltp over the weekend.
>>>>>>
>>>>>>Regards,
>>>>>>Dave
>>>>>>
>>>>>>
>>>>>>Dave Winchell wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Hi Dan, Deepak:
>>>>>>>
>>>>>>>Thanks for the data. Those drifts are severe - no wonder
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ntp couldn't
>>>>
>>>>
>>>>        
>>>>
>>>>>>>keep then in synch. I'll try to reproduce that behaviour
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>here, with
>>>>
>>>>
>>>>        
>>>>
>>>>>>>my code base.
>>>>>>>If I can't reproduce it, I'll try 3.2.
>>>>>>>
>>>>>>>If you can isolate what ltp is doing during the cliffs, 
>>>>>>>              
>>>>>>>
>>that would
>>    
>>
>>>>>>>be very
>>>>>>>helpful.
>>>>>>>
>>>>>>>thanks,
>>>>>>>Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Dan Magenheimer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>OK, Deepak repeated the test without ntpd and using
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>ntpdate -b before
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>the test.
>>>>>>>>
>>>>>>>>The attached graph shows his results: el5u1-64 (best=~0.07%),
>>>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>>>>>
>>>>>>>>We will continue to look at LTP to try to isolate.
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>Dan
>>>>>>>>
>>>>>>>>P.S. elXuY is essentially RHEL XuY with some patches.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>-----Original Message-----
>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
>>>>>>>>>To: Deepak Patel
>>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>Dave Winchell
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>>>>pending
>>>>>>>>>missed ticks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Dan, Deeepak,
>>>>>>>>>
>>>>>>>>>It may be that the underlying clock error is too great for ntp
>>>>>>>>>to handle. It would be useful if you did not run ntpd
>>>>>>>>>and, instead did ntpdate -b <timeserver> at the start
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>of the test
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>for each guest. Then capture the data as you have been doing.
>>>>>>>>>If the drift is greater than .05%, then we need to 
>>>>>>>>>                  
>>>>>>>>>
>>address that.
>>    
>>
>>>>>>>>>Another option is, when running ntpd, to enable loop
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>statistics in
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>/etc/ntp.conf
>>>>>>>>>by adding this to the file:
>>>>>>>>>
>>>>>>>>>statistics loopstats
>>>>>>>>>statsdir /var/lib/ntp/
>>>>>>>>>
>>>>>>>>>Then you will see loop data in that directory.
>>>>>>>>>Correlating the data in the loopstats files with the
>>>>>>>>>peaks in skew would be interesting. You will see
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>entries of the form
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>239.735511 10
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>Where the second to last column is the Allan Deviation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>When that
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>gets over 1000, ntpd is working pretty hard. However, 
>>>>>>>>>                  
>>>>>>>>>
>>I have not
>>    
>>
>>>>>>>>>seen ntpd
>>>>>>>>>completely loose it like you have.
>>>>>>>>>
>>>>>>>>>I'm on vacation until Monday, and won't be reading
>>>>>>>>>email.
>>>>>>>>>
>>>>>>>>>Thanks for all your work on this!
>>>>>>>>>
>>>>>>>>>-Dave
>>>>>>>>>
>>>>>>>>>Deepak Patel wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>I do not know which graph was attached with this. But
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>I saw this
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>guests when I
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>was running ltp tests continuously.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>What was the behaviour of the other guests running?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>All pvm guests are fine. But behavior of most of the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>hvm guests were
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>as described.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>If they had spikes, were they at the same wall time?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>No. They are not at the same wall time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>Were the other guests running ltp as well?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>>>>>>>>continuously.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>How are you measuring skew?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>I was collecting output of "ntpdate -q  <timeserver> every
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>300 seconds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>(5 minutes) and have created graph based on that.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>Are you running ntpd?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>Yes. ntp was running on all the guests.
>>>>>>>>>>
>>>>>>>>>>I am investigating what causes this spikes and let everyone
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>know what
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>are my findings.
>>>>>>>>>>
>>>>>>>>>>Thanks,
>>>>>>>>>>Deepak
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>Anything that you can discover that would be in sync with
>>>>>>>>>>>the spikes would be very helpful!
>>>>>>>>>>>
>>>>>>>>>>>The code that I test with is our product code, which is based
>>>>>>>>>>>on 3.1. So it is possible that something in 3.2 other
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>than vpt.c
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>is the cause. I can test with 3.2, if necessary.
>>>>>>>>>>>
>>>>>>>>>>>thanks,
>>>>>>>>>>>Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>Hi Dave (Keir, see suggestion below) --
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>>Turning off vhpet certainly helps a lot (though see below).
>>>>>>>>>>>>
>>>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>should be
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>until it is
>>    
>>
>>>>>>>>>>>>fixed?  (I have a patch that defaults it off, can post it if
>>>>>>>>>>>>there is agreement on the above point.)  The whole
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>point of an
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>HPET is to provide more precise timekeeping and if vhpet is
>>>>>>>>>>>>worse than vpit, it can only confuse users.  Comments?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>In your testing, are you just measuring % skew over a long
>>>>>>>>>>>>period of time?
>>>>>>>>>>>>We are graphing the skew continuously and
>>>>>>>>>>>>seeing periodic behavior that is unsettling, even with pit.
>>>>>>>>>>>>See attached.  Though your algorithm recovers, the "cliffs"
>>>>>>>>>>>>could still cause real user problems.  I wonder if there is
>>>>>>>>>>>>anything that can be done to make the "recovery" more
>>>>>>>>>>>>responsive?
>>>>>>>>>>>>
>>>>>>>>>>>>We are looking into what part(s) of LTP is causing
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>the cliffs.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>Dan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>>>>>>>deepak.patel@oracle.com;
>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>that disables
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>pending
>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Dan,
>>>>>>>>>>>>>
>>>>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage.
>>>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>should specify
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>>>>>>>
>>>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are.
>>>>>>>>>>>>>The xen and guest clocksources do not need to be the same.
>>>>>>>>>>>>>In my tests, xen is using the hpet for its timekeeping and
>>>>>>>>>>>>>that appears to be the default.
>>>>>>>>>>>>>
>>>>>>>>>>>>>When you boot the guests you should see
>>>>>>>>>>>>> time.c: Using PIT/TSC based timekeeping.
>>>>>>>>>>>>>on the rh4u5-64 guest, and something similar on the others.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>>>>>>14.318MHz HPET.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>This appears to be the xen state, which is fine.
>>>>>>>>>>>>>I was wrongly assuming that this was the guest state.
>>>>>>>>>>>>>You might want to look in your guest logs and see
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>what they were
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>picking
>>>>>>>>>>>>>for a clock source.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thanks, I hadn't realized that!  No wonder we didn't
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>see the same
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>>improvement you saw!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Try specifying clock=pit on the linux boot line...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>command line or
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>guest (or
>>    
>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>dom0?) command
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>line?  Or both places?  Since the tests take awhile, it
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>would be nice
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>to get this right the first time.  Do the Xen and guest
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>clocksources need
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>to be the same?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>deepak.patel@oracle.com;
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>that disables
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>>pending missed ticks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hpet timer does have a fairly large error, as I
>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>trying this
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> one recently.
>>>>>>>>>>>>>> I don't remember what I got for error, but 1% sounds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>about right.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>the module
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Keir and I did
>>>>>>>>>>>>>> all the recent work in, for its periodic timer 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>needs. Try
>>    
>>
>>>>>>>>>>>>>> specifying clock=pit
>>>>>>>>>>>>>> on the linux boot line. If it still picks the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>hpet, which it
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> might, let me know
>>>>>>>>>>>>>> and I'll tell you how to get around this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>--------------------------------------------------------------
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>----------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> *From:* Dan Magenheimer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>[mailto:dan.magenheimer@oracle.com]
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser
>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>deepak.patel@oracle.com;
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> akira.ijuin@oracle.com
>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>that disables
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> pending missed ticks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry for the very late followup on this but we finally
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>were able
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> to get our testing set up again on stable 3.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>bits and have
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>order of 1%.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> Test enviroment was a 4-socket dual core machine
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>with 24GB of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>plus two pv.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> All six guests were running LTP simultaneously.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>The four hvm
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>RHEL4u5-64.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>32-bit guests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> All four hvm guests experienced skew around -1%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>even the 32-bit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> guest.  Less intensive testing didn't exhibit much
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>skew at all.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> A representative graph is attached.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dave, I wonder if some portion of your patches
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>didn't end up in
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> the xen trees?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>Platform timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > -----Original Message-----
>>>>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > Dave Winchell
>>>>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>>>>> > To: Keir Fraser
>>>>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > Winchell
>>>>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>>>>> > disables pending
>>>>>>>>>>>>>> > missed ticks
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Hi Keir,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > The latest change, c/s 16690, looks fine.
>>>>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to
>>>>>>>>>>>>>> > the code I submitted. Also, your version is more
>>>>>>>>>>>>>> > concise.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > The error tests confirm the equivalence. With
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>overnight cpu loads,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > the checked in version was accurate to +.048% for sles
>>>>>>>>>>>>>> > and +.038% for red hat. My version was +.046%
>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>+.032% in a
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > 2 hour test.
>>>>>>>>>>>>>> > I don't think the difference is significant.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > i/o loads produced errors of +.01%.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thanks for all your efforts on this issue.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Regards,
>>>>>>>>>>>>>> > Dave
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Keir Fraser wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > >Applied as c/s 16690, although the 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>checked-in patch is
>>    
>>
>>>>>>>>>>>>>> > smaller. I think the
>>>>>>>>>>>>>> > >only important fix is to pt_intr_post() and the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>only bit of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > the patch I
>>>>>>>>>>>>>> > >totally omitted was the change to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>pt_process_missed_ticks().
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > I don't think
>>>>>>>>>>>>>> > >that change can be important, but let's see what
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>happens to the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> error
>>>>>>>>>>>>>> > >percentage...
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > -- Keir
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >>Hi Dan and Keir,
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>Attached is a patch that fixes some issues with the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>SYNC policy
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>(no_missed_ticks_pending).
>>>>>>>>>>>>>> > >>I have not tried to make the change the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>minimal one, but,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > rather, just
>>>>>>>>>>>>>> > >>ported into
>>>>>>>>>>>>>> > >>the new code what I know to work well. The error for
>>>>>>>>>>>>>> > >>no_missed_ticks_pending goes from
>>>>>>>>>>>>>> > >>over 3% to .03% with this change according
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>to my testing.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>Regards,
>>>>>>>>>>>>>> > >>Dave
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>Dan Magenheimer wrote:
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>>Hi Dave --
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>Did you get your correction ported?  If so,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>it would be
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > nice to see this get
>>>>>>>>>>>>>> > >>>into 3.1.3.
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>Note that I just did some very limited testing with
>>>>>>>>>>>>>> > timer_mode=2(=SYNC=no
>>>>>>>>>>>>>> > >>>missed ticks pending)
>>>>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>guest) and the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > worst error I've
>>>>>>>>>>>>>> > >>>seen so far
>>>>>>>>>>>>>> > >>>is 0.012%.  But I haven't tried any exotic
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>loads, just LTP.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>Thanks,
>>>>>>>>>>>>>> > >>>Dan
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>>-----Original Message-----
>>>>>>>>>>>>>> > >>>>From: Dave Winchell
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>[mailto:dwinchell@virtualiron.com]
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>timer mode that
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>disables pending
>>>>>>>>>>>>>> > >>>>missed ticks
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>Dan,
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>SYNC method
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>(now called
>>>>>>>>>>>>>> > >>>>no_missed_ticks_pending)
>>>>>>>>>>>>>> > >>>>and found the error to be very high, much larger
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>than 1 %, as
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>I recall.
>>>>>>>>>>>>>> > >>>>I have not had a chance to submit a correction. I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>will try to
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>do it later
>>>>>>>>>>>>>> > >>>>this week or the first week in January. My
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>version of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> constant tsc
>>>>>>>>>>>>>> > >>>>offset SYNC method
>>>>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>that into the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>current code.
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>The error you got for both of those kernels is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>what I would
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>Regards,
>>>>>>>>>>>>>> > >>>>Dave
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>saw a loss of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>about 0.2% with no load.  This was
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>xen-unstable tip today
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>with no options specified.  32-bit was 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>about 0.01%.
>>    
>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>>I think I missed something... how do I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>run the various
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>accounting choices and which ones are known to be
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>appropriate
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>for which kernels?
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>>Thanks,
>>>>>>>>>>>>>> > >>>>>Dan
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>>-----Original Message-----
>>>>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>Keir Fraser
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>>> > >>>>>>To: Dave Winchell
>>>>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>xen-devel@lists.xensource.com; Dong,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > Eddie; Jiang,
>>>>>>>>>>>>>> > >>>>>>Yunhong
>>>>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>mode that
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>disables pending
>>>>>>>>>>>>>> > >>>>>>missed ticks
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>changeset 16545.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>-- Keir
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>Keir,
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>loads for the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>various time protocols follows. In
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>addition, the data
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>for cpu loads is shown.
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>processor AMD
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> box.
>>>>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>vcpu each.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>>> > >>>>>>>(usex is available at
>>>>>>>>>>>>>> http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>of=/dev/null.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>instances of dd.
>>    
>>
>>>>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>as i/o-32
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>instances of dd.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>>>+4.42
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec -.006%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > +.005% cpu
>>>>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec, -.001%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > +.012% cpu
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec, -.009%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> -.004% cpu
>>>>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.005%, -.005% cpu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.008%, -.003% cpu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.040% cpu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec, -.034%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > -.031% cpu
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec,-55.7 sec, -.01%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.09% i/o-8
>>>>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec,-14.0 sec, -.015%
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.14% i/o-8
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec, -.017%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.022% i/o-8
>>>>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.017%, -.018% i/o-8
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec, -.020%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.029% i/o-8
>>>>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec, -.023%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > -.031% i/o-8
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.04% i/o-32
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec, -.011%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.005% i/o-32
>>>>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.11% i/o-32
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>13. sec -.07%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > .003% i/o-4/32
>>>>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>sec, -.017%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > .01% i/o-4/32
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>Overhead measurements:
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>through a fixed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>system workload
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>>>>> > >>>>>>>The workload was usex -b48.
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>Conclusions:
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>>>>> > requirement for ntp
>>>>>>>>>>>>>> > >>>>>>>tracking under the loads
>>>>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>accuracies for
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>SYNC, MIXED,
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>and ASYNC
>>>>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>method by only
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>scheduling the extra
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>wakeups if a certain number
>>>>>>>>>>>>>> > >>>>>>>of ticks are missed.
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>Regards,
>>>>>>>>>>>>>> > >>>>>>>Dave
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>ASYNC method a
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>couple of days ago,
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>there may have
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > been something
>>>>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>days ago for
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>ASYNC. It may have been
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>after context
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>switch in.
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>or ASYNC give
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>acceptable accuracy,
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>each running consistently around or under
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>.01%. MIXED has
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>a fairly high
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>error of
>>>>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>to .05% ntp
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>threshold for comfort.
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>plan to leave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>SYNC running
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>leave MIXED
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>running instead.
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>I can run
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>more overnight tests
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>next week.
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>effects of the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>SYNC+run_timer
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>cause higher
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>system-wide CPU
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>through the
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>accurate than
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>approaches,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>and each interrupt
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>favourites and
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>if the latter is
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>actually more accurate then I can
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>simply revert the
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > changeset that
>>>>>>>>>>>>>> > >>>>>>>>implemented MIXED.
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>workloads you
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>could try idle
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>large disc reads
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>to /dev/null)? We
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>CPU bound, so
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>that's really an
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>-- Keir
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>> > >>>>>>_______________________________________________
>>>>>>>>>>>>>> > >>>>>>Xen-devel mailing list
>>>>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>2007 +0000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>2008 -0500
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>pt_process_missed_ticks(stru
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>pt->period + 1;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>no_missed_ticks_pending) )
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>>> > >>+        pt->do_not_freeze = 1;
>>>>>>>>>>>>>> > >>     else
>>>>>>>>>>>>>> > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>>> > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>pt_timer_fn(void *data)
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>     pt_lock(pt);
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>-    pt->pending_intr_nr++;
>>>>>>>>>>>>>> > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>+        pt->pending_intr_nr = 1;
>>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 0;
>>>>>>>>>>>>>> > >>+    }
>>>>>>>>>>>>>> > >>+    else
>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr++;
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>     if ( !pt->one_shot )
>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>vcpu *v, struct
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>         return;
>>>>>>>>>>>>>> > >>     }
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>-    pt->do_not_freeze = 0;
>>>>>>>>>>>>>> > >>-
>>>>>>>>>>>>>> > >>     if ( pt->one_shot )
>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>> > >>         pt->enabled = 0;
>>>>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>*v, struct
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>             pt->last_plt_gtime =
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>hvm_get_guest_time(v);
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>             pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>'collapse' all
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > missed ticks */
>>>>>>>>>>>>>> > >>         }
>>>>>>>>>>>>>> > >>+ else if ( mode_is(v->domain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>+     pt->pending_intr_nr--;
>>>>>>>>>>>>>> > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>>> > >>+ }
>>>>>>>>>>>>>> > >>         else
>>>>>>>>>>>>>> > >>         {
>>>>>>>>>>>>>> > >>             pt->last_plt_gtime += 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>pt->period_cycles;
>>    
>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>> > Xen-devel mailing list
>>>>>>>>>>>>>> > Xen-devel@lists.xensource.com
>>>>>>>>>>>>>> > http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>
>>>>>          
>>>>>
>>>>--------------------------------------------------------------
>>>>----------
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

ltp runs by default loading up only one vcpu.
The -x option can be used to run multiple instances, though
in this mode you will get test failures.
I ran 8 instances on each guest for 16 hours, 25 min
and the time error was -11 sec (-.019%) on each guest.

Regards,
Dave

Dave Winchell wrote:

> Hi Dan,
>
> Mine was oversubscribed.
> 8 physical cpu, 2 guests, each with 8 vcpu.
> I ran one instance of ltp on each guest, continuously. I hope ltp
> loaded up all the vcpus. I seem to recall that it did, but I
> could be wrong. If it didn't, that would be a major difference
> between our tests. I'll verify this afternoon and run multiple instances,
> if necessary.
>
> Thanks,
> Dave
>
>
>
> Dan Magenheimer wrote:
>
>> Hi Dave --
>>
>> No new results yet but one other question:
>>
>> The problems we've seen with our testing have been with a heavily
>> oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
>> running LTP simultaneously.
>>
>> Was your LTP testing oversubscribed or just a single guest?
>>
>> Thanks,
>> Dan
>>
>>  
>>
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Thursday, February 14, 2008 10:56 AM
>>> To: dan.magenheimer@oracle.com
>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
>>> Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending
>>> missed ticks
>>>
>>>
>>> Dan,
>>>
>>> Here are some boot snipets for rh4u564 on xen 3.2.
>>>
>>>
>>> #1:
>>>
>>> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
>>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
>>> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
>>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>> ...
>>> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
>>> console=ttyS0 clocksource=pit nohpet
>>> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
>>> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11,
>>> 65536 bytes)
>>> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
>>> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
>>> ...
>>> Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8
>>> CPUs: passed.
>>> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
>>> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use of PM timer
>>> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
>>>
>>>
>>>
>>> #2:
>>>
>>> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
>>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
>>> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
>>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>> ...
>>> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
>>> console=ttyS0 clocksource=pit nohpet nopmtimer
>>> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
>>> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11,
>>> 65536 bytes)
>>> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
>>> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
>>> ...
>>> Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8
>>> CPUs: passed.
>>> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
>>> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.
>>>
>>>
>>> As you can see, I only get the pit if I specify nopmtimer.
>>>
>>> Dan Magenheimer wrote:
>>>
>>>   
>>>
>>>> Hi Dave --
>>>>
>>>> Thanks for continuing to run tests!
>>>>
>>>> Hmmm... I thought I had noticed that even though Linux will     
>>>
>>> acknowledge
>>>   
>>>
>>>> the existence of the pmtimer, it still prints:
>>>>
>>>> time.c: Using PIT/TSC based timekeeping.
>>>>
>>>> I will check again, but assuming the clocksource for our tests is
>>>> indeed pit, the huge difference in the results (yours vs ours) is
>>>> baffling. I wonder if the difference may be the underlying hardware.
>>>> Maybe we will try to ensure we can duplicate the results on     
>>>
>>> a different
>>>   
>>>
>>>> box.
>>>>
>>>>
>>>> So your testing was with stock 3.2.0 xen bits (what cset?) without
>>>> any of your [quote from below] "clock related tweaks that I haven't
>>>> submitted, because I'm still characterizing them"?
>>>>
>>>>
>>>>     
>>>
>>> None of the tweaks I mentioned are in this test.
>>> It was stock with some patches. However, none of the patches are time
>>> related to
>>> my knowledge and I checked vpt.c to make sure that it is the same as
>>> what's in unstable.
>>> The only difference is in pt_intr_post, where I set the timer mode.
>>> I don't have timer mode tied into our config process yet, which
>>> is different than official xen method.
>>>
>>>
>>> (In pt_intr_post)
>>>      else
>>>      {
>>> +       if(v->arch.paging.mode->guest_levels == 4)
>>> +           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
>>> HVMPTM_no_missed_ticks_pending;
>>> +       else
>>> +           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
>>> HVMPTM_delay_for_missed_ticks;
>>>          if ( mode_is(v->domain, one_missed_tick_pending) ||
>>>               mode_is(v->domain, no_missed_ticks_pending) )
>>>          {
>>>
>>>   
>>>
>>>> Could you also send detail on the rhel4u4-64 kernel you
>>>> are testing with, just to ensure we are not comparing apples
>>>> and oranges?  (Perhaps there's some way we can even share the
>>>> identical disk image and vm.cfg file?)
>>>>
>>>> And if our problem is indeed the pmtimer, I will need to submit
>>>> another patch to Keir to add an hvm pmtimer platform variable.
>>>> (Hmmm... I don't think he's even accepted the hpet variable patch
>>>> yet.  I'll have to check.)
>>>>
>>>> Thanks,
>>>> Dan
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>> -----Original Message-----
>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>> Sent: Thursday, February 14, 2008 9:00 AM
>>>>> To: dan.magenheimer@oracle.com
>>>>> Cc: Dave Winchell; Keir Fraser;       
>>>>
>>> xen-devel@lists.xensource.com; Deepak
>>>   
>>>
>>>>> Patel
>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>> disables pending
>>>>> missed ticks
>>>>>
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>> I ran the ltp tests with 3.2 and found the errors
>>>>> for a 16 hour run to be:
>>>>>
>>>>> rh4u564 -9.9 sec (-.017%)
>>>>> rh4u464 -7.3 sec (-.013%)
>>>>>
>>>>> There were no cliffs and the drift was linear.
>>>>>
>>>>> I think the problem you had may be due to the use of the
>>>>> pm timer. If you still have the boot log, it would tell you.
>>>>>
>>>>> When I first tried a guest on 3.2 with "clocksource=pit nohpet"
>>>>> I noticed that it picked the pm timer. Adding "nopmtimer", the
>>>>> guest will pick the pit.
>>>>>
>>>>> The reason I didn't have the problem with our 3.1 base is that
>>>>> I had disabled the hpet and the pmtimer by not advertising them
>>>>> in the acpi tables. I did this so long ago, I forgot that I had to
>>>>> disable pmtimer as well as hpet.
>>>>>
>>>>> So, can you re-run your test with "clocksource=pit nohpet       
>>>>
>>> nopmtimer"?
>>>   
>>>
>>>>> You should see this in the boot messages:
>>>>>
>>>>> time.c: Using PIT/TSC based timekeeping.
>>>>>
>>>>> Thanks,
>>>>> Dave
>>>>>
>>>>>
>>>>> Dave Winchell wrote:
>>>>>
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>> Over the weekend the drift was +18 seconds for each guest (no ntp).
>>>>>> The duration was 3900 minutes, so the error for each was +.0077%.
>>>>>> Looking back through the data, it appears to drift linearly at
>>>>>> this rate. I've attached a plot for rh4u5-64.
>>>>>>
>>>>>> This accuracy is better than what I've seen before (.03-.05%).
>>>>>> This may be due to the different load (ltp vs usex) or to         
>>>>>
>>> one of the
>>>   
>>>
>>>>>> changes I've made recently. I'll do some experimentation to see if
>>>>>> there is
>>>>>> a fix I should propose.
>>>>>>
>>>>>> This still doesn't address the radical drift you saw.
>>>>>> The next step for me is to run 3.2 and see if I can reproduce it.
>>>>>>
>>>>>> Regards,
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dave Winchell wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> Sorry it took me so long, but I finally ran an ltp test today.
>>>>>>> Its on rh4u4-64. I'm using the defaults for ltp and using a script
>>>>>>> called runltp. I had a usex load on rh4u5-64. No ntpd.
>>>>>>> virtual processors / physical processors = 2.
>>>>>>>
>>>>>>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
>>>>>>> for -.005% and .008%.
>>>>>>>
>>>>>>> I'm running a 3.1 based hypervisor with some clock related
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>
>>>>> tweaks that
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>> I haven't submitted, because I'm still characterizing them.
>>>>>>>
>>>>>>> I'm stopping the usex load on 4u5-64 now and replacing it with ltp
>>>>>>> and will leave the two guests running ltp over the weekend.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>> Dave Winchell wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> Hi Dan, Deepak:
>>>>>>>>
>>>>>>>> Thanks for the data. Those drifts are severe - no wonder
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>> ntp couldn't
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>> keep then in synch. I'll try to reproduce that behaviour
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>> here, with
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>> my code base.
>>>>>>>> If I can't reproduce it, I'll try 3.2.
>>>>>>>>
>>>>>>>> If you can isolate what ltp is doing during the cliffs, 
>>>>>>>>             
>>>>>>>
>>> that would
>>>   
>>>
>>>>>>>> be very
>>>>>>>> helpful.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>> OK, Deepak repeated the test without ntpd and using
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>
>>>>> ntpdate -b before
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>> the test.
>>>>>>>>>
>>>>>>>>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>>>>>>>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>>>>>>
>>>>>>>>> We will continue to look at LTP to try to isolate.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>> P.S. elXuY is essentially RHEL XuY with some patches.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>>>>>>>>> To: Deepak Patel
>>>>>>>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>>>>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> Dave Winchell
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>>>>>>>> pending
>>>>>>>>>> missed ticks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dan, Deeepak,
>>>>>>>>>>
>>>>>>>>>> It may be that the underlying clock error is too great for ntp
>>>>>>>>>> to handle. It would be useful if you did not run ntpd
>>>>>>>>>> and, instead did ntpdate -b <timeserver> at the start
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> of the test
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> for each guest. Then capture the data as you have been doing.
>>>>>>>>>> If the drift is greater than .05%, then we need to 
>>>>>>>>>>                 
>>>>>>>>>
>>> address that.
>>>   
>>>
>>>>>>>>>> Another option is, when running ntpd, to enable loop
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> statistics in
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> /etc/ntp.conf
>>>>>>>>>> by adding this to the file:
>>>>>>>>>>
>>>>>>>>>> statistics loopstats
>>>>>>>>>> statsdir /var/lib/ntp/
>>>>>>>>>>
>>>>>>>>>> Then you will see loop data in that directory.
>>>>>>>>>> Correlating the data in the loopstats files with the
>>>>>>>>>> peaks in skew would be interesting. You will see
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> entries of the form
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> 239.735511 10
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> Where the second to last column is the Allan Deviation.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> When that
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> gets over 1000, ntpd is working pretty hard. However, 
>>>>>>>>>>                 
>>>>>>>>>
>>> I have not
>>>   
>>>
>>>>>>>>>> seen ntpd
>>>>>>>>>> completely loose it like you have.
>>>>>>>>>>
>>>>>>>>>> I'm on vacation until Monday, and won't be reading
>>>>>>>>>> email.
>>>>>>>>>>
>>>>>>>>>> Thanks for all your work on this!
>>>>>>>>>>
>>>>>>>>>> -Dave
>>>>>>>>>>
>>>>>>>>>> Deepak Patel wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> I do not know which graph was attached with this. But
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>> I saw this
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>> guests when I
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>> was running ltp tests continuously.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> What was the behaviour of the other guests running?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> All pvm guests are fine. But behavior of most of the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>> hvm guests were
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>> as described.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> If they had spikes, were they at the same wall time?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> No. They are not at the same wall time.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> Were the other guests running ltp as well?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>>>>>>>>> continuously.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> How are you measuring skew?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> I was collecting output of "ntpdate -q  <timeserver> every
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>>>>>>> 300 seconds
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>> (5 minutes) and have created graph based on that.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> Are you running ntpd?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> Yes. ntp was running on all the guests.
>>>>>>>>>>>
>>>>>>>>>>> I am investigating what causes this spikes and let everyone
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>>>>>>> know what
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>> are my findings.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Deepak
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> Anything that you can discover that would be in sync with
>>>>>>>>>>>> the spikes would be very helpful!
>>>>>>>>>>>>
>>>>>>>>>>>> The code that I test with is our product code, which is based
>>>>>>>>>>>> on 3.1. So it is possible that something in 3.2 other
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>> than vpt.c
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>>>>>>>>
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> Dave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Turning off vhpet certainly helps a lot (though see below).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I wonder if timekeeping with vhpet is so bad that it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>> should be
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>> turned off by default (in 3.1, 3.2, and unstable) 
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>> until it is
>>>   
>>>
>>>>>>>>>>>>> fixed?  (I have a patch that defaults it off, can post it if
>>>>>>>>>>>>> there is agreement on the above point.)  The whole
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>> point of an
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>> HPET is to provide more precise timekeeping and if vhpet is
>>>>>>>>>>>>> worse than vpit, it can only confuse users.  Comments?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In your testing, are you just measuring % skew over a long
>>>>>>>>>>>>> period of time?
>>>>>>>>>>>>> We are graphing the skew continuously and
>>>>>>>>>>>>> seeing periodic behavior that is unsettling, even with pit.
>>>>>>>>>>>>> See attached.  Though your algorithm recovers, the "cliffs"
>>>>>>>>>>>>> could still cause real user problems.  I wonder if there is
>>>>>>>>>>>>> anything that can be done to make the "recovery" more
>>>>>>>>>>>>> responsive?
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are looking into what part(s) of LTP is causing
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>> the cliffs.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>>>>>>>> deepak.patel@oracle.com;
>>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>> that disables
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> pending
>>>>>>>>>>>>>> missed ticks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I guess I'm a bit out of date calling for clock= usage.
>>>>>>>>>>>>>> Looking at linux 2.6.20.4 sources, I think you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>> should specify
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can leave the xen and dom0 bootlines as they are.
>>>>>>>>>>>>>> The xen and guest clocksources do not need to be the same.
>>>>>>>>>>>>>> In my tests, xen is using the hpet for its timekeeping and
>>>>>>>>>>>>>> that appears to be the default.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When you boot the guests you should see
>>>>>>>>>>>>>> time.c: Using PIT/TSC based timekeeping.
>>>>>>>>>>>>>> on the rh4u5-64 guest, and something similar on the others.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This appears to be the xen state, which is fine.
>>>>>>>>>>>>>> I was wrongly assuming that this was the guest state.
>>>>>>>>>>>>>> You might want to look in your guest logs and see
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>> what they were
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> picking
>>>>>>>>>>>>>> for a clock source.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, I hadn't realized that!  No wonder we didn't
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> see the same
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> improvement you saw!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Try specifying clock=pit on the linux boot line...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> command line or
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the 
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>> guest (or
>>>   
>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dom0?) command
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> line?  Or both places?  Since the tests take awhile, it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> would be nice
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to get this right the first time.  Do the Xen and guest
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> clocksources need
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to be the same?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> deepak.patel@oracle.com;
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> that disables
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> pending missed ticks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hpet timer does have a fairly large error, as I
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> trying this
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> one recently.
>>>>>>>>>>>>>>> I don't remember what I got for error, but 1% sounds
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> about right.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the module
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Keir and I did
>>>>>>>>>>>>>>> all the recent work in, for its periodic timer 
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>> needs. Try
>>>   
>>>
>>>>>>>>>>>>>>> specifying clock=pit
>>>>>>>>>>>>>>> on the linux boot line. If it still picks the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> hpet, which it
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> might, let me know
>>>>>>>>>>>>>>> and I'll tell you how to get around this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>> --------------------------------------------------------------
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> ----------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *From:* Dan Magenheimer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> [mailto:dan.magenheimer@oracle.com]
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser
>>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> deepak.patel@oracle.com;
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> akira.ijuin@oracle.com
>>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that disables
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pending missed ticks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry for the very late followup on this but we finally
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> were able
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to get our testing set up again on stable 3.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> bits and have
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> order of 1%.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> Test enviroment was a 4-socket dual core machine
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> with 24GB of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> plus two pv.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> All six guests were running LTP simultaneously.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> The four hvm
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> RHEL4u5-64.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 32-bit guests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All four hvm guests experienced skew around -1%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> even the 32-bit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> guest.  Less intensive testing didn't exhibit much
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> skew at all.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> A representative graph is attached.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Dave, I wonder if some portion of your patches
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> didn't end up in
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> the xen trees?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> Platform timer
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > -----Original Message-----
>>>>>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > Dave Winchell
>>>>>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>>>>>> > To: Keir Fraser
>>>>>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > Winchell
>>>>>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>>>>>> > disables pending
>>>>>>>>>>>>>>> > missed ticks
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Hi Keir,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > The latest change, c/s 16690, looks fine.
>>>>>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to
>>>>>>>>>>>>>>> > the code I submitted. Also, your version is more
>>>>>>>>>>>>>>> > concise.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > The error tests confirm the equivalence. With
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> overnight cpu loads,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > the checked in version was accurate to +.048% for sles
>>>>>>>>>>>>>>> > and +.038% for red hat. My version was +.046%
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> +.032% in a
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > 2 hour test.
>>>>>>>>>>>>>>> > I don't think the difference is significant.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > i/o loads produced errors of +.01%.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Thanks for all your efforts on this issue.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Regards,
>>>>>>>>>>>>>>> > Dave
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Keir Fraser wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > >Applied as c/s 16690, although the 
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>> checked-in patch is
>>>   
>>>
>>>>>>>>>>>>>>> > smaller. I think the
>>>>>>>>>>>>>>> > >only important fix is to pt_intr_post() and the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> only bit of
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > the patch I
>>>>>>>>>>>>>>> > >totally omitted was the change to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> pt_process_missed_ticks().
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > I don't think
>>>>>>>>>>>>>>> > >that change can be important, but let's see what
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> happens to the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>> > >percentage...
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > -- Keir
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >>Hi Dan and Keir,
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>Attached is a patch that fixes some issues with the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SYNC policy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>(no_missed_ticks_pending).
>>>>>>>>>>>>>>> > >>I have not tried to make the change the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> minimal one, but,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > rather, just
>>>>>>>>>>>>>>> > >>ported into
>>>>>>>>>>>>>>> > >>the new code what I know to work well. The error for
>>>>>>>>>>>>>>> > >>no_missed_ticks_pending goes from
>>>>>>>>>>>>>>> > >>over 3% to .03% with this change according
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> to my testing.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>Regards,
>>>>>>>>>>>>>>> > >>Dave
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>Dan Magenheimer wrote:
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>>Hi Dave --
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>Did you get your correction ported?  If so,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> it would be
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > nice to see this get
>>>>>>>>>>>>>>> > >>>into 3.1.3.
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>Note that I just did some very limited testing with
>>>>>>>>>>>>>>> > timer_mode=2(=SYNC=no
>>>>>>>>>>>>>>> > >>>missed ticks pending)
>>>>>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> guest) and the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > worst error I've
>>>>>>>>>>>>>>> > >>>seen so far
>>>>>>>>>>>>>>> > >>>is 0.012%.  But I haven't tried any exotic
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> loads, just LTP.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>Thanks,
>>>>>>>>>>>>>>> > >>>Dan
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>>-----Original Message-----
>>>>>>>>>>>>>>> > >>>>From: Dave Winchell
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> [mailto:dwinchell@virtualiron.com]
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> timer mode that
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>disables pending
>>>>>>>>>>>>>>> > >>>>missed ticks
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>Dan,
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SYNC method
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>(now called
>>>>>>>>>>>>>>> > >>>>no_missed_ticks_pending)
>>>>>>>>>>>>>>> > >>>>and found the error to be very high, much larger
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> than 1 %, as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>I recall.
>>>>>>>>>>>>>>> > >>>>I have not had a chance to submit a correction. I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> will try to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>do it later
>>>>>>>>>>>>>>> > >>>>this week or the first week in January. My
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> version of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> constant tsc
>>>>>>>>>>>>>>> > >>>>offset SYNC method
>>>>>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> that into the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>current code.
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>The error you got for both of those kernels is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> what I would
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode.
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>Regards,
>>>>>>>>>>>>>>> > >>>>Dave
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> saw a loss of
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>about 0.2% with no load.  This was
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> xen-unstable tip today
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>with no options specified.  32-bit was 
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>> about 0.01%.
>>>   
>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>>I think I missed something... how do I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> run the various
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>accounting choices and which ones are known to be
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>for which kernels?
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>>Thanks,
>>>>>>>>>>>>>>> > >>>>>Dan
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>>-----Original Message-----
>>>>>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>>>>>>>                                     
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                   
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                 
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>Keir Fraser
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>>>> > >>>>>>To: Dave Winchell
>>>>>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > Eddie; Jiang,
>>>>>>>>>>>>>>> > >>>>>>Yunhong
>>>>>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> mode that
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>disables pending
>>>>>>>>>>>>>>> > >>>>>>missed ticks
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> changeset 16545.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>-- Keir
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Keir,
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> loads for the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>various time protocols follows. In
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> addition, the data
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>for cpu loads is shown.
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> processor AMD
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> box.
>>>>>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> vcpu each.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>>>> > >>>>>>>(usex is available at
>>>>>>>>>>>>>>> http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> of=/dev/null.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>> instances of dd.
>>>   
>>>
>>>>>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> as i/o-32
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> instances of dd.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>>>> +4.42
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> sec -.006%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > +.005% cpu
>>>>>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> sec, -.001%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > +.012% cpu
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec, -.009%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> -.004% cpu
>>>>>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.005%, -.005% cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.008%, -.003% cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.040% cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> sec, -.034%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > -.031% cpu
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec,-55.7 sec, -.01%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.09% i/o-8
>>>>>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec,-14.0 sec, -.015%
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.14% i/o-8
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec, -.017%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.022% i/o-8
>>>>>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.017%, -.018% i/o-8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec, -.020%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.029% i/o-8
>>>>>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> sec, -.023%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > -.031% i/o-8
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.04% i/o-32
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec, -.011%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.005% i/o-32
>>>>>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.11% i/o-32
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> 13. sec -.07%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > .003% i/o-4/32
>>>>>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> sec, -.017%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > .01% i/o-4/32
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Overhead measurements:
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> through a fixed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>system workload
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>>>>>> > >>>>>>>The workload was usex -b48.
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Conclusions:
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>>>>>> > requirement for ntp
>>>>>>>>>>>>>>> > >>>>>>>tracking under the loads
>>>>>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> accuracies for
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>SYNC, MIXED,
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>and ASYNC
>>>>>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> method by only
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>scheduling the extra
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>wakeups if a certain number
>>>>>>>>>>>>>>> > >>>>>>>of ticks are missed.
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Regards,
>>>>>>>>>>>>>>> > >>>>>>>Dave
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ASYNC method a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>couple of days ago,
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> there may have
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > been something
>>>>>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> days ago for
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>ASYNC. It may have been
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> after context
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>switch in.
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> or ASYNC give
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>acceptable accuracy,
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>each running consistently around or under
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> .01%. MIXED has
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>a fairly high
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>error of
>>>>>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> to .05% ntp
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>threshold for comfort.
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> plan to leave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>SYNC running
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> leave MIXED
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>running instead.
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can run
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>more overnight tests
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>next week.
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> effects of the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>SYNC+run_timer
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cause higher
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>system-wide CPU
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> through the
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> accurate than
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> approaches,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>and each interrupt
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> favourites and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>if the latter is
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>actually more accurate then I can
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> simply revert the
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > changeset that
>>>>>>>>>>>>>>> > >>>>>>>>implemented MIXED.
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> workloads you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>could try idle
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> large disc reads
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>to /dev/null)? We
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CPU bound, so
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>that's really an
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>-- Keir
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>>>> > >>>>>>_______________________________________________
>>>>>>>>>>>>>>> > >>>>>>Xen-devel mailing list
>>>>>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> 2007 +0000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> 2008 -0500
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> pt_process_missed_ticks(stru
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>     missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pt->period + 1;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>     if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> no_missed_ticks_pending) )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>>>> > >>+        pt->do_not_freeze = 1;
>>>>>>>>>>>>>>> > >>     else
>>>>>>>>>>>>>>> > >>         pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>>>> > >>     pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> pt_timer_fn(void *data)
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>     pt_lock(pt);
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>-    pt->pending_intr_nr++;
>>>>>>>>>>>>>>> > >>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>+        pt->pending_intr_nr = 1;
>>>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 0;
>>>>>>>>>>>>>>> > >>+    }
>>>>>>>>>>>>>>> > >>+    else
>>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr++;
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>     if ( !pt->one_shot )
>>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> vcpu *v, struct
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>         return;
>>>>>>>>>>>>>>> > >>     }
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>-    pt->do_not_freeze = 0;
>>>>>>>>>>>>>>> > >>-
>>>>>>>>>>>>>>> > >>     if ( pt->one_shot )
>>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>>> > >>         pt->enabled = 0;
>>>>>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *v, struct
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>             pt->last_plt_gtime =
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> hvm_get_guest_time(v);
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>             pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>> 'collapse' all
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > missed ticks */
>>>>>>>>>>>>>>> > >>         }
>>>>>>>>>>>>>>> > >>+ else if ( mode_is(v->domain,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>+     pt->pending_intr_nr--;
>>>>>>>>>>>>>>> > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>>>> > >>+ }
>>>>>>>>>>>>>>> > >>         else
>>>>>>>>>>>>>>> > >>         {
>>>>>>>>>>>>>>> > >>             pt->last_plt_gtime += 
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>> pt->period_cycles;
>>>   
>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>> > Xen-devel mailing list
>>>>>>>>>>>>>>> > Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>> > http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                           
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>>>
>>>>>>         
>>>>>
>>>>> --------------------------------------------------------------
>>>>> ----------
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>       
>>>>
>>>>
>>>>     
>>>
>>>   
>>
>>
>>  
>>
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Hi Dave --

Thanks for that observation on ltp running on one vcpu!

With "clocksource=pit nohpet nopmtimer" our clock skew
problems seem to have been reduced to a reasonable
percentage.  However, our 32-bit clock skew seems to
show a measureable problem now.  As a result,
I've been doing some digging into kernel sources and
have observed the following relative to RHEL4 (2.6.9-based)
kernels and RHEL5 (2.6.18-based) kernels and thought I
would document them for posterity.  Some of
our confusion arises from the fact that invalid command
line parameters are silently ignored.

RHEL4:
- clock= is a valid parameter for RHEL4-32
- clocksource= is not a valid parameter for RHEL4-xx
- nohpet is a valid parameter for RHEL4-64, not RHEL4-32
- nopmtimer is not a valid parameter for RHEL4-xx
- notsc is a valid parameter for RHEL4-32, not RHEL4-64
- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently

For Xen RHEL4 HVM guests:
- I *think* clock=pit is sufficient for RHEL4-32 [1]
- I *think* nohpet is sufficient for RHEL4-64 [1]

RHEL5:
- there are two kinds of timekeeping, WALL and gtod
- clocksource= is a valid parameter for RHEL5-xx
- clock= is a valid but deprecated parameter for RHEL5-xx
- clock= and clocksource= are essentially equivalent
- nohpet is a valid parameter for RHEL5-64, not RHEL5-32
- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32
- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1]
- clock=pit changes the gtod source but not the WALL source[2]
- nohpet nopmtimer changes the WALL source to PIT
- /sys/devices/system/clocksource/clocksource0/...
  available_clocksource lists the possible clock sources
  current_clocksource lists the chosen clock source
  ..but neither of these works in a RHEL5 guest!

For Xen RHEL5 HVM guests:
- I *think* clock=pit is sufficient for RHEL5-32
- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64

Other info:
- As of 2.6.24.2, clock= is still valid (though still deprecated)

So, some open questions:
[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
    (I *think* not as it has never come up in any email.)
[2] In RHEL5, I *think* it is the WALL source that we care about?

And finally, since invalid command line parameters are ignored.
I think specifying:
	clock=pit nohpet nopmtimer
will force the guest clock sources into the optimal state for
all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the
question above on tsc).  And we should keep an eye on
kernel/time/clocksource.c to ensure the __setup("clock="...)
line doesn't go away before RHEL6.

Note that if hpet=0 and pmtimer=0 were the default hvm platform
parameters for all xen hvm guests (on all versions of xen),
specifying kernel command line parameters would be unnecessary,
but c'est la vie.

Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests
and timer_mode=2 for 64-bit RHEL guests.

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Tuesday, February 19, 2008 8:27 AM
> To: dan.magenheimer@oracle.com
> Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
> Patel
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> 
> Hi Dan,
> 
> ltp runs by default loading up only one vcpu.
> The -x option can be used to run multiple instances, though
> in this mode you will get test failures.
> I ran 8 instances on each guest for 16 hours, 25 min
> and the time error was -11 sec (-.019%) on each guest.
> 
> Regards,
> Dave
> 
> Dave Winchell wrote:
> 
> > Hi Dan,
> >
> > Mine was oversubscribed.
> > 8 physical cpu, 2 guests, each with 8 vcpu.
> > I ran one instance of ltp on each guest, continuously. I hope ltp
> > loaded up all the vcpus. I seem to recall that it did, but I
> > could be wrong. If it didn't, that would be a major difference
> > between our tests. I'll verify this afternoon and run 
> multiple instances,
> > if necessary.
> >
> > Thanks,
> > Dave
> >
> >
> >
> > Dan Magenheimer wrote:
> >
> >> Hi Dave --
> >>
> >> No new results yet but one other question:
> >>
> >> The problems we've seen with our testing have been with a heavily
> >> oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
> >> running LTP simultaneously.
> >>
> >> Was your LTP testing oversubscribed or just a single guest?
> >>
> >> Thanks,
> >> Dan
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>> Sent: Thursday, February 14, 2008 10:56 AM
> >>> To: dan.magenheimer@oracle.com
> >>> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
> >>> Winchell
> >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> >>> missed ticks
> >>>
> >>>
> >>> Dan,
> >>>
> >>> Here are some boot snipets for rh4u564 on xen 3.2.
> >>>
> >>>
> >>> #1:
> >>>
> >>> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
> >>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
> >>> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
> >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
> 3.4.6 20060404
> >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>> ...
> >>> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
> >>> console=ttyS0 clocksource=pit nohpet
> >>> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
> >>> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 
> 2048 (order: 11,
> >>> 65536 bytes)
> >>> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
> >>> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 
> MHz processor.
> >>> ...
> >>> Feb 14 10:45:00 vs076 kernel: checking TSC 
> synchronization across 8
> >>> CPUs: passed.
> >>> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
> >>> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to 
> use of PM timer
> >>> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
> >>>
> >>>
> >>>
> >>> #2:
> >>>
> >>> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
> >>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
> >>> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
> >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
> 3.4.6 20060404
> >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>> ...
> >>> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
> >>> console=ttyS0 clocksource=pit nohpet nopmtimer
> >>> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
> >>> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 
> 2048 (order: 11,
> >>> 65536 bytes)
> >>> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz 
> PIT timer.
> >>> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 
> MHz processor.
> >>> ...
> >>> Feb 14 10:47:59 vs076 kernel: checking TSC 
> synchronization across 8
> >>> CPUs: passed.
> >>> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
> >>> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based 
> timekeeping.
> >>>
> >>>
> >>> As you can see, I only get the pit if I specify nopmtimer.
> >>>
> >>> Dan Magenheimer wrote:
> >>>
> >>>
> >>>
> >>>> Hi Dave --
> >>>>
> >>>> Thanks for continuing to run tests!
> >>>>
> >>>> Hmmm... I thought I had noticed that even though Linux will
> >>>
> >>> acknowledge
> >>>
> >>>
> >>>> the existence of the pmtimer, it still prints:
> >>>>
> >>>> time.c: Using PIT/TSC based timekeeping.
> >>>>
> >>>> I will check again, but assuming the clocksource for our tests is
> >>>> indeed pit, the huge difference in the results (yours vs ours) is
> >>>> baffling. I wonder if the difference may be the 
> underlying hardware.
> >>>> Maybe we will try to ensure we can duplicate the results on
> >>>
> >>> a different
> >>>
> >>>
> >>>> box.
> >>>>
> >>>>
> >>>> So your testing was with stock 3.2.0 xen bits (what 
> cset?) without
> >>>> any of your [quote from below] "clock related tweaks 
> that I haven't
> >>>> submitted, because I'm still characterizing them"?
> >>>>
> >>>>
> >>>>
> >>>
> >>> None of the tweaks I mentioned are in this test.
> >>> It was stock with some patches. However, none of the 
> patches are time
> >>> related to
> >>> my knowledge and I checked vpt.c to make sure that it is 
> the same as
> >>> what's in unstable.
> >>> The only difference is in pt_intr_post, where I set the 
> timer mode.
> >>> I don't have timer mode tied into our config process yet, which
> >>> is different than official xen method.
> >>>
> >>>
> >>> (In pt_intr_post)
> >>>      else
> >>>      {
> >>> +       if(v->arch.paging.mode->guest_levels == 4)
> >>> +           
> v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
> >>> HVMPTM_no_missed_ticks_pending;
> >>> +       else
> >>> +           
> v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
> >>> HVMPTM_delay_for_missed_ticks;
> >>>          if ( mode_is(v->domain, one_missed_tick_pending) ||
> >>>               mode_is(v->domain, no_missed_ticks_pending) )
> >>>          {
> >>>
> >>>
> >>>
> >>>> Could you also send detail on the rhel4u4-64 kernel you
> >>>> are testing with, just to ensure we are not comparing apples
> >>>> and oranges?  (Perhaps there's some way we can even share the
> >>>> identical disk image and vm.cfg file?)
> >>>>
> >>>> And if our problem is indeed the pmtimer, I will need to submit
> >>>> another patch to Keir to add an hvm pmtimer platform variable.
> >>>> (Hmmm... I don't think he's even accepted the hpet variable patch
> >>>> yet.  I'll have to check.)
> >>>>
> >>>> Thanks,
> >>>> Dan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>> Sent: Thursday, February 14, 2008 9:00 AM
> >>>>> To: dan.magenheimer@oracle.com
> >>>>> Cc: Dave Winchell; Keir Fraser;
> >>>>
> >>> xen-devel@lists.xensource.com; Deepak
> >>>
> >>>
> >>>>> Patel
> >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>> disables pending
> >>>>> missed ticks
> >>>>>
> >>>>>
> >>>>> Hi Dan,
> >>>>>
> >>>>> I ran the ltp tests with 3.2 and found the errors
> >>>>> for a 16 hour run to be:
> >>>>>
> >>>>> rh4u564 -9.9 sec (-.017%)
> >>>>> rh4u464 -7.3 sec (-.013%)
> >>>>>
> >>>>> There were no cliffs and the drift was linear.
> >>>>>
> >>>>> I think the problem you had may be due to the use of the
> >>>>> pm timer. If you still have the boot log, it would tell you.
> >>>>>
> >>>>> When I first tried a guest on 3.2 with "clocksource=pit nohpet"
> >>>>> I noticed that it picked the pm timer. Adding "nopmtimer", the
> >>>>> guest will pick the pit.
> >>>>>
> >>>>> The reason I didn't have the problem with our 3.1 base is that
> >>>>> I had disabled the hpet and the pmtimer by not advertising them
> >>>>> in the acpi tables. I did this so long ago, I forgot 
> that I had to
> >>>>> disable pmtimer as well as hpet.
> >>>>>
> >>>>> So, can you re-run your test with "clocksource=pit nohpet
> >>>>
> >>> nopmtimer"?
> >>>
> >>>
> >>>>> You should see this in the boot messages:
> >>>>>
> >>>>> time.c: Using PIT/TSC based timekeeping.
> >>>>>
> >>>>> Thanks,
> >>>>> Dave
> >>>>>
> >>>>>
> >>>>> Dave Winchell wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Hi Dan,
> >>>>>>
> >>>>>> Over the weekend the drift was +18 seconds for each 
> guest (no ntp).
> >>>>>> The duration was 3900 minutes, so the error for each 
> was +.0077%.
> >>>>>> Looking back through the data, it appears to drift linearly at
> >>>>>> this rate. I've attached a plot for rh4u5-64.
> >>>>>>
> >>>>>> This accuracy is better than what I've seen before (.03-.05%).
> >>>>>> This may be due to the different load (ltp vs usex) or to
> >>>>>
> >>> one of the
> >>>
> >>>
> >>>>>> changes I've made recently. I'll do some 
> experimentation to see if
> >>>>>> there is
> >>>>>> a fix I should propose.
> >>>>>>
> >>>>>> This still doesn't address the radical drift you saw.
> >>>>>> The next step for me is to run 3.2 and see if I can 
> reproduce it.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Dave
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Dave Winchell wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Hi Dan,
> >>>>>>>
> >>>>>>> Sorry it took me so long, but I finally ran an ltp test today.
> >>>>>>> Its on rh4u4-64. I'm using the defaults for ltp and 
> using a script
> >>>>>>> called runltp. I had a usex load on rh4u5-64. No ntpd.
> >>>>>>> virtual processors / physical processors = 2.
> >>>>>>>
> >>>>>>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 
> 300 minutes
> >>>>>>> for -.005% and .008%.
> >>>>>>>
> >>>>>>> I'm running a 3.1 based hypervisor with some clock related
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>> tweaks that
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> I haven't submitted, because I'm still characterizing them.
> >>>>>>>
> >>>>>>> I'm stopping the usex load on 4u5-64 now and 
> replacing it with ltp
> >>>>>>> and will leave the two guests running ltp over the weekend.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Dave
> >>>>>>>
> >>>>>>>
> >>>>>>> Dave Winchell wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hi Dan, Deepak:
> >>>>>>>>
> >>>>>>>> Thanks for the data. Those drifts are severe - no wonder
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>> ntp couldn't
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>> keep then in synch. I'll try to reproduce that behaviour
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>> here, with
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>> my code base.
> >>>>>>>> If I can't reproduce it, I'll try 3.2.
> >>>>>>>>
> >>>>>>>> If you can isolate what ltp is doing during the cliffs,
> >>>>>>>>
> >>>>>>>
> >>> that would
> >>>
> >>>
> >>>>>>>> be very
> >>>>>>>> helpful.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Dave
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> OK, Deepak repeated the test without ntpd and using
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>> ntpdate -b before
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>> the test.
> >>>>>>>>>
> >>>>>>>>> The attached graph shows his results: el5u1-64 
> (best=~0.07%),
> >>>>>>>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
> >>>>>>>>>
> >>>>>>>>> We will continue to look at LTP to try to isolate.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Dan
> >>>>>>>>>
> >>>>>>>>> P.S. elXuY is essentially RHEL XuY with some patches.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>> Sent: Wednesday, January 30, 2008 2:45 PM
> >>>>>>>>>> To: Deepak Patel
> >>>>>>>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
> >>>>>>>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> Dave Winchell
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode 
> that disables
> >>>>>>>>>> pending
> >>>>>>>>>> missed ticks
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Dan, Deeepak,
> >>>>>>>>>>
> >>>>>>>>>> It may be that the underlying clock error is too 
> great for ntp
> >>>>>>>>>> to handle. It would be useful if you did not run ntpd
> >>>>>>>>>> and, instead did ntpdate -b <timeserver> at the start
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> of the test
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> for each guest. Then capture the data as you have 
> been doing.
> >>>>>>>>>> If the drift is greater than .05%, then we need to
> >>>>>>>>>>
> >>>>>>>>>
> >>> address that.
> >>>
> >>>
> >>>>>>>>>> Another option is, when running ntpd, to enable loop
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> statistics in
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> /etc/ntp.conf
> >>>>>>>>>> by adding this to the file:
> >>>>>>>>>>
> >>>>>>>>>> statistics loopstats
> >>>>>>>>>> statsdir /var/lib/ntp/
> >>>>>>>>>>
> >>>>>>>>>> Then you will see loop data in that directory.
> >>>>>>>>>> Correlating the data in the loopstats files with the
> >>>>>>>>>> peaks in skew would be interesting. You will see
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> entries of the form
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> 239.735511 10
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> Where the second to last column is the Allan Deviation.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> When that
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> gets over 1000, ntpd is working pretty hard. However,
> >>>>>>>>>>
> >>>>>>>>>
> >>> I have not
> >>>
> >>>
> >>>>>>>>>> seen ntpd
> >>>>>>>>>> completely loose it like you have.
> >>>>>>>>>>
> >>>>>>>>>> I'm on vacation until Monday, and won't be reading
> >>>>>>>>>> email.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for all your work on this!
> >>>>>>>>>>
> >>>>>>>>>> -Dave
> >>>>>>>>>>
> >>>>>>>>>> Deepak Patel wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> Is the graph for RHEL5u1-64? (I've never tested 
> this one.)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I do not know which graph was attached with this. But
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>> I saw this
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>> guests when I
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>> was running ltp tests continuously.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> What was the behaviour of the other guests running?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> All pvm guests are fine. But behavior of most of the
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>> hvm guests were
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>> as described.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> If they had spikes, were they at the same wall time?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> No. They are not at the same wall time.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Were the other guests running ltp as well?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are 
> running ltp
> >>>>>>>>>>> continuously.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> How are you measuring skew?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I was collecting output of "ntpdate -q  <timeserver> every
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 300 seconds
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> (5 minutes) and have created graph based on that.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Are you running ntpd?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Yes. ntp was running on all the guests.
> >>>>>>>>>>>
> >>>>>>>>>>> I am investigating what causes this spikes and 
> let everyone
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> know what
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> are my findings.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Deepak
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Anything that you can discover that would be in sync with
> >>>>>>>>>>>> the spikes would be very helpful!
> >>>>>>>>>>>>
> >>>>>>>>>>>> The code that I test with is our product code, 
> which is based
> >>>>>>>>>>>> on 3.1. So it is possible that something in 3.2 other
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>> than vpt.c
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>> is the cause. I can test with 3.2, if necessary.
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks,
> >>>>>>>>>>>> Dave
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Dave (Keir, see suggestion below) --
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Turning off vhpet certainly helps a lot (though 
> see below).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wonder if timekeeping with vhpet is so bad that it
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> should be
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>> turned off by default (in 3.1, 3.2, and unstable)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>> until it is
> >>>
> >>>
> >>>>>>>>>>>>> fixed?  (I have a patch that defaults it off, 
> can post it if
> >>>>>>>>>>>>> there is agreement on the above point.)  The whole
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> point of an
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>> HPET is to provide more precise timekeeping and 
> if vhpet is
> >>>>>>>>>>>>> worse than vpit, it can only confuse users.  Comments?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In your testing, are you just measuring % skew 
> over a long
> >>>>>>>>>>>>> period of time?
> >>>>>>>>>>>>> We are graphing the skew continuously and
> >>>>>>>>>>>>> seeing periodic behavior that is unsettling, 
> even with pit.
> >>>>>>>>>>>>> See attached.  Though your algorithm recovers, 
> the "cliffs"
> >>>>>>>>>>>>> could still cause real user problems.  I wonder 
> if there is
> >>>>>>>>>>>>> anything that can be done to make the "recovery" more
> >>>>>>>>>>>>> responsive?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We are looking into what part(s) of LTP is causing
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> the cliffs.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Dan
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
> >>>>>>>>>>>>>> To: dan.magenheimer@oracle.com
> >>>>>>>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>> deepak.patel@oracle.com;
> >>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> that disables
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> pending
> >>>>>>>>>>>>>> missed ticks
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dan,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I guess I'm a bit out of date calling for clock= usage.
> >>>>>>>>>>>>>> Looking at linux 2.6.20.4 sources, I think you
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> should specify
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> You can leave the xen and dom0 bootlines as they are.
> >>>>>>>>>>>>>> The xen and guest clocksources do not need to 
> be the same.
> >>>>>>>>>>>>>> In my tests, xen is using the hpet for its 
> timekeeping and
> >>>>>>>>>>>>>> that appears to be the default.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When you boot the guests you should see
> >>>>>>>>>>>>>> time.c: Using PIT/TSC based timekeeping.
> >>>>>>>>>>>>>> on the rh4u5-64 guest, and something similar 
> on the others.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, 
> Platform timer
> >>>>>>>>>>>>>>> 14.318MHz HPET.)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This appears to be the xen state, which is fine.
> >>>>>>>>>>>>>> I was wrongly assuming that this was the guest state.
> >>>>>>>>>>>>>> You might want to look in your guest logs and see
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> what they were
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> picking
> >>>>>>>>>>>>>> for a clock source.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, I hadn't realized that!  No wonder we didn't
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> see the same
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> improvement you saw!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Try specifying clock=pit on the linux boot line...
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'm confused... do you mean "clocksource=pit" 
> on the Xen
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> command line or
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> guest (or
> >>>
> >>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> dom0?) command
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> line?  Or both places?  Since the tests take 
> awhile, it
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> would be nice
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to get this right the first time.  Do the Xen 
> and guest
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> clocksources need
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to be the same?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Dan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>> *From:* Dave Winchell 
> [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
> >>>>>>>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> deepak.patel@oracle.com;
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> that disables
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> pending missed ticks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Dan,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hpet timer does have a fairly large error, as I
> >>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> trying this
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> one recently.
> >>>>>>>>>>>>>>> I don't remember what I got for error, but 1% sounds
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> about right.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> the module
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Keir and I did
> >>>>>>>>>>>>>>> all the recent work in, for its periodic timer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> needs. Try
> >>>
> >>>
> >>>>>>>>>>>>>>> specifying clock=pit
> >>>>>>>>>>>>>>> on the linux boot line. If it still picks the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> hpet, which it
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> might, let me know
> >>>>>>>>>>>>>>> and I'll tell you how to get around this.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>> Dave
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> --------------------------------------------------------------
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> ----------
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *From:* Dan Magenheimer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:dan.magenheimer@oracle.com]
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM
> >>>>>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser
> >>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> deepak.patel@oracle.com;
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> akira.ijuin@oracle.com
> >>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> that disables
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> pending missed ticks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sorry for the very late followup on this but 
> we finally
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> were able
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to get our testing set up again on stable 3.1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> bits and have
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> order of 1%.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> Test enviroment was a 4-socket dual core machine
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> with 24GB of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> plus two pv.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> All six guests were running LTP simultaneously.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> The four hvm
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> RHEL4u5-64.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 32-bit guests.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> All four hvm guests experienced skew around -1%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> even the 32-bit
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> guest.  Less intensive testing didn't exhibit much
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> skew at all.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> A representative graph is attached.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Dave, I wonder if some portion of your patches
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> didn't end up in
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> the xen trees?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> Platform timer
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> 14.318MHz HPET.)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Dan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for 
> running tests.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > -----Original Message-----
> >>>>>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > Dave Winchell
> >>>>>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM
> >>>>>>>>>>>>>>> > To: Keir Fraser
> >>>>>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> xen-devel@lists.xensource.com; Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > Winchell
> >>>>>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a 
> timer mode that
> >>>>>>>>>>>>>>> > disables pending
> >>>>>>>>>>>>>>> > missed ticks
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Hi Keir,
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > The latest change, c/s 16690, looks fine.
> >>>>>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to
> >>>>>>>>>>>>>>> > the code I submitted. Also, your version is more
> >>>>>>>>>>>>>>> > concise.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > The error tests confirm the equivalence. With
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> overnight cpu loads,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > the checked in version was accurate to 
> +.048% for sles
> >>>>>>>>>>>>>>> > and +.038% for red hat. My version was +.046%
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> +.032% in a
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > 2 hour test.
> >>>>>>>>>>>>>>> > I don't think the difference is significant.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > i/o loads produced errors of +.01%.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Thanks for all your efforts on this issue.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Regards,
> >>>>>>>>>>>>>>> > Dave
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Keir Fraser wrote:
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > >Applied as c/s 16690, although the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> checked-in patch is
> >>>
> >>>
> >>>>>>>>>>>>>>> > smaller. I think the
> >>>>>>>>>>>>>>> > >only important fix is to pt_intr_post() and the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> only bit of
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > the patch I
> >>>>>>>>>>>>>>> > >totally omitted was the change to
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> pt_process_missed_ticks().
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > I don't think
> >>>>>>>>>>>>>>> > >that change can be important, but let's see what
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> happens to the
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>> > >percentage...
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > > -- Keir
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell"
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> <dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >>Hi Dan and Keir,
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>Attached is a patch that fixes some 
> issues with the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> SYNC policy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>(no_missed_ticks_pending).
> >>>>>>>>>>>>>>> > >>I have not tried to make the change the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> minimal one, but,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > rather, just
> >>>>>>>>>>>>>>> > >>ported into
> >>>>>>>>>>>>>>> > >>the new code what I know to work well. 
> The error for
> >>>>>>>>>>>>>>> > >>no_missed_ticks_pending goes from
> >>>>>>>>>>>>>>> > >>over 3% to .03% with this change according
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> to my testing.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>Regards,
> >>>>>>>>>>>>>>> > >>Dave
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>>Hi Dave --
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>Did you get your correction ported?  If so,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> it would be
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > nice to see this get
> >>>>>>>>>>>>>>> > >>>into 3.1.3.
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>Note that I just did some very limited 
> testing with
> >>>>>>>>>>>>>>> > timer_mode=2(=SYNC=no
> >>>>>>>>>>>>>>> > >>>missed ticks pending)
> >>>>>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> guest) and the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > worst error I've
> >>>>>>>>>>>>>>> > >>>seen so far
> >>>>>>>>>>>>>>> > >>>is 0.012%.  But I haven't tried any exotic
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> loads, just LTP.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>Thanks,
> >>>>>>>>>>>>>>> > >>>Dan
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>>-----Original Message-----
> >>>>>>>>>>>>>>> > >>>>From: Dave Winchell
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:dwinchell@virtualiron.com]
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> timer mode that
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>disables pending
> >>>>>>>>>>>>>>> > >>>>missed ticks
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>Dan,
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> SYNC method
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>(now called
> >>>>>>>>>>>>>>> > >>>>no_missed_ticks_pending)
> >>>>>>>>>>>>>>> > >>>>and found the error to be very high, much larger
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> than 1 %, as
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>I recall.
> >>>>>>>>>>>>>>> > >>>>I have not had a chance to submit a 
> correction. I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> will try to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>do it later
> >>>>>>>>>>>>>>> > >>>>this week or the first week in January. My
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> version of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> constant tsc
> >>>>>>>>>>>>>>> > >>>>offset SYNC method
> >>>>>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> that into the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>current code.
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>The error you got for both of those kernels is
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> what I would
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> expect
> >>>>>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks.
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the 
> time mode.
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>Regards,
> >>>>>>>>>>>>>>> > >>>>Dave
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch?
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> saw a loss of
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>about 0.2% with no load.  This was
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> xen-unstable tip today
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>with no options specified.  32-bit was
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> about 0.01%.
> >>>
> >>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>>I think I missed something... how do I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>accounting choices and which ones are 
> known to be
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>for which kernels?
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>>Thanks,
> >>>>>>>>>>>>>>> > >>>>>Dan
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>>-----Original Message-----
> >>>>>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>Keir Fraser
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>>>>>> > >>>>>>To: Dave Winchell
> >>>>>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > Eddie; Jiang,
> >>>>>>>>>>>>>>> > >>>>>>Yunhong
> >>>>>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> mode that
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>disables pending
> >>>>>>>>>>>>>>> > >>>>>>missed ticks
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> changeset 16545.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>-- Keir
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Keir,
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> loads for the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>various time protocols follows. In
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> addition, the data
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>for cpu loads is shown.
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> processor AMD
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> box.
> >>>>>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> vcpu each.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>>>>>> > >>>>>>>(usex is available at
> >>>>>>>>>>>>>>> http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> of=/dev/null.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> instances of dd.
> >>>
> >>>
> >>>>>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> as i/o-32
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> instances of dd.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>>>>>>> +4.42
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec -.006%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > +.005% cpu
> >>>>>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec, -.001%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > +.012% cpu
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.009%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> -.004% cpu
> >>>>>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.005%, -.005% cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.008%, -.003% cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.040% cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec, -.034%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > -.031% cpu
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec,-55.7 sec, -.01%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.09% i/o-8
> >>>>>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec,-14.0 sec, -.015%
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.14% i/o-8
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.017%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.022% i/o-8
> >>>>>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.017%, -.018% i/o-8
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.020%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.029% i/o-8
> >>>>>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec, -.023%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > -.031% i/o-8
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 
> sec, -.12%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.04% i/o-32
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.011%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.005% i/o-32
> >>>>>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 
> sec -.10%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.11% i/o-32
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> 13. sec -.07%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > .003% i/o-4/32
> >>>>>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.017%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > .01% i/o-4/32
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Overhead measurements:
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> through a fixed
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>system workload
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu 
> sles idle.
> >>>>>>>>>>>>>>> > >>>>>>>The workload was usex -b48.
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Conclusions:
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>The only protocol which meets the 
> .05% accuracy
> >>>>>>>>>>>>>>> > requirement for ntp
> >>>>>>>>>>>>>>> > >>>>>>>tracking under the loads
> >>>>>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> accuracies for
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>SYNC, MIXED,
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>and ASYNC
> >>>>>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> method by only
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>scheduling the extra
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>wakeups if a certain number
> >>>>>>>>>>>>>>> > >>>>>>>of ticks are missed.
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Regards,
> >>>>>>>>>>>>>>> > >>>>>>>Dave
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>Keir Fraser wrote:
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ASYNC method a
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>couple of days ago,
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> there may have
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > been something
> >>>>>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> days ago for
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>ASYNC. It may have been
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> after context
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>switch in.
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> or ASYNC give
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>acceptable accuracy,
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>each running consistently around or under
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> .01%. MIXED has
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>a fairly high
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>error of
> >>>>>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> to .05% ntp
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>threshold for comfort.
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> plan to leave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>SYNC running
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> leave MIXED
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>running instead.
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the 
> protocol and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I can run
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>more overnight tests
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>next week.
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> effects of the
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>SYNC+run_timer
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> cause higher
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>system-wide CPU
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> through the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> accurate than
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than 
> the other
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> approaches,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>and each interrupt
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> favourites and
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>if the latter is
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>actually more accurate then I can
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> simply revert the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > changeset that
> >>>>>>>>>>>>>>> > >>>>>>>>implemented MIXED.
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> workloads you
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>could try idle
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> large disc reads
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>to /dev/null)? We
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CPU bound, so
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>that's really an
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>-- Keir
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > >>>>>>>>
> >>>>>>>>>>>>>>> > 
> >>>>>>_______________________________________________
> >>>>>>>>>>>>>>> > >>>>>>Xen-devel mailing list
> >>>>>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>>
> >>>>>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> 2007 +0000
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> 2008 -0500
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> pt_process_missed_ticks(stru
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>     missed_ticks = missed_ticks / (s_time_t)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> pt->period + 1;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>     if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> no_missed_ticks_pending) )
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>>>>>> > >>+        pt->do_not_freeze = 1;
> >>>>>>>>>>>>>>> > >>     else
> >>>>>>>>>>>>>>> > >>         pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>>>>>> > >>     pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> pt_timer_fn(void *data)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>     pt_lock(pt);
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>-    pt->pending_intr_nr++;
> >>>>>>>>>>>>>>> > >>+    if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> no_missed_ticks_pending) ) {
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>+        pt->pending_intr_nr = 1;
> >>>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 0;
> >>>>>>>>>>>>>>> > >>+    }
> >>>>>>>>>>>>>>> > >>+    else
> >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr++;
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>     if ( !pt->one_shot )
> >>>>>>>>>>>>>>> > >>     {
> >>>>>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> vcpu *v, struct
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>         return;
> >>>>>>>>>>>>>>> > >>     }
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>-    pt->do_not_freeze = 0;
> >>>>>>>>>>>>>>> > >>-
> >>>>>>>>>>>>>>> > >>     if ( pt->one_shot )
> >>>>>>>>>>>>>>> > >>     {
> >>>>>>>>>>>>>>> > >>         pt->enabled = 0;
> >>>>>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *v, struct
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>             pt->last_plt_gtime =
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> hvm_get_guest_time(v);
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>             pt->pending_intr_nr = 0; /*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> 'collapse' all
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > missed ticks */
> >>>>>>>>>>>>>>> > >>         }
> >>>>>>>>>>>>>>> > >>+ else if ( mode_is(v->domain,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> no_missed_ticks_pending) ) {
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>+     pt->pending_intr_nr--;
> >>>>>>>>>>>>>>> > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>>>>>> > >>+ }
> >>>>>>>>>>>>>>> > >>         else
> >>>>>>>>>>>>>>> > >>         {
> >>>>>>>>>>>>>>> > >>             pt->last_plt_gtime +=
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> pt->period_cycles;
> >>>
> >>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > _______________________________________________
> >>>>>>>>>>>>>>> > Xen-devel mailing list
> >>>>>>>>>>>>>>> > Xen-devel@lists.xensource.com
> >>>>>>>>>>>>>>> > http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --------------------------------------------------------------
> >>>>> ----------
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 19/2/08 17:55, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> Note that if hpet=0 and pmtimer=0 were the default hvm platform
> parameters for all xen hvm guests (on all versions of xen),
> specifying kernel command line parameters would be unnecessary,
> but c'est la vie.

I will get round to pmtimer at some point. I haven't put the hpet patch into
3.2 or 3.1 because it did not trivially backport. Someone else will have to
do that work if they want it (including 3.2, if you want 3.1!).

 -- Keir

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

Thanks for all the investigation you've done!

-Dave


Dan Magenheimer wrote:

>Hi Dave --
>
>Thanks for that observation on ltp running on one vcpu!
>
>With "clocksource=pit nohpet nopmtimer" our clock skew
>problems seem to have been reduced to a reasonable
>percentage.  However, our 32-bit clock skew seems to
>show a measureable problem now.
>
For the 32 bit guest, which timesource did it pick?

>  As a result,
>I've been doing some digging into kernel sources and
>have observed the following relative to RHEL4 (2.6.9-based)
>kernels and RHEL5 (2.6.18-based) kernels and thought I
>would document them for posterity.  Some of
>our confusion arises from the fact that invalid command
>line parameters are silently ignored.
>
>RHEL4:
>- clock= is a valid parameter for RHEL4-32
>- clocksource= is not a valid parameter for RHEL4-xx
>- nohpet is a valid parameter for RHEL4-64, not RHEL4-32
>- nopmtimer is not a valid parameter for RHEL4-xx
>- notsc is a valid parameter for RHEL4-32, not RHEL4-64
>- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently
>
>For Xen RHEL4 HVM guests:
>- I *think* clock=pit is sufficient for RHEL4-32 [1]
>- I *think* nohpet is sufficient for RHEL4-64 [1]
>
>RHEL5:
>- there are two kinds of timekeeping, WALL and gtod
>- clocksource= is a valid parameter for RHEL5-xx
>- clock= is a valid but deprecated parameter for RHEL5-xx
>- clock= and clocksource= are essentially equivalent
>- nohpet is a valid parameter for RHEL5-64, not RHEL5-32
>- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32
>- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1]
>- clock=pit changes the gtod source but not the WALL source[2]
>- nohpet nopmtimer changes the WALL source to PIT
>- /sys/devices/system/clocksource/clocksource0/...
>  available_clocksource lists the possible clock sources
>  current_clocksource lists the chosen clock source
>  ..but neither of these works in a RHEL5 guest!
>
>For Xen RHEL5 HVM guests:
>- I *think* clock=pit is sufficient for RHEL5-32
>  
>
But still poor accuracy, right?

>- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64
>
>Other info:
>- As of 2.6.24.2, clock= is still valid (though still deprecated)
>
>So, some open questions:
>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>    (I *think* not as it has never come up in any email.)
>  
>
I have not investigated this yet.

>[2] In RHEL5, I *think* it is the WALL source that we care about?
>  
>
I'll have to check on this too.

>And finally, since invalid command line parameters are ignored.
>I think specifying:
>	clock=pit nohpet nopmtimer
>will force the guest clock sources into the optimal state for
>all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the
>question above on tsc).  And we should keep an eye on
>kernel/time/clocksource.c to ensure the __setup("clock="...)
>line doesn't go away before RHEL6.
>
>Note that if hpet=0 and pmtimer=0 were the default hvm platform
>parameters for all xen hvm guests (on all versions of xen),
>specifying kernel command line parameters would be unnecessary,
>but c'est la vie.
>
>Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests
>and timer_mode=2 for 64-bit RHEL guests.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Tuesday, February 19, 2008 8:27 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
>>Patel
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>
>>Hi Dan,
>>
>>ltp runs by default loading up only one vcpu.
>>The -x option can be used to run multiple instances, though
>>in this mode you will get test failures.
>>I ran 8 instances on each guest for 16 hours, 25 min
>>and the time error was -11 sec (-.019%) on each guest.
>>
>>Regards,
>>Dave
>>
>>Dave Winchell wrote:
>>
>>    
>>
>>>Hi Dan,
>>>
>>>Mine was oversubscribed.
>>>8 physical cpu, 2 guests, each with 8 vcpu.
>>>I ran one instance of ltp on each guest, continuously. I hope ltp
>>>loaded up all the vcpus. I seem to recall that it did, but I
>>>could be wrong. If it didn't, that would be a major difference
>>>between our tests. I'll verify this afternoon and run 
>>>      
>>>
>>multiple instances,
>>    
>>
>>>if necessary.
>>>
>>>Thanks,
>>>Dave
>>>
>>>
>>>
>>>Dan Magenheimer wrote:
>>>
>>>      
>>>
>>>>Hi Dave --
>>>>
>>>>No new results yet but one other question:
>>>>
>>>>The problems we've seen with our testing have been with a heavily
>>>>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
>>>>running LTP simultaneously.
>>>>
>>>>Was your LTP testing oversubscribed or just a single guest?
>>>>
>>>>Thanks,
>>>>Dan
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>-----Original Message-----
>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>Sent: Thursday, February 14, 2008 10:56 AM
>>>>>To: dan.magenheimer@oracle.com
>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
>>>>>Winchell
>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>>>>          
>>>>>
>>disables pending
>>    
>>
>>>>>missed ticks
>>>>>
>>>>>
>>>>>Dan,
>>>>>
>>>>>Here are some boot snipets for rh4u564 on xen 3.2.
>>>>>
>>>>>
>>>>>#1:
>>>>>
>>>>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
>>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
>>>>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
>>>>>          
>>>>>
>>3.4.6 20060404
>>    
>>
>>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>>>>...
>>>>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
>>>>>console=ttyS0 clocksource=pit nohpet
>>>>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
>>>>>Feb 14 10:44:59 vs076 kernel: PID hash table entries: 
>>>>>          
>>>>>
>>2048 (order: 11,
>>    
>>
>>>>>65536 bytes)
>>>>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
>>>>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 
>>>>>          
>>>>>
>>MHz processor.
>>    
>>
>>>>>...
>>>>>Feb 14 10:45:00 vs076 kernel: checking TSC 
>>>>>          
>>>>>
>>synchronization across 8
>>    
>>
>>>>>CPUs: passed.
>>>>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
>>>>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to 
>>>>>          
>>>>>
>>use of PM timer
>>    
>>
>>>>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
>>>>>
>>>>>
>>>>>
>>>>>#2:
>>>>>
>>>>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
>>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
>>>>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
>>>>>          
>>>>>
>>3.4.6 20060404
>>    
>>
>>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>>>>...
>>>>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
>>>>>console=ttyS0 clocksource=pit nohpet nopmtimer
>>>>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
>>>>>Feb 14 10:47:58 vs076 kernel: PID hash table entries: 
>>>>>          
>>>>>
>>2048 (order: 11,
>>    
>>
>>>>>65536 bytes)
>>>>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz 
>>>>>          
>>>>>
>>PIT timer.
>>    
>>
>>>>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 
>>>>>          
>>>>>
>>MHz processor.
>>    
>>
>>>>>...
>>>>>Feb 14 10:47:59 vs076 kernel: checking TSC 
>>>>>          
>>>>>
>>synchronization across 8
>>    
>>
>>>>>CPUs: passed.
>>>>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
>>>>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based 
>>>>>          
>>>>>
>>timekeeping.
>>    
>>
>>>>>As you can see, I only get the pit if I specify nopmtimer.
>>>>>
>>>>>Dan Magenheimer wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Hi Dave --
>>>>>>
>>>>>>Thanks for continuing to run tests!
>>>>>>
>>>>>>Hmmm... I thought I had noticed that even though Linux will
>>>>>>            
>>>>>>
>>>>>acknowledge
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>the existence of the pmtimer, it still prints:
>>>>>>
>>>>>>time.c: Using PIT/TSC based timekeeping.
>>>>>>
>>>>>>I will check again, but assuming the clocksource for our tests is
>>>>>>indeed pit, the huge difference in the results (yours vs ours) is
>>>>>>baffling. I wonder if the difference may be the 
>>>>>>            
>>>>>>
>>underlying hardware.
>>    
>>
>>>>>>Maybe we will try to ensure we can duplicate the results on
>>>>>>            
>>>>>>
>>>>>a different
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>box.
>>>>>>
>>>>>>
>>>>>>So your testing was with stock 3.2.0 xen bits (what 
>>>>>>            
>>>>>>
>>cset?) without
>>    
>>
>>>>>>any of your [quote from below] "clock related tweaks 
>>>>>>            
>>>>>>
>>that I haven't
>>    
>>
>>>>>>submitted, because I'm still characterizing them"?
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>None of the tweaks I mentioned are in this test.
>>>>>It was stock with some patches. However, none of the 
>>>>>          
>>>>>
>>patches are time
>>    
>>
>>>>>related to
>>>>>my knowledge and I checked vpt.c to make sure that it is 
>>>>>          
>>>>>
>>the same as
>>    
>>
>>>>>what's in unstable.
>>>>>The only difference is in pt_intr_post, where I set the 
>>>>>          
>>>>>
>>timer mode.
>>    
>>
>>>>>I don't have timer mode tied into our config process yet, which
>>>>>is different than official xen method.
>>>>>
>>>>>
>>>>>(In pt_intr_post)
>>>>>     else
>>>>>     {
>>>>>+       if(v->arch.paging.mode->guest_levels == 4)
>>>>>+           
>>>>>          
>>>>>
>>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
>>    
>>
>>>>>HVMPTM_no_missed_ticks_pending;
>>>>>+       else
>>>>>+           
>>>>>          
>>>>>
>>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
>>    
>>
>>>>>HVMPTM_delay_for_missed_ticks;
>>>>>         if ( mode_is(v->domain, one_missed_tick_pending) ||
>>>>>              mode_is(v->domain, no_missed_ticks_pending) )
>>>>>         {
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Could you also send detail on the rhel4u4-64 kernel you
>>>>>>are testing with, just to ensure we are not comparing apples
>>>>>>and oranges?  (Perhaps there's some way we can even share the
>>>>>>identical disk image and vm.cfg file?)
>>>>>>
>>>>>>And if our problem is indeed the pmtimer, I will need to submit
>>>>>>another patch to Keir to add an hvm pmtimer platform variable.
>>>>>>(Hmmm... I don't think he's even accepted the hpet variable patch
>>>>>>yet.  I'll have to check.)
>>>>>>
>>>>>>Thanks,
>>>>>>Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>Sent: Thursday, February 14, 2008 9:00 AM
>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>Cc: Dave Winchell; Keir Fraser;
>>>>>>>              
>>>>>>>
>>>>>xen-devel@lists.xensource.com; Deepak
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>Patel
>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>>disables pending
>>>>>>>missed ticks
>>>>>>>
>>>>>>>
>>>>>>>Hi Dan,
>>>>>>>
>>>>>>>I ran the ltp tests with 3.2 and found the errors
>>>>>>>for a 16 hour run to be:
>>>>>>>
>>>>>>>rh4u564 -9.9 sec (-.017%)
>>>>>>>rh4u464 -7.3 sec (-.013%)
>>>>>>>
>>>>>>>There were no cliffs and the drift was linear.
>>>>>>>
>>>>>>>I think the problem you had may be due to the use of the
>>>>>>>pm timer. If you still have the boot log, it would tell you.
>>>>>>>
>>>>>>>When I first tried a guest on 3.2 with "clocksource=pit nohpet"
>>>>>>>I noticed that it picked the pm timer. Adding "nopmtimer", the
>>>>>>>guest will pick the pit.
>>>>>>>
>>>>>>>The reason I didn't have the problem with our 3.1 base is that
>>>>>>>I had disabled the hpet and the pmtimer by not advertising them
>>>>>>>in the acpi tables. I did this so long ago, I forgot 
>>>>>>>              
>>>>>>>
>>that I had to
>>    
>>
>>>>>>>disable pmtimer as well as hpet.
>>>>>>>
>>>>>>>So, can you re-run your test with "clocksource=pit nohpet
>>>>>>>              
>>>>>>>
>>>>>nopmtimer"?
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>You should see this in the boot messages:
>>>>>>>
>>>>>>>time.c: Using PIT/TSC based timekeeping.
>>>>>>>
>>>>>>>Thanks,
>>>>>>>Dave
>>>>>>>
>>>>>>>
>>>>>>>Dave Winchell wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>Hi Dan,
>>>>>>>>
>>>>>>>>Over the weekend the drift was +18 seconds for each 
>>>>>>>>                
>>>>>>>>
>>guest (no ntp).
>>    
>>
>>>>>>>>The duration was 3900 minutes, so the error for each 
>>>>>>>>                
>>>>>>>>
>>was +.0077%.
>>    
>>
>>>>>>>>Looking back through the data, it appears to drift linearly at
>>>>>>>>this rate. I've attached a plot for rh4u5-64.
>>>>>>>>
>>>>>>>>This accuracy is better than what I've seen before (.03-.05%).
>>>>>>>>This may be due to the different load (ltp vs usex) or to
>>>>>>>>                
>>>>>>>>
>>>>>one of the
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>changes I've made recently. I'll do some 
>>>>>>>>                
>>>>>>>>
>>experimentation to see if
>>    
>>
>>>>>>>>there is
>>>>>>>>a fix I should propose.
>>>>>>>>
>>>>>>>>This still doesn't address the radical drift you saw.
>>>>>>>>The next step for me is to run 3.2 and see if I can 
>>>>>>>>                
>>>>>>>>
>>reproduce it.
>>    
>>
>>>>>>>>Regards,
>>>>>>>>Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Dave Winchell wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Hi Dan,
>>>>>>>>>
>>>>>>>>>Sorry it took me so long, but I finally ran an ltp test today.
>>>>>>>>>Its on rh4u4-64. I'm using the defaults for ltp and 
>>>>>>>>>                  
>>>>>>>>>
>>using a script
>>    
>>
>>>>>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
>>>>>>>>>virtual processors / physical processors = 2.
>>>>>>>>>
>>>>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 
>>>>>>>>>                  
>>>>>>>>>
>>300 minutes
>>    
>>
>>>>>>>>>for -.005% and .008%.
>>>>>>>>>
>>>>>>>>>I'm running a 3.1 based hypervisor with some clock related
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>tweaks that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>I haven't submitted, because I'm still characterizing them.
>>>>>>>>>
>>>>>>>>>I'm stopping the usex load on 4u5-64 now and 
>>>>>>>>>                  
>>>>>>>>>
>>replacing it with ltp
>>    
>>
>>>>>>>>>and will leave the two guests running ltp over the weekend.
>>>>>>>>>
>>>>>>>>>Regards,
>>>>>>>>>Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Dave Winchell wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>Hi Dan, Deepak:
>>>>>>>>>>
>>>>>>>>>>Thanks for the data. Those drifts are severe - no wonder
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>ntp couldn't
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>keep then in synch. I'll try to reproduce that behaviour
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>here, with
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>my code base.
>>>>>>>>>>If I can't reproduce it, I'll try 3.2.
>>>>>>>>>>
>>>>>>>>>>If you can isolate what ltp is doing during the cliffs,
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>that would
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>be very
>>>>>>>>>>helpful.
>>>>>>>>>>
>>>>>>>>>>thanks,
>>>>>>>>>>Dave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>OK, Deepak repeated the test without ntpd and using
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>ntpdate -b before
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>the test.
>>>>>>>>>>>
>>>>>>>>>>>The attached graph shows his results: el5u1-64 
>>>>>>>>>>>                      
>>>>>>>>>>>
>>(best=~0.07%),
>>    
>>
>>>>>>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>>>>>>>>
>>>>>>>>>>>We will continue to look at LTP to try to isolate.
>>>>>>>>>>>
>>>>>>>>>>>Thanks,
>>>>>>>>>>>Dan
>>>>>>>>>>>
>>>>>>>>>>>P.S. elXuY is essentially RHEL XuY with some patches.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
>>>>>>>>>>>>To: Deepak Patel
>>>>>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>>>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>Dave Winchell
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>that disables
>>    
>>
>>>>>>>>>>>>pending
>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Dan, Deeepak,
>>>>>>>>>>>>
>>>>>>>>>>>>It may be that the underlying clock error is too 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>great for ntp
>>    
>>
>>>>>>>>>>>>to handle. It would be useful if you did not run ntpd
>>>>>>>>>>>>and, instead did ntpdate -b <timeserver> at the start
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>of the test
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>for each guest. Then capture the data as you have 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>been doing.
>>    
>>
>>>>>>>>>>>>If the drift is greater than .05%, then we need to
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>address that.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>Another option is, when running ntpd, to enable loop
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>statistics in
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>/etc/ntp.conf
>>>>>>>>>>>>by adding this to the file:
>>>>>>>>>>>>
>>>>>>>>>>>>statistics loopstats
>>>>>>>>>>>>statsdir /var/lib/ntp/
>>>>>>>>>>>>
>>>>>>>>>>>>Then you will see loop data in that directory.
>>>>>>>>>>>>Correlating the data in the loopstats files with the
>>>>>>>>>>>>peaks in skew would be interesting. You will see
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>entries of the form
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>239.735511 10
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>Where the second to last column is the Allan Deviation.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>When that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>gets over 1000, ntpd is working pretty hard. However,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>I have not
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>seen ntpd
>>>>>>>>>>>>completely loose it like you have.
>>>>>>>>>>>>
>>>>>>>>>>>>I'm on vacation until Monday, and won't be reading
>>>>>>>>>>>>email.
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks for all your work on this!
>>>>>>>>>>>>
>>>>>>>>>>>>-Dave
>>>>>>>>>>>>
>>>>>>>>>>>>Deepak Patel wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>this one.)
>>    
>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>I do not know which graph was attached with this. But
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>I saw this
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>guests when I
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>was running ltp tests continuously.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>What was the behaviour of the other guests running?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>All pvm guests are fine. But behavior of most of the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>hvm guests were
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>as described.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>If they had spikes, were they at the same wall time?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>No. They are not at the same wall time.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Were the other guests running ltp as well?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are 
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>running ltp
>>    
>>
>>>>>>>>>>>>>continuously.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>How are you measuring skew?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>I was collecting output of "ntpdate -q  <timeserver> every
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>300 seconds
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>(5 minutes) and have created graph based on that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Are you running ntpd?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>Yes. ntp was running on all the guests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>I am investigating what causes this spikes and 
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>let everyone
>>    
>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>know what
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>are my findings.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>Deepak
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Anything that you can discover that would be in sync with
>>>>>>>>>>>>>>the spikes would be very helpful!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>The code that I test with is our product code, 
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>which is based
>>    
>>
>>>>>>>>>>>>>>on 3.1. So it is possible that something in 3.2 other
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>than vpt.c
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>is the cause. I can test with 3.2, if necessary.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>thanks,
>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                            
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Hi Dave (Keir, see suggestion below) --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Turning off vhpet certainly helps a lot (though 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>see below).
>>    
>>
>>>>>>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>should be
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>until it is
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>fixed?  (I have a patch that defaults it off, 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>can post it if
>>    
>>
>>>>>>>>>>>>>>>there is agreement on the above point.)  The whole
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>point of an
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>HPET is to provide more precise timekeeping and 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>if vhpet is
>>    
>>
>>>>>>>>>>>>>>>worse than vpit, it can only confuse users.  Comments?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>In your testing, are you just measuring % skew 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>over a long
>>    
>>
>>>>>>>>>>>>>>>period of time?
>>>>>>>>>>>>>>>We are graphing the skew continuously and
>>>>>>>>>>>>>>>seeing periodic behavior that is unsettling, 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>even with pit.
>>    
>>
>>>>>>>>>>>>>>>See attached.  Though your algorithm recovers, 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>the "cliffs"
>>    
>>
>>>>>>>>>>>>>>>could still cause real user problems.  I wonder 
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>if there is
>>    
>>
>>>>>>>>>>>>>>>anything that can be done to make the "recovery" more
>>>>>>>>>>>>>>>responsive?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>We are looking into what part(s) of LTP is causing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>the cliffs.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>>deepak.patel@oracle.com;
>>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>that disables
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>pending
>>>>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Dan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage.
>>>>>>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>should specify
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are.
>>>>>>>>>>>>>>>>The xen and guest clocksources do not need to 
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>be the same.
>>    
>>
>>>>>>>>>>>>>>>>In my tests, xen is using the hpet for its 
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>timekeeping and
>>    
>>
>>>>>>>>>>>>>>>>that appears to be the default.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>When you boot the guests you should see
>>>>>>>>>>>>>>>>time.c: Using PIT/TSC based timekeeping.
>>>>>>>>>>>>>>>>on the rh4u5-64 guest, and something similar 
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>on the others.
>>    
>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>Platform timer
>>    
>>
>>>>>>>>>>>>>>>>>14.318MHz HPET.)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>This appears to be the xen state, which is fine.
>>>>>>>>>>>>>>>>I was wrongly assuming that this was the guest state.
>>>>>>>>>>>>>>>>You might want to look in your guest logs and see
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>what they were
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>picking
>>>>>>>>>>>>>>>>for a clock source.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thanks, I hadn't realized that!  No wonder we didn't
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>see the same
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>improvement you saw!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Try specifying clock=pit on the linux boot line...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>on the Xen
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>command line or
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>guest (or
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>dom0?) command
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>line?  Or both places?  Since the tests take 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>awhile, it
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>would be nice
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>to get this right the first time.  Do the Xen 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>and guest
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>clocksources need
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>to be the same?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>>>>>*From:* Dave Winchell 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>[mailto:dwinchell@virtualiron.com]
>>    
>>
>>>>>>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>deepak.patel@oracle.com;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>that disables
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>pending missed ticks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hi Dan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hpet timer does have a fairly large error, as I
>>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>trying this
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>one recently.
>>>>>>>>>>>>>>>>>I don't remember what I got for error, but 1% sounds
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>about right.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>The problem is that hpet is not built on top of vpt.c,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>the module
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Keir and I did
>>>>>>>>>>>>>>>>>all the recent work in, for its periodic timer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>needs. Try
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>specifying clock=pit
>>>>>>>>>>>>>>>>>on the linux boot line. If it still picks the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>hpet, which it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>might, let me know
>>>>>>>>>>>>>>>>>and I'll tell you how to get around this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>--------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>----------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>*From:* Dan Magenheimer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:dan.magenheimer@oracle.com]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>*Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>>>>>>>>*To:* Dave Winchell; Keir Fraser
>>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>deepak.patel@oracle.com;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>akira.ijuin@oracle.com
>>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>that disables
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>pending missed ticks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Sorry for the very late followup on this but 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>we finally
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>were able
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>to get our testing set up again on stable 3.1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>bits and have
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>seen some very bad results on 3.1.3-rc1, on the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>order of 1%.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>Test enviroment was a 4-socket dual core machine
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>with 24GB of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>memory running six two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>plus two pv.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>All six guests were running LTP simultaneously.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>The four hvm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>RHEL4u5-64.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Timer_mode was set to 2 for 64-bit guests and 0 for
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>32-bit guests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>All four hvm guests experienced skew around -1%,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>even the 32-bit
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>guest.  Less intensive testing didn't exhibit much
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>skew at all.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>A representative graph is attached.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Dave, I wonder if some portion of your patches
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>didn't end up in
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>the xen trees?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>Platform timer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>14.318MHz HPET.)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>P.S. Many thanks to Deepak and Akira for 
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>running tests.
>>    
>>
>>>>>>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>Dave Winchell
>>>>>>>>>>>>>>>>>>Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>>>>>>>>>To: Keir Fraser
>>>>>>>>>>>>>>>>>>Cc: dan.magenheimer@oracle.com;
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Winchell
>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a 
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>timer mode that
>>    
>>
>>>>>>>>>>>>>>>>>>disables pending
>>>>>>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Hi Keir,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>The latest change, c/s 16690, looks fine.
>>>>>>>>>>>>>>>>>>I agree that the code in c/s 16690 is equivalent to
>>>>>>>>>>>>>>>>>>the code I submitted. Also, your version is more
>>>>>>>>>>>>>>>>>>concise.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>The error tests confirm the equivalence. With
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>overnight cpu loads,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>the checked in version was accurate to 
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>+.048% for sles
>>    
>>
>>>>>>>>>>>>>>>>>>and +.038% for red hat. My version was +.046%
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>+.032% in a
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>2 hour test.
>>>>>>>>>>>>>>>>>>I don't think the difference is significant.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>i/o loads produced errors of +.01%.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Thanks for all your efforts on this issue.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Applied as c/s 16690, although the
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>checked-in patch is
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>smaller. I think the
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>only important fix is to pt_intr_post() and the
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>only bit of
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>the patch I
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>totally omitted was the change to
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>pt_process_missed_ticks().
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>I don't think
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>that change can be important, but let's see what
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>happens to the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>error
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>percentage...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>-- Keir
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>On 4/1/08 23:24, "Dave Winchell"
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Hi Dan and Keir,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Attached is a patch that fixes some 
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>issues with the
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>SYNC policy
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>(no_missed_ticks_pending).
>>>>>>>>>>>>>>>>>>>>I have not tried to make the change the
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>minimal one, but,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>rather, just
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>ported into
>>>>>>>>>>>>>>>>>>>>the new code what I know to work well. 
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>The error for
>>    
>>
>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending goes from
>>>>>>>>>>>>>>>>>>>>over 3% to .03% with this change according
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>to my testing.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Hi Dave --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Did you get your correction ported?  If so,
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>it would be
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>nice to see this get
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>into 3.1.3.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Note that I just did some very limited 
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>testing with
>>    
>>
>>>>>>>>>>>>>>>>>>timer_mode=2(=SYNC=no
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>missed ticks pending)
>>>>>>>>>>>>>>>>>>>>>on tip of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>guest) and the
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>worst error I've
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>seen so far
>>>>>>>>>>>>>>>>>>>>>is 0.012%.  But I haven't tried any exotic
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>loads, just LTP.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>>>>>>>>>>From: Dave Winchell
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:dwinchell@virtualiron.com]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>>>>>>>>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>timer mode that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>disables pending
>>>>>>>>>>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Dan,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>I did some testing with the constant tsc offset
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>SYNC method
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>(now called
>>>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending)
>>>>>>>>>>>>>>>>>>>>>>and found the error to be very high, much larger
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>than 1 %, as
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>I recall.
>>>>>>>>>>>>>>>>>>>>>>I have not had a chance to submit a 
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>correction. I
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>will try to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>do it later
>>>>>>>>>>>>>>>>>>>>>>this week or the first week in January. My
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>version of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>constant tsc
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>offset SYNC method
>>>>>>>>>>>>>>>>>>>>>>produces .02 % error, so I just need to port
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>that into the
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>current code.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>The error you got for both of those kernels is
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>what I would
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>expect
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>I'll let Keir answer on how to set the 
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>time mode.
>>    
>>
>>>>>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>>>>>>>>>>>>                                              
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>saw a loss of
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                              
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>about 0.2% with no load.  This was
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>xen-unstable tip today
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>with no options specified.  32-bit was
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>about 0.01%.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>I think I missed something... how do I
>>>>>>>>>>>>>>>>>>>>>>>                                              
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>run the various
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                              
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>accounting choices and which ones are 
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>known to be
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>appropriate
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>for which kernels?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                              
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Keir Fraser
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>>>>>>>>>>>>>To: Dave Winchell
>>>>>>>>>>>>>>>>>>>>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Eddie; Jiang,
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Yunhong
>>>>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>mode that
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>disables pending
>>>>>>>>>>>>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Please take a look at xen-unstable
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>changeset 16545.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>-- Keir
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Keir,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The accuracy data I've collected for i/o
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>loads for the
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>various time protocols follows. In
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>addition, the data
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>for cpu loads is shown.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>processor AMD
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>box.
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Two guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>vcpu each.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>>>>>>>>>>>>>>(usex is available at
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>of=/dev/null.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-32 are 32
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>instances of dd.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>>>>>>>>>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>>>>>>>>>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-4/32 are the same
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>as i/o-32
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>instances of dd.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>+4.42
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec -.006%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>+.005% cpu
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec, -.001%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>+.012% cpu
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.009%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>-.004% cpu
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.005%, -.005% cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.008%, -.003% cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.040% cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec, -.034%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>-.031% cpu
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec,-55.7 sec, -.01%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.09% i/o-8
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec,-14.0 sec, -.015%
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.14% i/o-8
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.017%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.022% i/o-8
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.017%, -.018% i/o-8
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.020%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.029% i/o-8
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec, -.023%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>-.031% i/o-8
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>sec, -.12%,
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.04% i/o-32
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.011%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.005% i/o-32
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>sec -.10%,
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.11% i/o-32
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>13. sec -.07%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>.003% i/o-4/32
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.017%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>.01% i/o-4/32
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Overhead measurements:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>through a fixed
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>system workload
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>on an 8 vcpu red hat with an 8 vcpu 
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>sles idle.
>>    
>>
>>>>>>>>>>>>>>>>>>>>>>>>>The workload was usex -b48.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Conclusions:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The only protocol which meets the 
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>.05% accuracy
>>    
>>
>>>>>>>>>>>>>>>>>>requirement for ntp
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>tracking under the loads
>>>>>>>>>>>>>>>>>>>>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>accuracies for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC, MIXED,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>and ASYNC
>>>>>>>>>>>>>>>>>>>>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>We could reduce the cost of the SYNC
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>method by only
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>scheduling the extra
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>wakeups if a certain number
>>>>>>>>>>>>>>>>>>>>>>>>>of ticks are missed.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>                                                  
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>ASYNC method a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>couple of days ago,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>I ran another ASYNC test. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>there may have
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>been something
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>days ago for
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>ASYNC. It may have been
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>after context
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>switch in.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>My results indicate that either SYNC
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>acceptable accuracy,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>each running consistently around or under
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>.01%. MIXED has
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>a fairly high
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>error of
>>>>>>>>>>>>>>>>>>>>>>>>>>>greater than .03%. Probably too close
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>threshold for comfort.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>I don't have an overnight run with SYNC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>plan to leave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC running
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>over the weekend. If you'd rather I can
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>leave MIXED
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>running instead.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>It may be too early to pick the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>protocol and
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I can run
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>more overnight tests
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>                                                      
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>I'm a bit worried about any unwanted side
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>effects of the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC+run_timer
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>cause higher
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>system-wide CPU
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>contention. I find it easier to think
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>through the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>implications of ASYNC. I'm
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>accurate than
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>delivers more timer interrupts than 
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>the other
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>approaches,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>and each interrupt
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>favourites and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>if the latter is
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>actually more accurate then I can
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>simply revert the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>changeset that
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>implemented MIXED.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>workloads you
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>could try idle
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>large disc reads
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>to /dev/null)? We
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>don't have any data on workloads that aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>CPU bound, so
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>that's really an
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>-- Keir
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>                                                    
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>                
>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Xen-devel mailing list
>>>>>>>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>                                                
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>                                              
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                                            
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>                                          
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>>>>>>>>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>2007 +0000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>2008 -0500
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>@@ -58,7 +58,7 @@ static void
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>pt_process_missed_ticks(stru
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>pt->period + 1;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>no_missed_ticks_pending) )
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>>>>>>>>>+        pt->do_not_freeze = 1;
>>>>>>>>>>>>>>>>>>>>    else
>>>>>>>>>>>>>>>>>>>>        pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>>>>>>>>>    pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>>>>>>>>>@@ -127,7 +127,12 @@ static void
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>pt_timer_fn(void *data)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>    pt_lock(pt);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>-    pt->pending_intr_nr++;
>>>>>>>>>>>>>>>>>>>>+    if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>+        pt->pending_intr_nr = 1;
>>>>>>>>>>>>>>>>>>>>+ pt->do_not_freeze = 0;
>>>>>>>>>>>>>>>>>>>>+    }
>>>>>>>>>>>>>>>>>>>>+    else
>>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr++;
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    if ( !pt->one_shot )
>>>>>>>>>>>>>>>>>>>>    {
>>>>>>>>>>>>>>>>>>>>@@ -221,8 +226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>vcpu *v, struct
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>        return;
>>>>>>>>>>>>>>>>>>>>    }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>-    pt->do_not_freeze = 0;
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>    if ( pt->one_shot )
>>>>>>>>>>>>>>>>>>>>    {
>>>>>>>>>>>>>>>>>>>>        pt->enabled = 0;
>>>>>>>>>>>>>>>>>>>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>*v, struct
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>            pt->last_plt_gtime =
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>hvm_get_guest_time(v);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>            pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>'collapse' all
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>missed ticks */
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>>>+ else if ( mode_is(v->domain,
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>+     pt->pending_intr_nr--;
>>>>>>>>>>>>>>>>>>>>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>>>>>>>>>+ }
>>>>>>>>>>>>>>>>>>>>        else
>>>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>>>            pt->last_plt_gtime +=
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>pt->period_cycles;
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>>>                                        
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                      
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>_______________________________________________
>>>>>>>>>>>>>>>>>>Xen-devel mailing list
>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                    
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                  
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>--------------------------------------------------------------
>>>>>>>----------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>          
>>>>>
>>>>
>>>>
>>>>        
>>>>
>>    
>>
>
>  
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

> >percentage.  However, our 32-bit clock skew seems to
> >show a measureable problem now.
> >
> For the 32 bit guest, which timesource did it pick?

The dmesg output is hard to interpret on a 32-bit guest,
but based on what we've seen, I think it was selecting
hpet as timesource (because we specified clocksource=pit
which would have been ignored on RHEL4-32).  We are
running another test with "clock=pit" to see if the
skew goes away.

> >For Xen RHEL5 HVM guests:
> >- I *think* clock=pit is sufficient for RHEL5-32
> >
> But still poor accuracy, right?

Unproven yet but I hope not.  The nohpet and nopmtimer
parameters are ignored on RHEL5-32 so the clock=pit
(or clocksource=pit) is the only way to choose the
clock source, and thus the only way to get good
accuracy on RHEL5-32.

Oops, I see from my long list that I neglected to say
that the two clocks (WALL and GTOD) on RHEL5 are
only reported on RHEL5-64.  RHELx-32 looks to have
only the one, which is overridden with clock=.

> >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> >    (I *think* not as it has never come up in any email.)
> >
> I have not investigated this yet.

My *think* is based on: 1) observation of dmesg output for
RHEL5-64 where specifying "nohpet nopmtimer" seems to
select PIT for WALL timer; 2) no mention of tsc in
the generic clocksource.c code nor in the i386-specific
time code.

Still I would sleep better if this were definitive.

> >[2] In RHEL5, I *think* it is the WALL source that we care about?
> >
> I'll have to check on this too.

My *think* is based on our observations to date that clock=pit
is insufficient to fix the skew problem (and doesn't change
the dmesg WALL source output on RHELx-64)... nohpet and nopmtimer
is required to change the WALL source output and fix the skew.

Again I would sleep better if this were definitive.

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Tuesday, February 19, 2008 1:50 PM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Dan,
> 
> Thanks for all the investigation you've done!
> 
> -Dave
> 
> 
> Dan Magenheimer wrote:
> 
> >Hi Dave --
> >
> >Thanks for that observation on ltp running on one vcpu!
> >
> >With "clocksource=pit nohpet nopmtimer" our clock skew
> >problems seem to have been reduced to a reasonable
> >percentage.  However, our 32-bit clock skew seems to
> >show a measureable problem now.
> >
> For the 32 bit guest, which timesource did it pick?
> 
> >  As a result,
> >I've been doing some digging into kernel sources and
> >have observed the following relative to RHEL4 (2.6.9-based)
> >kernels and RHEL5 (2.6.18-based) kernels and thought I
> >would document them for posterity.  Some of
> >our confusion arises from the fact that invalid command
> >line parameters are silently ignored.
> >
> >RHEL4:
> >- clock= is a valid parameter for RHEL4-32
> >- clocksource= is not a valid parameter for RHEL4-xx
> >- nohpet is a valid parameter for RHEL4-64, not RHEL4-32
> >- nopmtimer is not a valid parameter for RHEL4-xx
> >- notsc is a valid parameter for RHEL4-32, not RHEL4-64
> >- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently
> >
> >For Xen RHEL4 HVM guests:
> >- I *think* clock=pit is sufficient for RHEL4-32 [1]
> >- I *think* nohpet is sufficient for RHEL4-64 [1]
> >
> >RHEL5:
> >- there are two kinds of timekeeping, WALL and gtod
> >- clocksource= is a valid parameter for RHEL5-xx
> >- clock= is a valid but deprecated parameter for RHEL5-xx
> >- clock= and clocksource= are essentially equivalent
> >- nohpet is a valid parameter for RHEL5-64, not RHEL5-32
> >- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32
> >- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1]
> >- clock=pit changes the gtod source but not the WALL source[2]
> >- nohpet nopmtimer changes the WALL source to PIT
> >- /sys/devices/system/clocksource/clocksource0/...
> >  available_clocksource lists the possible clock sources
> >  current_clocksource lists the chosen clock source
> >  ..but neither of these works in a RHEL5 guest!
> >
> >For Xen RHEL5 HVM guests:
> >- I *think* clock=pit is sufficient for RHEL5-32
> >
> >
> But still poor accuracy, right?
> 
> >- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64
> >
> >Other info:
> >- As of 2.6.24.2, clock= is still valid (though still deprecated)
> >
> >So, some open questions:
> >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> >    (I *think* not as it has never come up in any email.)
> >
> >
> I have not investigated this yet.
> 
> >[2] In RHEL5, I *think* it is the WALL source that we care about?
> >
> >
> I'll have to check on this too.
> 
> >And finally, since invalid command line parameters are ignored.
> >I think specifying:
> >	clock=pit nohpet nopmtimer
> >will force the guest clock sources into the optimal state for
> >all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the
> >question above on tsc).  And we should keep an eye on
> >kernel/time/clocksource.c to ensure the __setup("clock="...)
> >line doesn't go away before RHEL6.
> >
> >Note that if hpet=0 and pmtimer=0 were the default hvm platform
> >parameters for all xen hvm guests (on all versions of xen),
> >specifying kernel command line parameters would be unnecessary,
> >but c'est la vie.
> >
> >Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests
> >and timer_mode=2 for 64-bit RHEL guests.
> >
> >Thanks,
> >Dan
> >
> >
> >
> >>-----Original Message-----
> >>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>Sent: Tuesday, February 19, 2008 8:27 AM
> >>To: dan.magenheimer@oracle.com
> >>Cc: Dave Winchell; Keir Fraser; 
> xen-devel@lists.xensource.com; Deepak
> >>Patel
> >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>disables pending
> >>missed ticks
> >>
> >>
> >>
> >>Hi Dan,
> >>
> >>ltp runs by default loading up only one vcpu.
> >>The -x option can be used to run multiple instances, though
> >>in this mode you will get test failures.
> >>I ran 8 instances on each guest for 16 hours, 25 min
> >>and the time error was -11 sec (-.019%) on each guest.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dave Winchell wrote:
> >>
> >>
> >>
> >>>Hi Dan,
> >>>
> >>>Mine was oversubscribed.
> >>>8 physical cpu, 2 guests, each with 8 vcpu.
> >>>I ran one instance of ltp on each guest, continuously. I hope ltp
> >>>loaded up all the vcpus. I seem to recall that it did, but I
> >>>could be wrong. If it didn't, that would be a major difference
> >>>between our tests. I'll verify this afternoon and run
> >>>
> >>>
> >>multiple instances,
> >>
> >>
> >>>if necessary.
> >>>
> >>>Thanks,
> >>>Dave
> >>>
> >>>
> >>>
> >>>Dan Magenheimer wrote:
> >>>
> >>>
> >>>
> >>>>Hi Dave --
> >>>>
> >>>>No new results yet but one other question:
> >>>>
> >>>>The problems we've seen with our testing have been with a heavily
> >>>>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
> >>>>running LTP simultaneously.
> >>>>
> >>>>Was your LTP testing oversubscribed or just a single guest?
> >>>>
> >>>>Thanks,
> >>>>Dan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>-----Original Message-----
> >>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>Sent: Thursday, February 14, 2008 10:56 AM
> >>>>>To: dan.magenheimer@oracle.com
> >>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak 
> Patel; Dave
> >>>>>Winchell
> >>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>
> >>>>>
> >>disables pending
> >>
> >>
> >>>>>missed ticks
> >>>>>
> >>>>>
> >>>>>Dan,
> >>>>>
> >>>>>Here are some boot snipets for rh4u564 on xen 3.2.
> >>>>>
> >>>>>
> >>>>>#1:
> >>>>>
> >>>>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
> >>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
> >>>>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
> >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version
> >>>>>
> >>>>>
> >>3.4.6 20060404
> >>
> >>
> >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>>>>...
> >>>>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro 
> root=LABEL=/
> >>>>>console=ttyS0 clocksource=pit nohpet
> >>>>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
> >>>>>Feb 14 10:44:59 vs076 kernel: PID hash table entries:
> >>>>>
> >>>>>
> >>2048 (order: 11,
> >>
> >>
> >>>>>65536 bytes)
> >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz 
> PM timer.
> >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050
> >>>>>
> >>>>>
> >>MHz processor.
> >>
> >>
> >>>>>...
> >>>>>Feb 14 10:45:00 vs076 kernel: checking TSC
> >>>>>
> >>>>>
> >>synchronization across 8
> >>
> >>
> >>>>>CPUs: passed.
> >>>>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
> >>>>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to
> >>>>>
> >>>>>
> >>use of PM timer
> >>
> >>
> >>>>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
> >>>>>
> >>>>>
> >>>>>
> >>>>>#2:
> >>>>>
> >>>>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
> >>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
> >>>>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
> >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version
> >>>>>
> >>>>>
> >>3.4.6 20060404
> >>
> >>
> >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>>>>...
> >>>>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro 
> root=LABEL=/
> >>>>>console=ttyS0 clocksource=pit nohpet nopmtimer
> >>>>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
> >>>>>Feb 14 10:47:58 vs076 kernel: PID hash table entries:
> >>>>>
> >>>>>
> >>2048 (order: 11,
> >>
> >>
> >>>>>65536 bytes)
> >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz
> >>>>>
> >>>>>
> >>PIT timer.
> >>
> >>
> >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959
> >>>>>
> >>>>>
> >>MHz processor.
> >>
> >>
> >>>>>...
> >>>>>Feb 14 10:47:59 vs076 kernel: checking TSC
> >>>>>
> >>>>>
> >>synchronization across 8
> >>
> >>
> >>>>>CPUs: passed.
> >>>>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
> >>>>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based
> >>>>>
> >>>>>
> >>timekeeping.
> >>
> >>
> >>>>>As you can see, I only get the pit if I specify nopmtimer.
> >>>>>
> >>>>>Dan Magenheimer wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Hi Dave --
> >>>>>>
> >>>>>>Thanks for continuing to run tests!
> >>>>>>
> >>>>>>Hmmm... I thought I had noticed that even though Linux will
> >>>>>>
> >>>>>>
> >>>>>acknowledge
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>the existence of the pmtimer, it still prints:
> >>>>>>
> >>>>>>time.c: Using PIT/TSC based timekeeping.
> >>>>>>
> >>>>>>I will check again, but assuming the clocksource for 
> our tests is
> >>>>>>indeed pit, the huge difference in the results (yours 
> vs ours) is
> >>>>>>baffling. I wonder if the difference may be the
> >>>>>>
> >>>>>>
> >>underlying hardware.
> >>
> >>
> >>>>>>Maybe we will try to ensure we can duplicate the results on
> >>>>>>
> >>>>>>
> >>>>>a different
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>box.
> >>>>>>
> >>>>>>
> >>>>>>So your testing was with stock 3.2.0 xen bits (what
> >>>>>>
> >>>>>>
> >>cset?) without
> >>
> >>
> >>>>>>any of your [quote from below] "clock related tweaks
> >>>>>>
> >>>>>>
> >>that I haven't
> >>
> >>
> >>>>>>submitted, because I'm still characterizing them"?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>None of the tweaks I mentioned are in this test.
> >>>>>It was stock with some patches. However, none of the
> >>>>>
> >>>>>
> >>patches are time
> >>
> >>
> >>>>>related to
> >>>>>my knowledge and I checked vpt.c to make sure that it is
> >>>>>
> >>>>>
> >>the same as
> >>
> >>
> >>>>>what's in unstable.
> >>>>>The only difference is in pt_intr_post, where I set the
> >>>>>
> >>>>>
> >>timer mode.
> >>
> >>
> >>>>>I don't have timer mode tied into our config process yet, which
> >>>>>is different than official xen method.
> >>>>>
> >>>>>
> >>>>>(In pt_intr_post)
> >>>>>     else
> >>>>>     {
> >>>>>+       if(v->arch.paging.mode->guest_levels == 4)
> >>>>>+
> >>>>>
> >>>>>
> >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
> >>
> >>
> >>>>>HVMPTM_no_missed_ticks_pending;
> >>>>>+       else
> >>>>>+
> >>>>>
> >>>>>
> >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =
> >>
> >>
> >>>>>HVMPTM_delay_for_missed_ticks;
> >>>>>         if ( mode_is(v->domain, one_missed_tick_pending) ||
> >>>>>              mode_is(v->domain, no_missed_ticks_pending) )
> >>>>>         {
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Could you also send detail on the rhel4u4-64 kernel you
> >>>>>>are testing with, just to ensure we are not comparing apples
> >>>>>>and oranges?  (Perhaps there's some way we can even share the
> >>>>>>identical disk image and vm.cfg file?)
> >>>>>>
> >>>>>>And if our problem is indeed the pmtimer, I will need to submit
> >>>>>>another patch to Keir to add an hvm pmtimer platform variable.
> >>>>>>(Hmmm... I don't think he's even accepted the hpet 
> variable patch
> >>>>>>yet.  I'll have to check.)
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Dan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>-----Original Message-----
> >>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>Sent: Thursday, February 14, 2008 9:00 AM
> >>>>>>>To: dan.magenheimer@oracle.com
> >>>>>>>Cc: Dave Winchell; Keir Fraser;
> >>>>>>>
> >>>>>>>
> >>>>>xen-devel@lists.xensource.com; Deepak
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>Patel
> >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>>>disables pending
> >>>>>>>missed ticks
> >>>>>>>
> >>>>>>>
> >>>>>>>Hi Dan,
> >>>>>>>
> >>>>>>>I ran the ltp tests with 3.2 and found the errors
> >>>>>>>for a 16 hour run to be:
> >>>>>>>
> >>>>>>>rh4u564 -9.9 sec (-.017%)
> >>>>>>>rh4u464 -7.3 sec (-.013%)
> >>>>>>>
> >>>>>>>There were no cliffs and the drift was linear.
> >>>>>>>
> >>>>>>>I think the problem you had may be due to the use of the
> >>>>>>>pm timer. If you still have the boot log, it would tell you.
> >>>>>>>
> >>>>>>>When I first tried a guest on 3.2 with "clocksource=pit nohpet"
> >>>>>>>I noticed that it picked the pm timer. Adding "nopmtimer", the
> >>>>>>>guest will pick the pit.
> >>>>>>>
> >>>>>>>The reason I didn't have the problem with our 3.1 base is that
> >>>>>>>I had disabled the hpet and the pmtimer by not advertising them
> >>>>>>>in the acpi tables. I did this so long ago, I forgot
> >>>>>>>
> >>>>>>>
> >>that I had to
> >>
> >>
> >>>>>>>disable pmtimer as well as hpet.
> >>>>>>>
> >>>>>>>So, can you re-run your test with "clocksource=pit nohpet
> >>>>>>>
> >>>>>>>
> >>>>>nopmtimer"?
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>You should see this in the boot messages:
> >>>>>>>
> >>>>>>>time.c: Using PIT/TSC based timekeeping.
> >>>>>>>
> >>>>>>>Thanks,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>
> >>>>>>>Dave Winchell wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>Hi Dan,
> >>>>>>>>
> >>>>>>>>Over the weekend the drift was +18 seconds for each
> >>>>>>>>
> >>>>>>>>
> >>guest (no ntp).
> >>
> >>
> >>>>>>>>The duration was 3900 minutes, so the error for each
> >>>>>>>>
> >>>>>>>>
> >>was +.0077%.
> >>
> >>
> >>>>>>>>Looking back through the data, it appears to drift linearly at
> >>>>>>>>this rate. I've attached a plot for rh4u5-64.
> >>>>>>>>
> >>>>>>>>This accuracy is better than what I've seen before (.03-.05%).
> >>>>>>>>This may be due to the different load (ltp vs usex) or to
> >>>>>>>>
> >>>>>>>>
> >>>>>one of the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>changes I've made recently. I'll do some
> >>>>>>>>
> >>>>>>>>
> >>experimentation to see if
> >>
> >>
> >>>>>>>>there is
> >>>>>>>>a fix I should propose.
> >>>>>>>>
> >>>>>>>>This still doesn't address the radical drift you saw.
> >>>>>>>>The next step for me is to run 3.2 and see if I can
> >>>>>>>>
> >>>>>>>>
> >>reproduce it.
> >>
> >>
> >>>>>>>>Regards,
> >>>>>>>>Dave
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>Dave Winchell wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Hi Dan,
> >>>>>>>>>
> >>>>>>>>>Sorry it took me so long, but I finally ran an ltp 
> test today.
> >>>>>>>>>Its on rh4u4-64. I'm using the defaults for ltp and
> >>>>>>>>>
> >>>>>>>>>
> >>using a script
> >>
> >>
> >>>>>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
> >>>>>>>>>virtual processors / physical processors = 2.
> >>>>>>>>>
> >>>>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in
> >>>>>>>>>
> >>>>>>>>>
> >>300 minutes
> >>
> >>
> >>>>>>>>>for -.005% and .008%.
> >>>>>>>>>
> >>>>>>>>>I'm running a 3.1 based hypervisor with some clock related
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>tweaks that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>I haven't submitted, because I'm still characterizing them.
> >>>>>>>>>
> >>>>>>>>>I'm stopping the usex load on 4u5-64 now and
> >>>>>>>>>
> >>>>>>>>>
> >>replacing it with ltp
> >>
> >>
> >>>>>>>>>and will leave the two guests running ltp over the weekend.
> >>>>>>>>>
> >>>>>>>>>Regards,
> >>>>>>>>>Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>Dave Winchell wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>Hi Dan, Deepak:
> >>>>>>>>>>
> >>>>>>>>>>Thanks for the data. Those drifts are severe - no wonder
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>ntp couldn't
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>keep then in synch. I'll try to reproduce that behaviour
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>here, with
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>my code base.
> >>>>>>>>>>If I can't reproduce it, I'll try 3.2.
> >>>>>>>>>>
> >>>>>>>>>>If you can isolate what ltp is doing during the cliffs,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>that would
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>be very
> >>>>>>>>>>helpful.
> >>>>>>>>>>
> >>>>>>>>>>thanks,
> >>>>>>>>>>Dave
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>OK, Deepak repeated the test without ntpd and using
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>ntpdate -b before
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>the test.
> >>>>>>>>>>>
> >>>>>>>>>>>The attached graph shows his results: el5u1-64
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>(best=~0.07%),
> >>
> >>
> >>>>>>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
> >>>>>>>>>>>
> >>>>>>>>>>>We will continue to look at LTP to try to isolate.
> >>>>>>>>>>>
> >>>>>>>>>>>Thanks,
> >>>>>>>>>>>Dan
> >>>>>>>>>>>
> >>>>>>>>>>>P.S. elXuY is essentially RHEL XuY with some patches.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
> >>>>>>>>>>>>To: Deepak Patel
> >>>>>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
> >>>>>>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>Dave Winchell
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>that disables
> >>
> >>
> >>>>>>>>>>>>pending
> >>>>>>>>>>>>missed ticks
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>Dan, Deeepak,
> >>>>>>>>>>>>
> >>>>>>>>>>>>It may be that the underlying clock error is too
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>great for ntp
> >>
> >>
> >>>>>>>>>>>>to handle. It would be useful if you did not run ntpd
> >>>>>>>>>>>>and, instead did ntpdate -b <timeserver> at the start
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>of the test
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>for each guest. Then capture the data as you have
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>been doing.
> >>
> >>
> >>>>>>>>>>>>If the drift is greater than .05%, then we need to
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>address that.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>Another option is, when running ntpd, to enable loop
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>statistics in
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>/etc/ntp.conf
> >>>>>>>>>>>>by adding this to the file:
> >>>>>>>>>>>>
> >>>>>>>>>>>>statistics loopstats
> >>>>>>>>>>>>statsdir /var/lib/ntp/
> >>>>>>>>>>>>
> >>>>>>>>>>>>Then you will see loop data in that directory.
> >>>>>>>>>>>>Correlating the data in the loopstats files with the
> >>>>>>>>>>>>peaks in skew would be interesting. You will see
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>entries of the form
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>239.735511 10
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>Where the second to last column is the Allan Deviation.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>When that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>gets over 1000, ntpd is working pretty hard. However,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>I have not
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>seen ntpd
> >>>>>>>>>>>>completely loose it like you have.
> >>>>>>>>>>>>
> >>>>>>>>>>>>I'm on vacation until Monday, and won't be reading
> >>>>>>>>>>>>email.
> >>>>>>>>>>>>
> >>>>>>>>>>>>Thanks for all your work on this!
> >>>>>>>>>>>>
> >>>>>>>>>>>>-Dave
> >>>>>>>>>>>>
> >>>>>>>>>>>>Deepak Patel wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>this one.)
> >>
> >>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>I do not know which graph was attached with this. But
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>I saw this
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>guests when I
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>was running ltp tests continuously.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>What was the behaviour of the other guests running?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>All pvm guests are fine. But behavior of most of the
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>hvm guests were
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>as described.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>If they had spikes, were they at the same wall time?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>No. They are not at the same wall time.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>Were the other guests running ltp as well?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>running ltp
> >>
> >>
> >>>>>>>>>>>>>continuously.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>How are you measuring skew?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>I was collecting output of "ntpdate -q  
> <timeserver> every
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>300 seconds
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>(5 minutes) and have created graph based on that.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>Are you running ntpd?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>Yes. ntp was running on all the guests.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>I am investigating what causes this spikes and
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>let everyone
> >>
> >>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>know what
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>are my findings.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>Deepak
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>Anything that you can discover that would be in 
> sync with
> >>>>>>>>>>>>>>the spikes would be very helpful!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>The code that I test with is our product code,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>which is based
> >>
> >>
> >>>>>>>>>>>>>>on 3.1. So it is possible that something in 3.2 other
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>than vpt.c
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>is the cause. I can test with 3.2, if necessary.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>thanks,
> >>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Hi Dave (Keir, see suggestion below) --
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Turning off vhpet certainly helps a lot (though
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>see below).
> >>
> >>
> >>>>>>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>should be
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>until it is
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>fixed?  (I have a patch that defaults it off,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>can post it if
> >>
> >>
> >>>>>>>>>>>>>>>there is agreement on the above point.)  The whole
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>point of an
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>HPET is to provide more precise timekeeping and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>if vhpet is
> >>
> >>
> >>>>>>>>>>>>>>>worse than vpit, it can only confuse users.  Comments?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>In your testing, are you just measuring % skew
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>over a long
> >>
> >>
> >>>>>>>>>>>>>>>period of time?
> >>>>>>>>>>>>>>>We are graphing the skew continuously and
> >>>>>>>>>>>>>>>seeing periodic behavior that is unsettling,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>even with pit.
> >>
> >>
> >>>>>>>>>>>>>>>See attached.  Though your algorithm recovers,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>the "cliffs"
> >>
> >>
> >>>>>>>>>>>>>>>could still cause real user problems.  I wonder
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>if there is
> >>
> >>
> >>>>>>>>>>>>>>>anything that can be done to make the "recovery" more
> >>>>>>>>>>>>>>>responsive?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>We are looking into what part(s) of LTP is causing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>the cliffs.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM
> >>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>>deepak.patel@oracle.com;
> >>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>that disables
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>pending
> >>>>>>>>>>>>>>>>missed ticks
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Dan,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>I guess I'm a bit out of date calling for 
> clock= usage.
> >>>>>>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>should specify
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are.
> >>>>>>>>>>>>>>>>The xen and guest clocksources do not need to
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>be the same.
> >>
> >>
> >>>>>>>>>>>>>>>>In my tests, xen is using the hpet for its
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>timekeeping and
> >>
> >>
> >>>>>>>>>>>>>>>>that appears to be the default.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>When you boot the guests you should see
> >>>>>>>>>>>>>>>>time.c: Using PIT/TSC based timekeeping.
> >>>>>>>>>>>>>>>>on the rh4u5-64 guest, and something similar
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>on the others.
> >>
> >>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>Platform timer
> >>
> >>
> >>>>>>>>>>>>>>>>>14.318MHz HPET.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>This appears to be the xen state, which is fine.
> >>>>>>>>>>>>>>>>I was wrongly assuming that this was the guest state.
> >>>>>>>>>>>>>>>>You might want to look in your guest logs and see
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>what they were
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>picking
> >>>>>>>>>>>>>>>>for a clock source.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Thanks, I hadn't realized that!  No wonder we didn't
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>see the same
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>improvement you saw!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Try specifying clock=pit on the linux boot line...
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>I'm confused... do you mean "clocksource=pit"
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>on the Xen
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>command line or
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>guest (or
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>dom0?) command
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>line?  Or both places?  Since the tests take
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>awhile, it
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>would be nice
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>to get this right the first time.  Do the Xen
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>and guest
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>clocksources need
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>to be the same?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>>>>>>*From:* Dave Winchell
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>[mailto:dwinchell@virtualiron.com]
> >>
> >>
> >>>>>>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
> >>>>>>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>deepak.patel@oracle.com;
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>that disables
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>pending missed ticks
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Hi Dan,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Hpet timer does have a fairly large error, as I
> >>>>>>>>>>>>>>>>>was
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>trying this
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>one recently.
> >>>>>>>>>>>>>>>>>I don't remember what I got for error, but 1% sounds
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>about right.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>The problem is that hpet is not built on top 
> of vpt.c,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>the module
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Keir and I did
> >>>>>>>>>>>>>>>>>all the recent work in, for its periodic timer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>needs. Try
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>specifying clock=pit
> >>>>>>>>>>>>>>>>>on the linux boot line. If it still picks the
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>hpet, which it
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>might, let me know
> >>>>>>>>>>>>>>>>>and I'll tell you how to get around this.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>--------------------------------------------------------------
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>----------
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>*From:* Dan Magenheimer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>[mailto:dan.magenheimer@oracle.com]
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>*Sent:* Fri 1/25/2008 6:50 PM
> >>>>>>>>>>>>>>>>>*To:* Dave Winchell; Keir Fraser
> >>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>deepak.patel@oracle.com;
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>akira.ijuin@oracle.com
> >>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>that disables
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>pending missed ticks
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Sorry for the very late followup on this but
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>we finally
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>were able
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>to get our testing set up again on stable 3.1
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>bits and have
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>seen some very bad results on 3.1.3-rc1, on the
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>order of 1%.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>Test enviroment was a 4-socket dual core machine
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>with 24GB of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>memory running six two-vcpu 2GB domains, four hvm
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>plus two pv.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>All six guests were running LTP simultaneously.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>The four hvm
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>RHEL4u5-64.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Timer_mode was set to 2 for 64-bit guests and 0 for
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>32-bit guests.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>All four hvm guests experienced skew around -1%,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>even the 32-bit
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>guest.  Less intensive testing didn't exhibit much
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>skew at all.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>A representative graph is attached.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Dave, I wonder if some portion of your patches
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>didn't end up in
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>the xen trees?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>Platform timer
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>14.318MHz HPET.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>P.S. Many thanks to Deepak and Akira for
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>running tests.
> >>
> >>
> >>>>>>>>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>Dave Winchell
> >>>>>>>>>>>>>>>>>>Sent: Wednesday, January 09, 2008 9:53 AM
> >>>>>>>>>>>>>>>>>>To: Keir Fraser
> >>>>>>>>>>>>>>>>>>Cc: dan.magenheimer@oracle.com;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dave
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Winchell
> >>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>timer mode that
> >>
> >>
> >>>>>>>>>>>>>>>>>>disables pending
> >>>>>>>>>>>>>>>>>>missed ticks
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Hi Keir,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>The latest change, c/s 16690, looks fine.
> >>>>>>>>>>>>>>>>>>I agree that the code in c/s 16690 is equivalent to
> >>>>>>>>>>>>>>>>>>the code I submitted. Also, your version is more
> >>>>>>>>>>>>>>>>>>concise.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>The error tests confirm the equivalence. With
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>overnight cpu loads,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>the checked in version was accurate to
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>+.048% for sles
> >>
> >>
> >>>>>>>>>>>>>>>>>>and +.038% for red hat. My version was +.046%
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>and
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>+.032% in a
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>2 hour test.
> >>>>>>>>>>>>>>>>>>I don't think the difference is significant.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>i/o loads produced errors of +.01%.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Thanks for all your efforts on this issue.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Keir Fraser wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>Applied as c/s 16690, although the
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>checked-in patch is
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>>smaller. I think the
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>only important fix is to pt_intr_post() and the
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>only bit of
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>the patch I
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>totally omitted was the change to
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>pt_process_missed_ticks().
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>I don't think
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>that change can be important, but let's see what
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>happens to the
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>error
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>percentage...
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>-- Keir
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>On 4/1/08 23:24, "Dave Winchell"
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>Hi Dan and Keir,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>Attached is a patch that fixes some
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>issues with the
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>SYNC policy
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>(no_missed_ticks_pending).
> >>>>>>>>>>>>>>>>>>>>I have not tried to make the change the
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>minimal one, but,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>rather, just
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>ported into
> >>>>>>>>>>>>>>>>>>>>the new code what I know to work well.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>The error for
> >>
> >>
> >>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending goes from
> >>>>>>>>>>>>>>>>>>>>over 3% to .03% with this change according
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>to my testing.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>Hi Dave --
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>Did you get your correction ported?  If so,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>it would be
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>nice to see this get
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>into 3.1.3.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>Note that I just did some very limited
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>testing with
> >>
> >>
> >>>>>>>>>>>>>>>>>>timer_mode=2(=SYNC=no
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>missed ticks pending)
> >>>>>>>>>>>>>>>>>>>>>on tip of xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>guest) and the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>worst error I've
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>seen so far
> >>>>>>>>>>>>>>>>>>>>>is 0.012%.  But I haven't tried any exotic
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>loads, just LTP.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>>>>>>>>>>>From: Dave Winchell
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>[mailto:dwinchell@virtualiron.com]
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>>>>>>>>>>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>timer mode that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>disables pending
> >>>>>>>>>>>>>>>>>>>>>>missed ticks
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>Dan,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>I did some testing with the constant tsc offset
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>SYNC method
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>(now called
> >>>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending)
> >>>>>>>>>>>>>>>>>>>>>>and found the error to be very high, much larger
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>than 1 %, as
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>I recall.
> >>>>>>>>>>>>>>>>>>>>>>I have not had a chance to submit a
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>correction. I
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>will try to
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>do it later
> >>>>>>>>>>>>>>>>>>>>>>this week or the first week in January. My
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>version of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>constant tsc
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>offset SYNC method
> >>>>>>>>>>>>>>>>>>>>>>produces .02 % error, so I just need to port
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>that into the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>current code.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>The error you got for both of those kernels is
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>what I would
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>expect
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>for the default mode, delay_for_missed_ticks.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>I'll let Keir answer on how to set the
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>time mode.
> >>
> >>
> >>>>>>>>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>Anyone make measurements on the final patch?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>saw a loss of
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>about 0.2% with no load.  This was
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>xen-unstable tip today
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>with no options specified.  32-bit was
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>about 0.01%.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>I think I missed something... how do I
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>run the various
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>accounting choices and which ones are
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>known to be
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>appropriate
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>for which kernels?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>Keir Fraser
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>>>>>>>>>>>>>>>To: Dave Winchell
> >>>>>>>>>>>>>>>>>>>>>>>>Cc: Shan, Haitao;
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Eddie; Jiang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>Yunhong
> >>>>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>mode that
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>disables pending
> >>>>>>>>>>>>>>>>>>>>>>>>missed ticks
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>Please take a look at xen-unstable
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>changeset 16545.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>-- Keir
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Keir,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>The accuracy data I've collected for i/o
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>loads for the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>various time protocols follows. In
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>addition, the data
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>for cpu loads is shown.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>processor AMD
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>box.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Two guests, red hat and sles 64 bit, 8
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>vcpu each.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>>>>>>>>>>>>>>>>(usex is available at
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>of=/dev/null.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-32 are 32
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>instances of dd.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>>>>>>>>>>>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>>>>>>>>>>>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-4/32 are the same
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>as i/o-32
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>instances of dd.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>+4.42
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec -.006%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>+.005% cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec, -.001%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>+.012% cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.009%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>-.004% cpu
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.005%, -.005% cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.008%, -.003% cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.040% cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec, -.034%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>-.031% cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec,-55.7 sec, -.01%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>-.09% i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec,-14.0 sec, -.015%
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>-.14% i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.017%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>-.022% i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.017%, -.018% i/o-8
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.020%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>-.029% i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec, -.023%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>-.031% i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/21 28 min MIXED -2.01 sec, -.67
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>sec, -.12%,
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.04% i/o-32
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.011%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>-.005% i/o-32
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>sec -.10%,
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.11% i/o-32
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>13. sec -.07%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>.003% i/o-4/32
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.017%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>.01% i/o-4/32
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Overhead measurements:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Progress in terms of number of passes
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>through a fixed
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>system workload
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>on an 8 vcpu red hat with an 8 vcpu
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>sles idle.
> >>
> >>
> >>>>>>>>>>>>>>>>>>>>>>>>>The workload was usex -b48.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>>>>>>>>>>>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>>>>>>>>>>>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>>>>>>>>>>>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Conclusions:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>The only protocol which meets the
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>.05% accuracy
> >>
> >>
> >>>>>>>>>>>>>>>>>>requirement for ntp
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>tracking under the loads
> >>>>>>>>>>>>>>>>>>>>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>accuracies for
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>SYNC, MIXED,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>and ASYNC
> >>>>>>>>>>>>>>>>>>>>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>We could reduce the cost of the SYNC
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>method by only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>scheduling the extra
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>wakeups if a certain number
> >>>>>>>>>>>>>>>>>>>>>>>>>of ticks are missed.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>Keir Fraser wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>ASYNC method a
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>couple of days ago,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>I ran another ASYNC test. I think
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>there may have
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>been something
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>days ago for
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>ASYNC. It may have been
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>after context
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>switch in.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>My results indicate that either SYNC
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>or ASYNC give
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>acceptable accuracy,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>each running consistently around or under
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>.01%. MIXED has
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>a fairly high
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>error of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>greater than .03%. Probably too close
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>to .05% ntp
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>threshold for comfort.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>I don't have an overnight run with SYNC. I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>plan to leave
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>SYNC running
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>over the weekend. If you'd rather I can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>leave MIXED
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>running instead.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>It may be too early to pick the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>protocol and
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>I can run
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>more overnight tests
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>I'm a bit worried about any unwanted side
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>effects of the
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>SYNC+run_timer
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>cause higher
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>system-wide CPU
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>contention. I find it easier to think
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>through the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>implications of ASYNC. I'm
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>accurate than
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>ASYNC. Perhaps it
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>delivers more timer interrupts than
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>the other
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>approaches,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>and each interrupt
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>favourites and
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>if the latter is
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>actually more accurate then I can
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>simply revert the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>changeset that
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>implemented MIXED.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>workloads you
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>could try idle
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>large disc reads
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>to /dev/null)? We
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>don't have any data on workloads that aren't
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>CPU bound, so
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>that's really an
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>-- Keir
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>_______________________________________________
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>Xen-devel mailing list
> >>>>>>>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>>>>>>>>>>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>2007 +0000
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>2008 -0500
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>@@ -58,7 +58,7 @@ static void
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>pt_process_missed_ticks(stru
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>    missed_ticks = missed_ticks / (s_time_t)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>pt->period + 1;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>    if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>no_missed_ticks_pending) )
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>>>>>>>>>>>+        pt->do_not_freeze = 1;
> >>>>>>>>>>>>>>>>>>>>    else
> >>>>>>>>>>>>>>>>>>>>        pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>>>>>>>>>>>    pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>>>>>>>>>>>@@ -127,7 +127,12 @@ static void
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>pt_timer_fn(void *data)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>    pt_lock(pt);
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>-    pt->pending_intr_nr++;
> >>>>>>>>>>>>>>>>>>>>+    if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>no_missed_ticks_pending) ) {
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>+        pt->pending_intr_nr = 1;
> >>>>>>>>>>>>>>>>>>>>+ pt->do_not_freeze = 0;
> >>>>>>>>>>>>>>>>>>>>+    }
> >>>>>>>>>>>>>>>>>>>>+    else
> >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr++;
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>    if ( !pt->one_shot )
> >>>>>>>>>>>>>>>>>>>>    {
> >>>>>>>>>>>>>>>>>>>>@@ -221,8 +226,6 @@ void pt_intr_post(struct
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>vcpu *v, struct
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>        return;
> >>>>>>>>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>-    pt->do_not_freeze = 0;
> >>>>>>>>>>>>>>>>>>>>-
> >>>>>>>>>>>>>>>>>>>>    if ( pt->one_shot )
> >>>>>>>>>>>>>>>>>>>>    {
> >>>>>>>>>>>>>>>>>>>>        pt->enabled = 0;
> >>>>>>>>>>>>>>>>>>>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>*v, struct
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>            pt->last_plt_gtime =
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>hvm_get_guest_time(v);
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>            pt->pending_intr_nr = 0; /*
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>'collapse' all
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>missed ticks */
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>>>>>>>>+ else if ( mode_is(v->domain,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>no_missed_ticks_pending) ) {
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>+     pt->pending_intr_nr--;
> >>>>>>>>>>>>>>>>>>>>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>>>>>>>>>>>+ }
> >>>>>>>>>>>>>>>>>>>>        else
> >>>>>>>>>>>>>>>>>>>>        {
> >>>>>>>>>>>>>>>>>>>>            pt->last_plt_gtime +=
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>pt->period_cycles;
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>_______________________________________________
> >>>>>>>>>>>>>>>>>>Xen-devel mailing list
> >>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>--------------------------------------------------------------
> >>>>>>>----------
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >
> >
> >
> 
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

> > >percentage.  However, our 32-bit clock skew seems to
> > >show a measureable problem now.
> > >
> > For the 32 bit guest, which timesource did it pick?
> 
> The dmesg output is hard to interpret on a 32-bit guest,
> but based on what we've seen, I think it was selecting
> hpet as timesource (because we specified clocksource=pit
> which would have been ignored on RHEL4-32).  We are
> running another test with "clock=pit" to see if the
> skew goes away.

Yep, that was the problem.  No clock skew with clock=pit
on RHEL4-32 anymore.

> > >For Xen RHEL5 HVM guests:
> > >- I *think* clock=pit is sufficient for RHEL5-32
> > >
> > But still poor accuracy, right?
> 
> Unproven yet but I hope not.  The nohpet and nopmtimer
> parameters are ignored on RHEL5-32 so the clock=pit
> (or clocksource=pit) is the only way to choose the
> clock source, and thus the only way to get good
> accuracy on RHEL5-32.

We had an interesting firedrill with RHEL5-32.
It seems that the RHEL-specific "divider=10" doesn't
get along very well with "clock=pit" and causes the
system clock to go whacko, gaining literally HOURS
every second.  Until Deepak discovered this, it was
impossible to boot.  So we haven't tested this
hypothesis yet, but the dmesg output looks promising.

Dan

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

Hi Dave --

I've looked into RHEL5-64 a bit more and would appreciate
any thoughts you might have:

> >So, some open questions:
> >[2] In RHEL5, I *think* it is the WALL source that we care about?
> >
> >
> I'll have to check on this too.

On second look, I may be wrong.  The GTOD clock seems to be
the one associated with vxtime.mode and vxtime.mode is used in
linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
determine if a tick is lost or not.  Is this the code that we
want timer_mode=2 to influence?

> >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> >    (I *think* not as it has never come up in any email.)
> >
> >
> I have not investigated this yet.

Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
it appears whether notsc is required or not depends in part on the
underlying virtual AND physical system.

notsc definitely is involved in the selection of GTOD time
but notsc can get set not only by kernel command line parameter
but also by the result of unsynchronized_tsc():

if (apic_is_clustered_box) notsc=1
if (box_is_Intel) {
	/* C3 delay is worst case HW latency to enter/exit C3 state */
	if (C3 delay < 1000) notsc=1
}
else { /* e.g. AMD */
	if (num_present_cpus() > 1) notsc=1
}

I'm not sure what constitutes a clustered HVM guest or how that
C3 state latency is determined under Xen, but its clear that the
clocksource can be influenced not only by what clock hardware is
present and command-line parameters but also by the physical CPU
and number of guest vcpus.

Yuck!

Thanks,
Dan

(progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Dan,

I've added a few comments below ...

Lately I've been busy getting the hpet emulation to be accurate.
Right now I have the hpet based clock error down to under .02% with
a limited amount of testing. (The error was 1% or more before my changes.)
I hope to submit a patch soon so others can try this out. I need to do more
testing and reduce my changes to the minimal set required.

One thing I really like about the hpet is that a test designed to see if
the clock will go backwards passes on the hpet where it fails on
the pit/tsc clock. I tried for quite a while to keep pit/tsc from going
backwards, but was unsuccessful. It ended up being easier to fix the hpet
emulation for accuracy.

The clock going backwards test is simply a call to gettimeofday() in a loop,
with no delay between calls except to check for new_time < prev_time.
I run this from a driver calling do_gettimeofday() to make it
more stressful. Then I run an instance of this on all (8) vcpus on
8 physical * 2 guests for a total of 16 vcpus running the test.
The test does a yield now and then so you can actually log into the 
system, etc.

Regards,
Dave

Dan Magenheimer wrote:

>Hi Dave --
>
>I've looked into RHEL5-64 a bit more and would appreciate
>any thoughts you might have:
>
>  
>
>>>So, some open questions:
>>>[2] In RHEL5, I *think* it is the WALL source that we care about?
>>>
>>>
>>>      
>>>
>>I'll have to check on this too.
>>    
>>
>
>On second look, I may be wrong.  The GTOD clock seems to be
>the one associated with vxtime.mode and vxtime.mode is used in
>linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
>determine if a tick is lost or not.  Is this the code that we
>want timer_mode=2 to influence?
>  
>
Yes, it is.

>  
>
>>>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>>>   (I *think* not as it has never come up in any email.)
>>>
>>>
>>>      
>>>
>>I have not investigated this yet.
>>    
>>
>
>Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
>it appears whether notsc is required or not depends in part on the
>underlying virtual AND physical system.
>
>notsc definitely is involved in the selection of GTOD time
>but notsc can get set not only by kernel command line parameter
>but also by the result of unsynchronized_tsc():
>  
>
According to the comment in time.c, unsynchronized_tsc() is guessing
whether the tsc is synchronized across processors. On most physical 
platforms they are,
and, on those physical platforms the virtualization layer will present the
same error as the hardware exibits. Right now we don't have code in xen to
make a unsynchronized host look synchronized to the guest, though that
wouldn't be very hard to do as long as the error between physical processors
remained constant.

>if (apic_is_clustered_box) notsc=1
>if (box_is_Intel) {
>	/* C3 delay is worst case HW latency to enter/exit C3 state */
>	if (C3 delay < 1000) notsc=1
>}
>else { /* e.g. AMD */
>	if (num_present_cpus() > 1) notsc=1
>}
>
>I'm not sure what constitutes a clustered HVM guest or how that
>C3 state latency is determined under Xen, but its clear that the
>clocksource can be influenced not only by what clock hardware is
>present and command-line parameters but also by the physical CPU
>and number of guest vcpus.
>  
>
I haven't looked into

>Yuck!
>
>Thanks,
>Dan
>
>
>
>  
>

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

If this is an integration of hpet into the vpt.c infrastructure then that
would be very welcome.

 -- Keir

On 25/2/08 20:01, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Hi Dan,
> 
> I've added a few comments below ...
> 
> Lately I've been busy getting the hpet emulation to be accurate.
> Right now I have the hpet based clock error down to under .02% with
> a limited amount of testing. (The error was 1% or more before my changes.)
> I hope to submit a patch soon so others can try this out. I need to do more
> testing and reduce my changes to the minimal set required.
> 
> One thing I really like about the hpet is that a test designed to see if
> the clock will go backwards passes on the hpet where it fails on
> the pit/tsc clock. I tried for quite a while to keep pit/tsc from going
> backwards, but was unsuccessful. It ended up being easier to fix the hpet
> emulation for accuracy.
> 
> The clock going backwards test is simply a call to gettimeofday() in a loop,
> with no delay between calls except to check for new_time < prev_time.
> I run this from a driver calling do_gettimeofday() to make it
> more stressful. Then I run an instance of this on all (8) vcpus on
> 8 physical * 2 guests for a total of 16 vcpus running the test.
> The test does a yield now and then so you can actually log into the
> system, etc.
> 
> Regards,
> Dave
> 
> Dan Magenheimer wrote:
> 
>> Hi Dave --
>> 
>> I've looked into RHEL5-64 a bit more and would appreciate
>> any thoughts you might have:
>> 
>>  
>> 
>>>> So, some open questions:
>>>> [2] In RHEL5, I *think* it is the WALL source that we care about?
>>>> 
>>>> 
>>>>      
>>>> 
>>> I'll have to check on this too.
>>>    
>>> 
>> 
>> On second look, I may be wrong.  The GTOD clock seems to be
>> the one associated with vxtime.mode and vxtime.mode is used in
>> linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
>> determine if a tick is lost or not.  Is this the code that we
>> want timer_mode=2 to influence?
>>  
>> 
> Yes, it is.
> 
>>  
>> 
>>>> [1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>>>>   (I *think* not as it has never come up in any email.)
>>>> 
>>>> 
>>>>      
>>>> 
>>> I have not investigated this yet.
>>>    
>>> 
>> 
>> Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
>> it appears whether notsc is required or not depends in part on the
>> underlying virtual AND physical system.
>> 
>> notsc definitely is involved in the selection of GTOD time
>> but notsc can get set not only by kernel command line parameter
>> but also by the result of unsynchronized_tsc():
>>  
>> 
> According to the comment in time.c, unsynchronized_tsc() is guessing
> whether the tsc is synchronized across processors. On most physical
> platforms they are,
> and, on those physical platforms the virtualization layer will present the
> same error as the hardware exibits. Right now we don't have code in xen to
> make a unsynchronized host look synchronized to the guest, though that
> wouldn't be very hard to do as long as the error between physical processors
> remained constant.
> 
>> if (apic_is_clustered_box) notsc=1
>> if (box_is_Intel) {
>> /* C3 delay is worst case HW latency to enter/exit C3 state */
>> if (C3 delay < 1000) notsc=1
>> }
>> else { /* e.g. AMD */
>> if (num_present_cpus() > 1) notsc=1
>> }
>> 
>> I'm not sure what constitutes a clustered HVM guest or how that
>> C3 state latency is determined under Xen, but its clear that the
>> clocksource can be influenced not only by what clock hardware is
>> present and command-line parameters but also by the physical CPU
>> and number of guest vcpus.
>>  
>> 
> I haven't looked into
> 
>> Yuck!
>> 
>> Thanks,
>> Dan
>> 
>> 
>> 
>>  
>> 
> 

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Hi Keir -

Keir Fraser wrote:

>If this is an integration of hpet into the vpt.c infrastructure then that
>would be very welcome.
>  
>
So far it is not, however I may head in that direction.
Last night's test had an error of .065% on one guest so I still have some
work to do.

-Dave

> -- Keir
>
>On 25/2/08 20:01, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>Hi Dan,
>>
>>I've added a few comments below ...
>>
>>Lately I've been busy getting the hpet emulation to be accurate.
>>Right now I have the hpet based clock error down to under .02% with
>>a limited amount of testing. (The error was 1% or more before my changes.)
>>I hope to submit a patch soon so others can try this out. I need to do more
>>testing and reduce my changes to the minimal set required.
>>
>>One thing I really like about the hpet is that a test designed to see if
>>the clock will go backwards passes on the hpet where it fails on
>>the pit/tsc clock. I tried for quite a while to keep pit/tsc from going
>>backwards, but was unsuccessful. It ended up being easier to fix the hpet
>>emulation for accuracy.
>>
>>The clock going backwards test is simply a call to gettimeofday() in a loop,
>>with no delay between calls except to check for new_time < prev_time.
>>I run this from a driver calling do_gettimeofday() to make it
>>more stressful. Then I run an instance of this on all (8) vcpus on
>>8 physical * 2 guests for a total of 16 vcpus running the test.
>>The test does a yield now and then so you can actually log into the
>>system, etc.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Hi Dave --
>>>
>>>I've looked into RHEL5-64 a bit more and would appreciate
>>>any thoughts you might have:
>>>
>>> 
>>>
>>>      
>>>
>>>>>So, some open questions:
>>>>>[2] In RHEL5, I *think* it is the WALL source that we care about?
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>I'll have to check on this too.
>>>>   
>>>>
>>>>        
>>>>
>>>On second look, I may be wrong.  The GTOD clock seems to be
>>>the one associated with vxtime.mode and vxtime.mode is used in
>>>linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
>>>determine if a tick is lost or not.  Is this the code that we
>>>want timer_mode=2 to influence?
>>> 
>>>
>>>      
>>>
>>Yes, it is.
>>
>>    
>>
>>> 
>>>
>>>      
>>>
>>>>>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>>>>>  (I *think* not as it has never come up in any email.)
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>I have not investigated this yet.
>>>>   
>>>>
>>>>        
>>>>
>>>Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
>>>it appears whether notsc is required or not depends in part on the
>>>underlying virtual AND physical system.
>>>
>>>notsc definitely is involved in the selection of GTOD time
>>>but notsc can get set not only by kernel command line parameter
>>>but also by the result of unsynchronized_tsc():
>>> 
>>>
>>>      
>>>
>>According to the comment in time.c, unsynchronized_tsc() is guessing
>>whether the tsc is synchronized across processors. On most physical
>>platforms they are,
>>and, on those physical platforms the virtualization layer will present the
>>same error as the hardware exibits. Right now we don't have code in xen to
>>make a unsynchronized host look synchronized to the guest, though that
>>wouldn't be very hard to do as long as the error between physical processors
>>remained constant.
>>
>>    
>>
>>>if (apic_is_clustered_box) notsc=1
>>>if (box_is_Intel) {
>>>/* C3 delay is worst case HW latency to enter/exit C3 state */
>>>if (C3 delay < 1000) notsc=1
>>>}
>>>else { /* e.g. AMD */
>>>if (num_present_cpus() > 1) notsc=1
>>>}
>>>
>>>I'm not sure what constitutes a clustered HVM guest or how that
>>>C3 state latency is determined under Xen, but its clear that the
>>>clocksource can be influenced not only by what clock hardware is
>>>present and command-line parameters but also by the physical CPU
>>>and number of guest vcpus.
>>> 
>>>
>>>      
>>>
>>I haven't looked into
>>
>>    
>>
>>>Yuck!
>>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>> 
>>>
>>>      
>>>
>
>
>  
>

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 26/2/08 14:45, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

>> If this is an integration of hpet into the vpt.c infrastructure then that
>> would be very welcome.
>>
> So far it is not, however I may head in that direction.
> Last night's test had an error of .065% on one guest so I still have some
> work to do.

Then I hope that the accuracy improvements have come from a set of changes
whose effects are both explicable and generally for the good (rather than
artefacts of how a particular sub-point release of Linux drives the HPET).

 -- Keir

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir Fraser wrote:

>On 26/2/08 14:45, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>>If this is an integration of hpet into the vpt.c infrastructure then that
>>>would be very welcome.
>>>
>>>      
>>>
>>So far it is not, however I may head in that direction.
>>Last night's test had an error of .065% on one guest so I still have some
>>work to do.
>>    
>>
>
>Then I hope that the accuracy improvements have come from a set of changes
>whose effects are both explicable and generally for the good (rather than
>artefacts of how a particular sub-point release of Linux drives the HPET).
>  
>
I believe that my changes meet these requirements. Your suggestion to layer
on vpt.c is a good one that I will pursue.

- Dave

> -- Keir
>
>
>  
>

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir, Dan:

This is an update on the hpet work. With limited testing, the accuracy
of this method appears to be at least 10 times better than the
pit/tsc method for usex loads. Also, as mentioned before, it does
not have the time going backwards problem. Another interesting property
is that the same policy is used for a 32 bit Linux guest and 64 bit Linux.

Two recent 14-16 hour tests were run, one with usex e48 and the other with
usex b48 as loads. The same load was run on three 8 vcpu guests on an 8 cpu
platform. No ntpd running. ntpdate -b used for initial setting of clock and
ntpdate -q used for monitoring drift. The guests were 4u464, 4u564, 4u432
Linux. The results are:

test      duration   drift (secs) 4u464,4u564,4u432      drift %

usex b48    16 hrs       -.68, -.60, -.68                 -.0012
usex e48    15 hrs       -.58, -.55, -.58                 -.0011


The drift with pit/tsc as checked in reported a few months ago:

>> The error tests confirm the equivalence. With overnight cpu loads,
>> the checked in version was accurate to +.048% for sles
>> and +.038% for red hat.

Recall that .05% is the goal for accuracy so that ntpd can synchronize.
With pit/tsc we are rather close to that limit. So far, with hpet,
we are well under that goal.

As for the code structure, I tried layering on vpt.c and had trouble
doing so. Therefore, I focused on a direct approach. I'll describe the approach
shortly, after I run tests with other loads and guests.


Regards,
Dave



Keir Fraser wrote:

>On 26/2/08 14:45, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>>If this is an integration of hpet into the vpt.c infrastructure then that
>>>would be very welcome.
>>>
>>>      
>>>
>>So far it is not, however I may head in that direction.
>>Last night's test had an error of .065% on one guest so I still have some
>>work to do.
>>    
>>
>
>Then I hope that the accuracy improvements have come from a set of changes
>whose effects are both explicable and generally for the good (rather than
>artefacts of how a particular sub-point release of Linux drives the HPET).
>
> -- Keir
>
>
>  
>

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 5/3/08 15:06, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> As for the code structure, I tried layering on vpt.c and had trouble
> doing so. Therefore, I focused on a direct approach. I'll describe the
> approach
> shortly, after I run tests with other loads and guests.

Are you targeting only 64-bit Linux guests? 32-bit Linux cannot tolerate
missed ticks, and I want only one missed-tick algorithm in Xen: currently
that is vpt.c.

 -- Keir

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir Fraser wrote:

>On 5/3/08 15:06, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>As for the code structure, I tried layering on vpt.c and had trouble
>>doing so. Therefore, I focused on a direct approach. I'll describe the
>>approach
>>shortly, after I run tests with other loads and guests.
>>    
>>
>
>Are you targeting only 64-bit Linux guests? 32-bit Linux cannot tolerate
>missed ticks, and I want only one missed-tick algorithm in Xen: currently
>that is vpt.c.
>  
>
No, my results are for 32 bit Linux as well, at least 4u432.
It turns out that for hpet, (rh4u4) 32 bit Linux does handle missed ticks.
I need to try other 32 bit Linux versions, as well as Windows, to see how
broad this behaviour is.

Looking at the code for 2.6.20, it appears to me that if CONFIG_GENERIC_TIME
is set then missed ticks are handled for hpet for 32 bit and 64
bit Linux (see update_wall_time() and read_hpet()).

In 2.6.9, it looks like cur_timer->mark_offset() call from 
timer_interrupt in
32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(), 
which computes
missed ticks based on hpet counter. mark_offset_pit() does nothing.
mark_offset_tsc() does compute missed ticks.

In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet
reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks.

So from the code perspective, it looks like missed ticks are computed 
for 32
and 64 bit Linux using hpet clocksource.

regards,
Dave

> -- Keir
>
>
>  
>

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Keir Fraser]

On 5/3/08 17:25, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> In 2.6.9, it looks like cur_timer->mark_offset() call from
> timer_interrupt in
> 32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(),
> which computes
> missed ticks based on hpet counter. mark_offset_pit() does nothing.
> mark_offset_tsc() does compute missed ticks.
> 
> In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet
> reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks.
> 
> So from the code perspective, it looks like missed ticks are computed
> for 32
> and 64 bit Linux using hpet clocksource.

Ah. I looked at 2.6.18 which seems to have neither the mark_offset nor the
GENERIC_TIME approach in its arch/i386 time code. But yeah, it does look
like in general Linux 2.6 is robust to missed ticks when using hpet. That's
good.

Do you see and simply ignore warning messages from 64-bit Linux when using
hpet (or otherwise not doing missed-tick handling in Xen), by the way? I
know 64-bit Linux is keen to warn about missed ticks, although it does look
like at least the warning is one shot.

 -- Keir

Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir Fraser wrote:

>On 5/3/08 17:25, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>
>  
>
>>In 2.6.9, it looks like cur_timer->mark_offset() call from
>>timer_interrupt in
>>32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(),
>>which computes
>>missed ticks based on hpet counter. mark_offset_pit() does nothing.
>>mark_offset_tsc() does compute missed ticks.
>>
>>In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet
>>reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks.
>>
>>So from the code perspective, it looks like missed ticks are computed
>>for 32
>>and 64 bit Linux using hpet clocksource.
>>    
>>
>
>Ah. I looked at 2.6.18 which seems to have neither the mark_offset nor the
>GENERIC_TIME approach in its arch/i386 time code. But yeah, it does look
>like in general Linux 2.6 is robust to missed ticks when using hpet. That's
>good.
>
>Do you see and simply ignore warning messages from 64-bit Linux when using
>hpet (or otherwise not doing missed-tick handling in Xen), by the way? I
>know 64-bit Linux is keen to warn about missed ticks, although it does look
>like at least the warning is one shot.
>  
>
I see the one shot messages on 64 bit and no complaints at all on 4u432 
Linux.
And I have been ignoring them.

> -- Keir
>
>
>  
>

RE: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

> >Do you see and simply ignore warning messages from 64-bit 
> Linux when using
> >hpet (or otherwise not doing missed-tick handling in Xen), 
> by the way? I
> >know 64-bit Linux is keen to warn about missed ticks, 
> although it does look
> >like at least the warning is one shot.
> >
> >
> I see the one shot messages on 64 bit and no complaints at 
> all on 4u432
> Linux.
> And I have been ignoring them.

Code to print the complaints is x86_64-specific in 2.6.9 and 2.6.18.
On 64-bit, a command line parameter of "report_lost_ticks" will
enable a complaint for every lost tick.

As of 2.6.24, the lost ticks message seems to have gone away
completely.

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Dan Magenheimer]

(One more time on this old thread...)

Hi Dave --

We have a strange but alarming result:  Running rhel5ga-64 with
timer_mode=0 has no skew but with timer_mode=2 has very bad skew.
With rhel5u1-64 (and all rhel4-64 guests) the opposite is true.
We repeated the tests to verify.  In all cases, dmesg shows:

time.c: Using 1.193182 MHz WALL PIT GTOD PIT/TSC timer.

Could you try your heavy load tests with all four combinations
of rhel5ga-64/rhel5u1-64 and timer_mode=0/2 to see if you
can reproduce our results.

Thanks,
Dan

> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: Monday, February 25, 2008 9:42 AM
> To: 'Dave Winchell'
> Cc: 'Keir Fraser'; 'xen-devel@lists.xensource.com'; 'Deepak Patel'
> Subject: RE: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Dave --
> 
> I've looked into RHEL5-64 a bit more and would appreciate
> any thoughts you might have:
> 
> > >So, some open questions:
> > >[2] In RHEL5, I *think* it is the WALL source that we care about?
> > >
> > >
> > I'll have to check on this too.
> 
> On second look, I may be wrong.  The GTOD clock seems to be
> the one associated with vxtime.mode and vxtime.mode is used in
> linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
> determine if a tick is lost or not.  Is this the code that we
> want timer_mode=2 to influence?
> 
> > >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> > >    (I *think* not as it has never come up in any email.)
> > >
> > >
> > I have not investigated this yet.
> 
> Well for RHEL5-64, looking at 
> linux-2.6.18.8/arch/x86_64/kernel/time.c,
> it appears whether notsc is required or not depends in part on the
> underlying virtual AND physical system.
> 
> notsc definitely is involved in the selection of GTOD time
> but notsc can get set not only by kernel command line parameter
> but also by the result of unsynchronized_tsc():
> 
> if (apic_is_clustered_box) notsc=1
> if (box_is_Intel) {
> 	/* C3 delay is worst case HW latency to enter/exit C3 state */
> 	if (C3 delay < 1000) notsc=1
> }
> else { /* e.g. AMD */
> 	if (num_present_cpus() > 1) notsc=1
> }
> 
> I'm not sure what constitutes a clustered HVM guest or how that
> C3 state latency is determined under Xen, but its clear that the
> clocksource can be influenced not only by what clock hardware is
> present and command-line parameters but also by the physical CPU
> and number of guest vcpus.
> 
> Yuck!
> 
> Thanks,
> Dan
> 
> 
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

> and which ones are known to be appropriate for which kernels?

The default, delay_for_missed_ticks is good for 32 bit Linux.
no_missed_ticks_pending is the most accurate for 64 bit Linux, but the version
of no_missed_ticks_pending checked in has some errors in it, as I mentioned.

Last I checked, before the most recent changeset, the version now called
one_missed_tick_pending had an error > .05 % for some loads.
This is called the MIXED method in the data I've published.

Regards,
Dave



Dan Magenheimer wrote:

>Anyone make measurements on the final patch?
>
>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no load.  This was xen-unstable tip today with no options specified.  32-bit was about 0.01%.
>
>I think I missed something... how do I run the various accounting choices and which ones are known to be appropriate for which kernels?
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
>>Sent: Thursday, December 06, 2007 4:57 AM
>>To: Dave Winchell
>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>Yunhong
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Please take a look at xen-unstable changeset 16545.
>>
>> -- Keir
>>
>>On 26/11/07 20:57, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>
>>    
>>
>>>Keir,
>>>
>>>The accuracy data I've collected for i/o loads for the
>>>various time protocols follows. In addition, the data
>>>for cpu loads is shown.
>>>
>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>The cpu load is usex -e36 on each guest.
>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>
>>>The loads labeled i/o-32 are 32 instances of dd.
>>>Also, these are run on 4 cpu AMD box.
>>>In addition, there is an idle rh-32bit guest.
>>>All three guests are 8vcpu.
>>>
>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>except that the redhat-64 guest has 4 instances of dd.
>>>
>>>Date Duration Protocol sles, rhat error load
>>>
>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>
>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>
>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>
>>>
>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>
>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>
>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>
>>>
>>>
>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>
>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>
>>>
>>>Overhead measurements:
>>>
>>>Progress in terms of number of passes through a fixed 
>>>      
>>>
>>system workload
>>    
>>
>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>The workload was usex -b48.
>>>
>>>
>>>ASYNC 167 min 145 passes .868 passes/min
>>>SYNC 167 min 144 passes .862 passes/min
>>>SYNC 1065 min 919 passes .863 passes/min
>>>MIXED 221 min 196 passes .887 passes/min
>>>
>>>
>>>Conclusions:
>>>
>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>tracking under the loads
>>>above is the SYNC protocol. The worst case accuracies for 
>>>      
>>>
>>SYNC, MIXED,
>>    
>>
>>>and ASYNC
>>>are .022%, .12%, and .14%, respectively.
>>>
>>>We could reduce the cost of the SYNC method by only 
>>>      
>>>
>>scheduling the extra
>>    
>>
>>>wakeups if a certain number
>>>of ticks are missed.
>>>
>>>Regards,
>>>Dave
>>>
>>>Keir Fraser wrote:
>>>
>>>      
>>>
>>>>On 9/11/07 19:22, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Since I had a high error (~.03%) for the ASYNC method a 
>>>>>          
>>>>>
>>couple of days ago,
>>    
>>
>>>>>I ran another ASYNC test. I think there may have been something
>>>>>wrong with the code I used a couple of days ago for 
>>>>>          
>>>>>
>>ASYNC. It may have been
>>    
>>
>>>>>missing the immediate delivery of interrupt after context 
>>>>>          
>>>>>
>>switch in.
>>    
>>
>>>>>My results indicate that either SYNC or ASYNC give 
>>>>>          
>>>>>
>>acceptable accuracy,
>>    
>>
>>>>>each running consistently around or under .01%. MIXED has 
>>>>>          
>>>>>
>>a fairly high
>>    
>>
>>>>>error of
>>>>>greater than .03%. Probably too close to .05% ntp 
>>>>>          
>>>>>
>>threshold for comfort.
>>    
>>
>>>>>I don't have an overnight run with SYNC. I plan to leave 
>>>>>          
>>>>>
>>SYNC running
>>    
>>
>>>>>over the weekend. If you'd rather I can leave MIXED 
>>>>>          
>>>>>
>>running instead.
>>    
>>
>>>>>It may be too early to pick the protocol and I can run 
>>>>>          
>>>>>
>>more overnight tests
>>    
>>
>>>>>next week.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>I'm a bit worried about any unwanted side effects of the 
>>>>        
>>>>
>>SYNC+run_timer
>>    
>>
>>>>approach -- e.g., whether timer wakeups will cause higher 
>>>>        
>>>>
>>system-wide CPU
>>    
>>
>>>>contention. I find it easier to think through the 
>>>>        
>>>>
>>implications of ASYNC. I'm
>>    
>>
>>>>surprised that MIXED loses time, and is less accurate than 
>>>>        
>>>>
>>ASYNC. Perhaps it
>>    
>>
>>>>delivers more timer interrupts than the other approaches, 
>>>>        
>>>>
>>and each interrupt
>>    
>>
>>>>event causes a small accumulated error?
>>>>
>>>>Overall I would consider MIXED and ASYNC as favourites and 
>>>>        
>>>>
>>if the latter is
>>    
>>
>>>>actually more accurate then I can simply revert the changeset that
>>>>implemented MIXED.
>>>>
>>>>Perhaps rather than running more of the same workloads you 
>>>>        
>>>>
>>could try idle
>>    
>>
>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads 
>>>>        
>>>>
>>to /dev/null)? We
>>    
>>
>>>>don't have any data on workloads that aren't CPU bound, so 
>>>>        
>>>>
>>that's really an
>>    
>>
>>>>obvious place to put any further effort imo.
>>>>
>>>>-- Keir
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>>    
>>
>
>  
>

Re: [PATCH] Add a timer mode that disables pending missed ticks

[Dave Winchell]

Keir,

I ran a 24 hour (23hr:40min) test. The usual setup. Protocol was ASYNC.

Errors:
  sles9sp3-64:  -4.96 sec  -.0058%
  rh4u4-64:     +4.42 sec  +.0052%

So, lets leave it ASYNC unless someone produces some test cases
where the error gets up to close to .05%.
I'll do some testing here with overnight runs or, perhaps,
different loads.

thanks,
Dave

Dave Winchell wrote:

> Hi Keir,
>
> I've added comments below.
> See my next mail on some interesting performance numbers.
>
> thanks,
> Dave
>
> Keir Fraser wrote:
>
>> On 7/11/07 19:38, "Dave Winchell" <dwinchell@virtualiron.com> wrote:
>>
>>  
>>
>>> My feeling is that we should go full SYNC. Yes, in theory the
>>> guests should be able to handle ASYNC, but in reality it appears that
>>> some do not. Since it is easy for us to give them SYNC,
>>> lets just do it and not stress them out.
>>>   
>>
>>
>> One problem with pure SYNC is there's a fair chance you won't deliver 
>> any
>> ticks at all for a long time, if the guest only runs in short bursts 
>> (e.g.,
>> I/O bound) and happens not to be running on any tick boundary. I'm 
>> not sure
>> how much that matters. It could cause time goes backwards if the time
>> extrapolation via the TSC is not perfectly accurate, or cause 
>> problems if
>> there are any assumptions that TSC delta since last tick fits in 32 bits
>> (less likely in x64 code I suppose). Anyway, my point is that only 
>> testing
>> VCPUs under full load may cause us to optimise in ways that have nasty
>> unexpected effects for other workloads.
>>  
>>
> I agree that this could be a problem. I have an idea that could give 
> us full
> SYNC and eliminate the long periods without clock interrupts.
> In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1.
> In pt_save_timer():
>
>    list_for_each_entry ( pt, head, list )
>        if(!pt->run_timer)
>             stop_timer(&pt->timer);
>
> And in pt_timer_fn():
>
>    pt->run_timer = 0;
>
> So, for a guest that misses a tick, we will interrupt him once from the
> descheduled state and then leave him alone in the descheduled state.
>
>>  
>>
>>> For default mode as checked into unstable is now,
>>> 64 bit guests should run quite fast as missed is calculated and then 
>>> a bunch
>>> of additional interrupts are delivered. On the other hand
>>> 32bit guests very well in default mode.
>>>
>>> For the original code, before we put in the constant tsc offset 
>>> business,
>>> 64bit guests run poorly and 32bit quests very well time-wise.
>>>   
>>
>>
>> The default mode hasn't changed. Are you under the impression that
>> missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 
>> guests
>> run badly with that because they treat every one of the missed ticks 
>> they
>> receive as a full tick.
>>  
>>
> Sorry, I was confused.
> However, the default mode will still run poorly for 64 bit guests because
> of the pending_nr's accumulated while the guest has interrupts disabled.
> As I recall, the effect is quite large, on the order of 10% error.
> I'll get you a number later today.
>
>> -- Keir
>>
>>  
>>
>>>> Or is the lack of
>>>> synchronization of TSCs across VCPUs causing issues that you're 
>>>> trying to
>>>> avoid?
>>>>     
>>>
>>> This does cause issues, but its not the only contributor to poor 
>>> timing.
>>> Having TSCs synchronized across vcpus will help some of the time going
>>> backwards problems we have seen, I think.
>>>
>>> Regards,
>>> Dave
>>>
>>> Keir Fraser wrote:
>>>
>>>   
>>>
>>>> On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>> So, you can see we send an interrupt immediately (and ASYNC) if 
>>>>> any ticks
>>>>> have been missed, but then successive ticks are delivered 'on the 
>>>>> beat'. A
>>>>> possible middleground? Or perhaps we should just go with SYNC 
>>>>> after all...
>>>>>  
>>>>>       
>>>>
>>>> How do these Linux x64 guests fare with the original and default 
>>>> timer mode,
>>>> by the way? I would expect that time should be accounted pretty 
>>>> accurately
>>>> in that mode, albeit with more interrupts than you'd like. Or is 
>>>> the lack of
>>>> synchronisation of TSCs across VCPUs causing issues that you're 
>>>> trying to
>>>> avoid?
>>>>
>>>> -- Keir
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>
>>
>>
>>  
>>
>

RE: [PATCH] Add a timer mode that disables pending missed ticks

[Shan, Haitao]

[-- Attachment #1.1: Type: text/plain, Size: 1317 bytes --]

Hi, Keir, Dave
 
Sorry, I made a silly mistake for my last patch. The attached will fix it.
 
This piece of code should actually be added in pt_restore_timer rather than pt_timer_fn.
+        else if ( (NOW() - pt->scheduled) >= 0 )
+        {
+            pt->pending_intr_nr++;
+            pt->scheduled = NOW() + pt->period;
+        }

Best Regards 
Haitao Shan 

 

________________________________

From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
Sent: 2007年10月31日 0:13
To: Shan, Haitao
Cc: xen-devel@lists.xensource.com; Dong, Eddie; Jiang, Yunhong; Dave Winchell
Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticks



Applied as c/s 16274. Please take a look and make sure the mode works as you expect.

 -- Keir

On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:



	Hi, Keir, 
	
	This patch adds a new timer mode, in which no missed ticks is calculated. This can be used with latest x86_64 linux guest, since it can pick up missed ticks themselves.
	
	 <<no_missed_ticks.patch>> 
	
	Best Regards 
	Haitao Shan 
	
	
________________________________

	_______________________________________________
	Xen-devel mailing list
	Xen-devel@lists.xensource.com
	http://lists.xensource.com/xen-devel
	




[-- Attachment #1.2: Type: text/html, Size: 3771 bytes --]

[-- Attachment #2: no_missed_ticks_fix.patch --]
[-- Type: application/octet-stream, Size: 1604 bytes --]

diff -r 7eb68d995aa7 xen/arch/x86/hvm/hvm.c
--- a/xen/arch/x86/hvm/hvm.c	Tue Oct 30 16:25:58 2007 +0000
+++ b/xen/arch/x86/hvm/hvm.c	Tue Oct 30 19:53:33 2007 +0800
@@ -1849,7 +1849,8 @@ long do_hvm_op(unsigned long op, XEN_GUE
             case HVM_PARAM_TIMER_MODE:
                 rc = -EINVAL;
                 if ( (a.value != HVMPTM_delay_for_missed_ticks) &&
-                     (a.value != HVMPTM_no_delay_for_missed_ticks) )
+                     (a.value != HVMPTM_no_delay_for_missed_ticks) &&
+                     (a.value != HVMPTM_no_missed_tick_accounting) )
                     goto param_fail;
                 break;
             }
diff -r 7eb68d995aa7 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Tue Oct 30 16:25:58 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c	Tue Oct 30 19:04:12 2007 +0800
@@ -119,6 +119,11 @@ void pt_restore_timer(struct vcpu *v)
     {
         if ( !mode_is(v->domain, no_missed_tick_accounting) )
             pt_process_missed_ticks(pt);
+        else if ( (NOW() - pt->scheduled) >= 0 )
+        {
+            pt->pending_intr_nr++;
+            pt->scheduled = NOW() + pt->period;
+        }
         set_timer(&pt->timer, pt->scheduled);
     }
 
@@ -141,11 +146,6 @@ static void pt_timer_fn(void *data)
         if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
         {
             pt_process_missed_ticks(pt);
-        }
-        else if ( (NOW() - pt->scheduled) >= 0 )
-        {
-            pt->pending_intr_nr++;
-            pt->scheduled = NOW() + pt->period;
         }
         set_timer(&pt->timer, pt->scheduled);
     }

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel