Re: [PATCH] [RFC] Building guests on monotonic Xen system time

[Keir Fraser <keir.fraser@eu.citrix.com>]

[-- Attachment #1: Type: text/plain, Size: 2894 bytes --]

Attached is a modified version (really just renames). However the new
hvm_set_guest_time() interface doesn't work for vpt.c timer modes which
allow progress of time to temporarily deviate across VCPUs. I guess you
don't use those modes so might not notice this issue. I think hvm guest time
handling may need to be integrated into vpt.c so that correct handling can
be applied for the various different types of timer mode. Having a blanket
per-domain stime offset and enforcing monotonicity regardless of timer mode
doesn't fly.

 -- Keir

On 21/5/08 20:01, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> OK, I think this version is ready for prime-time.
> 
> Keir, please check-in to unstable.
> 
> [PATCH] Build hvm guest platform timers on monotonic Xen system time
> 
> This patch moves hvm platform timers from underlying physical CPU TSC
> to Xen system time and ensures monotonicity.  TSC on many systems may
> skew between processors, thus hvm platform timer reads were not
> monotonic which led to "Time going backwards" problems.
> 
> Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Tuesday, May 20, 2008 1:37 AM
>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> Subject: Re: [Xen-devel] [PATCH] [RFC] Building guests on
>> monotonic Xen
>> system time
>> 
>> 
>> On 19/5/08 19:27, "Dan Magenheimer"
>> <dan.magenheimer@oracle.com> wrote:
>> 
>>> Why? Scaling and adjusting of xen-time-based-tsc will
>>> be very difficult to coordinate with processor-based-tsc.
>>> We need to always ensure that A < B < C for a guest
>>> executing:
>>> 
>>> rdtsc(A) /* untrapped */
>>> emulated_rdtsc(B)
>>> rdtsc(C) /* untrapped */
>>> 
>>> Further, OS's use TSC as a highest-resolution time source
>>> with knowledge that TSCs on different processors may
>>> not be synchronized, whereas they assume that a platform
>>> timer is one-per-system and monotonically increasing.
>>> 
>>> Keir, if you disagree and see guest-TSC-on-Xen-system-time
>>> as an absolute must, please let me know.
>> 
>> I am inclined to say we should have a
>> guest-TSC-on-system-time mode where
>> *all* RDTSC executions get trapped. This would at least be useful as a
>> baseline for tracking down guest time problems, and also provide a
>> guaranteed stable TSC timesource for those who care about
>> that more than
>> pure performance.
>> 
>> *However* I would agree that, with TSC virtualisation as it
>> currently is,
>> there actually isn't really a way to build guest TSC on Xen
>> system time
>> without periodically warping the TSC back or forth. The guest
>> TSC runs at
>> the host TSC rate and that is that!
>> 
>> I think my original point was that at least we should not
>> build all our
>> other virtual time sources on this dodgy 'guest TSC'. :-)
>> 
>>  -- Keir
>> 
>> 
>> 


[-- Attachment #2: stime.patch --]
[-- Type: application/octet-stream, Size: 13413 bytes --]

diff -r e64c3a8c60e1 xen/arch/x86/hvm/hpet.c
--- a/xen/arch/x86/hvm/hpet.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/hpet.c	Thu May 22 09:38:58 2008 +0100
@@ -29,9 +29,9 @@
 #define S_TO_NS  1000000000ULL           /* 1s  = 10^9  ns */
 #define S_TO_FS  1000000000000000ULL     /* 1s  = 10^15 fs */
 
-/* Frequency_of_TSC / frequency_of_HPET = 32 */
-#define TSC_PER_HPET_TICK 32
-#define guest_time_hpet(v) (hvm_get_guest_time(v) / TSC_PER_HPET_TICK)
+/* Frequency_of_Xen_systeme_time / frequency_of_HPET = 16 */
+#define STIME_PER_HPET_TICK 16
+#define guest_time_hpet(v) (hvm_get_guest_time(v) / STIME_PER_HPET_TICK)
 
 #define HPET_ID         0x000
 #define HPET_PERIOD     0x004
@@ -192,7 +192,7 @@ static void hpet_stop_timer(HPETState *h
 
 /* the number of HPET tick that stands for
  * 1/(2^10) second, namely, 0.9765625 milliseconds */
-#define  HPET_TINY_TIME_SPAN  ((h->tsc_freq >> 10) / TSC_PER_HPET_TICK)
+#define  HPET_TINY_TIME_SPAN  ((h->stime_freq >> 10) / STIME_PER_HPET_TICK)
 
 static void hpet_set_timer(HPETState *h, unsigned int tn)
 {
@@ -558,17 +558,17 @@ void hpet_init(struct vcpu *v)
     spin_lock_init(&h->lock);
 
     h->vcpu = v;
-    h->tsc_freq = ticks_per_sec(v);
-
-    h->hpet_to_ns_scale = ((S_TO_NS * TSC_PER_HPET_TICK) << 10) / h->tsc_freq;
+    h->stime_freq = S_TO_NS;
+
+    h->hpet_to_ns_scale = ((S_TO_NS * STIME_PER_HPET_TICK) << 10) / h->stime_freq;
     h->hpet_to_ns_limit = ~0ULL / h->hpet_to_ns_scale;
 
     /* 64-bit main counter; 3 timers supported; LegacyReplacementRoute. */
     h->hpet.capability = 0x8086A201ULL;
 
     /* This is the number of femptoseconds per HPET tick. */
-    /* Here we define HPET's frequency to be 1/32 of the TSC's */
-    h->hpet.capability |= ((S_TO_FS*TSC_PER_HPET_TICK/h->tsc_freq) << 32);
+    /* Here we define HPET's frequency to be 1/16 of Xen system time */
+    h->hpet.capability |= ((S_TO_FS*STIME_PER_HPET_TICK/h->stime_freq) << 32);
 
     for ( i = 0; i < HPET_TIMER_NUM; i++ )
     {
diff -r e64c3a8c60e1 xen/arch/x86/hvm/hvm.c
--- a/xen/arch/x86/hvm/hvm.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/hvm.c	Thu May 22 09:39:40 2008 +0100
@@ -152,6 +152,49 @@ u64 hvm_get_guest_tsc(struct vcpu *v)
     return host_tsc + v->arch.hvm_vcpu.cache_tsc_offset;
 }
 
+/*
+ * Guest time is Xen system time (measured in nsec) plus an offset.
+ * It is guaranteed to be monotonic
+ */
+void hvm_init_guest_time(struct hvm_domain *hd)
+{
+    struct pl_time *pl = &hd->pl_time;
+
+    spin_lock_init(&pl->pl_time_lock);
+    pl->stime_offset = -(u64)get_s_time_mono();
+    pl->last_guest_time = 0ULL;
+}
+
+void hvm_set_guest_time(struct vcpu *v, u64 guest_time)
+{
+    struct pl_time *pl = &v->domain->arch.hvm_domain.pl_time;
+    u64 now;
+
+    spin_lock(&pl->pl_time_lock);
+    now = (u64)get_s_time_mono() + pl->stime_offset;
+    if (now <= guest_time) {
+         pl->stime_offset = guest_time - now;
+         pl->last_guest_time = guest_time - now;
+    }
+    /* ignore attempts to set guest time backwards */
+    spin_unlock(&pl->pl_time_lock);
+}
+
+u64 hvm_get_guest_time(struct vcpu *v)
+{
+    struct pl_time *pl = &v->domain->arch.hvm_domain.pl_time;
+    u64 now;
+
+    spin_lock(&pl->pl_time_lock);
+    now = get_s_time_mono() + pl->stime_offset;
+    if ( now > pl->last_guest_time )
+        pl->last_guest_time = now;
+    else
+        now = pl->last_guest_time;
+    spin_unlock(&pl->pl_time_lock);
+    return now;
+}
+
 void hvm_migrate_timers(struct vcpu *v)
 {
     rtc_migrate_timers(v);
@@ -295,6 +338,8 @@ int hvm_domain_initialise(struct domain 
     spin_lock_init(&d->arch.hvm_domain.pbuf_lock);
     spin_lock_init(&d->arch.hvm_domain.irq_lock);
     spin_lock_init(&d->arch.hvm_domain.uc_lock);
+
+    hvm_init_guest_time(&d->arch.hvm_domain);
 
     d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
 
@@ -661,7 +706,7 @@ int hvm_vcpu_initialise(struct vcpu *v)
         hpet_init(v);
  
         /* Init guest TSC to start from zero. */
-        hvm_set_guest_time(v, 0);
+        hvm_set_guest_tsc(v, 0);
 
         /* Can start up without SIPI-SIPI or setvcpucontext domctl. */
         v->is_initialised = 1;
@@ -1632,7 +1677,7 @@ int hvm_msr_read_intercept(struct cpu_us
     switch ( ecx )
     {
     case MSR_IA32_TSC:
-        msr_content = hvm_get_guest_time(v);
+        msr_content = hvm_get_guest_tsc(v);
         break;
 
     case MSR_IA32_APICBASE:
@@ -1725,7 +1770,7 @@ int hvm_msr_write_intercept(struct cpu_u
     switch ( ecx )
     {
      case MSR_IA32_TSC:
-        hvm_set_guest_time(v, msr_content);
+        hvm_set_guest_tsc(v, msr_content);
         pt_reset(v);
         break;
 
diff -r e64c3a8c60e1 xen/arch/x86/hvm/i8254.c
--- a/xen/arch/x86/hvm/i8254.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/i8254.c	Thu May 22 09:38:09 2008 +0100
@@ -31,6 +31,7 @@
 #include <xen/lib.h>
 #include <xen/errno.h>
 #include <xen/sched.h>
+#include <asm/time.h>
 #include <asm/hvm/hvm.h>
 #include <asm/hvm/io.h>
 #include <asm/hvm/support.h>
@@ -87,7 +88,7 @@ static int pit_get_count(PITState *pit, 
     ASSERT(spin_is_locked(&pit->lock));
 
     d = muldiv64(hvm_get_guest_time(v) - pit->count_load_time[channel],
-                 PIT_FREQ, ticks_per_sec(v));
+                 PIT_FREQ, SYSTEM_TIME_HZ);
 
     switch ( c->mode )
     {
@@ -118,7 +119,7 @@ static int pit_get_out(PITState *pit, in
     ASSERT(spin_is_locked(&pit->lock));
 
     d = muldiv64(hvm_get_guest_time(v) - pit->count_load_time[channel], 
-                 PIT_FREQ, ticks_per_sec(v));
+                 PIT_FREQ, SYSTEM_TIME_HZ);
 
     switch ( s->mode )
     {
@@ -195,11 +196,11 @@ static void pit_load_count(PITState *pit
         val = 0x10000;
 
     if ( v == NULL )
-        rdtscll(pit->count_load_time[channel]);
+        pit->count_load_time[channel] = 0;
     else
         pit->count_load_time[channel] = hvm_get_guest_time(v);
     s->count = val;
-    period = DIV_ROUND((val * 1000000000ULL), PIT_FREQ);
+    period = DIV_ROUND(val * SYSTEM_TIME_HZ, PIT_FREQ);
 
     if ( (v == NULL) || !is_hvm_vcpu(v) || (channel != 0) )
         return;
diff -r e64c3a8c60e1 xen/arch/x86/hvm/pmtimer.c
--- a/xen/arch/x86/hvm/pmtimer.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/pmtimer.c	Thu May 22 09:38:50 2008 +0100
@@ -257,7 +257,7 @@ void pmtimer_init(struct vcpu *v)
 
     spin_lock_init(&s->lock);
 
-    s->scale = ((uint64_t)FREQUENCE_PMTIMER << 32) / ticks_per_sec(v);
+    s->scale = ((uint64_t)FREQUENCE_PMTIMER << 32) / SYSTEM_TIME_HZ;
     s->vcpu = v;
 
     /* Intercept port I/O (need two handlers because PM1a_CNT is between
diff -r e64c3a8c60e1 xen/arch/x86/hvm/svm/svm.c
--- a/xen/arch/x86/hvm/svm/svm.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/svm/svm.c	Wed May 21 20:49:53 2008 +0100
@@ -299,7 +299,7 @@ static void svm_save_cpu_state(struct vc
     data->msr_efer         = v->arch.hvm_vcpu.guest_efer;
     data->msr_flags        = -1ULL;
 
-    data->tsc = hvm_get_guest_time(v);
+    data->tsc = hvm_get_guest_tsc(v);
 }
 
 
@@ -315,7 +315,7 @@ static void svm_load_cpu_state(struct vc
     v->arch.hvm_vcpu.guest_efer = data->msr_efer;
     svm_update_guest_efer(v);
 
-    hvm_set_guest_time(v, data->tsc);
+    hvm_set_guest_tsc(v, data->tsc);
 }
 
 static void svm_save_vmcb_ctxt(struct vcpu *v, struct hvm_hw_cpu *ctxt)
diff -r e64c3a8c60e1 xen/arch/x86/hvm/vlapic.c
--- a/xen/arch/x86/hvm/vlapic.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/vlapic.c	Thu May 22 09:38:40 2008 +0100
@@ -474,7 +474,6 @@ static uint32_t vlapic_get_tmcct(struct 
     uint64_t counter_passed;
 
     counter_passed = ((hvm_get_guest_time(v) - vlapic->timer_last_update)
-                      * 1000000000ULL / ticks_per_sec(v)
                       / APIC_BUS_CYCLE_NS / vlapic->hw.timer_divisor);
     tmcct = tmict - counter_passed;
 
diff -r e64c3a8c60e1 xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Wed May 21 20:49:53 2008 +0100
@@ -607,7 +607,7 @@ static void vmx_save_cpu_state(struct vc
     data->msr_syscall_mask = guest_state->msrs[VMX_INDEX_MSR_SYSCALL_MASK];
 #endif
 
-    data->tsc = hvm_get_guest_time(v);
+    data->tsc = hvm_get_guest_tsc(v);
 }
 
 static void vmx_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
@@ -625,7 +625,7 @@ static void vmx_load_cpu_state(struct vc
     v->arch.hvm_vmx.shadow_gs = data->shadow_gs;
 #endif
 
-    hvm_set_guest_time(v, data->tsc);
+    hvm_set_guest_tsc(v, data->tsc);
 }
 
 
diff -r e64c3a8c60e1 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/hvm/vpt.c	Thu May 22 09:38:29 2008 +0100
@@ -348,7 +348,7 @@ void create_periodic_time(
     pt->vcpu = v;
     pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
     pt->irq = irq;
-    pt->period_cycles = (u64)period * cpu_khz / 1000000L;
+    pt->period_cycles = (u64)period;
     pt->one_shot = one_shot;
     pt->scheduled = NOW() + period;
     /*
diff -r e64c3a8c60e1 xen/arch/x86/time.c
--- a/xen/arch/x86/time.c	Wed May 21 16:55:11 2008 +0100
+++ b/xen/arch/x86/time.c	Wed May 21 20:49:53 2008 +0100
@@ -501,6 +501,8 @@ static s_time_t read_platform_stime(void
     return stime;
 }
 
+extern void get_s_time_mono_init(void);
+
 static void platform_time_calibration(void)
 {
     u64 count;
@@ -559,6 +561,7 @@ static void init_platform_timer(void)
     plt_overflow(NULL);
 
     platform_timer_stamp = plt_stamp64;
+    get_s_time_mono_init();
 
     printk("Platform timer is %s %s\n",
            freq_string(pts->frequency), pts->name);
@@ -694,6 +697,29 @@ s_time_t get_s_time(void)
     rdtscll(tsc);
     delta = tsc - t->local_tsc_stamp;
     now = t->stime_local_stamp + scale_delta(delta, &t->tsc_scale);
+
+    return now;
+}
+
+spinlock_t get_s_time_mono_lock;
+
+void get_s_time_mono_init(void)
+{
+    spin_lock_init(&get_s_time_mono_lock);
+}
+
+s_time_t get_s_time_mono(void)
+{
+    s_time_t now;
+    static s_time_t last = (s_time_t)0;
+
+    spin_lock(&get_s_time_mono_lock);
+    now = get_s_time();
+    if (now > last)
+        last = now;
+    else
+        now = last;
+    spin_unlock(&get_s_time_mono_lock);
 
     return now;
 }
diff -r e64c3a8c60e1 xen/include/asm-x86/hvm/hvm.h
--- a/xen/include/asm-x86/hvm/hvm.h	Wed May 21 16:55:11 2008 +0100
+++ b/xen/include/asm-x86/hvm/hvm.h	Thu May 22 09:37:49 2008 +0100
@@ -147,8 +147,8 @@ void hvm_send_assist_req(struct vcpu *v)
 
 void hvm_set_guest_tsc(struct vcpu *v, u64 guest_tsc);
 u64 hvm_get_guest_tsc(struct vcpu *v);
-#define hvm_set_guest_time(vcpu, gtime) hvm_set_guest_tsc(vcpu, gtime)
-#define hvm_get_guest_time(vcpu)        hvm_get_guest_tsc(vcpu)
+void hvm_set_guest_time(struct vcpu *v, u64 guest_time);
+u64 hvm_get_guest_time(struct vcpu *v);
 
 #define hvm_paging_enabled(v) \
     (!!((v)->arch.hvm_vcpu.guest_cr[0] & X86_CR0_PG))
diff -r e64c3a8c60e1 xen/include/asm-x86/hvm/vmx/vmx.h
--- a/xen/include/asm-x86/hvm/vmx/vmx.h	Wed May 21 16:55:11 2008 +0100
+++ b/xen/include/asm-x86/hvm/vmx/vmx.h	Wed May 21 20:49:53 2008 +0100
@@ -49,7 +49,6 @@ void vmx_asm_do_vmentry(void);
 void vmx_asm_do_vmentry(void);
 void vmx_intr_assist(void);
 void vmx_do_resume(struct vcpu *);
-void set_guest_time(struct vcpu *v, u64 gtime);
 void vmx_vlapic_msr_changed(struct vcpu *v);
 void vmx_realmode(struct cpu_user_regs *regs);
 
diff -r e64c3a8c60e1 xen/include/asm-x86/hvm/vpt.h
--- a/xen/include/asm-x86/hvm/vpt.h	Wed May 21 16:55:11 2008 +0100
+++ b/xen/include/asm-x86/hvm/vpt.h	Wed May 21 20:49:53 2008 +0100
@@ -57,7 +57,7 @@ typedef struct HPETState {
 typedef struct HPETState {
     struct hpet_registers hpet;
     struct vcpu *vcpu;
-    uint64_t tsc_freq;
+    uint64_t stime_freq;
     uint64_t hpet_to_ns_scale; /* hpet ticks to ns (multiplied by 2^10) */
     uint64_t hpet_to_ns_limit; /* max hpet ticks convertable to ns      */
     uint64_t mc_offset;
@@ -137,6 +137,9 @@ struct pl_time {    /* platform time */
     struct RTCState  vrtc;
     struct HPETState vhpet;
     struct PMTState  vpmt;
+    int64_t stime_offset;       /* guest_time = Xen sys time + stime_offset */
+    uint64_t last_guest_time;   /* ensure monotonicity */
+    spinlock_t pl_time_lock;
 };
 
 #define ticks_per_sec(v) (v->domain->arch.hvm_domain.tsc_frequency)
diff -r e64c3a8c60e1 xen/include/xen/time.h
--- a/xen/include/xen/time.h	Wed May 21 16:55:11 2008 +0100
+++ b/xen/include/xen/time.h	Thu May 22 09:36:35 2008 +0100
@@ -24,6 +24,8 @@ struct domain;
  * System Time
  * 64 bit value containing the nanoseconds elapsed since boot time.
  * This value is adjusted by frequency drift.
+ * mono version guarantees that consecutive reads are non-decreasing,
+ * (e.g. when first read is on a different processor than second read)
  * NOW() returns the current time.
  * The other macros are for convenience to approximate short intervals
  * of real time into system time 
@@ -32,6 +34,7 @@ typedef s64 s_time_t;
 typedef s64 s_time_t;
 
 s_time_t get_s_time(void);
+s_time_t get_s_time_mono(void);
 unsigned long get_localtime(struct domain *d);
 
 struct tm {
@@ -47,6 +50,7 @@ struct tm {
 };
 struct tm gmtime(unsigned long t);
 
+#define SYSTEM_TIME_HZ  1000000000ULL
 #define NOW()           ((s_time_t)get_s_time())
 #define SECONDS(_s)     ((s_time_t)((_s)  * 1000000000ULL))
 #define MILLISECS(_ms)  ((s_time_t)((_ms) * 1000000ULL))

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel