[PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

["Dan Magenheimer" <dan.magenheimer@oracle.com>]

[-- Attachment #1: Type: text/plain, Size: 8557 bytes --]

The attached patch adds a new Xen boot option "softtsc" that
causes all hvm domains to trap and emulate all rdtsc instructions
and fetch Xen system time instead of the hardware tsc.  As
a result, the processor will appear to be running at 1000.xyz MHz
(where xyz indicates some fraction that seems to vary slightly
across different boots).  I've also added a line of output to
"xm debug-key t" to report softsc and tsc-related CPU features.

(Keir, though 3.3 functionality freeze is past, this is more of a
diagnostic aid and defaults off, so I don't know if you would
consider it for 3.3.)

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>

Dan

P.S. Thanks to Li Zhang for coding the foundation for this patch.
Li, do you want to provide a signed-off-by line too?

> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: Tuesday, July 08, 2008 6:30 PM
> To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
> Cc: 'Tian, Kevin'
> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> 
> 
> Looks great!  I am continuing to code and test.  Some 
> preliminary results:
> 
> 1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() 
> seems to just work!  The hvm
>     guest believes it is running on a 1000.185 MHz clock.
> 
> 2) On my test box, a guest rdtsc without softtsc takes about 
> 30ns.  With softtsc,
>     rdtsc takes about 1us.
> 
> 3) On my user-land test program, I am seeing rare "Time goes 
> backwards", even with
>    hvm_get_guest_time().  I'm trying to track this down.
> 
> Dan
> 
> > -----Original Message-----
> > From: Zhang, Li [mailto:li.zhang@intel.com]
> > Sent: Tuesday, July 08, 2008 3:47 AM
> > To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > Cc: Tian, Kevin
> > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> > feedback please)
> > 
> > 
> > Hi, Dan
> > Please ignore previous comments. The above attachment is OK. 
> > Sorry for that. :(
> > 
> > >  -----Original Message-----
> > >  From: xen-devel-bounces@lists.xensource.com
> > >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf 
> > Of Zhang, Li
> > >  Sent: Tuesday, July 08, 2008 2:58 PM
> > >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > >  Cc: Tian, Kevin
> > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> > feedback please)
> > >
> > >  In fact, the issue is from guest. In the guest's terminal, 
> > it shows that "your
> > >  time source seems to be instable..."
> > >
> > >  >  -----Original Message-----
> > >  >  From: xen-devel-bounces@lists.xensource.com
> > >  >  [mailto:xen-devel-bounces@lists.xensource.com] On 
> > Behalf Of Zhang, Li
> > >  >  Sent: Tuesday, July 08, 2008 2:49 PM
> > >  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > >  >  Cc: Tian, Kevin
> > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and 
> > AMD feedback please)
> > >  >
> > >  >  Hi, Dan
> > >  >
> > >  >  This is the patch which I think it has been completed 
> > VT. But there is an
> > >  >  instable issue. The reason may be from that this 
> > interception is not bound
> > >  >  to timer mode yet.
> > >  >
> > >  >  Thanks
> > >  >  --Li
> > >  >
> > >  >  >  -----Original Message-----
> > >  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> > >  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
> > >  >  >  To: Zhang, Li; Xen-Devel (E-mail)
> > >  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
> > >  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel 
> > and AMD feedback
> > >  please)
> > >  >  >
> > >  >  >  Thanks Kevin and Li --
> > >  >  >
> > >  >  >  A couple of questions:
> > >  >  >
> > >  >  >  1) If the EXITING flag to be set in vmcs.c is to be 
> > controlled by a xen
> > >  >  boot
> > >  >  >  option, would this work:
> > >  >  >
> > >  >  >  static int opt_softtsc = 0;
> > >  >  >  boolean_param("softtsc", opt_softtsc);
> > >  >  >    :
> > >  >  >  min = /* original code */
> > >  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
> > >  >  >
> > >  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) 
> > if we change the
> > >  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and 
> > then set regs->edx
> > >  amd
> > >  >  >  regs->eax from tmptsc, are we almost done?
> > >  >  >
> > >  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if 
> > so, what code gets
> > >  >  called
> > >  >  >  to force the trap?
> > >  >  >
> > >  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is 
> > there already code
> > >  >  >  somewhere to emulate WRMSR of tsc?
> > >  >  >
> > >  >  >  Sorry I am not very familar with the details of the 
> > ia32 instruction
> > >  set.
> > >  >  >
> > >  >  >  Thanks,
> > >  >  >  Dan
> > >  >  >
> > >  >  >  > -----Original Message-----
> > >  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
> > >  >  >  > Sent: Monday, July 07, 2008 9:00 PM
> > >  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > >  >  >  > Cc: Tian, Kevin
> > >  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
> > >  >  >  > feedback please)
> > >  >  >  >
> > >  >  >  >
> > >  >  >  > Hi, Dan
> > >  >  >  > This seems not to be enough. I will cook another patch.
> > >  >  >  >
> > >  >  >  > >  -----Original Message-----
> > >  >  >  > >  From: Zhang, Li
> > >  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
> > >  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 
> > 'Xen-Devel
> > >  (E-mail)'
> > >  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen 
> > (Intel and AMD
> > >  >  >  > feedback please)
> > >  >  >  > >
> > >  >  >  > >  Hi, Dan
> > >  >  >  > >
> > >  >  >  > >  I just add some code to produce vmexit. The function
> > >  >  >  > vmx_rdtsc_intercept()
> > >  >  >  > >  is not completed. It is needed to add some code 
> > to get the
> > >  >  >  > guest tsc.
> > >  >  >  > >
> > >  >  >  > >  Thanks
> > >  >  >  > >  --Li
> > >  >  >  > >
> > >  >  >  > >  >  -----Original Message-----
> > >  >  >  > >  >  From: Tian, Kevin
> > >  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
> > >  >  >  > >  >  To: 'dan.magenheimer@oracle.com'; 
> Xen-Devel (E-mail)
> > >  >  >  > >  >  Cc: Zhang, Li
> > >  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen 
> (Intel and
> > >  >  >  > AMD feedback please)
> > >  >  >  > >  >
> > >  >  >  > >  >  >From: Dan Magenheimer 
> > [mailto:dan.magenheimer@oracle.com]
> > >  >  >  > >  >  >Sent: 2008年7月5日 1:32
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >> >Is this something that you (or Intel 
> in general)
> > >  >  >  > could look at?
> > >  >  >  > >  >  >> >I would be happy to participate but I 
> > don't think I
> > >  >  >  > understand
> > >  >  >  > >  >  >> >VT well enough.  Once the trap occurs, 
> I suppose
> > >  >  >  > Xen system time
> > >  >  >  > >  >  >> >could be used as the virtual TSC, 
> > possibly scaled up.
> > >  >  >  > >  >  >> >
> > >  >  >  > >  >  >>
> > >  >  >  > >  >  >> There should be tiny related to VT, as 
> > only turning
> > >  >  >  > on some bit to
> > >  >  >  > >  >  >> allow RDTSC trapping and then the rest 
> > stuff should be common
> > >  >  >  > >  >  >> how to handle it. We'll take a look, but can't
> > >  >  >  > commit the time due
> > >  >  >  > >  >  >> to other scheduled bandwidth. But if 
> you'd like to
> > >  >  >  > jump in early
> > >  >  >  > >  >  >> we definitely can help with VT side.
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >If you can post a patch with code that:
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
> > >  >  >  > >  >  >2) if the variable is set, a rdtsc 
> > instruction in any hvm
> > >  >  >  > >  >  >   domain causes a trap
> > >  >  >  > >  >  >3) the trap handler just does a physical 
> > rdtsc and returns
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >then I could probably take it from there.
> > >  >  >  > >  >
> > >  >  >  > >  >  OK, and Li in CC will take a look and bake a 
> > patch for you.
> > >  >  >  > >  >
> > >  >  >  > >  >  Thanks,
> > >  >  >  > >  >  Kevin
> > >  >  >  >
> > >
> > >  _______________________________________________
> > >  Xen-devel mailing list
> > >  Xen-devel@lists.xensource.com
> > >  http://lists.xensource.com/xen-devel
> > 
> > 

[-- Attachment #2: softtsc.patch --]
[-- Type: application/octet-stream, Size: 7521 bytes --]

diff -r bd97e45e073a xen/arch/x86/cpu/amd.c
--- a/xen/arch/x86/cpu/amd.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/cpu/amd.c	Wed Jul 09 15:13:55 2008 -0600
@@ -327,8 +327,10 @@ static void __init init_amd(struct cpuin
 
 	if (cpuid_eax(0x80000000) >= 0x80000007) {
 		c->x86_power = cpuid_edx(0x80000007);
-		if (c->x86_power & (1<<8))
+		if (c->x86_power & (1<<8)) {
 			set_bit(X86_FEATURE_CONSTANT_TSC, c->x86_capability);
+			constant_tsc = 1;
+		}
 	}
 
 #ifdef CONFIG_X86_HT
diff -r bd97e45e073a xen/arch/x86/cpu/common.c
--- a/xen/arch/x86/cpu/common.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/cpu/common.c	Wed Jul 09 15:13:55 2008 -0600
@@ -16,6 +16,8 @@
 
 #define tsc_disable 0
 #define disable_pse 0
+
+int constant_tsc = 0;
 
 static int cachesize_override __cpuinitdata = -1;
 static int disable_x86_fxsr __cpuinitdata;
diff -r bd97e45e073a xen/arch/x86/cpu/intel.c
--- a/xen/arch/x86/cpu/intel.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/cpu/intel.c	Wed Jul 09 15:13:55 2008 -0600
@@ -184,8 +184,10 @@ static void __devinit init_intel(struct 
 	if (c->x86 == 6) 
 		set_bit(X86_FEATURE_P3, c->x86_capability);
 	if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
-		(c->x86 == 0x6 && c->x86_model >= 0x0e))
+		(c->x86 == 0x6 && c->x86_model >= 0x0e)) {
 		set_bit(X86_FEATURE_CONSTANT_TSC, c->x86_capability);
+		constant_tsc = 1;
+	}
 
 	start_vmx();
 }
diff -r bd97e45e073a xen/arch/x86/hvm/hvm.c
--- a/xen/arch/x86/hvm/hvm.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/hvm.c	Wed Jul 09 15:13:55 2008 -0600
@@ -56,6 +56,8 @@ int hvm_enabled __read_mostly;
 
 unsigned int opt_hvm_debug_level __read_mostly;
 integer_param("hvm_debug", opt_hvm_debug_level);
+int opt_softtsc = 0;
+boolean_param("softtsc", opt_softtsc);
 
 struct hvm_function_table hvm_funcs __read_mostly;
 
@@ -148,7 +150,10 @@ u64 hvm_get_guest_tsc(struct vcpu *v)
 {
     u64 host_tsc;
 
-    rdtscll(host_tsc);
+    if ( opt_softtsc )
+        host_tsc = hvm_get_guest_time(v);
+    else
+        rdtscll(host_tsc);
     return host_tsc + v->arch.hvm_vcpu.cache_tsc_offset;
 }
 
@@ -1640,6 +1645,16 @@ void hvm_cpuid(unsigned int input, unsig
     }
 }
 
+void hvm_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = (uint32_t)tsc;
+    regs->edx = (uint32_t)(tsc >> 32);
+}
+
 int hvm_msr_read_intercept(struct cpu_user_regs *regs)
 {
     uint32_t ecx = regs->ecx;
diff -r bd97e45e073a xen/arch/x86/hvm/svm/svm.c
--- a/xen/arch/x86/hvm/svm/svm.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/svm/svm.c	Wed Jul 09 15:13:55 2008 -0600
@@ -52,6 +52,8 @@
 #include <asm/hap.h>
 
 u32 svm_feature_flags;
+
+extern int opt_softtsc;
 
 #define set_segment_register(name, value)  \
     asm volatile ( "movw %%ax ,%%" STR(name) "" : : "a" (value) )
@@ -1340,6 +1342,13 @@ asmlinkage void svm_vmexit_handler(struc
         hvm_triple_fault();
         break;
 
+    case VMEXIT_RDTSC:
+        if ( opt_softtsc )
+        {
+            hvm_rdtsc_intercept(regs);
+            break;
+        }
+        /* else fall through */
     case VMEXIT_RDTSCP:
     case VMEXIT_MONITOR:
     case VMEXIT_MWAIT:
diff -r bd97e45e073a xen/arch/x86/hvm/svm/vmcb.c
--- a/xen/arch/x86/hvm/svm/vmcb.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/svm/vmcb.c	Wed Jul 09 15:13:55 2008 -0600
@@ -39,6 +39,8 @@
 #include <xen/keyhandler.h>
 
 extern int svm_dbg_on;
+
+extern int opt_softtsc;
 
 #define IOPM_SIZE   (12 * 1024)
 #define MSRPM_SIZE  (8  * 1024)
@@ -165,6 +167,9 @@ static int construct_vmcb(struct vcpu *v
 
     /* TSC. */
     vmcb->tsc_offset = 0;
+    if ( opt_softtsc )
+        vmcb->general1_intercepts |= GENERAL1_INTERCEPT_RDTSC;
+
     
     /* Guest EFER: *must* contain SVME or VMRUN will fail. */
     vmcb->efer = EFER_SVME;
diff -r bd97e45e073a xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Wed Jul 09 15:13:55 2008 -0600
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+extern int opt_softtsc;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@ static void vmx_init_vmcs_config(void)
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (opt_softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
diff -r bd97e45e073a xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Wed Jul 09 15:13:55 2008 -0600
@@ -2202,6 +2202,11 @@ asmlinkage void vmx_vmexit_handler(struc
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        hvm_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;
diff -r bd97e45e073a xen/arch/x86/time.c
--- a/xen/arch/x86/time.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/time.c	Wed Jul 09 15:13:55 2008 -0600
@@ -68,7 +68,7 @@ static DEFINE_PER_CPU(struct cpu_time, c
 static DEFINE_PER_CPU(struct cpu_time, cpu_time);
 
 /* TSC is invariant on C state entry? */
-static bool_t tsc_invariant;
+bool_t tsc_invariant;
 
 /*
  * We simulate a 32-bit platform timer from the 16-bit PIT ch2 counter.
diff -r bd97e45e073a xen/common/keyhandler.c
--- a/xen/common/keyhandler.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/common/keyhandler.c	Wed Jul 09 15:13:55 2008 -0600
@@ -251,6 +251,8 @@ static void read_clocks(unsigned char ke
     unsigned int cpu = smp_processor_id(), min_cpu, max_cpu;
     u64 min, max, dif, difus;
     static DEFINE_SPINLOCK(lock);
+    extern int opt_softtsc, constant_tsc;
+    extern bool_t tsc_invariant;
 
     spin_lock(&lock);
 
@@ -284,6 +286,8 @@ static void read_clocks(unsigned char ke
     printk("Min = %"PRIu64" ; Max = %"PRIu64" ; Diff = %"PRIu64
            " (%"PRIu64" microseconds)\n",
            min, max, dif, difus);
+    printk("softtsc = %d, constant_tsc = %d, tsc_invariant = %d\n",
+         opt_softtsc, constant_tsc, (int)tsc_invariant);
 }
 
 extern void dump_runq(unsigned char key);
diff -r bd97e45e073a xen/include/asm-x86/cpufeature.h
--- a/xen/include/asm-x86/cpufeature.h	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/include/asm-x86/cpufeature.h	Wed Jul 09 15:13:55 2008 -0600
@@ -74,6 +74,8 @@
 #define X86_FEATURE_P3		(3*32+ 6) /* P3 */
 #define X86_FEATURE_P4		(3*32+ 7) /* P4 */
 #define X86_FEATURE_CONSTANT_TSC (3*32+ 8) /* TSC ticks at a constant rate */
+
+extern int constant_tsc;
 
 /* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */
 #define X86_FEATURE_XMM3	(4*32+ 0) /* Streaming SIMD Extensions-3 */
diff -r bd97e45e073a xen/include/asm-x86/hvm/support.h
--- a/xen/include/asm-x86/hvm/support.h	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/include/asm-x86/hvm/support.h	Wed Jul 09 15:13:55 2008 -0600
@@ -125,6 +125,8 @@ void hvm_hlt(unsigned long rflags);
 void hvm_hlt(unsigned long rflags);
 void hvm_triple_fault(void);
 
+void hvm_rdtsc_intercept(struct cpu_user_regs *regs);
+
 /* These functions all return X86EMUL return codes. */
 int hvm_set_efer(uint64_t value);
 int hvm_set_cr0(unsigned long value);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel