Guest TSC and Xen (Intel and AMD feedback please)

Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

Various versions of Linux under various circumstances select
TSC as the primary clocksource for the kernel.  This is
especially true for uniprocessor kernels, but also in some
cases for multiprocessor kernels.  In most cases, this
is because a processor bit (tsc_invariant? constant_tsc?)
is passed through directly from the hardware via Xen and
tested by the hvm guest and the result implies that the
TSC is "stable".

I'd like to propose that, for a Xen hvm guest, TSC should
NEVER be considered stable.  For at least these reasons:

1) Often this test is done once at guest boot; if the guest
   migrates to another machine without the bit set, time
   will be erratic.
2) I *think* that in some cases even within the same system,
   TSC values will skew somewhat.  Since a uniprocessor guest
   will often be rescheduled to a different pcpu by Xen,
   the underlying tsc may appear erratic.

Comments (especially from Intel and AMD)?

If agreed, could Intel and AMD provide patches so that hvm
reads of the bits return "false"?  Or will this cause other
problems?

Another alternative would be to trap all rdtsc's and emulate
them but this probably will not be easy and may have
significant performance implications.  But perhaps it should
be an option?

Dan

===================================
Thanks... for the memory
I really could use more / My throughput's on the floor
The balloon is flat / My swap disk's fat / I've OOM's in store
Overcommitted so much
(with apologies to the late great Bob Hope)

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Guest TSC and Xen (Intel and AMD feedback please)

[Keir Fraser]

On 1/7/08 18:33, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> Another alternative would be to trap all rdtsc's and emulate
> them but this probably will not be easy and may have
> significant performance implications.  But perhaps it should
> be an option?

I think this would be useful, even if just for diagnosing guest time issues.

 -- Keir

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer
>Sent: 2008年7月2日 1:33
>
>Various versions of Linux under various circumstances select
>TSC as the primary clocksource for the kernel.  This is
>especially true for uniprocessor kernels, but also in some
>cases for multiprocessor kernels.  In most cases, this
>is because a processor bit (tsc_invariant? constant_tsc?)
>is passed through directly from the hardware via Xen and
>tested by the hvm guest and the result implies that the
>TSC is "stable".

constant_tsc is one of the factors affecting tsc stability, and
the keypoint is that one internal variable 'tsc_unstable' is 
initialized to zero. That means, kernel considers TSC as
stable by default, and you need trigger some factors upon 
which guest kernel'd like to mark tsc as unstable when 
checking them. However by far none of those factors seem 
appliable to all circumstances.

For example, even by hidding constant_tsc bit from cpuid,
UP guest will still consider tsc as stable, and so does 32bit
SMP guest running on Intel processors.

There's one proposal in KVM side, by exposing a faked C2
states into guest. ACPI processor idle path will mark TSC
as unstable when trying to enter C2. However that requires
guest to enable ACPI processor support, and need some
tricky logic to further mark C2 invalid to avoid performance
influence after purpose is achieved...

When saying this, I'm not opposing your proposal since it
does be a light option for some type of guests. My point is
just that no reliabe option to mark tsc as unstable, and
whether we have done all what can be done in Xen side 
to tackle this issue... Sorry that I didn't follow up your 
long discussion in this area before and thus may lose some 
context.

>
>I'd like to propose that, for a Xen hvm guest, TSC should
>NEVER be considered stable.  For at least these reasons:
>
>1) Often this test is done once at guest boot; if the guest
>   migrates to another machine without the bit set, time
>   will be erratic.

Since it's 'if', we should treat it as an option, instead of 'NEVER',
right? User may not have migration requirement in some cases,
and or the migratable backups are with same bits exactly... :-)

>2) I *think* that in some cases even within the same system,
>   TSC values will skew somewhat.  Since a uniprocessor guest
>   will often be rescheduled to a different pcpu by Xen,
>   the underlying tsc may appear erratic.

Can't be the tsc kept monotonic to guest, with some tweak on
tsc offset? Also Xen is always trying to sync tsc among cores.
As long as you don't run on a box with multiple bus crystals, 
or a box without constant tsc upon freq change, the tsc drift
among cores should be negligible considering the overhead of
migration. Then for cases out of 'as long as', we should mark
TSC as unstable, still as an option.

>
>Comments (especially from Intel and AMD)?
>
>If agreed, could Intel and AMD provide patches so that hvm
>reads of the bits return "false"?  Or will this cause other
>problems?

I vaguely recalled some posts about CPUID virtualization
framework. There's no need to a seperate category, and it
looks like this option can be aligned with that part?

Thanks,
Kevin

>
>Another alternative would be to trap all rdtsc's and emulate
>them but this probably will not be easy and may have
>significant performance implications.  But perhaps it should
>be an option?
>

Agree.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

Hi Kevin --

> constant_tsc is one of the factors affecting tsc stability, and
> the keypoint is that one internal variable 'tsc_unstable' is
> initialized to zero. That means, kernel considers TSC as
> stable by default, and you need trigger some factors upon
> which guest kernel'd like to mark tsc as unstable when
> checking them. However by far none of those factors seem
> appliable to all circumstances.
>
> For example, even by hidding constant_tsc bit from cpuid,
> UP guest will still consider tsc as stable, and so does 32bit
> SMP guest running on Intel processors.

Yes, looking over some Linux code, I see you are right.  In
one version of RH, if the processor vendor is Intel, it
always assumes tsc is stable!

> Since it's 'if', we should treat it as an option, instead of 'NEVER',
> right? User may not have migration requirement in some cases,
> and or the migratable backups are with same bits exactly... :-)

Yes, if it could be done, I agree an option would be better
than 'NEVER'.  But Keir is not too keen on more time-related
options.

> >2) I *think* that in some cases even within the same system,
> >   TSC values will skew somewhat.  Since a uniprocessor guest
> >   will often be rescheduled to a different pcpu by Xen,
> >   the underlying tsc may appear erratic.
>
> Can't be the tsc kept monotonic to guest, with some tweak on
> tsc offset? Also Xen is always trying to sync tsc among cores.
> As long as you don't run on a box with multiple bus crystals,
> or a box without constant tsc upon freq change, the tsc drift
> among cores should be negligible considering the overhead of
> migration. Then for cases out of 'as long as', we should mark
> TSC as unstable, still as an option.

No Xen doesn't actally sync the tsc's, just maintains its own
software offset variables.  I suppose it could periodically check
to make sure the tsc skew is within some reasonable value,
but after the guest boots, it is probably too late.

> I vaguely recalled some posts about CPUID virtualization
> framework. There's no need to a seperate category, and it
> looks like this option can be aligned with that part?

Yes, good point.  I wasn't following that closely either but
this could be covered by it.

> >Another alternative would be to trap all rdtsc's and emulate
> >them but this probably will not be easy and may have
> >significant performance implications.  But perhaps it should
> >be an option?
>
> Agree.

Is this something that you (or Intel in general) could look at?
I would be happy to participate but I don't think I understand
VT well enough.  Once the trap occurs, I suppose Xen system time
could be used as the virtual TSC, possibly scaled up.

Thanks,
Dan

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
>Sent: 2008年7月3日 9:21
>>
>> Can't be the tsc kept monotonic to guest, with some tweak on
>> tsc offset? Also Xen is always trying to sync tsc among cores.
>> As long as you don't run on a box with multiple bus crystals,
>> or a box without constant tsc upon freq change, the tsc drift
>> among cores should be negligible considering the overhead of
>> migration. Then for cases out of 'as long as', we should mark
>> TSC as unstable, still as an option.
>
>No Xen doesn't actally sync the tsc's, just maintains its own
>software offset variables.  I suppose it could periodically check
>to make sure the tsc skew is within some reasonable value,
>but after the guest boots, it is probably too late.

You are right. Xen only sync TSCs at boot time and then calibrate
TSC pace independently on each cpu. If some factors drag TSC
in the middle, then we may observe larger drift. But now I'm not
sure whether periodically sync among cpus is light enough, if taking
a similar convergence algothrim as boot time. But still, if only TSC
drift exists (caused by occasional events instead of a constantly
increasing drift from freq difference), this monotonic can be easily
exposed to the guest, like derived from xen system time to adjust
TSC offset at migration...

>
>> >Another alternative would be to trap all rdtsc's and emulate
>> >them but this probably will not be easy and may have
>> >significant performance implications.  But perhaps it should
>> >be an option?
>>
>> Agree.
>
>Is this something that you (or Intel in general) could look at?
>I would be happy to participate but I don't think I understand
>VT well enough.  Once the trap occurs, I suppose Xen system time
>could be used as the virtual TSC, possibly scaled up.
>

There should be tiny related to VT, as only turning on some bit to
allow RDTSC trapping and then the rest stuff should be common
how to handle it. We'll take a look, but can't commit the time due
to other scheduled bandwidth. But if you'd like to jump in early
we definitely can help with VT side.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

> >Is this something that you (or Intel in general) could look at?
> >I would be happy to participate but I don't think I understand
> >VT well enough.  Once the trap occurs, I suppose Xen system time
> >could be used as the virtual TSC, possibly scaled up.
> >
>
> There should be tiny related to VT, as only turning on some bit to
> allow RDTSC trapping and then the rest stuff should be common
> how to handle it. We'll take a look, but can't commit the time due
> to other scheduled bandwidth. But if you'd like to jump in early
> we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm
   domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

Dan

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
>Sent: 2008年7月5日 1:32
>
>> >Is this something that you (or Intel in general) could look at?
>> >I would be happy to participate but I don't think I understand
>> >VT well enough.  Once the trap occurs, I suppose Xen system time
>> >could be used as the virtual TSC, possibly scaled up.
>> >
>>
>> There should be tiny related to VT, as only turning on some bit to
>> allow RDTSC trapping and then the rest stuff should be common
>> how to handle it. We'll take a look, but can't commit the time due
>> to other scheduled bandwidth. But if you'd like to jump in early
>> we definitely can help with VT side.
>
>If you can post a patch with code that:
>
>1) declares a boolean global variable: softtsc = 0
>2) if the variable is set, a rdtsc instruction in any hvm
>   domain causes a trap
>3) the trap handler just does a physical rdtsc and returns
>
>then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

Hi, Dan

I just add some code to produce vmexit. The function vmx_rdtsc_intercept() is not completed. It is needed to add some code to get the guest tsc.

Thanks 
--Li

>  -----Original Message-----
>  From: Tian, Kevin
>  Sent: Tuesday, July 08, 2008 9:39 AM
>  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  Cc: Zhang, Li
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >Sent: 2008年7月5日 1:32
>  >
>  >> >Is this something that you (or Intel in general) could look at?
>  >> >I would be happy to participate but I don't think I understand
>  >> >VT well enough.  Once the trap occurs, I suppose Xen system time
>  >> >could be used as the virtual TSC, possibly scaled up.
>  >> >
>  >>
>  >> There should be tiny related to VT, as only turning on some bit to
>  >> allow RDTSC trapping and then the rest stuff should be common
>  >> how to handle it. We'll take a look, but can't commit the time due
>  >> to other scheduled bandwidth. But if you'd like to jump in early
>  >> we definitely can help with VT side.
>  >
>  >If you can post a patch with code that:
>  >
>  >1) declares a boolean global variable: softtsc = 0
>  >2) if the variable is set, a rdtsc instruction in any hvm
>  >   domain causes a trap
>  >3) the trap handler just does a physical rdtsc and returns
>  >
>  >then I could probably take it from there.
>  
>  OK, and Li in CC will take a look and bake a patch for you.
>  
>  Thanks,
>  Kevin

[-- Attachment #2: tsc.patch --]
[-- Type: application/octet-stream, Size: 1265 bytes --]

diff -r 5eaec8296a22 xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Sun Jul 06 20:16:24 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 18:08:39 2008 +0800
@@ -89,6 +89,7 @@
 
     min = (CPU_BASED_HLT_EXITING |
            CPU_BASED_INVLPG_EXITING |
+           CPU_BASED_RDTSC_EXITING |
            CPU_BASED_CR3_LOAD_EXITING |
            CPU_BASED_CR3_STORE_EXITING |
            CPU_BASED_MONITOR_EXITING |
diff -r 5eaec8296a22 xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Sun Jul 06 20:16:24 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 18:08:39 2008 +0800
@@ -1374,6 +1374,13 @@
     HVMTRACE_LONG_2D(INVLPG, curr, /*invlpga=*/ 0, TRC_PAR_LONG(vaddr));
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
+}
+
+static void vmx_rdtsc_intercept(struct vcpu *vcpu)
+{
+    uint64_t tsc; 
+    rdtscll(tsc);
+    return;    
 }
 
 #define CASE_SET_REG(REG, reg)      \
@@ -2193,6 +2200,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(v);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

Hi, Dan
This seems not to be enough. I will cook another patch. 

>  -----Original Message-----
>  From: Zhang, Li
>  Sent: Tuesday, July 08, 2008 10:28 AM
>  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  Hi, Dan
>  
>  I just add some code to produce vmexit. The function vmx_rdtsc_intercept()
>  is not completed. It is needed to add some code to get the guest tsc.
>  
>  Thanks
>  --Li
>  
>  >  -----Original Message-----
>  >  From: Tian, Kevin
>  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  >  Cc: Zhang, Li
>  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  >
>  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  >Sent: 2008年7月5日 1:32
>  >  >
>  >  >> >Is this something that you (or Intel in general) could look at?
>  >  >> >I would be happy to participate but I don't think I understand
>  >  >> >VT well enough.  Once the trap occurs, I suppose Xen system time
>  >  >> >could be used as the virtual TSC, possibly scaled up.
>  >  >> >
>  >  >>
>  >  >> There should be tiny related to VT, as only turning on some bit to
>  >  >> allow RDTSC trapping and then the rest stuff should be common
>  >  >> how to handle it. We'll take a look, but can't commit the time due
>  >  >> to other scheduled bandwidth. But if you'd like to jump in early
>  >  >> we definitely can help with VT side.
>  >  >
>  >  >If you can post a patch with code that:
>  >  >
>  >  >1) declares a boolean global variable: softtsc = 0
>  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  >  >   domain causes a trap
>  >  >3) the trap handler just does a physical rdtsc and returns
>  >  >
>  >  >then I could probably take it from there.
>  >
>  >  OK, and Li in CC will take a look and bake a patch for you.
>  >
>  >  Thanks,
>  >  Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 3250 bytes --]

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work:

static int opt_softtsc = 0;
boolean_param("softtsc", opt_softtsc);
  :
min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction set.

Thanks,
Dan

> -----Original Message-----
> From: Zhang, Li [mailto:li.zhang@intel.com]
> Sent: Monday, July 07, 2008 9:00 PM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Cc: Tian, Kevin
> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> 
> 
> Hi, Dan
> This seems not to be enough. I will cook another patch.
> 
> >  -----Original Message-----
> >  From: Zhang, Li
> >  Sent: Tuesday, July 08, 2008 10:28 AM
> >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
> >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> >
> >  Hi, Dan
> >
> >  I just add some code to produce vmexit. The function 
> vmx_rdtsc_intercept()
> >  is not completed. It is needed to add some code to get the 
> guest tsc.
> >
> >  Thanks
> >  --Li
> >
> >  >  -----Original Message-----
> >  >  From: Tian, Kevin
> >  >  Sent: Tuesday, July 08, 2008 9:39 AM
> >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
> >  >  Cc: Zhang, Li
> >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and 
> AMD feedback please)
> >  >
> >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >  >  >Sent: 2008年7月5日 1:32
> >  >  >
> >  >  >> >Is this something that you (or Intel in general) 
> could look at?
> >  >  >> >I would be happy to participate but I don't think I 
> understand
> >  >  >> >VT well enough.  Once the trap occurs, I suppose 
> Xen system time
> >  >  >> >could be used as the virtual TSC, possibly scaled up.
> >  >  >> >
> >  >  >>
> >  >  >> There should be tiny related to VT, as only turning 
> on some bit to
> >  >  >> allow RDTSC trapping and then the rest stuff should be common
> >  >  >> how to handle it. We'll take a look, but can't 
> commit the time due
> >  >  >> to other scheduled bandwidth. But if you'd like to 
> jump in early
> >  >  >> we definitely can help with VT side.
> >  >  >
> >  >  >If you can post a patch with code that:
> >  >  >
> >  >  >1) declares a boolean global variable: softtsc = 0
> >  >  >2) if the variable is set, a rdtsc instruction in any hvm
> >  >  >   domain causes a trap
> >  >  >3) the trap handler just does a physical rdtsc and returns
> >  >  >
> >  >  >then I could probably take it from there.
> >  >
> >  >  OK, and Li in CC will take a look and bake a patch for you.
> >  >
> >  >  Thanks,
> >  >  Kevin
> 

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
>Sent: 2008年7月8日 11:34
>
>1) If the EXITING flag to be set in vmcs.c is to be controlled 
>by a xen boot option, would this work:
>
>static int opt_softtsc = 0;
>boolean_param("softtsc", opt_softtsc);
>  :
>min = /* original code */
>if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

yes.

>
>2) In vmx_rdtsc_intercept(/* need regs as param*/) if we 
>change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and 
>then set regs->edx amd regs->eax from tmptsc, are we almost done?

Yes, and similar example is under case EXIT_REASON_CPUID.

>
>3) Des Linux or Windows use the CR4.TSD flag and, if so, what 
>code gets called to force the trap?

My feeling is that both Linux and Windows allows user directly read to
TSC, and thus CR4.TSD is unlikely to be set. But it needs confirmation.
If you really want to trap such setting, hvm_set_cr4 is the point and
you may further reference vmx_update_guest_cr if it's vmx specific (but
I guess it should be common).

But I doubt the point to support such model, since two levels trapping
(Xen trap, inject fault, guest kernel read, xen trap, ...) which adds much
delay on Xen maintained guest tsc.

>
>4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there 
>already code somewhere to emulate WRMSR of tsc?

I'm not sure about this one, and others may answer.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

[-- Attachment #1: Type: text/plain, Size: 3932 bytes --]

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet. 

Thanks
--Li

>  -----Original Message-----
>  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  Sent: Tuesday, July 08, 2008 11:34 AM
>  To: Zhang, Li; Xen-Devel (E-mail)
>  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  Thanks Kevin and Li --
>  
>  A couple of questions:
>  
>  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot
>  option, would this work:
>  
>  static int opt_softtsc = 0;
>  boolean_param("softtsc", opt_softtsc);
>    :
>  min = /* original code */
>  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  
>  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd
>  regs->eax from tmptsc, are we almost done?
>  
>  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called
>  to force the trap?
>  
>  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>  somewhere to emulate WRMSR of tsc?
>  
>  Sorry I am not very familar with the details of the ia32 instruction set.
>  
>  Thanks,
>  Dan
>  
>  > -----Original Message-----
>  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  > Sent: Monday, July 07, 2008 9:00 PM
>  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  > Cc: Tian, Kevin
>  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > feedback please)
>  >
>  >
>  > Hi, Dan
>  > This seems not to be enough. I will cook another patch.
>  >
>  > >  -----Original Message-----
>  > >  From: Zhang, Li
>  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
>  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > feedback please)
>  > >
>  > >  Hi, Dan
>  > >
>  > >  I just add some code to produce vmexit. The function
>  > vmx_rdtsc_intercept()
>  > >  is not completed. It is needed to add some code to get the
>  > guest tsc.
>  > >
>  > >  Thanks
>  > >  --Li
>  > >
>  > >  >  -----Original Message-----
>  > >  >  From: Tian, Kevin
>  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  > >  >  Cc: Zhang, Li
>  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  > AMD feedback please)
>  > >  >
>  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  > >  >  >Sent: 2008年7月5日 1:32
>  > >  >  >
>  > >  >  >> >Is this something that you (or Intel in general)
>  > could look at?
>  > >  >  >> >I would be happy to participate but I don't think I
>  > understand
>  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>  > Xen system time
>  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>  > >  >  >> >
>  > >  >  >>
>  > >  >  >> There should be tiny related to VT, as only turning
>  > on some bit to
>  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>  > >  >  >> how to handle it. We'll take a look, but can't
>  > commit the time due
>  > >  >  >> to other scheduled bandwidth. But if you'd like to
>  > jump in early
>  > >  >  >> we definitely can help with VT side.
>  > >  >  >
>  > >  >  >If you can post a patch with code that:
>  > >  >  >
>  > >  >  >1) declares a boolean global variable: softtsc = 0
>  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  > >  >  >   domain causes a trap
>  > >  >  >3) the trap handler just does a physical rdtsc and returns
>  > >  >  >
>  > >  >  >then I could probably take it from there.
>  > >  >
>  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>  > >  >
>  > >  >  Thanks,
>  > >  >  Kevin
>  >

[-- Attachment #2: tsc.patch --]
[-- Type: application/octet-stream, Size: 1673 bytes --]

diff -r be52424a543e xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 22:26:57 2008 +0800
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+int softtsc = 1;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 22:26:57 2008 +0800
@@ -1375,6 +1375,17 @@
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
 }
+
+static void vmx_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = tsc & 0xffff;
+    regs->edx = tsc >> 32 & 0xffff;
+}
+
 
 #define CASE_SET_REG(REG, reg)      \
     case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."

>  -----Original Message-----
>  From: xen-devel-bounces@lists.xensource.com
>  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>  Sent: Tuesday, July 08, 2008 2:49 PM
>  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  Cc: Tian, Kevin
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  Hi, Dan
>  
>  This is the patch which I think it has been completed VT. But there is an
>  instable issue. The reason may be from that this interception is not bound
>  to timer mode yet.
>  
>  Thanks
>  --Li
>  
>  >  -----Original Message-----
>  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  Sent: Tuesday, July 08, 2008 11:34 AM
>  >  To: Zhang, Li; Xen-Devel (E-mail)
>  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  >
>  >  Thanks Kevin and Li --
>  >
>  >  A couple of questions:
>  >
>  >  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>  boot
>  >  option, would this work:
>  >
>  >  static int opt_softtsc = 0;
>  >  boolean_param("softtsc", opt_softtsc);
>  >    :
>  >  min = /* original code */
>  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  >
>  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd
>  >  regs->eax from tmptsc, are we almost done?
>  >
>  >  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>  called
>  >  to force the trap?
>  >
>  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>  >  somewhere to emulate WRMSR of tsc?
>  >
>  >  Sorry I am not very familar with the details of the ia32 instruction set.
>  >
>  >  Thanks,
>  >  Dan
>  >
>  >  > -----Original Message-----
>  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  >  > Sent: Monday, July 07, 2008 9:00 PM
>  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  >  > Cc: Tian, Kevin
>  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  > feedback please)
>  >  >
>  >  >
>  >  > Hi, Dan
>  >  > This seems not to be enough. I will cook another patch.
>  >  >
>  >  > >  -----Original Message-----
>  >  > >  From: Zhang, Li
>  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
>  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  > feedback please)
>  >  > >
>  >  > >  Hi, Dan
>  >  > >
>  >  > >  I just add some code to produce vmexit. The function
>  >  > vmx_rdtsc_intercept()
>  >  > >  is not completed. It is needed to add some code to get the
>  >  > guest tsc.
>  >  > >
>  >  > >  Thanks
>  >  > >  --Li
>  >  > >
>  >  > >  >  -----Original Message-----
>  >  > >  >  From: Tian, Kevin
>  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  >  > >  >  Cc: Zhang, Li
>  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  >  > AMD feedback please)
>  >  > >  >
>  >  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  > >  >  >Sent: 2008年7月5日 1:32
>  >  > >  >  >
>  >  > >  >  >> >Is this something that you (or Intel in general)
>  >  > could look at?
>  >  > >  >  >> >I would be happy to participate but I don't think I
>  >  > understand
>  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>  >  > Xen system time
>  >  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>  >  > >  >  >> >
>  >  > >  >  >>
>  >  > >  >  >> There should be tiny related to VT, as only turning
>  >  > on some bit to
>  >  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>  >  > >  >  >> how to handle it. We'll take a look, but can't
>  >  > commit the time due
>  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
>  >  > jump in early
>  >  > >  >  >> we definitely can help with VT side.
>  >  > >  >  >
>  >  > >  >  >If you can post a patch with code that:
>  >  > >  >  >
>  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>  >  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  >  > >  >  >   domain causes a trap
>  >  > >  >  >3) the trap handler just does a physical rdtsc and returns
>  >  > >  >  >
>  >  > >  >  >then I could probably take it from there.
>  >  > >  >
>  >  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>  >  > >  >
>  >  > >  >  Thanks,
>  >  > >  >  Kevin
>  >  >

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

[-- Attachment #1: Type: text/plain, Size: 5706 bytes --]

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

>  -----Original Message-----
>  From: xen-devel-bounces@lists.xensource.com
>  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>  Sent: Tuesday, July 08, 2008 2:58 PM
>  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  Cc: Tian, Kevin
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  In fact, the issue is from guest. In the guest's terminal, it shows that "your
>  time source seems to be instable..."
>  
>  >  -----Original Message-----
>  >  From: xen-devel-bounces@lists.xensource.com
>  >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>  >  Sent: Tuesday, July 08, 2008 2:49 PM
>  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  >  Cc: Tian, Kevin
>  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  >
>  >  Hi, Dan
>  >
>  >  This is the patch which I think it has been completed VT. But there is an
>  >  instable issue. The reason may be from that this interception is not bound
>  >  to timer mode yet.
>  >
>  >  Thanks
>  >  --Li
>  >
>  >  >  -----Original Message-----
>  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
>  >  >  To: Zhang, Li; Xen-Devel (E-mail)
>  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback
>  please)
>  >  >
>  >  >  Thanks Kevin and Li --
>  >  >
>  >  >  A couple of questions:
>  >  >
>  >  >  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>  >  boot
>  >  >  option, would this work:
>  >  >
>  >  >  static int opt_softtsc = 0;
>  >  >  boolean_param("softtsc", opt_softtsc);
>  >  >    :
>  >  >  min = /* original code */
>  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  >  >
>  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx
>  amd
>  >  >  regs->eax from tmptsc, are we almost done?
>  >  >
>  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>  >  called
>  >  >  to force the trap?
>  >  >
>  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>  >  >  somewhere to emulate WRMSR of tsc?
>  >  >
>  >  >  Sorry I am not very familar with the details of the ia32 instruction
>  set.
>  >  >
>  >  >  Thanks,
>  >  >  Dan
>  >  >
>  >  >  > -----Original Message-----
>  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  >  >  > Sent: Monday, July 07, 2008 9:00 PM
>  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  >  >  > Cc: Tian, Kevin
>  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  >  > feedback please)
>  >  >  >
>  >  >  >
>  >  >  > Hi, Dan
>  >  >  > This seems not to be enough. I will cook another patch.
>  >  >  >
>  >  >  > >  -----Original Message-----
>  >  >  > >  From: Zhang, Li
>  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel
>  (E-mail)'
>  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  >  > feedback please)
>  >  >  > >
>  >  >  > >  Hi, Dan
>  >  >  > >
>  >  >  > >  I just add some code to produce vmexit. The function
>  >  >  > vmx_rdtsc_intercept()
>  >  >  > >  is not completed. It is needed to add some code to get the
>  >  >  > guest tsc.
>  >  >  > >
>  >  >  > >  Thanks
>  >  >  > >  --Li
>  >  >  > >
>  >  >  > >  >  -----Original Message-----
>  >  >  > >  >  From: Tian, Kevin
>  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  >  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  >  >  > >  >  Cc: Zhang, Li
>  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  >  >  > AMD feedback please)
>  >  >  > >  >
>  >  >  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  >  > >  >  >Sent: 2008年7月5日 1:32
>  >  >  > >  >  >
>  >  >  > >  >  >> >Is this something that you (or Intel in general)
>  >  >  > could look at?
>  >  >  > >  >  >> >I would be happy to participate but I don't think I
>  >  >  > understand
>  >  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>  >  >  > Xen system time
>  >  >  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>  >  >  > >  >  >> >
>  >  >  > >  >  >>
>  >  >  > >  >  >> There should be tiny related to VT, as only turning
>  >  >  > on some bit to
>  >  >  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>  >  >  > >  >  >> how to handle it. We'll take a look, but can't
>  >  >  > commit the time due
>  >  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
>  >  >  > jump in early
>  >  >  > >  >  >> we definitely can help with VT side.
>  >  >  > >  >  >
>  >  >  > >  >  >If you can post a patch with code that:
>  >  >  > >  >  >
>  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>  >  >  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  >  >  > >  >  >   domain causes a trap
>  >  >  > >  >  >3) the trap handler just does a physical rdtsc and returns
>  >  >  > >  >  >
>  >  >  > >  >  >then I could probably take it from there.
>  >  >  > >  >
>  >  >  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>  >  >  > >  >
>  >  >  > >  >  Thanks,
>  >  >  > >  >  Kevin
>  >  >  >
>  
>  _______________________________________________
>  Xen-devel mailing list
>  Xen-devel@lists.xensource.com
>  http://lists.xensource.com/xen-devel

[-- Attachment #2: tsc.patch --]
[-- Type: application/octet-stream, Size: 1681 bytes --]

diff -r be52424a543e xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 22:26:57 2008 +0800
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+int softtsc = 1;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 22:26:57 2008 +0800
@@ -1375,6 +1375,17 @@
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
 }
+
+static void vmx_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = tsc & 0xffffffff;
+    regs->edx = tsc >> 32 & 0xffffffff;
+}
+
 
 #define CASE_SET_REG(REG, reg)      \
     case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Guest TSC and Xen (Intel and AMD feedback please)

[Dave Winchell]

Hi Li,

Shouldn't this

+ regs->eax = tsc & 0xffff;
+ regs->edx = tsc >> 32 & 0xffff;

be

+ regs->eax = tsc & 0xffffffff;
+ regs->edx = tsc >> 32 & 0xffffffff;

Thanks,
Dave


Zhang, Li wrote:

>Hi, Dan
>Please ignore previous comments. The above attachment is OK. Sorry for that. :(
>
>  
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>> Sent: Tuesday, July 08, 2008 2:58 PM
>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> Cc: Tian, Kevin
>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>> 
>> In fact, the issue is from guest. In the guest's terminal, it shows that "your
>> time source seems to be instable..."
>> 
>> >  -----Original Message-----
>> >  From: xen-devel-bounces@lists.xensource.com
>> >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>> >  Sent: Tuesday, July 08, 2008 2:49 PM
>> >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> >  Cc: Tian, Kevin
>> >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>> >
>> >  Hi, Dan
>> >
>> >  This is the patch which I think it has been completed VT. But there is an
>> >  instable issue. The reason may be from that this interception is not bound
>> >  to timer mode yet.
>> >
>> >  Thanks
>> >  --Li
>> >
>> >  >  -----Original Message-----
>> >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>> >  >  Sent: Tuesday, July 08, 2008 11:34 AM
>> >  >  To: Zhang, Li; Xen-Devel (E-mail)
>> >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>> >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback
>> please)
>> >  >
>> >  >  Thanks Kevin and Li --
>> >  >
>> >  >  A couple of questions:
>> >  >
>> >  >  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>> >  boot
>> >  >  option, would this work:
>> >  >
>> >  >  static int opt_softtsc = 0;
>> >  >  boolean_param("softtsc", opt_softtsc);
>> >  >    :
>> >  >  min = /* original code */
>> >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>> >  >
>> >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>> >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx
>> amd
>> >  >  regs->eax from tmptsc, are we almost done?
>> >  >
>> >  >  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>> >  called
>> >  >  to force the trap?
>> >  >
>> >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>> >  >  somewhere to emulate WRMSR of tsc?
>> >  >
>> >  >  Sorry I am not very familar with the details of the ia32 instruction
>> set.
>> >  >
>> >  >  Thanks,
>> >  >  Dan
>> >  >
>> >  >  > -----Original Message-----
>> >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>> >  >  > Sent: Monday, July 07, 2008 9:00 PM
>> >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> >  >  > Cc: Tian, Kevin
>> >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>> >  >  > feedback please)
>> >  >  >
>> >  >  >
>> >  >  > Hi, Dan
>> >  >  > This seems not to be enough. I will cook another patch.
>> >  >  >
>> >  >  > >  -----Original Message-----
>> >  >  > >  From: Zhang, Li
>> >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>> >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel
>> (E-mail)'
>> >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>> >  >  > feedback please)
>> >  >  > >
>> >  >  > >  Hi, Dan
>> >  >  > >
>> >  >  > >  I just add some code to produce vmexit. The function
>> >  >  > vmx_rdtsc_intercept()
>> >  >  > >  is not completed. It is needed to add some code to get the
>> >  >  > guest tsc.
>> >  >  > >
>> >  >  > >  Thanks
>> >  >  > >  --Li
>> >  >  > >
>> >  >  > >  >  -----Original Message-----
>> >  >  > >  >  From: Tian, Kevin
>> >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>> >  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>> >  >  > >  >  Cc: Zhang, Li
>> >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>> >  >  > AMD feedback please)
>> >  >  > >  >
>> >  >  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>> >  >  > >  >  >Sent: 2008年7月5日 1:32
>> >  >  > >  >  >
>> >  >  > >  >  >> >Is this something that you (or Intel in general)
>> >  >  > could look at?
>> >  >  > >  >  >> >I would be happy to participate but I don't think I
>> >  >  > understand
>> >  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>> >  >  > Xen system time
>> >  >  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>> >  >  > >  >  >> >
>> >  >  > >  >  >>
>> >  >  > >  >  >> There should be tiny related to VT, as only turning
>> >  >  > on some bit to
>> >  >  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>> >  >  > >  >  >> how to handle it. We'll take a look, but can't
>> >  >  > commit the time due
>> >  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
>> >  >  > jump in early
>> >  >  > >  >  >> we definitely can help with VT side.
>> >  >  > >  >  >
>> >  >  > >  >  >If you can post a patch with code that:
>> >  >  > >  >  >
>> >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>> >  >  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>> >  >  > >  >  >   domain causes a trap
>> >  >  > >  >  >3) the trap handler just does a physical rdtsc and returns
>> >  >  > >  >  >
>> >  >  > >  >  >then I could probably take it from there.
>> >  >  > >  >
>> >  >  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>> >  >  > >  >
>> >  >  > >  >  Thanks,
>> >  >  > >  >  Kevin
>> >  >  >
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>    
>>
>>------------------------------------------------------------------------
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>    
>>

Re: Guest TSC and Xen (Intel and AMD feedback please)

[Keir Fraser]

I would go for (uint32_t)tsc and (uint32_t)(tsc >> 32) as being even
clearer.

 -- Keir

On 8/7/08 15:48, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Hi Li,
> 
> Shouldn't this
> 
> + regs->eax = tsc & 0xffff;
> + regs->edx = tsc >> 32 & 0xffff;
> 
> be
> 
> + regs->eax = tsc & 0xffffffff;
> + regs->edx = tsc >> 32 & 0xffffffff;
> 
> Thanks,
> Dave
> 
> 
> Zhang, Li wrote:
> 
>> Hi, Dan
>> Please ignore previous comments. The above attachment is OK. Sorry for that.
>> :(
>> 
>>  
>> 
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>>> Sent: Tuesday, July 08, 2008 2:58 PM
>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>> Cc: Tian, Kevin
>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>>> 
>>> In fact, the issue is from guest. In the guest's terminal, it shows that
>>> "your
>>> time source seems to be instable..."
>>> 
>>>>  -----Original Message-----
>>>>  From: xen-devel-bounces@lists.xensource.com
>>>>  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>>>>  Sent: Tuesday, July 08, 2008 2:49 PM
>>>>  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>>>  Cc: Tian, Kevin
>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>>>> 
>>>>  Hi, Dan
>>>> 
>>>>  This is the patch which I think it has been completed VT. But there is an
>>>>  instable issue. The reason may be from that this interception is not bound
>>>>  to timer mode yet.
>>>> 
>>>>  Thanks
>>>>  --Li
>>>> 
>>>>>  -----Original Message-----
>>>>>  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>  Sent: Tuesday, July 08, 2008 11:34 AM
>>>>>  To: Zhang, Li; Xen-Devel (E-mail)
>>>>>  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback
>>> please)
>>>>> 
>>>>>  Thanks Kevin and Li --
>>>>> 
>>>>>  A couple of questions:
>>>>> 
>>>>>  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>>>>  boot
>>>>>  option, would this work:
>>>>> 
>>>>>  static int opt_softtsc = 0;
>>>>>  boolean_param("softtsc", opt_softtsc);
>>>>>    :
>>>>>  min = /* original code */
>>>>>  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>>>>> 
>>>>>  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>>>>>  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx
>>> amd
>>>>>  regs->eax from tmptsc, are we almost done?
>>>>> 
>>>>>  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>>>>  called
>>>>>  to force the trap?
>>>>> 
>>>>>  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>>>>>  somewhere to emulate WRMSR of tsc?
>>>>> 
>>>>>  Sorry I am not very familar with the details of the ia32 instruction
>>> set.
>>>>> 
>>>>>  Thanks,
>>>>>  Dan
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Zhang, Li [mailto:li.zhang@intel.com]
>>>>>> Sent: Monday, July 07, 2008 9:00 PM
>>>>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>>>>> Cc: Tian, Kevin
>>>>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>>>>>> feedback please)
>>>>>> 
>>>>>> 
>>>>>> Hi, Dan
>>>>>> This seems not to be enough. I will cook another patch.
>>>>>> 
>>>>>>>  -----Original Message-----
>>>>>>>  From: Zhang, Li
>>>>>>>  Sent: Tuesday, July 08, 2008 10:28 AM
>>>>>>>  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel
>>> (E-mail)'
>>>>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>>>>>> feedback please)
>>>>>>> 
>>>>>>>  Hi, Dan
>>>>>>> 
>>>>>>>  I just add some code to produce vmexit. The function
>>>>>> vmx_rdtsc_intercept()
>>>>>>>  is not completed. It is needed to add some code to get the
>>>>>> guest tsc.
>>>>>>> 
>>>>>>>  Thanks
>>>>>>>  --Li
>>>>>>> 
>>>>>>>>  -----Original Message-----
>>>>>>>>  From: Tian, Kevin
>>>>>>>>  Sent: Tuesday, July 08, 2008 9:39 AM
>>>>>>>>  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>>>>>>>>  Cc: Zhang, Li
>>>>>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>>>>>> AMD feedback please)
>>>>>>>> 
>>>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>>>>> Sent: 2008年7月5日 1:32
>>>>>>>>> 
>>>>>>>>>>> Is this something that you (or Intel in general)
>>>>>> could look at?
>>>>>>>>>>> I would be happy to participate but I don't think I
>>>>>> understand
>>>>>>>>>>> VT well enough.  Once the trap occurs, I suppose
>>>>>> Xen system time
>>>>>>>>>>> could be used as the virtual TSC, possibly scaled up.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> There should be tiny related to VT, as only turning
>>>>>> on some bit to
>>>>>>>>>> allow RDTSC trapping and then the rest stuff should be common
>>>>>>>>>> how to handle it. We'll take a look, but can't
>>>>>> commit the time due
>>>>>>>>>> to other scheduled bandwidth. But if you'd like to
>>>>>> jump in early
>>>>>>>>>> we definitely can help with VT side.
>>>>>>>>> 
>>>>>>>>> If you can post a patch with code that:
>>>>>>>>> 
>>>>>>>>> 1) declares a boolean global variable: softtsc = 0
>>>>>>>>> 2) if the variable is set, a rdtsc instruction in any hvm
>>>>>>>>>   domain causes a trap
>>>>>>>>> 3) the trap handler just does a physical rdtsc and returns
>>>>>>>>> 
>>>>>>>>> then I could probably take it from there.
>>>>>>>> 
>>>>>>>>  OK, and Li in CC will take a look and bake a patch for you.
>>>>>>>> 
>>>>>>>>  Thanks,
>>>>>>>>  Kevin
>>>>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>    
>>> 
>>> ------------------------------------------------------------------------
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>    
>>> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 7042 bytes --]

Looks great!  I am continuing to code and test.  Some preliminary results:

1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work!  The hvm
    guest believes it is running on a 1000.185 MHz clock.

2) On my test box, a guest rdtsc without softtsc takes about 30ns.  With softtsc,
    rdtsc takes about 1us.

3) On my user-land test program, I am seeing rare "Time goes backwards", even with
   hvm_get_guest_time().  I'm trying to track this down.

Dan

> -----Original Message-----
> From: Zhang, Li [mailto:li.zhang@intel.com]
> Sent: Tuesday, July 08, 2008 3:47 AM
> To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Cc: Tian, Kevin
> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> 
> 
> Hi, Dan
> Please ignore previous comments. The above attachment is OK. 
> Sorry for that. :(
> 
> >  -----Original Message-----
> >  From: xen-devel-bounces@lists.xensource.com
> >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf 
> Of Zhang, Li
> >  Sent: Tuesday, July 08, 2008 2:58 PM
> >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >  Cc: Tian, Kevin
> >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> >
> >  In fact, the issue is from guest. In the guest's terminal, 
> it shows that "your
> >  time source seems to be instable..."
> >
> >  >  -----Original Message-----
> >  >  From: xen-devel-bounces@lists.xensource.com
> >  >  [mailto:xen-devel-bounces@lists.xensource.com] On 
> Behalf Of Zhang, Li
> >  >  Sent: Tuesday, July 08, 2008 2:49 PM
> >  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >  >  Cc: Tian, Kevin
> >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and 
> AMD feedback please)
> >  >
> >  >  Hi, Dan
> >  >
> >  >  This is the patch which I think it has been completed 
> VT. But there is an
> >  >  instable issue. The reason may be from that this 
> interception is not bound
> >  >  to timer mode yet.
> >  >
> >  >  Thanks
> >  >  --Li
> >  >
> >  >  >  -----Original Message-----
> >  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
> >  >  >  To: Zhang, Li; Xen-Devel (E-mail)
> >  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
> >  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel 
> and AMD feedback
> >  please)
> >  >  >
> >  >  >  Thanks Kevin and Li --
> >  >  >
> >  >  >  A couple of questions:
> >  >  >
> >  >  >  1) If the EXITING flag to be set in vmcs.c is to be 
> controlled by a xen
> >  >  boot
> >  >  >  option, would this work:
> >  >  >
> >  >  >  static int opt_softtsc = 0;
> >  >  >  boolean_param("softtsc", opt_softtsc);
> >  >  >    :
> >  >  >  min = /* original code */
> >  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
> >  >  >
> >  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) 
> if we change the
> >  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and 
> then set regs->edx
> >  amd
> >  >  >  regs->eax from tmptsc, are we almost done?
> >  >  >
> >  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if 
> so, what code gets
> >  >  called
> >  >  >  to force the trap?
> >  >  >
> >  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is 
> there already code
> >  >  >  somewhere to emulate WRMSR of tsc?
> >  >  >
> >  >  >  Sorry I am not very familar with the details of the 
> ia32 instruction
> >  set.
> >  >  >
> >  >  >  Thanks,
> >  >  >  Dan
> >  >  >
> >  >  >  > -----Original Message-----
> >  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
> >  >  >  > Sent: Monday, July 07, 2008 9:00 PM
> >  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >  >  >  > Cc: Tian, Kevin
> >  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
> >  >  >  > feedback please)
> >  >  >  >
> >  >  >  >
> >  >  >  > Hi, Dan
> >  >  >  > This seems not to be enough. I will cook another patch.
> >  >  >  >
> >  >  >  > >  -----Original Message-----
> >  >  >  > >  From: Zhang, Li
> >  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
> >  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 
> 'Xen-Devel
> >  (E-mail)'
> >  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen 
> (Intel and AMD
> >  >  >  > feedback please)
> >  >  >  > >
> >  >  >  > >  Hi, Dan
> >  >  >  > >
> >  >  >  > >  I just add some code to produce vmexit. The function
> >  >  >  > vmx_rdtsc_intercept()
> >  >  >  > >  is not completed. It is needed to add some code 
> to get the
> >  >  >  > guest tsc.
> >  >  >  > >
> >  >  >  > >  Thanks
> >  >  >  > >  --Li
> >  >  >  > >
> >  >  >  > >  >  -----Original Message-----
> >  >  >  > >  >  From: Tian, Kevin
> >  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
> >  >  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
> >  >  >  > >  >  Cc: Zhang, Li
> >  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
> >  >  >  > AMD feedback please)
> >  >  >  > >  >
> >  >  >  > >  >  >From: Dan Magenheimer 
> [mailto:dan.magenheimer@oracle.com]
> >  >  >  > >  >  >Sent: 2008年7月5日 1:32
> >  >  >  > >  >  >
> >  >  >  > >  >  >> >Is this something that you (or Intel in general)
> >  >  >  > could look at?
> >  >  >  > >  >  >> >I would be happy to participate but I 
> don't think I
> >  >  >  > understand
> >  >  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
> >  >  >  > Xen system time
> >  >  >  > >  >  >> >could be used as the virtual TSC, 
> possibly scaled up.
> >  >  >  > >  >  >> >
> >  >  >  > >  >  >>
> >  >  >  > >  >  >> There should be tiny related to VT, as 
> only turning
> >  >  >  > on some bit to
> >  >  >  > >  >  >> allow RDTSC trapping and then the rest 
> stuff should be common
> >  >  >  > >  >  >> how to handle it. We'll take a look, but can't
> >  >  >  > commit the time due
> >  >  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
> >  >  >  > jump in early
> >  >  >  > >  >  >> we definitely can help with VT side.
> >  >  >  > >  >  >
> >  >  >  > >  >  >If you can post a patch with code that:
> >  >  >  > >  >  >
> >  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
> >  >  >  > >  >  >2) if the variable is set, a rdtsc 
> instruction in any hvm
> >  >  >  > >  >  >   domain causes a trap
> >  >  >  > >  >  >3) the trap handler just does a physical 
> rdtsc and returns
> >  >  >  > >  >  >
> >  >  >  > >  >  >then I could probably take it from there.
> >  >  >  > >  >
> >  >  >  > >  >  OK, and Li in CC will take a look and bake a 
> patch for you.
> >  >  >  > >  >
> >  >  >  > >  >  Thanks,
> >  >  >  > >  >  Kevin
> >  >  >  >
> >
> >  _______________________________________________
> >  Xen-devel mailing list
> >  Xen-devel@lists.xensource.com
> >  http://lists.xensource.com/xen-devel
> 
> 

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 8557 bytes --]

The attached patch adds a new Xen boot option "softtsc" that
causes all hvm domains to trap and emulate all rdtsc instructions
and fetch Xen system time instead of the hardware tsc.  As
a result, the processor will appear to be running at 1000.xyz MHz
(where xyz indicates some fraction that seems to vary slightly
across different boots).  I've also added a line of output to
"xm debug-key t" to report softsc and tsc-related CPU features.

(Keir, though 3.3 functionality freeze is past, this is more of a
diagnostic aid and defaults off, so I don't know if you would
consider it for 3.3.)

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>

Dan

P.S. Thanks to Li Zhang for coding the foundation for this patch.
Li, do you want to provide a signed-off-by line too?

> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: Tuesday, July 08, 2008 6:30 PM
> To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
> Cc: 'Tian, Kevin'
> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> 
> 
> Looks great!  I am continuing to code and test.  Some 
> preliminary results:
> 
> 1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() 
> seems to just work!  The hvm
>     guest believes it is running on a 1000.185 MHz clock.
> 
> 2) On my test box, a guest rdtsc without softtsc takes about 
> 30ns.  With softtsc,
>     rdtsc takes about 1us.
> 
> 3) On my user-land test program, I am seeing rare "Time goes 
> backwards", even with
>    hvm_get_guest_time().  I'm trying to track this down.
> 
> Dan
> 
> > -----Original Message-----
> > From: Zhang, Li [mailto:li.zhang@intel.com]
> > Sent: Tuesday, July 08, 2008 3:47 AM
> > To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > Cc: Tian, Kevin
> > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> > feedback please)
> > 
> > 
> > Hi, Dan
> > Please ignore previous comments. The above attachment is OK. 
> > Sorry for that. :(
> > 
> > >  -----Original Message-----
> > >  From: xen-devel-bounces@lists.xensource.com
> > >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf 
> > Of Zhang, Li
> > >  Sent: Tuesday, July 08, 2008 2:58 PM
> > >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > >  Cc: Tian, Kevin
> > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> > feedback please)
> > >
> > >  In fact, the issue is from guest. In the guest's terminal, 
> > it shows that "your
> > >  time source seems to be instable..."
> > >
> > >  >  -----Original Message-----
> > >  >  From: xen-devel-bounces@lists.xensource.com
> > >  >  [mailto:xen-devel-bounces@lists.xensource.com] On 
> > Behalf Of Zhang, Li
> > >  >  Sent: Tuesday, July 08, 2008 2:49 PM
> > >  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > >  >  Cc: Tian, Kevin
> > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and 
> > AMD feedback please)
> > >  >
> > >  >  Hi, Dan
> > >  >
> > >  >  This is the patch which I think it has been completed 
> > VT. But there is an
> > >  >  instable issue. The reason may be from that this 
> > interception is not bound
> > >  >  to timer mode yet.
> > >  >
> > >  >  Thanks
> > >  >  --Li
> > >  >
> > >  >  >  -----Original Message-----
> > >  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> > >  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
> > >  >  >  To: Zhang, Li; Xen-Devel (E-mail)
> > >  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
> > >  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel 
> > and AMD feedback
> > >  please)
> > >  >  >
> > >  >  >  Thanks Kevin and Li --
> > >  >  >
> > >  >  >  A couple of questions:
> > >  >  >
> > >  >  >  1) If the EXITING flag to be set in vmcs.c is to be 
> > controlled by a xen
> > >  >  boot
> > >  >  >  option, would this work:
> > >  >  >
> > >  >  >  static int opt_softtsc = 0;
> > >  >  >  boolean_param("softtsc", opt_softtsc);
> > >  >  >    :
> > >  >  >  min = /* original code */
> > >  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
> > >  >  >
> > >  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) 
> > if we change the
> > >  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and 
> > then set regs->edx
> > >  amd
> > >  >  >  regs->eax from tmptsc, are we almost done?
> > >  >  >
> > >  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if 
> > so, what code gets
> > >  >  called
> > >  >  >  to force the trap?
> > >  >  >
> > >  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is 
> > there already code
> > >  >  >  somewhere to emulate WRMSR of tsc?
> > >  >  >
> > >  >  >  Sorry I am not very familar with the details of the 
> > ia32 instruction
> > >  set.
> > >  >  >
> > >  >  >  Thanks,
> > >  >  >  Dan
> > >  >  >
> > >  >  >  > -----Original Message-----
> > >  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
> > >  >  >  > Sent: Monday, July 07, 2008 9:00 PM
> > >  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> > >  >  >  > Cc: Tian, Kevin
> > >  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
> > >  >  >  > feedback please)
> > >  >  >  >
> > >  >  >  >
> > >  >  >  > Hi, Dan
> > >  >  >  > This seems not to be enough. I will cook another patch.
> > >  >  >  >
> > >  >  >  > >  -----Original Message-----
> > >  >  >  > >  From: Zhang, Li
> > >  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
> > >  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 
> > 'Xen-Devel
> > >  (E-mail)'
> > >  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen 
> > (Intel and AMD
> > >  >  >  > feedback please)
> > >  >  >  > >
> > >  >  >  > >  Hi, Dan
> > >  >  >  > >
> > >  >  >  > >  I just add some code to produce vmexit. The function
> > >  >  >  > vmx_rdtsc_intercept()
> > >  >  >  > >  is not completed. It is needed to add some code 
> > to get the
> > >  >  >  > guest tsc.
> > >  >  >  > >
> > >  >  >  > >  Thanks
> > >  >  >  > >  --Li
> > >  >  >  > >
> > >  >  >  > >  >  -----Original Message-----
> > >  >  >  > >  >  From: Tian, Kevin
> > >  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
> > >  >  >  > >  >  To: 'dan.magenheimer@oracle.com'; 
> Xen-Devel (E-mail)
> > >  >  >  > >  >  Cc: Zhang, Li
> > >  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen 
> (Intel and
> > >  >  >  > AMD feedback please)
> > >  >  >  > >  >
> > >  >  >  > >  >  >From: Dan Magenheimer 
> > [mailto:dan.magenheimer@oracle.com]
> > >  >  >  > >  >  >Sent: 2008年7月5日 1:32
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >> >Is this something that you (or Intel 
> in general)
> > >  >  >  > could look at?
> > >  >  >  > >  >  >> >I would be happy to participate but I 
> > don't think I
> > >  >  >  > understand
> > >  >  >  > >  >  >> >VT well enough.  Once the trap occurs, 
> I suppose
> > >  >  >  > Xen system time
> > >  >  >  > >  >  >> >could be used as the virtual TSC, 
> > possibly scaled up.
> > >  >  >  > >  >  >> >
> > >  >  >  > >  >  >>
> > >  >  >  > >  >  >> There should be tiny related to VT, as 
> > only turning
> > >  >  >  > on some bit to
> > >  >  >  > >  >  >> allow RDTSC trapping and then the rest 
> > stuff should be common
> > >  >  >  > >  >  >> how to handle it. We'll take a look, but can't
> > >  >  >  > commit the time due
> > >  >  >  > >  >  >> to other scheduled bandwidth. But if 
> you'd like to
> > >  >  >  > jump in early
> > >  >  >  > >  >  >> we definitely can help with VT side.
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >If you can post a patch with code that:
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
> > >  >  >  > >  >  >2) if the variable is set, a rdtsc 
> > instruction in any hvm
> > >  >  >  > >  >  >   domain causes a trap
> > >  >  >  > >  >  >3) the trap handler just does a physical 
> > rdtsc and returns
> > >  >  >  > >  >  >
> > >  >  >  > >  >  >then I could probably take it from there.
> > >  >  >  > >  >
> > >  >  >  > >  >  OK, and Li in CC will take a look and bake a 
> > patch for you.
> > >  >  >  > >  >
> > >  >  >  > >  >  Thanks,
> > >  >  >  > >  >  Kevin
> > >  >  >  >
> > >
> > >  _______________________________________________
> > >  Xen-devel mailing list
> > >  Xen-devel@lists.xensource.com
> > >  http://lists.xensource.com/xen-devel
> > 
> > 

[-- Attachment #2: softtsc.patch --]
[-- Type: application/octet-stream, Size: 7521 bytes --]

diff -r bd97e45e073a xen/arch/x86/cpu/amd.c
--- a/xen/arch/x86/cpu/amd.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/cpu/amd.c	Wed Jul 09 15:13:55 2008 -0600
@@ -327,8 +327,10 @@ static void __init init_amd(struct cpuin
 
 	if (cpuid_eax(0x80000000) >= 0x80000007) {
 		c->x86_power = cpuid_edx(0x80000007);
-		if (c->x86_power & (1<<8))
+		if (c->x86_power & (1<<8)) {
 			set_bit(X86_FEATURE_CONSTANT_TSC, c->x86_capability);
+			constant_tsc = 1;
+		}
 	}
 
 #ifdef CONFIG_X86_HT
diff -r bd97e45e073a xen/arch/x86/cpu/common.c
--- a/xen/arch/x86/cpu/common.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/cpu/common.c	Wed Jul 09 15:13:55 2008 -0600
@@ -16,6 +16,8 @@
 
 #define tsc_disable 0
 #define disable_pse 0
+
+int constant_tsc = 0;
 
 static int cachesize_override __cpuinitdata = -1;
 static int disable_x86_fxsr __cpuinitdata;
diff -r bd97e45e073a xen/arch/x86/cpu/intel.c
--- a/xen/arch/x86/cpu/intel.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/cpu/intel.c	Wed Jul 09 15:13:55 2008 -0600
@@ -184,8 +184,10 @@ static void __devinit init_intel(struct 
 	if (c->x86 == 6) 
 		set_bit(X86_FEATURE_P3, c->x86_capability);
 	if ((c->x86 == 0xf && c->x86_model >= 0x03) ||
-		(c->x86 == 0x6 && c->x86_model >= 0x0e))
+		(c->x86 == 0x6 && c->x86_model >= 0x0e)) {
 		set_bit(X86_FEATURE_CONSTANT_TSC, c->x86_capability);
+		constant_tsc = 1;
+	}
 
 	start_vmx();
 }
diff -r bd97e45e073a xen/arch/x86/hvm/hvm.c
--- a/xen/arch/x86/hvm/hvm.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/hvm.c	Wed Jul 09 15:13:55 2008 -0600
@@ -56,6 +56,8 @@ int hvm_enabled __read_mostly;
 
 unsigned int opt_hvm_debug_level __read_mostly;
 integer_param("hvm_debug", opt_hvm_debug_level);
+int opt_softtsc = 0;
+boolean_param("softtsc", opt_softtsc);
 
 struct hvm_function_table hvm_funcs __read_mostly;
 
@@ -148,7 +150,10 @@ u64 hvm_get_guest_tsc(struct vcpu *v)
 {
     u64 host_tsc;
 
-    rdtscll(host_tsc);
+    if ( opt_softtsc )
+        host_tsc = hvm_get_guest_time(v);
+    else
+        rdtscll(host_tsc);
     return host_tsc + v->arch.hvm_vcpu.cache_tsc_offset;
 }
 
@@ -1640,6 +1645,16 @@ void hvm_cpuid(unsigned int input, unsig
     }
 }
 
+void hvm_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = (uint32_t)tsc;
+    regs->edx = (uint32_t)(tsc >> 32);
+}
+
 int hvm_msr_read_intercept(struct cpu_user_regs *regs)
 {
     uint32_t ecx = regs->ecx;
diff -r bd97e45e073a xen/arch/x86/hvm/svm/svm.c
--- a/xen/arch/x86/hvm/svm/svm.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/svm/svm.c	Wed Jul 09 15:13:55 2008 -0600
@@ -52,6 +52,8 @@
 #include <asm/hap.h>
 
 u32 svm_feature_flags;
+
+extern int opt_softtsc;
 
 #define set_segment_register(name, value)  \
     asm volatile ( "movw %%ax ,%%" STR(name) "" : : "a" (value) )
@@ -1340,6 +1342,13 @@ asmlinkage void svm_vmexit_handler(struc
         hvm_triple_fault();
         break;
 
+    case VMEXIT_RDTSC:
+        if ( opt_softtsc )
+        {
+            hvm_rdtsc_intercept(regs);
+            break;
+        }
+        /* else fall through */
     case VMEXIT_RDTSCP:
     case VMEXIT_MONITOR:
     case VMEXIT_MWAIT:
diff -r bd97e45e073a xen/arch/x86/hvm/svm/vmcb.c
--- a/xen/arch/x86/hvm/svm/vmcb.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/svm/vmcb.c	Wed Jul 09 15:13:55 2008 -0600
@@ -39,6 +39,8 @@
 #include <xen/keyhandler.h>
 
 extern int svm_dbg_on;
+
+extern int opt_softtsc;
 
 #define IOPM_SIZE   (12 * 1024)
 #define MSRPM_SIZE  (8  * 1024)
@@ -165,6 +167,9 @@ static int construct_vmcb(struct vcpu *v
 
     /* TSC. */
     vmcb->tsc_offset = 0;
+    if ( opt_softtsc )
+        vmcb->general1_intercepts |= GENERAL1_INTERCEPT_RDTSC;
+
     
     /* Guest EFER: *must* contain SVME or VMRUN will fail. */
     vmcb->efer = EFER_SVME;
diff -r bd97e45e073a xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Wed Jul 09 15:13:55 2008 -0600
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+extern int opt_softtsc;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@ static void vmx_init_vmcs_config(void)
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (opt_softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
diff -r bd97e45e073a xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Wed Jul 09 15:13:55 2008 -0600
@@ -2202,6 +2202,11 @@ asmlinkage void vmx_vmexit_handler(struc
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        hvm_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;
diff -r bd97e45e073a xen/arch/x86/time.c
--- a/xen/arch/x86/time.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/arch/x86/time.c	Wed Jul 09 15:13:55 2008 -0600
@@ -68,7 +68,7 @@ static DEFINE_PER_CPU(struct cpu_time, c
 static DEFINE_PER_CPU(struct cpu_time, cpu_time);
 
 /* TSC is invariant on C state entry? */
-static bool_t tsc_invariant;
+bool_t tsc_invariant;
 
 /*
  * We simulate a 32-bit platform timer from the 16-bit PIT ch2 counter.
diff -r bd97e45e073a xen/common/keyhandler.c
--- a/xen/common/keyhandler.c	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/common/keyhandler.c	Wed Jul 09 15:13:55 2008 -0600
@@ -251,6 +251,8 @@ static void read_clocks(unsigned char ke
     unsigned int cpu = smp_processor_id(), min_cpu, max_cpu;
     u64 min, max, dif, difus;
     static DEFINE_SPINLOCK(lock);
+    extern int opt_softtsc, constant_tsc;
+    extern bool_t tsc_invariant;
 
     spin_lock(&lock);
 
@@ -284,6 +286,8 @@ static void read_clocks(unsigned char ke
     printk("Min = %"PRIu64" ; Max = %"PRIu64" ; Diff = %"PRIu64
            " (%"PRIu64" microseconds)\n",
            min, max, dif, difus);
+    printk("softtsc = %d, constant_tsc = %d, tsc_invariant = %d\n",
+         opt_softtsc, constant_tsc, (int)tsc_invariant);
 }
 
 extern void dump_runq(unsigned char key);
diff -r bd97e45e073a xen/include/asm-x86/cpufeature.h
--- a/xen/include/asm-x86/cpufeature.h	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/include/asm-x86/cpufeature.h	Wed Jul 09 15:13:55 2008 -0600
@@ -74,6 +74,8 @@
 #define X86_FEATURE_P3		(3*32+ 6) /* P3 */
 #define X86_FEATURE_P4		(3*32+ 7) /* P4 */
 #define X86_FEATURE_CONSTANT_TSC (3*32+ 8) /* TSC ticks at a constant rate */
+
+extern int constant_tsc;
 
 /* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */
 #define X86_FEATURE_XMM3	(4*32+ 0) /* Streaming SIMD Extensions-3 */
diff -r bd97e45e073a xen/include/asm-x86/hvm/support.h
--- a/xen/include/asm-x86/hvm/support.h	Tue Jul 08 09:28:50 2008 +0100
+++ b/xen/include/asm-x86/hvm/support.h	Wed Jul 09 15:13:55 2008 -0600
@@ -125,6 +125,8 @@ void hvm_hlt(unsigned long rflags);
 void hvm_hlt(unsigned long rflags);
 void hvm_triple_fault(void);
 
+void hvm_rdtsc_intercept(struct cpu_user_regs *regs);
+
 /* These functions all return X86EMUL return codes. */
 int hvm_set_efer(uint64_t value);
 int hvm_set_cr0(unsigned long value);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

[Zhang, Li]

OK. 
Signed-off-by: Li Zhang <li.zhang@intel.com>

Thanks 
>  -----Original Message-----
>  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  Sent: Thursday, July 10, 2008 5:33 AM
>  To: dan.magenheimer@oracle.com; Zhang, Li; Xen-Devel (E-mail)
>  Cc: Tian, Kevin
>  Subject: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and
>  AMD feedback please))
>  
>  The attached patch adds a new Xen boot option "softtsc" that
>  causes all hvm domains to trap and emulate all rdtsc instructions
>  and fetch Xen system time instead of the hardware tsc.  As
>  a result, the processor will appear to be running at 1000.xyz MHz
>  (where xyz indicates some fraction that seems to vary slightly
>  across different boots).  I've also added a line of output to
>  "xm debug-key t" to report softsc and tsc-related CPU features.
>  
>  (Keir, though 3.3 functionality freeze is past, this is more of a
>  diagnostic aid and defaults off, so I don't know if you would
>  consider it for 3.3.)
>  
>  Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
>  
>  Dan
>  
>  P.S. Thanks to Li Zhang for coding the foundation for this patch.
>  Li, do you want to provide a signed-off-by line too?
>  
>  > -----Original Message-----
>  > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  > Sent: Tuesday, July 08, 2008 6:30 PM
>  > To: 'Zhang, Li'; 'Xen-Devel (E-mail)'
>  > Cc: 'Tian, Kevin'
>  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > feedback please)
>  >
>  >
>  > Looks great!  I am continuing to code and test.  Some
>  > preliminary results:
>  >
>  > 1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc()
>  > seems to just work!  The hvm
>  >     guest believes it is running on a 1000.185 MHz clock.
>  >
>  > 2) On my test box, a guest rdtsc without softtsc takes about
>  > 30ns.  With softtsc,
>  >     rdtsc takes about 1us.
>  >
>  > 3) On my user-land test program, I am seeing rare "Time goes
>  > backwards", even with
>  >    hvm_get_guest_time().  I'm trying to track this down.
>  >
>  > Dan
>  >
>  > > -----Original Message-----
>  > > From: Zhang, Li [mailto:li.zhang@intel.com]
>  > > Sent: Tuesday, July 08, 2008 3:47 AM
>  > > To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  > > Cc: Tian, Kevin
>  > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > > feedback please)
>  > >
>  > >
>  > > Hi, Dan
>  > > Please ignore previous comments. The above attachment is OK.
>  > > Sorry for that. :(
>  > >
>  > > >  -----Original Message-----
>  > > >  From: xen-devel-bounces@lists.xensource.com
>  > > >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf
>  > > Of Zhang, Li
>  > > >  Sent: Tuesday, July 08, 2008 2:58 PM
>  > > >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  > > >  Cc: Tian, Kevin
>  > > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > > feedback please)
>  > > >
>  > > >  In fact, the issue is from guest. In the guest's terminal,
>  > > it shows that "your
>  > > >  time source seems to be instable..."
>  > > >
>  > > >  >  -----Original Message-----
>  > > >  >  From: xen-devel-bounces@lists.xensource.com
>  > > >  >  [mailto:xen-devel-bounces@lists.xensource.com] On
>  > > Behalf Of Zhang, Li
>  > > >  >  Sent: Tuesday, July 08, 2008 2:49 PM
>  > > >  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  > > >  >  Cc: Tian, Kevin
>  > > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  > > AMD feedback please)
>  > > >  >
>  > > >  >  Hi, Dan
>  > > >  >
>  > > >  >  This is the patch which I think it has been completed
>  > > VT. But there is an
>  > > >  >  instable issue. The reason may be from that this
>  > > interception is not bound
>  > > >  >  to timer mode yet.
>  > > >  >
>  > > >  >  Thanks
>  > > >  >  --Li
>  > > >  >
>  > > >  >  >  -----Original Message-----
>  > > >  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  > > >  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
>  > > >  >  >  To: Zhang, Li; Xen-Devel (E-mail)
>  > > >  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  > > >  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel
>  > > and AMD feedback
>  > > >  please)
>  > > >  >  >
>  > > >  >  >  Thanks Kevin and Li --
>  > > >  >  >
>  > > >  >  >  A couple of questions:
>  > > >  >  >
>  > > >  >  >  1) If the EXITING flag to be set in vmcs.c is to be
>  > > controlled by a xen
>  > > >  >  boot
>  > > >  >  >  option, would this work:
>  > > >  >  >
>  > > >  >  >  static int opt_softtsc = 0;
>  > > >  >  >  boolean_param("softtsc", opt_softtsc);
>  > > >  >  >    :
>  > > >  >  >  min = /* original code */
>  > > >  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  > > >  >  >
>  > > >  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/)
>  > > if we change the
>  > > >  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and
>  > > then set regs->edx
>  > > >  amd
>  > > >  >  >  regs->eax from tmptsc, are we almost done?
>  > > >  >  >
>  > > >  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if
>  > > so, what code gets
>  > > >  >  called
>  > > >  >  >  to force the trap?
>  > > >  >  >
>  > > >  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is
>  > > there already code
>  > > >  >  >  somewhere to emulate WRMSR of tsc?
>  > > >  >  >
>  > > >  >  >  Sorry I am not very familar with the details of the
>  > > ia32 instruction
>  > > >  set.
>  > > >  >  >
>  > > >  >  >  Thanks,
>  > > >  >  >  Dan
>  > > >  >  >
>  > > >  >  >  > -----Original Message-----
>  > > >  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  > > >  >  >  > Sent: Monday, July 07, 2008 9:00 PM
>  > > >  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  > > >  >  >  > Cc: Tian, Kevin
>  > > >  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > > >  >  >  > feedback please)
>  > > >  >  >  >
>  > > >  >  >  >
>  > > >  >  >  > Hi, Dan
>  > > >  >  >  > This seems not to be enough. I will cook another patch.
>  > > >  >  >  >
>  > > >  >  >  > >  -----Original Message-----
>  > > >  >  >  > >  From: Zhang, Li
>  > > >  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  > > >  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com';
>  > > 'Xen-Devel
>  > > >  (E-mail)'
>  > > >  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen
>  > > (Intel and AMD
>  > > >  >  >  > feedback please)
>  > > >  >  >  > >
>  > > >  >  >  > >  Hi, Dan
>  > > >  >  >  > >
>  > > >  >  >  > >  I just add some code to produce vmexit. The function
>  > > >  >  >  > vmx_rdtsc_intercept()
>  > > >  >  >  > >  is not completed. It is needed to add some code
>  > > to get the
>  > > >  >  >  > guest tsc.
>  > > >  >  >  > >
>  > > >  >  >  > >  Thanks
>  > > >  >  >  > >  --Li
>  > > >  >  >  > >
>  > > >  >  >  > >  >  -----Original Message-----
>  > > >  >  >  > >  >  From: Tian, Kevin
>  > > >  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  > > >  >  >  > >  >  To: 'dan.magenheimer@oracle.com';
>  > Xen-Devel (E-mail)
>  > > >  >  >  > >  >  Cc: Zhang, Li
>  > > >  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen
>  > (Intel and
>  > > >  >  >  > AMD feedback please)
>  > > >  >  >  > >  >
>  > > >  >  >  > >  >  >From: Dan Magenheimer
>  > > [mailto:dan.magenheimer@oracle.com]
>  > > >  >  >  > >  >  >Sent: 2008年7月5日 1:32
>  > > >  >  >  > >  >  >
>  > > >  >  >  > >  >  >> >Is this something that you (or Intel
>  > in general)
>  > > >  >  >  > could look at?
>  > > >  >  >  > >  >  >> >I would be happy to participate but I
>  > > don't think I
>  > > >  >  >  > understand
>  > > >  >  >  > >  >  >> >VT well enough.  Once the trap occurs,
>  > I suppose
>  > > >  >  >  > Xen system time
>  > > >  >  >  > >  >  >> >could be used as the virtual TSC,
>  > > possibly scaled up.
>  > > >  >  >  > >  >  >> >
>  > > >  >  >  > >  >  >>
>  > > >  >  >  > >  >  >> There should be tiny related to VT, as
>  > > only turning
>  > > >  >  >  > on some bit to
>  > > >  >  >  > >  >  >> allow RDTSC trapping and then the rest
>  > > stuff should be common
>  > > >  >  >  > >  >  >> how to handle it. We'll take a look, but can't
>  > > >  >  >  > commit the time due
>  > > >  >  >  > >  >  >> to other scheduled bandwidth. But if
>  > you'd like to
>  > > >  >  >  > jump in early
>  > > >  >  >  > >  >  >> we definitely can help with VT side.
>  > > >  >  >  > >  >  >
>  > > >  >  >  > >  >  >If you can post a patch with code that:
>  > > >  >  >  > >  >  >
>  > > >  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>  > > >  >  >  > >  >  >2) if the variable is set, a rdtsc
>  > > instruction in any hvm
>  > > >  >  >  > >  >  >   domain causes a trap
>  > > >  >  >  > >  >  >3) the trap handler just does a physical
>  > > rdtsc and returns
>  > > >  >  >  > >  >  >
>  > > >  >  >  > >  >  >then I could probably take it from there.
>  > > >  >  >  > >  >
>  > > >  >  >  > >  >  OK, and Li in CC will take a look and bake a
>  > > patch for you.
>  > > >  >  >  > >  >
>  > > >  >  >  > >  >  Thanks,
>  > > >  >  >  > >  >  Kevin
>  > > >  >  >  >
>  > > >
>  > > >  _______________________________________________
>  > > >  Xen-devel mailing list
>  > > >  Xen-devel@lists.xensource.com
>  > > >  http://lists.xensource.com/xen-devel
>  > >
>  > >

Re: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

[Keir Fraser]

On 9/7/08 22:32, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> The attached patch adds a new Xen boot option "softtsc" that
> causes all hvm domains to trap and emulate all rdtsc instructions
> and fetch Xen system time instead of the hardware tsc.  As
> a result, the processor will appear to be running at 1000.xyz MHz
> (where xyz indicates some fraction that seems to vary slightly
> across different boots).  I've also added a line of output to
> "xm debug-key t" to report softsc and tsc-related CPU features.
> 
> (Keir, though 3.3 functionality freeze is past, this is more of a
> diagnostic aid and defaults off, so I don't know if you would
> consider it for 3.3.)

Yes, I will take it, but have the following comments.
 1. Why do you define new boolean 'constant_tsc'? Can you just use
test_bit(X86_FEATURE_CONSTANT_TSC)?
 2. Your change in common/keyhandler.c breaks ia64. :-)
 3. Your change to arch/x86/time.c looks unnecessary.
 4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?

 -- Keir

RE: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

[Dan Magenheimer]

> Yes, I will take it, but have the following comments.
>  2. Your change in common/keyhandler.c breaks ia64. :-)

Oops! ;-)  What's the best way to handle this?  It would
be unfortunate to lose valuable debug data just because its
arch-dependent but I don't see any other arch-dependent code
in keyhandler.c and I'll bet you don't want to start adding
ifdef's nor introduce arch/xxx/keyhandler.c just for this.

>  1. Why do you define new boolean 'constant_tsc'? Can you just use
> test_bit(X86_FEATURE_CONSTANT_TSC)?
>  3. Your change to arch/x86/time.c looks unnecessary.

I was thinking that the tests for these features should probably
be abstracted (e.g. static inline in a header file or a
global function), but wasn't sure about the best way to deal
with the datatypes (e.g. struct cpuinfo_x86) so defaulted to
global variables.

Both globals are simply for debug output in keyhandler.c
so depending on the answer to (2) above, those patch-parts
could just go away.

>  4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?

I thought I remembered seeing code that reported/lied to guests
that the rdtscp feature was not present?

Thanks,
Dan

Re: [PATCH] softtsc (was RE: Guest TSC and Xen (Intel and AMD feedback please))

[Keir Fraser]

Hmm.. okay I think I will throw away the debug-output pieces of the patch.
They are logically separate and distinct anyway.

 -- Keir

On 10/7/08 15:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

>> Yes, I will take it, but have the following comments.
>>  2. Your change in common/keyhandler.c breaks ia64. :-)
> 
> Oops! ;-)  What's the best way to handle this?  It would
> be unfortunate to lose valuable debug data just because its
> arch-dependent but I don't see any other arch-dependent code
> in keyhandler.c and I'll bet you don't want to start adding
> ifdef's nor introduce arch/xxx/keyhandler.c just for this.
> 
>>  1. Why do you define new boolean 'constant_tsc'? Can you just use
>> test_bit(X86_FEATURE_CONSTANT_TSC)?
>>  3. Your change to arch/x86/time.c looks unnecessary.
> 
> I was thinking that the tests for these features should probably
> be abstracted (e.g. static inline in a header file or a
> global function), but wasn't sure about the best way to deal
> with the datatypes (e.g. struct cpuinfo_x86) so defaulted to
> global variables.
> 
> Both globals are simply for debug output in keyhandler.c
> so depending on the answer to (2) above, those patch-parts
> could just go away.
> 
>>  4. Should you catch SVM's RDTSCP vmexit as well as RDTSC?
> 
> I thought I remembered seeing code that reported/lied to guests
> that the rdtscp feature was not present?
> 
> Thanks,
> Dan
> 

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

Hi Andy --

Thanks for the reply.

No, I don't think the TSC offset capabilities in VT are sufficient.
If you are migrating from a TSC-synchronized SMP system and TSC
was selected as the clocksource by the guest at boot *because* TSC
is always synchronized, and then you migrate to a system where TSC
is not synchronized, Xen can synchronize it once at migrate time
but then the TSC's on the target system will immediately start to
diverge.  So TSC might be a reasonable clocksource on the first
system but not on the target system.  One could of course use
CPUID to disallow a TSC-unsynchronized host as a suitable target
for a TSC-synchronized-assumed guest, but that seems overly restrictive,
especially if TSC wasn't selected as the clocksource for the guest
and/or the guest (and its apps) isn't particularly time-sensitive.

> I don't know why you want to single out TSC here.

I'm singleing it out because it is a per-cpu clock rather
than a platform timer... a platform timer can be (and indeed
is) offset'ed on migration and that is sufficient if it is
selected as the clocksource.

> That is what Linux is testing for anyways. If it decides it is
> ok it is fine.

Not sure... if Linux thinks it is running on a uniprocessor,
but Xen reschedules this uniprocessor Linux guest on a different
processor on the same physical SMP system, does Xen adjust the
potential TSC difference?  I could be wrong, but I think not.

> The reason why it is an advantage to try to make TSC btw
> is that it is *much* faster than any other timer and there
> are definitely workloads that are very timer intensive.

Yes, understood, but if a timer-intensive application makes
the assumption that TSC is synchronized and thus will never
go backwards, but TSC is not synchronized and it DOES (apparently)
go backwards due to Xen scheduler or migration, a slower timer
might have been preferred.

Dan

> -----Original Message-----
> From: Andi Kleen [mailto:andi@firstfloor.org]
> Sent: Tuesday, July 01, 2008 2:42 PM
> To: dan.magenheimer@oracle.com
> Cc: Xen-Devel (E-mail)
> Subject: Re: Guest TSC and Xen (Intel and AMD feedback please)
> 
> 
> "Dan Magenheimer" <dan.magenheimer@oracle.com> writes:
> >
> > 1) Often this test is done once at guest boot; if the guest
> >    migrates to another machine without the bit set, time
> >    will be erratic.
> 
> First it doesn't have to be. Xen could use the TSC offset capabilities
> in VT to simulate a relatively [you would likely get a small hickup
> during the migration, but nothing really bad] stable TSC between
> machines per guest (assuming the TSC does not diverge by itself).
> That would require a TSC negotation phase during migration.
> 
> Then the tests are just a special instance of a CPUID bit test
> [although they sometimes check other things too, but from this
> perspective they are the same] and when your CPUID and other
> configuration values are not controlled between different systems you
> migrate to you cannot safely migrate anyways because everything else
> that depends on CPUID features will fail too.
> 
> I don't know why you want to single out TSC here.
> 
> > 2) I *think* that in some cases even within the same system,
> >    TSC values will skew somewhat.  Since a uniprocessor guest
> >    will often be rescheduled to a different pcpu by Xen,
> >    the underlying tsc may appear erratic.
> 
> That is what Linux is testing for anyways. If it decides it is
> ok it is fine.
> 
> The reason why it is an advantage to try to make TSC btw
> is that it is *much* faster than any other timer and there
> are definitely workloads that are very timer intensive.
> 
> -Andi
>