RE: Guest TSC and Xen (Intel and AMD feedback please)

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

Hi Andy --

Thanks for the reply.

No, I don't think the TSC offset capabilities in VT are sufficient.
If you are migrating from a TSC-synchronized SMP system and TSC
was selected as the clocksource by the guest at boot *because* TSC
is always synchronized, and then you migrate to a system where TSC
is not synchronized, Xen can synchronize it once at migrate time
but then the TSC's on the target system will immediately start to
diverge.  So TSC might be a reasonable clocksource on the first
system but not on the target system.  One could of course use
CPUID to disallow a TSC-unsynchronized host as a suitable target
for a TSC-synchronized-assumed guest, but that seems overly restrictive,
especially if TSC wasn't selected as the clocksource for the guest
and/or the guest (and its apps) isn't particularly time-sensitive.

> I don't know why you want to single out TSC here.

I'm singleing it out because it is a per-cpu clock rather
than a platform timer... a platform timer can be (and indeed
is) offset'ed on migration and that is sufficient if it is
selected as the clocksource.

> That is what Linux is testing for anyways. If it decides it is
> ok it is fine.

Not sure... if Linux thinks it is running on a uniprocessor,
but Xen reschedules this uniprocessor Linux guest on a different
processor on the same physical SMP system, does Xen adjust the
potential TSC difference?  I could be wrong, but I think not.

> The reason why it is an advantage to try to make TSC btw
> is that it is *much* faster than any other timer and there
> are definitely workloads that are very timer intensive.

Yes, understood, but if a timer-intensive application makes
the assumption that TSC is synchronized and thus will never
go backwards, but TSC is not synchronized and it DOES (apparently)
go backwards due to Xen scheduler or migration, a slower timer
might have been preferred.

Dan

> -----Original Message-----
> From: Andi Kleen [mailto:andi@firstfloor.org]
> Sent: Tuesday, July 01, 2008 2:42 PM
> To: dan.magenheimer@oracle.com
> Cc: Xen-Devel (E-mail)
> Subject: Re: Guest TSC and Xen (Intel and AMD feedback please)
> 
> 
> "Dan Magenheimer" <dan.magenheimer@oracle.com> writes:
> >
> > 1) Often this test is done once at guest boot; if the guest
> >    migrates to another machine without the bit set, time
> >    will be erratic.
> 
> First it doesn't have to be. Xen could use the TSC offset capabilities
> in VT to simulate a relatively [you would likely get a small hickup
> during the migration, but nothing really bad] stable TSC between
> machines per guest (assuming the TSC does not diverge by itself).
> That would require a TSC negotation phase during migration.
> 
> Then the tests are just a special instance of a CPUID bit test
> [although they sometimes check other things too, but from this
> perspective they are the same] and when your CPUID and other
> configuration values are not controlled between different systems you
> migrate to you cannot safely migrate anyways because everything else
> that depends on CPUID features will fail too.
> 
> I don't know why you want to single out TSC here.
> 
> > 2) I *think* that in some cases even within the same system,
> >    TSC values will skew somewhat.  Since a uniprocessor guest
> >    will often be rescheduled to a different pcpu by Xen,
> >    the underlying tsc may appear erratic.
> 
> That is what Linux is testing for anyways. If it decides it is
> ok it is fine.
> 
> The reason why it is an advantage to try to make TSC btw
> is that it is *much* faster than any other timer and there
> are definitely workloads that are very timer intensive.
> 
> -Andi
>

Re: RE: Guest TSC and Xen (Intel and AMD feedback please)

[Jeremy Fitzhardinge]

Dan Magenheimer wrote:
> Hi Andy --
>
> Thanks for the reply.
>
> No, I don't think the TSC offset capabilities in VT are sufficient.
> If you are migrating from a TSC-synchronized SMP system and TSC
> was selected as the clocksource by the guest at boot *because* TSC
> is always synchronized, and then you migrate to a system where TSC
> is not synchronized, Xen can synchronize it once at migrate time
> but then the TSC's on the target system will immediately start to
> diverge.  So TSC might be a reasonable clocksource on the first
> system but not on the target system.  One could of course use
> CPUID to disallow a TSC-unsynchronized host as a suitable target
> for a TSC-synchronized-assumed guest, but that seems overly restrictive,
> especially if TSC wasn't selected as the clocksource for the guest
> and/or the guest (and its apps) isn't particularly time-sensitive.
>   

Linux (as in, relatively modern kernels, at least) will downgrade the 
tsc if it starts misbehaving.  This can happen, for example, if the tsc 
seems fine on initial inspection, but then starts misbehaving when the 
cpu frequency changes.

On the other hand, if its a modern kernel you may as well use a pv linux 
kernel and get a proper pv clocksource.

This is an area in which kernel behaviour has been changing a lot over 
the last few years, so its hard to guess what any given kernel version 
(+hacked by vendor) will do in this case...

    J

RE: RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer
>Sent: 2008年7月2日 5:43
>
>Hi Andy --
>
>Thanks for the reply.
>
>No, I don't think the TSC offset capabilities in VT are sufficient.
>If you are migrating from a TSC-synchronized SMP system and TSC
>was selected as the clocksource by the guest at boot *because* TSC
>is always synchronized, and then you migrate to a system where TSC
>is not synchronized, Xen can synchronize it once at migrate time
>but then the TSC's on the target system will immediately start to

If dest and src boxes are with same bits, saying same tsc freq, then 
the divergence is a fixed value and TSC offset adjustment is sufficient.
Or else TSC offset can't help.

>diverge.  So TSC might be a reasonable clocksource on the first
>system but not on the target system.  One could of course use
>CPUID to disallow a TSC-unsynchronized host as a suitable target
>for a TSC-synchronized-assumed guest, but that seems overly 
>restrictive,
>especially if TSC wasn't selected as the clocksource for the guest
>and/or the guest (and its apps) isn't particularly time-sensitive.

So if it's an option instead of a 'NEVER' clause, I agree as:

a) user may mark TSC unstable to migrate among boxes with 
mismatching TSC bits (bus crystal, cpu freq impact, etc.)

b) user may always use TSC as clocksource and then trap RDTSC
when migrating to a box with mismatching TSC bits

c) user may always use TSC as clocksource when migrating to a
box with same TSC bits, by adjusting TSC offset

d) migration may be prevented since no reliable methods to ensure
a)'s effect. Such prevention then falls into generic CPUID comparison
involved in migration

>
>> I don't know why you want to single out TSC here.
>
>I'm singleing it out because it is a per-cpu clock rather
>than a platform timer... a platform timer can be (and indeed
>is) offset'ed on migration and that is sufficient if it is
>selected as the clocksource.

The problem is not per-cpu vs platform, IMO. Instead, it's the 
problem that currently guest TSC is conveyed by host TSC plus 
an offset approach, without read trap. If you also virtualize a 
platform clocksource by a real one, like dedicating a HPET ch,
same concern also raises.

>
>> That is what Linux is testing for anyways. If it decides it is
>> ok it is fine.
>
>Not sure... if Linux thinks it is running on a uniprocessor,
>but Xen reschedules this uniprocessor Linux guest on a different
>processor on the same physical SMP system, does Xen adjust the
>potential TSC difference?  I could be wrong, but I think not.

Xen can do and should be, since SMP system is driven by same
crystal and thus host TSC is synced. But I guess by far Xen hasn't
do it, since the TSC drift (dozen of cycles) is smaller than the overhead
to migrate a vcpu. Thus guest won't observe a backward value in
theory.

>
>> The reason why it is an advantage to try to make TSC btw
>> is that it is *much* faster than any other timer and there
>> are definitely workloads that are very timer intensive.

Curiously, how much downgrade using a platform clock source may be,
for a time-intensive workload?

>
>Yes, understood, but if a timer-intensive application makes
>the assumption that TSC is synchronized and thus will never
>go backwards, but TSC is not synchronized and it DOES (apparently)
>go backwards due to Xen scheduler or migration, a slower timer
>might have been preferred.

Shouldn't this be a software bug instead?

Thanks,
Kevin

RE: RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 3193 bytes --]

Hi Kevin --

> a) user may mark TSC unstable to migrate among boxes with
> mismatching TSC bits (bus crystal, cpu freq impact, etc.)
>
> b) user may always use TSC as clocksource and then trap RDTSC
> when migrating to a box with mismatching TSC bits
>
> c) user may always use TSC as clocksource when migrating to a
> box with same TSC bits, by adjusting TSC offset
>
> d) migration may be prevented since no reliable methods to ensure
> a)'s effect. Such prevention then falls into generic CPUID comparison
> involved in migration

I think this is an excellent summary of the choices.

> >> I don't know why you want to single out TSC here.
> >
> >I'm singleing it out because it is a per-cpu clock rather
> >than a platform timer... a platform timer can be (and indeed
> >is) offset'ed on migration and that is sufficient if it is
> >selected as the clocksource.
>
> The problem is not per-cpu vs platform, IMO. Instead, it's the
> problem that currently guest TSC is conveyed by host TSC plus
> an offset approach, without read trap. If you also virtualize a
> platform clocksource by a real one, like dedicating a HPET ch,
> same concern also raises.

Yes, you are right.

> >> That is what Linux is testing for anyways. If it decides it is
> >> ok it is fine.
> >
> >Not sure... if Linux thinks it is running on a uniprocessor,
> >but Xen reschedules this uniprocessor Linux guest on a different
> >processor on the same physical SMP system, does Xen adjust the
> >potential TSC difference?  I could be wrong, but I think not.
>
> Xen can do and should be, since SMP system is driven by same
> crystal and thus host TSC is synced. But I guess by far Xen hasn't
> do it, since the TSC drift (dozen of cycles) is smaller than
> the overhead
> to migrate a vcpu. Thus guest won't observe a backward value in
> theory.

I'm not sure this is always the case (though the patch I posted
earlier today may indicate there is something else going on that
has led to my assumption that TSC was skewing worse than dozens
of cycles).

> >> The reason why it is an advantage to try to make TSC btw
> >> is that it is *much* faster than any other timer and there
> >> are definitely workloads that are very timer intensive.
>
> Curiously, how much downgrade using a platform clock source may be,
> for a time-intensive workload?

A good question.  We have a workload that spends >10% of its time
doing gettimeoffset_tsc()... not sure if that is realistic but
it would be interesting to measure that if it used an hpet instead
or if rdtsc was fully emulated.

> >Yes, understood, but if a timer-intensive application makes
> >the assumption that TSC is synchronized and thus will never
> >go backwards, but TSC is not synchronized and it DOES (apparently)
> >go backwards due to Xen scheduler or migration, a slower timer
> >might have been preferred.
>
> Shouldn't this be a software bug instead?

If the application is smart enough to check the TSC bits when
it launches, but stability changes later due to migration/scheduling,
I'm not sure htis is an application bug.  Or did I misunderstand
your question?

Thanks,
Dan

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

Various versions of Linux under various circumstances select
TSC as the primary clocksource for the kernel.  This is
especially true for uniprocessor kernels, but also in some
cases for multiprocessor kernels.  In most cases, this
is because a processor bit (tsc_invariant? constant_tsc?)
is passed through directly from the hardware via Xen and
tested by the hvm guest and the result implies that the
TSC is "stable".

I'd like to propose that, for a Xen hvm guest, TSC should
NEVER be considered stable.  For at least these reasons:

1) Often this test is done once at guest boot; if the guest
   migrates to another machine without the bit set, time
   will be erratic.
2) I *think* that in some cases even within the same system,
   TSC values will skew somewhat.  Since a uniprocessor guest
   will often be rescheduled to a different pcpu by Xen,
   the underlying tsc may appear erratic.

Comments (especially from Intel and AMD)?

If agreed, could Intel and AMD provide patches so that hvm
reads of the bits return "false"?  Or will this cause other
problems?

Another alternative would be to trap all rdtsc's and emulate
them but this probably will not be easy and may have
significant performance implications.  But perhaps it should
be an option?

Dan

===================================
Thanks... for the memory
I really could use more / My throughput's on the floor
The balloon is flat / My swap disk's fat / I've OOM's in store
Overcommitted so much
(with apologies to the late great Bob Hope)

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Guest TSC and Xen (Intel and AMD feedback please)

[Keir Fraser]

On 1/7/08 18:33, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> Another alternative would be to trap all rdtsc's and emulate
> them but this probably will not be easy and may have
> significant performance implications.  But perhaps it should
> be an option?

I think this would be useful, even if just for diagnosing guest time issues.

 -- Keir

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer
>Sent: 2008年7月2日 1:33
>
>Various versions of Linux under various circumstances select
>TSC as the primary clocksource for the kernel.  This is
>especially true for uniprocessor kernels, but also in some
>cases for multiprocessor kernels.  In most cases, this
>is because a processor bit (tsc_invariant? constant_tsc?)
>is passed through directly from the hardware via Xen and
>tested by the hvm guest and the result implies that the
>TSC is "stable".

constant_tsc is one of the factors affecting tsc stability, and
the keypoint is that one internal variable 'tsc_unstable' is 
initialized to zero. That means, kernel considers TSC as
stable by default, and you need trigger some factors upon 
which guest kernel'd like to mark tsc as unstable when 
checking them. However by far none of those factors seem 
appliable to all circumstances.

For example, even by hidding constant_tsc bit from cpuid,
UP guest will still consider tsc as stable, and so does 32bit
SMP guest running on Intel processors.

There's one proposal in KVM side, by exposing a faked C2
states into guest. ACPI processor idle path will mark TSC
as unstable when trying to enter C2. However that requires
guest to enable ACPI processor support, and need some
tricky logic to further mark C2 invalid to avoid performance
influence after purpose is achieved...

When saying this, I'm not opposing your proposal since it
does be a light option for some type of guests. My point is
just that no reliabe option to mark tsc as unstable, and
whether we have done all what can be done in Xen side 
to tackle this issue... Sorry that I didn't follow up your 
long discussion in this area before and thus may lose some 
context.

>
>I'd like to propose that, for a Xen hvm guest, TSC should
>NEVER be considered stable.  For at least these reasons:
>
>1) Often this test is done once at guest boot; if the guest
>   migrates to another machine without the bit set, time
>   will be erratic.

Since it's 'if', we should treat it as an option, instead of 'NEVER',
right? User may not have migration requirement in some cases,
and or the migratable backups are with same bits exactly... :-)

>2) I *think* that in some cases even within the same system,
>   TSC values will skew somewhat.  Since a uniprocessor guest
>   will often be rescheduled to a different pcpu by Xen,
>   the underlying tsc may appear erratic.

Can't be the tsc kept monotonic to guest, with some tweak on
tsc offset? Also Xen is always trying to sync tsc among cores.
As long as you don't run on a box with multiple bus crystals, 
or a box without constant tsc upon freq change, the tsc drift
among cores should be negligible considering the overhead of
migration. Then for cases out of 'as long as', we should mark
TSC as unstable, still as an option.

>
>Comments (especially from Intel and AMD)?
>
>If agreed, could Intel and AMD provide patches so that hvm
>reads of the bits return "false"?  Or will this cause other
>problems?

I vaguely recalled some posts about CPUID virtualization
framework. There's no need to a seperate category, and it
looks like this option can be aligned with that part?

Thanks,
Kevin

>
>Another alternative would be to trap all rdtsc's and emulate
>them but this probably will not be easy and may have
>significant performance implications.  But perhaps it should
>be an option?
>

Agree.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

Hi Kevin --

> constant_tsc is one of the factors affecting tsc stability, and
> the keypoint is that one internal variable 'tsc_unstable' is
> initialized to zero. That means, kernel considers TSC as
> stable by default, and you need trigger some factors upon
> which guest kernel'd like to mark tsc as unstable when
> checking them. However by far none of those factors seem
> appliable to all circumstances.
>
> For example, even by hidding constant_tsc bit from cpuid,
> UP guest will still consider tsc as stable, and so does 32bit
> SMP guest running on Intel processors.

Yes, looking over some Linux code, I see you are right.  In
one version of RH, if the processor vendor is Intel, it
always assumes tsc is stable!

> Since it's 'if', we should treat it as an option, instead of 'NEVER',
> right? User may not have migration requirement in some cases,
> and or the migratable backups are with same bits exactly... :-)

Yes, if it could be done, I agree an option would be better
than 'NEVER'.  But Keir is not too keen on more time-related
options.

> >2) I *think* that in some cases even within the same system,
> >   TSC values will skew somewhat.  Since a uniprocessor guest
> >   will often be rescheduled to a different pcpu by Xen,
> >   the underlying tsc may appear erratic.
>
> Can't be the tsc kept monotonic to guest, with some tweak on
> tsc offset? Also Xen is always trying to sync tsc among cores.
> As long as you don't run on a box with multiple bus crystals,
> or a box without constant tsc upon freq change, the tsc drift
> among cores should be negligible considering the overhead of
> migration. Then for cases out of 'as long as', we should mark
> TSC as unstable, still as an option.

No Xen doesn't actally sync the tsc's, just maintains its own
software offset variables.  I suppose it could periodically check
to make sure the tsc skew is within some reasonable value,
but after the guest boots, it is probably too late.

> I vaguely recalled some posts about CPUID virtualization
> framework. There's no need to a seperate category, and it
> looks like this option can be aligned with that part?

Yes, good point.  I wasn't following that closely either but
this could be covered by it.

> >Another alternative would be to trap all rdtsc's and emulate
> >them but this probably will not be easy and may have
> >significant performance implications.  But perhaps it should
> >be an option?
>
> Agree.

Is this something that you (or Intel in general) could look at?
I would be happy to participate but I don't think I understand
VT well enough.  Once the trap occurs, I suppose Xen system time
could be used as the virtual TSC, possibly scaled up.

Thanks,
Dan

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
>Sent: 2008年7月3日 9:21
>>
>> Can't be the tsc kept monotonic to guest, with some tweak on
>> tsc offset? Also Xen is always trying to sync tsc among cores.
>> As long as you don't run on a box with multiple bus crystals,
>> or a box without constant tsc upon freq change, the tsc drift
>> among cores should be negligible considering the overhead of
>> migration. Then for cases out of 'as long as', we should mark
>> TSC as unstable, still as an option.
>
>No Xen doesn't actally sync the tsc's, just maintains its own
>software offset variables.  I suppose it could periodically check
>to make sure the tsc skew is within some reasonable value,
>but after the guest boots, it is probably too late.

You are right. Xen only sync TSCs at boot time and then calibrate
TSC pace independently on each cpu. If some factors drag TSC
in the middle, then we may observe larger drift. But now I'm not
sure whether periodically sync among cpus is light enough, if taking
a similar convergence algothrim as boot time. But still, if only TSC
drift exists (caused by occasional events instead of a constantly
increasing drift from freq difference), this monotonic can be easily
exposed to the guest, like derived from xen system time to adjust
TSC offset at migration...

>
>> >Another alternative would be to trap all rdtsc's and emulate
>> >them but this probably will not be easy and may have
>> >significant performance implications.  But perhaps it should
>> >be an option?
>>
>> Agree.
>
>Is this something that you (or Intel in general) could look at?
>I would be happy to participate but I don't think I understand
>VT well enough.  Once the trap occurs, I suppose Xen system time
>could be used as the virtual TSC, possibly scaled up.
>

There should be tiny related to VT, as only turning on some bit to
allow RDTSC trapping and then the rest stuff should be common
how to handle it. We'll take a look, but can't commit the time due
to other scheduled bandwidth. But if you'd like to jump in early
we definitely can help with VT side.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

> >Is this something that you (or Intel in general) could look at?
> >I would be happy to participate but I don't think I understand
> >VT well enough.  Once the trap occurs, I suppose Xen system time
> >could be used as the virtual TSC, possibly scaled up.
> >
>
> There should be tiny related to VT, as only turning on some bit to
> allow RDTSC trapping and then the rest stuff should be common
> how to handle it. We'll take a look, but can't commit the time due
> to other scheduled bandwidth. But if you'd like to jump in early
> we definitely can help with VT side.

If you can post a patch with code that:

1) declares a boolean global variable: softtsc = 0
2) if the variable is set, a rdtsc instruction in any hvm
   domain causes a trap
3) the trap handler just does a physical rdtsc and returns

then I could probably take it from there.

Dan

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
>Sent: 2008年7月5日 1:32
>
>> >Is this something that you (or Intel in general) could look at?
>> >I would be happy to participate but I don't think I understand
>> >VT well enough.  Once the trap occurs, I suppose Xen system time
>> >could be used as the virtual TSC, possibly scaled up.
>> >
>>
>> There should be tiny related to VT, as only turning on some bit to
>> allow RDTSC trapping and then the rest stuff should be common
>> how to handle it. We'll take a look, but can't commit the time due
>> to other scheduled bandwidth. But if you'd like to jump in early
>> we definitely can help with VT side.
>
>If you can post a patch with code that:
>
>1) declares a boolean global variable: softtsc = 0
>2) if the variable is set, a rdtsc instruction in any hvm
>   domain causes a trap
>3) the trap handler just does a physical rdtsc and returns
>
>then I could probably take it from there.

OK, and Li in CC will take a look and bake a patch for you.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

Hi, Dan

I just add some code to produce vmexit. The function vmx_rdtsc_intercept() is not completed. It is needed to add some code to get the guest tsc.

Thanks 
--Li

>  -----Original Message-----
>  From: Tian, Kevin
>  Sent: Tuesday, July 08, 2008 9:39 AM
>  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  Cc: Zhang, Li
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >Sent: 2008年7月5日 1:32
>  >
>  >> >Is this something that you (or Intel in general) could look at?
>  >> >I would be happy to participate but I don't think I understand
>  >> >VT well enough.  Once the trap occurs, I suppose Xen system time
>  >> >could be used as the virtual TSC, possibly scaled up.
>  >> >
>  >>
>  >> There should be tiny related to VT, as only turning on some bit to
>  >> allow RDTSC trapping and then the rest stuff should be common
>  >> how to handle it. We'll take a look, but can't commit the time due
>  >> to other scheduled bandwidth. But if you'd like to jump in early
>  >> we definitely can help with VT side.
>  >
>  >If you can post a patch with code that:
>  >
>  >1) declares a boolean global variable: softtsc = 0
>  >2) if the variable is set, a rdtsc instruction in any hvm
>  >   domain causes a trap
>  >3) the trap handler just does a physical rdtsc and returns
>  >
>  >then I could probably take it from there.
>  
>  OK, and Li in CC will take a look and bake a patch for you.
>  
>  Thanks,
>  Kevin

[-- Attachment #2: tsc.patch --]
[-- Type: application/octet-stream, Size: 1265 bytes --]

diff -r 5eaec8296a22 xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Sun Jul 06 20:16:24 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 18:08:39 2008 +0800
@@ -89,6 +89,7 @@
 
     min = (CPU_BASED_HLT_EXITING |
            CPU_BASED_INVLPG_EXITING |
+           CPU_BASED_RDTSC_EXITING |
            CPU_BASED_CR3_LOAD_EXITING |
            CPU_BASED_CR3_STORE_EXITING |
            CPU_BASED_MONITOR_EXITING |
diff -r 5eaec8296a22 xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Sun Jul 06 20:16:24 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 18:08:39 2008 +0800
@@ -1374,6 +1374,13 @@
     HVMTRACE_LONG_2D(INVLPG, curr, /*invlpga=*/ 0, TRC_PAR_LONG(vaddr));
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
+}
+
+static void vmx_rdtsc_intercept(struct vcpu *vcpu)
+{
+    uint64_t tsc; 
+    rdtscll(tsc);
+    return;    
 }
 
 #define CASE_SET_REG(REG, reg)      \
@@ -2193,6 +2200,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(v);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

Hi, Dan
This seems not to be enough. I will cook another patch. 

>  -----Original Message-----
>  From: Zhang, Li
>  Sent: Tuesday, July 08, 2008 10:28 AM
>  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  Hi, Dan
>  
>  I just add some code to produce vmexit. The function vmx_rdtsc_intercept()
>  is not completed. It is needed to add some code to get the guest tsc.
>  
>  Thanks
>  --Li
>  
>  >  -----Original Message-----
>  >  From: Tian, Kevin
>  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  >  Cc: Zhang, Li
>  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  >
>  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  >Sent: 2008年7月5日 1:32
>  >  >
>  >  >> >Is this something that you (or Intel in general) could look at?
>  >  >> >I would be happy to participate but I don't think I understand
>  >  >> >VT well enough.  Once the trap occurs, I suppose Xen system time
>  >  >> >could be used as the virtual TSC, possibly scaled up.
>  >  >> >
>  >  >>
>  >  >> There should be tiny related to VT, as only turning on some bit to
>  >  >> allow RDTSC trapping and then the rest stuff should be common
>  >  >> how to handle it. We'll take a look, but can't commit the time due
>  >  >> to other scheduled bandwidth. But if you'd like to jump in early
>  >  >> we definitely can help with VT side.
>  >  >
>  >  >If you can post a patch with code that:
>  >  >
>  >  >1) declares a boolean global variable: softtsc = 0
>  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  >  >   domain causes a trap
>  >  >3) the trap handler just does a physical rdtsc and returns
>  >  >
>  >  >then I could probably take it from there.
>  >
>  >  OK, and Li in CC will take a look and bake a patch for you.
>  >
>  >  Thanks,
>  >  Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 3250 bytes --]

Thanks Kevin and Li --

A couple of questions:

1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work:

static int opt_softtsc = 0;
boolean_param("softtsc", opt_softtsc);
  :
min = /* original code */
if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done?

3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap?

4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code somewhere to emulate WRMSR of tsc?

Sorry I am not very familar with the details of the ia32 instruction set.

Thanks,
Dan

> -----Original Message-----
> From: Zhang, Li [mailto:li.zhang@intel.com]
> Sent: Monday, July 07, 2008 9:00 PM
> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Cc: Tian, Kevin
> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> 
> 
> Hi, Dan
> This seems not to be enough. I will cook another patch.
> 
> >  -----Original Message-----
> >  From: Zhang, Li
> >  Sent: Tuesday, July 08, 2008 10:28 AM
> >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
> >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> >
> >  Hi, Dan
> >
> >  I just add some code to produce vmexit. The function 
> vmx_rdtsc_intercept()
> >  is not completed. It is needed to add some code to get the 
> guest tsc.
> >
> >  Thanks
> >  --Li
> >
> >  >  -----Original Message-----
> >  >  From: Tian, Kevin
> >  >  Sent: Tuesday, July 08, 2008 9:39 AM
> >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
> >  >  Cc: Zhang, Li
> >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and 
> AMD feedback please)
> >  >
> >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >  >  >Sent: 2008年7月5日 1:32
> >  >  >
> >  >  >> >Is this something that you (or Intel in general) 
> could look at?
> >  >  >> >I would be happy to participate but I don't think I 
> understand
> >  >  >> >VT well enough.  Once the trap occurs, I suppose 
> Xen system time
> >  >  >> >could be used as the virtual TSC, possibly scaled up.
> >  >  >> >
> >  >  >>
> >  >  >> There should be tiny related to VT, as only turning 
> on some bit to
> >  >  >> allow RDTSC trapping and then the rest stuff should be common
> >  >  >> how to handle it. We'll take a look, but can't 
> commit the time due
> >  >  >> to other scheduled bandwidth. But if you'd like to 
> jump in early
> >  >  >> we definitely can help with VT side.
> >  >  >
> >  >  >If you can post a patch with code that:
> >  >  >
> >  >  >1) declares a boolean global variable: softtsc = 0
> >  >  >2) if the variable is set, a rdtsc instruction in any hvm
> >  >  >   domain causes a trap
> >  >  >3) the trap handler just does a physical rdtsc and returns
> >  >  >
> >  >  >then I could probably take it from there.
> >  >
> >  >  OK, and Li in CC will take a look and bake a patch for you.
> >  >
> >  >  Thanks,
> >  >  Kevin
> 

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
>Sent: 2008年7月8日 11:34
>
>1) If the EXITING flag to be set in vmcs.c is to be controlled 
>by a xen boot option, would this work:
>
>static int opt_softtsc = 0;
>boolean_param("softtsc", opt_softtsc);
>  :
>min = /* original code */
>if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;

yes.

>
>2) In vmx_rdtsc_intercept(/* need regs as param*/) if we 
>change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and 
>then set regs->edx amd regs->eax from tmptsc, are we almost done?

Yes, and similar example is under case EXIT_REASON_CPUID.

>
>3) Des Linux or Windows use the CR4.TSD flag and, if so, what 
>code gets called to force the trap?

My feeling is that both Linux and Windows allows user directly read to
TSC, and thus CR4.TSD is unlikely to be set. But it needs confirmation.
If you really want to trap such setting, hvm_set_cr4 is the point and
you may further reference vmx_update_guest_cr if it's vmx specific (but
I guess it should be common).

But I doubt the point to support such model, since two levels trapping
(Xen trap, inject fault, guest kernel read, xen trap, ...) which adds much
delay on Xen maintained guest tsc.

>
>4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there 
>already code somewhere to emulate WRMSR of tsc?

I'm not sure about this one, and others may answer.

Thanks,
Kevin

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

[-- Attachment #1: Type: text/plain, Size: 3932 bytes --]

Hi, Dan

This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet. 

Thanks
--Li

>  -----Original Message-----
>  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  Sent: Tuesday, July 08, 2008 11:34 AM
>  To: Zhang, Li; Xen-Devel (E-mail)
>  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  Thanks Kevin and Li --
>  
>  A couple of questions:
>  
>  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot
>  option, would this work:
>  
>  static int opt_softtsc = 0;
>  boolean_param("softtsc", opt_softtsc);
>    :
>  min = /* original code */
>  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  
>  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd
>  regs->eax from tmptsc, are we almost done?
>  
>  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called
>  to force the trap?
>  
>  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>  somewhere to emulate WRMSR of tsc?
>  
>  Sorry I am not very familar with the details of the ia32 instruction set.
>  
>  Thanks,
>  Dan
>  
>  > -----Original Message-----
>  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  > Sent: Monday, July 07, 2008 9:00 PM
>  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  > Cc: Tian, Kevin
>  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > feedback please)
>  >
>  >
>  > Hi, Dan
>  > This seems not to be enough. I will cook another patch.
>  >
>  > >  -----Original Message-----
>  > >  From: Zhang, Li
>  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
>  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  > feedback please)
>  > >
>  > >  Hi, Dan
>  > >
>  > >  I just add some code to produce vmexit. The function
>  > vmx_rdtsc_intercept()
>  > >  is not completed. It is needed to add some code to get the
>  > guest tsc.
>  > >
>  > >  Thanks
>  > >  --Li
>  > >
>  > >  >  -----Original Message-----
>  > >  >  From: Tian, Kevin
>  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  > >  >  Cc: Zhang, Li
>  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  > AMD feedback please)
>  > >  >
>  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  > >  >  >Sent: 2008年7月5日 1:32
>  > >  >  >
>  > >  >  >> >Is this something that you (or Intel in general)
>  > could look at?
>  > >  >  >> >I would be happy to participate but I don't think I
>  > understand
>  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>  > Xen system time
>  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>  > >  >  >> >
>  > >  >  >>
>  > >  >  >> There should be tiny related to VT, as only turning
>  > on some bit to
>  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>  > >  >  >> how to handle it. We'll take a look, but can't
>  > commit the time due
>  > >  >  >> to other scheduled bandwidth. But if you'd like to
>  > jump in early
>  > >  >  >> we definitely can help with VT side.
>  > >  >  >
>  > >  >  >If you can post a patch with code that:
>  > >  >  >
>  > >  >  >1) declares a boolean global variable: softtsc = 0
>  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  > >  >  >   domain causes a trap
>  > >  >  >3) the trap handler just does a physical rdtsc and returns
>  > >  >  >
>  > >  >  >then I could probably take it from there.
>  > >  >
>  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>  > >  >
>  > >  >  Thanks,
>  > >  >  Kevin
>  >

[-- Attachment #2: tsc.patch --]
[-- Type: application/octet-stream, Size: 1673 bytes --]

diff -r be52424a543e xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 22:26:57 2008 +0800
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+int softtsc = 1;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 22:26:57 2008 +0800
@@ -1375,6 +1375,17 @@
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
 }
+
+static void vmx_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = tsc & 0xffff;
+    regs->edx = tsc >> 32 & 0xffff;
+}
+
 
 #define CASE_SET_REG(REG, reg)      \
     case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

In fact, the issue is from guest. In the guest's terminal, it shows that "your time source seems to be instable..."

>  -----Original Message-----
>  From: xen-devel-bounces@lists.xensource.com
>  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>  Sent: Tuesday, July 08, 2008 2:49 PM
>  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  Cc: Tian, Kevin
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  Hi, Dan
>  
>  This is the patch which I think it has been completed VT. But there is an
>  instable issue. The reason may be from that this interception is not bound
>  to timer mode yet.
>  
>  Thanks
>  --Li
>  
>  >  -----Original Message-----
>  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  Sent: Tuesday, July 08, 2008 11:34 AM
>  >  To: Zhang, Li; Xen-Devel (E-mail)
>  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  >
>  >  Thanks Kevin and Li --
>  >
>  >  A couple of questions:
>  >
>  >  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>  boot
>  >  option, would this work:
>  >
>  >  static int opt_softtsc = 0;
>  >  boolean_param("softtsc", opt_softtsc);
>  >    :
>  >  min = /* original code */
>  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  >
>  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd
>  >  regs->eax from tmptsc, are we almost done?
>  >
>  >  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>  called
>  >  to force the trap?
>  >
>  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>  >  somewhere to emulate WRMSR of tsc?
>  >
>  >  Sorry I am not very familar with the details of the ia32 instruction set.
>  >
>  >  Thanks,
>  >  Dan
>  >
>  >  > -----Original Message-----
>  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  >  > Sent: Monday, July 07, 2008 9:00 PM
>  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  >  > Cc: Tian, Kevin
>  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  > feedback please)
>  >  >
>  >  >
>  >  > Hi, Dan
>  >  > This seems not to be enough. I will cook another patch.
>  >  >
>  >  > >  -----Original Message-----
>  >  > >  From: Zhang, Li
>  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)'
>  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  > feedback please)
>  >  > >
>  >  > >  Hi, Dan
>  >  > >
>  >  > >  I just add some code to produce vmexit. The function
>  >  > vmx_rdtsc_intercept()
>  >  > >  is not completed. It is needed to add some code to get the
>  >  > guest tsc.
>  >  > >
>  >  > >  Thanks
>  >  > >  --Li
>  >  > >
>  >  > >  >  -----Original Message-----
>  >  > >  >  From: Tian, Kevin
>  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  >  > >  >  Cc: Zhang, Li
>  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  >  > AMD feedback please)
>  >  > >  >
>  >  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  > >  >  >Sent: 2008年7月5日 1:32
>  >  > >  >  >
>  >  > >  >  >> >Is this something that you (or Intel in general)
>  >  > could look at?
>  >  > >  >  >> >I would be happy to participate but I don't think I
>  >  > understand
>  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>  >  > Xen system time
>  >  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>  >  > >  >  >> >
>  >  > >  >  >>
>  >  > >  >  >> There should be tiny related to VT, as only turning
>  >  > on some bit to
>  >  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>  >  > >  >  >> how to handle it. We'll take a look, but can't
>  >  > commit the time due
>  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
>  >  > jump in early
>  >  > >  >  >> we definitely can help with VT side.
>  >  > >  >  >
>  >  > >  >  >If you can post a patch with code that:
>  >  > >  >  >
>  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>  >  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  >  > >  >  >   domain causes a trap
>  >  > >  >  >3) the trap handler just does a physical rdtsc and returns
>  >  > >  >  >
>  >  > >  >  >then I could probably take it from there.
>  >  > >  >
>  >  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>  >  > >  >
>  >  > >  >  Thanks,
>  >  > >  >  Kevin
>  >  >

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Zhang, Li]

[-- Attachment #1: Type: text/plain, Size: 5706 bytes --]

Hi, Dan
Please ignore previous comments. The above attachment is OK. Sorry for that. :(

>  -----Original Message-----
>  From: xen-devel-bounces@lists.xensource.com
>  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>  Sent: Tuesday, July 08, 2008 2:58 PM
>  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  Cc: Tian, Kevin
>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  
>  In fact, the issue is from guest. In the guest's terminal, it shows that "your
>  time source seems to be instable..."
>  
>  >  -----Original Message-----
>  >  From: xen-devel-bounces@lists.xensource.com
>  >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>  >  Sent: Tuesday, July 08, 2008 2:49 PM
>  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  >  Cc: Tian, Kevin
>  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>  >
>  >  Hi, Dan
>  >
>  >  This is the patch which I think it has been completed VT. But there is an
>  >  instable issue. The reason may be from that this interception is not bound
>  >  to timer mode yet.
>  >
>  >  Thanks
>  >  --Li
>  >
>  >  >  -----Original Message-----
>  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
>  >  >  To: Zhang, Li; Xen-Devel (E-mail)
>  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback
>  please)
>  >  >
>  >  >  Thanks Kevin and Li --
>  >  >
>  >  >  A couple of questions:
>  >  >
>  >  >  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>  >  boot
>  >  >  option, would this work:
>  >  >
>  >  >  static int opt_softtsc = 0;
>  >  >  boolean_param("softtsc", opt_softtsc);
>  >  >    :
>  >  >  min = /* original code */
>  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>  >  >
>  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx
>  amd
>  >  >  regs->eax from tmptsc, are we almost done?
>  >  >
>  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>  >  called
>  >  >  to force the trap?
>  >  >
>  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>  >  >  somewhere to emulate WRMSR of tsc?
>  >  >
>  >  >  Sorry I am not very familar with the details of the ia32 instruction
>  set.
>  >  >
>  >  >  Thanks,
>  >  >  Dan
>  >  >
>  >  >  > -----Original Message-----
>  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>  >  >  > Sent: Monday, July 07, 2008 9:00 PM
>  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>  >  >  > Cc: Tian, Kevin
>  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  >  > feedback please)
>  >  >  >
>  >  >  >
>  >  >  > Hi, Dan
>  >  >  > This seems not to be enough. I will cook another patch.
>  >  >  >
>  >  >  > >  -----Original Message-----
>  >  >  > >  From: Zhang, Li
>  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel
>  (E-mail)'
>  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>  >  >  > feedback please)
>  >  >  > >
>  >  >  > >  Hi, Dan
>  >  >  > >
>  >  >  > >  I just add some code to produce vmexit. The function
>  >  >  > vmx_rdtsc_intercept()
>  >  >  > >  is not completed. It is needed to add some code to get the
>  >  >  > guest tsc.
>  >  >  > >
>  >  >  > >  Thanks
>  >  >  > >  --Li
>  >  >  > >
>  >  >  > >  >  -----Original Message-----
>  >  >  > >  >  From: Tian, Kevin
>  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>  >  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>  >  >  > >  >  Cc: Zhang, Li
>  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>  >  >  > AMD feedback please)
>  >  >  > >  >
>  >  >  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>  >  >  > >  >  >Sent: 2008年7月5日 1:32
>  >  >  > >  >  >
>  >  >  > >  >  >> >Is this something that you (or Intel in general)
>  >  >  > could look at?
>  >  >  > >  >  >> >I would be happy to participate but I don't think I
>  >  >  > understand
>  >  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>  >  >  > Xen system time
>  >  >  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>  >  >  > >  >  >> >
>  >  >  > >  >  >>
>  >  >  > >  >  >> There should be tiny related to VT, as only turning
>  >  >  > on some bit to
>  >  >  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>  >  >  > >  >  >> how to handle it. We'll take a look, but can't
>  >  >  > commit the time due
>  >  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
>  >  >  > jump in early
>  >  >  > >  >  >> we definitely can help with VT side.
>  >  >  > >  >  >
>  >  >  > >  >  >If you can post a patch with code that:
>  >  >  > >  >  >
>  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>  >  >  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>  >  >  > >  >  >   domain causes a trap
>  >  >  > >  >  >3) the trap handler just does a physical rdtsc and returns
>  >  >  > >  >  >
>  >  >  > >  >  >then I could probably take it from there.
>  >  >  > >  >
>  >  >  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>  >  >  > >  >
>  >  >  > >  >  Thanks,
>  >  >  > >  >  Kevin
>  >  >  >
>  
>  _______________________________________________
>  Xen-devel mailing list
>  Xen-devel@lists.xensource.com
>  http://lists.xensource.com/xen-devel

[-- Attachment #2: tsc.patch --]
[-- Type: application/octet-stream, Size: 1681 bytes --]

diff -r be52424a543e xen/arch/x86/hvm/vmx/vmcs.c
--- a/xen/arch/x86/hvm/vmx/vmcs.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmcs.c	Tue Jul 08 22:26:57 2008 +0800
@@ -40,6 +40,7 @@
 
 static int opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
+int softtsc = 1;
 
 /* Dynamic (run-time adjusted) execution control flags. */
 u32 vmx_pin_based_exec_control __read_mostly;
@@ -96,6 +97,8 @@
            CPU_BASED_MOV_DR_EXITING |
            CPU_BASED_ACTIVATE_IO_BITMAP |
            CPU_BASED_USE_TSC_OFFSETING);
+    if (softtsc)
+        min |= CPU_BASED_RDTSC_EXITING;
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
diff -r be52424a543e xen/arch/x86/hvm/vmx/vmx.c
--- a/xen/arch/x86/hvm/vmx/vmx.c	Fri Jun 27 14:15:11 2008 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c	Tue Jul 08 22:26:57 2008 +0800
@@ -1375,6 +1375,17 @@
     if ( paging_invlpg(curr, vaddr) )
         vpid_sync_vcpu_gva(curr, vaddr);
 }
+
+static void vmx_rdtsc_intercept(struct cpu_user_regs *regs)
+{
+    uint64_t tsc;
+    struct vcpu *v = current; 
+
+    tsc = hvm_get_guest_tsc(v);
+    regs->eax = tsc & 0xffffffff;
+    regs->edx = tsc >> 32 & 0xffffffff;
+}
+
 
 #define CASE_SET_REG(REG, reg)      \
     case VMX_CONTROL_REG_ACCESS_GPR_ ## REG: regs->reg = value; break
@@ -2192,6 +2203,11 @@
         vmx_invlpg_intercept(exit_qualification);
         break;
     }
+    case EXIT_REASON_RDTSC:
+        inst_len = __get_instruction_length();
+        __update_guest_eip(inst_len);
+        vmx_rdtsc_intercept(regs);
+        break;
     case EXIT_REASON_VMCALL:
     {
         int rc;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Guest TSC and Xen (Intel and AMD feedback please)

[Dave Winchell]

Hi Li,

Shouldn't this

+ regs->eax = tsc & 0xffff;
+ regs->edx = tsc >> 32 & 0xffff;

be

+ regs->eax = tsc & 0xffffffff;
+ regs->edx = tsc >> 32 & 0xffffffff;

Thanks,
Dave


Zhang, Li wrote:

>Hi, Dan
>Please ignore previous comments. The above attachment is OK. Sorry for that. :(
>
>  
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>> Sent: Tuesday, July 08, 2008 2:58 PM
>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> Cc: Tian, Kevin
>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>> 
>> In fact, the issue is from guest. In the guest's terminal, it shows that "your
>> time source seems to be instable..."
>> 
>> >  -----Original Message-----
>> >  From: xen-devel-bounces@lists.xensource.com
>> >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>> >  Sent: Tuesday, July 08, 2008 2:49 PM
>> >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> >  Cc: Tian, Kevin
>> >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>> >
>> >  Hi, Dan
>> >
>> >  This is the patch which I think it has been completed VT. But there is an
>> >  instable issue. The reason may be from that this interception is not bound
>> >  to timer mode yet.
>> >
>> >  Thanks
>> >  --Li
>> >
>> >  >  -----Original Message-----
>> >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>> >  >  Sent: Tuesday, July 08, 2008 11:34 AM
>> >  >  To: Zhang, Li; Xen-Devel (E-mail)
>> >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>> >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback
>> please)
>> >  >
>> >  >  Thanks Kevin and Li --
>> >  >
>> >  >  A couple of questions:
>> >  >
>> >  >  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>> >  boot
>> >  >  option, would this work:
>> >  >
>> >  >  static int opt_softtsc = 0;
>> >  >  boolean_param("softtsc", opt_softtsc);
>> >  >    :
>> >  >  min = /* original code */
>> >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>> >  >
>> >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>> >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx
>> amd
>> >  >  regs->eax from tmptsc, are we almost done?
>> >  >
>> >  >  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>> >  called
>> >  >  to force the trap?
>> >  >
>> >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>> >  >  somewhere to emulate WRMSR of tsc?
>> >  >
>> >  >  Sorry I am not very familar with the details of the ia32 instruction
>> set.
>> >  >
>> >  >  Thanks,
>> >  >  Dan
>> >  >
>> >  >  > -----Original Message-----
>> >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
>> >  >  > Sent: Monday, July 07, 2008 9:00 PM
>> >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>> >  >  > Cc: Tian, Kevin
>> >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>> >  >  > feedback please)
>> >  >  >
>> >  >  >
>> >  >  > Hi, Dan
>> >  >  > This seems not to be enough. I will cook another patch.
>> >  >  >
>> >  >  > >  -----Original Message-----
>> >  >  > >  From: Zhang, Li
>> >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
>> >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel
>> (E-mail)'
>> >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>> >  >  > feedback please)
>> >  >  > >
>> >  >  > >  Hi, Dan
>> >  >  > >
>> >  >  > >  I just add some code to produce vmexit. The function
>> >  >  > vmx_rdtsc_intercept()
>> >  >  > >  is not completed. It is needed to add some code to get the
>> >  >  > guest tsc.
>> >  >  > >
>> >  >  > >  Thanks
>> >  >  > >  --Li
>> >  >  > >
>> >  >  > >  >  -----Original Message-----
>> >  >  > >  >  From: Tian, Kevin
>> >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
>> >  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>> >  >  > >  >  Cc: Zhang, Li
>> >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>> >  >  > AMD feedback please)
>> >  >  > >  >
>> >  >  > >  >  >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>> >  >  > >  >  >Sent: 2008年7月5日 1:32
>> >  >  > >  >  >
>> >  >  > >  >  >> >Is this something that you (or Intel in general)
>> >  >  > could look at?
>> >  >  > >  >  >> >I would be happy to participate but I don't think I
>> >  >  > understand
>> >  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
>> >  >  > Xen system time
>> >  >  > >  >  >> >could be used as the virtual TSC, possibly scaled up.
>> >  >  > >  >  >> >
>> >  >  > >  >  >>
>> >  >  > >  >  >> There should be tiny related to VT, as only turning
>> >  >  > on some bit to
>> >  >  > >  >  >> allow RDTSC trapping and then the rest stuff should be common
>> >  >  > >  >  >> how to handle it. We'll take a look, but can't
>> >  >  > commit the time due
>> >  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
>> >  >  > jump in early
>> >  >  > >  >  >> we definitely can help with VT side.
>> >  >  > >  >  >
>> >  >  > >  >  >If you can post a patch with code that:
>> >  >  > >  >  >
>> >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
>> >  >  > >  >  >2) if the variable is set, a rdtsc instruction in any hvm
>> >  >  > >  >  >   domain causes a trap
>> >  >  > >  >  >3) the trap handler just does a physical rdtsc and returns
>> >  >  > >  >  >
>> >  >  > >  >  >then I could probably take it from there.
>> >  >  > >  >
>> >  >  > >  >  OK, and Li in CC will take a look and bake a patch for you.
>> >  >  > >  >
>> >  >  > >  >  Thanks,
>> >  >  > >  >  Kevin
>> >  >  >
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>    
>>
>>------------------------------------------------------------------------
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>    
>>

Re: Guest TSC and Xen (Intel and AMD feedback please)

[Keir Fraser]

I would go for (uint32_t)tsc and (uint32_t)(tsc >> 32) as being even
clearer.

 -- Keir

On 8/7/08 15:48, "Dave Winchell" <dwinchell@virtualiron.com> wrote:

> Hi Li,
> 
> Shouldn't this
> 
> + regs->eax = tsc & 0xffff;
> + regs->edx = tsc >> 32 & 0xffff;
> 
> be
> 
> + regs->eax = tsc & 0xffffffff;
> + regs->edx = tsc >> 32 & 0xffffffff;
> 
> Thanks,
> Dave
> 
> 
> Zhang, Li wrote:
> 
>> Hi, Dan
>> Please ignore previous comments. The above attachment is OK. Sorry for that.
>> :(
>> 
>>  
>> 
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>>> Sent: Tuesday, July 08, 2008 2:58 PM
>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>> Cc: Tian, Kevin
>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>>> 
>>> In fact, the issue is from guest. In the guest's terminal, it shows that
>>> "your
>>> time source seems to be instable..."
>>> 
>>>>  -----Original Message-----
>>>>  From: xen-devel-bounces@lists.xensource.com
>>>>  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li
>>>>  Sent: Tuesday, July 08, 2008 2:49 PM
>>>>  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>>>  Cc: Tian, Kevin
>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>>>> 
>>>>  Hi, Dan
>>>> 
>>>>  This is the patch which I think it has been completed VT. But there is an
>>>>  instable issue. The reason may be from that this interception is not bound
>>>>  to timer mode yet.
>>>> 
>>>>  Thanks
>>>>  --Li
>>>> 
>>>>>  -----Original Message-----
>>>>>  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>  Sent: Tuesday, July 08, 2008 11:34 AM
>>>>>  To: Zhang, Li; Xen-Devel (E-mail)
>>>>>  Cc: Tian, Kevin; dan.magenheimer@oracle.com
>>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback
>>> please)
>>>>> 
>>>>>  Thanks Kevin and Li --
>>>>> 
>>>>>  A couple of questions:
>>>>> 
>>>>>  1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen
>>>>  boot
>>>>>  option, would this work:
>>>>> 
>>>>>  static int opt_softtsc = 0;
>>>>>  boolean_param("softtsc", opt_softtsc);
>>>>>    :
>>>>>  min = /* original code */
>>>>>  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
>>>>> 
>>>>>  2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the
>>>>>  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx
>>> amd
>>>>>  regs->eax from tmptsc, are we almost done?
>>>>> 
>>>>>  3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets
>>>>  called
>>>>>  to force the trap?
>>>>> 
>>>>>  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is there already code
>>>>>  somewhere to emulate WRMSR of tsc?
>>>>> 
>>>>>  Sorry I am not very familar with the details of the ia32 instruction
>>> set.
>>>>> 
>>>>>  Thanks,
>>>>>  Dan
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Zhang, Li [mailto:li.zhang@intel.com]
>>>>>> Sent: Monday, July 07, 2008 9:00 PM
>>>>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
>>>>>> Cc: Tian, Kevin
>>>>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>>>>>> feedback please)
>>>>>> 
>>>>>> 
>>>>>> Hi, Dan
>>>>>> This seems not to be enough. I will cook another patch.
>>>>>> 
>>>>>>>  -----Original Message-----
>>>>>>>  From: Zhang, Li
>>>>>>>  Sent: Tuesday, July 08, 2008 10:28 AM
>>>>>>>  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel
>>> (E-mail)'
>>>>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
>>>>>> feedback please)
>>>>>>> 
>>>>>>>  Hi, Dan
>>>>>>> 
>>>>>>>  I just add some code to produce vmexit. The function
>>>>>> vmx_rdtsc_intercept()
>>>>>>>  is not completed. It is needed to add some code to get the
>>>>>> guest tsc.
>>>>>>> 
>>>>>>>  Thanks
>>>>>>>  --Li
>>>>>>> 
>>>>>>>>  -----Original Message-----
>>>>>>>>  From: Tian, Kevin
>>>>>>>>  Sent: Tuesday, July 08, 2008 9:39 AM
>>>>>>>>  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
>>>>>>>>  Cc: Zhang, Li
>>>>>>>>  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
>>>>>> AMD feedback please)
>>>>>>>> 
>>>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>>>>>> Sent: 2008年7月5日 1:32
>>>>>>>>> 
>>>>>>>>>>> Is this something that you (or Intel in general)
>>>>>> could look at?
>>>>>>>>>>> I would be happy to participate but I don't think I
>>>>>> understand
>>>>>>>>>>> VT well enough.  Once the trap occurs, I suppose
>>>>>> Xen system time
>>>>>>>>>>> could be used as the virtual TSC, possibly scaled up.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> There should be tiny related to VT, as only turning
>>>>>> on some bit to
>>>>>>>>>> allow RDTSC trapping and then the rest stuff should be common
>>>>>>>>>> how to handle it. We'll take a look, but can't
>>>>>> commit the time due
>>>>>>>>>> to other scheduled bandwidth. But if you'd like to
>>>>>> jump in early
>>>>>>>>>> we definitely can help with VT side.
>>>>>>>>> 
>>>>>>>>> If you can post a patch with code that:
>>>>>>>>> 
>>>>>>>>> 1) declares a boolean global variable: softtsc = 0
>>>>>>>>> 2) if the variable is set, a rdtsc instruction in any hvm
>>>>>>>>>   domain causes a trap
>>>>>>>>> 3) the trap handler just does a physical rdtsc and returns
>>>>>>>>> 
>>>>>>>>> then I could probably take it from there.
>>>>>>>> 
>>>>>>>>  OK, and Li in CC will take a look and bake a patch for you.
>>>>>>>> 
>>>>>>>>  Thanks,
>>>>>>>>  Kevin
>>>>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>    
>>> 
>>> ------------------------------------------------------------------------
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>    
>>> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 7042 bytes --]

Looks great!  I am continuing to code and test.  Some preliminary results:

1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work!  The hvm
    guest believes it is running on a 1000.185 MHz clock.

2) On my test box, a guest rdtsc without softtsc takes about 30ns.  With softtsc,
    rdtsc takes about 1us.

3) On my user-land test program, I am seeing rare "Time goes backwards", even with
   hvm_get_guest_time().  I'm trying to track this down.

Dan

> -----Original Message-----
> From: Zhang, Li [mailto:li.zhang@intel.com]
> Sent: Tuesday, July 08, 2008 3:47 AM
> To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Cc: Tian, Kevin
> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> 
> 
> Hi, Dan
> Please ignore previous comments. The above attachment is OK. 
> Sorry for that. :(
> 
> >  -----Original Message-----
> >  From: xen-devel-bounces@lists.xensource.com
> >  [mailto:xen-devel-bounces@lists.xensource.com] On Behalf 
> Of Zhang, Li
> >  Sent: Tuesday, July 08, 2008 2:58 PM
> >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >  Cc: Tian, Kevin
> >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD 
> feedback please)
> >
> >  In fact, the issue is from guest. In the guest's terminal, 
> it shows that "your
> >  time source seems to be instable..."
> >
> >  >  -----Original Message-----
> >  >  From: xen-devel-bounces@lists.xensource.com
> >  >  [mailto:xen-devel-bounces@lists.xensource.com] On 
> Behalf Of Zhang, Li
> >  >  Sent: Tuesday, July 08, 2008 2:49 PM
> >  >  To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >  >  Cc: Tian, Kevin
> >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and 
> AMD feedback please)
> >  >
> >  >  Hi, Dan
> >  >
> >  >  This is the patch which I think it has been completed 
> VT. But there is an
> >  >  instable issue. The reason may be from that this 
> interception is not bound
> >  >  to timer mode yet.
> >  >
> >  >  Thanks
> >  >  --Li
> >  >
> >  >  >  -----Original Message-----
> >  >  >  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >  >  >  Sent: Tuesday, July 08, 2008 11:34 AM
> >  >  >  To: Zhang, Li; Xen-Devel (E-mail)
> >  >  >  Cc: Tian, Kevin; dan.magenheimer@oracle.com
> >  >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel 
> and AMD feedback
> >  please)
> >  >  >
> >  >  >  Thanks Kevin and Li --
> >  >  >
> >  >  >  A couple of questions:
> >  >  >
> >  >  >  1) If the EXITING flag to be set in vmcs.c is to be 
> controlled by a xen
> >  >  boot
> >  >  >  option, would this work:
> >  >  >
> >  >  >  static int opt_softtsc = 0;
> >  >  >  boolean_param("softtsc", opt_softtsc);
> >  >  >    :
> >  >  >  min = /* original code */
> >  >  >  if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;
> >  >  >
> >  >  >  2) In vmx_rdtsc_intercept(/* need regs as param*/) 
> if we change the
> >  >  >  rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and 
> then set regs->edx
> >  amd
> >  >  >  regs->eax from tmptsc, are we almost done?
> >  >  >
> >  >  >  3) Des Linux or Windows use the CR4.TSD flag and, if 
> so, what code gets
> >  >  called
> >  >  >  to force the trap?
> >  >  >
> >  >  >  4) Does Linux or Windows use RDMSR/WRMSR of tsc?  Is 
> there already code
> >  >  >  somewhere to emulate WRMSR of tsc?
> >  >  >
> >  >  >  Sorry I am not very familar with the details of the 
> ia32 instruction
> >  set.
> >  >  >
> >  >  >  Thanks,
> >  >  >  Dan
> >  >  >
> >  >  >  > -----Original Message-----
> >  >  >  > From: Zhang, Li [mailto:li.zhang@intel.com]
> >  >  >  > Sent: Monday, July 07, 2008 9:00 PM
> >  >  >  > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> >  >  >  > Cc: Tian, Kevin
> >  >  >  > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD
> >  >  >  > feedback please)
> >  >  >  >
> >  >  >  >
> >  >  >  > Hi, Dan
> >  >  >  > This seems not to be enough. I will cook another patch.
> >  >  >  >
> >  >  >  > >  -----Original Message-----
> >  >  >  > >  From: Zhang, Li
> >  >  >  > >  Sent: Tuesday, July 08, 2008 10:28 AM
> >  >  >  > >  To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 
> 'Xen-Devel
> >  (E-mail)'
> >  >  >  > >  Subject: RE: [Xen-devel] Guest TSC and Xen 
> (Intel and AMD
> >  >  >  > feedback please)
> >  >  >  > >
> >  >  >  > >  Hi, Dan
> >  >  >  > >
> >  >  >  > >  I just add some code to produce vmexit. The function
> >  >  >  > vmx_rdtsc_intercept()
> >  >  >  > >  is not completed. It is needed to add some code 
> to get the
> >  >  >  > guest tsc.
> >  >  >  > >
> >  >  >  > >  Thanks
> >  >  >  > >  --Li
> >  >  >  > >
> >  >  >  > >  >  -----Original Message-----
> >  >  >  > >  >  From: Tian, Kevin
> >  >  >  > >  >  Sent: Tuesday, July 08, 2008 9:39 AM
> >  >  >  > >  >  To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail)
> >  >  >  > >  >  Cc: Zhang, Li
> >  >  >  > >  >  Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and
> >  >  >  > AMD feedback please)
> >  >  >  > >  >
> >  >  >  > >  >  >From: Dan Magenheimer 
> [mailto:dan.magenheimer@oracle.com]
> >  >  >  > >  >  >Sent: 2008年7月5日 1:32
> >  >  >  > >  >  >
> >  >  >  > >  >  >> >Is this something that you (or Intel in general)
> >  >  >  > could look at?
> >  >  >  > >  >  >> >I would be happy to participate but I 
> don't think I
> >  >  >  > understand
> >  >  >  > >  >  >> >VT well enough.  Once the trap occurs, I suppose
> >  >  >  > Xen system time
> >  >  >  > >  >  >> >could be used as the virtual TSC, 
> possibly scaled up.
> >  >  >  > >  >  >> >
> >  >  >  > >  >  >>
> >  >  >  > >  >  >> There should be tiny related to VT, as 
> only turning
> >  >  >  > on some bit to
> >  >  >  > >  >  >> allow RDTSC trapping and then the rest 
> stuff should be common
> >  >  >  > >  >  >> how to handle it. We'll take a look, but can't
> >  >  >  > commit the time due
> >  >  >  > >  >  >> to other scheduled bandwidth. But if you'd like to
> >  >  >  > jump in early
> >  >  >  > >  >  >> we definitely can help with VT side.
> >  >  >  > >  >  >
> >  >  >  > >  >  >If you can post a patch with code that:
> >  >  >  > >  >  >
> >  >  >  > >  >  >1) declares a boolean global variable: softtsc = 0
> >  >  >  > >  >  >2) if the variable is set, a rdtsc 
> instruction in any hvm
> >  >  >  > >  >  >   domain causes a trap
> >  >  >  > >  >  >3) the trap handler just does a physical 
> rdtsc and returns
> >  >  >  > >  >  >
> >  >  >  > >  >  >then I could probably take it from there.
> >  >  >  > >  >
> >  >  >  > >  >  OK, and Li in CC will take a look and bake a 
> patch for you.
> >  >  >  > >  >
> >  >  >  > >  >  Thanks,
> >  >  >  > >  >  Kevin
> >  >  >  >
> >
> >  _______________________________________________
> >  Xen-devel mailing list
> >  Xen-devel@lists.xensource.com
> >  http://lists.xensource.com/xen-devel
> 
> 

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel