Saving/Restoring IA32_TSC_AUX MSR

Saving/Restoring IA32_TSC_AUX MSR

[Nakajima, Jun]

I see the code like (in arch/x86/time.c), and wondering how IA32_TSC_AUX MSR is saved/restored at domain switch time.

    if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
         boot_cpu_has(X86_FEATURE_RDTSCP) )
        write_rdtscp_aux(d->arch.incarnation);

BTW,

include/asm-x86/msr.h
#define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)

We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding 
+#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
in include/asm-x86/msr-index.h

Thanks,
Jun
---
Intel Open Source Technology Center

RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

Hi Jun --

Xen doesn't expose the TSC rdtscp bit so assumes that
no guests depend on it.  So no save/restore of TSC_AUX
is necessary.  Xen could provide support for the TSC
rdtscp bit and allow a guest OS to manage TSC_AUX, but
the existing use of TSC_AUX by Linux would fail to
provide the desired result across migration, so there's
little point.  Also the pvrdtscp algorithm now assumes
that Xen itself is responsible for updating TSC_AUX
whenever a migration (across physical machines) occurs.

The #define for write_rdtscp_aux is from Linux source,
so I didn't change the code and define the constant.

Dan

> -----Original Message-----
> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> Sent: Wednesday, December 09, 2009 9:42 AM
> To: xen-devel@lists.xensource.com
> Cc: Dan Magenheimer
> Subject: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> I see the code like (in arch/x86/time.c), and wondering how 
> IA32_TSC_AUX MSR is saved/restored at domain switch time.
> 
>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
>          boot_cpu_has(X86_FEATURE_RDTSCP) )
>         write_rdtscp_aux(d->arch.incarnation);
> 
> BTW,
> 
> include/asm-x86/msr.h
> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
> 
> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding 
> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
> in include/asm-x86/msr-index.h
> 
> Thanks,
> Jun
> ---
> Intel Open Source Technology Center
> 
>

RE: Saving/Restoring IA32_TSC_AUX MSR

[Nakajima, Jun]

Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:

> Hi Jun --
> 

Dan,

> Xen doesn't expose the TSC rdtscp bit so assumes that
> no guests depend on it.  So no save/restore of TSC_AUX
> is necessary.  Xen could provide support for the TSC

But it's possible that multiple domains use the pvrdtscp algorithm, and the incarnation number is domain specific. We also have the issue when adding RDTSCP support for HVM guests.

> rdtscp bit and allow a guest OS to manage TSC_AUX, but
> the existing use of TSC_AUX by Linux would fail to
> provide the desired result across migration, so there's
> little point.  Also the pvrdtscp algorithm now assumes
> that Xen itself is responsible for updating TSC_AUX
> whenever a migration (across physical machines) occurs.
> 
> The #define for write_rdtscp_aux is from Linux source,
> so I didn't change the code and define the constant.
> 
> Dan
> 
>> -----Original Message-----
>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
>> Sent: Wednesday, December 09, 2009 9:42 AM
>> To: xen-devel@lists.xensource.com
>> Cc: Dan Magenheimer
>> Subject: Saving/Restoring IA32_TSC_AUX MSR
>> 
>> 
>> I see the code like (in arch/x86/time.c), and wondering how
>> IA32_TSC_AUX MSR is saved/restored at domain switch time.
>> 
>>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
>>          boot_cpu_has(X86_FEATURE_RDTSCP) )
>>         write_rdtscp_aux(d->arch.incarnation);
>> 
>> BTW,
>> 
>> include/asm-x86/msr.h
>> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
>> 
>> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
>> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
>> in include/asm-x86/msr-index.h
>> 
>> Thanks,
>> Jun
>> ---
>> Intel Open Source Technology Center
>> 
>>

Jun
___
Intel Open Source Technology Center

RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

Hi Jun --

> But it's possible that multiple domains use the pvrdtscp 
> algorithm, and the incarnation number is domain specific.

OK, I see.  The code for writing TSC_AUX is in
__update_vcpu_system_time() not in context switch.

> We also have the issue when adding RDTSCP support for
> HVM guests.

Only if you expose the rdtscp bit via cpuid.  This could
certainly be done but, as I said, is probably pointless.
(The pvrdtscp algorithm uses the instruction whether or
not the rdtscp bit is set in cpuid, since Xen emulates
it -- for PV domains only now -- if the physical machine
doesn't support the instruction.

Dan

> -----Original Message-----
> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> Sent: Wednesday, December 09, 2009 10:08 AM
> To: Dan Magenheimer; xen-devel@lists.xensource.com
> Subject: RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:
> 
> > Hi Jun --
> > 
> 
> Dan,
> 
> > Xen doesn't expose the TSC rdtscp bit so assumes that
> > no guests depend on it.  So no save/restore of TSC_AUX
> > is necessary.  Xen could provide support for the TSC
> 
> But it's possible that multiple domains use the pvrdtscp 
> algorithm, and the incarnation number is domain specific. We 
> also have the issue when adding RDTSCP support for HVM guests.
> 
> > rdtscp bit and allow a guest OS to manage TSC_AUX, but
> > the existing use of TSC_AUX by Linux would fail to
> > provide the desired result across migration, so there's
> > little point.  Also the pvrdtscp algorithm now assumes
> > that Xen itself is responsible for updating TSC_AUX
> > whenever a migration (across physical machines) occurs.
> > 
> > The #define for write_rdtscp_aux is from Linux source,
> > so I didn't change the code and define the constant.
> > 
> > Dan
> > 
> >> -----Original Message-----
> >> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> >> Sent: Wednesday, December 09, 2009 9:42 AM
> >> To: xen-devel@lists.xensource.com
> >> Cc: Dan Magenheimer
> >> Subject: Saving/Restoring IA32_TSC_AUX MSR
> >> 
> >> 
> >> I see the code like (in arch/x86/time.c), and wondering how
> >> IA32_TSC_AUX MSR is saved/restored at domain switch time.
> >> 
> >>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
> >>          boot_cpu_has(X86_FEATURE_RDTSCP) )
> >>         write_rdtscp_aux(d->arch.incarnation);
> >> 
> >> BTW,
> >> 
> >> include/asm-x86/msr.h
> >> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
> >> 
> >> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
> >> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
> >> in include/asm-x86/msr-index.h
> >> 
> >> Thanks,
> >> Jun
> >> ---
> >> Intel Open Source Technology Center
> >> 
> >>
> 
> Jun
> ___
> Intel Open Source Technology Center
> 
> 
> 
>

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Xu, Dongxiao]

Hi, Dan, 
	I am now trying to add the rdtscp support for Xen HVM guest. 
	I have some questions about your pvrdtscp patch. See below. 

Dan Magenheimer wrote:
> Hi Jun --
> 
>> But it's possible that multiple domains use the pvrdtscp
>> algorithm, and the incarnation number is domain specific.
> 
> OK, I see.  The code for writing TSC_AUX is in
> __update_vcpu_system_time() not in context switch.

Will you modify the place where Hypervisor writes TSC_AUX MSR? 
In the current pvrdtscp logic, I think this MSR should be written while
vcpu context switch. Also, this will make HVM support much easier 
because that MSR would not be modified by Hypervisor time to time.

> 
>> We also have the issue when adding RDTSCP support for
>> HVM guests.
> 
> Only if you expose the rdtscp bit via cpuid.  This could
> certainly be done but, as I said, is probably pointless.
> (The pvrdtscp algorithm uses the instruction whether or
> not the rdtscp bit is set in cpuid, since Xen emulates
> it -- for PV domains only now -- if the physical machine
> doesn't support the instruction.

We are planning to add HVM support for RDTSCP, and the behavior for this instruction
will follow the native way. 
This caused a problem that RDTSCP instruction in application has different experience
upon PV and HVM domains. Do you have any comment about this? Thanks!

Thanks!
Dongxiao

> 
> Dan
> 
>> -----Original Message-----
>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
>> Sent: Wednesday, December 09, 2009 10:08 AM
>> To: Dan Magenheimer; xen-devel@lists.xensource.com
>> Subject: RE: Saving/Restoring IA32_TSC_AUX MSR
>> 
>> 
>> Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:
>> 
>>> Hi Jun --
>>> 
>> 
>> Dan,
>> 
>>> Xen doesn't expose the TSC rdtscp bit so assumes that
>>> no guests depend on it.  So no save/restore of TSC_AUX
>>> is necessary.  Xen could provide support for the TSC
>> 
>> But it's possible that multiple domains use the pvrdtscp
>> algorithm, and the incarnation number is domain specific. We
>> also have the issue when adding RDTSCP support for HVM guests.
>> 
>>> rdtscp bit and allow a guest OS to manage TSC_AUX, but
>>> the existing use of TSC_AUX by Linux would fail to
>>> provide the desired result across migration, so there's
>>> little point.  Also the pvrdtscp algorithm now assumes
>>> that Xen itself is responsible for updating TSC_AUX
>>> whenever a migration (across physical machines) occurs.
>>> 
>>> The #define for write_rdtscp_aux is from Linux source,
>>> so I didn't change the code and define the constant.
>>> 
>>> Dan
>>> 
>>>> -----Original Message-----
>>>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
>>>> Sent: Wednesday, December 09, 2009 9:42 AM
>>>> To: xen-devel@lists.xensource.com
>>>> Cc: Dan Magenheimer
>>>> Subject: Saving/Restoring IA32_TSC_AUX MSR
>>>> 
>>>> 
>>>> I see the code like (in arch/x86/time.c), and wondering how
>>>> IA32_TSC_AUX MSR is saved/restored at domain switch time.
>>>> 
>>>>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
>>>>          boot_cpu_has(X86_FEATURE_RDTSCP) )
>>>>         write_rdtscp_aux(d->arch.incarnation);
>>>> 
>>>> BTW,
>>>> 
>>>> include/asm-x86/msr.h
>>>> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
>>>> 
>>>> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
>>>> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
>>>> in include/asm-x86/msr-index.h
>>>> 
>>>> Thanks,
>>>> Jun
>>>> ---
>>>> Intel Open Source Technology Center
>>>> 
>>>> 
>> 
>> Jun
>> ___
>> Intel Open Source Technology Center
>> 
>> 
>> 
>> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

Hi Dongxiao --

There are two approaches to adding rdtscp support:

1) Faithful full implementation of rdtscp instruction
2) Support pvrtdtscp algorithm

For (1), you would enable the rdtscp bit in cpuid.  Then
on hardware that supports rdtscp, you would do context
switching of TSC_AUX.  On hardware that doesn't support
rdtscp, you would intercept the illegal instruction trap
and emulate the instruction.  (TSC_AUX emulation
could be handled "lazily", no need to do context
switch for that.)

BUT if you look at how TSC_AUX is used by a native
OS**, the OS sets TSC_AUX to each physical CPU number
so an application can easily determine if successive
rdtscp instructions were not executed on the same
processor.  (This was important on older processors
that did not have invariant TSC.)  Unfortunately,
on Xen, this mechanism is worthless and misleading
because the OS believes it is setting TSC_AUX to
a physical CPU number but it is actually setting
it to a virtual CPU number, and the physical CPU
number may change at any time due to scheduling
or migration.  So an app using rdtscp will get a
wrong answer.

As a result, I do NOT recommend (1) and do recommend
that Xen should continue to return zero for the rdtscp
bit in cpuid.

For (2), setting TSC_AUX in __update_vcpu_system_time()
is fine (I think).  On hardware that supports, for HVM
you would need to ensure that the rdtscp instruction
works natively (even though the rdtscp bit in cpuid
is not turned on for the guest).  On hardware that
does not support rdtscp, you would intercept the illegal
instruction trap and call the existing code in
pv_soft_rdtsc().

Does that make sense?

Thanks,
Dan

** I've looked at RHEL5.  Windows actually always
returns 0 for TSC_AUX.

> -----Original Message-----
> From: Xu, Dongxiao [mailto:dongxiao.xu@intel.com]
> Sent: Thursday, December 10, 2009 4:22 AM
> To: Dan Magenheimer; Nakajima, Jun; 
> xen-devel@lists.xensource.com; Keir
> Fraser
> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> Hi, Dan, 
> 	I am now trying to add the rdtscp support for Xen HVM guest. 
> 	I have some questions about your pvrdtscp patch. See below. 
> 
> Dan Magenheimer wrote:
> > Hi Jun --
> > 
> >> But it's possible that multiple domains use the pvrdtscp
> >> algorithm, and the incarnation number is domain specific.
> > 
> > OK, I see.  The code for writing TSC_AUX is in
> > __update_vcpu_system_time() not in context switch.
> 
> Will you modify the place where Hypervisor writes TSC_AUX MSR? 
> In the current pvrdtscp logic, I think this MSR should be 
> written while
> vcpu context switch. Also, this will make HVM support much easier 
> because that MSR would not be modified by Hypervisor time to time.
> 
> > 
> >> We also have the issue when adding RDTSCP support for
> >> HVM guests.
> > 
> > Only if you expose the rdtscp bit via cpuid.  This could
> > certainly be done but, as I said, is probably pointless.
> > (The pvrdtscp algorithm uses the instruction whether or
> > not the rdtscp bit is set in cpuid, since Xen emulates
> > it -- for PV domains only now -- if the physical machine
> > doesn't support the instruction.
> 
> We are planning to add HVM support for RDTSCP, and the 
> behavior for this instruction
> will follow the native way. 
> This caused a problem that RDTSCP instruction in application 
> has different experience
> upon PV and HVM domains. Do you have any comment about this? Thanks!
> 
> Thanks!
> Dongxiao
> 
> > 
> > Dan
> > 
> >> -----Original Message-----
> >> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> >> Sent: Wednesday, December 09, 2009 10:08 AM
> >> To: Dan Magenheimer; xen-devel@lists.xensource.com
> >> Subject: RE: Saving/Restoring IA32_TSC_AUX MSR
> >> 
> >> 
> >> Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:
> >> 
> >>> Hi Jun --
> >>> 
> >> 
> >> Dan,
> >> 
> >>> Xen doesn't expose the TSC rdtscp bit so assumes that
> >>> no guests depend on it.  So no save/restore of TSC_AUX
> >>> is necessary.  Xen could provide support for the TSC
> >> 
> >> But it's possible that multiple domains use the pvrdtscp
> >> algorithm, and the incarnation number is domain specific. We
> >> also have the issue when adding RDTSCP support for HVM guests.
> >> 
> >>> rdtscp bit and allow a guest OS to manage TSC_AUX, but
> >>> the existing use of TSC_AUX by Linux would fail to
> >>> provide the desired result across migration, so there's
> >>> little point.  Also the pvrdtscp algorithm now assumes
> >>> that Xen itself is responsible for updating TSC_AUX
> >>> whenever a migration (across physical machines) occurs.
> >>> 
> >>> The #define for write_rdtscp_aux is from Linux source,
> >>> so I didn't change the code and define the constant.
> >>> 
> >>> Dan
> >>> 
> >>>> -----Original Message-----
> >>>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> >>>> Sent: Wednesday, December 09, 2009 9:42 AM
> >>>> To: xen-devel@lists.xensource.com
> >>>> Cc: Dan Magenheimer
> >>>> Subject: Saving/Restoring IA32_TSC_AUX MSR
> >>>> 
> >>>> 
> >>>> I see the code like (in arch/x86/time.c), and wondering how
> >>>> IA32_TSC_AUX MSR is saved/restored at domain switch time.
> >>>> 
> >>>>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
> >>>>          boot_cpu_has(X86_FEATURE_RDTSCP) )
> >>>>         write_rdtscp_aux(d->arch.incarnation);
> >>>> 
> >>>> BTW,
> >>>> 
> >>>> include/asm-x86/msr.h
> >>>> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
> >>>> 
> >>>> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
> >>>> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
> >>>> in include/asm-x86/msr-index.h
> >>>> 
> >>>> Thanks,
> >>>> Jun
> >>>> ---
> >>>> Intel Open Source Technology Center
> >>>> 
> >>>> 
> >> 
> >> Jun
> >> ___
> >> Intel Open Source Technology Center
> >> 
> >> 
> >> 
> >> 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Xu, Dongxiao]

Dan, 
	Thanks for reply, some comments below. 

Best Regards,
-- Dongxiao

Dan Magenheimer wrote:
> Hi Dongxiao --
> 
> There are two approaches to adding rdtscp support:
> 
> 1) Faithful full implementation of rdtscp instruction
> 2) Support pvrtdtscp algorithm
> 
> For (1), you would enable the rdtscp bit in cpuid.  Then
> on hardware that supports rdtscp, you would do context
> switching of TSC_AUX.  On hardware that doesn't support
> rdtscp, you would intercept the illegal instruction trap
> and emulate the instruction.  (TSC_AUX emulation
> could be handled "lazily", no need to do context
> switch for that.)
> 
> BUT if you look at how TSC_AUX is used by a native
> OS**, the OS sets TSC_AUX to each physical CPU number
> so an application can easily determine if successive
> rdtscp instructions were not executed on the same
> processor.  (This was important on older processors
> that did not have invariant TSC.)  Unfortunately,
> on Xen, this mechanism is worthless and misleading
> because the OS believes it is setting TSC_AUX to
> a physical CPU number but it is actually setting
> it to a virtual CPU number, and the physical CPU
> number may change at any time due to scheduling
> or migration.  So an app using rdtscp will get a
> wrong answer.

However for HVM, we should keep its behavior the same as
on native machine. So if hardware support rdtscp, we will also
support it in HVM; if not, we will not expose that bit in cpuid 
to guest. 

> 
> As a result, I do NOT recommend (1) and do recommend
> that Xen should continue to return zero for the rdtscp
> bit in cpuid.
> 
> For (2), setting TSC_AUX in __update_vcpu_system_time()
> is fine (I think).  On hardware that supports, for HVM
> you would need to ensure that the rdtscp instruction
> works natively (even though the rdtscp bit in cpuid
> is not turned on for the guest).  On hardware that
> does not support rdtscp, you would intercept the illegal
> instruction trap and call the existing code in
> pv_soft_rdtsc().

Put the writing of TSC_AUX MSR in __update_vcpu_system_time()
has a problem that, Hypervisor will overwrite the value time to time,
( For example, at do_softirq()->local_time_calibration() ), even if the
value didn't change (Currently the domain incarnation value only
increase at save/restore/migration). This makes HVM support a bit
Tricky because we need to save/restore guest/host TSC_AUX at every
VMEXIT/VMENTRY. If both PV/HVM could put TSC_AUX writing in 
context_switch(), then things will become easier for HVM support. 
Do you have idea about It? Thanks!  :-)

> 
> Does that make sense?
> 
> Thanks,
> Dan
> 
> ** I've looked at RHEL5.  Windows actually always
> returns 0 for TSC_AUX.
> 
>> -----Original Message-----
>> From: Xu, Dongxiao [mailto:dongxiao.xu@intel.com]
>> Sent: Thursday, December 10, 2009 4:22 AM
>> To: Dan Magenheimer; Nakajima, Jun;
>> xen-devel@lists.xensource.com; Keir
>> Fraser
>> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
>> 
>> 
>> Hi, Dan,
>> 	I am now trying to add the rdtscp support for Xen HVM guest.
>> 	I have some questions about your pvrdtscp patch. See below.
>> 
>> Dan Magenheimer wrote:
>>> Hi Jun --
>>> 
>>>> But it's possible that multiple domains use the pvrdtscp
>>>> algorithm, and the incarnation number is domain specific.
>>> 
>>> OK, I see.  The code for writing TSC_AUX is in
>>> __update_vcpu_system_time() not in context switch.
>> 
>> Will you modify the place where Hypervisor writes TSC_AUX MSR?
>> In the current pvrdtscp logic, I think this MSR should be
>> written while
>> vcpu context switch. Also, this will make HVM support much easier
>> because that MSR would not be modified by Hypervisor time to time.
>> 
>>> 
>>>> We also have the issue when adding RDTSCP support for
>>>> HVM guests.
>>> 
>>> Only if you expose the rdtscp bit via cpuid.  This could
>>> certainly be done but, as I said, is probably pointless.
>>> (The pvrdtscp algorithm uses the instruction whether or
>>> not the rdtscp bit is set in cpuid, since Xen emulates
>>> it -- for PV domains only now -- if the physical machine
>>> doesn't support the instruction.
>> 
>> We are planning to add HVM support for RDTSCP, and the
>> behavior for this instruction
>> will follow the native way.
>> This caused a problem that RDTSCP instruction in application
>> has different experience
>> upon PV and HVM domains. Do you have any comment about this? Thanks!
>> 
>> Thanks!
>> Dongxiao
>> 
>>> 
>>> Dan
>>> 
>>>> -----Original Message-----
>>>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
>>>> Sent: Wednesday, December 09, 2009 10:08 AM
>>>> To: Dan Magenheimer; xen-devel@lists.xensource.com
>>>> Subject: RE: Saving/Restoring IA32_TSC_AUX MSR
>>>> 
>>>> 
>>>> Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:
>>>> 
>>>>> Hi Jun --
>>>>> 
>>>> 
>>>> Dan,
>>>> 
>>>>> Xen doesn't expose the TSC rdtscp bit so assumes that
>>>>> no guests depend on it.  So no save/restore of TSC_AUX
>>>>> is necessary.  Xen could provide support for the TSC
>>>> 
>>>> But it's possible that multiple domains use the pvrdtscp
>>>> algorithm, and the incarnation number is domain specific. We
>>>> also have the issue when adding RDTSCP support for HVM guests.
>>>> 
>>>>> rdtscp bit and allow a guest OS to manage TSC_AUX, but
>>>>> the existing use of TSC_AUX by Linux would fail to
>>>>> provide the desired result across migration, so there's
>>>>> little point.  Also the pvrdtscp algorithm now assumes
>>>>> that Xen itself is responsible for updating TSC_AUX
>>>>> whenever a migration (across physical machines) occurs.
>>>>> 
>>>>> The #define for write_rdtscp_aux is from Linux source,
>>>>> so I didn't change the code and define the constant.
>>>>> 
>>>>> Dan
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
>>>>>> Sent: Wednesday, December 09, 2009 9:42 AM
>>>>>> To: xen-devel@lists.xensource.com
>>>>>> Cc: Dan Magenheimer
>>>>>> Subject: Saving/Restoring IA32_TSC_AUX MSR
>>>>>> 
>>>>>> 
>>>>>> I see the code like (in arch/x86/time.c), and wondering how
>>>>>> IA32_TSC_AUX MSR is saved/restored at domain switch time.
>>>>>> 
>>>>>>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
>>>>>>          boot_cpu_has(X86_FEATURE_RDTSCP) )
>>>>>>         write_rdtscp_aux(d->arch.incarnation);
>>>>>> 
>>>>>> BTW,
>>>>>> 
>>>>>> include/asm-x86/msr.h
>>>>>> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
>>>>>> 
>>>>>> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
>>>>>> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
>>>>>> in include/asm-x86/msr-index.h
>>>>>> 
>>>>>> Thanks,
>>>>>> Jun
>>>>>> ---
>>>>>> Intel Open Source Technology Center
>>>>>> 
>>>>>> 
>>>> 
>>>> Jun
>>>> ___
>>>> Intel Open Source Technology Center
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> However for HVM, we should keep its behavior the same as
> on native machine. So if hardware support rdtscp, we will also
> support it in HVM; if not, we will not expose that bit in cpuid 
> to guest. 

As I said, I think this a very bad idea because there
is no way to ensure the behavior of an app/OS in a VM
gives the same results as in a physical machine.
So I think the cpuid rdtscp bit should always be off. 

> increase at save/restore/migration). This makes HVM support a bit
> Tricky because we need to save/restore guest/host TSC_AUX at every
> VMEXIT/VMENTRY. If both PV/HVM could put TSC_AUX writing in 
> context_switch(), then things will become easier for HVM support. 

If you are doing a full faithful implementation of
rdtscp (as if cpuid rdtscp bit is on), I agree this
is a problem.  If not, and the only use of TSC_AUX
is for the pvrdtscp algorithm, I think setting
TSC_AUX in __update_vcpu_system_time() is fine
because TSC_AUX is not part of a VM's context,
it is a communication of information from system
software (Xen) to applications.

I expect that Keir will not support putting TSC_AUX
in the context switch code unless it is absolutely
necessary, as it is certainly expensive to read and
write to TSC_AUX and this cost will add to every
context switch of every VM even though very few will
actually use rdtscp/TSC_AUX.

So I think we need to decide first about approach (1),
the full faithful implementation of rdtscp.

> -----Original Message-----
> From: Xu, Dongxiao [mailto:dongxiao.xu@intel.com]
> Sent: Thursday, December 10, 2009 6:23 PM
> To: Dan Magenheimer; Nakajima, Jun; 
> xen-devel@lists.xensource.com; Keir
> Fraser
> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> Dan, 
> 	Thanks for reply, some comments below. 
> 
> Best Regards,
> -- Dongxiao
> 
> Dan Magenheimer wrote:
> > Hi Dongxiao --
> > 
> > There are two approaches to adding rdtscp support:
> > 
> > 1) Faithful full implementation of rdtscp instruction
> > 2) Support pvrtdtscp algorithm
> > 
> > For (1), you would enable the rdtscp bit in cpuid.  Then
> > on hardware that supports rdtscp, you would do context
> > switching of TSC_AUX.  On hardware that doesn't support
> > rdtscp, you would intercept the illegal instruction trap
> > and emulate the instruction.  (TSC_AUX emulation
> > could be handled "lazily", no need to do context
> > switch for that.)
> > 
> > BUT if you look at how TSC_AUX is used by a native
> > OS**, the OS sets TSC_AUX to each physical CPU number
> > so an application can easily determine if successive
> > rdtscp instructions were not executed on the same
> > processor.  (This was important on older processors
> > that did not have invariant TSC.)  Unfortunately,
> > on Xen, this mechanism is worthless and misleading
> > because the OS believes it is setting TSC_AUX to
> > a physical CPU number but it is actually setting
> > it to a virtual CPU number, and the physical CPU
> > number may change at any time due to scheduling
> > or migration.  So an app using rdtscp will get a
> > wrong answer.
> 
> However for HVM, we should keep its behavior the same as
> on native machine. So if hardware support rdtscp, we will also
> support it in HVM; if not, we will not expose that bit in cpuid 
> to guest. 
> 
> > 
> > As a result, I do NOT recommend (1) and do recommend
> > that Xen should continue to return zero for the rdtscp
> > bit in cpuid.
> > 
> > For (2), setting TSC_AUX in __update_vcpu_system_time()
> > is fine (I think).  On hardware that supports, for HVM
> > you would need to ensure that the rdtscp instruction
> > works natively (even though the rdtscp bit in cpuid
> > is not turned on for the guest).  On hardware that
> > does not support rdtscp, you would intercept the illegal
> > instruction trap and call the existing code in
> > pv_soft_rdtsc().
> 
> Put the writing of TSC_AUX MSR in __update_vcpu_system_time()
> has a problem that, Hypervisor will overwrite the value time to time,
> ( For example, at do_softirq()->local_time_calibration() ), 
> even if the
> value didn't change (Currently the domain incarnation value only
> increase at save/restore/migration). This makes HVM support a bit
> Tricky because we need to save/restore guest/host TSC_AUX at every
> VMEXIT/VMENTRY. If both PV/HVM could put TSC_AUX writing in 
> context_switch(), then things will become easier for HVM support. 
> Do you have idea about It? Thanks!  :-)
> 
> > 
> > Does that make sense?
> > 
> > Thanks,
> > Dan
> > 
> > ** I've looked at RHEL5.  Windows actually always
> > returns 0 for TSC_AUX.
> > 
> >> -----Original Message-----
> >> From: Xu, Dongxiao [mailto:dongxiao.xu@intel.com]
> >> Sent: Thursday, December 10, 2009 4:22 AM
> >> To: Dan Magenheimer; Nakajima, Jun;
> >> xen-devel@lists.xensource.com; Keir
> >> Fraser
> >> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> >> 
> >> 
> >> Hi, Dan,
> >> 	I am now trying to add the rdtscp support for Xen HVM guest.
> >> 	I have some questions about your pvrdtscp patch. See below.
> >> 
> >> Dan Magenheimer wrote:
> >>> Hi Jun --
> >>> 
> >>>> But it's possible that multiple domains use the pvrdtscp
> >>>> algorithm, and the incarnation number is domain specific.
> >>> 
> >>> OK, I see.  The code for writing TSC_AUX is in
> >>> __update_vcpu_system_time() not in context switch.
> >> 
> >> Will you modify the place where Hypervisor writes TSC_AUX MSR?
> >> In the current pvrdtscp logic, I think this MSR should be
> >> written while
> >> vcpu context switch. Also, this will make HVM support much easier
> >> because that MSR would not be modified by Hypervisor time to time.
> >> 
> >>> 
> >>>> We also have the issue when adding RDTSCP support for
> >>>> HVM guests.
> >>> 
> >>> Only if you expose the rdtscp bit via cpuid.  This could
> >>> certainly be done but, as I said, is probably pointless.
> >>> (The pvrdtscp algorithm uses the instruction whether or
> >>> not the rdtscp bit is set in cpuid, since Xen emulates
> >>> it -- for PV domains only now -- if the physical machine
> >>> doesn't support the instruction.
> >> 
> >> We are planning to add HVM support for RDTSCP, and the
> >> behavior for this instruction
> >> will follow the native way.
> >> This caused a problem that RDTSCP instruction in application
> >> has different experience
> >> upon PV and HVM domains. Do you have any comment about 
> this? Thanks!
> >> 
> >> Thanks!
> >> Dongxiao
> >> 
> >>> 
> >>> Dan
> >>> 
> >>>> -----Original Message-----
> >>>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> >>>> Sent: Wednesday, December 09, 2009 10:08 AM
> >>>> To: Dan Magenheimer; xen-devel@lists.xensource.com
> >>>> Subject: RE: Saving/Restoring IA32_TSC_AUX MSR
> >>>> 
> >>>> 
> >>>> Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:
> >>>> 
> >>>>> Hi Jun --
> >>>>> 
> >>>> 
> >>>> Dan,
> >>>> 
> >>>>> Xen doesn't expose the TSC rdtscp bit so assumes that
> >>>>> no guests depend on it.  So no save/restore of TSC_AUX
> >>>>> is necessary.  Xen could provide support for the TSC
> >>>> 
> >>>> But it's possible that multiple domains use the pvrdtscp
> >>>> algorithm, and the incarnation number is domain specific. We
> >>>> also have the issue when adding RDTSCP support for HVM guests.
> >>>> 
> >>>>> rdtscp bit and allow a guest OS to manage TSC_AUX, but
> >>>>> the existing use of TSC_AUX by Linux would fail to
> >>>>> provide the desired result across migration, so there's
> >>>>> little point.  Also the pvrdtscp algorithm now assumes
> >>>>> that Xen itself is responsible for updating TSC_AUX
> >>>>> whenever a migration (across physical machines) occurs.
> >>>>> 
> >>>>> The #define for write_rdtscp_aux is from Linux source,
> >>>>> so I didn't change the code and define the constant.
> >>>>> 
> >>>>> Dan
> >>>>> 
> >>>>>> -----Original Message-----
> >>>>>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> >>>>>> Sent: Wednesday, December 09, 2009 9:42 AM
> >>>>>> To: xen-devel@lists.xensource.com
> >>>>>> Cc: Dan Magenheimer
> >>>>>> Subject: Saving/Restoring IA32_TSC_AUX MSR
> >>>>>> 
> >>>>>> 
> >>>>>> I see the code like (in arch/x86/time.c), and wondering how
> >>>>>> IA32_TSC_AUX MSR is saved/restored at domain switch time.
> >>>>>> 
> >>>>>>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
> >>>>>>          boot_cpu_has(X86_FEATURE_RDTSCP) )
> >>>>>>         write_rdtscp_aux(d->arch.incarnation);
> >>>>>> 
> >>>>>> BTW,
> >>>>>> 
> >>>>>> include/asm-x86/msr.h
> >>>>>> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
> >>>>>> 
> >>>>>> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
> >>>>>> +#define MSR_TSC_AUX		0xc0000103 /* Auxiliary TSC */
> >>>>>> in include/asm-x86/msr-index.h
> >>>>>> 
> >>>>>> Thanks,
> >>>>>> Jun
> >>>>>> ---
> >>>>>> Intel Open Source Technology Center
> >>>>>> 
> >>>>>> 
> >>>> 
> >>>> Jun
> >>>> ___
> >>>> Intel Open Source Technology Center
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >>> _______________________________________________
> >>> Xen-devel mailing list
> >>> Xen-devel@lists.xensource.com
> >>> http://lists.xensource.com/xen-devel

Re: RE: Saving/Restoring IA32_TSC_AUX MSR

[Keir Fraser]

On 11/12/2009 02:00, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> I expect that Keir will not support putting TSC_AUX
> in the context switch code unless it is absolutely
> necessary, as it is certainly expensive to read and
> write to TSC_AUX and this cost will add to every
> context switch of every VM even though very few will
> actually use rdtscp/TSC_AUX.

Well, you'd make it dependent on the guest using TSC_AUX, I suppose. I think
that's going to be pretty rare.

> So I think we need to decide first about approach (1),
> the full faithful implementation of rdtscp.

The question has to be: what win do we get for faithful virtualisation of
RDTSCP in a virtualised environment? Supporting CPU instructions just
because they're there is not a useful effort.

 -- Keir

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Zhang, Xiantao]

Keir Fraser wrote:
> On 11/12/2009 02:00, "Dan Magenheimer" <dan.magenheimer@oracle.com>
> wrote: 
> 
>> I expect that Keir will not support putting TSC_AUX
>> in the context switch code unless it is absolutely
>> necessary, as it is certainly expensive to read and
>> write to TSC_AUX and this cost will add to every
>> context switch of every VM even though very few will
>> actually use rdtscp/TSC_AUX.
> 
> Well, you'd make it dependent on the guest using TSC_AUX, I suppose.
> I think that's going to be pretty rare.
> 
>> So I think we need to decide first about approach (1),
>> the full faithful implementation of rdtscp.
> 
> The question has to be: what win do we get for faithful
> virtualisation of RDTSCP in a virtualised environment? Supporting CPU
> instructions just because they're there is not a useful effort.

As I know, RDTSCP can used to implment fast vgetcpu in newer Linux kernel.  Current node and cpu info is saved in the MSR, and applications or libraries can get this info at ring3 through this instruction. If enable this instruction for vmx non-root mode,  it should benefit these kernels I think. 
Thanks!
Xiantao

Re: RE: Saving/Restoring IA32_TSC_AUX MSR

[Keir Fraser]

On 11/12/2009 08:43, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:

>> The question has to be: what win do we get for faithful
>> virtualisation of RDTSCP in a virtualised environment? Supporting CPU
>> instructions just because they're there is not a useful effort.
> 
> As I know, RDTSCP can used to implment fast vgetcpu in newer Linux kernel.
> Current node and cpu info is saved in the MSR, and applications or libraries
> can get this info at ring3 through this instruction. If enable this
> instruction for vmx non-root mode,  it should benefit these kernels I think.

Sounds reasonable. Obviously this will be incompatible with pvrdtscp, but
the latter is off by default so this isn't a too serious problem I think.
Pvrdtscp will simply trump ordinary RDTSCP emulation when it is enabled.

You can put your meddling with TSC_AUX MSR in the context-switch path,
regradless of whether pvrdtscp's stays in __update_vcpu_system_time().

In short: have at it.

 -- Keir

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> As I know, RDTSCP can used to implment fast vgetcpu in 
> newer Linux kernel.

Yes, but code which uses fast vgetcpu is expecting
to get physical cpu and physical node number.  Since
an HVM guest OS only has access to virtual cpu and
virtual node number, the information written to TSC_AUX
by a guest OS is misleading and may silently break any
userland code that assumes it is getting physical
information.

I continue to think this is a bad idea and, to use Keir's
words, is "Supporting CPU instructions just because
they're there".

But, if I am overruled, I'd like to see some measurement
of the cycle cost for writing to TSC_AUX.  Since
Linux only writes it once at __cpuinit time, I wouldn't
be surprised to find out that it is horribly slow
and adding it to every context switch would be slowing
down all users of Xen for a handful of applications --
that are getting incorrect information (vcpu vs pcpu)
anyway.

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Friday, December 11, 2009 2:22 AM
> To: Zhang, Xiantao; Dan Magenheimer; Xu, Dongxiao; Nakajima, Jun;
> xen-devel@lists.xensource.com
> Cc: Dugger, Donald D
> Subject: Re: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> On 11/12/2009 08:43, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
> 
> >> The question has to be: what win do we get for faithful
> >> virtualisation of RDTSCP in a virtualised environment? 
> Supporting CPU
> >> instructions just because they're there is not a useful effort.
> > 
> > As I know, RDTSCP can used to implment fast vgetcpu in 
> newer Linux kernel.
> > Current node and cpu info is saved in the MSR, and 
> applications or libraries
> > can get this info at ring3 through this instruction. If enable this
> > instruction for vmx non-root mode,  it should benefit these 
> kernels I think.
> 
> Sounds reasonable. Obviously this will be incompatible with 
> pvrdtscp, but
> the latter is off by default so this isn't a too serious 
> problem I think.
> Pvrdtscp will simply trump ordinary RDTSCP emulation when it 
> is enabled.
> 
> You can put your meddling with TSC_AUX MSR in the context-switch path,
> regradless of whether pvrdtscp's stays in __update_vcpu_system_time().
> 
> In short: have at it.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Xu, Dongxiao]

Dan Magenheimer wrote:
>> As I know, RDTSCP can used to implment fast vgetcpu in
>> newer Linux kernel.
> 
> Yes, but code which uses fast vgetcpu is expecting
> to get physical cpu and physical node number.  Since
> an HVM guest OS only has access to virtual cpu and
> virtual node number, the information written to TSC_AUX
> by a guest OS is misleading and may silently break any
> userland code that assumes it is getting physical
> information.

This is depend on how the node info is virtualized. 
If the virtual node could reflect the physical
node info, what rdtscp returns is valuable to applications. 

> 
> I continue to think this is a bad idea and, to use Keir's
> words, is "Supporting CPU instructions just because
> they're there".
> 
> But, if I am overruled, I'd like to see some measurement
> of the cycle cost for writing to TSC_AUX.  Since
> Linux only writes it once at __cpuinit time, I wouldn't
> be surprised to find out that it is horribly slow
> and adding it to every context switch would be slowing
> down all users of Xen for a handful of applications --
> that are getting incorrect information (vcpu vs pcpu)
> anyway.

According to the current PVRDTSC logic, write_rdtscp_aux()
is called in each scheduling ( schedule()->
update_vcpu_system_time()->__update_vcpu_system_time()->
write_rdtscp_aux() ), which is more frequent than
__context_switch(). 

> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Friday, December 11, 2009 2:22 AM
>> To: Zhang, Xiantao; Dan Magenheimer; Xu, Dongxiao; Nakajima, Jun;
>> xen-devel@lists.xensource.com Cc: Dugger, Donald D
>> Subject: Re: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
>> 
>> 
>> On 11/12/2009 08:43, "Zhang, Xiantao" <xiantao.zhang@intel.com>
>> wrote: 
>> 
>>>> The question has to be: what win do we get for faithful
>>>> virtualisation of RDTSCP in a virtualised environment? Supporting
>>>> CPU instructions just because they're there is not a useful effort.
>>> 
>>> As I know, RDTSCP can used to implment fast vgetcpu in newer Linux
>>> kernel. Current node and cpu info is saved in the MSR, and
>>> applications or libraries can get this info at ring3 through this
>>> instruction. If enable this instruction for vmx non-root mode,  it
>>> should benefit these kernels I think. 
>> 
>> Sounds reasonable. Obviously this will be incompatible with
>> pvrdtscp, but
>> the latter is off by default so this isn't a too serious
>> problem I think.
>> Pvrdtscp will simply trump ordinary RDTSCP emulation when it
>> is enabled.
>> 
>> You can put your meddling with TSC_AUX MSR in the context-switch
>> path, regradless of whether pvrdtscp's stays in
>> __update_vcpu_system_time(). 
>> 
>> In short: have at it.
>> 
>>  -- Keir
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel



Best Regards,
-- Dongxiao

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> > Yes, but code which uses fast vgetcpu is expecting
> > to get physical cpu and physical node number.  Since
> > an HVM guest OS only has access to virtual cpu and
> > virtual node number, the information written to TSC_AUX
> > by a guest OS is misleading and may silently break any
> > userland code that assumes it is getting physical
> > information.
> 
> This is depend on how the node info is virtualized. 
> If the virtual node could reflect the physical
> node info, what rdtscp returns is valuable to applications. 

If it is possible to ensure that the cpu/node info
is virtualized so that TSC_AUX always correctly provides the
information needed by apps, I agree this would be
valuable.  I don't see how this is possible, but maybe
you have some creative ideas?

> According to the current PVRDTSC logic, write_rdtscp_aux()
> is called in each scheduling ( schedule()->
> update_vcpu_system_time()->__update_vcpu_system_time()->
> write_rdtscp_aux() ), which is more frequent than
> __context_switch(). 

OK, I see.  Then I am OK with moving the call to write_rdtscp_aux()

Dan

Re: RE: Saving/Restoring IA32_TSC_AUX MSR

[Jeremy Fitzhardinge]

On 12/11/09 07:09, Dan Magenheimer wrote:
>> As I know, RDTSCP can used to implment fast vgetcpu in
>> newer Linux kernel.
>>      
> Yes, but code which uses fast vgetcpu is expecting
> to get physical cpu and physical node number.  Since
> an HVM guest OS only has access to virtual cpu and
> virtual node number, the information written to TSC_AUX
> by a guest OS is misleading and may silently break any
> userland code that assumes it is getting physical
> information.
>    

It will fall back to using the segment limit trick to get vcpu+vnode 
info if rdtscp isn't available, so they'll get the info either way.

It's not clear how many apps make good use of the numa node info, but 
presumably some do.  So long as the virtual numa info bears some vague 
resemblance to the real topology then they could still make use of it in 
a Xen domain.  Whether or not Xen currently implements that is a 
separate question.

However, the vcpu number is definitely useful to usermode apps, so they 
can get some idea how they're moved between (v)cpus.  I don't think it 
will matter to them that it isn't pcpu.

     J

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> However, the vcpu number is definitely useful to usermode 
> apps, so they 
> can get some idea how they're moved between (v)cpus.  I don't 
> think it 
> will matter to them that it isn't pcpu.

My point is that an app running on native Linux can
safely assume that, if TSC_AUX==3 at time T1 and
TSC_AUX is still 3 at time T2,it is running
on the same processor and the same node at both T1
and T2.  In a virtual environment it cannot even
assume it is running on the same machine.
Further if the app sees that TSC_AUX==2 at time T3
and TSC_AUX==3 at time T4, on native Linux it
can safely assume that it is running on a different
processor.  While rarer, in a virtual environment,
this may also be a false assumption.

That's why I say the information is misleading.

> -----Original Message-----
> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> Sent: Friday, December 11, 2009 11:21 AM
> To: Dan Magenheimer
> Cc: Keir Fraser; Zhang, Xiantao; Xu, Dongxiao; Nakajima, Jun;
> xen-devel@lists.xensource.com; Dugger, Donald D
> Subject: Re: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> On 12/11/09 07:09, Dan Magenheimer wrote:
> >> As I know, RDTSCP can used to implment fast vgetcpu in
> >> newer Linux kernel.
> >>      
> > Yes, but code which uses fast vgetcpu is expecting
> > to get physical cpu and physical node number.  Since
> > an HVM guest OS only has access to virtual cpu and
> > virtual node number, the information written to TSC_AUX
> > by a guest OS is misleading and may silently break any
> > userland code that assumes it is getting physical
> > information.
> >    
> 
> It will fall back to using the segment limit trick to get vcpu+vnode 
> info if rdtscp isn't available, so they'll get the info either way.
> 
> It's not clear how many apps make good use of the numa node info, but 
> presumably some do.  So long as the virtual numa info bears 
> some vague 
> resemblance to the real topology then they could still make 
> use of it in 
> a Xen domain.  Whether or not Xen currently implements that is a 
> separate question.
> 
> However, the vcpu number is definitely useful to usermode 
> apps, so they 
> can get some idea how they're moved between (v)cpus.  I don't 
> think it 
> will matter to them that it isn't pcpu.
> 
>      J
>

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Nakajima, Jun]

Dan Magenheimer wrote on Fri, 11 Dec 2009 at 08:12:00:

>>> Yes, but code which uses fast vgetcpu is expecting
>>> to get physical cpu and physical node number.  Since
>>> an HVM guest OS only has access to virtual cpu and
>>> virtual node number, the information written to TSC_AUX
>>> by a guest OS is misleading and may silently break any
>>> userland code that assumes it is getting physical
>>> information.
>> 
>> This is depend on how the node info is virtualized.
>> If the virtual node could reflect the physical
>> node info, what rdtscp returns is valuable to applications.
> 
> If it is possible to ensure that the cpu/node info
> is virtualized so that TSC_AUX always correctly provides the
> information needed by apps, I agree this would be
> valuable.  I don't see how this is possible, but maybe
> you have some creative ideas?

It's possible, and the way guest NUMA supposed to be. We are working on that.

Jun
___
Intel Open Source Technology Center

Re: RE: Saving/Restoring IA32_TSC_AUX MSR

[Jeremy Fitzhardinge]

On 12/11/09 10:35, Dan Magenheimer wrote:
>> However, the vcpu number is definitely useful to usermode
>> apps, so they
>> can get some idea how they're moved between (v)cpus.  I don't
>> think it
>> will matter to them that it isn't pcpu.
>>      
> My point is that an app running on native Linux can
> safely assume that, if TSC_AUX==3 at time T1 and
> TSC_AUX is still 3 at time T2,it is running
> on the same processor and the same node at both T1
> and T2.  In a virtual environment it cannot even
> assume it is running on the same machine.
> Further if the app sees that TSC_AUX==2 at time T3
> and TSC_AUX==3 at time T4, on native Linux it
> can safely assume that it is running on a different
> processor.  While rarer, in a virtual environment,
> this may also be a false assumption.
>
> That's why I say the information is misleading.
>    

Sure, but that info is, at best, of heuristic value, and won't cause any 
correctness problems if it is wrong.  The performance may suck, but 
that's part of the larger problem of running NUMA-aware code in a 
virtual environment.

     J

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Nakajima, Jun]

Jeremy Fitzhardinge wrote on Fri, 11 Dec 2009 at 10:50:29:

> On 12/11/09 10:35, Dan Magenheimer wrote:
>>> However, the vcpu number is definitely useful to usermode
>>> apps, so they
>>> can get some idea how they're moved between (v)cpus.  I don't
>>> think it
>>> will matter to them that it isn't pcpu.
>>> 
>> My point is that an app running on native Linux can
>> safely assume that, if TSC_AUX==3 at time T1 and
>> TSC_AUX is still 3 at time T2,it is running
>> on the same processor and the same node at both T1
>> and T2.  In a virtual environment it cannot even
>> assume it is running on the same machine.
>> Further if the app sees that TSC_AUX==2 at time T3
>> and TSC_AUX==3 at time T4, on native Linux it
>> can safely assume that it is running on a different
>> processor.  While rarer, in a virtual environment,
>> this may also be a false assumption.
>> 
>> That's why I say the information is misleading.
>> 
>  Sure, but that info is, at best, of heuristic value, and won't cause
> any correctness problems if it is wrong.  The performance may suck, but
> that's part of the larger problem of running NUMA-aware code in a
> virtual environment.
> 
 
And to utilize various NUMA optimizations in the kernel/apps in the guest, we need "the virtual numa info bears some vague resemblance to the real topology" (from Jeremy's email) with the vcpus bound to the CPU/node.

I understand that enabling RDTSCP in HVM will disable the pvrdtscp algorithm if used by the kernel. One way is to mask off the feature in CPUID (by default). Then kernel won't use it. 

Jun
___
Intel Open Source Technology Center

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> It's possible, and the way guest NUMA supposed to be. We are 
> working on that.

I'd be very interested in learning more about your plans.

> -----Original Message-----
> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> Sent: Friday, December 11, 2009 11:38 AM
> To: Dan Magenheimer; Xu, Dongxiao; Keir Fraser; Zhang, Xiantao;
> xen-devel@lists.xensource.com
> Cc: Dugger, Donald D
> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> Dan Magenheimer wrote on Fri, 11 Dec 2009 at 08:12:00:
> 
> >>> Yes, but code which uses fast vgetcpu is expecting
> >>> to get physical cpu and physical node number.  Since
> >>> an HVM guest OS only has access to virtual cpu and
> >>> virtual node number, the information written to TSC_AUX
> >>> by a guest OS is misleading and may silently break any
> >>> userland code that assumes it is getting physical
> >>> information.
> >> 
> >> This is depend on how the node info is virtualized.
> >> If the virtual node could reflect the physical
> >> node info, what rdtscp returns is valuable to applications.
> > 
> > If it is possible to ensure that the cpu/node info
> > is virtualized so that TSC_AUX always correctly provides the
> > information needed by apps, I agree this would be
> > valuable.  I don't see how this is possible, but maybe
> > you have some creative ideas?
> 
> It's possible, and the way guest NUMA supposed to be. We are 
> working on that.
> 
> Jun
> ___
> Intel Open Source Technology Center
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

Well, although it might be nice to be able to use
rdtscp and TSC_AUX to determine pcpu/vcpu/pnode/vnode
information, I think Jeremy and Jan convinced me in
another thread a couple of months ago that in userland:

x = vgetcpu()
do_other_stuff();
y = vgetcpu()

if x==1 and y==2, there's no way to determine that
do_other_stuff() was executed on cpu 1 vs cpu 2,
or (though unlikely) even on cpu 3.  And if
x==y==4, there's  no guarantee that do_other_stuff()
is executed on cpu 4.

If this is true the only safe use of TSC_AUX is for
its originally designed intent: To determine if two
successive rdtscp instructions were or were not
executed on the same processor.  Since this cannot
be guaranteed in a VM, that's a reasonable argument
that TSC_AUX shouldn't be exposed at all (meaning the
rdtscp bit in cpuid should be turned off by Xen).

True, as long as the information is ONLY used
heuristically to obtain pcpu/vcpu/pnode/vnode info,
and no guarantee of correctness is implied or expected,
it might be useful some of the time.

But frankly, if "performance sucks" when the heuristic
fails due to the fact that the app is running on
a VM instead of native OS, I'd see that as a problem
and suggest the proper way to fix that is to define
more App-to-Xen ABIs so that the app can get the
real information, not a heuristic.  Which also argues
for Xen leaving the rdtscp bit in cpuid turned off

Dan

> -----Original Message-----
> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
> Sent: Friday, December 11, 2009 12:30 PM
> To: Jeremy Fitzhardinge; Dan Magenheimer
> Cc: Keir Fraser; Zhang, Xiantao; Xu, Dongxiao;
> xen-devel@lists.xensource.com; Dugger, Donald D
> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> Jeremy Fitzhardinge wrote on Fri, 11 Dec 2009 at 10:50:29:
> 
> > On 12/11/09 10:35, Dan Magenheimer wrote:
> >>> However, the vcpu number is definitely useful to usermode
> >>> apps, so they
> >>> can get some idea how they're moved between (v)cpus.  I don't
> >>> think it
> >>> will matter to them that it isn't pcpu.
> >>> 
> >> My point is that an app running on native Linux can
> >> safely assume that, if TSC_AUX==3 at time T1 and
> >> TSC_AUX is still 3 at time T2,it is running
> >> on the same processor and the same node at both T1
> >> and T2.  In a virtual environment it cannot even
> >> assume it is running on the same machine.
> >> Further if the app sees that TSC_AUX==2 at time T3
> >> and TSC_AUX==3 at time T4, on native Linux it
> >> can safely assume that it is running on a different
> >> processor.  While rarer, in a virtual environment,
> >> this may also be a false assumption.
> >> 
> >> That's why I say the information is misleading.
> >> 
> >  Sure, but that info is, at best, of heuristic value, and 
> won't cause
> > any correctness problems if it is wrong.  The performance 
> may suck, but
> > that's part of the larger problem of running NUMA-aware code in a
> > virtual environment.
> > 
>  
> And to utilize various NUMA optimizations in the kernel/apps 
> in the guest, we need "the virtual numa info bears some vague 
> resemblance to the real topology" (from Jeremy's email) with 
> the vcpus bound to the CPU/node.
> 
> I understand that enabling RDTSCP in HVM will disable the 
> pvrdtscp algorithm if used by the kernel. One way is to mask 
> off the feature in CPUID (by default). Then kernel won't use it. 
> 
> Jun
> ___
> Intel Open Source Technology Center
> 
> 
> 
>

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Nakajima, Jun]

Dan Magenheimer wrote on Fri, 11 Dec 2009 at 14:23:02:

> Well, although it might be nice to be able to use
> rdtscp and TSC_AUX to determine pcpu/vcpu/pnode/vnode
> information, I think Jeremy and Jan convinced me in
> another thread a couple of months ago that in userland:
> 
> x = vgetcpu()
> do_other_stuff();
> y = vgetcpu()
> 
> if x==1 and y==2, there's no way to determine that
> do_other_stuff() was executed on cpu 1 vs cpu 2,
> or (though unlikely) even on cpu 3.  And if
> x==y==4, there's  no guarantee that do_other_stuff()
> is executed on cpu 4.
> 
> If this is true the only safe use of TSC_AUX is for
> its originally designed intent: To determine if two
> successive rdtscp instructions were or were not
> executed on the same processor.  Since this cannot
> be guaranteed in a VM, that's a reasonable argument
> that TSC_AUX shouldn't be exposed at all (meaning the
> rdtscp bit in cpuid should be turned off by Xen).

This should work if you bind (i.e. pin) each vcpu to each CPU, as I suggested.

Jun
___
Intel Open Source Technology Center

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> > If this is true the only safe use of TSC_AUX is for
> > its originally designed intent: To determine if two
> > successive rdtscp instructions were or were not
> > executed on the same processor.  Since this cannot
> > be guaranteed in a VM, that's a reasonable argument
> > that TSC_AUX shouldn't be exposed at all (meaning the
> > rdtscp bit in cpuid should be turned off by Xen).
> 
> This should work if you bind (i.e. pin) each vcpu to each 
> CPU, as I suggested.

Yes, it does.  If there were a reasonable way for an
application to check "am I running on a VM for which
each vcpu has been pinned?" this might be a reasonable
constraint as, if the app isn't, it could fail or at least
log a message.  But if the app will randomly fail
(or perform horribly) depending on whether the
underlying VM is pinned or not (which might even
change across a migration or if a sysadmin is
"tuning" his data center), I don't think
enterprise customers would appreciate that.

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Xu, Dongxiao]

Dan Magenheimer wrote:
>>> If this is true the only safe use of TSC_AUX is for
>>> its originally designed intent: To determine if two
>>> successive rdtscp instructions were or were not
>>> executed on the same processor.  Since this cannot
>>> be guaranteed in a VM, that's a reasonable argument
>>> that TSC_AUX shouldn't be exposed at all (meaning the
>>> rdtscp bit in cpuid should be turned off by Xen).
>> 
>> This should work if you bind (i.e. pin) each vcpu to each
>> CPU, as I suggested.
> 
> Yes, it does.  If there were a reasonable way for an
> application to check "am I running on a VM for which
> each vcpu has been pinned?" this might be a reasonable
> constraint as, if the app isn't, it could fail or at least
> log a message.  But if the app will randomly fail
> (or perform horribly) depending on whether the
> underlying VM is pinned or not (which might even
> change across a migration or if a sysadmin is
> "tuning" his data center), I don't think
> enterprise customers would appreciate that.

Dan, 
	If later guest NUMA is implemented, both APP and 
Hypervisor/Guest are NUMA awared. APP could get benefit
>From the information of node/processor which is got from
RDTSCP. But how to implement guest NUMA is another story,
either we can use pin, or something other creative idea.

Best Regards,
-- Dongxiao

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> > Yes, it does.  If there were a reasonable way for an
> > application to check "am I running on a VM for which
> > each vcpu has been pinned?" this might be a reasonable
> > constraint as, if the app isn't, it could fail or at least
> > log a message.  But if the app will randomly fail
> > (or perform horribly) depending on whether the
> > underlying VM is pinned or not (which might even
> > change across a migration or if a sysadmin is
> > "tuning" his data center), I don't think
> > enterprise customers would appreciate that.
> 
> Dan, 
> 	If later guest NUMA is implemented, both APP and 
> Hypervisor/Guest are NUMA awared. APP could get benefit
> From the information of node/processor which is got from
> RDTSCP. But how to implement guest NUMA is another story,
> either we can use pin, or something other creative idea.

Right.  A guest NUMA implementation could use:

1) rdtscp+tsc_aux, which is very fast but unreliable
   (unless the app can be certain the guest is permanently
   pinned), or
2) some other yet-to-be-designed mechanism, likely involving
   system calls and/or hypercalls, which is slower but can be
   designed to be always reliable

In my experience in the enterprise world, "slow but
reliable" is always better than "fast but unreliable",
except possibly in well-understood constrained situations.
So I am suggesting we do not implement (1) by NOT
enabling rdtscp-bit-in-cpuid and instead concentrate
on (2).  I guess for the special cases where unreliable
is acceptable, (1) could be an option, but I don't
think it should be turned on by default.

Dan

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Xu, Dongxiao]

Dan Magenheimer wrote:
>>> Yes, it does.  If there were a reasonable way for an
>>> application to check "am I running on a VM for which
>>> each vcpu has been pinned?" this might be a reasonable
>>> constraint as, if the app isn't, it could fail or at least
>>> log a message.  But if the app will randomly fail
>>> (or perform horribly) depending on whether the
>>> underlying VM is pinned or not (which might even
>>> change across a migration or if a sysadmin is
>>> "tuning" his data center), I don't think
>>> enterprise customers would appreciate that.
>> 
>> Dan,
>> 	If later guest NUMA is implemented, both APP and
>> Hypervisor/Guest are NUMA awared. APP could get benefit
>> From the information of node/processor which is got from
>> RDTSCP. But how to implement guest NUMA is another story,
>> either we can use pin, or something other creative idea.
> 
> Right.  A guest NUMA implementation could use:
> 
> 1) rdtscp+tsc_aux, which is very fast but unreliable
>    (unless the app can be certain the guest is permanently
>    pinned), or
> 2) some other yet-to-be-designed mechanism, likely involving
>    system calls and/or hypercalls, which is slower but can be
>    designed to be always reliable

Here is my simple understanding of guest NUMA: it means that
Hypervisor will present the correct NUMA information to 
Guest kernel/app. So once guest NUMA is implemented,
the information got from RDTSCP is both reliable and fast. 

Thanks!
Dongxiao

> 
> In my experience in the enterprise world, "slow but
> reliable" is always better than "fast but unreliable",
> except possibly in well-understood constrained situations.
> So I am suggesting we do not implement (1) by NOT
> enabling rdtscp-bit-in-cpuid and instead concentrate
> on (2).  I guess for the special cases where unreliable
> is acceptable, (1) could be an option, but I don't
> think it should be turned on by default.
> 
> Dan

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Zhang, Xiantao]

Dan Magenheimer wrote:
> Well, although it might be nice to be able to use
> rdtscp and TSC_AUX to determine pcpu/vcpu/pnode/vnode
> information, I think Jeremy and Jan convinced me in
> another thread a couple of months ago that in userland:
> 
> x = vgetcpu()
> do_other_stuff();
> y = vgetcpu()
> 
> if x==1 and y==2, there's no way to determine that
> do_other_stuff() was executed on cpu 1 vs cpu 2,
> or (though unlikely) even on cpu 3.  And if
> x==y==4, there's  no guarantee that do_other_stuff()
> is executed on cpu 4.
> 
> If this is true the only safe use of TSC_AUX is for
> its originally designed intent: To determine if two
> successive rdtscp instructions were or were not
> executed on the same processor.  Since this cannot
> be guaranteed in a VM, that's a reasonable argument
> that TSC_AUX shouldn't be exposed at all (meaning the
> rdtscp bit in cpuid should be turned off by Xen).

Why do you think this is the design intent of this instruction ?  

For guest NUMA support,  it should be a must to pin each vcpu of one VM to some logical proceossors which belong to one specific node(disable vcpu migration between nodes), I think, otherwise, virutal numa may suffer from performance loss.  For example, in a numa system which has two nodes and each node has 4G memory and 8 logical processors. And in this Xen-configured system,  if we carete a VM with 2 G memory with4  vcpu support,  Xen system may allocate 1 G memory from physical node 0 and another 1 G memory from physical node 1.  And in this case, if we virtualize numa for this VM, vcpu0 and vcpu1 can be assinged to virtual node0 , vcpu2 and vcpu3 can be configured for virtual node1, certainly, we also can safely pin vcpu0 and vpcu1 to the physical node0's 8 locial processors and accordingly pin vcpu2 and vcpu3 to the physical node1's 8 physical processors.  Since virtual TSC_AUX is virtualized for each vcpu, and the value is saved/restored for the vcpu when its migration occurs, so if one application always runs on a virtual processors, it should get a fixed value when it calls vgetcpu, envn if this vcpu often migrates among logical processors of one node.   

Back to this topic, in all,  we can't mix the virtual  TSC_AUX of guest with the host's TSC_AUX.  If switch to HVM's vcpu context,  load this vcpu's virtual  TSC_AUX_MSR to physical TSC_AUX_MSR, and when it is sheduled out,  host's TSC_AUX_MSR(which maybe used for pv guests) is loaded.  


> True, as long as the information is ONLY used
> heuristically to obtain pcpu/vcpu/pnode/vnode info,
> and no guarantee of correctness is implied or expected,
> it might be useful some of the time.
> 
> But frankly, if "performance sucks" when the heuristic
> fails due to the fact that the app is running on
> a VM instead of native OS, I'd see that as a problem
> and suggest the proper way to fix that is to define
> more App-to-Xen ABIs so that the app can get the
> real information, not a heuristic.  Which also argues
> for Xen leaving the rdtscp bit in cpuid turned off
> 
> Dan
> 
>> -----Original Message-----
>> From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
>> Sent: Friday, December 11, 2009 12:30 PM
>> To: Jeremy Fitzhardinge; Dan Magenheimer
>> Cc: Keir Fraser; Zhang, Xiantao; Xu, Dongxiao;
>> xen-devel@lists.xensource.com; Dugger, Donald D
>> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
>> 
>> 
>> Jeremy Fitzhardinge wrote on Fri, 11 Dec 2009 at 10:50:29:
>> 
>>> On 12/11/09 10:35, Dan Magenheimer wrote:
>>>>> However, the vcpu number is definitely useful to usermode apps,
>>>>> so they can get some idea how they're moved between (v)cpus.  I
>>>>> don't think it will matter to them that it isn't pcpu.
>>>>> 
>>>> My point is that an app running on native Linux can
>>>> safely assume that, if TSC_AUX==3 at time T1 and
>>>> TSC_AUX is still 3 at time T2,it is running
>>>> on the same processor and the same node at both T1
>>>> and T2.  In a virtual environment it cannot even
>>>> assume it is running on the same machine.
>>>> Further if the app sees that TSC_AUX==2 at time T3
>>>> and TSC_AUX==3 at time T4, on native Linux it
>>>> can safely assume that it is running on a different
>>>> processor.  While rarer, in a virtual environment,
>>>> this may also be a false assumption.
>>>> 
>>>> That's why I say the information is misleading.
>>>> 
>>>  Sure, but that info is, at best, of heuristic value, and won't
>>> cause any correctness problems if it is wrong.  The performance may
>>> suck, but that's part of the larger problem of running NUMA-aware
>>> code in a virtual environment. 
>>> 
>> 
>> And to utilize various NUMA optimizations in the kernel/apps
>> in the guest, we need "the virtual numa info bears some vague
>> resemblance to the real topology" (from Jeremy's email) with
>> the vcpus bound to the CPU/node.
>> 
>> I understand that enabling RDTSCP in HVM will disable the
>> pvrdtscp algorithm if used by the kernel. One way is to mask
>> off the feature in CPUID (by default). Then kernel won't use it.
>> 
>> Jun
>> ___
>> Intel Open Source Technology Center

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer]

> > If this is true the only safe use of TSC_AUX is for
> > its originally designed intent: To determine if two
> > successive rdtscp instructions were or were not
> > executed on the same processor.  Since this cannot
> > be guaranteed in a VM, that's a reasonable argument
> > that TSC_AUX shouldn't be exposed at all (meaning the
> > rdtscp bit in cpuid should be turned off by Xen).
> 
> Why do you think this is the design intent of this instruction ?  

The instruction was designed by AMD for this purpose
a few years ago in order to allow applications to
detect (and correct) possible TSC skew between processors.

> For guest NUMA support,  it should be a must to pin each vcpu 
> of one VM to some logical proceossors which belong to one 
> specific node(disable vcpu migration between nodes), I think, 
> otherwise, virutal numa may suffer from performance loss.

I agree that, for guest NUMA support, restricting all
vcpus to the same physical node is important.  However,
PINNING each vcpu to a fixed pcpu (and never allowing
migration) greatly reduces the value of virtualization.

> For example, in a numa system which has two nodes and each 
> node has 4G memory and 8 logical processors. And in this 
> Xen-configured system,  if we carete a VM with 2 G memory 
> with4  vcpu support,  Xen system may allocate 1 G memory from 
> physical node 0 and another 1 G memory from physical node 1.  
> And in this case, if we virtualize numa for this VM, vcpu0 
> and vcpu1 can be assinged to virtual node0 , vcpu2 and vcpu3 
> can be configured for virtual node1, certainly, we also can 
> safely pin vcpu0 and vpcu1 to the physical node0's 8 locial 
> processors and accordingly pin vcpu2 and vcpu3 to the 
> physical node1's 8 physical processors.  Since virtual 
> TSC_AUX is virtualized for each vcpu, and the value is 
> saved/restored for the vcpu when its migration occurs, so if 
> one application always runs on a virtual processors, it 
> should get a fixed value when it calls vgetcpu, envn if this 
> vcpu often migrates among logical processors of one node.   

I agree there are some cases where the TSC_AUX value
set by a guest OS may be useful.  But ensuring that its
is always useful (NEVER incorrect) requires too many restrictions,
such as pinning.

> Back to this topic, in all,  we can't mix the virtual  
> TSC_AUX of guest with the host's TSC_AUX.  If switch to HVM's 
> vcpu context,  load this vcpu's virtual  TSC_AUX_MSR to 
> physical TSC_AUX_MSR, and when it is sheduled out,  host's 
> TSC_AUX_MSR(which maybe used for pv guests) is loaded.  

I agree they can't be mixed.  My position is that a guest
does not have sufficient information to always correctly set
TSC_AUX, so the best way to avoid the issue is to tell
the guest OS that TSC_AUX doesn't exist (i.e. cpuid-rdtscp
bit is off).  Xen can still set TSC_AUX (and even emulate
it on processors that don't support it) and this information
can still be used (correctly) by virtualization-and-NUMA-aware
OS's and applications.

Re: RE: Saving/Restoring IA32_TSC_AUX MSR

[Jeremy Fitzhardinge]

On 12/13/09 10:06, Dan Magenheimer wrote:
> I agree there are some cases where the TSC_AUX value
> set by a guest OS may be useful.  But ensuring that its
> is always useful (NEVER incorrect) requires too many restrictions,
> such as pinning.
>    

At least with respect to Linux guests [*], this objection to rdtscp is 
moot, because if it isn't present then Linux will fall back to another 
mechanism which is always present.  Guest usermode will get the same 
info, good/bad/misleading/whatever, either way; rdtscp can't make it 
worse.  The only question is whether specifically adding rdtscp/TSC_AUX 
support adds any overall improvement.

(* I don't know if any other rdtscp-users attempt to put NUMA or other 
physical topology info into TSC_AUX.  If they just stick to 
setting/using the cpu number, then they will get a net win from rdtscp.)

     J

RE: RE: Saving/Restoring IA32_TSC_AUX MSR

[Xu, Dongxiao]

Jeremy Fitzhardinge wrote:
> On 12/13/09 10:06, Dan Magenheimer wrote:
>> I agree there are some cases where the TSC_AUX value
>> set by a guest OS may be useful.  But ensuring that its
>> is always useful (NEVER incorrect) requires too many restrictions,
>> such as pinning. 
>> 
> 
> At least with respect to Linux guests [*], this objection to rdtscp is
> moot, because if it isn't present then Linux will fall back to another
> mechanism which is always present.  Guest usermode will get the same
> info, good/bad/misleading/whatever, either way; rdtscp can't make it
> worse.  The only question is whether specifically adding
> rdtscp/TSC_AUX support adds any overall improvement.
> 
> (* I don't know if any other rdtscp-users attempt to put NUMA or other
> physical topology info into TSC_AUX.  If they just stick to
> setting/using the cpu number, then they will get a net win from
> rdtscp.) 

Just have a glance at the open-solaris code, in its mp_startup() function, 
it will write the cpu_id value into the TSC_AUX MSR. Therefore I think 
open-solaris also uses this feature. 

Thanks,
Dongxiao

> 
>      J