[PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Mark Langsdorf]

The Xen hypervisor uses the TSC as a time source.  1st and 2nd generation AMD 
Operton processors have TSCs that change with frequency under power management.
The Xen hypervisor currently cannot cope with the way the TSCs change and the
time will regularly go backwards on multicore systems.  

This patch disables PowerNow! on SMP/CMP 1st and 2nd generation Opteron 
processors and their Athlon 64, Turion, and Sempron equivalents.  3rd generation
Opteron, Athlon 64, and Phenom parts have pstate invariant TSCs and do not
suffer from the time regression.

There is an optional override by passing powernow-k8.xen_tsc_override=1 on
the kernel command line, for users who are willing to trade potential time
regression or drift for improved power usage.

Signed-off-by: Mark Langsdorf <mark.langsdorf@amd.com>

diff -r 47916bd012c6 arch/i386/kernel/cpu/cpufreq/powernow-k8.c
--- a/arch/i386/kernel/cpu/cpufreq/powernow-k8.c	Fri Jan 18 13:37:53 2008 +0000
+++ b/arch/i386/kernel/cpu/cpufreq/powernow-k8.c	Tue Jan 22 11:14:47 2008 -0600
@@ -60,6 +60,7 @@ static cpumask_t cpu_core_map[1];
 static cpumask_t cpu_core_map[1];
 #endif
 
+static int xen_tsc_override;
 /* Return a frequency in MHz, given an input fid */
 static u32 find_freq_from_fid(u32 fid)
 {
@@ -1317,6 +1318,10 @@ static int __cpuinit powernowk8_init(voi
 			supported_cpus++;
 	}
 
+	if (!xen_tsc_override && 
+		disable_if_tsc_problems(cpu_family, supported_cpus))
+		return -ENODEV;
+
 	if (supported_cpus == num_online_cpus()) {
 		powernow_k8_cpu_preinit_acpi();
 		printk(KERN_INFO PFX "Found %d %s "
@@ -1340,5 +1345,8 @@ MODULE_DESCRIPTION("AMD Athlon 64 and Op
 MODULE_DESCRIPTION("AMD Athlon 64 and Opteron processor frequency driver.");
 MODULE_LICENSE("GPL");
 
+module_param(xen_tsc_override,  int, 0);
+MODULE_PARM_DESC(xen_tsc_override, "Allow Xen PowerNow! even if the TSCs are variable.  May cause time to go backwards.");
+
 late_initcall(powernowk8_init);
 module_exit(powernowk8_exit);
diff -r 47916bd012c6 arch/i386/kernel/cpu/cpufreq/powernow-k8.h
--- a/arch/i386/kernel/cpu/cpufreq/powernow-k8.h	Fri Jan 18 13:37:53 2008 +0000
+++ b/arch/i386/kernel/cpu/cpufreq/powernow-k8.h	Tue Jan 22 11:14:47 2008 -0600
@@ -221,3 +221,21 @@ static inline void define_siblings(int c
 	cpu_set(0, cpu_sharedcore_mask[0]);
 }
 #endif
+
+#ifdef CONFIG_XEN
+static inline int disable_if_tsc_problems(unsigned int cpu_family, unsigned int supported_cpus)
+{
+	/* Disable support for multi-core RevF on Xen due to tsc issues */
+	if ((cpu_family == CPU_OPTERON) &&
+	    (supported_cpus > 1))
+		return -ENODEV;
+	else
+		return 0;
+}
+#else
+static inline int disable_if_tsc_problems()
+{
+	return 0;
+}
+#endif
+

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Ian Pratt]

> The Xen hypervisor uses the TSC as a time source.  1st and 2nd
> generation AMD
> Operton processors have TSCs that change with frequency under power
> management.
> The Xen hypervisor currently cannot cope with the way the TSCs change
> and the time will regularly go backwards on multicore systems.

Surely it should be possible to inform Xen of frequency changes (other
that T-states due to thermal throttling) and hence allow Xen to ensure
monotonicity? Xen itself does not require that the CPUs run with the
same rate TSC. It does need to know what frequency they're running at,
and when they change.

Ian

> This patch disables PowerNow! on SMP/CMP 1st and 2nd generation
Opteron
> processors and their Athlon 64, Turion, and Sempron equivalents.  3rd
> generation
> Opteron, Athlon 64, and Phenom parts have pstate invariant TSCs and do
> not
> suffer from the time regression.
> 
> There is an optional override by passing
powernow-k8.xen_tsc_override=1
> on
> the kernel command line, for users who are willing to trade potential
> time
> regression or drift for improved power usage.
> 
> Signed-off-by: Mark Langsdorf <mark.langsdorf@amd.com>
> 
> diff -r 47916bd012c6 arch/i386/kernel/cpu/cpufreq/powernow-k8.c
> --- a/arch/i386/kernel/cpu/cpufreq/powernow-k8.c	Fri Jan 18
> 13:37:53 2008 +0000
> +++ b/arch/i386/kernel/cpu/cpufreq/powernow-k8.c	Tue Jan 22
> 11:14:47 2008 -0600
> @@ -60,6 +60,7 @@ static cpumask_t cpu_core_map[1];
>  static cpumask_t cpu_core_map[1];
>  #endif
> 
> +static int xen_tsc_override;
>  /* Return a frequency in MHz, given an input fid */
>  static u32 find_freq_from_fid(u32 fid)
>  {
> @@ -1317,6 +1318,10 @@ static int __cpuinit powernowk8_init(voi
>  			supported_cpus++;
>  	}
> 
> +	if (!xen_tsc_override &&
> +		disable_if_tsc_problems(cpu_family, supported_cpus))
> +		return -ENODEV;
> +
>  	if (supported_cpus == num_online_cpus()) {
>  		powernow_k8_cpu_preinit_acpi();
>  		printk(KERN_INFO PFX "Found %d %s "
> @@ -1340,5 +1345,8 @@ MODULE_DESCRIPTION("AMD Athlon 64 and Op
>  MODULE_DESCRIPTION("AMD Athlon 64 and Opteron processor frequency
> driver.");
>  MODULE_LICENSE("GPL");
> 
> +module_param(xen_tsc_override,  int, 0);
> +MODULE_PARM_DESC(xen_tsc_override, "Allow Xen PowerNow! even if the
> TSCs are variable.  May cause time to go backwards.");
> +
>  late_initcall(powernowk8_init);
>  module_exit(powernowk8_exit);
> diff -r 47916bd012c6 arch/i386/kernel/cpu/cpufreq/powernow-k8.h
> --- a/arch/i386/kernel/cpu/cpufreq/powernow-k8.h	Fri Jan 18
> 13:37:53 2008 +0000
> +++ b/arch/i386/kernel/cpu/cpufreq/powernow-k8.h	Tue Jan 22
> 11:14:47 2008 -0600
> @@ -221,3 +221,21 @@ static inline void define_siblings(int c
>  	cpu_set(0, cpu_sharedcore_mask[0]);
>  }
>  #endif
> +
> +#ifdef CONFIG_XEN
> +static inline int disable_if_tsc_problems(unsigned int cpu_family,
> unsigned int supported_cpus)
> +{
> +	/* Disable support for multi-core RevF on Xen due to tsc issues
> */
> +	if ((cpu_family == CPU_OPTERON) &&
> +	    (supported_cpus > 1))
> +		return -ENODEV;
> +	else
> +		return 0;
> +}
> +#else
> +static inline int disable_if_tsc_problems()
> +{
> +	return 0;
> +}
> +#endif
> +
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 22/1/08 21:16, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:

>> The Xen hypervisor uses the TSC as a time source.  1st and 2nd
>> generation AMD
>> Operton processors have TSCs that change with frequency under power
>> management.
>> The Xen hypervisor currently cannot cope with the way the TSCs change
>> and the time will regularly go backwards on multicore systems.
> 
> Surely it should be possible to inform Xen of frequency changes (other
> that T-states due to thermal throttling) and hence allow Xen to ensure
> monotonicity? Xen itself does not require that the CPUs run with the
> same rate TSC. It does need to know what frequency they're running at,
> and when they change.

Yes, and that is already done, hooked off the cpu notifier callback chain.
Nobbling the C- and P-state logic shouldn't be necessary.

 -- Keir

RE: [PATCH] Disable Xen PowerNow! support on Opteron2nd gen and earlier processors

[Tian, Kevin]

>From:Keir Fraser
>Sent: 2008年1月23日 5:52
>On 22/1/08 21:16, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:
>
>>> The Xen hypervisor uses the TSC as a time source.  1st and 2nd
>>> generation AMD
>>> Operton processors have TSCs that change with frequency under power
>>> management.
>>> The Xen hypervisor currently cannot cope with the way the 
>TSCs change
>>> and the time will regularly go backwards on multicore systems.
>> 
>> Surely it should be possible to inform Xen of frequency 
>changes (other
>> that T-states due to thermal throttling) and hence allow Xen 
>to ensure
>> monotonicity? Xen itself does not require that the CPUs run with the
>> same rate TSC. It does need to know what frequency they're 
>running at,
>> and when they change.
>
>Yes, and that is already done, hooked off the cpu notifier 
>callback chain.
>Nobbling the C- and P-state logic shouldn't be necessary.
>
> -- Keir

I remembered Mark is the one to add that notification interface,
and so also interested why this patch is required... unless those
early stepping as some processor-initiated freq change... 

Thanks,
Kevin

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> > Surely it should be possible to inform Xen of frequency 
> changes (other
> > that T-states due to thermal throttling) and hence allow 
> Xen to ensure
> > monotonicity? Xen itself does not require that the CPUs run with the
> > same rate TSC. It does need to know what frequency they're 
> running at,
> > and when they change.
> 
> Yes, and that is already done, hooked off the cpu notifier 
> callback chain.
> Nobbling the C- and P-state logic shouldn't be necessary.

And yet, people are still reporting "Time went backwards"
messages upwards of 30-40 times an hour on RevF Opterons
when PowerNow! is enabled on Xen.

I spent a month trying to debug this and I made no significant
progress.  If someone who understands Xen timekeeping wants to
help me dive into this, I'll be glad to take another stab at
it.  As it is, even with informing Xen of frequency changes,
Xen does not ensure monotinicity.

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 23/1/08 15:17, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> Yes, and that is already done, hooked off the cpu notifier
>> callback chain.
>> Nobbling the C- and P-state logic shouldn't be necessary.
> 
> And yet, people are still reporting "Time went backwards"
> messages upwards of 30-40 times an hour on RevF Opterons
> when PowerNow! is enabled on Xen.
> 
> I spent a month trying to debug this and I made no significant
> progress.  If someone who understands Xen timekeeping wants to
> help me dive into this, I'll be glad to take another stab at
> it.  As it is, even with informing Xen of frequency changes,
> Xen does not ensure monotinicity.

Seeing them at that frequency (less than one a minute) is not too serious.
Perhaps we should get rid of the message, or reduce its priority.

 -- Keir

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> >> Yes, and that is already done, hooked off the cpu notifier
> >> callback chain.
> >> Nobbling the C- and P-state logic shouldn't be necessary.
> > 
> > And yet, people are still reporting "Time went backwards"
> > messages upwards of 30-40 times an hour on RevF Opterons
> > when PowerNow! is enabled on Xen.
> > 
> > I spent a month trying to debug this and I made no significant
> > progress.  If someone who understands Xen timekeeping wants to
> > help me dive into this, I'll be glad to take another stab at
> > it.  As it is, even with informing Xen of frequency changes,
> > Xen does not ensure monotinicity.
> 
> Seeing them at that frequency (less than one a minute) is not 
> too serious.
> Perhaps we should get rid of the message, or reduce its priority.

Red Hat engineering is requesting the disable because they've
gotten STREAM results with the test finishing before it started.

For myself, I don't mind the frequency of the messages.  In the
real world, it's serious, especially on financial servers.

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 23/1/08 15:31, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> Seeing them at that frequency (less than one a minute) is not
>> too serious.
>> Perhaps we should get rid of the message, or reduce its priority.
> 
> Red Hat engineering is requesting the disable because they've
> gotten STREAM results with the test finishing before it started.
> 
> For myself, I don't mind the frequency of the messages.  In the
> real world, it's serious, especially on financial servers.

Firstly, Red Hat can apply this patch to their own tree, regardless of
whether or not it's in 2.6.18-xen.hg. Secondly, the cpufreq stuff only kicks
in if you put cpufreq=dom0-kernel on the Xen command line. Perhaps they
shouldn't do that? They could even omit it dynamically at install time,
based on install-hardware configuration.

 -- Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Stephen C. Tweedie]

Hi,

On Wed, 2008-01-23 at 15:26 +0000, Keir Fraser wrote:

> > I spent a month trying to debug this and I made no significant
> > progress.  If someone who understands Xen timekeeping wants to
> > help me dive into this, I'll be glad to take another stab at
> > it.  As it is, even with informing Xen of frequency changes,
> > Xen does not ensure monotinicity.
> 
> Seeing them at that frequency (less than one a minute) is not too serious.
> Perhaps we should get rid of the message, or reduce its priority.

They are most definitely serious --- gettimeofday() going backwards
breaks Unix semantics.  It breaks applications.  It _will_ corrupt
application data --- things like database rollback logs that rely on
timestamp monotonicity to order things correctly will break.

I think this is something that Xen upstream ought to be pretty concerned
about.  If we can't fix it properly, then the workaround of disabling
powernow seems to be a must-have.

--Stephen

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[John Levon]

On Fri, Jan 25, 2008 at 05:19:53PM +0000, Stephen C. Tweedie wrote:

> > Seeing them at that frequency (less than one a minute) is not too serious.
> > Perhaps we should get rid of the message, or reduce its priority.
> 
> They are most definitely serious --- gettimeofday() going backwards
> breaks Unix semantics.  It breaks applications.  It _will_ corrupt
> application data --- things like database rollback logs that rely on
> timestamp monotonicity to order things correctly will break.

I thought Linux now enforced a monotonic clock the same way we were
forced to - that is, globally synchronise and make sure we return last
value + 1, even when the Xen clock goes backwards across CPUs, which it
does regularly, powernow or not?

regards
john

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 25/1/08 17:46, "John Levon" <levon@movementarian.org> wrote:

> On Fri, Jan 25, 2008 at 05:19:53PM +0000, Stephen C. Tweedie wrote:
> 
>>> Seeing them at that frequency (less than one a minute) is not too serious.
>>> Perhaps we should get rid of the message, or reduce its priority.
>> 
>> They are most definitely serious --- gettimeofday() going backwards
>> breaks Unix semantics.  It breaks applications.  It _will_ corrupt
>> application data --- things like database rollback logs that rely on
>> timestamp monotonicity to order things correctly will break.
> 
> I thought Linux now enforced a monotonic clock the same way we were
> forced to - that is, globally synchronise and make sure we return last
> value + 1, even when the Xen clock goes backwards across CPUs, which it
> does regularly, powernow or not?

This is also true. Although in this case the backwards deltas of >100ms are
a bit extreme and will cause application-visible progress of time to stall
for that length of time. We've suggested some leads for Mark to look into,
so hopefully that will improve matters. Ultimately, the work being done to
put the C/P-state governors in Xen is the better approach, once that's
ready.

 -- Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 25/1/08 17:19, "Stephen C. Tweedie" <sct@redhat.com> wrote:

>> Seeing them at that frequency (less than one a minute) is not too serious.
>> Perhaps we should get rid of the message, or reduce its priority.
> 
> They are most definitely serious --- gettimeofday() going backwards
> breaks Unix semantics.  It breaks applications.  It _will_ corrupt
> application data --- things like database rollback logs that rely on
> timestamp monotonicity to order things correctly will break.
> 
> I think this is something that Xen upstream ought to be pretty concerned
> about.  If we can't fix it properly, then the workaround of disabling
> powernow seems to be a must-have.

It's a primary reason why we hide it behind an explicit Xen boot parameter.

 -- Keir

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Ian Pratt]

> > Yes, and that is already done, hooked off the cpu notifier
> > callback chain.
> > Nobbling the C- and P-state logic shouldn't be necessary.
> 
> And yet, people are still reporting "Time went backwards"
> messages upwards of 30-40 times an hour on RevF Opterons
> when PowerNow! is enabled on Xen.
> 
> I spent a month trying to debug this and I made no significant
> progress.  If someone who understands Xen timekeeping wants to
> help me dive into this, I'll be glad to take another stab at
> it.  As it is, even with informing Xen of frequency changes,
> Xen does not ensure monotinicity.

I do think we need to understand what's going on here.

Are the 'time went backwards' messages correlated with power management
operations? How far is time stepping backwards? ns, us, or ms? 

Are you sure the systems aren't overheating and performing emergency
T-state thermal throttling?  

Ian

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> > > Yes, and that is already done, hooked off the cpu notifier
> > > callback chain. Nobbling the C- and P-state logic
< shouldn't be necessary.
> > 
> > And yet, people are still reporting "Time went backwards"
> > messages upwards of 30-40 times an hour on RevF Opterons
> > when PowerNow! is enabled on Xen.
> > 
> > I spent a month trying to debug this and I made no significant
> > progress.  If someone who understands Xen timekeeping wants to
> > help me dive into this, I'll be glad to take another stab at
> > it.  As it is, even with informing Xen of frequency changes,
> > Xen does not ensure monotinicity.
> 
> I do think we need to understand what's going on here.
> 
> Are the 'time went backwards' messages correlated with power 
> management operations?

Yes, I can regularly generate messages by switching the governor
from powersave (minimum frequency) to performance (maximum
frequency).

> How far is time stepping backwards?
> ns, us, or ms? 
 
Rolf's message is typical:
> Jan 19 00:11:09 server kernel: printk: 14 messages suppressed.
> Jan 19 00:11:09 server kernel: Timer ISR/0: Time went backwards:
> delta=-123862858 delta_cpu=-123862858 shadow=
> 2993571302800 off=705014593 processed=2994400167600
cpu_processed=2994400167600
> Jan 19 00:11:09 server kernel:  0: 2994400167600
> Jan 19 00:11:09 server kernel:  1: 2994213786880
> Jan 19 00:11:09 server kernel: Timer ISR/1: Time went backwards:
> delta=-50043804 delta_cpu=136336916 shadow=29
> 93571302800 off=778821372 processed=2994400167600
cpu_processed=2994213786880
> Jan 19 00:11:09 server kernel:  0: 2994400167600
> Jan 19 00:11:09 server kernel:  1: 2994213786880
> Jan 19 00:11:09 server kernel: Timer ISR/1: Time went backwards:
> delta=-70033718 delta_cpu=66347002 shadow=299
> 3571302800 off=838831281 processed=2994480167600
cpu_processed=2994343786880
> Jan 19 00:11:09 server kernel:  0: 2994480167600
> Jan 19 00:11:09 server kernel:  1: 2994343786880
>
> This happens also if I switch manually with
> cpufreq-set -f 2.60GHz of cpufrequtils.

It seems to happen most reliably when there's a large jump in
frequency (2.6 GHz to 800 MHz or vice versa) but I've seen it
happen for smaller switches.  Usually there's 1 large backward
movement, and then 2-4 smaller ones.  It doesn't always happen,
especially if there are multiple shifts in a short period (ie,
constant changes on the ondemand governor) and it seems to 
happen more consistently when increasing frequency.

> Are you sure the systems aren't overheating and performing emergency
> T-state thermal throttling?  

Yes.  The same machine that instantly has time going backwards
after a pstate change can stay at that pstate for hours without
any time issues, whether its a low pstate or a high pstate.

-Mark Langsdorf
Operating System Research Center
AMD

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Ian Pratt]

> Yes, I can regularly generate messages by switching the governor
> from powersave (minimum frequency) to performance (maximum
> frequency).
> 
> > How far is time stepping backwards?
> > ns, us, or ms?
> 
> Rolf's message is typical:
> > Jan 19 00:11:09 server kernel: printk: 14 messages suppressed.
> > Jan 19 00:11:09 server kernel: Timer ISR/0: Time went backwards:
> > delta=-123862858 delta_cpu=-123862858 shadow=
> > 2993571302800 off=705014593 processed=2994400167600
> cpu_processed=2994400167600
>
> It seems to happen most reliably when there's a large jump in
> frequency (2.6 GHz to 800 MHz or vice versa) but I've seen it
> happen for smaller switches.  Usually there's 1 large backward
> movement, and then 2-4 smaller ones.  It doesn't always happen,
> especially if there are multiple shifts in a short period (ie,
> constant changes on the ondemand governor) and it seems to
> happen more consistently when increasing frequency.

Increasing the frequency requires (1) instructing the regulator to
increasing the voltage, (2) waiting for it to settle, (3) increasing the
multipler, (4) wait for the PLL to settle, (5) unpause the CPU.

Which of these steps are under software control on PowerNow?  

Since we probably don't know how long we had to wait between when the
multipler was increased and when the CPU was un-paused, it will be
necessary to recalibrate the CPU to the global time source (PIT/HPET).

Ian

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> > It seems to happen most reliably when there's a large jump in
> > frequency (2.6 GHz to 800 MHz or vice versa) but I've seen it
> > happen for smaller switches.  Usually there's 1 large backward
> > movement, and then 2-4 smaller ones.  It doesn't always happen,
> > especially if there are multiple shifts in a short period (ie,
> > constant changes on the ondemand governor) and it seems to
> > happen more consistently when increasing frequency.
> 
> Increasing the frequency requires (1) instructing the regulator to
> increasing the voltage, (2) waiting for it to settle, (3) 
> increasing the multipler, (4) wait for the PLL to settle,
> (5) unpause the CPU.

Essentially, except that frequency shifts have to be performed in
steps of 200 MHz, aside from an elavator jump from 800 or 1000 MHz
to 1600 or 1800 MHz.  So steps 3-5 have to happen repeatedly,
and there are steps 6) instruct the regulator to decrease the
voltage
> 
> Which of these steps are under software control on PowerNow?  
 
For this generation of parts, all steps are under active software
control except unpausing the CPU.

> Since we probably don't know how long we had to wait between when the
> multipler was increased and when the CPU was un-paused, it will be
> necessary to recalibrate the CPU to the global time source (PIT/HPET).

How is that performed?

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 23/1/08 22:34, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:

> Increasing the frequency requires (1) instructing the regulator to
> increasing the voltage, (2) waiting for it to settle, (3) increasing the
> multipler, (4) wait for the PLL to settle, (5) unpause the CPU.
> 
> Which of these steps are under software control on PowerNow?
> 
> Since we probably don't know how long we had to wait between when the
> multipler was increased and when the CPU was un-paused, it will be
> necessary to recalibrate the CPU to the global time source (PIT/HPET).

Yes, actually that is a good point. Possibly we would get better results by
setting t->stime_local_stamp = t->stime_master_stamp = read_platform_stime()
in xen/arch/x86/time.c:cpu_frequency_change(). Currently we set the local
stamp to get_s_time(), but get_s_time() may not be trustworthy.

 -- Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Mark Langsdorf]

[-- Attachment #1: Type: text/plain, Size: 2726 bytes --]

On Wednesday 23 January 2008, Keir Fraser wrote:
> On 23/1/08 22:34, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:
> 
> > Since we probably don't know how long we had to wait between when the
> > multipler was increased and when the CPU was un-paused, it will be
> > necessary to recalibrate the CPU to the global time source (PIT/HPET).
> 
> Yes, actually that is a good point. Possibly we would get better results by
> setting t->stime_local_stamp = t->stime_master_stamp = read_platform_stime()
> in xen/arch/x86/time.c:cpu_frequency_change(). Currently we set the local
> stamp to get_s_time(), but get_s_time() may not be trustworthy.

I tried this suggestion and also implemented the trace function in
cpu_frequency_change() and local_time_calibration() you suggested in
a previous message.

It made no discernable difference in my tests.  I did high to low 
and low to high changes roughly every five minutes, and always got
an excessive difference inside of Xen and usually got error messages
in Linux.

Changed code follows and the xm dmesg log is attached.
diff -r 62fc84adc8ed xen/arch/x86/time.c
--- a/xen/arch/x86/time.c	Fri Jan 18 13:43:26 2008 +0000
+++ b/xen/arch/x86/time.c	Fri Jan 25 16:45:03 2008 -0600
@@ -724,6 +724,20 @@ void update_domain_wallclock_time(struct
     spin_unlock(&wc_lock);
 }
 
+void checktime(char *buf);
+void checktime(char *buf)
+{
+        u64 s_tmp, p_tmp, diff;
+        s_tmp = get_s_time();
+        p_tmp = read_platform_stime();
+        if (s_tmp > p_tmp)
+                diff = s_tmp - p_tmp;
+        else
+                diff = p_tmp - s_tmp;
+        if (diff > 50000000)
+                printk(KERN_ERR "%s:time differed by %lu platform time %lu, systime %lu\n", buf, diff, p_tmp, s_tmp);
+}
+
 int cpu_frequency_change(u64 freq)
 {
     struct cpu_time *t = &this_cpu(cpu_time);
@@ -732,11 +746,14 @@ int cpu_frequency_change(u64 freq)
     local_irq_disable();
     rdtscll(curr_tsc);
     t->local_tsc_stamp = curr_tsc;
-    t->stime_local_stamp = get_s_time();
+//    t->stime_local_stamp = get_s_time();
     t->stime_master_stamp = read_platform_stime();
+    t->stime_local_stamp = t->stime_master_stamp;
     set_time_scale(&t->tsc_scale, freq);
+checktime("frequency_change");
     local_irq_enable();
 
+printk(KERN_ERR "freq change at %lu\n", t->stime_master_stamp);
     /* A full epoch should pass before we check for deviation. */
     set_timer(&t->calibration_timer, NOW() + EPOCH);
     if ( smp_processor_id() == 0 )
@@ -808,6 +825,7 @@ static void local_time_calibration(void 
     rdtscll(curr_tsc);
     curr_local_stime  = get_s_time();
     curr_master_stime = read_platform_stime();
+checktime("calibration");
     local_irq_enable();
 
 #if 0

[-- Attachment #2: xen_revf_time.log.gz --]
[-- Type: application/x-gzip, Size: 3118 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 25/1/08 22:58, "Mark Langsdorf" <mark.langsdorf@amd.com> wrote:

> I tried this suggestion and also implemented the trace function in
> cpu_frequency_change() and local_time_calibration() you suggested in
> a previous message.
> 
> It made no discernable difference in my tests.  I did high to low
> and low to high changes roughly every five minutes, and always got
> an excessive difference inside of Xen and usually got error messages
> in Linux.

It's a bit confusing as to what is going on since there are 4 CPUs in the
system. Are arbitrary or all CPUs changing frequency, or just one of them?
Is the single-step PSTATE_CTRL MSR being used, or the multi-step FID_VID
method where voltage adjustments are under software control? What is the
system's CPU topology, and do any of the cores have their frequency controls
tied together?

It'd be useful to have each tracing line stamped with the CPU that it comes
from. As it is the tracing looks very suspect because the systimes on
adjacent lines are much less than one second, and calibration is supposed to
happen no more than once per second.

 -- Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Xavian-Anderson Macpherson]

[-- Attachment #1.1: Type: text/plain, Size: 1508 bytes --]

Are we in fact talking about the new Barcelona chips? If so, their 
frequency settings are independent per core. Unless I have misunderstood 
the question.

Shingoshi



Keir Fraser wrote:
> On 25/1/08 22:58, "Mark Langsdorf" <mark.langsdorf@amd.com> wrote:
>
>   
>> I tried this suggestion and also implemented the trace function in
>> cpu_frequency_change() and local_time_calibration() you suggested in
>> a previous message.
>>
>> It made no discernable difference in my tests.  I did high to low
>> and low to high changes roughly every five minutes, and always got
>> an excessive difference inside of Xen and usually got error messages
>> in Linux.
>>     
>
> It's a bit confusing as to what is going on since there are 4 CPUs in the
> system. Are arbitrary or all CPUs changing frequency, or just one of them?
> Is the single-step PSTATE_CTRL MSR being used, or the multi-step FID_VID
> method where voltage adjustments are under software control? What is the
> system's CPU topology, and do any of the cores have their frequency controls
> tied together?
>
> It'd be useful to have each tracing line stamped with the CPU that it comes
> from. As it is the tracing looks very suspect because the systimes on
> adjacent lines are much less than one second, and calibration is supposed to
> happen no more than once per second.
>
>  -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>   

[-- Attachment #1.2.1: Type: text/html, Size: 2298 bytes --]

[-- Attachment #1.2.2: 175753_mail.png --]
[-- Type: image/png, Size: 7620 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Disable Xen PowerNow! support on Opteron2nd gen and earlier processors

[Ian Pratt]

> Are we in fact talking about the new Barcelona chips? If so, their
> frequency settings are independent per core. Unless I have
> misunderstood the question.

That shouldn't be an issue either way.

Ian

 
> Shingoshi
> 
> 
> 
> 
> 
> Keir Fraser wrote:
> 
> 	On 25/1/08 22:58, "Mark Langsdorf" <mark.langsdorf@amd.com>
> <mailto:mark.langsdorf@amd.com>  wrote:
> 
> 
> 
> 		I tried this suggestion and also implemented the trace
> function in
> 		cpu_frequency_change() and local_time_calibration() you
> suggested in
> 		a previous message.
> 
> 		It made no discernable difference in my tests.  I did
high
> to low
> 		and low to high changes roughly every five minutes, and
> always got
> 		an excessive difference inside of Xen and usually got
error
> messages
> 		in Linux.
> 
> 
> 
> 	It's a bit confusing as to what is going on since there are 4
> CPUs in the
> 	system. Are arbitrary or all CPUs changing frequency, or just
one
> of them?
> 	Is the single-step PSTATE_CTRL MSR being used, or the multi-step
> FID_VID
> 	method where voltage adjustments are under software control?
What
> is the
> 	system's CPU topology, and do any of the cores have their
> frequency controls
> 	tied together?
> 
> 	It'd be useful to have each tracing line stamped with the CPU
> that it comes
> 	from. As it is the tracing looks very suspect because the
> systimes on
> 	adjacent lines are much less than one second, and calibration is
> supposed to
> 	happen no more than once per second.
> 
> 	 -- Keir
> 
> 
> 
> 	_______________________________________________
> 	Xen-devel mailing list
> 	Xen-devel@lists.xensource.com
> 	http://lists.xensource.com/xen-devel
> 
> 

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

> > It made no discernable difference in my tests.  I did high to low
> > and low to high changes roughly every five minutes, and always got
> > an excessive difference inside of Xen and usually got error messages
> > in Linux.
> 
> It's a bit confusing as to what is going on since there are 4 
> CPUs in the system.

It's two dual-core processors.  Each dual core processor
shares frequency and voltage for both cores, so only
two transitions are being sent to Xen.

> Are arbitrary or all CPUs changing frequency, or just 
> one of them?

All processors are changing frequency.  There's probably
a delay in the command that changes the frequency, so cores 0,1
go at the same time, then 2,3 slightly later.

> Is the single-step PSTATE_CTRL MSR being used, or the 
> multi-step FID_VID method where voltage adjustments
> are under software control? 

Multistep.  Processors with single step also have pstate
invariant TSCs and don't seem to run into this issue.

> It'd be useful to have each tracing line stamped with the CPU 
> that it comes from.

See attached xm dmesg log in gzip'd form.  I cycled all processors
between highest and lowest frequency, with a change request
occuring for all processors every fifteen seconds.  The start 
of the sequence is marked in the log.

One thing that stands out is that although the dom0 notifier
event is firing for each core, Xen is running cpu_frequency_change
for cores 0 and 2.  The out of sync errors are occuring on
cores 1 and 3.  I'm looking into that now.

-Mark Langsdorf
Operating System Research Center
AMD

[-- Attachment #2: xen_revf_time-2.log.gz --]
[-- Type: application/x-gzip, Size: 3818 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> It'd be useful to have each tracing line stamped with the CPU 
> that it comes from. As it is the tracing looks very suspect
> because the systimes on adjacent lines are much less than
> one second, and calibration is supposed to happen no more
> than once per second.

Keir -

There seems to be a bug in continue_hypercall_on_cpu().
I'm getting vcpu_set_affinity() failures with error code
-22.  Any idea what that is?

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 26/1/08 00:54, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> It'd be useful to have each tracing line stamped with the CPU
>> that it comes from. As it is the tracing looks very suspect
>> because the systimes on adjacent lines are much less than
>> one second, and calibration is supposed to happen no more
>> than once per second.
> 
> Keir -
> 
> There seems to be a bug in continue_hypercall_on_cpu().
> I'm getting vcpu_set_affinity() failures with error code
> -22.  Any idea what that is?

Oh crap, of course:
 * cpufreq=dom0-kernel implies dom0_vcpus_pin
 * dom0_vcpus_pin stops dom0 from migrating its VCPUs among physical CPUs
 * The check is enforced in vcpu_set_affinity(), and unfortunately causes
continue_hypercall_on_cpu() to fail

That is such an obvious bug. :-(

vcpu_set_affinity() needs to not check dom0_vcpus_pin when called from
continue_hypercall_on_cpu(). I will cook up a patch...

 Thanks!
 Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 26/1/08 08:16, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> vcpu_set_affinity() needs to not check dom0_vcpus_pin when called from
> continue_hypercall_on_cpu(). I will cook up a patch...

Please test with latest xen-unstable. As of changeset 16897 I have fixed the
vcpu affinity change, so that should actually work, and also I have included
the modification to set local_stamp to master_stamp in
cpu_frequency_change() (since we do not trust TSC value immediately after a
frequency change).

I'm pretty hopeful these fixes will solve the nasty backwards deltas.

 Thanks,
 Keir

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> Please test with latest xen-unstable. As of changeset 16897 I 
> have fixed the vcpu affinity change, so that should actually
> work,

It doesn't, for some reason.  It seems to lock up on the

	v->arch.schedule_tail = continue_hypercall_on_cpu_helper

for no reason that I can tell.  If I comment out that line,
the system boots fine, but if that is in Xen, the machine
softlocks shortly after starting cpufreq.  Unfortunately,
cpufreq starts well before any console, so I can't get a
lot of information out the system.

I'll continue working on this, but any advance would be 
appreciated.

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

I'll try and kick that code path myself tomorrow. Since it's only triggered
by cpufreq and ACPI S3 I haven't really done any run testing so far! With a
bit of luck it's just a silly mistake.

 -- Keir

On 28/1/08 21:52, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> Please test with latest xen-unstable. As of changeset 16897 I
>> have fixed the vcpu affinity change, so that should actually
>> work,
> 
> It doesn't, for some reason.  It seems to lock up on the
> 
> v->arch.schedule_tail = continue_hypercall_on_cpu_helper
> 
> for no reason that I can tell.  If I comment out that line,
> the system boots fine, but if that is in Xen, the machine
> softlocks shortly after starting cpufreq.  Unfortunately,
> cpufreq starts well before any console, so I can't get a
> lot of information out the system.
> 
> I'll continue working on this, but any advance would be
> appreciated.
> 
> -Mark Langsdorf
> Operating System Research Center
> AMD
> 
> 

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 28/1/08 21:52, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> Please test with latest xen-unstable. As of changeset 16897 I
>> have fixed the vcpu affinity change, so that should actually
>> work,
> 
> It doesn't, for some reason.  It seems to lock up on the
> 
> v->arch.schedule_tail = continue_hypercall_on_cpu_helper
> 
> for no reason that I can tell.  If I comment out that line,
> the system boots fine, but if that is in Xen, the machine
> softlocks shortly after starting cpufreq.  Unfortunately,
> cpufreq starts well before any console, so I can't get a
> lot of information out the system.
> 
> I'll continue working on this, but any advance would be
> appreciated.

As promised I've instrumented and kicked that code path, and I execute it
with no problems, with the VCPU concerned being correctly migrated before
and after the continue_hypercall_on_cpu() is executed. I've tried it with
'dom0_vcpus_pin' and also I've tried making fairly extreme calls to
cpu_frequency_change() inside the hypercall-continuation 'helper' function.
None of this causes my dom0 boot to hang. So it looks like something a bit
subtle must be going wrong on your test system.

 -- Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 29/1/08 09:53, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> As promised I've instrumented and kicked that code path, and I execute it
> with no problems, with the VCPU concerned being correctly migrated before
> and after the continue_hypercall_on_cpu() is executed. I've tried it with
> 'dom0_vcpus_pin' and also I've tried making fairly extreme calls to
> cpu_frequency_change() inside the hypercall-continuation 'helper' function.
> None of this causes my dom0 boot to hang. So it looks like something a bit
> subtle must be going wrong on your test system.

Actually I've now eyeballed the problem. We access a stack variable after
the stack frame is gone away. Probably we therefore end up setting a CPU's
frequency to zero, which is not good at all. I'll work up a fix.

 -- Keir

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 29/1/08 10:32, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> On 29/1/08 09:53, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> As promised I've instrumented and kicked that code path, and I execute it
>> with no problems, with the VCPU concerned being correctly migrated before
>> and after the continue_hypercall_on_cpu() is executed. I've tried it with
>> 'dom0_vcpus_pin' and also I've tried making fairly extreme calls to
>> cpu_frequency_change() inside the hypercall-continuation 'helper' function.
>> None of this causes my dom0 boot to hang. So it looks like something a bit
>> subtle must be going wrong on your test system.
> 
> Actually I've now eyeballed the problem. We access a stack variable after
> the stack frame is gone away. Probably we therefore end up setting a CPU's
> frequency to zero, which is not good at all. I'll work up a fix.

Changeset 16927 should fix the problem for you.

 -- Keir

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> > Actually I've now eyeballed the problem. We access a stack 
> > variable after the stack frame is gone away. Probably we
> > therefore end up setting a CPU's frequency to zero, which
> > is not good at all. I'll work up a fix.
> 
> Changeset 16927 should fix the problem for you.

16927 seems to have resolved the problem.  I'll be running
a stress test in the background all day, but the prelimary
results show that:

a) the machine isn't locking up on boot any more
b) pstate transitions are possible without having the time
	go backwards on a regular basis

Ian was right about the design of Xen's time-keeping.

-Mark Langsdorf
Operating System Research Center
AMD

PS - Ralk, could you please test this on your machine?

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Ralf Schenk]

Langsdorf, Mark schrieb:
>>> Actually I've now eyeballed the problem. We access a stack 
>>> variable after the stack frame is gone away. Probably we
>>> therefore end up setting a CPU's frequency to zero, which
>>> is not good at all. I'll work up a fix.
>> Changeset 16927 should fix the problem for you.
> 
> 16927 seems to have resolved the problem.  I'll be running
> a stress test in the background all day, but the prelimary
> results show that:
> 
> a) the machine isn't locking up on boot any more
> b) pstate transitions are possible without having the time
> 	go backwards on a regular basis
> 
> Ian was right about the design of Xen's time-keeping.
> 
> -Mark Langsdorf
> Operating System Research Center
> AMD
> 
> PS - Ralk, could you please test this on your machine?

Hello,

thanks for your discussion and efforts to solve the Problems.

For your information: I'm running ondemand scheduler in Dom0 now with

(XEN) Xen version 3.2.1-rc1-pre (root@office.databay.de) (gcc version 
4.0.3 (Ubuntu 4.0.3-1ubuntu5)) Sat Feb  9 13:07:11 CET 2008
(XEN) Latest ChangeSet: Wed Feb 06 10:06:35 2008 +0000 16759:d26b1b777fde

since Saturday and I haven't had "Time went backwards" messages in my 
DomU's so far. Switching is working depending on the CPU Load in Domu as 
expected. But mostly the system stays at 1Ghz.

 From cpuinfo:
processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 107
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 5000+
stepping        : 2
cpu MHz         : 1000.000
cache size      : 512 KB

Bye
-- 
__________________________________________________

Ralf Schenk
fon (02 41) 9 91 21-0
fax (02 41) 9 91 21-59
rs@databay.de

Databay AG
Hüttenstraße 7
D-52068 Aachen
www.databay.de

Vorstand: Jens Conze, Ralf Schenk, Aresch Yavari
Aufsichtsratvorsitzender: Ansgar Vögeli

Sitz/Amtsgericht Aachen
HRB:8437
USt-IdNr.: DE 210844202

Databay - einfach machen.

_________________________________________________

Diese E-Mail und etwa angehängte Dateien enthalten vertrauliche
Informationen und sind ausschließlich für den Adressaten bestimmt.
Sollten Sie irrtümlich diese E-Mail erhalten haben, bitten wir Sie,
uns darüber unter info@databay.de zu informieren und die E-Mail
aus Ihrem System zu löschen.

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
info@databay.de. If you are not the named recipient, you should return
this message and delete it from your system.

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> Changeset 16927 should fix the problem for you.

I'm getting about 5 "Time went backwards" messages a day right
now, which is a drastic improvement but still more than I'd 
like.  In three instances, the shadow time value does not
increase.

The weird thing is the cpu time values are always of the form:
0: xxxxxx7444952
1: yyyyyy7447746
2: yyyyyy7447746
3: yyyyyy7447746

the y values are usually but not always the same as x or
slightly higher.  When they are different, they differ
by 1000000, 2000000, or 3000000.

I suspect it is related to the fact that platform calibration
is only performed if smp_processor_id() == 0, but I can't
figure out why this is occurring.

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

What are the deltas when time goes backwards? Bear in mind we already
default to being silent for backwards deltas up to 10ms. You could just
fudge that a bit bigger if the powernow module is loaded. But if the deltas
you see are much bigger than 10ms then yes, there may be an issue to be
resolved.

The issue here could well be that if you take a timer interrupt towards the
end of a frequency change but before Xen has been told that the frequency
has changed, the time values that Xen and dom0 calculate will be off by
potentially quite a bit. It's a small window of opportunity, but clearly you
hit it a few times each day. I'm not sure how easy it is to close off that
window cleanly with the MSR twiddling being controlled from dom0...

 -- Keir


On 1/2/08 16:39, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> Changeset 16927 should fix the problem for you.
> 
> I'm getting about 5 "Time went backwards" messages a day right
> now, which is a drastic improvement but still more than I'd
> like.  In three instances, the shadow time value does not
> increase.
> 
> The weird thing is the cpu time values are always of the form:
> 0: xxxxxx7444952
> 1: yyyyyy7447746
> 2: yyyyyy7447746
> 3: yyyyyy7447746
> 
> the y values are usually but not always the same as x or
> slightly higher.  When they are different, they differ
> by 1000000, 2000000, or 3000000.
> 
> I suspect it is related to the fact that platform calibration
> is only performed if smp_processor_id() == 0, but I can't
> figure out why this is occurring.
> 
> -Mark Langsdorf
> Operating System Research Center
> AMD
> 
> 

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> What are the deltas when time goes backwards? Bear in mind we already
> default to being silent for backwards deltas up to 10ms. 

Largest observed delta was 36 ms; it was a bit of an outlier.
Most deltas were around 20 ms.  I totaled 45 deltas over 10 ms
in 6 days with a frequency change every 15 seconds.

I think we can do better than that.  There's lots of <1 ms
regressions happening, roughly 1 every other 4 transitions.

> The issue here could well be that if you take a timer 
> interrupt towards the
> end of a frequency change but before Xen has been told that 
> the frequency
> has changed, the time values that Xen and dom0 calculate will 
> be off by
> potentially quite a bit. It's a small window of opportunity, 
> but clearly you
> hit it a few times each day. I'm not sure how easy it is to 
> close off that
> window cleanly with the MSR twiddling being controlled from dom0...

Would turning off interrupts solve it?  It might be a bit brute
force, but there's certainly a prechange notifier event and a 
postchange notifier event to use to turn them on and off.

Alternately, the prechange event could sleep until woken by
the timer interrupt, which would increase the chance that all 
frequency changes take place at the start of an interrupt.

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 4/2/08 21:52, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

> Would turning off interrupts solve it?  It might be a bit brute
> force, but there's certainly a prechange notifier event and a
> postchange notifier event to use to turn them on and off.
> 
> Alternately, the prechange event could sleep until woken by
> the timer interrupt, which would increase the chance that all
> frequency changes take place at the start of an interrupt.

Perhaps you could find out if the big deltas happen on a timer interrupt
that happens between pre-notification and post-notification? If so you could
indeed disable event delivery and that might well fix it.

 -- Keir

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Ian Pratt]

> I tried this suggestion and also implemented the trace function in
> cpu_frequency_change() and local_time_calibration() you suggested in a
> previous message.

Please provide the output of the tracing. Presumably the
time-goes-backwards messages only come out when stepping frequency down?
Presumably it is the CPU that is down-stepping that thinks time has gone
backwards? The messages comes out shortly after the downstep?

Once the multiplier change has happened, you need to create a new time
record for the CPU based on the global clock PIT/HPET (and update the
frequency info in the record).

Implemented correctly, this really ought to work just fine.

Ian

 
> It made no discernable difference in my tests.  I did high to low and
> low to high changes roughly every five minutes, and always got an
> excessive difference inside of Xen and usually got error messages in
> Linux.
> 
> Changed code follows and the xm dmesg log is attached.
> diff -r 62fc84adc8ed xen/arch/x86/time.c
> --- a/xen/arch/x86/time.c	Fri Jan 18 13:43:26 2008 +0000
> +++ b/xen/arch/x86/time.c	Fri Jan 25 16:45:03 2008 -0600
> @@ -724,6 +724,20 @@ void update_domain_wallclock_time(struct
>      spin_unlock(&wc_lock);
>  }
> 
> +void checktime(char *buf);
> +void checktime(char *buf)
> +{
> +        u64 s_tmp, p_tmp, diff;
> +        s_tmp = get_s_time();
> +        p_tmp = read_platform_stime();
> +        if (s_tmp > p_tmp)
> +                diff = s_tmp - p_tmp;
> +        else
> +                diff = p_tmp - s_tmp;
> +        if (diff > 50000000)
> +                printk(KERN_ERR "%s:time differed by %lu platform
time
> +%lu, systime %lu\n", buf, diff, p_tmp, s_tmp); }
> +
>  int cpu_frequency_change(u64 freq)
>  {
>      struct cpu_time *t = &this_cpu(cpu_time); @@ -732,11 +746,14 @@
> int cpu_frequency_change(u64 freq)
>      local_irq_disable();
>      rdtscll(curr_tsc);
>      t->local_tsc_stamp = curr_tsc;
> -    t->stime_local_stamp = get_s_time();
> +//    t->stime_local_stamp = get_s_time();
>      t->stime_master_stamp = read_platform_stime();
> +    t->stime_local_stamp = t->stime_master_stamp;
>      set_time_scale(&t->tsc_scale, freq);
> +checktime("frequency_change");
>      local_irq_enable();
> 
> +printk(KERN_ERR "freq change at %lu\n", t->stime_master_stamp);
>      /* A full epoch should pass before we check for deviation. */
>      set_timer(&t->calibration_timer, NOW() + EPOCH);
>      if ( smp_processor_id() == 0 )
> @@ -808,6 +825,7 @@ static void local_time_calibration(void
>      rdtscll(curr_tsc);
>      curr_local_stime  = get_s_time();
>      curr_master_stime = read_platform_stime();
> +checktime("calibration");
>      local_irq_enable();
> 
>  #if 0

RE: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Langsdorf, Mark]

> > I tried this suggestion and also implemented the trace function in
> > cpu_frequency_change() and local_time_calibration() you 
> > suggested in a previous message.
> 
> Please provide the output of the tracing.

I did, in the attached gzip'd xm dmesg log.

> Presumably the time-goes-backwards messages only come out when
> stepping frequency down?

The time-goes-backwards messages appear in Xen when 
stepping frequency down, and when the frequency goes up.

> Presumably it is the CPU that is down-stepping that thinks 
> time has gone backwards?

> The messages comes out shortly after the downstep?

Yes, but the Xen trace
 
> Once the multiplier change has happened, you need to create a new time
> record for the CPU based on the global clock PIT/HPET (and update the
> frequency info in the record).

I thought I was doing that already.  The code that is in
there now was written on Keir based on my original design.
 
> Implemented correctly, this really ought to work just fine.

We seem to be missing something in the correct implementation.

-Mark Langsdorf
Operating System Research Center
AMD

Re: [PATCH] Disable Xen PowerNow! support on Opteron 2nd gen and earlier processors

[Keir Fraser]

On 23/1/08 21:34, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> Are the 'time went backwards' messages correlated with power
>> management operations?
> 
> Yes, I can regularly generate messages by switching the governor
> from powersave (minimum frequency) to performance (maximum
> frequency).
> 
>> How far is time stepping backwards?
>> ns, us, or ms? 
>  
> Rolf's message is typical:
>> Jan 19 00:11:09 server kernel: printk: 14 messages suppressed.
>> Jan 19 00:11:09 server kernel: Timer ISR/0: Time went backwards:
>> delta=-123862858 delta_cpu=-123862858 shadow=
>> 2993571302800 off=705014593 processed=2994400167600

That's a big deviation. I suggest adding tracing to xen/arch/x86/time.c. In
cpu_frequency_change() and local_time_calibration(), where we get_s_time()
and read_platform_stime() it is worth checking if those two values differ by
more than a few milliseconds and print a (platform timestamped) message if
so. Perhaps we can work out if the deviation from wall time is occurring
when the cpu frequency change occurs, or is creeping up over the next epoch
or two (e.g., because the frequency notified to Xen by dom0 is inaccurate?).

 -- Keir