x86 Physical CPUs at different frequencies - timer error messages

x86 Physical CPUs at different frequencies - timer error messages

[Langsdorf, Mark]

I'm experimenting with getting PowerNow! running on
SMP Opteron systems.  

One problem I'm having is that Xen assumes that all
physical processors are incrementing the TSCs at 
the same rate, and complains in the timer ISR that
time is going backward if this isn't the case:

"Timer ISR/1: Time went backwards: delta=BIGNUM1
delta_cpu=BIGNUM2 shadow=BIGNUM3 off=BIGNUM4
processed=BIGNUM5 cpu_processed=BIGNUM6
 0: BIGNUM5
 1: BIGNUM6"

I'm not surprised this is happening, since the TSC
values on processor 0 and processor 1 increment at
different rates and tend to have wildly differing
values.  Normally AMD strongly discourages using 
TSC for time-keeping with PowerNow! enabled for 
exactly this reason.

However, the worst effects of time skew that we
see on native systems is not happening on my dom0
with PowerNow!.  How much should I worry about this
and what can be done to mitigate it?

-Mark Langsdorf
AMD, Inc.

RE: x86 Physical CPUs at different frequencies - timer error messages

[Ian Pratt]

> I'm experimenting with getting PowerNow! running on
> SMP Opteron systems.
> 
> One problem I'm having is that Xen assumes that all
> physical processors are incrementing the TSCs at
> the same rate, and complains in the timer ISR that
> time is going backward if this isn't the case:

Xen *is* able to cope with TSCs running at different frequencies. It
calibrates each CPU every few seconds on the assumption that the
frequencies are basically stable but different. 

To integrate with PowerNow, after you've changed the frequency you need
to run the code to generate a new time record and propagate it to the
VCPU of the guest currently using the CPU. For the frequency field, you
are best off scaling the frequency measured in the last calibration
period rather than assuming the new target frequency is precise.

Ian

> "Timer ISR/1: Time went backwards: delta=BIGNUM1
> delta_cpu=BIGNUM2 shadow=BIGNUM3 off=BIGNUM4
> processed=BIGNUM5 cpu_processed=BIGNUM6
>  0: BIGNUM5
>  1: BIGNUM6"
> 
> I'm not surprised this is happening, since the TSC
> values on processor 0 and processor 1 increment at
> different rates and tend to have wildly differing
> values.  Normally AMD strongly discourages using
> TSC for time-keeping with PowerNow! enabled for
> exactly this reason.
> 
> However, the worst effects of time skew that we
> see on native systems is not happening on my dom0
> with PowerNow!.  How much should I worry about this
> and what can be done to mitigate it?
> 
> -Mark Langsdorf
> AMD, Inc.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: x86 Physical CPUs at different frequencies - timer error messages

[Langsdorf, Mark]

> > I'm experimenting with getting PowerNow! running on
> > SMP Opteron systems.
> > 
> > One problem I'm having is that Xen assumes that all
> > physical processors are incrementing the TSCs at
> > the same rate, and complains in the timer ISR that
> > time is going backward if this isn't the case:
> 
> Xen *is* able to cope with TSCs running at different frequencies. It
> calibrates each CPU every few seconds on the assumption that the
> frequencies are basically stable but different. 
> 
> To integrate with PowerNow, after you've changed the 
> frequency you need to run the code to generate a new
> time record and propagate it to the VCPU of the guest
> currently using the CPU. For the frequency field, you
> are best off scaling the frequency measured in the last
> calibration period rather than assuming the new target
> frequency is precise.

Could you give some rough hints on how to do that?  
Nothing in time-xen.c or the published hypercalls looks
more than vaguely related.  All the functions and
hypercalls deal with absolute offsets.

The only reference to a frequency field is in 
vcpu_time_info, but no functions take that as an 
argument.  vcpu_info doesn't have anything either.
What functions calls/hypercalls should I be dealing
with?

-Mark Langsdorf
Operating System Research Center
AMD

Re: x86 Physical CPUs at different frequencies - timer error messages

[Mark Langsdorf]

> Xen *is* able to cope with TSCs running at different frequencies. It
> calibrates each CPU every few seconds on the assumption that the
> frequencies are basically stable but different. 
> 
> To integrate with PowerNow, after you've changed the 
> frequency you need to run the code to generate a new
> time record and propagate it to the VCPU of the guest
> currently using the CPU. For the frequency field, you
> are best off scaling the frequency measured in the last
> calibration period rather than assuming the new target
> frequency is precise.

Is something like this what you meant?  I'm still
getting a half dozen Time error messages every time
I increase frequency.

diff -r a70de77dd8d3 arch/i386/kernel/time-xen.c
--- a/arch/i386/kernel/time-xen.c	Tue Jul 10 10:18:24 2007 +0100
+++ b/arch/i386/kernel/time-xen.c	Thu Jul 26 15:21:38 2007 -0500
@@ -50,6 +50,7 @@
 #include <linux/percpu.h>
 #include <linux/kernel_stat.h>
 #include <linux/posix-timers.h>
+#include <linux/cpufreq.h>
 
 #include <asm/io.h>
 #include <asm/smp.h>
@@ -1118,6 +1119,95 @@ void local_teardown_timer(unsigned int c
 	BUG_ON(cpu == 0);
 	unbind_from_irqhandler(per_cpu(timer_irq, cpu), NULL);
 }
+#endif
+
+#if CONFIG_CPU_FREQ
+/* 
+ * cpufreq scaling handling
+ */
+static int time_cpufreq_notifier(struct notifier_block *nb, unsigned long val, 
+				void *data)
+{
+	struct cpufreq_freqs *freq = data;
+        struct vcpu_time_info *info = &vcpu_info(freq->cpu)->time;
+	u64 new_mult;
+	char new_shift;
+
+	if (cpu_has(&cpu_data[freq->cpu], X86_FEATURE_CONSTANT_TSC))
+		return 0;
+
+	if (val == CPUFREQ_PRECHANGE)
+		return 0;
+
+	/* re order time per Ian's instructions */
+	{
+		u64 __cpu_khz = info->tsc_to_system_mul;
+		do_div(__cpu_khz, freq->new / 1000);
+		new_mult = __cpu_khz * freq->old / 1000;	
+		new_shift = info->tsc_shift;
+		while (new_mult > (1LL <<32))  {
+			new_shift += 1;
+			new_mult = new_mult >> 1;
+		}
+		while (new_mult < (1LL << 31)) {
+			new_shift -= 1;
+			new_mult = new_mult << 1;
+		}
+	}
+	do {
+		rmb();
+		info->tsc_to_system_mul = new_mult;
+		info->tsc_shift = new_shift;
+		rmb();
+	} while (info->version & 1);
+	get_time_values_from_xen(freq->cpu);
+
+	return 0;
+}
+
+static struct notifier_block time_cpufreq_notifier_block = {
+	.notifier_call = time_cpufreq_notifier
+};
+
+static int __init cpufreq_time_setup(void)
+{
+	if (!cpufreq_register_notifier(&time_cpufreq_notifier_block,
+			CPUFREQ_TRANSITION_NOTIFIER)) {
+		printk(KERN_ERR "failed to set up cpufreq notifier\n");
+		return -ENODEV;
+	}
+	return 0;
+}
+
+core_initcall(cpufreq_time_setup);
 #endif
 
 /*

Re: x86 Physical CPUs at different frequencies - timer error messages

[Keir Fraser]

On 26/7/07 22:00, "Mark Langsdorf" <mark.langsdorf@amd.com> wrote:

> Is something like this what you meant?  I'm still
> getting a half dozen Time error messages every time
> I increase frequency.

No. The code belongs in Xen itself. Otherwise no other domains will pick up
the new scale factor, and even dom0 itself may lose it as soon as it
re-picks up the scale information from Xen.

 -- Keir

RE: x86 Physical CPUs at different frequencies - timer error messages

[Langsdorf, Mark]

> > Is something like this what you meant?  I'm still
> > getting a half dozen Time error messages every time
> > I increase frequency.
> 
> No. The code belongs in Xen itself. Otherwise no other 
> domains will pick up the new scale factor, and even
> dom0 itself may lose it as soon as it
> re-picks up the scale information from Xen.

Oops.

I take it there isn't currently a hypercall to 
support changing it in Xen, though?  Shouldn't
be too hard to add, but I want to confirm.

-Mark Langsdorf
Operating System Research Center
AMD

Re: x86 Physical CPUs at different frequencies - timer error messages

[Keir Fraser]

On 27/7/07 15:30, "Langsdorf, Mark" <mark.langsdorf@amd.com> wrote:

>> No. The code belongs in Xen itself. Otherwise no other
>> domains will pick up the new scale factor, and even
>> dom0 itself may lose it as soon as it
>> re-picks up the scale information from Xen.
> 
> Oops.
> 
> I take it there isn't currently a hypercall to
> support changing it in Xen, though?  Shouldn't
> be too hard to add, but I want to confirm.

Correct. Makes sense to add a platform_op for this.

 -- Keir

question about blk-attach

[tgh]

hi
I  have  attach a disk onto the vm
and now I try to understand the code relative to blk-attach
it seems like that device_creat()->createdevice()->addStoreEntries() 
,add store entry into xenstore , then backend and frontend will watch 
and do several related  work  to real add-device in backend and 
frontend  ,is it  right?  and  xenstore have watches to check the 
related entry ,is it right?
I am confused  ,which code or function to the real adding work, I guess 
it is the watches do it ,but I cannot find that fucntions

could you help me
Thanks in advance

question about dynamically add block device

[tgh]

hi
I try to understand the code relative to dynamically add block device in 
a running VM
it seems like that device_creat()->createdevice()->addStoreEntries() 
,add store entry into xenstore , then backend and frontend will watch 
and do several related  work  to real add-device in backend and 
frontend  ,is it  right?  and  xenstore have watches to check the 
related entry ,is it right?
I am confused  ,which code or function to the real adding work, I guess 
it is the watches do it ,but I cannot find that fucntions

could you help me
Thanks in advance

confusion about blk-attach

[tgh]

hi
I try to understand the code relative to blk-attach
it seems like that device_creat()->createdevice()->addStoreEntries() 
,add store entry into xenstore
then backend and frontend will watch and do several related  work  to 
real add-device in backend and frontend  ,is it  right?  and  xenstore 
have watches to check the related entry ,is it right?
I am confused  ,which code or function to the real adding work
I guess there is somekind watches which  do it ,but I cannot find that 
fucntions

I am confused about it

could you help me
Thanks in advance