Re: can TSC tick with different speeds on SMP?

Re: can TSC tick with different speeds on SMP?

[Andi Kleen]

"Kirill Korotaev"  <kksx@mail.ru> writes:

> I've got some stupid question to SMP gurus and would be very
> thankful for the details. I suddenly faced an SMP system where
> different P4 cpus were installed (with different steppings). This
> resulted in different CPU clock speeds and different speeds of time
> stamp counters on these CPUs. I faced the problem during some
> timings I measured in the kernel.

Yes, it is a quite common problem. You'll have to deal with it
somehow.  Somehow it can be fixed by disabling "cross spectrum
clocking" or similar in the BIOS setup, but on others it 
is unfixable.

Ultimatively the kernel will need to move to a per CPU time base
to deal with this better, but that is a future project.

For user space I would suggest to use gettimeofday() or better
clock_gettime(CLOCK_MONOTONIC, ...) in 2.6 to get time stamps
and let the kernel deal with it.

>
> So the question is "is such system compliant with SMP
> specification?".  In old kernels there was a code to syncronize TSCs
> and to detect if they were screwed up. Current kernels do not have
> such code. Is it intentional? I suppose there is some code in kernel
> which won't work find on such systems (real-time threads timing
> accounting and so on).

The usual way to is to turn off tsc support in gettimeofday.  This can
be done by booting with "notsc" That should be the only code relying
on it. The scheduler did in some 2.6 versions, but that has been also
fixed (it can tolerate non monotonous times now). Other code
who reads the TSC directly has to deal with it.

e.g. x86-64 just forces HPET gettimeofday when SMP is on, which slows
down gettimeofday greatly.

-Andi

can TSC tick with different speeds on SMP?

["Kirill Korotaev"]

Hello,

I've got some stupid question to SMP gurus and would be very thankful for the details. I suddenly faced an SMP system where different P4 cpus were installed (with different steppings). This resulted in different CPU clock speeds and different speeds of time stamp counters on these CPUs. I faced the problem during some timings I measured in the kernel.

So the question is "is such system compliant with SMP specification?".
In old kernels there was a code to syncronize TSCs and to detect if they were screwed up. Current kernels do not have such code. Is it intentional? I suppose there is some code in kernel which won't work find on such systems (real-time threads timing accounting and so on).

Thanks in advance,
Kirill

Re: can TSC tick with different speeds on SMP?

[Pasi Savolainen]

* Kirill Korotaev <kksx@mail.ru>:
> Hello,
>
> I've got some stupid question to SMP gurus and would be very thankful
> for the details. I suddenly faced an SMP system where different P4 cpus
> were installed (with different steppings). This resulted in different
> CPU clock speeds and different speeds of time stamp counters on these
> CPUs. I faced the problem during some timings I measured in the kernel.
>   
> So the question is "is such system compliant with SMP specification?".

Well, I can't tell if it's SMP-compliant, but I can tell that it's
certainly not rare to have such a situation.

For example on AMD K7-SMP system when using powersaving mode which
basically 'turns CPU off' for a while, TSC's become desynchronized. At
the moment 2.6 -kernels handle this very well. (I haven't got problems
with it for more than half a year).

Now, I don't know what will happen if the become desynchronized because
of totally different _length_ of single tick. Is this what you
experience?

FWIW, this is what I get for timer in dmesg:
- -
 ...
Using pmtmr for high-res timesource
 ...
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1545.0689 MHz.
..... host bus clock speed is 268.0815 MHz.
checking TSC synchronization across 2 CPUs: passed.
 ...
- -


-- 
   Psi -- <http://www.iki.fi/pasi.savolainen>

Re: can TSC tick with different speeds on SMP?

[Grzegorz Kulewski]

On Mon, 21 Jun 2004, [koi8-r] "Kirill Korotaev[koi8-r] "  wrote:

> Hello,
> 
> I've got some stupid question to SMP gurus and would be very thankful for the details. I suddenly faced an SMP system where different P4 cpus were installed (with different steppings). This resulted in different CPU clock speeds and different speeds of time stamp counters on these CPUs. I faced the problem during some timings I measured in the kernel.

IANSG (= I am not SMP guru).

> So the question is "is such system compliant with SMP specification?".
> In old kernels there was a code to syncronize TSCs and to detect if they were screwed up. Current kernels do not have such code. Is it intentional? I suppose there is some code in kernel which won't work find on such systems (real-time threads timing accounting and so on).

I have friend. He has SMP system with 2x P III. He put two different CPUs 
into it (on 700 and one 750). This resulted in fast growing difference 
between time on them. This can be potentially dangerous to databases or 
other time driven programs. But there is a fix: boot with notsc kernel 
parameter. But there is one problem. Everyting is ok as long as you have 
i386 (that means NOT i686) compiled glibc. If you have not, init will stop 
with segmentation fault or something like that. But we are using Gentoo, 
so his glibc was i686 compiled so not working. He do not wanted to 
recompile it to i386 so I fixed glibc to work with notsc and i686. But I 
will not provide any patch because I fixed it against some CVS version 
that Gentoo used at this time. Maybe he has more recent patch. I will ask 
him and I will post patch if he has any. But the change is very simple. 
You should grep glibc's sources against tsc related assembly instructions 
and then kill them in the smart way. I found two places. One was in some 
malloc debuging code and the other was in header. The header is simple to 
fix - you should overwrite it with its no-op version from i386 (or 
something like that) directory. And you should disable the second call in 
the malloc debuging code (for example by setting define that you have no 
precise timing source in your machine). Then you should recompile your 
glibc and you are done. My friend's server has something about 2 moths 
uptime and no clock problems so this is probably The Right Fix (TM). I am 
writing this words using it so I know...


Grzegorz Kulewski

Re: can TSC tick with different speeds on SMP?

[James Cleverdon]

IIRC, in the IA64 manuals Intel, by carefully not making any guarantees 
to the contrary, reserved the right to have the TSC-equivalent register 
not be synchronized either to the bus clock or the CPU clock.

This doesn't directly apply to IA32, but may give a hint as to their 
future intentions.


On Monday 21 June 2004 12:02 pm, Kirill Korotaev wrote:
> Hello,
>
> I've got some stupid question to SMP gurus and would be very thankful
> for the details. I suddenly faced an SMP system where different P4
> cpus were installed (with different steppings). This resulted in
> different CPU clock speeds and different speeds of time stamp
> counters on these CPUs. I faced the problem during some timings I
> measured in the kernel.
>
> So the question is "is such system compliant with SMP
> specification?". In old kernels there was a code to syncronize TSCs
> and to detect if they were screwed up. Current kernels do not have
> such code. Is it intentional? I suppose there is some code in kernel
> which won't work find on such systems (real-time threads timing
> accounting and so on).
>
> Thanks in advance,
> Kirill
>
> -
>

-- 
James Cleverdon
IBM LTC (xSeries Linux Solutions)
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot comm

Re: can TSC tick with different speeds on SMP?

[Zwane Mwaikambo]

On Mon, 21 Jun 2004, James Cleverdon wrote:

> IIRC, in the IA64 manuals Intel, by carefully not making any guarantees
> to the contrary, reserved the right to have the TSC-equivalent register
> not be synchronized either to the bus clock or the CPU clock.
>
> This doesn't directly apply to IA32, but may give a hint as to their
> future intentions.

The intel MP1.4 specification also allows for processors of
varying capabilities, this would include different clock speeds resulting
in differing TSC frequencies.

Re: can TSC tick with different speeds on SMP?

[john stultz]

On Mon, 2004-06-21 at 12:02, =?koi8-r?Q?=22?=Kirill
Korotaev=?koi8-r?Q?=22=20?= wrote:
> I've got some stupid question to SMP gurus and would be very thankful
> for the details. I suddenly faced an SMP system where different P4
> cpus were installed (with different steppings). This resulted in
> different CPU clock speeds and different speeds of time stamp counters
> on these CPUs. I faced the problem during some timings I measured in
> the kernel.
> So the question is "is such system compliant with SMP specification?".

Linux doesn't really support this configuration at the moment.
Historically Intel hasn't supported different stepping cpus in the same
system, much less different frequency cpus. I know the TSC time source
code won't handle it properly, and there are other users of rdtsc that
might cause problems. Other concerns would be users of cpu_khz
(scheduler?) and cpufreq code. 

I've heard that AMD has introduced this as a possible configuration, and
Intel may have changed their support story as well, so this may need to
be addressed. 

> In old kernels there was a code to syncronize TSCs and to detect if
> they were screwed up. Current kernels do not have such code. Is it
> intentional? I suppose there is some code in kernel which won't work
> find on such systems (real-time threads timing accounting and so on).

The TSC syncing code is still there, but it runs only once at boot. Thus
it will sync the cpus, but then won't account for the TSC drift between
them.

The only way to get around this is to use an alternate time source, such
as the ACPI-PM time source (clock=pmtmr), or even the PIT (clock=pit).
ACPI-PM still uses the TSC for delay() so it may still have issues, so
be warned.

thanks
-john