Re: do_gettimeofday vs. rdtsc in the scheduler

[James Cleverdon <jamesclv@us.ibm.com>]

On Tuesday 17 September 2002 02:18 pm, David S. Miller wrote:
>    From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>    Date: 17 Sep 2002 22:28:12 +0100
>
>    A bus clock - but things like the x440 have more than one bus clock. Its
>    NUMA. Also the bus clock and rdtsc clock are different - rdtsc is
>    dependant on the multiplier. Shove a celeron 300 and a celeron 450 in a
>    BP6 board with tsc on and enjoy
>
> That's mostly my point.
>
> If the bus clocks differ, then great create some system wide crystal
> oscillator.  That's a detail, the important bit is that you don't need
> to go out to the system bus to read the tick value, it must be cpu
> local to be effective and without serious performance impact.
> -

It's more than just a detail.  Sequent's last NUMA system (_not_ the NUMA-Q;  
never released) did exactly what you suggest.  The midplane card generated 
the bus clock for all quad modules.  We had requested this feature because it 
was such a pain dealing with clock drift between nodes in the OS.

The HW guys were able to give us synchronized bus clocks on a 16-way box, but 
warned us that it would not be practical on the 256-way.  Too much clock skew 
at those speeds, or something like that.  I suppose you could trade off 
interconnect rate for clock sync, but then performance would suffer.

I don't know how Sun and SGI manage with their larger systems.  Either they 
don't do clock sync, or they may have to make expensive tradeoffs.

Interestingly, Intel's IA64 manual does not guarantee that the CPU clock (and 
thus its TSC register) has anything to do with the bus clock rate.  Maybe 
they want to dabble with asynchronous logic or multiple clock domains in 
future CPUs.

Trivia:  NUMA-Q systems running Dynix/PTX can contain quads running at very 
different CPU speeds.  This made locating some race conditions quite easy.

-- 
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com