Re: [RFC] synchronized CPU count registers on SMP machines

[Jun Sun <jsun@mvista.com>]

On Wed, Jun 04, 2003 at 03:51:49PM -0700, Michael Uhler wrote:
<snip>
> > 
> > Apparently, this scheme won't work if any of the following 
> > conditions are true:
> > 
> > 1) clocks on different CPUs don't have the same frequency
> > 2) clocks on different CPUs drift to each other
> 
> Depending on the precise system configuration (including whether the
> CPUs are on different SOCs, different boards, where the PLLs are, and
> what the ultimate clock source is), I'd guess that drift is pretty
> likely on almost all systems unless the clocks are intentionally driven
> by some sort of synchronize source.
>

Yes, if there are physically multiple boards, it is likely to have
drifting clocks.

I don't suppose the count synchronization code will work for
all SMP systems.  I hope to get an idea whether it is worth the effort
for the systems that it does work.

For example, it will work on Broadcom's BCM91250 chip and probably a
couple of coming ones.

To give an idea of my original motivation on this RFC, I found out
it is pretty _hard_ just to get a micro-second resolution gettimeofday()
on a SMP system.  And apparently this is an issue on other arches
as well.  I think i386 uses synchronized TSCs, which is the same
idea as what we are talking about here.

> The biggest problem here is latency on the spinlocks and observation of
> the flag state changing.  Depending on the memory architecture, the
> point at which each CPU will see the change could be very different
> (consider a NUMA mesh architecture in which the data movement can take
> different paths).  As such, you could see on order of memory latency
> different in the clocks.
>

If it is a few clocks in difference, it should be acceptable.

Basically, imagine a spin lock protected read_count routine:

unsigned read_count() 
{
	spin_lock(&count_lock);
	count = read_c0_count();
	spin_unlock(&count_lock);
	return count;
}

and imagine there are randomly calls to this function by different
CPUs, the calls are serialized due to the spinlock.  As long as
the return values are monotonically increasing, we are fine.

Thanks for the reply.  And congrads on your new post. :)

Jun