Re: 2.6.14-rc4-rt7

[Ingo Molnar <mingo@elte.hu>]

John

i found one source of timekeeping bugs on SMP boxes, it's the 
non-monotonicity of the TSC:

... time warped from 1270809453 to 1270808096.
... MTSC warped from 0000000a731a8c3c [0] to 0000000a731a899c [2].
... MTSC warped from 0000000a7c93baec [0] to 0000000a7c93b7a8 [3].
... MTSC warped from 0000000a881d6afc [0] to 0000000a881d67d0 [2].
... MTSC warped from 0000000a924217a0 [0] to 0000000a924216ac [3].
... MTSC warped from 0000000a9c592788 [0] to 0000000a9c59232c [2].
... MTSC warped from 0000000aa7aa95c8 [0] to 0000000aa7aa9338 [3].
... MTSC warped from 0000000b33206d60 [0] to 0000000b33206a48 [3].
... time warped from 26699635824 to 26699633144.
... MTSC warped from 00000013f379cb88 [0] to 00000013f379c7e0 [3].
... MTSC warped from 0000001413df8660 [0] to 0000001413df8200 [3].
... MTSC warped from 00000014194f5360 [1] to 00000014194f51b0 [2].
... time warped from 60775269225 to 60775266727.

the number in square brackets is the CPU#. I.e. CPUs on this 4-CPU box 
have small TSC differences, which ends up leaking into the generic TOD 
code, causing real time warps, which causes ktimer weirdnesses (timers 
failed to expire, etc.).

(the above output tracks TSC results globally, under a spinlock. It also 
detects time-warps that propagate into the monotonic clock output.)

unfortunately, there's no easy solution for this. We could make 
cycle_last per-CPU, but that again brings up the question of how to set 
up the per-CPU 'TSC offset' values - those would need similar technique 
that the current clear-all-TSCs-on-all-CPUs code does - which as we can 
see failed ...

	Ingo