RE: Xen system skew MUCH worse than tsc skew (was RE: RE: [PATCH] record max stime skew (was RE: [PATCH] strictly increasing hvm guest time))

["Dan Magenheimer" <dan.magenheimer@oracle.com>]

> > Would you expect system load to impact stime skew between
> > processors (using hpet as a system timer)?  I can repeatably
> > watch skew get worse when I am launching an hvm domain.  It is
> > MUCH worse when the new domain is in its early stages of booting.
> > CPU load on domain0 has little or no impact but I/O load
> > on dom0 seems to make skew get worse.
> 
> Perhaps it makes a difference if it takes each CPU a bit 
> longer to execute
> the calibration function in softirq context? That could be 
> delayed by long
> hypercalls, for example (although long hypercalls should mostly be
> preemptible).

I'm not positive yet, but I think I have an explanation for
this.  The issue is not HOW LONG it takes to execute the
calibration function but WHEN relative to other processors
the calibration function executes.  If jitter on the platform
timer occurs and the (e.g. two) calibration functions are triggered
"temporally maximally distant" (e.g. cpu0 at 1.0, 2.0, 3.0
and cpu1 at 1.5, 2.5, 3.5), their differing slope during the
interim partial-second could result in greater skew.  Since activity
on a processor will result in different locks held, interrupts
on/off, etc, system load differences between processors is more
likely to cause distance to vary between the scheduled calibration
functions on each processor.

(Worse, could maximal distance maybe result in harmonic
resonance?  The fact that I can observe the effect seems to
imply that it stays bad for awhile.)

This is all still theoretical... I still have to figure out how to
measure this.  But does the theory make sense?

Perhaps some form of the proposed "deferrable timers" can
be used to ensure per-cpu calibration happens on different
processors at roughly the same moment?

Thanks,
Dan