RE: [RFC] Physical hot-add cpus and TSC

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> >> b) With the patch:
> >> After add:
> >> (XEN) TSC marked as reliable, warp = 407 (count=12)
> >> (XEN) TSC marked as reliable, warp = 444 (count=17)
> >> (XEN) TSC marked as reliable, warp = 525 (count=19)
> >
> >Hi Yunhong --
> >
> >Does this continue to grow?  I'm concerned that
> >the hot-added CPU might be skewing as well?
> >I didn't think this was possible with an Invariant
> >TSC machine but maybe something in the hot-add
> >isolation electronics changes the characteristics
> >of the clock signal?
> >Please try:
> >
> >for i in {0..1000}; do xm debug-key t; sleep 3; done; \
> >xm dmesg | tail
> >
> >then wait an hour, and see how large the warp is.
> >Hopefully the trend (407,444,525) is a coincidence.
> 
> I just looped 10 times, I will try with 1000 loops next Monday.
> I suspect there are any isolation electronics for hot-add. Basically
> each CPU can be hot-added except socket 0. But yes, I can have a look
> on it.

Hmmm... I'm not a system hardware expert, but the more I think
about this, the more likely it seems that any hot-plug
board must have separate QPI buses that are driven by separate
crystals.  And there would be some kind of bus bridge/repeater
to connect the two with a forwarding protocol.  That would
certainly explain a growing TSC skew.

If so, even two single socket boards connected like that at
initial boot (no hot-add) are really a "big NUMA" system due
to higher cross-node latencies and might deserve a separate
boot option anyway... this is really NOT a single system...
it is multiple systems glued together with a fast interconnect.
Xen (and users) should be warned that there is no free
lunch here and the performance degradation from TSC emulation
may be only a small part of the problem.

Some boot option like "multiboard_interconnect" (but shorter)
might be appropriate?  Or is there some way at boot-time
to determine that this box does, or might (via hot-add), or
definitely does not, go beyond point-to-point interconnect?
A boot-time decision on TSC emulation could be driven off
of that if it existed.