From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: [RFC] Physical hot-add cpus and TSC Date: Fri, 28 May 2010 13:24:50 -0700 (PDT) Message-ID: <2d0b4e25-6fa9-4d66-9efe-a1b9e27612f5@default> References: <789F9655DD1B8F43B48D77C5D30659731E78D370@shsmsx501.ccr.corp.intel.com> <789F9655DD1B8F43B48D77C5D30659731E78D500@shsmsx501.ccr.corp.intel.com> <3c99c55d-68ce-4150-b895-72fda1ff3b89@default> <789F9655DD1B8F43B48D77C5D30659731E78D89D@shsmsx501.ccr.corp.intel.com> <26342d1d-2141-4fb1-94ac-a398d7f553d6@default 789F9655DD1B8F43B48D77C5D30659731E78DA70@shsmsx501.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <789F9655DD1B8F43B48D77C5D30659731E78DA70@shsmsx501.ccr.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Jiang, Yunhong" , Keir Fraser , "Xen-Devel (xen-devel@lists.xensource.com)" , Ian Pratt List-Id: xen-devel@lists.xenproject.org > >> b) With the patch: > >> After add: > >> (XEN) TSC marked as reliable, warp =3D 407 (count=3D12) > >> (XEN) TSC marked as reliable, warp =3D 444 (count=3D17) > >> (XEN) TSC marked as reliable, warp =3D 525 (count=3D19) > > > >Hi Yunhong -- > > > >Does this continue to grow? I'm concerned that > >the hot-added CPU might be skewing as well? > >I didn't think this was possible with an Invariant > >TSC machine but maybe something in the hot-add > >isolation electronics changes the characteristics > >of the clock signal? > >Please try: > > > >for i in {0..1000}; do xm debug-key t; sleep 3; done; \ > >xm dmesg | tail > > > >then wait an hour, and see how large the warp is. > >Hopefully the trend (407,444,525) is a coincidence. >=20 > I just looped 10 times, I will try with 1000 loops next Monday. > I suspect there are any isolation electronics for hot-add. Basically > each CPU can be hot-added except socket 0. But yes, I can have a look > on it. Hmmm... I'm not a system hardware expert, but the more I think about this, the more likely it seems that any hot-plug board must have separate QPI buses that are driven by separate crystals. And there would be some kind of bus bridge/repeater to connect the two with a forwarding protocol. That would certainly explain a growing TSC skew. If so, even two single socket boards connected like that at initial boot (no hot-add) are really a "big NUMA" system due to higher cross-node latencies and might deserve a separate boot option anyway... this is really NOT a single system... it is multiple systems glued together with a fast interconnect. Xen (and users) should be warned that there is no free lunch here and the performance degradation from TSC emulation may be only a small part of the problem. Some boot option like "multiboard_interconnect" (but shorter) might be appropriate? Or is there some way at boot-time to determine that this box does, or might (via hot-add), or definitely does not, go beyond point-to-point interconnect? A boot-time decision on TSC emulation could be driven off of that if it existed.