Please send Xen boot output (xm dmesg). Get= ting it from Xen 3.2 as well
would be interesting, if you still have it installed on any of these
machines.

=A0-- Keir

On 23/02/2011 19:04, "Olivier Hanesse" <olivier.hanesse@gmail.com> wrote:

> I am sorry for the lack of information.
> Every domUs on the dom0 are affected by this bug at the exact same tim= e.
>
> And I had this bug on a dozen servers (all running on the same hw) sin= ce
> October (when I switched from Xen 3.2 to 4.0).
>
> Regards
>
> Olivier
>
> Le 23/02/2011 18:19, Keir Fraser a =E9crit :
>> On 23/02/2011 16:16, "Dan Magenheimer"<dan.magenheimer@oracle.com> =A0wrote= :
>>

>>> It=B9s very unlikely this is a problem with TSC. It is m= ost likely a Xen (or

>>> possibly a PV Linux) problem where a guest (= or dom0) either =B3goes out to

>>> lunch=B2 for a long period, or some other timer gets stu= ck. =A0The =B3clocksource
>>> tsc unstable=B2 message is a side effect of this... it=B9s ver= y likely the TSC

>>> that IS stable and correct and the other clo= cksource (pvclock) has
>>> lost/gained
>>> 50 minutes!
>>>

>>> Mark Adams cc=B9ed and his original xen-devel posting be= low. =A0The fact that

>>> two
>>> different users (possibly on the same processor/system type?) = have submitted
>>> the message with a delta so similar would lead me to believe t= here is some

>>> timer that is =B3wrapping=B2. =A0And since pvclock is us= ually the clocksource for
>>> dom0, and pvclock is driven! =A0by Xen=B9s =B3system time=B2, = a reasonable guess is

>>> that the timer that is wrapping is in Xen it= self.
>>>

>>> Mark=B9s delta =3D -2999660303788 ns

>>> Your delta =3D -299966033421= 1 ns
>>>
>>> Googling, I see the HPET wraparound is ~306 seconds and this d= elta is about
>>> 3000 seconds, so that may be a bad guess.
>>>
>>> Keir, any thoughts on this? =A0Do you recall any post-4.0 patc= hes that may
>>> have
>>> fixed this?
>> I've never seen a 3000s wrap, and I don't know of anything= that would have
>> fixed a bug like this. If this is a Xen time wrap of some kind the= n it would
>> affect all running guests; it's not clear here whether only on= e, or all,
>> guests see the wrap.
>>
>> =A0 K.
>>
>>> Thanks,
>>> Dan
>>>
>>> References:
>>> http://lists.xensource.com/archive= s/html/xen-devel/2010-10/msg00210.html
>>> https://lkml.org/lkml/2010/10/26/126
>>>
>>>
>>> From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]
>>> Sent: Wednesday, February 23, 2011 3:50 AM
>>> To: xen-devel@= lists.xensource.co! =A0m; Xen Users
>>> Subject: [Xen-devel] Xen 4 TSC problems
>>>
>>>
>>> Hello
>>>
>>>
>>>
>>> I've got an issue about time keeping with Xen 4.0 (Debian = squeeze release).
>>>
>>>
>>>
>>> My problem is here (hopefully I amn't the only one, so the= re might be a bug
>>> somewhere) : http://bugs.debian.org/cgi-bin/bu= greport.cgi?bug=3D599161#50
>>>
>>> After some times, =A0I got this error : Clocksource tsc unstab= le (delta =3D
>>> -2999660334211 ns). It has happened on several servers.
>>>
>>>
>>>
>>> Looking at the output of "xm debug-key s;"
>>>
>>>
>>>
>>> (XEN) TSC has constant rate, deep Cstates possible, so not rel= iable,
>>> warp=3D2850
>>> (count=3D3)
>>>
>>>
>>>
>>> I am using a "Intel(R) Xeon(R) CPU L5420 =A0@ 2.50GHz&quo= t;, which has the
>>> "constant_tsc", but not the "nonstop_tsc" = one.
>>>
>>> On other systems with a newer cpu with "nonstop_tsc"= , I don't have this
>>> issue
>>> (systems are running the same distros with same config).
>>>
>>>
>>>
>>> I tried to boot with "max_cstate=3D0", but nothing c= hanged, my TSC isn't
>>> reliable and after some times, I will got the "50min"= ; issue again.
>>>
>>>
>>>
>>> I don't unders! =A0tand how a system can do a jump of &quo= t;50min" in the future.
>>> Why
>>> 50min ? it is not 40min, not 1 hour, it is always 50min.
>>>
>>> I don't know how to make my TSC "reliable" (I al= ready disable everything
>>> about
>>> Powerstate in BIOS Settings).
>>>
>>>
>>>
>>> Any ideas ?
>>>
>>>
>>>
>>> Regards
>>>
>>>
>>>
>>> Olivier
>>>
>>
>

--0015175caae4859f9a049d0443f3-- --===============1147676405== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1147676405==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: Xen 4 TSC problems Date: Thu, 24 Feb 2011 10:59:30 +0000 Message-ID: <4D6648220200007800033769@vpn.id2.novell.com> References: <4D655A39.2040501@gmail.com>

Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Olivier Hanesse Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Jeremy Fitzhardinge , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org >>> On 24.02.11 at 10:59, Olivier Hanesse = wrote: > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more = times. >=20 > Output of xm debug-key s : >=20 > (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > warp=3D2684 (count=3D4) Did you try turning of use of C states ("cpuidle=3D0" on the Xen command line)? Jan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Thu, 24 Feb 2011 11:30:12 +0000 Message-ID: References: <4D6648220200007800033769@vpn.id2.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4D6648220200007800033769@vpn.id2.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Jan Beulich , Olivier Hanesse Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Dan Magenheimer , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On 24/02/2011 10:59, "Jan Beulich" wrote: >>>> On 24.02.11 at 10:59, Olivier Hanesse wrote: >> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. >> >> Output of xm debug-key s : >> >> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, >> warp=2684 (count=4) > > Did you try turning of use of C states ("cpuidle=0" on the Xen > command line)? Another thing to try is changing the platform timer that Xen uses. It's using HPET on your machines, so try clocksource=pit on Xen command line, and confirm that the 'Platform timer is xxx' message changes in xm dmesg. However, this bug looks more like a CPU's TSC jumping forward (or maybe backward) for some inexplicable reason. We added code post 3.2 to detect the platform timer counter wrapping, and to account for that based on trusting the CPU's 64-bit TSC. But if the TSC value is bogus then we can detect a wrap when it didn't happen and the new code will do more harm than good. It is not currently possible to disable the code via a boto parameter -- maybe we could add that. However, if the problem is a jumpy TSC then it is better to fix that as Xen relies so heavily on TSC for time handling. -- Keir > Jan > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Thu, 24 Feb 2011 12:57:15 +0100 Message-ID: References: <4D6648220200007800033769@vpn.id2.novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1556827642==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Keir Fraser Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Dan Magenheimer , Jan Beulich , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============1556827642== Content-Type: multipart/alternative; boundary=0016e64caa7e21f5f6049d05ea9e --0016e64caa7e21f5f6049d05ea9e Content-Type: text/plain; charset=ISO-8859-1 Jan : I tried to turn off cstates with max_cstate=0 without success (still "not reliable"). With cpuidle=0, I also got : (XEN) TSC has constant rate, deep Cstates possible, so not reliable, warp=3022 (count=1) xm info | grep command xen_commandline : dom0_mem=512M cpuidle=0 loglvl=all guest_loglvl=all dom0_max_vcpus=1 dom0_vcpus_pin console=vga,com1 com1=19200,8n1 Keir : Using clocksource=pit : (XEN) Platform timer is 1.193MHz PIT I also got : (XEN) TSC has constant rate, deep Cstates possible, so not reliable, warp=3262 (count=2) 2011/2/24 Keir Fraser > On 24/02/2011 10:59, "Jan Beulich" wrote: > > >>>> On 24.02.11 at 10:59, Olivier Hanesse > wrote: > >> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more > times. > >> > >> Output of xm debug-key s : > >> > >> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > >> warp=2684 (count=4) > > > > Did you try turning of use of C states ("cpuidle=0" on the Xen > > command line)? > > Another thing to try is changing the platform timer that Xen uses. It's > using HPET on your machines, so try clocksource=pit on Xen command line, > and > confirm that the 'Platform timer is xxx' message changes in xm dmesg. > > However, this bug looks more like a CPU's TSC jumping forward (or maybe > backward) for some inexplicable reason. We added code post 3.2 to detect > the > platform timer counter wrapping, and to account for that based on trusting > the CPU's 64-bit TSC. But if the TSC value is bogus then we can detect a > wrap when it didn't happen and the new code will do more harm than good. It > is not currently possible to disable the code via a boto parameter -- maybe > we could add that. However, if the problem is a jumpy TSC then it is better > to fix that as Xen relies so heavily on TSC for time handling. > > -- Keir > > > Jan > > > > > --0016e64caa7e21f5f6049d05ea9e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Jan :=A0

I tried to turn off cstates with max_cstate= =3D0=A0without success (still "not reliable").

With cpuidle=3D0, I also got :

(XEN) TSC has constant rate, deep Cstates possible, so = not reliable, warp=3D3022 (count=3D1)

xm info= | grep command

xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M = cpuidle=3D0 loglvl=3Dall guest_loglvl=3Dall dom0_max_vcpus=3D1 dom0_vcpus_p= in console=3Dvga,com1 com1=3D19200,8n1

Keir :=A0

Using clocksou= rce=3Dpit :

(XEN) Platform timer is 1.19= 3MHz PIT

I also got : =A0

(XEN) TSC has constant rate, deep Cstates possible, so not reliable, warp= =3D3262 (count=3D2)

2011/2/24 Keir Fraser <keir@xen.org>

On 24/02/= 2011 10:59, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> On 24.02.11 at 10:59, Olivier Hanesse <olivier.hanesse@gmail.com> wrote:
>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or mo= re times.
>>
>> Output of xm debug-key s :
>>
>> (XEN) TSC has constant rate, deep Cstates possible, so not reliabl= e,
>> warp=3D2684 (count=3D4)
>
> Did you try turning of use of C states ("cpuidle=3D0" on the= Xen
> command line)?

Another thing to try is changing the platform timer that Xen us= es. It's
using HPET on your machines, so try clocksource=3Dpit on Xen command line, = and
confirm that the 'Platform timer is xxx' message changes in xm dmes= g.

However, this bug looks more like a CPU's TSC jumping forward (or maybe=
backward) for some inexplicable reason. We added code post 3.2 to detect th= e
platform timer counter wrapping, and to account for that based on trusting<= br> the CPU's 64-bit TSC. But if the TSC value is bogus then we can detect = a
wrap when it didn't happen and the new code will do more harm than good= . It
is not currently possible to disable the code via a boto parameter -- maybe=
we could add that. However, if the problem is a jumpy TSC then it is better=
to fix that as Xen relies so heavily on TSC for time handling.

=A0-- Keir

> Jan
>

--0016e64caa7e21f5f6049d05ea9e-- --===============1556827642== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users --===============1556827642==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: Xen 4 TSC problems Date: Thu, 24 Feb 2011 12:37:53 +0000 Message-ID: <4D665F3102000078000337B6@vpn.id2.novell.com> References: <4D6648220200007800033769@vpn.id2.novell.com>

<4D665F3102000078000337B6@vpn.id2.novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1821090853==" Return-path: In-Reply-To: <4D665F3102000078000337B6@vpn.id2.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Jan Beulich Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Dan Magenheimer , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============1821090853== Content-Type: multipart/alternative; boundary=0016367f985ebb86cd049d07e863 --0016367f985ebb86cd049d07e863 Content-Type: text/plain; charset=ISO-8859-1 Both dom0 and domUs are affected by this" jump". I expect to see something like "TSC marked as reliable, warp = 0". I got this on newer hardware with same config/distros. Is there a way to measure if it is a TSC warp ? to point out a cpu tsc issue ? 2011/2/24 Jan Beulich > >>> On 24.02.11 at 12:57, Olivier Hanesse > wrote: > > I tried to turn off cstates with max_cstate=0 without success (still "not > > reliable"). > > > > With cpuidle=0, I also got : > > > > (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > > warp=3022 (count=1) > > This message by itself isn't telling much I believe. > > > xm info | grep command > > xen_commandline : dom0_mem=512M cpuidle=0 loglvl=all > guest_loglvl=all > > dom0_max_vcpus=1 dom0_vcpus_pin console=vga,com1 com1=19200,8n1 > > > > Keir : > > > > Using clocksource=pit : > > > > (XEN) Platform timer is 1.193MHz PIT > > > > I also got : > > > > (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > > warp=3262 (count=2) > > The question is whether any of this eliminates the time jumps seen > by your DomU-s (from your past mails I wasn't actually sure whether > Dom0 also experienced this problem, albeit it would be odd if it didn't). > > Jan > > Jan > > --0016367f985ebb86cd049d07e863 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Both dom0 and domUs are affected by this" jump".

I e= xpect to see something like "TSC marked as reliable, warp =3D 0".= =A0

I got this on newer hardware with same config/distros.

<= div>

Is there a way to measure if it is a TSC warp ? to point out= a cpu tsc issue ?

2011/2/24 Jan Beulich <JBeulich@novell.com>

>>> On 24.02.11 = at 12:57, Olivier Hanesse <= olivier.hanesse@gmail.com> wrote:
> I tried to turn off cstates with max_cstate=3D0 without success (still= "not
> reliable").
>
> With cpuidle=3D0, I also got :
>
> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > warp=3D3022 (count=3D1)

This message by itself isn't telling much I believe.

> xm info | grep command
> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle=3D0 loglvl=3D= all guest_loglvl=3Dall
> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1=3D19200,8n1<= br> >
> Keir :
>
> Using clocksource=3Dpit :
>
> (XEN) Platform timer is 1.193MHz PIT
>
> I also got :
>
> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > warp=3D3262 (count=3D2)

The question is whether any of this eliminates the time jumps seen by your DomU-s (from your past mails I wasn't actually sure whether
Dom0 also experienced this problem, albeit it would be odd if it didn't= ).

Jan

Jan

--0016367f985ebb86cd049d07e863-- --===============1821090853== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1821090853==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Thu, 24 Feb 2011 14:52:03 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Olivier Hanesse , Jan Beulich Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Dan Magenheimer , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On 24/02/2011 14:20, "Olivier Hanesse" wrote: > Both dom0 and domUs are affected by this" jump". >=20 > I expect to see something like "TSC marked as reliable, warp =3D 0".=A0 > I got this on newer hardware with same config/distros. It depends on the CPU itself, older CPUs do not have the super-stable TSC features. But that should never cause a massive 3000s time jump. > Is there a way to measure if it is a TSC warp ? to point out a cpu tsc is= sue ? The TSC warps or out-of-sync issues that we could reasonably expect would b= e on the order of microseconds. A 3000s warp is something else entirely. Xen is very confused and/or some TSC or platform timer has jumped a long way (indicating a hardware/firmware issue). -- Keir >=20 > 2011/2/24 Jan Beulich >>>>> On 24.02.11 at 12:57, Olivier Hanesse wro= te: >>> I tried to turn off cstates with max_cstate=3D0 without success (still "n= ot >>> reliable"). >>>=20 >>> With cpuidle=3D0, I also got : >>>=20 >>> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, >>> warp=3D3022 (count=3D1) >>=20 >> This message by itself isn't telling much I believe. >>=20 >>> xm info | grep command >>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall guest_loglv= l=3Dall >>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1=3D19200,8n1 >>>=20 >>> Keir : >>>=20 >>> Using clocksource=3Dpit : >>>=20 >>> (XEN) Platform timer is 1.193MHz PIT >>>=20 >>> I also got : >>>=20 >>> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, >>> warp=3D3262 (count=3D2) >>=20 >> The question is whether any of this eliminates the time jumps seen >> by your DomU-s (from your past mails I wasn't actually sure whether >> Dom0 also experienced this problem, albeit it would be odd if it didn't)= . >>=20 >> Jan >>=20 >> Jan >>=20 >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: [Xen-devel] Xen 4 TSC problems Date: Thu, 24 Feb 2011 09:43:22 -0800 (PST) Message-ID: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> References: Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Keir Fraser , Olivier Hanesse , Jan Beulich Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org Just a wild guess, but this in Olivier's posted output: (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. and the fact that a 32-bit HPET wrap is ~300 seconds and, with the "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue (or a complete red herring, but I thought it worth mentioning). Mark and Olivier, it would be interesting to know if you are using the same processor/system. > -----Original Message----- > From: Keir Fraser [mailto:keir.xen@gmail.com] > Sent: Thursday, February 24, 2011 7:52 AM > To: Olivier Hanesse; Jan Beulich > Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen > Users; Dan Magenheimer; Keir Fraser > Subject: Re: [Xen-devel] Xen 4 TSC problems >=20 > On 24/02/2011 14:20, "Olivier Hanesse" > wrote: >=20 > > Both dom0 and domUs are affected by this" jump". > > > > I expect to see something like "TSC marked as reliable, warp =3D 0". > > I got this on newer hardware with same config/distros. >=20 > It depends on the CPU itself, older CPUs do not have the super-stable > TSC > features. But that should never cause a massive 3000s time jump. >=20 > > Is there a way to measure if it is a TSC warp ? to point out a cpu > tsc issue ? >=20 > The TSC warps or out-of-sync issues that we could reasonably expect > would be > on the order of microseconds. A 3000s warp is something else entirely. > Xen > is very confused and/or some TSC or platform timer has jumped a long > way > (indicating a hardware/firmware issue). >=20 > -- Keir >=20 > > > > 2011/2/24 Jan Beulich > >>>>> On 24.02.11 at 12:57, Olivier Hanesse > wrote: > >>> I tried to turn off cstates with max_cstate=3D0 without success > (still "not > >>> reliable"). > >>> > >>> With cpuidle=3D0, I also got : > >>> > >>> (XEN) TSC has constant rate, deep Cstates possible, so not > reliable, > >>> warp=3D3022 (count=3D1) > >> > >> This message by itself isn't telling much I believe. > >> > >>> xm info | grep command > >>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle=3D0 loglvl= =3Dall > guest_loglvl=3Dall > >>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1=3D19200,8n1 > >>> > >>> Keir : > >>> > >>> Using clocksource=3Dpit : > >>> > >>> (XEN) Platform timer is 1.193MHz PIT > >>> > >>> I also got : > >>> > >>> (XEN) TSC has constant rate, deep Cstates possible, so not > reliable, > >>> warp=3D3262 (count=3D2) > >> > >> The question is whether any of this eliminates the time jumps seen > >> by your DomU-s (from your past mails I wasn't actually sure whether > >> Dom0 also experienced this problem, albeit it would be odd if it > didn't). > >> > >> Jan > >> > >> Jan > >> > > > > >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: Xen 4 TSC problems Date: Thu, 24 Feb 2011 18:58:08 +0100 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1539987911==" Return-path: In-Reply-To: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dan Magenheimer Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============1539987911== Content-Type: multipart/alternative; boundary=0016367f985ec8d256049d0af408 --0016367f985ec8d256049d0af408 Content-Type: text/plain; charset=ISO-8859-1 Mark is running with a E5620 Xeon processor. I got a L5420. What is very strange is that this jump is always 50min, not more, not less. And we are not alone with Mark to have this issue. So it might have an explanation somewhere (bad counter, overflow, bug or somethings). So maybe this 300 seconds * 10 is a lead. Another point, what is the number "warp" really means, in the output of "xm debug-key -s". Should I monitor this number ? Maybe I could predict a jump by watching this value ? 2011/2/24 Dan Magenheimer > Just a wild guess, but this in Olivier's posted output: > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > (or a complete red herring, but I thought it worth mentioning). > > Mark and Olivier, it would be interesting to know if you are > using the same processor/system. > > > -----Original Message----- > > From: Keir Fraser [mailto:keir.xen@gmail.com] > > Sent: Thursday, February 24, 2011 7:52 AM > > To: Olivier Hanesse; Jan Beulich > > Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen > > Users; Dan Magenheimer; Keir Fraser > > Subject: Re: [Xen-devel] Xen 4 TSC problems > > > > On 24/02/2011 14:20, "Olivier Hanesse" > > wrote: > > > > > Both dom0 and domUs are affected by this" jump". > > > > > > I expect to see something like "TSC marked as reliable, warp = 0". > > > I got this on newer hardware with same config/distros. > > > > It depends on the CPU itself, older CPUs do not have the super-stable > > TSC > > features. But that should never cause a massive 3000s time jump. > > > > > Is there a way to measure if it is a TSC warp ? to point out a cpu > > tsc issue ? > > > > The TSC warps or out-of-sync issues that we could reasonably expect > > would be > > on the order of microseconds. A 3000s warp is something else entirely. > > Xen > > is very confused and/or some TSC or platform timer has jumped a long > > way > > (indicating a hardware/firmware issue). > > > > -- Keir > > > > > > > > 2011/2/24 Jan Beulich > > >>>>> On 24.02.11 at 12:57, Olivier Hanesse > > wrote: > > >>> I tried to turn off cstates with max_cstate=0 without success > > (still "not > > >>> reliable"). > > >>> > > >>> With cpuidle=0, I also got : > > >>> > > >>> (XEN) TSC has constant rate, deep Cstates possible, so not > > reliable, > > >>> warp=3022 (count=1) > > >> > > >> This message by itself isn't telling much I believe. > > >> > > >>> xm info | grep command > > >>> xen_commandline : dom0_mem=512M cpuidle=0 loglvl=all > > guest_loglvl=all > > >>> dom0_max_vcpus=1 dom0_vcpus_pin console=vga,com1 com1=19200,8n1 > > >>> > > >>> Keir : > > >>> > > >>> Using clocksource=pit : > > >>> > > >>> (XEN) Platform timer is 1.193MHz PIT > > >>> > > >>> I also got : > > >>> > > >>> (XEN) TSC has constant rate, deep Cstates possible, so not > > reliable, > > >>> warp=3262 (count=2) > > >> > > >> The question is whether any of this eliminates the time jumps seen > > >> by your DomU-s (from your past mails I wasn't actually sure whether > > >> Dom0 also experienced this problem, albeit it would be odd if it > > didn't). > > >> > > >> Jan > > >> > > >> Jan > > >> > > > > > > > > > > > --0016367f985ec8d256049d0af408 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Mark is running with a=A0E5620 X= eon processor.

I got a=A0L5420.

What is very strange is that this jump is alwa= ys 50min, not more, not less. And we are not alone with Mark to have this i= ssue.

So it might have an explanation somewhere (bad counter, overflow, bug = or somethings).

So maybe this 300 seconds * 10 is a lead.=A0

Another point, what is the number "warp" rea= lly means, in the output of "xm debug-key -s".

Should I monitor this number ? =A0Maybe I could predict a jump by watc= hing this value ?

2011/2/24 Dan Ma= genheimer <dan.magenheimer@oracle.com>

Just a wild guess, but this in Olivier'= s posted output:

(XEN) Platform timer appears to have unexpectedly wrapped 10 or more times.=

and the fact that a 32-bit HPET wrap is ~300 seconds and, with the "10 or more times", 10 * 300 seconds is 3000 seconds, might be a = clue
(or a complete red herring, but I thought it worth mentioning).

Mark and Olivier, it would be interesting to know if you are
using the same processor/system.

> -----Original Message-----
> From: Keir Fraser [mailto:keir.x= en@gmail.com]
> Sent: Thursday, February 24, 2011 7:52 AM
> To: Olivier Hanesse; Jan Beulich
> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen
> Users; Dan Magenheimer; Keir Fraser
> Subject: Re: [Xen-devel] Xen 4 TSC problems
>
> On 24/02/2011 14:20, "Olivier Hanesse" <olivier.hanesse@gmail.com>
> wrote:
>
> > Both dom0 and domUs are affected by this" jump".
> >
> > I expect to see something like "TSC marked as reliable, warp= =3D 0".
> > I got this on newer hardware with same config/distros.
>
> It depends on the CPU itself, older CPUs do not have the super-stable<= br> > TSC
> features. But that should never cause a massive 3000s time jump.
>
> > Is there a way to measure if it is a TSC warp ? to point out a cp= u
> tsc issue ?
>
> The TSC warps or out-of-sync issues that we could reasonably expect > would be
> on the order of microseconds. A 3000s warp is something else entirely.=
> Xen
> is very confused and/or some TSC or platform timer has jumped a long > way
> (indicating a hardware/firmware issue).
>
> =A0-- Keir
>
> >
> > 2011/2/24 Jan Beulich <= JBeulich@novell.com>
> >>>>> On 24.02.11 at 12:57, Olivier Hanesse <olivier.hanesse@gmail.com>
> wrote:
> >>> I tried to turn off cstates with max_cstate=3D0 without s= uccess
> (still "not
> >>> reliable").
> >>>
> >>> With cpuidle=3D0, I also got :
> >>>
> >>> (XEN) TSC has constant rate, deep Cstates possible, so no= t
> reliable,
> >>> warp=3D3022 (count=3D1)
> >>
> >> This message by itself isn't telling much I believe.
> >>
> >>> xm info | grep command
> >>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle= =3D0 loglvl=3Dall
> guest_loglvl=3Dall
> >>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1= =3D19200,8n1
> >>>
> >>> Keir :
> >>>
> >>> Using clocksource=3Dpit :
> >>>
> >>> (XEN) Platform timer is 1.193MHz PIT
> >>>
> >>> I also got :
> >>>
> >>> (XEN) TSC has constant rate, deep Cstates possible, so no= t
> reliable,
> >>> warp=3D3262 (count=3D2)
> >>
> >> The question is whether any of this eliminates the time jumps= seen
> >> by your DomU-s (from your past mails I wasn't actually su= re whether
> >> Dom0 also experienced this problem, albeit it would be odd if= it
> didn't).
> >>
> >> Jan
> >>
> >> Jan
> >>
> >
> >
>
>

--0016367f985ec8d256049d0af408-- --===============1539987911== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1539987911==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Thu, 24 Feb 2011 11:01:58 -0800 Message-ID: <4D66AB26.9020000@goop.org> References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Dan Magenheimer Cc: xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Olivier Hanesse , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > Just a wild guess, but this in Olivier's posted output: > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > (or a complete red herring, but I thought it worth mentioning). > > Mark and Olivier, it would be interesting to know if you are > using the same processor/system. It definitely seems like some kind of problem on the host system rather than anything in the guests themselves. If the platform timer is misbehaving, then Xen could be completely screwing up the pvclock calibration which it then passes to guests. Could it be one of those "platform clock stops in certain power states" problems? J >> -----Original Message----- >> From: Keir Fraser [mailto:keir.xen@gmail.com] >> Sent: Thursday, February 24, 2011 7:52 AM >> To: Olivier Hanesse; Jan Beulich >> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen >> Users; Dan Magenheimer; Keir Fraser >> Subject: Re: [Xen-devel] Xen 4 TSC problems >> >> On 24/02/2011 14:20, "Olivier Hanesse" >> wrote: >> >>> Both dom0 and domUs are affected by this" jump". >>> >>> I expect to see something like "TSC marked as reliable, warp = 0". >>> I got this on newer hardware with same config/distros. >> It depends on the CPU itself, older CPUs do not have the super-stable >> TSC >> features. But that should never cause a massive 3000s time jump. >> >>> Is there a way to measure if it is a TSC warp ? to point out a cpu >> tsc issue ? >> >> The TSC warps or out-of-sync issues that we could reasonably expect >> would be >> on the order of microseconds. A 3000s warp is something else entirely. >> Xen >> is very confused and/or some TSC or platform timer has jumped a long >> way >> (indicating a hardware/firmware issue). >> >> -- Keir >> >>> 2011/2/24 Jan Beulich >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse >> wrote: >>>>> I tried to turn off cstates with max_cstate=0 without success >> (still "not >>>>> reliable"). >>>>> >>>>> With cpuidle=0, I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3022 (count=1) >>>> This message by itself isn't telling much I believe. >>>> >>>>> xm info | grep command >>>>> xen_commandline : dom0_mem=512M cpuidle=0 loglvl=all >> guest_loglvl=all >>>>> dom0_max_vcpus=1 dom0_vcpus_pin console=vga,com1 com1=19200,8n1 >>>>> >>>>> Keir : >>>>> >>>>> Using clocksource=pit : >>>>> >>>>> (XEN) Platform timer is 1.193MHz PIT >>>>> >>>>> I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3262 (count=2) >>>> The question is whether any of this eliminates the time jumps seen >>>> by your DomU-s (from your past mails I wasn't actually sure whether >>>> Dom0 also experienced this problem, albeit it would be odd if it >> didn't). >>>> Jan >>>> >>>> Jan >>>> >>> >> From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Mon, 28 Feb 2011 15:37:02 +0100 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <4D66AB26.9020000@goop.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0454445095==" Return-path: In-Reply-To: <4D66AB26.9020000@goop.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Jeremy Fitzhardinge Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============0454445095== Content-Type: multipart/alternative; boundary=0016e64caa7ef37658049d589c98 --0016e64caa7ef37658049d589c98 Content-Type: text/plain; charset=ISO-8859-1 Hello, It happened again twice this weekend. What about setting "tsc_mode=2" for my vms ? Should this mode prevent this bug (coming from a bad emulated tsc due to firmware issue ? is it possible ?) from affecting time in domUs ? Setting clocksource=pit, make 'tsc' available in "/sys/devices/system/clocksource/clocksource0/available_clocksource" (otherwise only xen is available, is it normal ? ). Should I bypass xen clocksource and use tsc as a clocksource for dom0/domU ? or will it be worsed ? Regards Olivier 2011/2/24 Jeremy Fitzhardinge > On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > > Just a wild guess, but this in Olivier's posted output: > > > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more > times. > > > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > > (or a complete red herring, but I thought it worth mentioning). > > > > Mark and Olivier, it would be interesting to know if you are > > using the same processor/system. > > It definitely seems like some kind of problem on the host system rather > than anything in the guests themselves. If the platform timer is > misbehaving, then Xen could be completely screwing up the pvclock > calibration which it then passes to guests. > > Could it be one of those "platform clock stops in certain power states" > problems? > > J > > >> -----Original Message----- > >> From: Keir Fraser [mailto:keir.xen@gmail.com] > >> Sent: Thursday, February 24, 2011 7:52 AM > >> To: Olivier Hanesse; Jan Beulich > >> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen > >> Users; Dan Magenheimer; Keir Fraser > >> Subject: Re: [Xen-devel] Xen 4 TSC problems > >> > >> On 24/02/2011 14:20, "Olivier Hanesse" > >> wrote: > >> > >>> Both dom0 and domUs are affected by this" jump". > >>> > >>> I expect to see something like "TSC marked as reliable, warp = 0". > >>> I got this on newer hardware with same config/distros. > >> It depends on the CPU itself, older CPUs do not have the super-stable > >> TSC > >> features. But that should never cause a massive 3000s time jump. > >> > >>> Is there a way to measure if it is a TSC warp ? to point out a cpu > >> tsc issue ? > >> > >> The TSC warps or out-of-sync issues that we could reasonably expect > >> would be > >> on the order of microseconds. A 3000s warp is something else entirely. > >> Xen > >> is very confused and/or some TSC or platform timer has jumped a long > >> way > >> (indicating a hardware/firmware issue). > >> > >> -- Keir > >> > >>> 2011/2/24 Jan Beulich > >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse > >> wrote: > >>>>> I tried to turn off cstates with max_cstate=0 without success > >> (still "not > >>>>> reliable"). > >>>>> > >>>>> With cpuidle=0, I also got : > >>>>> > >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not > >> reliable, > >>>>> warp=3022 (count=1) > >>>> This message by itself isn't telling much I believe. > >>>> > >>>>> xm info | grep command > >>>>> xen_commandline : dom0_mem=512M cpuidle=0 loglvl=all > >> guest_loglvl=all > >>>>> dom0_max_vcpus=1 dom0_vcpus_pin console=vga,com1 com1=19200,8n1 > >>>>> > >>>>> Keir : > >>>>> > >>>>> Using clocksource=pit : > >>>>> > >>>>> (XEN) Platform timer is 1.193MHz PIT > >>>>> > >>>>> I also got : > >>>>> > >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not > >> reliable, > >>>>> warp=3262 (count=2) > >>>> The question is whether any of this eliminates the time jumps seen > >>>> by your DomU-s (from your past mails I wasn't actually sure whether > >>>> Dom0 also experienced this problem, albeit it would be odd if it > >> didn't). > >>>> Jan > >>>> > >>>> Jan > >>>> > >>> > >> > > --0016e64caa7ef37658049d589c98 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hello,

It happened again twice this weekend.

<= br>

What about setting "tsc_mode=3D2" for my vms ? Shou= ld this mode prevent this bug (coming from a bad emulated tsc due to firmwa= re issue ? is it possible ?) from affecting time in domUs ?

Setting clocksource=3Dpit, make 'tsc' available= in "/sys/devices/system/clocksource/clocksource0/available_clocksourc= e" (otherwise only xen is available, is it normal ? ).=A0

Should I bypass xen clocksource and use tsc as a clocksource for= dom0/domU ? or =A0will it be worsed ?

Regards

Olivier

2011= /2/24 Jeremy Fitzhardinge <jeremy@goop.org>

On 02/24/2011 09:43 AM, D= an Magenheimer wrote:
> Just a wild guess, but this in Olivier's posted output:
>
> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more t= imes.
>
> and the fact that a 32-bit HPET wrap is ~300 seconds and, with the
> "10 or more times", 10 * 300 seconds is 3000 seconds, might = be a clue
> (or a complete red herring, but I thought it worth mentioning).
>
> Mark and Olivier, it would be interesting to know if you are
> using the same processor/system.

It definitely seems like some kind of problem on the host system rath= er
than anything in the guests themselves. =A0If the platform timer is
misbehaving, then Xen could be completely screwing up the pvclock
calibration which it then passes to guests.

Could it be one of those "platform clock stops in certain power states= "
problems?

=A0 =A0J

>> -----Original Message-----
>> From: Keir Fraser [mailto:ke= ir.xen@gmail.com]
>> Sent: Thursday, February 24, 2011 7:52 AM
>> To: Olivier Hanesse; Jan Beulich
>> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen
>> Users; Dan Magenheimer; Keir Fraser
>> Subject: Re: [Xen-devel] Xen 4 TSC problems
>>
>> On 24/02/2011 14:20, "Olivier Hanesse" <olivier.hanesse@gmail.com>
>> wrote:
>>
>>> Both dom0 and domUs are affected by this" jump".
>>>
>>> I expect to see something like "TSC marked as reliable, w= arp =3D 0".
>>> I got this on newer hardware with same config/distros.
>> It depends on the CPU itself, older CPUs do not have the super-sta= ble
>> TSC
>> features. But that should never cause a massive 3000s time jump. >>
>>> Is there a way to measure if it is a TSC warp ? to point out a= cpu
>> tsc issue ?
>>
>> The TSC warps or out-of-sync issues that we could reasonably expec= t
>> would be
>> on the order of microseconds. A 3000s warp is something else entir= ely.
>> Xen
>> is very confused and/or some TSC or platform timer has jumped a lo= ng
>> way
>> (indicating a hardware/firmware issue).
>>
>> =A0-- Keir
>>
>>> 2011/2/24 Jan Beulich <JBeulich@novell.com>
>>>>>>> On 24.02.11 at 12:57, Olivier Hanesse <olivier.hanesse@gmail.com><= br> >> wrote:
>>>>> I tried to turn off cstates with max_cstate=3D0 withou= t success
>> (still "not
>>>>> reliable").
>>>>>
>>>>> With cpuidle=3D0, I also got :
>>>>>
>>>>> (XEN) TSC has constant rate, deep Cstates possible, so= not
>> reliable,
>>>>> warp=3D3022 (count=3D1)
>>>> This message by itself isn't telling much I believe. >>>>
>>>>> xm info | grep command
>>>>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuid= le=3D0 loglvl=3Dall
>> guest_loglvl=3Dall
>>>>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 c= om1=3D19200,8n1
>>>>>
>>>>> Keir :
>>>>>
>>>>> Using clocksource=3Dpit :
>>>>>
>>>>> (XEN) Platform timer is 1.193MHz PIT
>>>>>
>>>>> I also got :
>>>>>
>>>>> (XEN) TSC has constant rate, deep Cstates possible, so= not
>> reliable,
>>>>> warp=3D3262 (count=3D2)
>>>> The question is whether any of this eliminates the time ju= mps seen
>>>> by your DomU-s (from your past mails I wasn't actually= sure whether
>>>> Dom0 also experienced this problem, albeit it would be odd= if it
>> didn't).
>>>> Jan
>>>>
>>>> Jan
>>>>
>>>
>>

--0016e64caa7ef37658049d589c98-- --===============0454445095== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users --===============0454445095==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: Xen 4 TSC problems Date: Mon, 28 Feb 2011 15:00:47 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Olivier Hanesse , Jeremy Fitzhardinge Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org The message about detecting wrapped platform timer on Xen console indicates a host problem rather than a guest configuration problem. Did you try running long term with changed platform timer source on Xen command line (clocksource=3Dpit), and also cpuidle=3D0? K. On 28/02/2011 14:37, "Olivier Hanesse" wrote: > Hello, >=20 > It happened again twice this weekend. >=20 > What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent thi= s bug > (coming from a bad emulated tsc due to firmware issue ? is it possible ?)= from > affecting time in domUs ? >=20 > Setting clocksource=3Dpit, make 'tsc' available in > "/sys/devices/system/clocksource/clocksource0/available_clocksource" > (otherwise only xen is available, is it normal ? ).=A0 >=20 > Should I bypass xen clocksource and use tsc as a clocksource for dom0/dom= U ? > or =A0will it be worsed ? >=20 > Regards >=20 > Olivier >=20 > 2011/2/24 Jeremy Fitzhardinge >> On 02/24/2011 09:43 AM, Dan Magenheimer wrote: >>> Just a wild guess, but this in Olivier's posted output: >>>=20 >>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more ti= mes. >>>=20 >>> and the fact that a 32-bit HPET wrap is ~300 seconds and, with the >>> "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue >>> (or a complete red herring, but I thought it worth mentioning). >>>=20 >>> Mark and Olivier, it would be interesting to know if you are >>> using the same processor/system. >>=20 >> It definitely seems like some kind of problem on the host system rather >> than anything in the guests themselves. =A0If the platform timer is >> misbehaving, then Xen could be completely screwing up the pvclock >> calibration which it then passes to guests. >>=20 >> Could it be one of those "platform clock stops in certain power states" >> problems? >>=20 >> =A0 =A0J >>=20 >>>> -----Original Message----- >>>> From: Keir Fraser [mailto:keir.xen@gmail.com] >>>> Sent: Thursday, February 24, 2011 7:52 AM >>>> To: Olivier Hanesse; Jan Beulich >>>> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xe= n >>>> Users; Dan Magenheimer; Keir Fraser >>>> Subject: Re: [Xen-devel] Xen 4 TSC problems >>>>=20 >>>> On 24/02/2011 14:20, "Olivier Hanesse" >>>> wrote: >>>>=20 >>>>> Both dom0 and domUs are affected by this" jump". >>>>>=20 >>>>> I expect to see something like "TSC marked as reliable, warp =3D 0". >>>>> I got this on newer hardware with same config/distros. >>>> It depends on the CPU itself, older CPUs do not have the super-stable >>>> TSC >>>> features. But that should never cause a massive 3000s time jump. >>>>=20 >>>>> Is there a way to measure if it is a TSC warp ? to point out a cpu >>>> tsc issue ? >>>>=20 >>>> The TSC warps or out-of-sync issues that we could reasonably expect >>>> would be >>>> on the order of microseconds. A 3000s warp is something else entirely. >>>> Xen >>>> is very confused and/or some TSC or platform timer has jumped a long >>>> way >>>> (indicating a hardware/firmware issue). >>>>=20 >>>> =A0-- Keir >>>>=20 >>>>> 2011/2/24 Jan Beulich >>>>>>>>> On 24.02.11 at 12:57, Olivier Hanesse >>>> wrote: >>>>>>> I tried to turn off cstates with max_cstate=3D0 without success >>>> (still "not >>>>>>> reliable"). >>>>>>>=20 >>>>>>> With cpuidle=3D0, I also got : >>>>>>>=20 >>>>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >>>> reliable, >>>>>>> warp=3D3022 (count=3D1) >>>>>> This message by itself isn't telling much I believe. >>>>>>=20 >>>>>>> xm info | grep command >>>>>>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall >>>> guest_loglvl=3Dall >>>>>>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1=3D19200,8n1 >>>>>>>=20 >>>>>>> Keir : >>>>>>>=20 >>>>>>> Using clocksource=3Dpit : >>>>>>>=20 >>>>>>> (XEN) Platform timer is 1.193MHz PIT >>>>>>>=20 >>>>>>> I also got : >>>>>>>=20 >>>>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >>>> reliable, >>>>>>> warp=3D3262 (count=3D2) >>>>>> The question is whether any of this eliminates the time jumps seen >>>>>> by your DomU-s (from your past mails I wasn't actually sure whether >>>>>> Dom0 also experienced this problem, albeit it would be odd if it >>>> didn't). >>>>>> Jan >>>>>>=20 >>>>>> Jan >>>>>>=20 >>>>>=20 >>>>=20 >>=20 >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: [Xen-devel] Xen 4 TSC problems Date: Mon, 28 Feb 2011 07:14:23 -0800 (PST) Message-ID: <49766a9d-0e37-46d2-9497-33c2e981a871@default> References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <4D66AB26.9020000@goop.org AANLkTi=NFebVsnj_09+SpCZUo_bNTnTyJLAoR5sfbwZa@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1911446809==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Olivier Hanesse , Jeremy Fitzhardinge Cc: xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Mark, Keir Fraser , Xen Users , Adams List-Id: xen-devel@lists.xenproject.org --===============1911446809== Content-Type: multipart/alternative; boundary="__129890608367518096abhmt020" --__129890608367518096abhmt020 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Hi Olivier - =20 It is the Xen clocksource that you want to try to change, not the dom0 cloc= ksource. To do this, you need to specify "clocksource=3Dpit" on the Xen bo= ot line (and reboot), not the dom0 boot line. =20 I believe Mark Adams played with tsc_mode to see if it solved his (similar?= identical?) problem last year, and it didn't make any difference. Please try booting Xen with "clocksource=3Dpit" and ensure that "Platform t= imer is 1.19MHz PIT" appears in the Xen boot messages. If the 50min jump d= oes not appear again, it would point to a problem in the hpet, either hardw= are or software. =20 Thanks, Dan =20 From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]=20 Sent: Monday, February 28, 2011 7:37 AM To: Jeremy Fitzhardinge Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; xen-devel@lists.= xensource.com; Xen Users; Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems =20 Hello, =20 It happened again twice this weekend. =20 What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent thi= s bug (coming from a bad emulated tsc due to firmware issue ? is it possibl= e ?) from affecting time in domUs ? =20 Setting clocksource=3Dpit, make 'tsc' available in "/sys/devices/system/clo= cksource/clocksource0/available_clocksource" (otherwise only xen is availab= le, is it normal ? ).=20 =20 Should I bypass xen clocksource and use tsc as a clocksource for dom0/domU = ? or will it be worsed ? =20 Regards =20 Olivier =20 2011/2/24 Jeremy Fitzhardinge On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > Just a wild guess, but this in Olivier's posted output: > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more time= s. > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > (or a complete red herring, but I thought it worth mentioning). > > Mark and Olivier, it would be interesting to know if you are > using the same processor/system. It definitely seems like some kind of problem on the host system rather than anything in the guests themselves. If the platform timer is misbehaving, then Xen could be completely screwing up the pvclock calibration which it then passes to guests. Could it be one of those "platform clock stops in certain power states" problems? J >> -----Original Message----- >> From: Keir Fraser [mailto:HYPERLINK "mailto:keir.xen@gmail.com"keir.xen@= gmail.com] >> Sent: Thursday, February 24, 2011 7:52 AM >> To: Olivier Hanesse; Jan Beulich >> Cc: Mark Adams; Jeremy Fitzhardinge; HYPERLINK "mailto:xen-devel@lists.x= ensource.com"xen-devel@lists.xensource.com; Xen >> Users; Dan Magenheimer; Keir Fraser >> Subject: Re: [Xen-devel] Xen 4 TSC problems >> >> On 24/02/2011 14:20, "Olivier Hanesse" >> wrote: >> >>> Both dom0 and domUs are affected by this" jump". >>> >>> I expect to see something like "TSC marked as reliable, warp =3D 0". >>> I got this on newer hardware with same config/distros. >> It depends on the CPU itself, older CPUs do not have the super-stable >> TSC >> features. But that should never cause a massive 3000s time jump. >> >>> Is there a way to measure if it is a TSC warp ? to point out a cpu >> tsc issue ? >> >> The TSC warps or out-of-sync issues that we could reasonably expect >> would be >> on the order of microseconds. A 3000s warp is something else entirely. >> Xen >> is very confused and/or some TSC or platform timer has jumped a long >> way >> (indicating a hardware/firmware issue). >> >> -- Keir >> >>> 2011/2/24 Jan Beulich >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse >> wrote: >>>>> I tried to turn off cstates with max_cstate=3D0 without success >> (still "not >>>>> reliable"). >>>>> >>>>> With cpuidle=3D0, I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3D3022 (count=3D1) >>>> This message by itself isn't telling much I believe. >>>> >>>>> xm info | grep command >>>>> xen_commandline : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall >> guest_loglvl=3Dall >>>>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1=3D19200,8n1 >>>>> >>>>> Keir : >>>>> >>>>> Using clocksource=3Dpit : >>>>> >>>>> (XEN) Platform timer is 1.193MHz PIT >>>>> >>>>> I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3D3262 (count=3D2) >>>> The question is whether any of this eliminates the time jumps seen >>>> by your DomU-s (from your past mails I wasn't actually sure whether >>>> Dom0 also experienced this problem, albeit it would be odd if it >> didn't). >>>> Jan >>>> >>>> Jan >>>> >>> >> =20 --__129890608367518096abhmt020 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: quoted-printable

Hi Ol= ivier –

&nb= sp;

It is the Xen clocksour= ce that you want to try to change, not the dom0 clocksource. To do = this, you need to specify “clocksource=3Dpit” on the Xen boot= line (and reboot), not the dom0 boot line.

I believe Mark Adams played with tsc_mode to see if it solved! =20 his (similar? identical?) problem last year, and it didn’t make any dif= ference.

Please tr= y booting Xen with “clocksource=3Dpit” and ensure that “= ;Platform timer is 1.19MHz PIT” appears in the Xen boot messages.&n= bsp; If the 50min jump does not appear again, it would point to a problem= in the hpet, either hardware or software.

Thanks,

Dan

From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]
Sen= t: Monday, February 28, 2011 7:37 AM
To: Jeremy Fitzharding= e
Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; xe= n-devel@lists.xensource.com; Xen Users; Keir Fraser
Subject: Re= : [Xen-devel] Xen 4 TSC problems

Hello,<= /o:p>

It happened again twice this weekend.

What about setting "tsc_mode=3D2" = for my vms ? Should this mode prevent this bug (coming from a bad emulate= d tsc due to firmware issue ? is it possible ?) from affecting time in do= mUs ?

Setting clocksource=3Dpit, make 'tsc' ava= ilable in "/sys/devices/system/clocksource/clocksource0/available_cl= ocksource" (otherwise only xen is available, is it normal ? ). =

Should I bypass xen clocksource and use tsc as= a clocksource for dom0/domU ? or will it be worsed ?

Regards

&= nbsp;

Olivier

<= o:p>

2011/2/24 Jeremy Fitzhardin= ge <jeremy@goop.org>

On 02/24/= 2011 09:43 AM, Dan Magenheimer wrote:
> Just a wild guess, but this= in Olivier's posted output:
>
> (XEN) Platform timer appears= to have unexpectedly wrapped 10 or more times.
>
> and the f= act that a 32-bit HPET wrap is ~300 seconds and, with the
> "1= 0 or more times", 10 * 300 seconds is 3000 seconds, might be a clue<= br>> (or a complete red herring, but I thought it worth mentioning).>
> Mark and Olivier, it would be interesting to know if you a= re
> using the same processor/system.

It definitely seems like some kind of problem on the host syste= m rather
than anything in the guests themselves. ! =20 If the platform timer is
misbehaving, then Xen could be completely screwi= ng up the pvclock
calibration which it then passes to guests.

C= ould it be one of those "platform clock stops in certain power state= s"
problems?

J

>> -----Original M= essage-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com]
>> Sent: Thursday, Februa= ry 24, 2011 7:52 AM
>> To: Olivier Hanesse; Jan Beulich
>&= gt; Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen
>> Users= ; Dan Magenheimer; Keir Fraser
>> Subject: Re: [Xen-devel] Xen 4= TSC problems
>>
>> On 24/02/2011 14:20, "Olivier = Hanesse" <olivier.ha= nesse@gmail.com>
>> wrote:
>>
>>> Both dom0 and domUs are affected by th= is" jump".
>>>
>>> I expect to see some= thing like "TSC marked as reliable, warp =3D 0".
>>>= ; I got this on newer hardware with same config/distros.
>> It d= epends on the CPU itself, older CPUs do not have the super-stable
>= > TSC
>> features. But that should never cause a massive 3000= s time jump.
>>
>>> Is there a way to measure if it = is a TSC warp ? to point out a cpu
>> tsc issue ?
>>>> The TSC warps or out-of-sync issues that we could reasonably ex= pect
>> would be
>> on the order of microseconds. A 300= 0s warp is something else entirely.
>> Xen
>> is very c= onfused and/or some TSC or platform timer has jumped a long
>> w= ay
>> (indicating a hardware/firmware issue).
>>
>= ;> -- Keir
>>
>>! =20 ;> 2011/2/24 Jan Beulich <JBeulich= @novell.com>
>>>>>>> On 24.02.11 at 12:57,= Olivier Hanesse <olivier= .hanesse@gmail.com>
>> wrote:
>>>>> I t= ried to turn off cstates with max_cstate=3D0 without success
>> = (still "not
>>>>> reliable").
>>>= >>
>>>>> With cpuidle=3D0, I also got :
>&g= t;>>>
>>>>> (XEN) TSC has constant rate, deep = Cstates possible, so not
>> reliable,
>>>>> wa= rp=3D3022 (count=3D1)
>>>> This message by itself isn't te= lling much I believe.
>>>>
>>>>> xm info= | grep command
>>>>> xen_commandline &nb= sp; : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall
>> guest_lo= glvl=3Dall
>>>>> dom0_max_vcpus=3D1 dom0_vcp! =20 us_pin console=3Dvga,com1 com1=3D19200,8n1
>>>>>
>>&g= t;>> Keir :
>>>>>
>>>>> Using c= locksource=3Dpit :
>>>>>
>>>>> (XEN) = Platform timer is 1.193MHz PIT
>>>>>
>>>>= ;> I also got :
>>>>>
>>>>> (XEN) = TSC has constant rate, deep Cstates possible, so not
>> reliable= ,
>>>>> warp=3D3262 (count=3D2)
>>>> The= question is whether any of this eliminates the time jumps seen
>&g= t;>> by your DomU-s (from your past mails I wasn't actually sure wh= ether
>>>> Dom0 also experienced this problem, albeit it w= ould be odd if it
>> didn't).
>>>> Jan
>>= ;>>
>>>> Jan
>>>>
>>>
= >>

--__129890608367518096abhmt020-- --===============1911446809== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users --===============1911446809==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Mon, 28 Feb 2011 16:23:07 +0100 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <49766a9d-0e37-46d2-9497-33c2e981a871@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1988755835==" Return-path: In-Reply-To: <49766a9d-0e37-46d2-9497-33c2e981a871@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Dan Magenheimer Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============1988755835== Content-Type: multipart/alternative; boundary=0015175cb0c6bbce5f049d59411e --0015175cb0c6bbce5f049d59411e Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Keir : Yes, it is "under progress". To make this change, I had to reboot every server, so it is taking time (production server :() So i was hoping to find a quick method to mitigate this issue on domUs whil= e rebooting servers. As this bug happens once or twice per server since October, I can't say tha= t right now that changing platform timer to PIT fixed it. I have to wait (I hope forever!) this bug to happen again on a 'patched' server ... But even with clcoksource=3Dpit, I am seeing some warp=3D3000+ in debug mes= sage ? I guess it is not a good sign, is it ? Jan : I was hoping to find a way to make the domU clocksource more "independent" like with xen3.2. 2011/2/28 Dan Magenheimer > Hi Olivier =96 > > > > It is the Xen clocksource that you want to try to change, not the dom0 > clocksource. To do this, you need to specify =93clocksource=3Dpit=94 on = the Xen > boot line (and reboot), not the dom0 boot line. > > > > I believe Mark Adams played with tsc_mode to see if it solved! his > (similar? identical?) problem last year, and it didn=92t make any differe= nce. > > > Please try booting Xen with =93clocksource=3Dpit=94 and ensure that =93Pl= atform > timer is 1.19MHz PIT=94 appears in the Xen boot messages. If the 50min j= ump > does not appear again, it would point to a problem in the hpet, either > hardware or software. > > > > Thanks, > > Dan > > > > *From:* Olivier Hanesse [mailto:olivier.hanesse@gmail.com] > *Sent:* Monday, February 28, 2011 7:37 AM > *To:* Jeremy Fitzhardinge > *Cc:* Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; > xen-devel@lists.xensource.com; Xen Users; Keir Fraser > > *Subject:* Re: [Xen-devel] Xen 4 TSC problems > > > > Hello, > > > > It happened again twice this weekend. > > > > What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent t= his > bug (coming from a bad emulated tsc due to firmware issue ? is it possibl= e > ?) from affecting time in domUs ? > > > > Setting clocksource=3Dpit, make 'tsc' available in > "/sys/devices/system/clocksource/clocksource0/available_clocksource" > (otherwise only xen is available, is it normal ? ). > > > > Should I bypass xen clocksource and use tsc as a clocksource for dom0/dom= U > ? or will it be worsed ? > > > > Regards > > > > Olivier > > > > 2011/2/24 Jeremy Fitzhardinge > > On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > > Just a wild guess, but this in Olivier's posted output: > > > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more > times. > > > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > > (or a complete red herring, but I thought it worth mentioning). > > > > Mark and Olivier, it would be interesting to know if you are > > using the same processor/system. > > It definitely seems like some kind of problem on the host system rather > than anything in the guests themselves. ! If the platform timer is > misbehaving, then Xen could be completely screwing up the pvclock > calibration which it then passes to guests. > > Could it be one of those "platform clock stops in certain power states" > problems? > > > J > > >> -----Original Message----- > >> From: Keir Fraser [mailto:keir.xen@gmail.com] > >> Sent: Thursday, February 24, 2011 7:52 AM > >> To: Olivier Hanesse; Jan Beulich > >> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xe= n > >> Users; Dan Magenheimer; Keir Fraser > >> Subject: Re: [Xen-devel] Xen 4 TSC problems > >> > >> On 24/02/2011 14:20, "Olivier Hanesse" > >> wrote: > >> > >>> Both dom0 and domUs are affected by this" jump". > >>> > >>> I expect to see something like "TSC marked as reliable, warp =3D 0". > >>> I got this on newer hardware with same config/distros. > >> It depends on the CPU itself, older CPUs do not have the super-stable > >> TSC > >> features. But that should never cause a massive 3000s time jump. > >> > >>> Is there a way to measure if it is a TSC warp ? to point out a cpu > >> tsc issue ? > >> > >> The TSC warps or out-of-sync issues that we could reasonably expect > >> would be > >> on the order of microseconds. A 3000s warp is something else entirely. > >> Xen > >> is very confused and/or some TSC or platform timer has jumped a long > >> way > >> (indicating a hardware/firmware issue). > >> > >> -- Keir > >> > >>! ;> 2011/2/24 Jan Beulich > > >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse > >> wrote: > >>>>> I tried to turn off cstates with max_cstate=3D0 without success > >> (still "not > >>>>> reliable"). > >>>>> > >>>>> With cpuidle=3D0, I also got : > >>>>> > >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not > >> reliable, > >>>>> warp=3D3022 (count=3D1) > >>>> This message by itself isn't telling much I believe. > >>>> > >>>>> xm info | grep command > >>>>> xen_commandline : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall > >> guest_loglvl=3Dall > >>>>> dom0_max_vcpus=3D1 dom0_vcp! us_pin console=3Dvga,com1 com1=3D19200= ,8n1 > > >>>>> > >>>>> Keir : > >>>>> > >>>>> Using clocksource=3Dpit : > >>>>> > >>>>> (XEN) Platform timer is 1.193MHz PIT > >>>>> > >>>>> I also got : > >>>>> > >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not > >> reliable, > >>>>> warp=3D3262 (count=3D2) > >>>> The question is whether any of this eliminates the time jumps seen > >>>> by your DomU-s (from your past mails I wasn't actually sure whether > >>>> Dom0 also experienced this problem, albeit it would be odd if it > >> didn't). > >>>> Jan > >>>> > >>>> Jan > >>>> > >>> > >> > > > --0015175cb0c6bbce5f049d59411e Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Keir :=A0

Yes, it is "under progress"= .=A0

To make this change, I had to reboot every server, so it is = taking time (production server :()

So i was hoping to find a quick method to mitigate this issue on domUs= while rebooting servers.

As this bug happens once= or twice per server since October, I can't say that right now that cha= nging platform timer to PIT fixed it. I have to wait (I hope forever!) this= bug to happen again on a 'patched' server ...=A0

But even with clcoksource=3Dpit, I am seeing some warp= =3D3000+ in debug message ? I guess it is not a good sign, is it ?

Jan : I was hoping to find a way to make the domU clocksou= rce more "independent" like with xen3.2.

2011/2/28 Dan Mage= nheimer <dan.magenheimer@oracle.com>

Hi Olivier =96
=A0<= /span>
It is the Xen clocksource th= at you want to try to change, not the dom0 clocksource.=A0 To do this, you = need to specify =93clocksource=3Dpit=94 on the Xen boot line (and reboot), = not the dom0 boot line.

=A0
I believe Mark Adams played with tsc_mode to see if it solved! his (similar? identical?) problem last year, and it didn=92t make any differenc= e.

Please try booting Xen with =93clocksource=3Dpit=94 and ensure = that =93Platform timer is 1.19MHz PIT=94 appears in the Xen boot messages.= =A0 If the 50min jump does not appear again, it would point to a problem in= the hpet, either hardware or software.

=A0
Thanks,
Dan

=A0

From:= Olivier Hanesse [mailto:olivier.hanesse@gmail.com= ]
Sent: Monday, February 28, 2011 7:37 AM
To: Jeremy Fitzhar= dinge
Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; = xen-deve= l@lists.xensource.com; Xen Users; Keir Fraser

Subject: Re: [Xen-devel] Xen 4= TSC problems
=A0
Hello,

=A0
It happened agai= n twice this weekend.
=A0
What about setting "tsc_mode=3D2" for my vms ? Should this = mode prevent this bug (coming from a bad emulated tsc due to firmware issue= ? is it possible ?) from affecting time in domUs ?

=A0
= Setting clocksource=3Dpit, make 'tsc' available in "/sys/devic= es/system/clocksource/clocksource0/available_clocksource" (otherwise o= nly xen is available, is it normal ? ).=A0

=A0
= Should I bypass xen clocksource and use tsc as a clocksource for dom0/domU = ? or =A0will it be worsed ?
=A0

Regards
=A0
=
Olivier
=A0
2011/2/24 Jeremy Fitzhardinge <jeremy@goop.org>

On 02/24/2011 09= :43 AM, Dan Magenheimer wrote:
> Just a wild guess, but this in Olivi= er's posted output:
>
> (XEN) Platform timer appears to hav= e unexpectedly wrapped 10 or more times.
>
> and the fact that a 32-bit HPET wrap is ~300 seconds and, with= the
> "10 or more times", 10 * 300 seconds is 3000 seconds= , might be a clue
> (or a complete red herring, but I thought it wort= h mentioning).
>
> Mark and Olivier, it would be interesting to know if you are> using the same processor/system.
It = definitely seems like some kind of problem on the host system rather
tha= n anything in the guests themselves. ! =A0If the platform timer is
misbehaving, then Xen could be completely screwing= up the pvclock
calibration which it then passes to guests.

Could= it be one of those "platform clock stops in certain power states"= ;
problems?

=A0 =A0J

>= ;> -----Original Message-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com]=
>> Sent: Thursday, February 24, 2011 7:52 AM
>> To: Olivier = Hanesse; Jan Beulich
>> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lis= ts.xensource.com; Xen
>> Users; Dan Magenheimer; Keir Fraser
>> Subject: Re: [Xen-= devel] Xen 4 TSC problems
>>
>> On 24/02/2011 14:20, &quo= t;Olivier Hanesse" <olivier.hanesse@gmail.com>
>> wrote:
>>
>>> Both dom0 and domUs are affected by this= " jump".
>>>
>>> I expect to see somethin= g like "TSC marked as reliable, warp =3D 0".
>>> I go= t this on newer hardware with same config/distros.
>> It depends on the CPU itself, older CPUs do not have the super-sta= ble
>> TSC
>> features. But that should never cause a mas= sive 3000s time jump.
>>
>>> Is there a way to measure= if it is a TSC warp ? to point out a cpu
>> tsc issue ?
>>
>> The TSC warps or out-of-sync i= ssues that we could reasonably expect
>> would be
>> on t= he order of microseconds. A 3000s warp is something else entirely.
>&= gt; Xen
>> is very confused and/or some TSC or platform timer has jumped a lo= ng
>> way
>> (indicating a hardware/firmware issue).
&= gt;>
>> =A0-- Keir
>>
>&gt! ;> 2011/2/24 Jan Beulich <JBeulich@novell.com>

>>>>&g= t;>> On 24.02.11 at 12:57, Olivier Hanesse <olivier.hanesse@gmail.com><= br> >> wrote:
>>>>> I tried to turn off cstates with ma= x_cstate=3D0 without success
>> (still "not
>>>&g= t;> reliable").
>>>>>
>>>>> Wit= h cpuidle=3D0, I also got :
>>>>>
>>>>> (XEN) TSC has constant rate, d= eep Cstates possible, so not
>> reliable,
>>>>> = warp=3D3022 (count=3D1)
>>>> This message by itself isn'= t telling much I believe.
>>>>
>>>>> xm info | grep command
>>= >>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle=3D0 l= oglvl=3Dall
>> guest_loglvl=3Dall
>>>>> do= m0_max_vcpus=3D1 dom0_vcp! us_pin console=3Dvga,com1 com1=3D19200,8n1

>>>>&g= t;
>>>>> Keir :
>>>>>
>>>&g= t;> Using clocksource=3Dpit :
>>>>>
>>>>= ;> (XEN) Platform timer is 1.193MHz PIT
>>>>>
>>>>> I also got :
>>>&g= t;>
>>>>> (XEN) TSC has constant rate, deep Cstates po= ssible, so not
>> reliable,
>>>>> warp=3D3262 (c= ount=3D2)
>>>> The question is whether any of this eliminates the time ju= mps seen
>>>> by your DomU-s (from your past mails I wasn= 9;t actually sure whether
>>>> Dom0 also experienced this pr= oblem, albeit it would be odd if it
>> didn't).
>>>> Jan
>>>>
>&g= t;>> Jan
>>>>
>>>
>>
=
=A0

--0015175cb0c6bbce5f049d59411e-- --===============1988755835== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users --===============1988755835==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: [Xen-devel] Xen 4 TSC problems Date: Mon, 28 Feb 2011 07:30:30 -0800 (PST) Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <49766a9d-0e37-46d2-9497-33c2e981a871@default AANLkTikptc2POrKQgJuoVZRdwJTo64DJ_hm12KuPky4D@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1891310359==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Olivier Hanesse Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============1891310359== Content-Type: multipart/alternative; boundary="__129890705141018905abhmt020" --__129890705141018905abhmt020 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Hi Olivier - =20 By "warp=3D3000+ in debug message" do you mean the Xen boot message "TSC h= as constant rate..., warp =3D NNNN"? =20 If so, this is a very different "warp" measured in cycles, not in seconds, = so 3000 is more like a microsecond not an hour, and this is normal (not a b= ad sign). =20 Dan =20 From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]=20 Sent: Monday, February 28, 2011 8:23 AM To: Dan Magenheimer Cc: Jeremy Fitzhardinge; Keir Fraser; Jan Beulich; Mark Adams; xen-devel@li= sts.xensource.com; Xen Users; Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems =20 Keir :=20 =20 Yes, it is "under progress".=20 To make this change, I had to reboot every server, so it is taking time (pr= oduction server :() So i was hoping to find a quick method to mitigate this issue on domUs whil= e rebooting servers. =20 As this bug happens once or twice per server since October, I can't say tha= t right now that changing platform timer to PIT fixed it. I have to wait (I= hope forever!) this bug to happen again on a 'patched' server ...=20 =20 But even with clcoksource=3Dpit, I am seeing some warp=3D3000+ in debug mes= sage ? I guess it is not a good sign, is it ? =20 Jan : I was hoping to find a way to make the domU clocksource more "indepen= dent" like with xen3.2. =20 =20 2011/2/28 Dan Magenheimer Hi Olivier - =20 It is the Xen clocksource that you want to try to change, not the dom0 cloc= ksource. To do this, you need to specify "clocksource=3Dpit" on the Xen bo= ot line (and reboot), not the dom0 boot line. =20 I believe Mark Adams played with tsc_mode to see if it solved! his (similar= ? identical?) problem last year, and it didn't make any difference. Please try booting Xen with "clocksource=3Dpit" and ensure that "Platform t= imer is 1.19MHz PIT" appears in the Xen boot messages. If the 50min jump d= oes not appear again, it would point to a problem in the hpet, either hardw= are or software. =20 Thanks, Dan =20 From: Olivier Hanesse [mailto:HYPERLINK "mailto:olivier.hanesse@gmail.com" = \nolivier.hanesse@gmail.com]=20 Sent: Monday, February 28, 2011 7:37 AM To: Jeremy Fitzhardinge Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; HYPERLINK "mailt= o:xen-devel@lists.xensource.com" \nxen-devel@lists.xensource.com; Xen Users= ; Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems =20 Hello, =20 It happened again twice this weekend. =20 What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent thi= s bug (coming from a bad emulated tsc due to firmware issue ? is it possibl= e ?) from affecting time in domUs ? =20 Setting clocksource=3Dpit, make 'tsc' available in "/sys/devices/system/clo= cksource/clocksource0/available_clocksource" (otherwise only xen is availab= le, is it normal ? ).=20 =20 Should I bypass xen clocksource and use tsc as a clocksource for dom0/domU = ? or will it be worsed ? =20 Regards =20 Olivier =20 2011/2/24 Jeremy Fitzhardinge On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > Just a wild guess, but this in Olivier's posted output: > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more time= s. > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > (or a complete red herring, but I thought it worth mentioning). > > Mark and Olivier, it would be interesting to know if you are > using the same processor/system. It definitely seems like some kind of problem on the host system rather than anything in the guests themselves. ! If the platform timer is misbehaving, then Xen could be completely screwing up the pvclock calibration which it then passes to guests. Could it be one of those "platform clock stops in certain power states" problems? J >> -----Original Message----- >> From: Keir Fraser [mailto:HYPERLINK "mailto:keir.xen@gmail.com" \nkeir.x= en@gmail.com] >> Sent: Thursday, February 24, 2011 7:52 AM >> To: Olivier Hanesse; Jan Beulich >> Cc: Mark Adams; Jeremy Fitzhardinge; HYPERLINK "mailto:xen-devel@lists.x= ensource.com" \nxen-devel@lists.xensource.com; Xen >> Users; Dan Magenheimer; Keir Fraser >> Subject: Re: [Xen-devel] Xen 4 TSC problems >> >> On 24/02/2011 14:20, "Olivier Hanesse" >> wrote: >> >>> Both dom0 and domUs are affected by this" jump". >>> >>> I expect to see something like "TSC marked as reliable, warp =3D 0". >>> I got this on newer hardware with same config/distros. >> It depends on the CPU itself, older CPUs do not have the super-stable >> TSC >> features. But that should never cause a massive 3000s time jump. >> >>> Is there a way to measure if it is a TSC warp ? to point out a cpu >> tsc issue ? >> >> The TSC warps or out-of-sync issues that we could reasonably expect >> would be >> on the order of microseconds. A 3000s warp is something else entirely. >> Xen >> is very confused and/or some TSC or platform timer has jumped a long >> way >> (indicating a hardware/firmware issue). >> >> -- Keir >> >>! ;> 2011/2/24 Jan Beulich >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse >> wrote: >>>>> I tried to turn off cstates with max_cstate=3D0 without success >> (still "not >>>>> reliable"). >>>>> >>>>> With cpuidle=3D0, I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3D3022 (count=3D1) >>>> This message by itself isn't telling much I believe. >>>> >>>>> xm info | grep command >>>>> xen_commandline : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall >> guest_loglvl=3Dall >>>>> dom0_max_vcpus=3D1 dom0_vcp! us_pin console=3Dvga,com1 com1=3D19200,8= n1 >>>>> >>>>> Keir : >>>>> >>>>> Using clocksource=3Dpit : >>>>> >>>>> (XEN) Platform timer is 1.193MHz PIT >>>>> >>>>> I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3D3262 (count=3D2) >>>> The question is whether any of this eliminates the time jumps seen >>>> by your DomU-s (from your past mails I wasn't actually sure whether >>>> Dom0 also experienced this problem, albeit it would be odd if it >> didn't). >>>> Jan >>>> >>>> Jan >>>> >>> >> =20 =20 --__129890705141018905abhmt020 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: quoted-printable
Hi Ol= ivier –
&nb= sp;
By “warp=3D3000+ = in debug message” do you mean the Xen boot message “TSC= has constant rate..., warp =3D NNNN”?

<= span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#= 1F497D'>If so, this is a very different “warp” measured in cy= cles, not in seconds, so 3000 is more like a microsecond not an hour, ! =20 and this is normal (not a bad sign).

Dan<= /o:p>

From: Olivier Hanesse [ma= ilto:olivier.hanesse@gmail.com]
Sent: Monday, February 28, 201= 1 8:23 AM
To: Dan Magenheimer
Cc: Jeremy Fitzhardinge= ; Keir Fraser; Jan Beulich; Mark Adams; xen-devel@lists.xensource.com; Xen Users; Keir Fraser
Subject: = Re: [Xen-devel] Xen 4 TSC problems

Keir :&nbs= p;

Yes, it is "under progress". =
To make this change, I had= to reboot every server, so it is taking time (production server :()=
So i was hoping to find a quick= method to mitigate this issue on domUs while rebooting servers.

As this bug happens once or twice per server since Octo= ber, I can't say that right now that changing platform timer to PIT fixed= it. I have to wait (I hope forever!) this bug to happen again on a 'patc= hed' server ...

Bu= t even with clcoksource=3Dpit, I am seeing some warp=3D3000+ in debug mes= sage ? I guess it is not a good sign, is it ?

J= an : I was hoping to find a way to make the domU clocksource more "i= ndependent" like with xen3.2.

<= /o:p>
2011/2/28 Dan Magenheimer <dan.magenheimer@oracle.com>
Hi Olivier –

It is the Xen clocks= ource that you want to try to change, not the dom0 clocksource. To = do this, you need to specify “clocksource=3Dpit” on the Xen b= oot line (and reboot), not the dom0 boot line.

I believe M= ark Adams played with tsc_mode to see if it solved! his (similar? identic= al?) problem last year, and it didn’t make any difference.

= Please try booting Xen with “clocksource=3Dpit” and ensu! =20 re that “Platform timer is 1.19MHz PIT” appears in the Xen boot messa= ges. If the 50min jump does not appear again, it would point to a p= roblem in the hpet, either hardware or software.

Thanks,
Dan

From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]
Se= nt: Monday, February 28, 2011 7:37 AM
To: Jeremy Fitzhardin= ge
Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; <= a href=3D"mailto:xen-devel@lists.xensource.com" target=3D"_blank">xen-dev= el@lists.xensource.com; Xen Users; Keir Fraser
<= div>

Subject: Re: [Xen-devel] Xen 4 T= SC problems
&nbs= p;
Hello,
<= /o:p>
It happened again twice this weekend.<= /p>

What about sett= ing "tsc_mode=3D2" for my vms ? Should this mode prevent this b= ug (coming from a bad emulated tsc due to firmware issue ? is it possible= ?) from affecting time in domUs ?
&nbs= p;
Setting clocksource=3Dpit, make 'tsc= ' available in "/sys/devices/system/clocksource/clocksource0/availab= le_clocksource" (otherwise only xen is available, is it norma! =20 l ? ).

Should I bypass xen clocksource and use tsc as a clocksource f= or dom0/domU ? or will it be worsed ?

Regards

Olivier

=
2011/2/24 Je= remy Fitzhardinge <jeremy@goop.org>
On 02/24/2011 09:43 AM, Da= n Magenheimer wrote:
> Just a wild guess, but this in Olivier's pos= ted output:
>
> (XEN) Platform timer appears to have unexpect= edly wrapped 10 or more times.
>
> and the fact that a 32-bit= HPET wrap is ~300 seconds and, with the
> "10 or more times&q= uot;, 10 * 300 seconds is 3000 seconds, might be a clue
> (or a com= plete red herring, but I thought it worth mentioning).
>
> Ma= rk and Olivier, it would be interesting to know if you are
> using = the same processor/system.
It definitely see= ms like some kind of problem on the host system rather
than anything in the guests themselves. ! If the platform= timer is
misbehaving, then Xen could be completely screwing up the pv= clock
calibration which it then passes to guests.

Could it be o= ne of those "platform clock stops in certain power states"
p= roblems?

J

>> -----Original Message-----
>&= gt; From: Keir Fraser [mailto:keir.xen@gmail.com]
>> Sent: Thursday, February= 24, 2011 7:52 AM
>> To: Olivier Hanesse; Jan Beulich
>>= ; Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen>> Users; Dan Magenheimer; Keir Fraser
>> Subject: Re: [= Xen-devel] Xen 4 TSC problems
>>
>> On 24/02/2011 14:20= , "Olivier Hanesse" <olivier.hanes= se@gmail.com>
>> wrote:
>>
>>> Both = dom0 and domUs are affected by this" jump".
>>>
= >>> I expect to see something like "TSC marked as reliable,= warp =3D 0".
>>> I got this on newer hardware with same= config/distros.
>> It depends on the CPU itself, older CPUs do = not have the super-stable
>> TSC
>> features. But that = should never cause a massive 3000s time jump.
>>
>>>= Is there a way to measure if it is a TSC warp ? to point out a cpu
&g= t;> tsc issue ?
>>
>> The TSC warps or out-of-sync i= ssues that we could reasonably expect
>> would be
>> on= the order of microseconds. A 3000s warp is something else entirely.
&= gt;> Xen
>> is very confused and/or some TSC or platform time= r has jumped a long
>> way
>> =20 (indicating a hardware/firmware issue).
>>
>> -- = Keir
>>
>&= gt! ;> 2011/2/24 Jan Beulich <JBeulich@novell.com>

>>>>>>> On 24.02.11 at 12:57, Oli= vier Hanesse <olivier.hanesse@gmail.com>
>> wrote:
>>&g= t;>> I tried to turn off cstates with max_cstate=3D0 without succes= s
>> (still "not
>>>>> reliable").>>>>>
>>>>> With cpuidle=3D0, I also g= ot :
>>>>>
>>>>> (XEN) TSC has consta= nt rate, deep Cstates possible, so not
>> reliable,
>>&= gt;>> warp=3D3022 (count=3D1)
>>>> This message by i= tself isn't telling much I believe.
>>>>
>>>>> xm info | grep comma= nd
>>>>> xen_commandline : d= om0_mem=3D512M cpuidle=3D0 loglvl=3Dall
>> guest_loglvl=3Dall
>>>>> dom0_max_vcpu= s=3D1 dom0_vcp! us_pin console=3Dvga,com1 com1=3D19200,8n1
=

>>>>>
>>>>>= ; Keir :
>>>>>
>>>>> Using clocksourc= e=3Dpit :
>>>>>
>>>>> (XEN) Platform = timer is 1.193MHz PIT
>>>>>
>>>>> I a= lso got :
>>>>>
>>>>> (XEN) TSC has c= onstant rate, deep Cstates possible, so not
>> reliable,
>= >>>> warp=3D3262 (count=3D2)
>>>> The question= is whether any of this eliminates the time jumps seen
>>>>= ; by your DomU-s (from your past mails I wasn't actually sure whether
>>>> Dom0 also experienced this problem, albeit it= would be odd if it
>> didn't).
>>>> Jan
>&= gt;>>
>>>> Jan
>>>>
>>>>>

--__129890705141018905abhmt020-- --===============1891310359== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users --===============1891310359==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Mon, 28 Feb 2011 15:39:48 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Olivier Hanesse , Dan Magenheimer Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On 28/02/2011 15:23, "Olivier Hanesse" wrote: > Keir :=A0 >=20 > Yes, it is "under progress".=A0 > To make this change, I had to reboot every server, so it is taking time > (production server :() > So i was hoping to find a quick method to mitigate this issue on domUs wh= ile > rebooting servers. >=20 > As this bug happens once or twice per server since October, I can't say t= hat > right now that changing platform timer to PIT fixed it. I have to wait (I= hope > forever!) this bug to happen again on a 'patched' server ...=A0 >=20 > But even with clcoksource=3Dpit, I am seeing some warp=3D3000+ in debug messa= ge ? > I guess it is not a good sign, is it ? Better not to have it, but honestly you're very unlikely to see any problem from it. It's totally unrelated to the 3000-second time jumps. -- Keir > Jan : I was hoping to find a way to make the domU clocksource more > "independent" like with xen3.2. >=20 >=20 > 2011/2/28 Dan Magenheimer >> Hi Olivier =AD >> =A0 >> It is the Xen clocksource that you want to try to change, not the dom0 >> clocksource.=A0 To do this, you need to specify =B3clocksource=3Dpit=B2 on the X= en >> boot line (and reboot), not the dom0 boot line. >> =A0 >> I believe Mark Adams played with tsc_mode to see if it solved! his (simi= lar? >> identical?) problem last year, and it didn=B9t make any difference. >>=20 >> Please try booting Xen with =B3clocksource=3Dpit=B2 and ensure that =B3Platform = timer >> is 1.19MHz PIT=B2 appears in the Xen boot messages.=A0 If the 50min jump doe= s not >> appear again, it would point to a problem in the hpet, either hardware o= r >> software. >> =A0 >> Thanks, >> Dan >> =A0 >>=20 >> From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com] >> Sent: Monday, February 28, 2011 7:37 AM >> To: Jeremy Fitzhardinge >> Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; >> xen-devel@lists.xensource.com; Xen Users; Keir Fraser >>=20 >>=20 >> Subject: Re: [Xen-devel] Xen 4 TSC problems >> =A0 >>=20 >> Hello, >>=20 >> =A0 >> It happened again twice this weekend. >>=20 >> =A0 >>=20 >> What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent th= is >> bug (coming from a bad emulated tsc due to firmware issue ? is it possib= le ?) >> from affecting time in domUs ? >>=20 >> =A0 >>=20 >> Setting clocksource=3Dpit, make 'tsc' available in >> "/sys/devices/system/clocksource/clocksource0/available_clocksource" >> (otherwise only xen is available, is it normal ? ).=A0 >>=20 >> =A0 >>=20 >> Should I bypass xen clocksource and use tsc as a clocksource for dom0/do= mU ? >> or =A0will it be worsed ? >>=20 >> =A0 >>=20 >> Regards >>=20 >> =A0 >>=20 >> Olivier >>=20 >> =A0 >>=20 >> 2011/2/24 Jeremy Fitzhardinge >>=20 >> On 02/24/2011 09:43 AM, Dan Magenheimer wrote: >>> Just a wild guess, but this in Olivier's posted output: >>>=20 >>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more ti= mes. >>>=20 >>> and the fact that a 32-bit HPET wrap is ~300 seconds and, with the >>> "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue >>> (or a complete red herring, but I thought it worth mentioning). >>>=20 >>> Mark and Olivier, it would be interesting to know if you are >>> using the same processor/system. >> It definitely seems like some kind of problem on the host system rather >> than anything in the guests themselves. ! =A0If the platform timer is >> misbehaving, then Xen could be completely screwing up the pvclock >> calibration which it then passes to guests. >>=20 >> Could it be one of those "platform clock stops in certain power states" >> problems? >>=20 >>=20 >> =A0 =A0J >>=20 >>>> -----Original Message----- >>>> From: Keir Fraser [mailto:keir.xen@gmail.com] >>>> Sent: Thursday, February 24, 2011 7:52 AM >>>> To: Olivier Hanesse; Jan Beulich >>>> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xe= n >>>> Users; Dan Magenheimer; Keir Fraser >>>> Subject: Re: [Xen-devel] Xen 4 TSC problems >>>>=20 >>>> On 24/02/2011 14:20, "Olivier Hanesse" >>>> wrote: >>>>=20 >>>>> Both dom0 and domUs are affected by this" jump". >>>>>=20 >>>>> I expect to see something like "TSC marked as reliable, warp =3D 0". >>>>> I got this on newer hardware with same config/distros. >>>> It depends on the CPU itself, older CPUs do not have the super-stable >>>> TSC >>>> features. But that should never cause a massive 3000s time jump. >>>>=20 >>>>> Is there a way to measure if it is a TSC warp ? to point out a cpu >>>> tsc issue ? >>>>=20 >>>> The TSC warps or out-of-sync issues that we could reasonably expect >>>> would be >>>> on the order of microseconds. A 3000s warp is something else entirely. >>>> Xen >>>> is very confused and/or some TSC or platform timer has jumped a long >>>> way >>>> (indicating a hardware/firmware issue). >>>>=20 >>>> =A0-- Keir >>>>=20 >>> >! ;> 2011/2/24 Jan Beulich >>=20 >>>>>>>>> On 24.02.11 at 12:57, Olivier Hanesse >>>> wrote: >>>>>>> I tried to turn off cstates with max_cstate=3D0 without success >>>> (still "not >>>>>>> reliable"). >>>>>>>=20 >>>>>>> With cpuidle=3D0, I also got : >>>>>>>=20 >>>>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >>>> reliable, >>>>>>> warp=3D3022 (count=3D1) >>>>>> This message by itself isn't telling much I believe. >>>>>>=20 >>>>>>> xm info | grep command >>>>>>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall >>>> guest_loglvl=3Dall >>>>>>> dom0_max_vcpus=3D1 dom0_vcp! us_pin console=3Dvga,com1 com1=3D19200,8n1 >>=20 >>>>>>>=20 >>>>>>> Keir : >>>>>>>=20 >>>>>>> Using clocksource=3Dpit : >>>>>>>=20 >>>>>>> (XEN) Platform timer is 1.193MHz PIT >>>>>>>=20 >>>>>>> I also got : >>>>>>>=20 >>>>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >>>> reliable, >>>>>>> warp=3D3262 (count=3D2) >>>>>> The question is whether any of this eliminates the time jumps seen >>>>>> by your DomU-s (from your past mails I wasn't actually sure whether >>>>>> Dom0 also experienced this problem, albeit it would be odd if it >>>> didn't). >>>>>> Jan >>>>>>=20 >>>>>> Jan >>>>>>=20 >>>>>=20 >>>>=20 >> =A0 >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: Xen 4 TSC problems Date: Mon, 28 Feb 2011 16:54:29 +0100 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0467043827==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dan Magenheimer Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============0467043827== Content-Type: multipart/alternative; boundary=90e6ba6e80fef3c66d049d59b13d --90e6ba6e80fef3c66d049d59b13d Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yes this is what I mean. I am glad to hear that it isn't a bad sign :) I thought of a bad sign, because on system with "reliable TSC", this counte= r is always 0. 2011/2/28 Dan Magenheimer > Hi Olivier =96 > > > > By =93warp=3D3000+ in debug message=94 do you mean the Xen boot message = =93TSC has > constant rate..., warp =3D NNNN=94? > > > > If so, this is a very different =93warp=94 measured in cycles, not in sec= onds, > so 3000 is more like a microsecond not an hour, ! and this is normal (not= a > bad sign). > > > > Dan > > > > *From:* Olivier Hanesse [mailto:olivier.hanesse@gmail.com] > *Sent:* Monday, February 28, 2011 8:23 AM > *To:* Dan Magenheimer > *Cc:* Jeremy Fitzhardinge; Keir Fraser; Jan Beulich; Mark Adams; > xen-devel@lists.xensource.com; Xen Users; Keir Fraser > > *Subject:* Re: [Xen-devel] Xen 4 TSC problems > > > > Keir : > > > > Yes, it is "under progress". > > To make this change, I had to reboot every server, so it is taking time > (production server :() > > So i was hoping to find a quick method to mitigate this issue on domUs > while rebooting servers. > > > > As this bug happens once or twice per server since October, I can't say > that right now that changing platform timer to PIT fixed it. I have to wa= it > (I hope forever!) this bug to happen again on a 'patched' server ... > > > > But even with clcoksource=3Dpit, I am seeing some warp=3D3000+ in debug m= essage > ? I guess it is not a good sign, is it ? > > > > Jan : I was hoping to find a way to make the domU clocksource more > "independent" like with xen3.2. > > > > > > 2011/2/28 Dan Magenheimer > > Hi Olivier =96 > > > > It is the Xen clocksource that you want to try to change, not the dom0 > clocksource. To do this, you need to specify =93clocksource=3Dpit=94 on = the Xen > boot line (and reboot), not the dom0 boot line. > > > > I believe Mark Adams played with tsc_mode to see if it solved! his > (similar? identical?) problem last year, and it didn=92t make any differe= nce. > > > Please try booting Xen with =93clocksource=3Dpit=94 and ensu! re that =93= Platform > timer is 1.19MHz PIT=94 appears in the Xen boot messages. If the 50min j= ump > does not appear again, it would point to a problem in the hpet, either > hardware or software. > > > > Thanks, > > Dan > > > > *From:* Olivier Hanesse [mailto:olivier.hanesse@gmail.com] > *Sent:* Monday, February 28, 2011 7:37 AM > *To:* Jeremy Fitzhardinge > *Cc:* Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; > xen-devel@lists.xensource.com; Xen Users; Keir Fraser > > > *Subject:* Re: [Xen-devel] Xen 4 TSC problems > > > > Hello, > > > > It happened again twice this weekend. > > > > What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent t= his > bug (coming from a bad emulated tsc due to firmware issue ? is it possibl= e > ?) from affecting time in domUs ? > > > > Setting clocksource=3Dpit, make 'tsc' available in > "/sys/devices/system/clocksource/clocksource0/available_clocksource" > (otherwise only xen is available, is it norma! l ? ). > > > > Should I bypass xen clocksource and use tsc as a clocksource for dom0/dom= U > ? or will it be worsed ? > > > > Regards > > > > Olivier > > > > 2011/2/24 Jeremy Fitzhardinge > > On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > > Just a wild guess, but this in Olivier's posted output: > > > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more > times. > > > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > > (or a complete red herring, but I thought it worth mentioning). > > > > Mark and Olivier, it would be interesting to know if you are > > using the same processor/system. > > It definitely seems like some kind of problem on the host system rather > than anything in the guests themselves. ! If the platform timer is > misbehaving, then Xen could be completely screwing up the pvclock > calibration which it then passes to guests. > > Could it be one of those "platform clock stops in certain power states" > problems? > > > J > > >> -----Original Message----- > >> From: Keir Fraser [mailto:keir.xen@gmail.com] > >> Sent: Thursday, February 24, 2011 7:52 AM > >> To: Olivier Hanesse; Jan Beulich > >> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xe= n > >> Users; Dan Magenheimer; Keir Fraser > >> Subject: Re: [Xen-devel] Xen 4 TSC problems > >> > >> On 24/02/2011 14:20, "Olivier Hanesse" > >> wrote: > >> > >>> Both dom0 and domUs are affected by this" jump". > >>> > >>> I expect to see something like "TSC marked as reliable, warp =3D 0". > >>> I got this on newer hardware with same config/distros. > >> It depends on the CPU itself, older CPUs do not have the super-stable > >> TSC > >> features. But that should never cause a massive 3000s time jump. > >> > >>> Is there a way to measure if it is a TSC warp ? to point out a cpu > >> tsc issue ? > >> > >> The TSC warps or out-of-sync issues that we could reasonably expect > >> would be > >> on the order of microseconds. A 3000s warp is something else entirely. > >> Xen > >> is very confused and/or some TSC or platform timer has jumped a long > >> way > >> (indicating a hardware/firmware issue). > >> > >> -- Keir > >> > > >>! ;> 2011/2/24 Jan Beulich > > > >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse > >> wrote: > >>>>> I tried to turn off cstates with max_cstate=3D0 without success > >> (still "not > >>>>> reliable"). > >>>>> > >>>>> With cpuidle=3D0, I also got : > >>>>> > >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not > >> reliable, > >>>>> warp=3D3022 (count=3D1) > >>>> This message by itself isn't telling much I believe. > >>>> > >>>>> xm info | grep command > >>>>> xen_commandline : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall > >> guest_loglvl=3Dall > > >>>>> dom0_max_vcpus=3D1 dom0_vcp! us_pin console=3Dvga,com1 com1=3D19200= ,8n1 > > > >>>>> > >>>>> Keir : > >>>>> > >>>>> Using clocksource=3Dpit : > >>>>> > >>>>> (XEN) Platform timer is 1.193MHz PIT > >>>>> > >>>>> I also got : > >>>>> > >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not > >> reliable, > >>>>> warp=3D3262 (count=3D2) > >>>> The question is whether any of this eliminates the time jumps seen > >>>> by your DomU-s (from your past mails I wasn't actually sure whether > >>>> Dom0 also experienced this problem, albeit it would be odd if it > >> didn't). > >>>> Jan > >>>> > >>>> Jan > >>>> > >>> > >> > > > > > --90e6ba6e80fef3c66d049d59b13d Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Ye= s this is what I mean.
I am glad to hear that it isn't a bad sign := )
I thought of a bad sign, because on system with "reliable TS= C", this counter is always 0.

2011/2/28 Dan Magenheimer <= span dir=3D"ltr"><dan.mage= nheimer@oracle.com>

Hi Olivier =96
=A0<= /span>
By =93warp=3D3000+ in=A0 deb= ug message=94 do you mean the Xen boot message =93TSC has constant rate...,= warp =3D NNNN=94?
=A0

If so= , this is a very different =93warp=94 measured in cycles, not in seconds, s= o 3000 is more like a microsecond not an hour, ! and this is normal (not a bad sign).
=A0
Dan
=A0

From: Olivier Hanesse [mai= lto:olivier.= hanesse@gmail.com]
Sent: Monday, February 28, 2011 8:23 AM
To: Dan Magenheime= r
Cc: Jeremy Fitzhardinge; Keir Fraser; Jan Beulich; Mark Adams; xen-deve= l@lists.xensource.com; Xen Users; Keir Fraser

Subject: Re: [Xen-devel] Xen 4 TSC problems

=A0
Keir :=A0
=A0
Yes, it is "under progress".=A0
To make this change, I had to reboot every server, so it is taking time (pr= oduction server :()
So i was hoping to= find a quick method to mitigate this issue on domUs while rebooting server= s.

=A0
= As this bug happens once or twice per server since October, I can't say= that right now that changing platform timer to PIT fixed it. I have to wai= t (I hope forever!) this bug to happen again on a 'patched' server = ...=A0

=A0
= But even with clcoksource=3Dpit, I am seeing some warp=3D3000+ in debug mes= sage ? I guess it is not a good sign, is it ?
=A0
Jan : I was hoping to find a way t= o make the domU clocksource more "independent" like with xen3.2.<= /p>
=A0
=A0

2011/2/28 Dan Mag= enheimer <dan.magenheimer@oracle.com>
<= p class=3D"MsoNormal"> Hi Olivier =96
=A0
It is the Xen clocksource that you want to try to change, not the dom0 clo= cksource.=A0 To do this, you need to specify =93clocksource=3Dpit=94 on the= Xen boot line (and reboot), not the dom0 boot line.

=A0
I believe Mark Adams played with tsc_mode to see if it solved! his (sim= ilar? identical?) problem last year, and it didn=92t make any difference.

Please try booting Xen with =93clocksource=3Dpit=94 and ensu! re that =93Platform timer is 1.19MHz PIT=94 appears in the Xen boot messages.=A0 If= the 50min jump does not appear again, it would point to a problem in the h= pet, either hardware or software.
=A0

Thank= s,
Dan
=A0

From: Ol= ivier Hanesse [mailto:olivier.hanesse@gmail.com]
Sent: Monday, February 28, 2011 7:37 AM
To: Jeremy Fitzhar= dinge
Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; = xen-deve= l@lists.xensource.com; Xen Users; Keir Fraser

Subject: Re: [Xen-devel] Xen 4 = TSC problems
<= p class=3D"MsoNormal">=A0
Hello,
=A0
It happened again twice this weekend.
=A0
What= about setting "tsc_mode=3D2" for my vms ? Should this mode preve= nt this bug (coming from a bad emulated tsc due to firmware issue ? is it p= ossible ?) from affecting time in domUs ?

=A0
Setting clocksource=3Dpit, make 'tsc' available in "/sys= /devices/system/clocksource/clocksource0/available_clocksource" (other= wise only xen is available, is it norma! l ? ).=A0
=A0
Should I bypass xen clocksource = and use tsc as a clocksource for dom0/domU ? or =A0will it be worsed ?
<= /div>

=A0
Regards
=
=A0
= Olivier
=
=A0
2011/2/24 Jeremy Fitzhardinge = <jeremy@goop.org>
On 02/2= 4/2011 09:43 AM, Dan Magenheimer wrote:
> Just a wild guess, but this in Olivier's posted output:
>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more = times.
>
> and the fact that a 32-bit HPET wrap is ~300 seconds= and, with the
> "10 or more times", 10 * 300 seconds is 3000 seconds, might = be a clue
> (or a complete red herring, but I thought it worth mentio= ning).
>
> Mark and Olivier, it would be interesting to know if= you are
> using the same processor/system.
It de= finitely seems like some kind of problem on the host system rather
than anything in the guests themselves. ! =A0If the platform time= r is
misbehaving, then Xen could be completely screwing up the pvclockcalibration which it then passes to guests.

Could it be one of tho= se "platform clock stops in certain power states"
problems?

=A0= =A0J

>> -----Original Message-----
>> From: Keir Fra= ser [mailto:keir.xe= n@gmail.com]
>> Sent: Thursday, February 24, 2011 7:52 AM
>> To: Olivier = Hanesse; Jan Beulich
>> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lis= ts.xensource.com; Xen
>> Users; Dan Magenheimer; Keir Fraser
>> Subject: Re: [Xen-= devel] Xen 4 TSC problems
>>
>> On 24/02/2011 14:20, &quo= t;Olivier Hanesse" <olivier.hanesse@gmail.com>
>> wrote:
>>
>>> Both dom0 and domUs are affecte= d by this" jump".
>>>
>>> I expect to see= something like "TSC marked as reliable, warp =3D 0".
>>= > I got this on newer hardware with same config/distros.
>> It depends on the CPU itself, older CPUs do not have the super-sta= ble
>> TSC
>> features. But that should never cause a mas= sive 3000s time jump.
>>
>>> Is there a way to measure= if it is a TSC warp ? to point out a cpu
>> tsc issue ?
>>
>> The TSC warps or out-of-sync i= ssues that we could reasonably expect
>> would be
>> on t= he order of microseconds. A 3000s warp is something else entirely.
>&= gt; Xen
>> is very confused and/or some TSC or platform timer has jumped a lo= ng
>> way
>> (indicating a hardware/firmware issue).
>>
>> =A0-- Keir<= br>>>
>&gt! ;> 2011/2= /24 Jan Beulich <JBeulich@novell.com>

>>>>>>> On 24.02.11 at= 12:57, Olivier Hanesse <olivier.hanesse@gmail.com>
>> wrote:
>= ;>>>> I tried to turn off cstates with max_cstate=3D0 without s= uccess
>> (still "not
>>>>> reliable").
>&= gt;>>>
>>>>> With cpuidle=3D0, I also got :
&= gt;>>>>
>>>>> (XEN) TSC has constant rate, de= ep Cstates possible, so not
>> reliable,
>>>>> warp=3D3022 (count=3D1)
>&= gt;>> This message by itself isn't telling much I believe.
>>>>
>>>>> xm info | grep command=
>>>>> xen_commandline =A0 =A0 =A0 =A0: dom0_mem=3D512M c= puidle=3D0 loglvl=3Dall
>> guest_loglvl=3Dall
>>>>> dom0_max_vcpus=3D1 dom0_vcp! us_pin consol= e=3Dvga,com1 com1=3D19200,8n1

>>>>>
>>>>>= ; Keir :
>>>>>
>>>>> Using clocksource= =3Dpit :
>>>>>
>>>>> (XEN) Platform tim= er is 1.193MHz PIT
>>>>>
>>>>> I also got :
>>>&g= t;>
>>>>> (XEN) TSC has constant rate, deep Cstates po= ssible, so not
>> reliable,
>>>>> warp=3D3262 (c= ount=3D2)
>>>> The question is whether any of this eliminates the time ju= mps seen
>>>> by your DomU-s (from your past mails I wasn= 9;t actually sure whether
>>>> Dom0 also experienced this problem, albeit it w= ould be odd if it
>> didn't).
>>>> Jan
>&= gt;>>
>>>> Jan
>>>>
>>>
>>
=A0
=A0
=

--90e6ba6e80fef3c66d049d59b13d-- --===============0467043827== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0467043827==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "andre.arnold" Subject: Re: Xen 4 TSC problems Date: Fri, 15 Apr 2011 00:51:46 -0700 (PDT) Message-ID: <1302853906683-4304962.post@n5.nabble.com> References: <4D665F3102000078000337B6@vpn.id2.novell.com> <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <4D66AB26.9020000@goop.org> <49766a9d-0e37-46d2-9497-33c2e981a871@default> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hi, we experienced the same problem with xen 4 under debian squeeze on our DELL PowerEdge R815 Servers. Does the "clocksource=pit" setting solve the problem? Cheers Andre -- View this message in context: http://xen.1045712.n5.nabble.com/Xen-4-TSC-problems-tp3396848p4304962.html Sent from the Xen - Dev mailing list archive at Nabble.com. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: Re: Xen 4 TSC problems Date: Fri, 15 Apr 2011 18:31:43 +0200 Message-ID: References: <4D665F3102000078000337B6@vpn.id2.novell.com> <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <4D66AB26.9020000@goop.org> <49766a9d-0e37-46d2-9497-33c2e981a871@default> <1302853906683-4304962.post@n5.nabble.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0276316951==" Return-path: In-Reply-To: <1302853906683-4304962.post@n5.nabble.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "andre.arnold" Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org --===============0276316951== Content-Type: multipart/alternative; boundary=bcaec51b1b4113b71c04a0f794ff --bcaec51b1b4113b71c04a0f794ff Content-Type: text/plain; charset=ISO-8859-1 So far yes. Let's wait another month to tell that setting clocksource=pit was the solution :) 2011/4/15 andre.arnold > Hi, > > we experienced the same problem with xen 4 under debian squeeze on > our DELL PowerEdge R815 Servers. > > Does the "clocksource=pit" setting solve the problem? > > Cheers > Andre > > -- > View this message in context: > http://xen.1045712.n5.nabble.com/Xen-4-TSC-problems-tp3396848p4304962.html > Sent from the Xen - Dev mailing list archive at Nabble.com. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > --bcaec51b1b4113b71c04a0f794ff Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable So far yes.
Let's wait another month to tell that setting clocksour= ce=3Dpit was the solution :)

2011/4/15 andre.arnold <andre.arnold@gmail.com>

Hi,

we experienced the same problem with xen 4 under debian squeeze on
our DELL PowerEdge R815 Servers.

Does the "clocksource=3Dpit" setting solve the problem?

Cheers
Andre

--
View this message in context: http://xen.1045= 712.n5.nabble.com/Xen-4-TSC-problems-tp3396848p4304962.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.= com
http://l= ists.xensource.com/xen-devel

--bcaec51b1b4113b71c04a0f794ff-- --===============0276316951== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0276316951==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Philippe Simonet Subject: Re: Xen 4 TSC problems Date: Tue, 13 Sep 2011 09:16:27 +0200 Message-ID: <4E6F034B.5040102@bluewin.ch> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hi Xen developers i just would like to inform you that I have exactly the same problem with Debian squeeze and xen, with 50 seconds time jump on my dom0 and domu. NTP is running on all dom0/domuU, clocksource is 'xen' everywhere. some messages : syslog : Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc unstable (delta = -2999662111513 ns) xm dmesg : ... (XEN) Platform timer is 14.318MHz HPET ... (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. (XEN) TSC marked as reliable, warp = 0 (count=2) ... I had some contact with Olivier Hanesse and it indicates that he doesn't have any solution for this problem, and all what was proposed in February didn't solved this problem. all suggestions are welcomed. Best regards Philippe config : -------------------------------------- Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 12:46:30 UTC 2011 x86_64 GNU/Linux -------------------------------------- HP DL385 -------------------------------------- vendor_id : AuthenticAMD cpu family : 16 model : 9 model name : AMD Opteron(tm) Processor 6174 stepping : 1 cpu MHz : 3058776.574 cache size : 512 KB fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid amd_dcm pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch nodeid_msr bogomips : 4409.03 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate -------------------------------------- -------------------------------------- PCI : 00:00.0 Host bridge: ATI Technologies Inc RD890 Northbridge only dual slot (2x16) PCI-e GFX Hydra part (rev 02) 00:04.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port D) 00:06.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port F) 00:0a.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (external gfx1 port A) 00:0b.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (NB-SB link) 00:0d.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (external gfx1 port B) 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [IDE mode] 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller 00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller 00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3d) 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control 00:19.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration 00:19.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map 00:19.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller 00:19.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control 00:19.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control 00:1a.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration 00:1a.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map 00:1a.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller 00:1a.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control 00:1a.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control 00:1b.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration 00:1b.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map 00:1b.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller 00:1b.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control 00:1b.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control 01:03.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) 02:00.0 System peripheral: Hewlett-Packard Company iLO3 Slave instrumentation & System support (rev 04) 02:00.2 System peripheral: Hewlett-Packard Company iLO3 Management Processor Support and Messaging (rev 04) 02:00.4 USB Controller: Hewlett-Packard Company Proliant iLO2/iLO3 virtual USB controller (rev 01) 03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01) 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 09:00.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb) 0a:04.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb) 0a:05.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb) 0a:06.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb) 0c:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0c:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0c:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0c:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0f:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0f:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0f:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 0f:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) -------------------------------------- On 8:59 PM, Olivier Hanesse wrote: > Hello > > I've got an issue about time keeping with Xen 4.0 (Debian squeeze release). > > My problem is here (hopefully I amn't the only one, so there might be a bug > somewhere) : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599161#50 > After some times, I got this error : Clocksource tsc unstable (delta = > -2999660334211 ns). It has happened on several servers. > > Looking at the output of "xm debug-key s;" > > (XEN) TSC has constant rate, deep Cstates possible, so not reliable, > warp=2850 (count=3) > > I am using a "Intel(R) Xeon(R) CPU L5420 @ 2.50GHz", which has the > "constant_tsc", but not the "nonstop_tsc" one. > On other systems with a newer cpu with "nonstop_tsc", I don't have this > issue (systems are running the same distros with same config). > > I tried to boot with "max_cstate=0", but nothing changed, my TSC isn't > reliable and after some times, I will got the "50min" issue again. > > I don't understand how a system can do a jump of "50min" in the future. Why > 50min ? it is not 40min, not 1 hour, it is always 50min. > I don't know how to make my TSC "reliable" (I already disable everything > about Powerstate in BIOS Settings). > > Any ideas ? > > Regards > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Xen 4 TSC problems Date: Thu, 15 Sep 2011 04:23:35 -0400 Message-ID: <20110915082335.GB24888@phenom.oracle.com> References: <4E6F034B.5040102@bluewin.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <4E6F034B.5040102@bluewin.ch> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Philippe Simonet Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On Tue, Sep 13, 2011 at 09:16:27AM +0200, Philippe Simonet wrote: > Hi Xen developers > > i just would like to inform you that I have exactly the same problem > with Debian squeeze and xen, with > 50 seconds time jump on my dom0 and domu. NTP is running on all > dom0/domuU, clocksource is 'xen' > everywhere. > > some messages : > syslog : > Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc > unstable (delta = -2999662111513 ns) > > xm dmesg : > ... > (XEN) Platform timer is 14.318MHz HPET > ... > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. > (XEN) TSC marked as reliable, warp = 0 (count=2) > ... > > I had some contact with Olivier Hanesse and it indicates that he > doesn't have any solution for this problem, > and all what was proposed in February didn't solved this problem. Which was the max_cstate=0 ? .. > config : > -------------------------------------- > Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 > 12:46:30 UTC 2011 x86_64 GNU/Linux > -------------------------------------- > HP DL385 > -------------------------------------- > vendor_id : AuthenticAMD > cpu family : 16 > model : 9 > model name : AMD Opteron(tm) Processor 6174 > stepping : 1 > cpu MHz : 3058776.574 OK, that is really messed up. Your house must be on fire for the machine to be running at 3058GHz! Jeremy, this sounds familiar - did we have a patch for this in your 2.6.32 tree? From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Xen 4 TSC problems Date: Thu, 15 Sep 2011 04:24:02 -0400 Message-ID: <20110915082402.GC24888@phenom.oracle.com> References: <4E6F034B.5040102@bluewin.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <4E6F034B.5040102@bluewin.ch> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Philippe Simonet , Jeremy Fitzhardinge Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On Tue, Sep 13, 2011 at 09:16:27AM +0200, Philippe Simonet wrote: > Hi Xen developers Lets try this again, this time Cc-ing Jeremy. > > i just would like to inform you that I have exactly the same problem > with Debian squeeze and xen, with > 50 seconds time jump on my dom0 and domu. NTP is running on all > dom0/domuU, clocksource is 'xen' > everywhere. > > some messages : > syslog : > Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc > unstable (delta = -2999662111513 ns) > > xm dmesg : > ... > (XEN) Platform timer is 14.318MHz HPET > ... > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. > (XEN) TSC marked as reliable, warp = 0 (count=2) > ... > > I had some contact with Olivier Hanesse and it indicates that he > doesn't have any solution for this problem, > and all what was proposed in February didn't solved this problem. Which was the max_cstate=0 ? .. > config : > -------------------------------------- > Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 > 12:46:30 UTC 2011 x86_64 GNU/Linux > -------------------------------------- > HP DL385 > -------------------------------------- > vendor_id : AuthenticAMD > cpu family : 16 > model : 9 > model name : AMD Opteron(tm) Processor 6174 > stepping : 1 > cpu MHz : 3058776.574 OK, that is really messed up. Your house must be on fire for the machine to be running at 3058GHz! Jeremy, this sounds familiar - did we have a patch for this in your 2.6.32 tree? From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: Xen 4 TSC problems Date: Thu, 15 Sep 2011 11:36:13 +0100 Message-ID: References: <4E6F034B.5040102@bluewin.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <4E6F034B.5040102@bluewin.ch> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Philippe Simonet Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On Tue, Sep 13, 2011 at 8:16 AM, Philippe Simonet wrote: > Hi Xen developers > > i just would like to inform you that I have exactly the same problem with > Debian squeeze and xen, with > 50 seconds time jump on my dom0 and domu. NTP is running on all dom0/domu= U, > clocksource is 'xen' > everywhere. > > some messages : > syslog : > Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc unstable > (delta =3D -2999662111513 ns) > > xm dmesg : > ... > (XEN) Platform timer is 14.318MHz HPET > ... > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more time= s. > (XEN) TSC marked as reliable, warp =3D 0 (count=3D2) > ... I haven't been following this conversation, so I don't know if this is relevant, but I've just discovered this morning that the TSC warp check in Xen is done at the wrong time (before any secondary cpus are brought up), and thus always returns warp=3D0. I've submitted a patch to do the check after secondary CPUs are brought up; that should cause Xen to do periodic synchronization of TSCs when there is drift. -George > > I had some contact with Olivier Hanesse and it indicates that he doesn't > have any solution for this problem, > and all what was proposed in February didn't solved this problem. > > all suggestions are welcomed. > > Best regards > > Philippe > > > config : > -------------------------------------- > Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 12:46:30 U= TC > 2011 x86_64 GNU/Linux > -------------------------------------- > HP DL385 > -------------------------------------- > vendor_id =A0 =A0 =A0 : AuthenticAMD > cpu family =A0 =A0 =A0: 16 > model =A0 =A0 =A0 =A0 =A0 : 9 > model name =A0 =A0 =A0: AMD Opteron(tm) Processor 6174 > stepping =A0 =A0 =A0 =A0: 1 > cpu MHz =A0 =A0 =A0 =A0 : 3058776.574 > cache size =A0 =A0 =A0: 512 KB > fpu =A0 =A0 =A0 =A0 =A0 =A0 : yes > fpu_exception =A0 : yes > cpuid level =A0 =A0 : 5 > wp =A0 =A0 =A0 =A0 =A0 =A0 =A0: yes > flags =A0 =A0 =A0 =A0 =A0 : fpu de tsc msr pae mce cx8 apic mtrr mca cmov= pat clflush > mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow > constant_tsc rep_good nonstop_tsc extd_apicid amd_dcm pni cx16 popcnt > hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse > 3dnowprefetch nodeid_msr > bogomips =A0 =A0 =A0 =A0: 4409.03 > TLB size =A0 =A0 =A0 =A0: 1024 4K pages > clflush size =A0 =A0: 64 > cache_alignment : 64 > address sizes =A0 : 48 bits physical, 48 bits virtual > power management: ts ttp tm stc 100mhzsteps hwpstate > -------------------------------------- > > -------------------------------------- > PCI : > 00:00.0 Host bridge: ATI Technologies Inc RD890 Northbridge only dual slo= t > (2x16) PCI-e GFX Hydra part (rev 02) > 00:04.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI > express gpp port D) > 00:06.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI > express gpp port F) > 00:0a.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (externa= l > gfx1 port A) > 00:0b.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (NB-SB > link) > 00:0d.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (externa= l > gfx1 port B) > 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller > [IDE mode] > 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 > Controller > 00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller > 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Control= ler > 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 > Controller > 00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller > 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Control= ler > 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3d) > 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller > 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller > 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge > 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > HyperTransport Configuration > 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Address Map > 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DR= AM > Controller > 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Miscellaneous Control > 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Li= nk > Control > 00:19.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > HyperTransport Configuration > 00:19.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Address Map > 00:19.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DR= AM > Controller > 00:19.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Miscellaneous Control > 00:19.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Li= nk > Control > 00:1a.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > HyperTransport Configuration > 00:1a.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Address Map > 00:1a.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DR= AM > Controller > 00:1a.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Miscellaneous Control > 00:1a.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Li= nk > Control > 00:1b.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > HyperTransport Configuration > 00:1b.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Address Map > 00:1b.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DR= AM > Controller > 00:1b.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor > Miscellaneous Control > 00:1b.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Li= nk > Control > 01:03.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) > 02:00.0 System peripheral: Hewlett-Packard Company iLO3 Slave > instrumentation & System support (rev 04) > 02:00.2 System peripheral: Hewlett-Packard Company iLO3 Management Proces= sor > Support and Messaging (rev 04) > 02:00.4 USB Controller: Hewlett-Packard Company Proliant iLO2/iLO3 virtua= l > USB controller (rev 01) > 03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 > controllers (rev 01) > 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 > Gigabit Ethernet (rev 20) > 04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 > Gigabit Ethernet (rev 20) > 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 > Gigabit Ethernet (rev 20) > 05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 > Gigabit Ethernet (rev 20) > 09:00.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev bb) > 0a:04.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev bb) > 0a:05.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev bb) > 0a:06.0 PCI bridge: PLX Technology, Inc. PEX 8616 16-lane, 4-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev bb) > 0c:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0c:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0c:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0c:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0f:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0f:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0f:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > 0f:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network > Connection (rev 01) > -------------------------------------- > > > > > On 8:59 PM, Olivier Hanesse wrote: >> >> Hello >> >> I've got an issue about time keeping with Xen 4.0 (Debian squeeze >> release). >> >> My problem is here (hopefully I amn't the only one, so there might be a >> bug >> somewhere) : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D599161#5= 0 >> After some times, =A0I got this error : Clocksource tsc unstable (delta = =3D >> -2999660334211 ns). It has happened on several servers. >> >> Looking at the output of "xm debug-key s;" >> >> (XEN) TSC has constant rate, deep Cstates possible, so not reliable, >> warp=3D2850 (count=3D3) >> >> I am using a "Intel(R) Xeon(R) CPU L5420 =A0@ 2.50GHz", which has the >> "constant_tsc", but not the "nonstop_tsc" one. >> On other systems with a newer cpu with "nonstop_tsc", I don't have this >> issue (systems are running the same distros with same config). >> >> I tried to boot with "max_cstate=3D0", but nothing changed, my TSC isn't >> reliable and after some times, I will got the "50min" issue again. >> >> I don't understand how a system can do a jump of "50min" in the future. >> Why >> 50min ? it is not 40min, not 1 hour, it is always 50min. >> I don't know how to make my TSC "reliable" (I already disable everything >> about Powerstate in BIOS Settings). >> >> Any ideas ? >> >> Regards >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Xen 4 TSC problems Date: Thu, 15 Sep 2011 09:24:43 -0700 Message-ID: <4E7226CB.2020706@goop.org> References: <4E6F034B.5040102@bluewin.ch> <20110915082402.GC24888@phenom.oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20110915082402.GC24888@phenom.oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: xen-devel@lists.xensource.com, Philippe Simonet List-Id: xen-devel@lists.xenproject.org On 09/15/2011 01:24 AM, Konrad Rzeszutek Wilk wrote: > On Tue, Sep 13, 2011 at 09:16:27AM +0200, Philippe Simonet wrote: >> Hi Xen developers > Lets try this again, this time Cc-ing Jeremy. >> i just would like to inform you that I have exactly the same problem >> with Debian squeeze and xen, with >> 50 seconds time jump on my dom0 and domu. NTP is running on all >> dom0/domuU, clocksource is 'xen' >> everywhere. >> >> some messages : >> syslog : >> Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc >> unstable (delta = -2999662111513 ns) >> >> xm dmesg : >> ... >> (XEN) Platform timer is 14.318MHz HPET >> ... >> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times. >> (XEN) TSC marked as reliable, warp = 0 (count=2) >> ... >> >> I had some contact with Olivier Hanesse and it indicates that he >> doesn't have any solution for this problem, >> and all what was proposed in February didn't solved this problem. That looks like Xen itself is having problems keeping track of time. If it can't manage it, then there's not much the guest kernels can do about it. > > Which was the max_cstate=0 ? > .. >> config : >> -------------------------------------- >> Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 >> 12:46:30 UTC 2011 x86_64 GNU/Linux >> -------------------------------------- >> HP DL385 >> -------------------------------------- >> vendor_id : AuthenticAMD >> cpu family : 16 >> model : 9 >> model name : AMD Opteron(tm) Processor 6174 >> stepping : 1 >> cpu MHz : 3058776.574 > OK, that is really messed up. Your house must be on fire for the machine > to be running at 3058GHz! > > Jeremy, this sounds familiar - did we have a patch for this in > your 2.6.32 tree? Not that I can think of. All I can suggest from the kernel side is that perhaps some of the ACPI power stuff isn't being set up properly, and that makes the CPU do very strange things with its TSC/power states in general. J From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: Xen 4 TSC problems Date: Thu, 15 Sep 2011 11:38:41 -0700 (PDT) Message-ID: <5224b434-5371-404d-8fed-2665e73dacce@default> References: <4E6F034B.5040102@bluewin.ch CAFLBxZZiGxef5CkH9r+VZeB4h7LPA0nJddumwDRovZaL4zOuvA@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: George Dunlap , Philippe Simonet Cc: jeremy@goop.org, xen-devel@lists.xensource.com, Keir Fraser , Konrad Wilk List-Id: xen-devel@lists.xenproject.org > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > Sent: Thursday, September 15, 2011 4:36 AM > To: Philippe Simonet > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Xen 4 TSC problems >=20 > On Tue, Sep 13, 2011 at 8:16 AM, Philippe Simonet > wrote: > > Hi Xen developers > > > > i just would like to inform you that I have exactly the same problem wi= th > > Debian squeeze and xen, with > > 50 seconds time jump on my dom0 and domu. NTP is running on all dom0/do= muU, > > clocksource is 'xen' > > everywhere. > > > > some messages : > > syslog : > > Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc unstabl= e > > (delta =3D -2999662111513 ns) > > > > xm dmesg : > > ... > > (XEN) Platform timer is 14.318MHz HPET > > ... > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more ti= mes. > > (XEN) TSC marked as reliable, warp =3D 0 (count=3D2) > > ... >=20 > I haven't been following this conversation, so I don't know if this is > relevant, but I've just discovered this morning that the TSC warp > check in Xen is done at the wrong time (before any secondary cpus are > brought up), and thus always returns warp=3D0. I've submitted a patch > to do the check after secondary CPUs are brought up; that should cause > Xen to do periodic synchronization of TSCs when there is drift. Wow, nice catch, George! I wonder if this is the underlying bug for many of the mysterious time problems that have been reported for a year or two now... at least on certain AMD boxes. Any idea when this was introduced? Or has it always been wrong? Dan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Subject: RE: Xen 4 TSC problems Date: Fri, 16 Sep 2011 06:03:22 +0000 Message-ID: References: <4E6F034B.5040102@bluewin.ch> <20110915082402.GC24888@phenom.oracle.com> <4E7226CB.2020706@goop.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <4E7226CB.2020706@goop.org> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: jeremy@goop.org, konrad.wilk@oracle.com Cc: xen-devel@lists.xensource.com, philippe.simonet@bluewin.ch List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Jeremy Fitzhardinge > Sent: Thursday, September 15, 2011 6:25 PM > To: Konrad Rzeszutek Wilk > Cc: xen-devel@lists.xensource.com; Philippe Simonet > Subject: Re: [Xen-devel] Xen 4 TSC problems >=20 > On 09/15/2011 01:24 AM, Konrad Rzeszutek Wilk wrote: > > On Tue, Sep 13, 2011 at 09:16:27AM +0200, Philippe Simonet wrote: > >> Hi Xen developers > > Lets try this again, this time Cc-ing Jeremy. > >> i just would like to inform you that I have exactly the same problem > >> with Debian squeeze and xen, with > >> 50 seconds time jump on my dom0 and domu. NTP is running on all > >> dom0/domuU, clocksource is 'xen' > >> everywhere. > >> > >> some messages : > >> syslog : > >> Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc > >> unstable (delta =3D -2999662111513 ns) > >> > >> xm dmesg : > >> ... > >> (XEN) Platform timer is 14.318MHz HPET ... > >> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more > times. > >> (XEN) TSC marked as reliable, warp =3D 0 (count=3D2) ... > >> > >> I had some contact with Olivier Hanesse and it indicates that he > >> doesn't have any solution for this problem, and all what was proposed > >> in February didn't solved this problem. >=20 > That looks like Xen itself is having problems keeping track of time. If = it can't > manage it, then there's not much the guest kernels can do about it. >=20 > > > > Which was the max_cstate=3D0 ? > > .. > >> config : > >> -------------------------------------- > >> Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 > >> 12:46:30 UTC 2011 x86_64 GNU/Linux > >> -------------------------------------- > >> HP DL385 > >> -------------------------------------- > >> vendor_id : AuthenticAMD > >> cpu family : 16 > >> model : 9 > >> model name : AMD Opteron(tm) Processor 6174 > >> stepping : 1 > >> cpu MHz : 3058776.574 > > OK, that is really messed up. Your house must be on fire for the > > machine to be running at 3058GHz! > > > > Jeremy, this sounds familiar - did we have a patch for this in your > > 2.6.32 tree? >=20 > Not that I can think of. All I can suggest from the kernel side is that = perhaps > some of the ACPI power stuff isn't being set up properly, and that makes = the > CPU do very strange things with its TSC/power states in general. >=20 how can i detect that ?=20 the /proc/acpi/processor path is empty,=20 find /proc/acpi /proc/acpi /proc/acpi/processor /proc/acpi/button /proc/acpi/button/power /proc/acpi/button/power/PWRF /proc/acpi/button/power/PWRF/info /proc/acpi/thermal_zone /proc/acpi/wakeup /proc/acpi/sleep /proc/acpi/fadt /proc/acpi/dsdt /proc/acpi/info /proc/acpi/power_resource /proc/acpi/embedded_controller dmesg | grep -I acpi [ 1.205647] hpet_acpi_add: no address or irqs in _CRS lsmod | grep -i acpi acpi_processor 5087 1 processor,[permanent] > J >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Xen 4 TSC problems Date: Fri, 16 Sep 2011 15:40:18 -0700 Message-ID: <4E73D052.3020901@goop.org> References: <4E6F034B.5040102@bluewin.ch> <20110915082402.GC24888@phenom.oracle.com> <4E7226CB.2020706@goop.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Philippe.Simonet@swisscom.com Cc: xen-devel@lists.xensource.com, philippe.simonet@bluewin.ch, konrad.wilk@oracle.com List-Id: xen-devel@lists.xenproject.org On 09/15/2011 11:03 PM, Philippe.Simonet@swisscom.com wrote: >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >> bounces@lists.xensource.com] On Behalf Of Jeremy Fitzhardinge >> Sent: Thursday, September 15, 2011 6:25 PM >> To: Konrad Rzeszutek Wilk >> Cc: xen-devel@lists.xensource.com; Philippe Simonet >> Subject: Re: [Xen-devel] Xen 4 TSC problems >> >> On 09/15/2011 01:24 AM, Konrad Rzeszutek Wilk wrote: >>> On Tue, Sep 13, 2011 at 09:16:27AM +0200, Philippe Simonet wrote: >>>> Hi Xen developers >>> Lets try this again, this time Cc-ing Jeremy. >>>> i just would like to inform you that I have exactly the same problem >>>> with Debian squeeze and xen, with >>>> 50 seconds time jump on my dom0 and domu. NTP is running on all >>>> dom0/domuU, clocksource is 'xen' >>>> everywhere. >>>> >>>> some messages : >>>> syslog : >>>> Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc >>>> unstable (delta = -2999662111513 ns) >>>> >>>> xm dmesg : >>>> ... >>>> (XEN) Platform timer is 14.318MHz HPET ... >>>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more >> times. >>>> (XEN) TSC marked as reliable, warp = 0 (count=2) ... >>>> >>>> I had some contact with Olivier Hanesse and it indicates that he >>>> doesn't have any solution for this problem, and all what was proposed >>>> in February didn't solved this problem. >> That looks like Xen itself is having problems keeping track of time. If it can't >> manage it, then there's not much the guest kernels can do about it. >> >>> Which was the max_cstate=0 ? >>> .. >>>> config : >>>> -------------------------------------- >>>> Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 >>>> 12:46:30 UTC 2011 x86_64 GNU/Linux >>>> -------------------------------------- >>>> HP DL385 >>>> -------------------------------------- >>>> vendor_id : AuthenticAMD >>>> cpu family : 16 >>>> model : 9 >>>> model name : AMD Opteron(tm) Processor 6174 >>>> stepping : 1 >>>> cpu MHz : 3058776.574 >>> OK, that is really messed up. Your house must be on fire for the >>> machine to be running at 3058GHz! >>> >>> Jeremy, this sounds familiar - did we have a patch for this in your >>> 2.6.32 tree? >> Not that I can think of. All I can suggest from the kernel side is that perhaps >> some of the ACPI power stuff isn't being set up properly, and that makes the >> CPU do very strange things with its TSC/power states in general. >> > how can i detect that ? > > the /proc/acpi/processor path is empty, > > find /proc/acpi > /proc/acpi > /proc/acpi/processor > /proc/acpi/button > /proc/acpi/button/power > /proc/acpi/button/power/PWRF > /proc/acpi/button/power/PWRF/info > /proc/acpi/thermal_zone > /proc/acpi/wakeup > /proc/acpi/sleep > /proc/acpi/fadt > /proc/acpi/dsdt > /proc/acpi/info > /proc/acpi/power_resource > /proc/acpi/embedded_controller > > dmesg | grep -I acpi > [ 1.205647] hpet_acpi_add: no address or irqs in _CRS > > lsmod | grep -i acpi > acpi_processor 5087 1 processor,[permanent] What does "xenpm start 5" say? J From mboxrd@z Thu Jan 1 00:00:00 1970 From: Philippe Simonet Subject: Re: Xen 4 TSC problems Date: Mon, 19 Sep 2011 07:45:22 +0200 Message-ID: <4E76D6F2.5010602@bluewin.ch> References: <4E6F034B.5040102@bluewin.ch> <20110915082402.GC24888@phenom.oracle.com> <4E7226CB.2020706@goop.org> <4E73D052.3020901@goop.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4E73D052.3020901@goop.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Jeremy Fitzhardinge Cc: Philippe.Simonet@swisscom.com, xen-devel@lists.xensource.com, konrad.wilk@oracle.com List-Id: xen-devel@lists.xenproject.org On 9/17/2011 12:40 AM, Jeremy Fitzhardinge wrote: > On 09/15/2011 11:03 PM, Philippe.Simonet@swisscom.com wrote: >>> -----Original Message----- >>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >>> bounces@lists.xensource.com] On Behalf Of Jeremy Fitzhardinge >>> Sent: Thursday, September 15, 2011 6:25 PM >>> To: Konrad Rzeszutek Wilk >>> Cc: xen-devel@lists.xensource.com; Philippe Simonet >>> Subject: Re: [Xen-devel] Xen 4 TSC problems >>> >>> On 09/15/2011 01:24 AM, Konrad Rzeszutek Wilk wrote: >>>> On Tue, Sep 13, 2011 at 09:16:27AM +0200, Philippe Simonet wrote: >>>>> Hi Xen developers >>>> Lets try this again, this time Cc-ing Jeremy. >>>>> i just would like to inform you that I have exactly the same problem >>>>> with Debian squeeze and xen, with >>>>> 50 seconds time jump on my dom0 and domu. NTP is running on all >>>>> dom0/domuU, clocksource is 'xen' >>>>> everywhere. >>>>> >>>>> some messages : >>>>> syslog : >>>>> Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc >>>>> unstable (delta = -2999662111513 ns) >>>>> >>>>> xm dmesg : >>>>> ... >>>>> (XEN) Platform timer is 14.318MHz HPET ... >>>>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more >>> times. >>>>> (XEN) TSC marked as reliable, warp = 0 (count=2) ... >>>>> >>>>> I had some contact with Olivier Hanesse and it indicates that he >>>>> doesn't have any solution for this problem, and all what was proposed >>>>> in February didn't solved this problem. >>> That looks like Xen itself is having problems keeping track of time. If it can't >>> manage it, then there's not much the guest kernels can do about it. >>> >>>> Which was the max_cstate=0 ? >>>> .. >>>>> config : >>>>> -------------------------------------- >>>>> Linux dnsit22.swissptt.ch 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 >>>>> 12:46:30 UTC 2011 x86_64 GNU/Linux >>>>> -------------------------------------- >>>>> HP DL385 >>>>> -------------------------------------- >>>>> vendor_id : AuthenticAMD >>>>> cpu family : 16 >>>>> model : 9 >>>>> model name : AMD Opteron(tm) Processor 6174 >>>>> stepping : 1 >>>>> cpu MHz : 3058776.574 >>>> OK, that is really messed up. Your house must be on fire for the >>>> machine to be running at 3058GHz! >>>> >>>> Jeremy, this sounds familiar - did we have a patch for this in your >>>> 2.6.32 tree? >>> Not that I can think of. All I can suggest from the kernel side is that perhaps >>> some of the ACPI power stuff isn't being set up properly, and that makes the >>> CPU do very strange things with its TSC/power states in general. >>> >> how can i detect that ? >> >> the /proc/acpi/processor path is empty, >> >> find /proc/acpi >> /proc/acpi >> /proc/acpi/processor >> /proc/acpi/button >> /proc/acpi/button/power >> /proc/acpi/button/power/PWRF >> /proc/acpi/button/power/PWRF/info >> /proc/acpi/thermal_zone >> /proc/acpi/wakeup >> /proc/acpi/sleep >> /proc/acpi/fadt >> /proc/acpi/dsdt >> /proc/acpi/info >> /proc/acpi/power_resource >> /proc/acpi/embedded_controller >> >> dmesg | grep -I acpi >> [ 1.205647] hpet_acpi_add: no address or irqs in _CRS >> >> lsmod | grep -i acpi >> acpi_processor 5087 1 processor,[permanent] > What does "xenpm start 5" say? > > J > here it is : root@dnsit22.swissptt.ch ~# xenpm start 5 Timeout set to 5 seconds Start sampling, waiting for CTRL-C or SIGINT or SIGALARM signal ... Elapsed time (ms): 5028 CPU0: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU1: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU2: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU3: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU4: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU5: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU6: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU7: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU8: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU9: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU10: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU11: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU12: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU13: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU14: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU15: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU16: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU17: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU18: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU19: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU20: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU21: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU22: Residency(ms) Avg Res(ms) Avg freq 18 KHz CPU23: Residency(ms) Avg Res(ms) Avg freq 18 KHz From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: Xen 4 TSC problems Date: Mon, 19 Sep 2011 11:39:49 +0100 Message-ID: References: <5224b434-5371-404d-8fed-2665e73dacce@default> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <5224b434-5371-404d-8fed-2665e73dacce@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dan Magenheimer Cc: Keir Fraser , jeremy@goop.org, xen-devel@lists.xensource.com, Philippe Simonet , Konrad Wilk List-Id: xen-devel@lists.xenproject.org On Thu, Sep 15, 2011 at 7:38 PM, Dan Magenheimer wrote: >> I haven't been following this conversation, so I don't know if this is >> relevant, but I've just discovered this morning that the TSC warp >> check in Xen is done at the wrong time (before any secondary cpus are >> brought up), and thus always returns warp=3D0. =A0I've submitted a patch >> to do the check after secondary CPUs are brought up; that should cause >> Xen to do periodic synchronization of TSCs when there is drift. > > Wow, nice catch, George! =A0I wonder if this is the underlying bug > for many of the mysterious time problems that have been reported > for a year or two now... at least on certain AMD boxes. > Any idea when this was introduced? =A0Or has it always been wrong? Well the comment in 20823:89907dab1aef seems to indicate that's where the "assume it's reliable on AMD until proven otherwise" started; that would be January 2010. I looked as far back as 20705:a74aca4b9386, and there the TSC reliability checks were again in init_xen_time(). Figuring out where things were before then is getting into archeology. :-) The comment at the top of init_xen_time() is correct now, but from the time it was first written through 4.1 is was just plain wrong -- it said init_xen_time() happened after all cpus were up, which has never been true. -George From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: Xen 4 TSC problems Date: Thu, 22 Sep 2011 13:07:21 +0100 Message-ID: <4E7B41190200007800057508@nat28.tlf.novell.com> References: <5224b434-5371-404d-8fed-2665e73dacce@default> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: George Dunlap Cc: jeremy@goop.org, xen-devel@lists.xensource.com, Keir Fraser , Konrad Wilk , Dan Magenheimer , Philippe Simonet List-Id: xen-devel@lists.xenproject.org >>> On 19.09.11 at 12:39, George Dunlap = wrote: > The comment at the top of init_xen_time() is correct now, but from the > time it was first written through 4.1 is was just plain wrong -- it > said init_xen_time() happened after all cpus were up, which has never > been true. Not really - CPUs got booted by that time originally (pre-4.0, which is what the old comment said), but not onlined. Prior to the use of smp_call_function() for the TSC reliability check I assume only CPU feature flags got looked at, which was available at that point prior to Keir's re-work of the SMP boot process. Jan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Subject: RE: Xen 4 TSC problems Date: Fri, 30 Sep 2011 06:33:00 +0000 Message-ID: References: <5224b434-5371-404d-8fed-2665e73dacce@default> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: George.Dunlap@eu.citrix.com, dan.magenheimer@oracle.com Cc: olivier.hanesse@gmail.com, jeremy@goop.org, xen-devel@lists.xensource.com, keir@xen.org, konrad.wilk@oracle.com List-Id: xen-devel@lists.xenproject.org Hi Xen developpers i need some good tips to go forward with my TSC problem :=20 first fast the problem :=20 - clock jump 50 minutes forward : (xm dmesg) (XEN) TSC is reliable, synchronization unnecessary (XEN) Platform timer is 14.318MHz HPET (XEN) Platform timer appears to have unexpectedly wrapped 10 or more time= s (syslog) Sep 28 17:45:06 dnsit11 kernel: [1970548.356130] Clocksource tsc unstable = (delta =3D -2999660112689 ns) Sep 11 13:56:50 dnsit22 kernel: [571603.359863] Clocksource tsc unstable (= delta =3D -2999662111513 ns) - I can't reproduce or force the problem - on 2 different HP DL 385 G7, with debian squeeze :=20 xen-hypervisor-4.0-amd64 4.0.1-2 dom0 : linux-image-2.6.32-5-xen-amd64 2.6.32-35 domus : 5 -> 15 debian machines 2 * 12-cores AMD Opteron(tm) Processor 6174 - i have this problem since begin of september, before, the machine were ru= nning since 3 month without problem begin of September, I have done an upgrade (dom0 and domus:) linux-image-2.6.32-5-xen-amd64:amd64 (2.6.32-31, automatic) -> linux-imag= e-2.6.32-5-xen-amd64:amd64 (2.6.32-31, 2.6.32-35) - what is strange : (don't know if there is a link with the problem) /proc/cpuinfo in dom0 gives me :=20 cpu MHz : 3249880.888 --or -- cpu MHz : 2300454.255 .... (different after each reboot) =09 in domu thi value is ok(cpu MHz : 2200.112), the bogomips is also = ok (bogomips : 4400.21) if I start the machine with a non-xen environment, the values are also ok =09 I have now exact the same machine where I can make some tests. Could you give me some tips that I could test or implement ? - hardware problem ? hypervisor problem ? dom0 problem ? - try other hypervisor version ?=20 - try linux-image-3.0.0-1-amd64 3.0.0-3 - try reproducing problem ? (how ?, log it ? ....) all your help is welcomed ! many thanks Philippe > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of George Dunlap > Sent: Monday, September 19, 2011 12:40 PM > To: Dan Magenheimer > Cc: Keir Fraser; jeremy@goop.org; xen-devel@lists.xensource.com; Philippe > Simonet; Konrad Wilk > Subject: Re: [Xen-devel] Xen 4 TSC problems >=20 > On Thu, Sep 15, 2011 at 7:38 PM, Dan Magenheimer > wrote: > >> I haven't been following this conversation, so I don't know if this > >> is relevant, but I've just discovered this morning that the TSC warp > >> check in Xen is done at the wrong time (before any secondary cpus are > >> brought up), and thus always returns warp=3D0. =A0I've submitted a pat= ch > >> to do the check after secondary CPUs are brought up; that should > >> cause Xen to do periodic synchronization of TSCs when there is drift. > > > > Wow, nice catch, George! =A0I wonder if this is the underlying bug for > > many of the mysterious time problems that have been reported for a > > year or two now... at least on certain AMD boxes. > > Any idea when this was introduced? =A0Or has it always been wrong? >=20 > Well the comment in 20823:89907dab1aef seems to indicate that's where the > "assume it's reliable on AMD until proven otherwise" started; that would = be > January 2010. >=20 > I looked as far back as 20705:a74aca4b9386, and there the TSC reliability > checks were again in init_xen_time(). Figuring out where things were bef= ore > then is getting into archeology. :-) >=20 > The comment at the top of init_xen_time() is correct now, but from the ti= me > it was first written through 4.1 is was just plain wrong -- it said > init_xen_time() happened after all cpus were up, which has never been tru= e. >=20 > -George >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: tommics Subject: RE: Xen 4 TSC problems Date: Fri, 30 Sep 2011 02:36:25 -0700 (PDT) Message-ID: <1317375385187-4856420.post@n5.nabble.com> References: <4E6F034B.5040102@bluewin.ch> <5224b434-5371-404d-8fed-2665e73dacce@default> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hey there, just wanted to report that we also experience the same problem since upgrading to xen 4.0.1 on debian squeeze. We are running latest debian stable release. ii libxenstore3.0 4.0.1-2 =20 ii linux-image-2.6.32-5-xen-amd64 2.6.32-35squeeze2 ii linux-image-xen-amd64 2.6.32+29 =20 ii xen-hypervisor-4.0-amd64 4.0.1-2 =20 ii xen-linux-system-2.6-xen-amd64 2.6.32+29 =20 ii xen-linux-system-2.6.32-5-xen-amd64 2.6.32-35squeeze2 ii xen-qemu-dm-4.0 4.0.1-2 =20 ii xen-tools 4.2-1 =20 ii xen-utils-4.0 4.0.1-2 =20 ii xen-utils-common 4.0.0-1 =20 ii xenstore-utils 4.0.1-2 =20 We also have the clock jumping 50 minutes into future.=20 We are running IBM Blades HS21 XM (Type 7995) with Intel(R) Xeon(R) CPU E5345 @ 2.33GHz. We are also running the same configuration on another machine with Intel(R) Xeon(R) CPU X7550 @ 2.00GHz where we dont experience this problems. Also w= e didnt had those bugs running xen 3.1.0 with 2.6.18-5-xen-amd64 kernel. We currently running one blade with disabled HPET, clocksoure=3Dpit and cpuidle=3D0 and another with HPET on and nothing else configured. The main problem debugging this, is to wait for the next error to appear. Until now both machines run fine without time jumping, but we did see that time jump on the machine which is running with hpet enabled and no other settings.=20 We could help debugging here. Regards Thomas P=C3=B6hler -- View this message in context: http://xen.1045712.n5.nabble.com/Xen-4-TSC-pr= oblems-tp3396848p4856420.html Sent from the Xen - Dev mailing list archive at Nabble.com. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: Xen 4 TSC problems Date: Fri, 30 Sep 2011 10:16:44 -0700 (PDT) Message-ID: References: <5224b434-5371-404d-8fed-2665e73dacce@default> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Philippe.Simonet@swisscom.com, George.Dunlap@eu.citrix.com Cc: olivier.hanesse@gmail.com, jeremy@goop.org, xen-devel@lists.xensource.com, keir@xen.org, Konrad Wilk List-Id: xen-devel@lists.xenproject.org > From: Philippe.Simonet@swisscom.com [mailto:Philippe.Simonet@swisscom.com= ] > Subject: RE: [Xen-devel] Xen 4 TSC problems >=20 > Hi Xen developpers >=20 > i need some good tips to go forward with my TSC problem : >=20 > Could you give me some tips that I could test or implement ? > =09- try other hypervisor version ? Hi Phillipe -- It would definitely be worthwhile to see if you can reproduce the problem on the latest xen-unstable bits. (Please make sure that the bug George reported below is fixed in your build.) A lot has changed since 4.0.1. Dan P.S. I will be mostly offline for the next week or so... > > -----Original Message----- > > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > > bounces@lists.xensource.com] On Behalf Of George Dunlap > > Sent: Monday, September 19, 2011 12:40 PM > > To: Dan Magenheimer > > Cc: Keir Fraser; jeremy@goop.org; xen-devel@lists.xensource.com; Philip= pe > > Simonet; Konrad Wilk > > Subject: Re: [Xen-devel] Xen 4 TSC problems > > > > On Thu, Sep 15, 2011 at 7:38 PM, Dan Magenheimer > > wrote: > > >> I haven't been following this conversation, so I don't know if this > > >> is relevant, but I've just discovered this morning that the TSC warp > > >> check in Xen is done at the wrong time (before any secondary cpus ar= e > > >> brought up), and thus always returns warp=3D0. =A0I've submitted a p= atch > > >> to do the check after secondary CPUs are brought up; that should > > >> cause Xen to do periodic synchronization of TSCs when there is drift= . > > > > > > Wow, nice catch, George! =A0I wonder if this is the underlying bug fo= r > > > many of the mysterious time problems that have been reported for a > > > year or two now... at least on certain AMD boxes. > > > Any idea when this was introduced? =A0Or has it always been wrong? > > > > Well the comment in 20823:89907dab1aef seems to indicate that's where t= he > > "assume it's reliable on AMD until proven otherwise" started; that woul= d be > > January 2010. > > > > I looked as far back as 20705:a74aca4b9386, and there the TSC reliabili= ty > > checks were again in init_xen_time(). Figuring out where things were b= efore > > then is getting into archeology. :-) > > > > The comment at the top of init_xen_time() is correct now, but from the = time > > it was first written through 4.1 is was just plain wrong -- it said > > init_xen_time() happened after all cpus were up, which has never been t= rue. > > > > -George > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mauro Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Thu, 27 Sep 2012 17:54:48 +0200 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xen.org Errors-To: xen-users-bounces@lists.xen.org To: Olivier Hanesse Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On 28 February 2011 16:54, Olivier Hanesse wrote: > Yes this is what I mean. > I am glad to hear that it isn't a bad sign :) > I thought of a bad sign, because on system with "reliable TSC", this counter > is always 0. Hey men. I have exactly the same problem. I have two cluster nodes. Server are two HP Proliant DL 580 G4 with four Quad Core Intel(R) Xeon(R) CPU E7330 @ 2.40GHz. I'm running debian squeeze in dom0s end domUs. xm info: host : xen-p01 release : 2.6.32-5-xen-amd64 version : #1 SMP Sun May 6 08:57:29 UTC 2012 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2400 hw_caps : bfebfbff:20100800:00000000:00000940:0004e3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 65532 free_memory : 40317 node_to_cpu : node0:0-15 node_to_memory : node0:40317 node_to_dma32_mem : node0:3256 max_node_id : 0 xen_major : 4 xen_minor : 0 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : placeholder dom0_mem=3072M loglvl=warning guest_loglvl=warning cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) cc_compile_by : ultrotter cc_compile_domain : debian.org cc_compile_date : Sat Sep 8 19:15:46 UTC 2012 xend_config_format : 4 I'm experiencing weekly a clock jump ahead of about 50 minutes on dom0. I'm seriously in trouble because it cause every time a reboot of one of the two nodes clusters. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: Re: [Xen-users] Re: Xen 4 TSC problems Date: Thu, 27 Sep 2012 12:27:53 -0700 (PDT) Message-ID: <65395f62-74e5-4910-b701-8df629c2ce3b@default> References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Mauro , Olivier Hanesse Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org > From: Mauro [mailto:mrsanna1@gmail.com] > Sent: Thursday, September 27, 2012 9:55 AM > To: Olivier Hanesse > Cc: Dan Magenheimer; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Keir Fraser; Jan Beulich; > Keir Fraser; Xen Users; Mark Adams > Subject: Re: [Xen-users] Re: [Xen-devel] Xen 4 TSC problems > > On 28 February 2011 16:54, Olivier Hanesse wrote: > > Yes this is what I mean. > > I am glad to hear that it isn't a bad sign :) > > I thought of a bad sign, because on system with "reliable TSC", this counter > > is always 0. > > Hey men. > I have exactly the same problem. > I have two cluster nodes. > Server are two HP Proliant DL 580 G4 with four Quad Core Intel(R) > Xeon(R) CPU E7330 @ 2.40GHz. > I'm running debian squeeze in dom0s end domUs. Hi Mauro -- There's been a lot of work on clocks since 4.0 (by other Xen developers, not me). I don't think this specific problem was ever reproduced by a developer so I don't think anyone knows if it has been already fixed or not, nor are there any plans to backport all the timer work to 4.0. You might try upgrading your Xen hypervisor to the just-released Xen 4.2 [1] and see if the problem goes away. If the problem still exists in 4.2, it may be easier to get some developer to pay attention to it. It may be specific hardware or processors or power management or firmware or even dom0 kernel, so the first thing to do is try later hypervisor bits. Sorry I can't be more helpful. Good luck! Dan [1] Sorry, I'm not familiar with the 4.0->4.2 upgrade process so you may want to confirm with others. > xm info: > > host : xen-p01 > release : 2.6.32-5-xen-amd64 > version : #1 SMP Sun May 6 08:57:29 UTC 2012 > machine : x86_64 > nr_cpus : 16 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 2400 > hw_caps : > bfebfbff:20100800:00000000:00000940:0004e3bd:00000000:00000001:00000000 > virt_caps : hvm > total_memory : 65532 > free_memory : 40317 > node_to_cpu : node0:0-15 > node_to_memory : node0:40317 > node_to_dma32_mem : node0:3256 > max_node_id : 0 > xen_major : 4 > xen_minor : 0 > xen_extra : .1 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > xen_commandline : placeholder dom0_mem=3072M loglvl=warning > guest_loglvl=warning > cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) > cc_compile_by : ultrotter > cc_compile_domain : debian.org > cc_compile_date : Sat Sep 8 19:15:46 UTC 2012 > xend_config_format : 4 > > I'm experiencing weekly a clock jump ahead of about 50 minutes on dom0. > I'm seriously in trouble because it cause every time a reboot of one > of the two nodes clusters. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: [Xen-users] Re: Xen 4 TSC problems Date: Thu, 27 Sep 2012 23:28:36 +0200 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <65395f62-74e5-4910-b701-8df629c2ce3b@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1496806874817463489==" Return-path: In-Reply-To: <65395f62-74e5-4910-b701-8df629c2ce3b@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dan Magenheimer Cc: Jeremy Fitzhardinge , xen-devel@lists.xensource.com, Keir Fraser , Keir Fraser , Mauro , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============1496806874817463489== Content-Type: multipart/alternative; boundary=e89a8f64347c3ab36704cab59f37 --e89a8f64347c3ab36704cab59f37 Content-Type: text/plain; charset=ISO-8859-1 Hello, >>From my point of view, this was a kind of xen hardware "incompatibility/bug" : I was able to reproduce this bug on more than 50 identical servers, but not on another farm of servers with a different hardware. Xen version, Debian Kernel was exactly the same on both farm. Regards Olivier 2012/9/27 Dan Magenheimer > > From: Mauro [mailto:mrsanna1@gmail.com] > > Sent: Thursday, September 27, 2012 9:55 AM > > To: Olivier Hanesse > > Cc: Dan Magenheimer; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; > Keir Fraser; Jan Beulich; > > Keir Fraser; Xen Users; Mark Adams > > Subject: Re: [Xen-users] Re: [Xen-devel] Xen 4 TSC problems > > > > On 28 February 2011 16:54, Olivier Hanesse > wrote: > > > Yes this is what I mean. > > > I am glad to hear that it isn't a bad sign :) > > > I thought of a bad sign, because on system with "reliable TSC", this > counter > > > is always 0. > > > > Hey men. > > I have exactly the same problem. > > I have two cluster nodes. > > Server are two HP Proliant DL 580 G4 with four Quad Core Intel(R) > > Xeon(R) CPU E7330 @ 2.40GHz. > > I'm running debian squeeze in dom0s end domUs. > > Hi Mauro -- > > There's been a lot of work on clocks since 4.0 (by other Xen developers, > not me). I don't think this specific problem was ever reproduced > by a developer so I don't think anyone knows if it has been > already fixed or not, nor are there any plans to backport all the > timer work to 4.0. > > You might try upgrading your Xen hypervisor to the just-released > Xen 4.2 [1] and see if the problem goes away. If the problem still > exists in 4.2, it may be easier to get some developer to pay attention > to it. It may be specific hardware or processors or power > management or firmware or even dom0 kernel, so the first thing > to do is try later hypervisor bits. > > Sorry I can't be more helpful. Good luck! > > Dan > > [1] Sorry, I'm not familiar with the 4.0->4.2 upgrade process > so you may want to confirm with others. > > > xm info: > > > > host : xen-p01 > > release : 2.6.32-5-xen-amd64 > > version : #1 SMP Sun May 6 08:57:29 UTC 2012 > > machine : x86_64 > > nr_cpus : 16 > > nr_nodes : 1 > > cores_per_socket : 4 > > threads_per_core : 1 > > cpu_mhz : 2400 > > hw_caps : > > bfebfbff:20100800:00000000:00000940:0004e3bd:00000000:00000001:00000000 > > virt_caps : hvm > > total_memory : 65532 > > free_memory : 40317 > > node_to_cpu : node0:0-15 > > node_to_memory : node0:40317 > > node_to_dma32_mem : node0:3256 > > max_node_id : 0 > > xen_major : 4 > > xen_minor : 0 > > xen_extra : .1 > > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > > hvm-3.0-x86_32p hvm-3.0-x86_64 > > xen_scheduler : credit > > xen_pagesize : 4096 > > platform_params : virt_start=0xffff800000000000 > > xen_changeset : unavailable > > xen_commandline : placeholder dom0_mem=3072M loglvl=warning > > guest_loglvl=warning > > cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) > > cc_compile_by : ultrotter > > cc_compile_domain : debian.org > > cc_compile_date : Sat Sep 8 19:15:46 UTC 2012 > > xend_config_format : 4 > > > > I'm experiencing weekly a clock jump ahead of about 50 minutes on dom0. > > I'm seriously in trouble because it cause every time a reboot of one > > of the two nodes clusters. > --e89a8f64347c3ab36704cab59f37 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello,=A0

From my point of view, this was a kind of xen = hardware "incompatibility/bug" : I was able to reproduce this bug= on more than 50 identical servers, but not on another farm of servers with= a different hardware.

Xen version, Debian Kernel was exactly the same on both farm.

Regards

Olivier

2012/9/27 Dan Magenheimer <dan.magenheime= r@oracle.com>

> From: Mauro [mailto:mrsanna1@gmail.com]
> Sent: Thursday, September 27, 2012 9:55 AM
> To: Olivier Hanesse
> Cc: Dan Magenheimer; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Keir Fraser; Jan Be= ulich;
> Keir Fraser; Xen Users; Mark Adams
> Subject: Re: [Xen-users] Re: [Xen-devel] Xen 4 TSC problems

>
> On 28 February 2011 16:54, Olivier Hanesse <olivier.hanesse@gmail.com> wrote:
> > Yes this is what I mean.
> > I am glad to hear that it isn't a bad sign :)
> > I thought of a bad sign, because on system with "reliable TS= C", this counter
> > is always 0.
>
> Hey men.
> I have exactly the same problem.
> I have two cluster nodes.
> Server are two HP Proliant DL 580 G4 with four Quad Core Intel(R)
> Xeon(R) CPU E7330 =A0@ 2.40GHz.
> I'm running debian squeeze in dom0s end domUs.

Hi Mauro --

There's been a lot of work on clocks since 4.0 (by other Xen developers= ,
not me). =A0I don't think this specific problem was ever reproduced
by a developer so I don't think anyone knows if it has been
already fixed or not, nor are there any plans to backport all the
timer work to 4.0.

You might try upgrading your Xen hypervisor to the just-released
Xen 4.2 [1] and see if the problem goes away. =A0If the problem still
exists in 4.2, it may be easier to get some developer to pay attention
to it. =A0It may be specific hardware or processors or power
management or firmware or even dom0 kernel, so the first thing
to do is try later hypervisor bits.

Sorry I can't be more helpful. =A0Good luck!

Dan

[1] Sorry, I'm not familiar with the 4.0->4.2 upgrade process
so you may want to confirm with others.

> xm info:
>
> host =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 : xen-p01
> release =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: 2.6.32-5-xen-amd64
> version =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: #1 SMP Sun May 6 08:57:29 UTC= 2012
> machine =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: x86_64
> nr_cpus =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: 16
> nr_nodes =A0 =A0 =A0 =A0 =A0 =A0 =A0 : 1
> cores_per_socket =A0 =A0 =A0 : 4
> threads_per_core =A0 =A0 =A0 : 1
> cpu_mhz =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0: 2400
> hw_caps =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0:
> bfebfbff:20100800:00000000:00000940:0004e3bd:00000000:00000001:0000000= 0
> virt_caps =A0 =A0 =A0 =A0 =A0 =A0 =A0: hvm
> total_memory =A0 =A0 =A0 =A0 =A0 : 65532
> free_memory =A0 =A0 =A0 =A0 =A0 =A0: 40317
> node_to_cpu =A0 =A0 =A0 =A0 =A0 =A0: node0:0-15
> node_to_memory =A0 =A0 =A0 =A0 : node0:40317
> node_to_dma32_mem =A0 =A0 =A0: node0:3256
> max_node_id =A0 =A0 =A0 =A0 =A0 =A0: 0
> xen_major =A0 =A0 =A0 =A0 =A0 =A0 =A0: 4
> xen_minor =A0 =A0 =A0 =A0 =A0 =A0 =A0: 0
> xen_extra =A0 =A0 =A0 =A0 =A0 =A0 =A0: .1
> xen_caps =A0 =A0 =A0 =A0 =A0 =A0 =A0 : xen-3.0-x86_64 xen-3.0-x86_32p = hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler =A0 =A0 =A0 =A0 =A0: credit
> xen_pagesize =A0 =A0 =A0 =A0 =A0 : 4096
> platform_params =A0 =A0 =A0 =A0: virt_start=3D0xffff800000000000
> xen_changeset =A0 =A0 =A0 =A0 =A0: unavailable
> xen_commandline =A0 =A0 =A0 =A0: placeholder dom0_mem=3D3072M loglvl= =3Dwarning
> guest_loglvl=3Dwarning
> cc_compiler =A0 =A0 =A0 =A0 =A0 =A0: gcc version 4.4.5 (Debian 4.4.5-8= )
> cc_compile_by =A0 =A0 =A0 =A0 =A0: ultrotter
> cc_compile_domain =A0 =A0 =A0: debian.org
> cc_compile_date =A0 =A0 =A0 =A0: Sat Sep =A08 19:15:46 UTC 2012
> xend_config_format =A0 =A0 : 4
>
> I'm experiencing weekly a clock jump ahead of about 50 minutes on = dom0.
> I'm seriously in trouble because it cause every time a reboot of o= ne
> of the two nodes clusters.

--e89a8f64347c3ab36704cab59f37-- --===============1496806874817463489== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1496806874817463489==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mauro Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Thu, 27 Sep 2012 23:42:04 +0200 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default>

<65395f62-74e5-4910-b701-8df629c2ce3b@default> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xen.org Errors-To: xen-users-bounces@lists.xen.org To: Olivier Hanesse Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On 27 September 2012 23:28, Olivier Hanesse wrote: > Hello, > > From my point of view, this was a kind of xen hardware "incompatibility/bug" > : I was able to reproduce this bug on more than 50 identical servers, but > not on another farm of servers with a different hardware. > Xen version, Debian Kernel was exactly the same on both farm. Yes I think so. The problem is where I use debian squeeze with xen 4.0. In another server with the same hardware but with debian lenny and xen 3.0 I have no problems. I've read that a workaround is to set clocksource=pit on the xen boot line in the grub conf, I hope this works because I can't change hardware. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: [Xen-users] Re: Xen 4 TSC problems Date: Sat, 29 Sep 2012 10:08:45 +0200 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default>

<65395f62-74e5-4910-b701-8df629c2ce3b@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0135414908100069518==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Mauro Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org --===============0135414908100069518== Content-Type: multipart/alternative; boundary=e89a8f23585f6dc74e04cad2ae5f --e89a8f23585f6dc74e04cad2ae5f Content-Type: text/plain; charset=ISO-8859-1 It didn't work for me :( clocksource=pit made another "time jump" (don't remember how much, but it was worst than 50min) 2012/9/27 Mauro > On 27 September 2012 23:28, Olivier Hanesse > wrote: > > Hello, > > > > From my point of view, this was a kind of xen hardware > "incompatibility/bug" > > : I was able to reproduce this bug on more than 50 identical servers, but > > not on another farm of servers with a different hardware. > > Xen version, Debian Kernel was exactly the same on both farm. > > Yes I think so. > The problem is where I use debian squeeze with xen 4.0. > In another server with the same hardware but with debian lenny and xen > 3.0 I have no problems. > I've read that a workaround is to set clocksource=pit on the xen boot > line in the grub conf, I hope this works because I can't change hardware. > --e89a8f23585f6dc74e04cad2ae5f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable It didn't work for me :(

clocksource=3Dpit made another "time = jump" (don't remember how much, but it was worst than 50min)

2012/9/27 Mauro <mrsanna1@gmail.com>=

On 27 September 2012 23:28= , Olivier Hanesse <olivier.= hanesse@gmail.com> wrote:

> Hello,
>
> From my point of view, this was a kind of xen hardware "incompati= bility/bug"
> : I was able to reproduce this bug on more than 50 identical servers, = but
> not on another farm of servers with a different hardware.
> Xen version, Debian Kernel was exactly the same on both farm.

Yes I think so.
The problem is where I use debian squeeze with xen 4.0.
In another server with the same hardware but with debian lenny and xen
3.0 I have no problems.
I've read that a workaround is to set clocksource=3Dpit on the xen boot=

line in the grub conf, I hope this works because I can't change h= ardware.

--e89a8f23585f6dc74e04cad2ae5f-- --===============0135414908100069518== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============0135414908100069518==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mauro Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Sat, 29 Sep 2012 11:41:39 +0200 Message-ID: References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default>

<65395f62-74e5-4910-b701-8df629c2ce3b@default>

Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xen.org Errors-To: xen-users-bounces@lists.xen.org To: Olivier Hanesse Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Jan Beulich , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org It's happened another time, system date 50 minutes ahead. There is really no solution? root@xen-p02:~# date sab 29 set 2012, 15.06.25, CEST root@xen-p02:~# hwclock --debug hwclock from util-linux-ng 2.17.2 Using /dev interface to clock. Last drift adjustment done at 1348816781 seconds after 1969 Last calibration done at 1348816781 seconds after 1969 Hardware clock is on UTC time Assuming hardware clock is kept in UTC time. Waiting for clock tick... ...got clock tick Time read from Hardware Clock: 2012/09/29 12:16:58 Hw clock time : 2012/09/29 12:16:58 = 1348921018 seconds since 1969 sab 29 set 2012 14:16:58 CEST -0.751536 seconds root@xen-p02:~# hwclock --show sab 29 set 2012 14:17:12 CEST -0.423643 seconds From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mauro Subject: Re: [Xen-devel] Xen 4 TSC problems Date: Sat, 29 Sep 2012 17:13:08 +0200 Message-ID: References: <4D655A39.2040501@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xen.org Errors-To: xen-users-bounces@lists.xen.org To: Keir Fraser Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Jeremy Fitzhardinge , Olivier Hanesse , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org T24gMjQgRmVicnVhcnkgMjAxMSAwODoxNiwgS2VpciBGcmFzZXIgPGtlaXIueGVuQGdtYWlsLmNv bT4gd3JvdGU6Cj4gUGxlYXNlIHNlbmQgWGVuIGJvb3Qgb3V0cHV0ICh4bSBkbWVzZykuIEdldHRp bmcgaXQgZnJvbSBYZW4gMy4yIGFzIHdlbGwKPiB3b3VsZCBiZSBpbnRlcmVzdGluZywgaWYgeW91 IHN0aWxsIGhhdmUgaXQgaW5zdGFsbGVkIG9uIGFueSBvZiB0aGVzZQo+IG1hY2hpbmVzLgoKSWYg aXQgY2FuIGJlIHVzZWZ1bCBoZXJlIGlzIHhtIGRtZXNmIG9mIHhlbiA0LjAgb24gYSBkZWJpYW4g c3F1ZWV6ZSBzeXN0ZW06CgooWEVOKSBYZW4gdmVyc2lvbiA0LjAuMSAoRGViaWFuIDQuMC4xLTUu NCkgKHVsdHJvdHRlckBkZWJpYW4ub3JnKSAoZ2NjCnZlcnNpb24gNC40LjUgKERlYmlhbiA0LjQu NS04KSApIFNhdCBTZXAgIDggMTk6MTU6NDYgVVRDIDIwMTIKKFhFTikgQm9vdGxvYWRlcjogR1JV QiAxLjk4KzIwMTAwODA0LTE0K3NxdWVlemUxCihYRU4pIENvbW1hbmQgbGluZTogcGxhY2Vob2xk ZXIgZG9tMF9tZW09MzA3Mk0gbG9nbHZsPXdhcm5pbmcKZ3Vlc3RfbG9nbHZsPXdhcm5pbmcKKFhF TikgVmlkZW8gaW5mb3JtYXRpb246CihYRU4pICBWR0EgaXMgdGV4dCBtb2RlIDgweDI1LCBmb250 IDh4MTYKKFhFTikgIFZCRS9EREMgbWV0aG9kczogVjI7IEVESUQgdHJhbnNmZXIgdGltZTogMiBz ZWNvbmRzCihYRU4pIERpc2MgaW5mb3JtYXRpb246CihYRU4pICBGb3VuZCAyIE1CUiBzaWduYXR1 cmVzCihYRU4pICBGb3VuZCAyIEVERCBpbmZvcm1hdGlvbiBzdHJ1Y3R1cmVzCihYRU4pIFhlbi1l ODIwIFJBTSBtYXA6CihYRU4pICAwMDAwMDAwMDAwMDAwMDAwIC0gMDAwMDAwMDAwMDA5ZjQwMCAo dXNhYmxlKQooWEVOKSAgMDAwMDAwMDAwMDA5ZjQwMCAtIDAwMDAwMDAwMDAwYTAwMDAgKHJlc2Vy dmVkKQooWEVOKSAgMDAwMDAwMDAwMDBmMDAwMCAtIDAwMDAwMDAwMDAxMDAwMDAgKHJlc2VydmVk KQooWEVOKSAgMDAwMDAwMDAwMDEwMDAwMCAtIDAwMDAwMDAwY2ZkNDMwMDAgKHVzYWJsZSkKKFhF TikgIDAwMDAwMDAwY2ZkNDMwMDAgLSAwMDAwMDAwMGNmZDRjMDAwIChBQ1BJIGRhdGEpCihYRU4p ICAwMDAwMDAwMGNmZDRjMDAwIC0gMDAwMDAwMDBjZmQ0ZDAwMCAodXNhYmxlKQooWEVOKSAgMDAw MDAwMDBjZmQ0ZDAwMCAtIDAwMDAwMDAwZDAwMDAwMDAgKHJlc2VydmVkKQooWEVOKSAgMDAwMDAw MDBlMDAwMDAwMCAtIDAwMDAwMDAwZjAwMDAwMDAgKHJlc2VydmVkKQooWEVOKSAgMDAwMDAwMDBm ZWMwMDAwMCAtIDAwMDAwMDAwZmVkMDAwMDAgKHJlc2VydmVkKQooWEVOKSAgMDAwMDAwMDBmZWUw MDAwMCAtIDAwMDAwMDAwZmVlMTAwMDAgKHJlc2VydmVkKQooWEVOKSAgMDAwMDAwMDBmZmMwMDAw MCAtIDAwMDAwMDAxMDAwMDAwMDAgKHJlc2VydmVkKQooWEVOKSAgMDAwMDAwMDEwMDAwMDAwMCAt IDAwMDAwMDEwMmZmZmYwMDAgKHVzYWJsZSkKKFhFTikgQUNQSTogUlNEUCAwMDBGNEYyMCwgMDAy NCAocjIgSFAgICAgKQooWEVOKSBBQ1BJOiBYU0RUIENGRDQzOTAwLCAwMDdDIChyMSBIUCAgICAg UHJvTGlhbnQgICAgICAgIDIgICDvv70gICAgIDE2MkUpCihYRU4pIEFDUEk6IEZBQ1AgQ0ZENDM5 QzAsIDAwRjQgKHIzIEhQICAgICBQcm9MaWFudCAgICAgICAgMiAgIO+/vSAgICAgMTYyRSkKKFhF TikgQUNQSTogRFNEVCBDRkQ0M0FDMCwgMzBDOSAocjEgSFAgICAgICAgICBEU0RUICAgICAgICAx IElOVEwgMjAwMzAyMjgpCihYRU4pIEFDUEk6IEZBQ1MgQ0ZENDMxMDAsIDAwNDAKKFhFTikgQUNQ STogU1BDUiBDRkQ0MzE0MCwgMDA1MCAocjEgSFAgICAgIFNQQ1JSQlNVICAgICAgICAxICAg77+9 ICAgICAxNjJFKQooWEVOKSBBQ1BJOiBNQ0ZHIENGRDQzMUMwLCAwMDNDIChyMSBIUCAgICAgUHJv TGlhbnQgICAgICAgIDEgICAgICAgICAgICAgMCkKKFhFTikgQUNQSTogSFBFVCBDRkQ0MzIwMCwg MDAzOCAocjEgSFAgICAgIFByb0xpYW50ICAgICAgICAyICAg77+9ICAgICAxNjJFKQooWEVOKSBB Q1BJOiBGRkZGIENGRDQzMjQwLCAwMDY0IChyMiBIUCAgICAgUDYxICAgICAgICAgICAgIDIgICDv v70gICAgIDE2MkUpCihYRU4pIEFDUEk6IFNQTUkgQ0ZENDMyQzAsIDAwNDAgKHI1IEhQICAgICBQ cm9MaWFudCAgICAgICAgMSAgIO+/vSAgICAgMTYyRSkKKFhFTikgQUNQSTogRVJTVCBDRkQ0MzMw MCwgMDFEMCAocjEgSFAgICAgIFByb0xpYW50ICAgICAgICAxICAg77+9ICAgICAxNjJFKQooWEVO KSBBQ1BJOiBBUElDIENGRDQzNTAwLCAwMTc2IChyMSBIUCAgICAgUHJvTGlhbnQgICAgICAgIDIg ICAgICAgICAgICAgMCkKKFhFTikgQUNQSTogRkZGRiBDRkQ0MzY4MCwgMDE3NiAocjEgSFAgICAg IFByb0xpYW50ICAgICAgICAxICAg77+9ICAgICAxNjJFKQooWEVOKSBBQ1BJOiBCRVJUIENGRDQz ODAwLCAwMDMwIChyMSBIUCAgICAgUHJvTGlhbnQgICAgICAgIDEgICDvv70gICAgIDE2MkUpCihY RU4pIEFDUEk6IEhFU1QgQ0ZENDM4NDAsIDAwQkMgKHIxIEhQICAgICBQcm9MaWFudCAgICAgICAg MSAgIO+/vSAgICAgMTYyRSkKKFhFTikgU3lzdGVtIFJBTTogNjU1MzJNQiAoNjcxMDU2NzJrQikK KFhFTikgRG9tYWluIGhlYXAgaW5pdGlhbGlzZWQKKFhFTikgUHJvY2Vzc29yICMwIDY6MTUgQVBJ QyB2ZXJzaW9uIDIwCihYRU4pIFByb2Nlc3NvciAjOCA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVO KSBQcm9jZXNzb3IgIzE2IDY6MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4pIFByb2Nlc3NvciAjMjQg NjoxNSBBUElDIHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMxIDY6MTUgQVBJQyB2ZXJzaW9u IDIwCihYRU4pIFByb2Nlc3NvciAjOSA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNz b3IgIzE3IDY6MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4pIFByb2Nlc3NvciAjMjUgNjoxNSBBUElD IHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMyIDY6MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4p IFByb2Nlc3NvciAjMTAgNjoxNSBBUElDIHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMxOCA2 OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNzb3IgIzI2IDY6MTUgQVBJQyB2ZXJzaW9u IDIwCihYRU4pIFByb2Nlc3NvciAjMyA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNz b3IgIzExIDY6MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4pIFByb2Nlc3NvciAjMTkgNjoxNSBBUElD IHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMyNyA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVO KSBJT0FQSUNbMF06IGFwaWNfaWQgMSwgdmVyc2lvbiAzMiwgYWRkcmVzcyAweGZlYzAwMDAwLCBH U0kgMC0yMwooWEVOKSBJT0FQSUNbMV06IGFwaWNfaWQgMiwgdmVyc2lvbiAzMiwgYWRkcmVzcyAw eGZlYzgwMDAwLCBHU0kgMjQtNDcKKFhFTikgSU9BUElDWzJdOiBhcGljX2lkIDMsIHZlcnNpb24g MzIsIGFkZHJlc3MgMHhmZWM4MTAwMCwgR1NJIDQ4LTcxCihYRU4pIElPQVBJQ1szXTogYXBpY19p ZCA0LCB2ZXJzaW9uIDMyLCBhZGRyZXNzIDB4ZmVjODE4MDAsIEdTSSA3Mi05NQooWEVOKSBFbmFi bGluZyBBUElDIG1vZGU6ICBQaHlzLiAgVXNpbmcgNCBJL08gQVBJQ3MKKFhFTikgVXNpbmcgc2No ZWR1bGVyOiBTTVAgQ3JlZGl0IFNjaGVkdWxlciAoY3JlZGl0KQooWEVOKSBEZXRlY3RlZCAyNDAw LjE0NSBNSHogcHJvY2Vzc29yLgooWEVOKSBJbml0aW5nIG1lbW9yeSBzaGFyaW5nLgooWEVOKSBW TVg6IFN1cHBvcnRlZCBhZHZhbmNlZCBmZWF0dXJlczoKKFhFTikgIC0gQVBJQyBNTUlPIGFjY2Vz cyB2aXJ0dWFsaXNhdGlvbgooWEVOKSAgLSBBUElDIFRQUiBzaGFkb3cKKFhFTikgIC0gVmlydHVh bCBOTUkKKFhFTikgIC0gTVNSIGRpcmVjdC1hY2Nlc3MgYml0bWFwCihYRU4pIEhWTTogQVNJRHMg ZGlzYWJsZWQuCihYRU4pIEhWTTogVk1YIGVuYWJsZWQKKFhFTikgSS9PIHZpcnR1YWxpc2F0aW9u IGRpc2FibGVkCihYRU4pIFRvdGFsIG9mIDE2IHByb2Nlc3NvcnMgYWN0aXZhdGVkLgooWEVOKSBF TkFCTElORyBJTy1BUElDIElSUXMKKFhFTikgIC0+IFVzaW5nIG5ldyBBQ0sgbWV0aG9kCihYRU4p IGNoZWNraW5nIFRTQyBzeW5jaHJvbml6YXRpb24gYWNyb3NzIDE2IENQVXM6IHBhc3NlZC4KKFhF TikgUGxhdGZvcm0gdGltZXIgaXMgMTQuMzE4TUh6IEhQRVQKKFhFTikgQWxsb2NhdGVkIGNvbnNv bGUgcmluZyBvZiAzMiBLaUIuCihYRU4pIEJyb3VnaHQgdXAgMTYgQ1BVcwooWEVOKSAqKiogTE9B RElORyBET01BSU4gMCAqKioKKFhFTikgIFhlbiAga2VybmVsOiA2NC1iaXQsIGxzYiwgY29tcGF0 MzIKKFhFTikgIERvbTAga2VybmVsOiA2NC1iaXQsIFBBRSwgbHNiLCBwYWRkciAweDEwMDAwMDAg LT4gMHgxNzA4MDAwCihYRU4pIFBIWVNJQ0FMIE1FTU9SWSBBUlJBTkdFTUVOVDoKKFhFTikgIERv bTAgYWxsb2MuOiAgIDAwMDAwMDA4M2MwMDAwMDAtPjAwMDAwMDA4NDAwMDAwMDAgKDc3MDA0OCBw YWdlcwp0byBiZSBhbGxvY2F0ZWQpCihYRU4pIFZJUlRVQUwgTUVNT1JZIEFSUkFOR0VNRU5UOgoo WEVOKSAgTG9hZGVkIGtlcm5lbDogZmZmZmZmZmY4MTAwMDAwMC0+ZmZmZmZmZmY4MTcwODAwMAoo WEVOKSAgSW5pdC4gcmFtZGlzazogZmZmZmZmZmY4MTcwODAwMC0+ZmZmZmZmZmY4MWVmYjAwMAoo WEVOKSAgUGh5cy1NYWNoIG1hcDogZmZmZmZmZmY4MWVmYjAwMC0+ZmZmZmZmZmY4MjRmYjAwMAoo WEVOKSAgU3RhcnQgaW5mbzogICAgZmZmZmZmZmY4MjRmYjAwMC0+ZmZmZmZmZmY4MjRmYjRiNAoo WEVOKSAgUGFnZSB0YWJsZXM6ICAgZmZmZmZmZmY4MjRmYzAwMC0+ZmZmZmZmZmY4MjUxMzAwMAoo WEVOKSAgQm9vdCBzdGFjazogICAgZmZmZmZmZmY4MjUxMzAwMC0+ZmZmZmZmZmY4MjUxNDAwMAoo WEVOKSAgVE9UQUw6ICAgICAgICAgZmZmZmZmZmY4MDAwMDAwMC0+ZmZmZmZmZmY4MjgwMDAwMAoo WEVOKSAgRU5UUlkgQUREUkVTUzogZmZmZmZmZmY4MTUzMTIwMAooWEVOKSBEb20wIGhhcyBtYXhp bXVtIDE2IFZDUFVzCihYRU4pIFNjcnViYmluZyBGcmVlIFJBTToKLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi5kb25lLgooWEVOKSBYZW4gdHJhY2UgYnVmZmVy czogZGlzYWJsZWQKKFhFTikgU3RkLiBMb2dsZXZlbDogRXJyb3JzIGFuZCB3YXJuaW5ncwooWEVO KSBHdWVzdCBMb2dsZXZlbDogRXJyb3JzIGFuZCB3YXJuaW5ncwooWEVOKSBYZW4gaXMgcmVsaW5x dWlzaGluZyBWR0EgY29uc29sZS4KKFhFTikgKioqIFNlcmlhbCBpbnB1dCAtPiBET00wICh0eXBl ICdDVFJMLWEnIHRocmVlIHRpbWVzIHRvIHN3aXRjaAppbnB1dCB0byBYZW4pCihYRU4pIEZyZWVk IDE3NmtCIGluaXQgbWVtb3J5LgooWEVOKSBQbGF0Zm9ybSB0aW1lciBhcHBlYXJzIHRvIGhhdmUg dW5leHBlY3RlZGx5IHdyYXBwZWQgMTAgb3IgbW9yZSB0aW1lcy4KCgphbmQgaGVyZSBpcyB4bSBk bWVzZyBvZiB4ZW4gMy4yIG9uIGEgZGViaWFuIGxlbm55IHN5c3RlbSBydW5uaW5nIG9uCnRoZSBz YW1lIGhhcmR3YXJlLCBvbiB0aGlzIHN5c3RlbSBJIGRvbid0IGhhdmUgY2xvY2sgcHJvYmxlbXM6 CgooWEVOKSBYZW4gdmVyc2lvbiAzLjItMSAoRGViaWFuIDMuMi4xLTIpICh3YWxkaUBkZWJpYW4u b3JnKSAoZ2NjCnZlcnNpb24gNC4zLjEgKERlYmlhbiA0LjMuMS0yKSApIFNhdCBKdW4gMjggMDk6 MzI6MTggVVRDIDIwMDgKKFhFTikgQ29tbWFuZCBsaW5lOgooWEVOKSBWaWRlbyBpbmZvcm1hdGlv bjoKKFhFTikgIFZHQSBpcyB0ZXh0IG1vZGUgODB4MjUsIGZvbnQgOHgxNgooWEVOKSAgVkJFL0RE QyBtZXRob2RzOiBWMjsgRURJRCB0cmFuc2ZlciB0aW1lOiAyIHNlY29uZHMKKFhFTikgRGlzYyBp bmZvcm1hdGlvbjoKKFhFTikgIEZvdW5kIDIgTUJSIHNpZ25hdHVyZXMKKFhFTikgIEZvdW5kIDIg RUREIGluZm9ybWF0aW9uIHN0cnVjdHVyZXMKKFhFTikgWGVuLWU4MjAgUkFNIG1hcDoKKFhFTikg IDAwMDAwMDAwMDAwMDAwMDAgLSAwMDAwMDAwMDAwMDlmNDAwICh1c2FibGUpCihYRU4pICAwMDAw MDAwMDAwMDlmNDAwIC0gMDAwMDAwMDAwMDBhMDAwMCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAw MDAwMGYwMDAwIC0gMDAwMDAwMDAwMDEwMDAwMCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAwMDAw MTAwMDAwIC0gMDAwMDAwMDBjZmQ0MzAwMCAodXNhYmxlKQooWEVOKSAgMDAwMDAwMDBjZmQ0MzAw MCAtIDAwMDAwMDAwY2ZkNGMwMDAgKEFDUEkgZGF0YSkKKFhFTikgIDAwMDAwMDAwY2ZkNGMwMDAg LSAwMDAwMDAwMGNmZDRkMDAwICh1c2FibGUpCihYRU4pICAwMDAwMDAwMGNmZDRkMDAwIC0gMDAw MDAwMDBkMDAwMDAwMCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAwMGUwMDAwMDAwIC0gMDAwMDAw MDBmMDAwMDAwMCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAwMGZlYzAwMDAwIC0gMDAwMDAwMDBm ZWQwMDAwMCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAwMGZlZTAwMDAwIC0gMDAwMDAwMDBmZWUx MDAwMCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAwMGZmYzAwMDAwIC0gMDAwMDAwMDEwMDAwMDAw MCAocmVzZXJ2ZWQpCihYRU4pICAwMDAwMDAwMTAwMDAwMDAwIC0gMDAwMDAwMGYyZmZmZjAwMCAo dXNhYmxlKQooWEVOKSBTeXN0ZW0gUkFNOiA2MTQzNk1CICg2MjkxMTM2OGtCKQooWEVOKSBYZW4g aGVhcDogMTJNQiAoMTMxMjhrQikKKFhFTikgRG9tYWluIGhlYXAgaW5pdGlhbGlzZWQ6IERNQSB3 aWR0aCAzMiBiaXRzCihYRU4pIFByb2Nlc3NvciAjMCA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVO KSBQcm9jZXNzb3IgIzggNjoxNSBBUElDIHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMxNiA2 OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNzb3IgIzI0IDY6MTUgQVBJQyB2ZXJzaW9u IDIwCihYRU4pIFByb2Nlc3NvciAjMSA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNz b3IgIzkgNjoxNSBBUElDIHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMxNyA2OjE1IEFQSUMg dmVyc2lvbiAyMAooWEVOKSBQcm9jZXNzb3IgIzI1IDY6MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4p IFByb2Nlc3NvciAjMiA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNzb3IgIzEwIDY6 MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4pIFByb2Nlc3NvciAjMTggNjoxNSBBUElDIHZlcnNpb24g MjAKKFhFTikgUHJvY2Vzc29yICMyNiA2OjE1IEFQSUMgdmVyc2lvbiAyMAooWEVOKSBQcm9jZXNz b3IgIzMgNjoxNSBBUElDIHZlcnNpb24gMjAKKFhFTikgUHJvY2Vzc29yICMxMSA2OjE1IEFQSUMg dmVyc2lvbiAyMAooWEVOKSBQcm9jZXNzb3IgIzE5IDY6MTUgQVBJQyB2ZXJzaW9uIDIwCihYRU4p IFByb2Nlc3NvciAjMjcgNjoxNSBBUElDIHZlcnNpb24gMjAKKFhFTikgSU9BUElDWzBdOiBhcGlj X2lkIDEsIHZlcnNpb24gMzIsIGFkZHJlc3MgMHhmZWMwMDAwMCwgR1NJIDAtMjMKKFhFTikgSU9B UElDWzFdOiBhcGljX2lkIDIsIHZlcnNpb24gMzIsIGFkZHJlc3MgMHhmZWM4MDAwMCwgR1NJIDI0 LTQ3CihYRU4pIElPQVBJQ1syXTogYXBpY19pZCAzLCB2ZXJzaW9uIDMyLCBhZGRyZXNzIDB4ZmVj ODEwMDAsIEdTSSA0OC03MQooWEVOKSBJT0FQSUNbM106IGFwaWNfaWQgNCwgdmVyc2lvbiAzMiwg YWRkcmVzcyAweGZlYzgxODAwLCBHU0kgNzItOTUKKFhFTikgRW5hYmxpbmcgQVBJQyBtb2RlOiAg UGh5cy4gIFVzaW5nIDQgSS9PIEFQSUNzCihYRU4pIFVzaW5nIHNjaGVkdWxlcjogU01QIENyZWRp dCBTY2hlZHVsZXIgKGNyZWRpdCkKKFhFTikgRGV0ZWN0ZWQgMjQwMC4xMzYgTUh6IHByb2Nlc3Nv ci4KKFhFTikgSFZNOiBWTVggZW5hYmxlZAooWEVOKSBDUFUwOiBJbnRlbChSKSBYZW9uKFIpIENQ VSAgICAgICAgICAgRTczMzAgIEAgMi40MEdIeiBzdGVwcGluZyAwYgooWEVOKSBCb290aW5nIHBy b2Nlc3NvciAxLzggZWlwIDhjMDAwCihYRU4pIENQVTE6IEludGVsKFIpIFhlb24oUikgQ1BVICAg ICAgICAgICBFNzMzMCAgQCAyLjQwR0h6IHN0ZXBwaW5nIDBiCihYRU4pIEJvb3RpbmcgcHJvY2Vz c29yIDIvMTYgZWlwIDhjMDAwCihYRU4pIENQVTI6IEludGVsKFIpIFhlb24oUikgQ1BVICAgICAg ICAgICBFNzMzMCAgQCAyLjQwR0h6IHN0ZXBwaW5nIDBiCihYRU4pIEJvb3RpbmcgcHJvY2Vzc29y IDMvMjQgZWlwIDhjMDAwCihYRU4pIENQVTM6IEludGVsKFIpIFhlb24oUikgQ1BVICAgICAgICAg ICBFNzMzMCAgQCAyLjQwR0h6IHN0ZXBwaW5nIDBiCihYRU4pIEJvb3RpbmcgcHJvY2Vzc29yIDQv MSBlaXAgOGMwMDAKKFhFTikgQ1BVNDogSW50ZWwoUikgWGVvbihSKSBDUFUgICAgICAgICAgIEU3 MzMwICBAIDIuNDBHSHogc3RlcHBpbmcgMGIKKFhFTikgQm9vdGluZyBwcm9jZXNzb3IgNS85IGVp cCA4YzAwMAooWEVOKSBDUFU1OiBJbnRlbChSKSBYZW9uKFIpIENQVSAgICAgICAgICAgRTczMzAg IEAgMi40MEdIeiBzdGVwcGluZyAwYgooWEVOKSBCb290aW5nIHByb2Nlc3NvciA2LzE3IGVpcCA4 YzAwMAooWEVOKSBDUFU2OiBJbnRlbChSKSBYZW9uKFIpIENQVSAgICAgICAgICAgRTczMzAgIEAg Mi40MEdIeiBzdGVwcGluZyAwYgooWEVOKSBCb290aW5nIHByb2Nlc3NvciA3LzI1IGVpcCA4YzAw MAooWEVOKSBDUFU3OiBJbnRlbChSKSBYZW9uKFIpIENQVSAgICAgICAgICAgRTczMzAgIEAgMi40 MEdIeiBzdGVwcGluZyAwYgooWEVOKSBCb290aW5nIHByb2Nlc3NvciA4LzIgZWlwIDhjMDAwCihY RU4pIENQVTg6IEludGVsKFIpIFhlb24oUikgQ1BVICAgICAgICAgICBFNzMzMCAgQCAyLjQwR0h6 IHN0ZXBwaW5nIDBiCihYRU4pIEJvb3RpbmcgcHJvY2Vzc29yIDkvMTAgZWlwIDhjMDAwCihYRU4p IENQVTk6IEludGVsKFIpIFhlb24oUikgQ1BVICAgICAgICAgICBFNzMzMCAgQCAyLjQwR0h6IHN0 ZXBwaW5nIDBiCihYRU4pIEJvb3RpbmcgcHJvY2Vzc29yIDEwLzE4IGVpcCA4YzAwMAooWEVOKSBD UFUxMDogSW50ZWwoUikgWGVvbihSKSBDUFUgICAgICAgICAgIEU3MzMwICBAIDIuNDBHSHogc3Rl cHBpbmcgMGIKKFhFTikgQm9vdGluZyBwcm9jZXNzb3IgMTEvMjYgZWlwIDhjMDAwCihYRU4pIENQ VTExOiBJbnRlbChSKSBYZW9uKFIpIENQVSAgICAgICAgICAgRTczMzAgIEAgMi40MEdIeiBzdGVw cGluZyAwYgooWEVOKSBCb290aW5nIHByb2Nlc3NvciAxMi8zIGVpcCA4YzAwMAooWEVOKSBDUFUx MjogSW50ZWwoUikgWGVvbihSKSBDUFUgICAgICAgICAgIEU3MzMwICBAIDIuNDBHSHogc3RlcHBp bmcgMGIKKFhFTikgQm9vdGluZyBwcm9jZXNzb3IgMTMvMTEgZWlwIDhjMDAwCihYRU4pIENQVTEz OiBJbnRlbChSKSBYZW9uKFIpIENQVSAgICAgICAgICAgRTczMzAgIEAgMi40MEdIeiBzdGVwcGlu ZyAwYgooWEVOKSBCb290aW5nIHByb2Nlc3NvciAxNC8xOSBlaXAgOGMwMDAKKFhFTikgQ1BVMTQ6 IEludGVsKFIpIFhlb24oUikgQ1BVICAgICAgICAgICBFNzMzMCAgQCAyLjQwR0h6IHN0ZXBwaW5n IDBiCihYRU4pIEJvb3RpbmcgcHJvY2Vzc29yIDE1LzI3IGVpcCA4YzAwMAooWEVOKSBDUFUxNTog SW50ZWwoUikgWGVvbihSKSBDUFUgICAgICAgICAgIEU3MzMwICBAIDIuNDBHSHogc3RlcHBpbmcg MGIKKFhFTikgVG90YWwgb2YgMTYgcHJvY2Vzc29ycyBhY3RpdmF0ZWQuCihYRU4pIEVOQUJMSU5H IElPLUFQSUMgSVJRcwooWEVOKSAgLT4gVXNpbmcgbmV3IEFDSyBtZXRob2QKKFhFTikgUGxhdGZv cm0gdGltZXIgb3ZlcmZsb3dzIGluIDE0OTk4IGppZmZpZXMuCihYRU4pIFBsYXRmb3JtIHRpbWVy IGlzIDE0LjMxOE1IeiBIUEVUCihYRU4pIEJyb3VnaHQgdXAgMTYgQ1BVcwooWEVOKSBBTUQgSU9N TVU6IERpc2FibGVkCihYRU4pICoqKiBMT0FESU5HIERPTUFJTiAwICoqKgooWEVOKSAgWGVuICBr ZXJuZWw6IDY0LWJpdCwgbHNiLCBjb21wYXQzMgooWEVOKSAgRG9tMCBrZXJuZWw6IDY0LWJpdCwg bHNiLCBwYWRkciAweDIwMDAwMCAtPiAweDYzMTkxOAooWEVOKSBQSFlTSUNBTCBNRU1PUlkgQVJS QU5HRU1FTlQ6CihYRU4pICBEb20wIGFsbG9jLjogICAwMDAwMDAwOGUwMDAwMDAwLT4wMDAwMDAw OGYwMDAwMDAwICgxNTQyMjU4NQpwYWdlcyB0byBiZSBhbGxvY2F0ZWQpCihYRU4pIFZJUlRVQUwg TUVNT1JZIEFSUkFOR0VNRU5UOgooWEVOKSAgTG9hZGVkIGtlcm5lbDogZmZmZmZmZmY4MDIwMDAw MC0+ZmZmZmZmZmY4MDYzMTkxOAooWEVOKSAgSW5pdC4gcmFtZGlzazogZmZmZmZmZmY4MDYzMjAw MC0+ZmZmZmZmZmY4MGQ0MGUwMAooWEVOKSAgUGh5cy1NYWNoIG1hcDogZmZmZmZmZmY4MGQ0MTAw MC0+ZmZmZmZmZmY4ODM2YjNjOAooWEVOKSAgU3RhcnQgaW5mbzogICAgZmZmZmZmZmY4ODM2YzAw MC0+ZmZmZmZmZmY4ODM2YzRhNAooWEVOKSAgUGFnZSB0YWJsZXM6ICAgZmZmZmZmZmY4ODM2ZDAw MC0+ZmZmZmZmZmY4ODNiNDAwMAooWEVOKSAgQm9vdCBzdGFjazogICAgZmZmZmZmZmY4ODNiNDAw MC0+ZmZmZmZmZmY4ODNiNTAwMAooWEVOKSAgVE9UQUw6ICAgICAgICAgZmZmZmZmZmY4MDAwMDAw MC0+ZmZmZmZmZmY4ODgwMDAwMAooWEVOKSAgRU5UUlkgQUREUkVTUzogZmZmZmZmZmY4MDIwMDAw MAooWEVOKSBEb20wIGhhcyBtYXhpbXVtIDE2IFZDUFVzCihYRU4pIEluaXRyZCBsZW4gMHg3MGVl MDAsIHN0YXJ0IGF0IDB4ZmZmZmZmZmY4MDYzMjAwMAooWEVOKSBTY3J1YmJpbmcgRnJlZSBSQU06 IC5kb25lLgooWEVOKSBYZW4gdHJhY2UgYnVmZmVyczogZGlzYWJsZWQKKFhFTikgU3RkLiBMb2ds ZXZlbDogRXJyb3JzIGFuZCB3YXJuaW5ncwooWEVOKSBHdWVzdCBMb2dsZXZlbDogTm90aGluZyAo UmF0ZS1saW1pdGVkOiBFcnJvcnMgYW5kIHdhcm5pbmdzKQooWEVOKSBYZW4gaXMgcmVsaW5xdWlz aGluZyBWR0EgY29uc29sZS4KKFhFTikgKioqIFNlcmlhbCBpbnB1dCAtPiBET00wICh0eXBlICdD VFJMLWEnIHRocmVlIHRpbWVzIHRvIHN3aXRjaAppbnB1dCB0byBYZW4pCihYRU4pIEZyZWVkIDEw NGtCIGluaXQgbWVtb3J5LgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KWGVuLXVzZXJzIG1haWxpbmcgbGlzdApYZW4tdXNlcnNAbGlzdHMueGVuLm9yZwpo dHRwOi8vbGlzdHMueGVuLm9yZy94ZW4tdXNlcnM= From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pasi =?iso-8859-1?Q?K=E4rkk=E4inen?= Subject: Re: [Xen-users] Re: Xen 4 TSC problems Date: Sun, 30 Sep 2012 18:13:14 +0300 Message-ID: <20120930151314.GU8912@reaktio.net> References:

<65395f62-74e5-4910-b701-8df629c2ce3b@default>

Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Mauro Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Olivier Hanesse , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org On Sat, Sep 29, 2012 at 02:19:55PM +0200, Mauro wrote: > It's happened another time, system date 50 minutes ahead. > There is really no solution? > Try with a recent Xen hypervisor version. Xen 4.1.3 or 4.2.0. It helps a lot to know if the issue is still in the latest hypervisor versions or not. 4.0.1 is quite old already.. and besides 4.0.4 is the latest version in 4.0 branch. -- Pasi > root@xen-p02:~# date > sab 29 set 2012, 15.06.25, CEST > > root@xen-p02:~# hwclock --debug > hwclock from util-linux-ng 2.17.2 > Using /dev interface to clock. > Last drift adjustment done at 1348816781 seconds after 1969 > Last calibration done at 1348816781 seconds after 1969 > Hardware clock is on UTC time > Assuming hardware clock is kept in UTC time. > Waiting for clock tick... > ...got clock tick > Time read from Hardware Clock: 2012/09/29 12:16:58 > Hw clock time : 2012/09/29 12:16:58 = 1348921018 seconds since 1969 > sab 29 set 2012 14:16:58 CEST -0.751536 seconds > > root@xen-p02:~# hwclock --show > sab 29 set 2012 14:17:12 CEST -0.423643 seconds > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mauro Subject: Re: [Xen-devel] Re: Xen 4 TSC problems Date: Sun, 30 Sep 2012 21:23:44 +0200 Message-ID: References:

<65395f62-74e5-4910-b701-8df629c2ce3b@default>

<20120930151314.GU8912@reaktio.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20120930151314.GU8912@reaktio.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xen.org Errors-To: xen-users-bounces@lists.xen.org To: =?UTF-8?B?UGFzaSBLw6Rya2vDpGluZW4=?= Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Olivier Hanesse , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org T24gMzAgU2VwdGVtYmVyIDIwMTIgMTc6MTMsIFBhc2kgS8Okcmtrw6RpbmVuIDxwYXNpa0Bpa2ku Zmk+IHdyb3RlOgo+IE9uIFNhdCwgU2VwIDI5LCAyMDEyIGF0IDAyOjE5OjU1UE0gKzAyMDAsIE1h dXJvIHdyb3RlOgo+PiBJdCdzIGhhcHBlbmVkIGFub3RoZXIgdGltZSwgc3lzdGVtIGRhdGUgNTAg bWludXRlcyBhaGVhZC4KPj4gVGhlcmUgaXMgcmVhbGx5IG5vIHNvbHV0aW9uPwo+Pgo+Cj4gVHJ5 IHdpdGggYSByZWNlbnQgWGVuIGh5cGVydmlzb3IgdmVyc2lvbi4gWGVuIDQuMS4zIG9yIDQuMi4w Lgo+IEl0IGhlbHBzIGEgbG90IHRvIGtub3cgaWYgdGhlIGlzc3VlIGlzIHN0aWxsIGluIHRoZSBs YXRlc3QgaHlwZXJ2aXNvciB2ZXJzaW9ucyBvciBub3QuCgpJJ20gdXNpbmcgZGViaWFuIHNxdWVl emUgeGVuIGtlcm5lbCBhbmQgdGhpcyBrZXJuZWwgaGFzIHhlbiA0LjAuCgpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpYZW4tdXNlcnMgbWFpbGluZyBsaXN0 Clhlbi11c2Vyc0BsaXN0cy54ZW4ub3JnCmh0dHA6Ly9saXN0cy54ZW4ub3JnL3hlbi11c2Vycw== From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mauro Subject: Re: [Xen-devel] Re: Xen 4 TSC problems Date: Sun, 30 Sep 2012 22:19:22 +0200 Message-ID: References:

<65395f62-74e5-4910-b701-8df629c2ce3b@default>

<20120930151314.GU8912@reaktio.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xen.org Errors-To: xen-users-bounces@lists.xen.org To: =?UTF-8?B?UGFzaSBLw6Rya2vDpGluZW4=?= Cc: Dan Magenheimer , xen-devel@lists.xensource.com, Keir Fraser , Jeremy Fitzhardinge , Olivier Hanesse , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org T24gMzAgU2VwdGVtYmVyIDIwMTIgMjE6MjMsIE1hdXJvIDxtcnNhbm5hMUBnbWFpbC5jb20+IHdy b3RlOgo+IE9uIDMwIFNlcHRlbWJlciAyMDEyIDE3OjEzLCBQYXNpIEvDpHJra8OkaW5lbiA8cGFz aWtAaWtpLmZpPiB3cm90ZToKPj4gT24gU2F0LCBTZXAgMjksIDIwMTIgYXQgMDI6MTk6NTVQTSAr MDIwMCwgTWF1cm8gd3JvdGU6Cj4+PiBJdCdzIGhhcHBlbmVkIGFub3RoZXIgdGltZSwgc3lzdGVt IGRhdGUgNTAgbWludXRlcyBhaGVhZC4KPj4+IFRoZXJlIGlzIHJlYWxseSBubyBzb2x1dGlvbj8K Pj4+Cj4+Cj4+IFRyeSB3aXRoIGEgcmVjZW50IFhlbiBoeXBlcnZpc29yIHZlcnNpb24uIFhlbiA0 LjEuMyBvciA0LjIuMC4KPj4gSXQgaGVscHMgYSBsb3QgdG8ga25vdyBpZiB0aGUgaXNzdWUgaXMg c3RpbGwgaW4gdGhlIGxhdGVzdCBoeXBlcnZpc29yIHZlcnNpb25zIG9yIG5vdC4KClRoZXJlIGlz IHNvbWVvbmUgdGhhdCBoYWQgdGhlIHByb2JsZW0gYW5kIHNvbHZlZCB1c2luZyBhIHJlY2VudCB4 ZW4gaHlwZXJ2aXNvcj8KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fClhlbi11c2VycyBtYWlsaW5nIGxpc3QKWGVuLXVzZXJzQGxpc3RzLnhlbi5vcmcKaHR0 cDovL2xpc3RzLnhlbi5vcmcveGVuLXVzZXJz From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zary Matej Subject: Re: [Xen-users] Re: Xen 4 TSC problems Date: Mon, 1 Oct 2012 13:39:11 +0200 Message-ID: <5DB0519124BB3D4DBEEB14426D4AC7EA6A72C9D58E@aries.space.cvtisr.sk> References:

<65395f62-74e5-4910-b701-8df629c2ce3b@default>

<20120930151314.GU8912@reaktio.net>, Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: Content-Language: en-GB List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Mauro , =?iso-8859-2?Q?Pasi_K=E4rkk=E4inen?= Cc: Dan Magenheimer , "xen-devel@lists.xensource.com" , Keir Fraser , Jeremy Fitzhardinge , Olivier Hanesse , Keir Fraser , Xen Users , Mark Adams List-Id: xen-devel@lists.xenproject.org >From: xen-users-bounces@lists.xen.org [xen-users-bounces@lists.xen.org] On= Behalf Of Mauro [mrsanna1@gmail.com] >On 30 September 2012 17:13, Pasi K=E4rkk=E4inen wrote: >> On Sat, Sep 29, 2012 at 02:19:55PM +0200, Mauro wrote: >>> It's happened another time, system date 50 minutes ahead. >>> There is really no solution? >>> >> >> Try with a recent Xen hypervisor version. Xen 4.1.3 or 4.2.0. >> It helps a lot to know if the issue is still in the latest hypervisor ve= rsions or not. > >I'm using debian squeeze xen kernel and this kernel has xen 4.0. > Xen and kernel are two different things, you can mix and match them in many= ways. If you don't want to compile from source, use Xen packages from Debi= an Wheezy (4.1.3-2), Sid (4.1.3-3) or Experimental (4.2.0-1). I doubt you w= ill move forward without trying newer versions no matter how many mails you= write to list (all the people) ... :) Matej From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olivier Hanesse Subject: Re: [Xen-users] Re: Xen 4 TSC problems Date: Mon, 15 Oct 2012 09:39:56 +0200 Message-ID: References:

<65395f62-74e5-4910-b701-8df629c2ce3b@default>

<20120930151314.GU8912@reaktio.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3773458365543498546==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: