From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Magenheimer Subject: RE: [Xen-devel] Xen 4 TSC problems Date: Mon, 28 Feb 2011 07:14:23 -0800 (PST) Message-ID: <49766a9d-0e37-46d2-9497-33c2e981a871@default> References: <68c41dd8-9195-41b0-83d7-9242b8eff809@default> <4D66AB26.9020000@goop.org AANLkTi=NFebVsnj_09+SpCZUo_bNTnTyJLAoR5sfbwZa@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1911446809==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-users-bounces@lists.xensource.com Errors-To: xen-users-bounces@lists.xensource.com To: Olivier Hanesse , Jeremy Fitzhardinge Cc: xen-devel@lists.xensource.com, Keir Fraser , Jan Beulich , Mark, Keir Fraser , Xen Users , Adams List-Id: xen-devel@lists.xenproject.org --===============1911446809== Content-Type: multipart/alternative; boundary="__129890608367518096abhmt020" --__129890608367518096abhmt020 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Hi Olivier - =20 It is the Xen clocksource that you want to try to change, not the dom0 cloc= ksource. To do this, you need to specify "clocksource=3Dpit" on the Xen bo= ot line (and reboot), not the dom0 boot line. =20 I believe Mark Adams played with tsc_mode to see if it solved his (similar?= identical?) problem last year, and it didn't make any difference. Please try booting Xen with "clocksource=3Dpit" and ensure that "Platform t= imer is 1.19MHz PIT" appears in the Xen boot messages. If the 50min jump d= oes not appear again, it would point to a problem in the hpet, either hardw= are or software. =20 Thanks, Dan =20 From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]=20 Sent: Monday, February 28, 2011 7:37 AM To: Jeremy Fitzhardinge Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; xen-devel@lists.= xensource.com; Xen Users; Keir Fraser Subject: Re: [Xen-devel] Xen 4 TSC problems =20 Hello, =20 It happened again twice this weekend. =20 What about setting "tsc_mode=3D2" for my vms ? Should this mode prevent thi= s bug (coming from a bad emulated tsc due to firmware issue ? is it possibl= e ?) from affecting time in domUs ? =20 Setting clocksource=3Dpit, make 'tsc' available in "/sys/devices/system/clo= cksource/clocksource0/available_clocksource" (otherwise only xen is availab= le, is it normal ? ).=20 =20 Should I bypass xen clocksource and use tsc as a clocksource for dom0/domU = ? or will it be worsed ? =20 Regards =20 Olivier =20 2011/2/24 Jeremy Fitzhardinge On 02/24/2011 09:43 AM, Dan Magenheimer wrote: > Just a wild guess, but this in Olivier's posted output: > > (XEN) Platform timer appears to have unexpectedly wrapped 10 or more time= s. > > and the fact that a 32-bit HPET wrap is ~300 seconds and, with the > "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue > (or a complete red herring, but I thought it worth mentioning). > > Mark and Olivier, it would be interesting to know if you are > using the same processor/system. It definitely seems like some kind of problem on the host system rather than anything in the guests themselves. If the platform timer is misbehaving, then Xen could be completely screwing up the pvclock calibration which it then passes to guests. Could it be one of those "platform clock stops in certain power states" problems? J >> -----Original Message----- >> From: Keir Fraser [mailto:HYPERLINK "mailto:keir.xen@gmail.com"keir.xen@= gmail.com] >> Sent: Thursday, February 24, 2011 7:52 AM >> To: Olivier Hanesse; Jan Beulich >> Cc: Mark Adams; Jeremy Fitzhardinge; HYPERLINK "mailto:xen-devel@lists.x= ensource.com"xen-devel@lists.xensource.com; Xen >> Users; Dan Magenheimer; Keir Fraser >> Subject: Re: [Xen-devel] Xen 4 TSC problems >> >> On 24/02/2011 14:20, "Olivier Hanesse" >> wrote: >> >>> Both dom0 and domUs are affected by this" jump". >>> >>> I expect to see something like "TSC marked as reliable, warp =3D 0". >>> I got this on newer hardware with same config/distros. >> It depends on the CPU itself, older CPUs do not have the super-stable >> TSC >> features. But that should never cause a massive 3000s time jump. >> >>> Is there a way to measure if it is a TSC warp ? to point out a cpu >> tsc issue ? >> >> The TSC warps or out-of-sync issues that we could reasonably expect >> would be >> on the order of microseconds. A 3000s warp is something else entirely. >> Xen >> is very confused and/or some TSC or platform timer has jumped a long >> way >> (indicating a hardware/firmware issue). >> >> -- Keir >> >>> 2011/2/24 Jan Beulich >>>>>>> On 24.02.11 at 12:57, Olivier Hanesse >> wrote: >>>>> I tried to turn off cstates with max_cstate=3D0 without success >> (still "not >>>>> reliable"). >>>>> >>>>> With cpuidle=3D0, I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3D3022 (count=3D1) >>>> This message by itself isn't telling much I believe. >>>> >>>>> xm info | grep command >>>>> xen_commandline : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall >> guest_loglvl=3Dall >>>>> dom0_max_vcpus=3D1 dom0_vcpus_pin console=3Dvga,com1 com1=3D19200,8n1 >>>>> >>>>> Keir : >>>>> >>>>> Using clocksource=3Dpit : >>>>> >>>>> (XEN) Platform timer is 1.193MHz PIT >>>>> >>>>> I also got : >>>>> >>>>> (XEN) TSC has constant rate, deep Cstates possible, so not >> reliable, >>>>> warp=3D3262 (count=3D2) >>>> The question is whether any of this eliminates the time jumps seen >>>> by your DomU-s (from your past mails I wasn't actually sure whether >>>> Dom0 also experienced this problem, albeit it would be odd if it >> didn't). >>>> Jan >>>> >>>> Jan >>>> >>> >> =20 --__129890608367518096abhmt020 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: quoted-printable

Hi Ol= ivier –

&nb= sp;

It is the Xen clocksour= ce that you want to try to change, not the dom0 clocksource.  To do = this, you need to specify “clocksource=3Dpit” on the Xen boot= line (and reboot), not the dom0 boot line.

 

I believe Mark Adams played with tsc_mode to see if it solved! =20 his (similar? identical?) problem last year, and it didn’t make any dif= ference.


Please tr= y booting Xen with “clocksource=3Dpit” and ensure that “= ;Platform timer is 1.19MHz PIT” appears in the Xen boot messages.&n= bsp; If the 50min jump does not appear again, it would point to a problem= in the hpet, either hardware or software.

 

Thanks,

Dan

 

From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]
Sen= t: Monday, February 28, 2011 7:37 AM
To: Jeremy Fitzharding= e
Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams; xe= n-devel@lists.xensource.com; Xen Users; Keir Fraser
Subject: Re= : [Xen-devel] Xen 4 TSC problems

 

Hello,<= /o:p>

 

It happened again twice this weekend.

 

What about setting "tsc_mode=3D2" = for my vms ? Should this mode prevent this bug (coming from a bad emulate= d tsc due to firmware issue ? is it possible ?) from affecting time in do= mUs ?

 

=

Setting clocksource=3Dpit, make 'tsc' ava= ilable in "/sys/devices/system/clocksource/clocksource0/available_cl= ocksource" (otherwise only xen is available, is it normal ? ). =

 

Should I bypass xen clocksource and use tsc as= a clocksource for dom0/domU ? or  will it be worsed ?

 

Regards

&= nbsp;

Olivier

<= o:p> 

2011/2/24 Jeremy Fitzhardin= ge <jeremy@goop.org>

On 02/24/= 2011 09:43 AM, Dan Magenheimer wrote:
> Just a wild guess, but this= in Olivier's posted output:
>
> (XEN) Platform timer appears= to have unexpectedly wrapped 10 or more times.
>
> and the f= act that a 32-bit HPET wrap is ~300 seconds and, with the
> "1= 0 or more times", 10 * 300 seconds is 3000 seconds, might be a clue<= br>> (or a complete red herring, but I thought it worth mentioning).>
> Mark and Olivier, it would be interesting to know if you a= re
> using the same processor/system.

It definitely seems like some kind of problem on the host syste= m rather
than anything in the guests themselves. ! =20  If the platform timer is
misbehaving, then Xen could be completely screwi= ng up the pvclock
calibration which it then passes to guests.

C= ould it be one of those "platform clock stops in certain power state= s"
problems?


   J

>> -----Original M= essage-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com]
>> Sent: Thursday, Februa= ry 24, 2011 7:52 AM
>> To: Olivier Hanesse; Jan Beulich
>&= gt; Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen
>> Users= ; Dan Magenheimer; Keir Fraser
>> Subject: Re: [Xen-devel] Xen 4= TSC problems
>>
>> On 24/02/2011 14:20, "Olivier = Hanesse" <olivier.ha= nesse@gmail.com>
>> wrote:
>>
>>> Both dom0 and domUs are affected by th= is" jump".
>>>
>>> I expect to see some= thing like "TSC marked as reliable, warp =3D 0".
>>>= ; I got this on newer hardware with same config/distros.
>> It d= epends on the CPU itself, older CPUs do not have the super-stable
>= > TSC
>> features. But that should never cause a massive 3000= s time jump.
>>
>>> Is there a way to measure if it = is a TSC warp ? to point out a cpu
>> tsc issue ?
>>>> The TSC warps or out-of-sync issues that we could reasonably ex= pect
>> would be
>> on the order of microseconds. A 300= 0s warp is something else entirely.
>> Xen
>> is very c= onfused and/or some TSC or platform timer has jumped a long
>> w= ay
>> (indicating a hardware/firmware issue).
>>
>= ;>  -- Keir
>>
>>! =20 ;> 2011/2/24 Jan Beulich <JBeulich= @novell.com>
>>>>>>> On 24.02.11 at 12:57,= Olivier Hanesse <olivier= .hanesse@gmail.com>
>> wrote:
>>>>> I t= ried to turn off cstates with max_cstate=3D0 without success
>> = (still "not
>>>>> reliable").
>>>= >>
>>>>> With cpuidle=3D0, I also got :
>&g= t;>>>
>>>>> (XEN) TSC has constant rate, deep = Cstates possible, so not
>> reliable,
>>>>> wa= rp=3D3022 (count=3D1)
>>>> This message by itself isn't te= lling much I believe.
>>>>
>>>>> xm info= | grep command
>>>>> xen_commandline     &nb= sp;  : dom0_mem=3D512M cpuidle=3D0 loglvl=3Dall
>> guest_lo= glvl=3Dall
>>>>> dom0_max_vcpus=3D1 dom0_vcp! =20 us_pin console=3Dvga,com1 com1=3D19200,8n1
>>>>>
>>&g= t;>> Keir :
>>>>>
>>>>> Using c= locksource=3Dpit :
>>>>>
>>>>> (XEN) = Platform timer is 1.193MHz PIT
>>>>>
>>>>= ;> I also got :
>>>>>
>>>>> (XEN) = TSC has constant rate, deep Cstates possible, so not
>> reliable= ,
>>>>> warp=3D3262 (count=3D2)
>>>> The= question is whether any of this eliminates the time jumps seen
>&g= t;>> by your DomU-s (from your past mails I wasn't actually sure wh= ether
>>>> Dom0 also experienced this problem, albeit it w= ould be odd if it
>> didn't).
>>>> Jan
>>= ;>>
>>>> Jan
>>>>
>>>
= >>

 =

--__129890608367518096abhmt020-- --===============1911446809== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users --===============1911446809==--