Re: [Xen-devel] Xen 4 TSC problems

[Keir Fraser <keir.xen@gmail.com>]

On 28/02/2011 15:23, "Olivier Hanesse" <olivier.hanesse@gmail.com> wrote:

> Keir : 
> 
> Yes, it is "under progress". 
> To make this change, I had to reboot every server, so it is taking time
> (production server :()
> So i was hoping to find a quick method to mitigate this issue on domUs while
> rebooting servers.
> 
> As this bug happens once or twice per server since October, I can't say that
> right now that changing platform timer to PIT fixed it. I have to wait (I hope
> forever!) this bug to happen again on a 'patched' server ... 
> 
> But even with clcoksource=pit, I am seeing some warp=3000+ in debug message ?
> I guess it is not a good sign, is it ?

Better not to have it, but honestly you're very unlikely to see any problem
from it. It's totally unrelated to the 3000-second time jumps.

 -- Keir

> Jan : I was hoping to find a way to make the domU clocksource more
> "independent" like with xen3.2.
> 
> 
> 2011/2/28 Dan Magenheimer <dan.magenheimer@oracle.com>
>> Hi Olivier ­
>>  
>> It is the Xen clocksource that you want to try to change, not the dom0
>> clocksource.  To do this, you need to specify ³clocksource=pit² on the Xen
>> boot line (and reboot), not the dom0 boot line.
>>  
>> I believe Mark Adams played with tsc_mode to see if it solved! his (similar?
>> identical?) problem last year, and it didn¹t make any difference.
>> 
>> Please try booting Xen with ³clocksource=pit² and ensure that ³Platform timer
>> is 1.19MHz PIT² appears in the Xen boot messages.  If the 50min jump does not
>> appear again, it would point to a problem in the hpet, either hardware or
>> software.
>>  
>> Thanks,
>> Dan
>>  
>> 
>> From: Olivier Hanesse [mailto:olivier.hanesse@gmail.com]
>> Sent: Monday, February 28, 2011 7:37 AM
>> To: Jeremy Fitzhardinge
>> Cc: Dan Magenheimer; Keir Fraser; Jan Beulich; Mark Adams;
>> xen-devel@lists.xensource.com; Xen Users; Keir Fraser
>> 
>> 
>> Subject: Re: [Xen-devel] Xen 4 TSC problems
>>  
>> 
>> Hello,
>> 
>>  
>> It happened again twice this weekend.
>> 
>>  
>> 
>> What about setting "tsc_mode=2" for my vms ? Should this mode prevent this
>> bug (coming from a bad emulated tsc due to firmware issue ? is it possible ?)
>> from affecting time in domUs ?
>> 
>>  
>> 
>> Setting clocksource=pit, make 'tsc' available in
>> "/sys/devices/system/clocksource/clocksource0/available_clocksource"
>> (otherwise only xen is available, is it normal ? ). 
>> 
>>  
>> 
>> Should I bypass xen clocksource and use tsc as a clocksource for dom0/domU ?
>> or  will it be worsed ?
>> 
>>  
>> 
>> Regards
>> 
>>  
>> 
>> Olivier
>> 
>>  
>> 
>> 2011/2/24 Jeremy Fitzhardinge <jeremy@goop.org>
>> 
>> On 02/24/2011 09:43 AM, Dan Magenheimer wrote:
>>> Just a wild guess, but this in Olivier's posted output:
>>> 
>>> (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times.
>>> 
>>> and the fact that a 32-bit HPET wrap is ~300 seconds and, with the
>>> "10 or more times", 10 * 300 seconds is 3000 seconds, might be a clue
>>> (or a complete red herring, but I thought it worth mentioning).
>>> 
>>> Mark and Olivier, it would be interesting to know if you are
>>> using the same processor/system.
>> It definitely seems like some kind of problem on the host system rather
>> than anything in the guests themselves. !  If the platform timer is
>> misbehaving, then Xen could be completely screwing up the pvclock
>> calibration which it then passes to guests.
>> 
>> Could it be one of those "platform clock stops in certain power states"
>> problems?
>> 
>> 
>>    J
>> 
>>>> -----Original Message-----
>>>> From: Keir Fraser [mailto:keir.xen@gmail.com]
>>>> Sent: Thursday, February 24, 2011 7:52 AM
>>>> To: Olivier Hanesse; Jan Beulich
>>>> Cc: Mark Adams; Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Xen
>>>> Users; Dan Magenheimer; Keir Fraser
>>>> Subject: Re: [Xen-devel] Xen 4 TSC problems
>>>> 
>>>> On 24/02/2011 14:20, "Olivier Hanesse" <olivier.hanesse@gmail.com>
>>>> wrote:
>>>> 
>>>>> Both dom0 and domUs are affected by this" jump".
>>>>> 
>>>>> I expect to see something like "TSC marked as reliable, warp = 0".
>>>>> I got this on newer hardware with same config/distros.
>>>> It depends on the CPU itself, older CPUs do not have the super-stable
>>>> TSC
>>>> features. But that should never cause a massive 3000s time jump.
>>>> 
>>>>> Is there a way to measure if it is a TSC warp ? to point out a cpu
>>>> tsc issue ?
>>>> 
>>>> The TSC warps or out-of-sync issues that we could reasonably expect
>>>> would be
>>>> on the order of microseconds. A 3000s warp is something else entirely.
>>>> Xen
>>>> is very confused and/or some TSC or platform timer has jumped a long
>>>> way
>>>> (indicating a hardware/firmware issue).
>>>> 
>>>>  -- Keir
>>>> 
>>> &gt! ;> 2011/2/24 Jan Beulich <JBeulich@novell.com>
>> 
>>>>>>>>> On 24.02.11 at 12:57, Olivier Hanesse <olivier.hanesse@gmail.com>
>>>> wrote:
>>>>>>> I tried to turn off cstates with max_cstate=0 without success
>>>> (still "not
>>>>>>> reliable").
>>>>>>> 
>>>>>>> With cpuidle=0, I also got :
>>>>>>> 
>>>>>>> (XEN) TSC has constant rate, deep Cstates possible, so not
>>>> reliable,
>>>>>>> warp=3022 (count=1)
>>>>>> This message by itself isn't telling much I believe.
>>>>>> 
>>>>>>> xm info | grep command
>>>>>>> xen_commandline        : dom0_mem=512M cpuidle=0 loglvl=all
>>>> guest_loglvl=all
>>>>>>> dom0_max_vcpus=1 dom0_vcp! us_pin console=vga,com1 com1=19200,8n1
>> 
>>>>>>> 
>>>>>>> Keir :
>>>>>>> 
>>>>>>> Using clocksource=pit :
>>>>>>> 
>>>>>>> (XEN) Platform timer is 1.193MHz PIT
>>>>>>> 
>>>>>>> I also got :
>>>>>>> 
>>>>>>> (XEN) TSC has constant rate, deep Cstates possible, so not
>>>> reliable,
>>>>>>> warp=3262 (count=2)
>>>>>> The question is whether any of this eliminates the time jumps seen
>>>>>> by your DomU-s (from your past mails I wasn't actually sure whether
>>>>>> Dom0 also experienced this problem, albeit it would be odd if it
>>>> didn't).
>>>>>> Jan
>>>>>> 
>>>>>> Jan
>>>>>> 
>>>>> 
>>>> 
>>  
> 
>