Xen 4.0.x 3,000 second time skew problem

Xen 4.0.x 3,000 second time skew problem

[Andy Smith]

Hi,

I have two installs of Xen 4.0.1 on Debian squeeze dom0s that
occasionally skew the time by 3000 seconds. It is always 3000
seconds and it always affects all guests.

Depending on the kernels in the guests they will say:

    tp0 kernel: [9339489.266542] Clocksource tsc unstable (delta = -2999660317697 ns)

or perhaps the kernel doesn't notice but ntpd does:

    ls0 ntpd[15009]: time reset -2999.683517 s

the point is it does affect all guests.

Back in February 2011 there was a thread started by Olivier Hanesse
which appears to be exactly the same issue:

    http://old-list-archives.xen.org/archives/html/xen-users/2011-02/msg00609.html

Like Olivier, one of my hosts used to run Xen 3.x and did not
experience this issue until after it was upgraded to Xen 4. The
other has only ever run 4.0.x so I don't know.

The above thread ends with Olivier using clocksource=pit on his Xen
command line and waiting to see if it fixed things. If you are
reading, Olivier, did it fix things for you?

These are Debian squeeze installs so using packaged versions of xen
and kernel:

ii  linux-image-2.6.32-5-xen-amd64 2.6.32-45 Linux 2.6.32 for 64-bit PCs, Xen dom0 support
ii  xen-hypervisor-4.0-amd64       4.0.1-4   The Xen Hypervisor on AMD64

One of the hosts affected is:

CPU: Xeon 5410
Motherboard: Supermicro X7DCL-i

The other is:

CPU: Xeon E5606
Motherboard: SUpermicro X8DTN+

but I do have other servers with same hardware as both of these that
aren't affected.

Cheers,
Andy

Re: Xen 4.0.x 3,000 second time skew problem

[Jan Beulich]

>>> On 08.06.12 at 16:18, Andy Smith <andy@strugglers.net> wrote:
> I have two installs of Xen 4.0.1 on Debian squeeze dom0s that
> occasionally skew the time by 3000 seconds. It is always 3000
> seconds and it always affects all guests.
> 
> Depending on the kernels in the guests they will say:
> 
>     tp0 kernel: [9339489.266542] Clocksource tsc unstable (delta = 
> -2999660317697 ns)
> 
> or perhaps the kernel doesn't notice but ntpd does:
> 
>     ls0 ntpd[15009]: time reset -2999.683517 s
> 
> the point is it does affect all guests.

So it ought to come from Dom0 (you didn't say whether there
was anything in its logs, or anything interesting going on on it
at the point where the time change happens), or would have
to be "invented" strait in the hypervisor (you didn't say
anything about its log either).

Also you didn't clarify how frequent "occasionally" is, and
whether this is e.g. always happening after a certain amount
of uptime, at some specific time during the day, etc.

> These are Debian squeeze installs so using packaged versions of xen
> and kernel:
> 
> ii  linux-image-2.6.32-5-xen-amd64 2.6.32-45 Linux 2.6.32 for 64-bit PCs, Xen dom0 
> support
> ii  xen-hypervisor-4.0-amd64       4.0.1-4   The Xen Hypervisor on AMD64

Clearly we'd want to know if this also occurs with current -unstable.

> One of the hosts affected is:
> 
> CPU: Xeon 5410
> Motherboard: Supermicro X7DCL-i
> 
> The other is:
> 
> CPU: Xeon E5606
> Motherboard: SUpermicro X8DTN+
> 
> but I do have other servers with same hardware as both of these that
> aren't affected.

Which hints towards something specific to the configurations of
the two affected hosts, suggesting that it would be a good idea
to narrow down the differences between affected and non-
affected, physically identical machines...

Jan

Re: Xen 4.0.x 3,000 second time skew problem

[Philippe.Simonet]

Hallo

i had the same  problem ** a lot of time **, on 4 different system, but not anymore since I reboot my machines at regular interval (each month ....)
what says your xm dmesg after the problem? what was your system uptime ? 

by me it occurs on 4 different  HP DL 385 with  AMD Opteron(tm) Processor 6174
but never on same machines but with Quad-Core AMD Opteron(tm) Processor 2384.

Regards

Philippe


> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Andy Smith
> Sent: Friday, June 08, 2012 4:19 PM
> To: xen-devel@lists.xen.org
> Cc: olivier.hanesse
> Subject: [Xen-devel] Xen 4.0.x 3,000 second time skew problem
> 
> Hi,
> 
> I have two installs of Xen 4.0.1 on Debian squeeze dom0s that occasionally
> skew the time by 3000 seconds. It is always 3000 seconds and it always
> affects all guests.
> 
> Depending on the kernels in the guests they will say:
> 
>     tp0 kernel: [9339489.266542] Clocksource tsc unstable (delta = -
> 2999660317697 ns)
> 
> or perhaps the kernel doesn't notice but ntpd does:
> 
>     ls0 ntpd[15009]: time reset -2999.683517 s
> 
> the point is it does affect all guests.
> 
> Back in February 2011 there was a thread started by Olivier Hanesse which
> appears to be exactly the same issue:
> 
>     http://old-list-archives.xen.org/archives/html/xen-users/2011-
> 02/msg00609.html
> 
> Like Olivier, one of my hosts used to run Xen 3.x and did not experience this
> issue until after it was upgraded to Xen 4. The other has only ever run 4.0.x
> so I don't know.
> 
> The above thread ends with Olivier using clocksource=pit on his Xen
> command line and waiting to see if it fixed things. If you are reading,
> Olivier, did it fix things for you?
> 
> These are Debian squeeze installs so using packaged versions of xen and
> kernel:
> 
> ii  linux-image-2.6.32-5-xen-amd64 2.6.32-45 Linux 2.6.32 for 64-bit PCs, Xen
> dom0 support
> ii  xen-hypervisor-4.0-amd64       4.0.1-4   The Xen Hypervisor on AMD64
> 
> One of the hosts affected is:
> 
> CPU: Xeon 5410
> Motherboard: Supermicro X7DCL-i
> 
> The other is:
> 
> CPU: Xeon E5606
> Motherboard: SUpermicro X8DTN+
> 
> but I do have other servers with same hardware as both of these that aren't
> affected.
> 
> Cheers,
> Andy
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel