domU time drifting

domU time drifting

[Carl Jones]

Hi,

I am trying to track down the cause of time drifting on our domUs.
We're using various versions of Xen, and various domU kernels.

dom0s all run ntpd and are in sync. No issues there.

dom0s are all running CentOS 5 64bit. domUs run various distributions
(the examples below happen to be Debian and Ubuntu only).

domUs that do not run ntpd tend to drift (both forwards and
backwards). domUs are using clocksource=xen. dom0s are using
clocksource=jiffies.

Some example setups:

Xen: 3.4.3
dom0: 2.6.18.8 x86_64 based on el5 sources
domU: 2.6.30.5 i386
domU uptime: 68 days

dom0: Sun Dec 12 23:45:59 UTC 2010
domU: Sun Dec 12 23:44:48 UTC 2010

Xen: 3.3.1
dom0 2.6.18.8 x86_64 from xen.org
domU: 2.6.33.3 i386
domU uptime: 34 days

dom0: Sun Dec 12 23:51:20 UTC 2010
domU: Sun Dec 12 23:50:39 UTC 2010

Xen: 3.4.3
dom0: 2.6.18.8 x86_64 based on el5 sources
domU: 2.6.30.5 i386
domU uptime: #1 28 days #2 26 days

dom0: Sun Dec 12 23:58:23 UTC 2010
domU #1: Sun Dec 12 23:58:03 UTC 2010
domU #2: Sun Dec 12 23:58:04 UTC 2010

The last example is interesting. Two domUs on the same host have the
same (incorrect) time. Same domU kernel. Different uptime.

I've tried suggestions in bug reports like these:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=534978
http://wiki.debian.org/Xen#A.27clocksource.2BAC8-0.3ATimewentbackwards.27
http://my.opera.com/marcomarongiu/blog/2010/08/18/debugging-ntp-again-part-4-and-last

But so far the problem persists.

Is there anything we can do on the host side of things to prevent
domUs from drifting? With clocksource=xen shouldn't the domUs stay in
sync provided the host keeps the correct time?

Please let me know if I can provide further details.

Regards,
Carl

Re: domU time drifting

[Keir Fraser]

If the domU kernels are pv_ops kernels, they don't keep their time in sync
with dom0 after boot. You have to run ntpd in every domU.

 -- Keir

On 13/12/2010 00:12, "Carl Jones" <carl@outerloop.net> wrote:

> Hi,
> 
> I am trying to track down the cause of time drifting on our domUs.
> We're using various versions of Xen, and various domU kernels.
> 
> dom0s all run ntpd and are in sync. No issues there.
> 
> dom0s are all running CentOS 5 64bit. domUs run various distributions
> (the examples below happen to be Debian and Ubuntu only).
> 
> domUs that do not run ntpd tend to drift (both forwards and
> backwards). domUs are using clocksource=xen. dom0s are using
> clocksource=jiffies.
> 
> Some example setups:
> 
> Xen: 3.4.3
> dom0: 2.6.18.8 x86_64 based on el5 sources
> domU: 2.6.30.5 i386
> domU uptime: 68 days
> 
> dom0: Sun Dec 12 23:45:59 UTC 2010
> domU: Sun Dec 12 23:44:48 UTC 2010
> 
> Xen: 3.3.1
> dom0 2.6.18.8 x86_64 from xen.org
> domU: 2.6.33.3 i386
> domU uptime: 34 days
> 
> dom0: Sun Dec 12 23:51:20 UTC 2010
> domU: Sun Dec 12 23:50:39 UTC 2010
> 
> Xen: 3.4.3
> dom0: 2.6.18.8 x86_64 based on el5 sources
> domU: 2.6.30.5 i386
> domU uptime: #1 28 days #2 26 days
> 
> dom0: Sun Dec 12 23:58:23 UTC 2010
> domU #1: Sun Dec 12 23:58:03 UTC 2010
> domU #2: Sun Dec 12 23:58:04 UTC 2010
> 
> The last example is interesting. Two domUs on the same host have the
> same (incorrect) time. Same domU kernel. Different uptime.
> 
> I've tried suggestions in bug reports like these:
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=534978
> http://wiki.debian.org/Xen#A.27clocksource.2BAC8-0.3ATimewentbackwards.27
> http://my.opera.com/marcomarongiu/blog/2010/08/18/debugging-ntp-again-part-4-a
> nd-last
> 
> But so far the problem persists.
> 
> Is there anything we can do on the host side of things to prevent
> domUs from drifting? With clocksource=xen shouldn't the domUs stay in
> sync provided the host keeps the correct time?
> 
> Please let me know if I can provide further details.
> 
> Regards,
> Carl
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: domU time drifting

[Carl Jones]

That explains it. Thanks Keir.

Regards,
Carl

On Mon, Dec 13, 2010 at 8:11 PM, Keir Fraser <keir@xen.org> wrote:
> If the domU kernels are pv_ops kernels, they don't keep their time in sync
> with dom0 after boot. You have to run ntpd in every domU.
>
>  -- Keir

Re: domU time drifting

[Jeremy Fitzhardinge]

On 12/12/2010 04:12 PM, Carl Jones wrote:
>  dom0s are using
> clocksource=jiffies.

Why?  I'm surprised you aren't seeing massive drift in dom0, beyond what
ntp can handle.

    J