Xen 3.1.2 domU clock goes out of sync

Xen 3.1.2 domU clock goes out of sync

[Tõnu Raitviir]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1558 bytes --]



Hello!
I have understood that when /proc/sys/xen/independent_wallclock is 0, then 
domU clock stays in sync with dom0 clock. Now I have a case with Xen 3.1.2 
(also occured with 3.1.1, but I haven't documented it) where domU clock 
runs significantly faster - after 65 hours domU clock is 20 seconds ahead 
of dom0 clock.
In this case dom0 is not running ntpd. When ntpd is running on dom0, domU 
clock also stays correct. Is this normal behaviour?

Some examples to show clock differences:

domU# uptime
  14:54:09 up 2 days, 17:05,  1 user,  load average: 0.75, 0.49, 0.43

domU# ntpdate -q ntp.eenet.ee
server 193.40.133.142, stratum 1, offset -20.856732, delay 0.02579
19 Nov 14:54:13 ntpdate[8467]: step time server 193.40.133.142 offset -20.856732 sec

dom0# uptime
  14:55:02 up 2 days, 17:06,  1 user,  load average: 0.07, 0.02, 0.00

dom0# ntpdate -q ntp.eenet.ee
server 193.40.133.142, stratum 1, offset -0.438055, delay 0.02589
19 Nov 14:55:17 ntpdate[32284]: adjust time server 193.40.133.142 offset -0.438055 sec


Xen version used:

Xen version 3.1.2 (jussuf@localdomain) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) Fri Nov 16 17:52:59 EET 2007
Latest ChangeSet: Wed Nov 14 23:35:43 2007 +0000 15502:c6776b6da8ee

Linux xen-2 2.6.18.8-xen0 #1 SMP Fri Nov 16 17:39:31 EET 2007 x86_64 GNU/Linux

-- 
Tõnu Raitviir                           tonu.raitviir@eenet.ee
EENet                                   http://www.eenet.ee/
BalticGrid Project                      http://www.balticgrid.org/

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen 3.1.2 domU clock goes out of sync

[Ben Guthro]

[-- Attachment #1.1: Type: text/plain, Size: 2528 bytes --]

Clock skew is a known problem in virtualized systems.

VMWare wrote a whitepaper a while back which goes into some of the 
problems that they faced while dealing with this same issue...
http://www.vmware.com/pdf/vmware_timekeeping.pdf

The Xen project is not immune to these problems - however, some efforts 
have been made recently on the list to address some of these issues

See these threads:
http://lists.xensource.com/archives/html/xen-devel/2007-10/msg01010.html
http://lists.xensource.com/archives/html/xen-devel/2007-11/msg00554.html

What it basically comes down to is that it is necessary to keep the 
clock skew under 0.05% for NTP to work - so that was the design goal.

The code introduced by some recent patches claims to be accurate to 
0.02% (for at least HVM guests)


Hope this helps.

Ben


Tõnu Raitviir wrote:
>
>
> Hello!
> I have understood that when /proc/sys/xen/independent_wallclock is 0, 
> then domU clock stays in sync with dom0 clock. Now I have a case with 
> Xen 3.1.2 (also occured with 3.1.1, but I haven't documented it) where 
> domU clock runs significantly faster - after 65 hours domU clock is 20 
> seconds ahead of dom0 clock.
> In this case dom0 is not running ntpd. When ntpd is running on dom0, 
> domU clock also stays correct. Is this normal behaviour?
>
> Some examples to show clock differences:
>
> domU# uptime
>  14:54:09 up 2 days, 17:05,  1 user,  load average: 0.75, 0.49, 0.43
>
> domU# ntpdate -q ntp.eenet.ee
> server 193.40.133.142, stratum 1, offset -20.856732, delay 0.02579
> 19 Nov 14:54:13 ntpdate[8467]: step time server 193.40.133.142 offset 
> -20.856732 sec
>
> dom0# uptime
>  14:55:02 up 2 days, 17:06,  1 user,  load average: 0.07, 0.02, 0.00
>
> dom0# ntpdate -q ntp.eenet.ee
> server 193.40.133.142, stratum 1, offset -0.438055, delay 0.02589
> 19 Nov 14:55:17 ntpdate[32284]: adjust time server 193.40.133.142 
> offset -0.438055 sec
>
>
> Xen version used:
>
> Xen version 3.1.2 (jussuf@localdomain) (gcc version 4.1.2 20061115 
> (prerelease) (Debian 4.1.1-21)) Fri Nov 16 17:52:59 EET 2007
> Latest ChangeSet: Wed Nov 14 23:35:43 2007 +0000 15502:c6776b6da8ee
>
> Linux xen-2 2.6.18.8-xen0 #1 SMP Fri Nov 16 17:39:31 EET 2007 x86_64 
> GNU/Linux
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   


[-- Attachment #1.2: Type: text/html, Size: 3437 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen 3.1.2 domU clock goes out of sync

[Keir Fraser]

Do all domUs stay in sync with each other, or do all guests drift randomly?

 -- Keir


On 20/11/07 13:33, "Tõnu Raitviir" <jussuf@eenet.ee> wrote:

> 
> 
> Hello!
> I have understood that when /proc/sys/xen/independent_wallclock is 0, then
> domU clock stays in sync with dom0 clock. Now I have a case with Xen 3.1.2
> (also occured with 3.1.1, but I haven't documented it) where domU clock
> runs significantly faster - after 65 hours domU clock is 20 seconds ahead
> of dom0 clock.
> In this case dom0 is not running ntpd. When ntpd is running on dom0, domU
> clock also stays correct. Is this normal behaviour?
> 
> Some examples to show clock differences:
> 
> domU# uptime
>   14:54:09 up 2 days, 17:05,  1 user,  load average: 0.75, 0.49, 0.43
> 
> domU# ntpdate -q ntp.eenet.ee
> server 193.40.133.142, stratum 1, offset -20.856732, delay 0.02579
> 19 Nov 14:54:13 ntpdate[8467]: step time server 193.40.133.142 offset
> -20.856732 sec
> 
> dom0# uptime
>   14:55:02 up 2 days, 17:06,  1 user,  load average: 0.07, 0.02, 0.00
> 
> dom0# ntpdate -q ntp.eenet.ee
> server 193.40.133.142, stratum 1, offset -0.438055, delay 0.02589
> 19 Nov 14:55:17 ntpdate[32284]: adjust time server 193.40.133.142 offset
> -0.438055 sec
> 
> 
> Xen version used:
> 
> Xen version 3.1.2 (jussuf@localdomain) (gcc version 4.1.2 20061115
> (prerelease) (Debian 4.1.1-21)) Fri Nov 16 17:52:59 EET 2007
> Latest ChangeSet: Wed Nov 14 23:35:43 2007 +0000 15502:c6776b6da8ee
> 
> Linux xen-2 2.6.18.8-xen0 #1 SMP Fri Nov 16 17:39:31 EET 2007 x86_64 GNU/Linux

Re: Xen 3.1.2 domU clock goes out of sync

[Tõnu Raitviir]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 581 bytes --]

On Tue, 20 Nov 2007, Keir Fraser wrote:

> Do all domUs stay in sync with each other, or do all guests drift randomly?

I didn't compare domU-s before, but they seem to be in sync. I stopped
ntpd on dom0 five hours ago, and now the results of ntpdate are:
dom0    offset -0.002353 sec
domU #1 offset -1.555219 sec
domU #2 offset -1.555261 sec
domU #3 offset -1.555243 sec

-- 
Tõnu Raitviir                           tonu.raitviir@eenet.ee
EENet                                   http://www.eenet.ee/
BalticGrid Project                      http://www.balticgrid.org/

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen 3.1.2 domU clock goes out of sync

[Keir Fraser]

I wonder if dom0 has remembered some ntp-derived adjustment values despite
ntpd no longer running. That would explain why it continues to keep good
time while all others drift. The others drifting is quite explicable -- they
are only as accurate as our estimate of platform timer (PIT or HPET)
frequency at boot time, and the accuracy of the crystal that drives that
timer is probably only good to about 100ppm.

So... running ntpd in dom0 is a good idea. :-)

 -- Keir

On 20/11/07 19:15, "Tõnu Raitviir" <jussuf@eenet.ee> wrote:

> On Tue, 20 Nov 2007, Keir Fraser wrote:
> 
>> Do all domUs stay in sync with each other, or do all guests drift randomly?
> 
> I didn't compare domU-s before, but they seem to be in sync. I stopped
> ntpd on dom0 five hours ago, and now the results of ntpdate are:
> dom0    offset -0.002353 sec
> domU #1 offset -1.555219 sec
> domU #2 offset -1.555261 sec
> domU #3 offset -1.555243 sec

Re: Xen 3.1.2 domU clock goes out of sync

[Tõnu Raitviir]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 878 bytes --]

On Tue, 20 Nov 2007, Keir Fraser wrote:

> I wonder if dom0 has remembered some ntp-derived adjustment values despite
> ntpd no longer running. That would explain why it continues to keep good
> time while all others drift.

Well... in the example in my first letter dom0 was booted without ntpd and 
after 2+ days its clock offset was only -0.438055 while domU had drifted 
far away. In that case there was no adjustment values to remember.

> So... running ntpd in dom0 is a good idea. :-)

I don't question that. But I wonder, isn't domU clock supposed to be 
always in sync with dom0, with or without ntpd? Or how exactly does ntpd 
in dom0 affect domU clock?

-- 
Tõnu Raitviir                           tonu.raitviir@eenet.ee
EENet                                   http://www.eenet.ee/
BalticGrid Project                      http://www.balticgrid.org/

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen 3.1.2 domU clock goes out of sync

[Keir Fraser]

On 20/11/07 21:52, "Tõnu Raitviir" <jussuf@eenet.ee> wrote:

>> I wonder if dom0 has remembered some ntp-derived adjustment values despite
>> ntpd no longer running. That would explain why it continues to keep good
>> time while all others drift.
> 
> Well... in the example in my first letter dom0 was booted without ntpd and
> after 2+ days its clock offset was only -0.438055 while domU had drifted
> far away. In that case there was no adjustment values to remember.

Could it not be a drift? e.g., Perhaps you ran ntpdate in dom0 but not the
domUs?

>> So... running ntpd in dom0 is a good idea. :-)
> 
> I don't question that. But I wonder, isn't domU clock supposed to be
> always in sync with dom0, with or without ntpd? Or how exactly does ntpd
> in dom0 affect domU clock?

dom0 updates Xen's shared notion of wallclock time periodically, but only if
dom0 is locked onto ntp.

 -- Keir

Re: Xen 3.1.2 domU clock goes out of sync

[Tõnu Raitviir]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 927 bytes --]

On Tue, 20 Nov 2007, Keir Fraser wrote:

> Could it not be a drift? e.g., Perhaps you ran ntpdate in dom0 but not the
> domUs?

That's possible, probably ntpdate was executed automatically on dom0. Now 
I did a new test, rebooted dom0 and made sure that no ntp* commands are
run. Results are somewhat different:
# uptime
  14:21:41 up 12:31,  1 user,  load average: 0.00, 0.00, 0.00

dom0	offset -7.190458
domU #1	offset -4.380381
domU #2	offset -4.380430
domU #3	offset -4.380401

So without any ntp interaction dom0 clock is running faster than domU 
clock and all domU clocks are in sync with each other, but still not 
with dom0.

Now I'm going to start ntpd again and make sure that it never stops :-)

-- 
Tõnu Raitviir                           tonu.raitviir@eenet.ee
EENet                                   http://www.eenet.ee/
BalticGrid Project                      http://www.balticgrid.org/

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel