domU clock over 23s off

domU clock over 23s off

[Saku Ytti]

Hey,

I have rather new installation of xen, dom0 is running ntpd and is
perfectly in time. All domU's are ~23s in future. Running ntpd on
the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
appears to have no effect. I'm running xen-unstable and 2.6.16-rc4 
in AMD64 box.

Any tip would be appricated.

Thanks,
-- 
  ++ytti

Re: domU clock over 23s off

[Saku Ytti]

On (2006-02-26 12:07 +0200), Saku Ytti wrote:

 Just noticed '[Xen-devel] domU time lagging'. And indeed 'date -s' 
in dom0 sets time right in domU's.
 So I guess I should try the Kim's patch next time I'm reloading
the box.

 Is this fix going to be pushed to mercurial anytime soon?

> Hey,
> 
> I have rather new installation of xen, dom0 is running ntpd and is
> perfectly in time. All domU's are ~23s in future. Running ntpd on
> the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> appears to have no effect. I'm running xen-unstable and 2.6.16-rc4 
> in AMD64 box.
> 
> Any tip would be appricated.
> 
> Thanks,
> -- 
>   ++ytti
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
  ++ytti

Re: domU clock over 23s off

[Saku Ytti]

On (2006-02-26 12:07 +0200), Saku Ytti wrote:

Just noticed '[Xen-devel] domU time lagging'. And indeed 'date -s'
in dom0 sets time right in domU's.
 So I guess I should try the Kim's patch next time I'm reloading
the box.

Is this fix going to be pushed to mercurial anytime soon?

> Hey,
> 
> I have rather new installation of xen, dom0 is running ntpd and is
> perfectly in time. All domU's are ~23s in future. Running ntpd on
> the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> appears to have no effect. I'm running xen-unstable and 2.6.16-rc4 
> in AMD64 box.
> 
> Any tip would be appricated.
> 
> Thanks,
> -- 
>   ++ytti
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
  ++ytti

Re: domU clock over 23s off

[Keir Fraser]

On 26 Feb 2006, at 10:07, Saku Ytti wrote:

> I have rather new installation of xen, dom0 is running ntpd and is
> perfectly in time. All domU's are ~23s in future. Running ntpd on
> the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> appears to have no effect. I'm running xen-unstable and 2.6.16-rc4
> in AMD64 box.
>
> Any tip would be appricated.

Booting the domU's with 'independent_wallclock' as a command-line 
parameter, and then running a standard vendor ntp init script (usually 
ntpdate followed by starting ntpd) certainly *should* have fixed the 
domU time problem.

Just setting independent_wallclock (no ntpd) would have no effect. ntpd 
without independent_wallclock also probably wouldn't work well.

I'll check in a (temporary) patch to periodically synchronise Xen's 
wallclock time with that of domain0. The best fix would be to move 
ntp's adtimex() syscall down into Xen, but that'll require some heavier 
lifting (I think it'll work well though, when it's done).

  -- Keir

Re: domU clock over 23s off

[Keir Fraser]

On 26 Feb 2006, at 14:47, Keir Fraser wrote:

> I'll check in a (temporary) patch to periodically synchronise Xen's 
> wallclock time with that of domain0.

The patch is now applied to the unstable and 3.0-testing trees.

  -- Keir

Re: domU clock over 23s off

[Saku Ytti]

On (2006-02-26 15:47 +0000), Keir Fraser wrote:
> 
> On 26 Feb 2006, at 14:47, Keir Fraser wrote:
> 
> >I'll check in a (temporary) patch to periodically synchronise Xen's 
> >wallclock time with that of domain0.
> 
> The patch is now applied to the unstable and 3.0-testing trees.

Thanks, will try it later, for now running 'date -s $(date +%H:%M:%S)'
in cron.hourly :)

-- 
  ++ytti

Re: domU clock over 23s off

[Saku Ytti]

On (2006-02-26 15:47 +0000), Keir Fraser wrote:

> >I'll check in a (temporary) patch to periodically synchronise Xen's 
> >wallclock time with that of domain0.
> 
> The patch is now applied to the unstable and 3.0-testing trees.

Thanks, will try it later, for now running 'date -s $(date +%H:%M:%S)'
in cron.hourly :)

-- 
  ++ytti

Re: domU clock over 23s off

[James Henderson]

Hello,

In Xen 2.0.7 I've had clock issues in domU when using OpenNTPD in dom0.
 When I switched to "legacy" ntpd the clocks went back in sync.

It doesn't sound like your server has this particular problem, but I
thought I'd mention it in case you're running OpenNTPD.

-James

Saku Ytti wrote:
> Hey,
> 
> I have rather new installation of xen, dom0 is running ntpd and is
> perfectly in time. All domU's are ~23s in future. Running ntpd on
> the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> appears to have no effect. I'm running xen-unstable and 2.6.16-rc4 
> in AMD64 box.
> 
> Any tip would be appricated.
> 
> Thanks,

Re: domU clock over 23s off

[Saku Ytti]

On (2006-02-26 16:55 -0700), James Henderson wrote:
> Hello,
> 
> In Xen 2.0.7 I've had clock issues in domU when using OpenNTPD in dom0.
>  When I switched to "legacy" ntpd the clocks went back in sync.
> 
> It doesn't sound like your server has this particular problem, but I
> thought I'd mention it in case you're running OpenNTPD.

thanks for the tip, bit indeed I'm using original NTPD.

> -James
> 
> Saku Ytti wrote:
> > Hey,
> > 
> > I have rather new installation of xen, dom0 is running ntpd and is
> > perfectly in time. All domU's are ~23s in future. Running ntpd on
> > the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> > appears to have no effect. I'm running xen-unstable and 2.6.16-rc4 
> > in AMD64 box.
> > 
> > Any tip would be appricated.
> > 
> > Thanks,
> 

-- 
  ++ytti

Re: domU clock over 23s off

[Harald Koenig]

Hi,

On Feb 26, Keir Fraser wrote:

> >I have rather new installation of xen, dom0 is running ntpd and is
> >perfectly in time. All domU's are ~23s in future. Running ntpd on
> >the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> >appears to have no effect. I'm running xen-unstable and 2.6.16-rc4
> >in AMD64 box.
>
> Booting the domU's with 'independent_wallclock' as a command-line 
> parameter, and then running a standard vendor ntp init script (usually 
> ntpdate followed by starting ntpd) certainly *should* have fixed the 
> domU time problem.

I run into the same problem (ntp not working), so I set independent_wallclock
and ntp un domU can set the system time.  thanks for the hint!

BUT!!!  with independent_wallclock I have a new problem:
now the hw clock can't be read anymore

	root@rhel4 # /sbin/hwclock --debug
	hwclock from util-linux-2.12a
	hwclock: Open of /dev/rtc failed, errno=19: No such device.
	No usable clock interface found.
	Cannot access the Hardware Clock via any known method.


this causes system to start up with system time "1 Jan 1970"

	# grep 1970 /var/log/boot.log
	Jan  1 01:00:10 os2-rhel4 date: Do Jan  1 01:00:10 CET 1970 
	Jan  1 01:00:10 os2-rhel4 rc.sysinit: Stelle Uhr  (localtime) auf: Do Jan  1 01:00:10 CET 1970 succeeded 


and in /var/log/messages:

	Jul 11 17:26:00 os2-rhel4 exiting on signal 15
	Jan  1 01:00:17 os2-rhel4 syslogd 1.4.1: restart.
	...
	Jan  1 01:00:26 os2-rhel4 xinetd[1307]: xinetd Version 2.3.13 started with libwrap loadavg options compiled in.
	Jan  1 01:00:26 os2-rhel4 xinetd[1307]: Started working: 2 available services
	Jul 11 17:31:56 os2-rhel4 ntpdate[1321]: step time server 10.0.5.7 offset 1152692789.758448 sec
	Jul 11 17:31:56 os2-rhel4 ntpd:  succeeded
	Jul 11 17:31:56 os2-rhel4 ntpd: Starten von ntpd succeeded



not only there are many bogus time stamps in log files, it also causes
a file system check at every startup because the fs was mounted last time
"so long ago":

	# /sbin/tune2fs -l /dev/hda1 |grep 1970  
	Last mount time:          Thu Jan  1 01:00:21 1970
	Last checked:             Thu Jan  1 01:01:36 1970



my current setup is:  AMD64 dual-core dual-opteron 

xen:
	Xen version 3.0.1 (xenod@cl.cam.ac.uk) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) Tue Jan 31 15:26:55 GMT 2006
	Latest ChangeSet: Tue Jan 31 11:04:12 2006 +0100 8738:eff96462fde8

dom0 and domU kernel:
	Linux version 2.6.12.6-xen (xenod@firebug.cl.cam.ac.uk) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #1 SMP Tue Jan 31 15:36:43 GMT 2006




will an update to a more recent XEN version fix these problems ?

if yes, what do I have to replace/update ? 
only domU kernel? or dom0+domU ? 
only hypervisor? or just everything ?



thanks for every help!!

Harald Koenig
-- 
"I hope to die                                      ___       _____
before I *have* to use Microsoft Word.",           0--,|    /OOOOOOO\
Donald E. Knuth, 02-Oct-2001 in Tuebingen.        <_/  /  /OOOOOOOOOOO\
                                                    \  \/OOOOOOOOOOOOOOO\
                                                      \ OOOOOOOOOOOOOOOOO|//
Harald Koenig                                          \/\/\/\/\/\/\/\/\/
science+computing ag                                    //  /     \\  \
koenig@science-computing.de                            ^^^^^       ^^^^^

Re: domU clock over 23s off

[Keir Fraser]

On 12 Jul 2006, at 10:00, Harald Koenig wrote:

> I run into the same problem (ntp not working), so I set 
> independent_wallclock
> and ntp un domU can set the system time.  thanks for the hint!
>
> BUT!!!  with independent_wallclock I have a new problem:
> now the hw clock can't be read anymore
>
> 	root@rhel4 # /sbin/hwclock --debug
> 	hwclock from util-linux-2.12a
> 	hwclock: Open of /dev/rtc failed, errno=19: No such device.
> 	No usable clock interface found.
> 	Cannot access the Hardware Clock via any known method.

RTC is inaccessible to domU regardless of independent_wallclock.

  -- Keir

Re: domU clock over 23s off

[Harald Koenig]

On Jul 12, Keir Fraser wrote:

> RTC is inaccessible to domU regardless of independent_wallclock.

so what's your suggestion for a setup which allows both 

- correct startup time (mount time, syslog start time etc), and
- working ntp ?


thanks,

Harald Koenig
-- 
"I hope to die                                      ___       _____
before I *have* to use Microsoft Word.",           0--,|    /OOOOOOO\
Donald E. Knuth, 02-Oct-2001 in Tuebingen.        <_/  /  /OOOOOOOOOOO\
                                                    \  \/OOOOOOOOOOOOOOO\
                                                      \ OOOOOOOOOOOOOOOOO|//
Harald Koenig                                          \/\/\/\/\/\/\/\/\/
science+computing ag                                    //  /     \\  \
koenig@science-computing.de                            ^^^^^       ^^^^^

Re: domU clock over 23s off

[Keir Fraser]

On 12 Jul 2006, at 10:56, Harald Koenig wrote:

>> RTC is inaccessible to domU regardless of independent_wallclock.
>
> so what's your suggestion for a setup which allows both
>
> - correct startup time (mount time, syslog start time etc), and
> - working ntp ?

Run ntpd in domain0, and don't use independent_wallclock.

  -- Keir

Re: domU clock over 23s off

[Harald Koenig]

On Jul 12, Keir Fraser wrote:

> >- correct startup time (mount time, syslog start time etc), and
> >- working ntp ?
> 
> Run ntpd in domain0, and don't use independent_wallclock.

that was my default setup I started with, until yesterday I realized that the
domUs don't keep the time in sync.  dom0 (FC4) is perfectly in sync running ntp. 

the domUs run ntp too, but there they don't manage to set/adjust the system time,
so the ntp on domU logs the time skew.  from syslog of a RH9 client:

Jun 26 20:08:25 os-rh9 ntpd[1666]: time reset +12.532031 s
Jun 26 20:24:51 os-rh9 ntpd[1666]: time reset +12.597371 s
Jun 26 20:41:07 os-rh9 ntpd[1666]: time reset +12.761633 s
Jun 26 21:02:54 os-rh9 ntpd[1666]: time reset +12.843185 s
Jun 26 21:19:18 os-rh9 ntpd[1666]: time reset +12.930329 s
Jun 26 21:38:59 os-rh9 ntpd[1666]: time reset +13.028862 s
Jun 26 21:56:19 os-rh9 ntpd[1666]: time reset +13.159422 s
...
Jun 27 20:05:22 os-rh9 ntpd[1666]: time reset +19.513893 s
Jun 27 20:22:37 os-rh9 ntpd[1666]: time reset +19.640442 s
Jun 27 20:37:47 os-rh9 ntpd[1666]: time reset +19.718607 s
Jun 27 20:55:09 os-rh9 ntpd[1666]: time reset +19.769548 s
Jun 27 21:13:31 os-rh9 ntpd[1666]: time reset +19.865546 s
Jun 27 21:30:49 os-rh9 ntpd[1666]: time reset +19.965029 s


does running ntp in a domU disable the clock synchronisation between dom0 and domU ?
 


Harald Koenig
-- 
"I hope to die                                      ___       _____
before I *have* to use Microsoft Word.",           0--,|    /OOOOOOO\
Donald E. Knuth, 02-Oct-2001 in Tuebingen.        <_/  /  /OOOOOOOOOOO\
                                                    \  \/OOOOOOOOOOOOOOO\
                                                      \ OOOOOOOOOOOOOOOOO|//
Harald Koenig                                          \/\/\/\/\/\/\/\/\/
science+computing ag                                    //  /     \\  \
koenig@science-computing.de                            ^^^^^       ^^^^^

Re: domU clock over 23s off

[Saku Ytti]

[-- Attachment #1.1: Type: text/plain, Size: 358 bytes --]

On (2006-07-12 15:37 +0200), Harald Koenig wrote:
 
> does running ntp in a domU disable the clock synchronisation between dom0 and domU ?

I do this:
[ytti@ytti.fi ~]% ssh root@herra.ip.fi "cat /etc/cron.hourly/ntp-hack"
#!/bin/sh

PATH=/bin

date -s $(date +%H:%M:%S) > /dev/null
[ytti@ytti.fi ~]%


Ugly, but works for me.

-- 
  ++ytti

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU clock over 23s off

[Harald Koenig]

On Jul 12, Saku Ytti wrote:

> On (2006-07-12 15:37 +0200), Harald Koenig wrote:
>  
> > does running ntp in a domU disable the clock synchronisation between dom0 and domU ?
> 
> I do this:
> [ytti@ytti.fi ~]% ssh root@herra.ip.fi "cat /etc/cron.hourly/ntp-hack"
> #!/bin/sh
> 
> PATH=/bin
> 
> date -s $(date +%H:%M:%S) > /dev/null

thanks for the hint!  

indeed, running this on dom0 also sets the clocks for domU.


BUT: it also causes clock jumps up to 0.999 secs every our because it can't 
handle fractions of a second.   and it always steps back the time
giving a non-monotonic system time -- too bad :-(((

that's not what time syncronisation and ntp is all about...


I tried to keep the fractions of seconds but this doesn't work:

	# date -s $( date +%H:%M:%S.%N )
	date: invalid date `21:38:53.976633000'


are there other methods  to trigger the time sync (push) between 
dom0 and domU without (re-) setting the system time in dom0 ? 


looks like I have to live with a jumpy time for now :-(


thanks for your help,

Harald Koenig
-- 
"I hope to die                                      ___       _____
before I *have* to use Microsoft Word.",           0--,|    /OOOOOOO\
Donald E. Knuth, 02-Oct-2001 in Tuebingen.        <_/  /  /OOOOOOOOOOO\
                                                    \  \/OOOOOOOOOOOOOOO\
                                                      \ OOOOOOOOOOOOOOOOO|//
Harald Koenig                                          \/\/\/\/\/\/\/\/\/
science+computing ag                                    //  /     \\  \
koenig@science-computing.de                            ^^^^^       ^^^^^

Re: domU clock over 23s off

[Keir Fraser]

On 12 Jul 2006, at 20:46, Harald Koenig wrote:

> I tried to keep the fractions of seconds but this doesn't work:
>
> 	# date -s $( date +%H:%M:%S.%N )
> 	date: invalid date `21:38:53.976633000'
>
>
> are there other methods  to trigger the time sync (push) between
> dom0 and domU without (re-) setting the system time in dom0 ?

It's supposed to happen automatically and with microsecond precision. 
Obviously something is wrong here.

  -- Keir

Re: domU clock over 23s off

[Harald Koenig]

On Jul 13, Keir Fraser wrote:

> 
> On 12 Jul 2006, at 20:46, Harald Koenig wrote:
> 
> >I tried to keep the fractions of seconds but this doesn't work:
> >
> >	# date -s $( date +%H:%M:%S.%N )
> >	date: invalid date `21:38:53.976633000'
> >
> >
> >are there other methods  to trigger the time sync (push) between
> >dom0 and domU without (re-) setting the system time in dom0 ?
> 
> It's supposed to happen automatically and with microsecond precision. 
> Obviously something is wrong here.

should this work already with xen 3.0.1, or do I have to upgrade ?

my current setup is:  AMD64 dual-core dual-opteron

xen:
        Xen version 3.0.1 (xenod@cl.cam.ac.uk) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) Tue Jan 31 15:26:55 GMT 2006
        Latest ChangeSet: Tue Jan 31 11:04:12 2006 +0100 8738:eff96462fde8

dom0 and domU kernel:
        Linux version 2.6.12.6-xen (xenod@firebug.cl.cam.ac.uk) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #1 SMP Tue Jan 31 15:36:43 GMT 2006

all from xensource.org...


Harald Koenig
-- 
"I hope to die                                      ___       _____
before I *have* to use Microsoft Word.",           0--,|    /OOOOOOO\
Donald E. Knuth, 02-Oct-2001 in Tuebingen.        <_/  /  /OOOOOOOOOOO\
                                                    \  \/OOOOOOOOOOOOOOO\
                                                      \ OOOOOOOOOOOOOOOOO|//
Harald Koenig                                          \/\/\/\/\/\/\/\/\/
science+computing ag                                    //  /     \\  \
koenig@science-computing.de                            ^^^^^       ^^^^^

Re: domU clock over 23s off

[Alexander Vallens]

I'm encountering similar issues on the current release of SuSE 10.1 on a
dual duo-core Opteron system. He this issue ever been fixed?

Thanks!
Alexander


On Feb 26, Keir Fraser wrote:

> >I have rather new installation of xen, dom0 is running ntpd and is
> >perfectly in time. All domU's are ~23s in future. Running ntpd on
> >the domU's seem to have no effect, /proc/sys/xen/independent_wallclock
> >appears to have no effect. I'm running xen-unstable and 2.6.16-rc4
> >in AMD64 box.
>
> Booting the domU's with 'independent_wallclock' as a command-line
> parameter, and then running a standard vendor ntp init script (usually
> ntpdate followed by starting ntpd) certainly *should* have fixed the
> domU time problem.

-- 
Alexander Vallens
System Administrator
Architectural Energy Corporation
http://www.archenergy.com
Voice:	303-459-7463
Fax:	303-444-4304