[PATCH 0/3] xen: fix various wallclock problems

[PATCH 0/3] xen: fix various wallclock problems

[David Vrabel]

This series (with some toolstack updates) corrects a number of cases
where a guest can see an incorrect wallclock time.

1. Systems with NTP won't periodically synchronize the hardware RTC so
the wallclock may be incorrect after a reboot.

2. The wallclock is always ~500 ms behind the correct time.

3. If the system time was stepped and NTP isn't synchronized yet, the
wallclock will still have the incorrect time.  The fix for this
requires the toolstack to synchronize the wallclock -- patch #3
provides the mechanism for this.

David

[PATCH 1/3] xen: sync the CMOS RTC as well as the Xen wallclock

[David Vrabel]

From: David Vrabel <david.vrabel@citrix.com>

If NTP is used in dom0 and it is synchronized to its clock source,
then the kernel will periodically synchronize the Xen wallclock with
the system time.  Updates to the Xen wallclock do not persist across
reboots, so also synchronize the CMOS RTC (as on bare metal).

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
 arch/x86/xen/time.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 0296a95..5e7f536 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -14,6 +14,7 @@
 #include <linux/kernel_stat.h>
 #include <linux/math64.h>
 #include <linux/gfp.h>
+#include <linux/mc146818rtc.h>
 
 #include <asm/pvclock.h>
 #include <asm/xen/hypervisor.h>
@@ -208,6 +209,10 @@ static int xen_set_wallclock(unsigned long now)
 	if (!xen_initial_domain())
 		return -1;
 
+	/* Set the hardware RTC. */
+	mach_set_rtc_mmss(now);
+
+	/* Set the Xen wallclock. */
 	op.cmd = XENPF_settime;
 	op.u.settime.secs = now;
 	op.u.settime.nsecs = 0;
-- 
1.7.2.5

[PATCH 2/3] xen: add correct 500 ms offset when setting Xen wallclock

[David Vrabel]

From: David Vrabel <david.vrabel@citrix.com>

update_persistent_wallclock() (and hence xet_set_wallclock()) is
called 500 ms after the second.  xen_set_wallclock() was not
considering this so the Xen wallclock would end up ~500 ms behind the
correct time.

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
 arch/x86/xen/time.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 5e7f536..11770d0 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -212,10 +212,15 @@ static int xen_set_wallclock(unsigned long now)
 	/* Set the hardware RTC. */
 	mach_set_rtc_mmss(now);
 
-	/* Set the Xen wallclock. */
+	/*
+	 * Set the Xen wallclock.
+	 *
+	 * update_persistent_wallclock() is called ~500 ms after 'now'
+	 * so add an extra 500 ms.
+	 */
 	op.cmd = XENPF_settime;
 	op.u.settime.secs = now;
-	op.u.settime.nsecs = 0;
+	op.u.settime.nsecs = NSEC_PER_SEC / 2;
 	op.u.settime.system_time = xen_clocksource_read();
 
 	rc = HYPERVISOR_dom0_op(&op);
-- 
1.7.2.5

[PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[David Vrabel]

From: David Vrabel <david.vrabel@citrix.com>

Add a new ioctl to synchronize Xen's wallclock with the current system
time.

This may be used by the tools to ensure that newly created domains see
the correct wallclock time if NTP is not used in dom0 or if domains
are started before NTP has synchronized.

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
 arch/x86/xen/time.c   |   25 ++++++++++++++++++-------
 drivers/xen/privcmd.c |   12 ++++++++++++
 include/xen/privcmd.h |    8 ++++++++
 include/xen/xen-ops.h |    2 ++
 4 files changed, 40 insertions(+), 7 deletions(-)

diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 11770d0..d481ac9 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -8,6 +8,7 @@
  * Jeremy Fitzhardinge <jeremy@xensource.com>, XenSource Inc, 2007
  */
 #include <linux/kernel.h>
+#include <linux/export.h>
 #include <linux/interrupt.h>
 #include <linux/clocksource.h>
 #include <linux/clockchips.h>
@@ -192,6 +193,19 @@ static void xen_read_wallclock(struct timespec *ts)
 	put_cpu_var(xen_vcpu);
 }
 
+int xen_write_wallclock(const struct timespec *ts)
+{
+	struct xen_platform_op op;
+
+	op.cmd = XENPF_settime;
+	op.u.settime.secs = ts->tv_sec;
+	op.u.settime.nsecs = ts->tv_nsec;
+	op.u.settime.system_time = xen_clocksource_read();
+
+	return HYPERVISOR_dom0_op(&op);
+}
+EXPORT_SYMBOL_GPL(xen_write_wallclock);
+
 static unsigned long xen_get_wallclock(void)
 {
 	struct timespec ts;
@@ -202,7 +216,7 @@ static unsigned long xen_get_wallclock(void)
 
 static int xen_set_wallclock(unsigned long now)
 {
-	struct xen_platform_op op;
+	struct timespec ts;
 	int rc;
 
 	/* do nothing for domU */
@@ -218,12 +232,9 @@ static int xen_set_wallclock(unsigned long now)
 	 * update_persistent_wallclock() is called ~500 ms after 'now'
 	 * so add an extra 500 ms.
 	 */
-	op.cmd = XENPF_settime;
-	op.u.settime.secs = now;
-	op.u.settime.nsecs = NSEC_PER_SEC / 2;
-	op.u.settime.system_time = xen_clocksource_read();
-
-	rc = HYPERVISOR_dom0_op(&op);
+	ts.tv_sec = now;
+	ts.tv_nsec = NSEC_PER_SEC / 2;
+	rc = xen_write_wallclock(&ts);
 	WARN(rc != 0, "XENPF_settime failed: now=%ld\n", now);
 
 	return rc;
diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
index ccee0f1..ed2caf3 100644
--- a/drivers/xen/privcmd.c
+++ b/drivers/xen/privcmd.c
@@ -338,6 +338,14 @@ out:
 	return ret;
 }
 
+static long privcmd_ioctl_sync_wallclock(void)
+{
+	struct timespec ts;
+
+	getnstimeofday(&ts);
+	return xen_write_wallclock(&ts);
+}
+
 static long privcmd_ioctl(struct file *file,
 			  unsigned int cmd, unsigned long data)
 {
@@ -357,6 +365,10 @@ static long privcmd_ioctl(struct file *file,
 		ret = privcmd_ioctl_mmap_batch(udata);
 		break;
 
+	case IOCTL_PRIVCMD_SYNC_WALLCLOCK:
+		ret = privcmd_ioctl_sync_wallclock();
+		break;
+
 	default:
 		ret = -EINVAL;
 		break;
diff --git a/include/xen/privcmd.h b/include/xen/privcmd.h
index 17857fb..d17d087 100644
--- a/include/xen/privcmd.h
+++ b/include/xen/privcmd.h
@@ -66,6 +66,12 @@ struct privcmd_mmapbatch {
  * @cmd: IOCTL_PRIVCMD_HYPERCALL
  * @arg: &privcmd_hypercall_t
  * Return: Value returned from execution of the specified hypercall.
+ *
+ * @cmd: IOCTL_PRIVCMD_SYNC_WALLCLOCK
+ * @arg: Unused.
+ * Synchronizes the Xen wallclock with the current system time.
+ * Return: 0 on success, or -1 on error with errno set to EPERM or
+ * EACCES.
  */
 #define IOCTL_PRIVCMD_HYPERCALL					\
 	_IOC(_IOC_NONE, 'P', 0, sizeof(struct privcmd_hypercall))
@@ -73,5 +79,7 @@ struct privcmd_mmapbatch {
 	_IOC(_IOC_NONE, 'P', 2, sizeof(struct privcmd_mmap))
 #define IOCTL_PRIVCMD_MMAPBATCH					\
 	_IOC(_IOC_NONE, 'P', 3, sizeof(struct privcmd_mmapbatch))
+#define IOCTL_PRIVCMD_SYNC_WALLCLOCK				\
+	_IOC(_IOC_NONE, 'P', 5, 0)
 
 #endif /* __LINUX_PUBLIC_PRIVCMD_H__ */
diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
index 6a198e4..eefed22 100644
--- a/include/xen/xen-ops.h
+++ b/include/xen/xen-ops.h
@@ -29,4 +29,6 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
 			       unsigned long mfn, int nr,
 			       pgprot_t prot, unsigned domid);
 
+int xen_write_wallclock(const struct timespec *ts);
+
 #endif /* INCLUDE_XEN_OPS_H */
-- 
1.7.2.5

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[Konrad Rzeszutek Wilk]

On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
> From: David Vrabel <david.vrabel@citrix.com>
> 
> Add a new ioctl to synchronize Xen's wallclock with the current system
> time.
> 
> This may be used by the tools to ensure that newly created domains see
> the correct wallclock time if NTP is not used in dom0 or if domains
> are started before NTP has synchronized.

So... how does this work with NTPD? As in does ntpd _not_ update the
hwclock enough?


> 
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> ---
>  arch/x86/xen/time.c   |   25 ++++++++++++++++++-------
>  drivers/xen/privcmd.c |   12 ++++++++++++
>  include/xen/privcmd.h |    8 ++++++++
>  include/xen/xen-ops.h |    2 ++
>  4 files changed, 40 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> index 11770d0..d481ac9 100644
> --- a/arch/x86/xen/time.c
> +++ b/arch/x86/xen/time.c
> @@ -8,6 +8,7 @@
>   * Jeremy Fitzhardinge <jeremy@xensource.com>, XenSource Inc, 2007
>   */
>  #include <linux/kernel.h>
> +#include <linux/export.h>
>  #include <linux/interrupt.h>
>  #include <linux/clocksource.h>
>  #include <linux/clockchips.h>
> @@ -192,6 +193,19 @@ static void xen_read_wallclock(struct timespec *ts)
>  	put_cpu_var(xen_vcpu);
>  }
>  
> +int xen_write_wallclock(const struct timespec *ts)
> +{
> +	struct xen_platform_op op;
> +
> +	op.cmd = XENPF_settime;
> +	op.u.settime.secs = ts->tv_sec;
> +	op.u.settime.nsecs = ts->tv_nsec;
> +	op.u.settime.system_time = xen_clocksource_read();
> +
> +	return HYPERVISOR_dom0_op(&op);
> +}
> +EXPORT_SYMBOL_GPL(xen_write_wallclock);
> +
>  static unsigned long xen_get_wallclock(void)
>  {
>  	struct timespec ts;
> @@ -202,7 +216,7 @@ static unsigned long xen_get_wallclock(void)
>  
>  static int xen_set_wallclock(unsigned long now)
>  {
> -	struct xen_platform_op op;
> +	struct timespec ts;
>  	int rc;
>  
>  	/* do nothing for domU */
> @@ -218,12 +232,9 @@ static int xen_set_wallclock(unsigned long now)
>  	 * update_persistent_wallclock() is called ~500 ms after 'now'
>  	 * so add an extra 500 ms.
>  	 */
> -	op.cmd = XENPF_settime;
> -	op.u.settime.secs = now;
> -	op.u.settime.nsecs = NSEC_PER_SEC / 2;
> -	op.u.settime.system_time = xen_clocksource_read();
> -
> -	rc = HYPERVISOR_dom0_op(&op);
> +	ts.tv_sec = now;
> +	ts.tv_nsec = NSEC_PER_SEC / 2;
> +	rc = xen_write_wallclock(&ts);
>  	WARN(rc != 0, "XENPF_settime failed: now=%ld\n", now);
>  
>  	return rc;
> diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
> index ccee0f1..ed2caf3 100644
> --- a/drivers/xen/privcmd.c
> +++ b/drivers/xen/privcmd.c
> @@ -338,6 +338,14 @@ out:
>  	return ret;
>  }
>  
> +static long privcmd_ioctl_sync_wallclock(void)
> +{
> +	struct timespec ts;
> +
> +	getnstimeofday(&ts);
> +	return xen_write_wallclock(&ts);
> +}
> +
>  static long privcmd_ioctl(struct file *file,
>  			  unsigned int cmd, unsigned long data)
>  {
> @@ -357,6 +365,10 @@ static long privcmd_ioctl(struct file *file,
>  		ret = privcmd_ioctl_mmap_batch(udata);
>  		break;
>  
> +	case IOCTL_PRIVCMD_SYNC_WALLCLOCK:
> +		ret = privcmd_ioctl_sync_wallclock();
> +		break;
> +
>  	default:
>  		ret = -EINVAL;
>  		break;
> diff --git a/include/xen/privcmd.h b/include/xen/privcmd.h
> index 17857fb..d17d087 100644
> --- a/include/xen/privcmd.h
> +++ b/include/xen/privcmd.h
> @@ -66,6 +66,12 @@ struct privcmd_mmapbatch {
>   * @cmd: IOCTL_PRIVCMD_HYPERCALL
>   * @arg: &privcmd_hypercall_t
>   * Return: Value returned from execution of the specified hypercall.
> + *
> + * @cmd: IOCTL_PRIVCMD_SYNC_WALLCLOCK
> + * @arg: Unused.
> + * Synchronizes the Xen wallclock with the current system time.
> + * Return: 0 on success, or -1 on error with errno set to EPERM or
> + * EACCES.
>   */
>  #define IOCTL_PRIVCMD_HYPERCALL					\
>  	_IOC(_IOC_NONE, 'P', 0, sizeof(struct privcmd_hypercall))
> @@ -73,5 +79,7 @@ struct privcmd_mmapbatch {
>  	_IOC(_IOC_NONE, 'P', 2, sizeof(struct privcmd_mmap))
>  #define IOCTL_PRIVCMD_MMAPBATCH					\
>  	_IOC(_IOC_NONE, 'P', 3, sizeof(struct privcmd_mmapbatch))
> +#define IOCTL_PRIVCMD_SYNC_WALLCLOCK				\
> +	_IOC(_IOC_NONE, 'P', 5, 0)
>  
>  #endif /* __LINUX_PUBLIC_PRIVCMD_H__ */
> diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
> index 6a198e4..eefed22 100644
> --- a/include/xen/xen-ops.h
> +++ b/include/xen/xen-ops.h
> @@ -29,4 +29,6 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
>  			       unsigned long mfn, int nr,
>  			       pgprot_t prot, unsigned domid);
>  
> +int xen_write_wallclock(const struct timespec *ts);
> +
>  #endif /* INCLUDE_XEN_OPS_H */
> -- 
> 1.7.2.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[David Vrabel]

On 12/10/12 14:41, Konrad Rzeszutek Wilk wrote:
> On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
>> From: David Vrabel <david.vrabel@citrix.com>
>>
>> Add a new ioctl to synchronize Xen's wallclock with the current system
>> time.
>>
>> This may be used by the tools to ensure that newly created domains see
>> the correct wallclock time if NTP is not used in dom0 or if domains
>> are started before NTP has synchronized.
> 
> So... how does this work with NTPD? As in does ntpd _not_ update the
> hwclock enough?

Once NTPD is synchronized then the kernel updates the wallclock (and the
RTC with patch #1) every 11 mins.  I assume this is often enough given
how NTP adjusts the system time.

You only really need the tools to sync wallclock if system time was
stepped at start of day.  e.g., init scripts could do something like:

ntpdate pool.ntp.org
hwclock --systohc
xen-wallclock --systowc

David

>> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
>> ---
>>  arch/x86/xen/time.c   |   25 ++++++++++++++++++-------
>>  drivers/xen/privcmd.c |   12 ++++++++++++
>>  include/xen/privcmd.h |    8 ++++++++
>>  include/xen/xen-ops.h |    2 ++
>>  4 files changed, 40 insertions(+), 7 deletions(-)
>>
>> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
>> index 11770d0..d481ac9 100644
>> --- a/arch/x86/xen/time.c
>> +++ b/arch/x86/xen/time.c
>> @@ -8,6 +8,7 @@
>>   * Jeremy Fitzhardinge <jeremy@xensource.com>, XenSource Inc, 2007
>>   */
>>  #include <linux/kernel.h>
>> +#include <linux/export.h>
>>  #include <linux/interrupt.h>
>>  #include <linux/clocksource.h>
>>  #include <linux/clockchips.h>
>> @@ -192,6 +193,19 @@ static void xen_read_wallclock(struct timespec *ts)
>>  	put_cpu_var(xen_vcpu);
>>  }
>>  
>> +int xen_write_wallclock(const struct timespec *ts)
>> +{
>> +	struct xen_platform_op op;
>> +
>> +	op.cmd = XENPF_settime;
>> +	op.u.settime.secs = ts->tv_sec;
>> +	op.u.settime.nsecs = ts->tv_nsec;
>> +	op.u.settime.system_time = xen_clocksource_read();
>> +
>> +	return HYPERVISOR_dom0_op(&op);
>> +}
>> +EXPORT_SYMBOL_GPL(xen_write_wallclock);
>> +
>>  static unsigned long xen_get_wallclock(void)
>>  {
>>  	struct timespec ts;
>> @@ -202,7 +216,7 @@ static unsigned long xen_get_wallclock(void)
>>  
>>  static int xen_set_wallclock(unsigned long now)
>>  {
>> -	struct xen_platform_op op;
>> +	struct timespec ts;
>>  	int rc;
>>  
>>  	/* do nothing for domU */
>> @@ -218,12 +232,9 @@ static int xen_set_wallclock(unsigned long now)
>>  	 * update_persistent_wallclock() is called ~500 ms after 'now'
>>  	 * so add an extra 500 ms.
>>  	 */
>> -	op.cmd = XENPF_settime;
>> -	op.u.settime.secs = now;
>> -	op.u.settime.nsecs = NSEC_PER_SEC / 2;
>> -	op.u.settime.system_time = xen_clocksource_read();
>> -
>> -	rc = HYPERVISOR_dom0_op(&op);
>> +	ts.tv_sec = now;
>> +	ts.tv_nsec = NSEC_PER_SEC / 2;
>> +	rc = xen_write_wallclock(&ts);
>>  	WARN(rc != 0, "XENPF_settime failed: now=%ld\n", now);
>>  
>>  	return rc;
>> diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
>> index ccee0f1..ed2caf3 100644
>> --- a/drivers/xen/privcmd.c
>> +++ b/drivers/xen/privcmd.c
>> @@ -338,6 +338,14 @@ out:
>>  	return ret;
>>  }
>>  
>> +static long privcmd_ioctl_sync_wallclock(void)
>> +{
>> +	struct timespec ts;
>> +
>> +	getnstimeofday(&ts);
>> +	return xen_write_wallclock(&ts);
>> +}
>> +
>>  static long privcmd_ioctl(struct file *file,
>>  			  unsigned int cmd, unsigned long data)
>>  {
>> @@ -357,6 +365,10 @@ static long privcmd_ioctl(struct file *file,
>>  		ret = privcmd_ioctl_mmap_batch(udata);
>>  		break;
>>  
>> +	case IOCTL_PRIVCMD_SYNC_WALLCLOCK:
>> +		ret = privcmd_ioctl_sync_wallclock();
>> +		break;
>> +
>>  	default:
>>  		ret = -EINVAL;
>>  		break;
>> diff --git a/include/xen/privcmd.h b/include/xen/privcmd.h
>> index 17857fb..d17d087 100644
>> --- a/include/xen/privcmd.h
>> +++ b/include/xen/privcmd.h
>> @@ -66,6 +66,12 @@ struct privcmd_mmapbatch {
>>   * @cmd: IOCTL_PRIVCMD_HYPERCALL
>>   * @arg: &privcmd_hypercall_t
>>   * Return: Value returned from execution of the specified hypercall.
>> + *
>> + * @cmd: IOCTL_PRIVCMD_SYNC_WALLCLOCK
>> + * @arg: Unused.
>> + * Synchronizes the Xen wallclock with the current system time.
>> + * Return: 0 on success, or -1 on error with errno set to EPERM or
>> + * EACCES.
>>   */
>>  #define IOCTL_PRIVCMD_HYPERCALL					\
>>  	_IOC(_IOC_NONE, 'P', 0, sizeof(struct privcmd_hypercall))
>> @@ -73,5 +79,7 @@ struct privcmd_mmapbatch {
>>  	_IOC(_IOC_NONE, 'P', 2, sizeof(struct privcmd_mmap))
>>  #define IOCTL_PRIVCMD_MMAPBATCH					\
>>  	_IOC(_IOC_NONE, 'P', 3, sizeof(struct privcmd_mmapbatch))
>> +#define IOCTL_PRIVCMD_SYNC_WALLCLOCK				\
>> +	_IOC(_IOC_NONE, 'P', 5, 0)
>>  
>>  #endif /* __LINUX_PUBLIC_PRIVCMD_H__ */
>> diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
>> index 6a198e4..eefed22 100644
>> --- a/include/xen/xen-ops.h
>> +++ b/include/xen/xen-ops.h
>> @@ -29,4 +29,6 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
>>  			       unsigned long mfn, int nr,
>>  			       pgprot_t prot, unsigned domid);
>>  
>> +int xen_write_wallclock(const struct timespec *ts);
>> +
>>  #endif /* INCLUDE_XEN_OPS_H */
>> -- 
>> 1.7.2.5
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[Konrad Rzeszutek Wilk]

On Fri, Oct 12, 2012 at 10:02 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> On 12/10/12 14:41, Konrad Rzeszutek Wilk wrote:
>> On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
>>> From: David Vrabel <david.vrabel@citrix.com>
>>>
>>> Add a new ioctl to synchronize Xen's wallclock with the current system
>>> time.
>>>
>>> This may be used by the tools to ensure that newly created domains see
>>> the correct wallclock time if NTP is not used in dom0 or if domains
>>> are started before NTP has synchronized.
>>
>> So... how does this work with NTPD? As in does ntpd _not_ update the
>> hwclock enough?
>
> Once NTPD is synchronized then the kernel updates the wallclock (and the
> RTC with patch #1) every 11 mins.  I assume this is often enough given
> how NTP adjusts the system time.
>
> You only really need the tools to sync wallclock if system time was
> stepped at start of day.  e.g., init scripts could do something like:
>
> ntpdate pool.ntp.org
> hwclock --systohc
> xen-wallclock --systowc

I think I am missing something. The hwclock should end up in the
xen_set_wallclock call. And from there on, the ntpd would update the
wallclock if it got skewed enough? Or is the system time not calling
the wall-clock enough? If that is the case, would just adding this in
the crontab be enough:

*/11 * * * * hwclock --systohc

?

Re: [PATCH 1/3] xen: sync the CMOS RTC as well as the Xen wallclock

[Konrad Rzeszutek Wilk]

On Fri, Oct 12, 2012 at 01:57:12PM +0100, David Vrabel wrote:
> From: David Vrabel <david.vrabel@citrix.com>
> 
> If NTP is used in dom0 and it is synchronized to its clock source,
> then the kernel will periodically synchronize the Xen wallclock with
> the system time.  Updates to the Xen wallclock do not persist across
> reboots, so also synchronize the CMOS RTC (as on bare metal).
> 
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> ---
>  arch/x86/xen/time.c |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> index 0296a95..5e7f536 100644
> --- a/arch/x86/xen/time.c
> +++ b/arch/x86/xen/time.c
> @@ -14,6 +14,7 @@
>  #include <linux/kernel_stat.h>
>  #include <linux/math64.h>
>  #include <linux/gfp.h>
> +#include <linux/mc146818rtc.h>
>  
>  #include <asm/pvclock.h>
>  #include <asm/xen/hypervisor.h>
> @@ -208,6 +209,10 @@ static int xen_set_wallclock(unsigned long now)
>  	if (!xen_initial_domain())
>  		return -1;
>  
> +	/* Set the hardware RTC. */
> +	mach_set_rtc_mmss(now);

So how does this work? Is the hypervisor traping on the RTC CMOS clock?

> +
> +	/* Set the Xen wallclock. */
>  	op.cmd = XENPF_settime;
>  	op.u.settime.secs = now;
>  	op.u.settime.nsecs = 0;
> -- 
> 1.7.2.5

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[David Vrabel]

On 12/10/12 15:29, Konrad Rzeszutek Wilk wrote:
> On Fri, Oct 12, 2012 at 10:02 AM, David Vrabel <david.vrabel@citrix.com> wrote:
>> On 12/10/12 14:41, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
>>>> From: David Vrabel <david.vrabel@citrix.com>
>>>>
>>>> Add a new ioctl to synchronize Xen's wallclock with the current system
>>>> time.
>>>>
>>>> This may be used by the tools to ensure that newly created domains see
>>>> the correct wallclock time if NTP is not used in dom0 or if domains
>>>> are started before NTP has synchronized.
>>>
>>> So... how does this work with NTPD? As in does ntpd _not_ update the
>>> hwclock enough?
>>
>> Once NTPD is synchronized then the kernel updates the wallclock (and the
>> RTC with patch #1) every 11 mins.  I assume this is often enough given
>> how NTP adjusts the system time.
>>
>> You only really need the tools to sync wallclock if system time was
>> stepped at start of day.  e.g., init scripts could do something like:
>>
>> ntpdate pool.ntp.org
>> hwclock --systohc
>> xen-wallclock --systowc
> 
> I think I am missing something. The hwclock should end up in the
> xen_set_wallclock call. And from there on, the ntpd would update the
> wallclock if it got skewed enough? Or is the system time not calling
> the wall-clock enough? If that is the case, would just adding this in
> the crontab be enough:

hwclock talks to /dev/rtc which writes to the CMOS directly and does not
call update_persistent_clock() (or xen_set_wallclock()).

David

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[Konrad Rzeszutek Wilk]

On Fri, Oct 12, 2012 at 03:59:25PM +0100, David Vrabel wrote:
> On 12/10/12 15:29, Konrad Rzeszutek Wilk wrote:
> > On Fri, Oct 12, 2012 at 10:02 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> >> On 12/10/12 14:41, Konrad Rzeszutek Wilk wrote:
> >>> On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
> >>>> From: David Vrabel <david.vrabel@citrix.com>
> >>>>
> >>>> Add a new ioctl to synchronize Xen's wallclock with the current system
> >>>> time.
> >>>>
> >>>> This may be used by the tools to ensure that newly created domains see
> >>>> the correct wallclock time if NTP is not used in dom0 or if domains
> >>>> are started before NTP has synchronized.
> >>>
> >>> So... how does this work with NTPD? As in does ntpd _not_ update the
> >>> hwclock enough?
> >>
> >> Once NTPD is synchronized then the kernel updates the wallclock (and the
> >> RTC with patch #1) every 11 mins.  I assume this is often enough given
> >> how NTP adjusts the system time.
> >>
> >> You only really need the tools to sync wallclock if system time was
> >> stepped at start of day.  e.g., init scripts could do something like:
> >>
> >> ntpdate pool.ntp.org
> >> hwclock --systohc
> >> xen-wallclock --systowc
> > 
> > I think I am missing something. The hwclock should end up in the
> > xen_set_wallclock call. And from there on, the ntpd would update the
> > wallclock if it got skewed enough? Or is the system time not calling
> > the wall-clock enough? If that is the case, would just adding this in
> > the crontab be enough:
> 
> hwclock talks to /dev/rtc which writes to the CMOS directly and does not
> call update_persistent_clock() (or xen_set_wallclock()).

/me scratches his head.

I recall that the update_persistent_clock() was being called.. with
hwclock -w (which is the same as --systohc).

That seems like a bug in the generic code - I would think that
hwclock would update the wallclock, not just the RTC.

Oh, it is whoever calls 'adjtimex' syscall ends up calling in
update_persistent_clock(). So .. ntpdate or ntpd don't call that?

Somebody does.. I hope?

Re: [PATCH 1/3] xen: sync the CMOS RTC as well as the Xen wallclock

[David Vrabel]

On 12/10/12 15:57, Konrad Rzeszutek Wilk wrote:
> On Fri, Oct 12, 2012 at 01:57:12PM +0100, David Vrabel wrote:
>> From: David Vrabel <david.vrabel@citrix.com>
>>
>> If NTP is used in dom0 and it is synchronized to its clock source,
>> then the kernel will periodically synchronize the Xen wallclock with
>> the system time.  Updates to the Xen wallclock do not persist across
>> reboots, so also synchronize the CMOS RTC (as on bare metal).
>>
>> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
>> ---
>>  arch/x86/xen/time.c |    5 +++++
>>  1 files changed, 5 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
>> index 0296a95..5e7f536 100644
>> --- a/arch/x86/xen/time.c
>> +++ b/arch/x86/xen/time.c
>> @@ -14,6 +14,7 @@
>>  #include <linux/kernel_stat.h>
>>  #include <linux/math64.h>
>>  #include <linux/gfp.h>
>> +#include <linux/mc146818rtc.h>
>>  
>>  #include <asm/pvclock.h>
>>  #include <asm/xen/hypervisor.h>
>> @@ -208,6 +209,10 @@ static int xen_set_wallclock(unsigned long now)
>>  	if (!xen_initial_domain())
>>  		return -1;
>>  
>> +	/* Set the hardware RTC. */
>> +	mach_set_rtc_mmss(now);
> 
> So how does this work? Is the hypervisor traping on the RTC CMOS clock?

It's works the same as with the RTC driver.  I think the hypervisor
allows dom0 access to the relevant I/O ports.

It's worth pointing out that mach_set_rtc_mmss() only sets the minutes
and seconds so it doesn't handle big step changes which is why
update_persistent_clock() is only called once NTP is synchronized.

David

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[David Vrabel]

On 12/10/12 16:02, Konrad Rzeszutek Wilk wrote:
> On Fri, Oct 12, 2012 at 03:59:25PM +0100, David Vrabel wrote:
>> On 12/10/12 15:29, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Oct 12, 2012 at 10:02 AM, David Vrabel <david.vrabel@citrix.com> wrote:
>>>> On 12/10/12 14:41, Konrad Rzeszutek Wilk wrote:
>>>>> On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
>>>>>> From: David Vrabel <david.vrabel@citrix.com>
>>>>>>
>>>>>> Add a new ioctl to synchronize Xen's wallclock with the current system
>>>>>> time.
>>>>>>
>>>>>> This may be used by the tools to ensure that newly created domains see
>>>>>> the correct wallclock time if NTP is not used in dom0 or if domains
>>>>>> are started before NTP has synchronized.
>>>>>
>>>>> So... how does this work with NTPD? As in does ntpd _not_ update the
>>>>> hwclock enough?
>>>>
>>>> Once NTPD is synchronized then the kernel updates the wallclock (and the
>>>> RTC with patch #1) every 11 mins.  I assume this is often enough given
>>>> how NTP adjusts the system time.
>>>>
>>>> You only really need the tools to sync wallclock if system time was
>>>> stepped at start of day.  e.g., init scripts could do something like:
>>>>
>>>> ntpdate pool.ntp.org
>>>> hwclock --systohc
>>>> xen-wallclock --systowc
>>>
>>> I think I am missing something. The hwclock should end up in the
>>> xen_set_wallclock call. And from there on, the ntpd would update the
>>> wallclock if it got skewed enough? Or is the system time not calling
>>> the wall-clock enough? If that is the case, would just adding this in
>>> the crontab be enough:
>>
>> hwclock talks to /dev/rtc which writes to the CMOS directly and does not
>> call update_persistent_clock() (or xen_set_wallclock()).
> 
> /me scratches his head.
> 
> I recall that the update_persistent_clock() was being called.. with
> hwclock -w (which is the same as --systohc).

No,

> That seems like a bug in the generic code - I would think that
> hwclock would update the wallclock, not just the RTC.

If we wanted hwclock to also update the xen wallclock we would need a
xen-specific rtc driver that updated both rtc and wallclock.

> Oh, it is whoever calls 'adjtimex' syscall ends up calling in
> update_persistent_clock(). So .. ntpdate or ntpd don't call that?

ntpd does call adjtimex so if you are running ntpd and it is
synchronized to its clock source you do get the periodic sync of the
wallclock.

David

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[Konrad Rzeszutek Wilk]

On Fri, Oct 12, 2012 at 05:14:16PM +0100, David Vrabel wrote:
> On 12/10/12 16:02, Konrad Rzeszutek Wilk wrote:
> > On Fri, Oct 12, 2012 at 03:59:25PM +0100, David Vrabel wrote:
> >> On 12/10/12 15:29, Konrad Rzeszutek Wilk wrote:
> >>> On Fri, Oct 12, 2012 at 10:02 AM, David Vrabel <david.vrabel@citrix.com> wrote:
> >>>> On 12/10/12 14:41, Konrad Rzeszutek Wilk wrote:
> >>>>> On Fri, Oct 12, 2012 at 01:57:14PM +0100, David Vrabel wrote:
> >>>>>> From: David Vrabel <david.vrabel@citrix.com>
> >>>>>>
> >>>>>> Add a new ioctl to synchronize Xen's wallclock with the current system
> >>>>>> time.
> >>>>>>
> >>>>>> This may be used by the tools to ensure that newly created domains see
> >>>>>> the correct wallclock time if NTP is not used in dom0 or if domains
> >>>>>> are started before NTP has synchronized.
> >>>>>
> >>>>> So... how does this work with NTPD? As in does ntpd _not_ update the
> >>>>> hwclock enough?
> >>>>
> >>>> Once NTPD is synchronized then the kernel updates the wallclock (and the
> >>>> RTC with patch #1) every 11 mins.  I assume this is often enough given
> >>>> how NTP adjusts the system time.
> >>>>
> >>>> You only really need the tools to sync wallclock if system time was
> >>>> stepped at start of day.  e.g., init scripts could do something like:
> >>>>
> >>>> ntpdate pool.ntp.org
> >>>> hwclock --systohc
> >>>> xen-wallclock --systowc
> >>>
> >>> I think I am missing something. The hwclock should end up in the
> >>> xen_set_wallclock call. And from there on, the ntpd would update the
> >>> wallclock if it got skewed enough? Or is the system time not calling
> >>> the wall-clock enough? If that is the case, would just adding this in
> >>> the crontab be enough:
> >>
> >> hwclock talks to /dev/rtc which writes to the CMOS directly and does not
> >> call update_persistent_clock() (or xen_set_wallclock()).
> > 
> > /me scratches his head.
> > 
> > I recall that the update_persistent_clock() was being called.. with
> > hwclock -w (which is the same as --systohc).
> 
> No,
> 
> > That seems like a bug in the generic code - I would think that
> > hwclock would update the wallclock, not just the RTC.
> 
> If we wanted hwclock to also update the xen wallclock we would need a
> xen-specific rtc driver that updated both rtc and wallclock.
> 
> > Oh, it is whoever calls 'adjtimex' syscall ends up calling in
> > update_persistent_clock(). So .. ntpdate or ntpd don't call that?
> 
> ntpd does call adjtimex so if you are running ntpd and it is
> synchronized to its clock source you do get the periodic sync of the
> wallclock.

So, having both ntpd (which calls call adjtimex) - which would run in the
background; and hwclock run at startup - (which updates the RTC, which ends
 - I hope - being trapped by the hypervisor), we end up with the correct
time, right?

Re: [PATCH 0/3] xen: fix various wallclock problems

[David Vrabel]

On 12/10/12 13:57, David Vrabel wrote:
> This series (with some toolstack updates) corrects a number of cases
> where a guest can see an incorrect wallclock time.
> 
> 1. Systems with NTP won't periodically synchronize the hardware RTC so
> the wallclock may be incorrect after a reboot.
> 
> 2. The wallclock is always ~500 ms behind the correct time.
> 
> 3. If the system time was stepped and NTP isn't synchronized yet, the
> wallclock will still have the incorrect time.  The fix for this
> requires the toolstack to synchronize the wallclock -- patch #3
> provides the mechanism for this.

These tables should help.

Before:

              |                Updates
Process	      | System Time?  Xen Wallclock?  Hardware Clock?
-------------------------------------------------------------
date -s       | X
hwclock -w    |                               X
ntpd          | X             X[*]
ntpdate       | X

After this (and the toolstack) patches:

              |                Updates
Process	      | System Time?  Xen Wallclock?  Hardware Clock?
-------------------------------------------------------------
date -s       | X
hwclock -w    |                               X
ntpd          | X             X[*]            X[*]
ntpdate       | X
xen-wallclock |               X

[*] every 11 minutes.

David

Re: [PATCH 3/3] xen/privcmd: add IOCTL_PRIVCMD_SYNC_WALLCLOCK to sync Xen's wallclock

[David Vrabel]

On 12/10/12 17:16, Konrad Rzeszutek Wilk wrote:
> 
> So, having both ntpd (which calls call adjtimex) - which would run in the
> background; and hwclock run at startup - (which updates the RTC, which ends
>  - I hope - being trapped by the hypervisor), we end up with the correct
> time, right?

Yes, unless you start a guest before ntpd has synchronized.

David

Re: [PATCH 2/3] xen: add correct 500 ms offset when setting Xen wallclock

[Ian Campbell]

On Fri, 2012-10-12 at 13:57 +0100, David Vrabel wrote:
> From: David Vrabel <david.vrabel@citrix.com>
> 
> update_persistent_wallclock() (and hence xet_set_wallclock()) is
> called 500 ms after the second.

The comment in sync_cmos_clock says it is called 500ms before the next
second.

I only mention it to double check that the right semantics for
set_wallclock are being implemented, i.e. that we are compensating in
the right direction.

>   xen_set_wallclock() was not
> considering this so the Xen wallclock would end up ~500 ms behind the
> correct time.
> 
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> ---
>  arch/x86/xen/time.c |    9 +++++++--
>  1 files changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> index 5e7f536..11770d0 100644
> --- a/arch/x86/xen/time.c
> +++ b/arch/x86/xen/time.c
> @@ -212,10 +212,15 @@ static int xen_set_wallclock(unsigned long now)
>  	/* Set the hardware RTC. */
>  	mach_set_rtc_mmss(now);
>  
> -	/* Set the Xen wallclock. */
> +	/*
> +	 * Set the Xen wallclock.
> +	 *
> +	 * update_persistent_wallclock() is called ~500 ms after 'now'
> +	 * so add an extra 500 ms.
> +	 */
>  	op.cmd = XENPF_settime;
>  	op.u.settime.secs = now;
> -	op.u.settime.nsecs = 0;
> +	op.u.settime.nsecs = NSEC_PER_SEC / 2;
>  	op.u.settime.system_time = xen_clocksource_read();
>  
>  	rc = HYPERVISOR_dom0_op(&op);

Re: [PATCH 0/3] xen: fix various wallclock problems

[Ian Campbell]

On Fri, 2012-10-12 at 17:25 +0100, David Vrabel wrote:
> On 12/10/12 13:57, David Vrabel wrote:
> > This series (with some toolstack updates) corrects a number of cases
> > where a guest can see an incorrect wallclock time.
> > 
> > 1. Systems with NTP won't periodically synchronize the hardware RTC so
> > the wallclock may be incorrect after a reboot.
> > 
> > 2. The wallclock is always ~500 ms behind the correct time.
> > 
> > 3. If the system time was stepped and NTP isn't synchronized yet, the
> > wallclock will still have the incorrect time.  The fix for this
> > requires the toolstack to synchronize the wallclock -- patch #3
> > provides the mechanism for this.
> 
> These tables should help.
> 
> Before:
> 
>               |                Updates
> Process	      | System Time?  Xen Wallclock?  Hardware Clock?
> -------------------------------------------------------------
> date -s       | X
> hwclock -w    |                               X
> ntpd          | X             X[*]
> ntpdate       | X

What does the equivalent table for native look like?

Is the "updated hardware clock" column in that caseis union of xen
wallclock and hardware clock columns here?

Ian.

Re: [PATCH 2/3] xen: add correct 500 ms offset when setting Xen wallclock

[David Vrabel]

On 15/10/12 10:25, Ian Campbell wrote:
> On Fri, 2012-10-12 at 13:57 +0100, David Vrabel wrote:
>> From: David Vrabel <david.vrabel@citrix.com>
>>
>> update_persistent_wallclock() (and hence xet_set_wallclock()) is
>> called 500 ms after the second.
> 
> The comment in sync_cmos_clock says it is called 500ms before the next
> second.

This is the same thing, right?

> I only mention it to double check that the right semantics for
> set_wallclock are being implemented, i.e. that we are compensating in
> the right direction.

I'll add some debug code to make sure.

David

Re: [PATCH 0/3] xen: fix various wallclock problems

[David Vrabel]

On 15/10/12 10:26, Ian Campbell wrote:
> On Fri, 2012-10-12 at 17:25 +0100, David Vrabel wrote:
>> On 12/10/12 13:57, David Vrabel wrote:
>>> This series (with some toolstack updates) corrects a number of cases
>>> where a guest can see an incorrect wallclock time.
>>>
>>> 1. Systems with NTP won't periodically synchronize the hardware RTC so
>>> the wallclock may be incorrect after a reboot.
>>>
>>> 2. The wallclock is always ~500 ms behind the correct time.
>>>
>>> 3. If the system time was stepped and NTP isn't synchronized yet, the
>>> wallclock will still have the incorrect time.  The fix for this
>>> requires the toolstack to synchronize the wallclock -- patch #3
>>> provides the mechanism for this.
>>
>> These tables should help.
>>
>> Before:
>>
>>               |                Updates
>> Process	      | System Time?  Xen Wallclock?  Hardware Clock?
>> -------------------------------------------------------------
>> date -s       | X
>> hwclock -w    |                               X
>> ntpd          | X             X[*]
>> ntpdate       | X
> 
> What does the equivalent table for native look like?
> 
> Is the "updated hardware clock" column in that caseis union of xen
> wallclock and hardware clock columns here?

Yes.

              |         Updates
Process	      | System Time?  Hardware Clock?
---------------------------------------------
date -s       | X
hwclock -w    |               X
ntpd          | X             X[*]
ntpdate       | X

David

Re: [PATCH 2/3] xen: add correct 500 ms offset when setting Xen wallclock

[Ian Campbell]

On Mon, 2012-10-15 at 13:21 +0100, David Vrabel wrote:
> On 15/10/12 10:25, Ian Campbell wrote:
> > On Fri, 2012-10-12 at 13:57 +0100, David Vrabel wrote:
> >> From: David Vrabel <david.vrabel@citrix.com>
> >>
> >> update_persistent_wallclock() (and hence xet_set_wallclock()) is
> >> called 500 ms after the second.
> > 
> > The comment in sync_cmos_clock says it is called 500ms before the next
> > second.
> 
> This is the same thing, right?

It matters in the native case, where, If I'm reading it right, things
are setup such thatthe time change happens on the next second tick over.
I just wanted to check it did/didn't matter here as well.

> 
> > I only mention it to double check that the right semantics for
> > set_wallclock are being implemented, i.e. that we are compensating in
> > the right direction.
> 
> I'll add some debug code to make sure.
> 
> David