Re: Wall-Clock-Time-Jump after migration [xen-4.1] [SOLVED]

[Philipp Hahn <hahn@univention.de>]

Hello,

to answer my own questions:

Am 16.02.2016 um 16:38 schrieb Philipp Hahn:
> Summary: When a Linux-PV-domU is migrated between two hosts, the
> "ntpdate" time jumps.
...
> 1. If I start a new domU (just kernel and InitRamFS with busybox as to
> minimize the processes involved), the wall-clock-time if off by ~283
> (=4m43s) seconds:
> 
> dom0: Tue Deb 16 14:48:12 UTC 2016 (before starting domU)
> domU: Tue Feb 16 14:43:32 UTC 2016 (after boot, which takes ~3s)
> 
> If I then run "ntpdate" that domU, the offset is corrected:
>> 16 Feb 14:51:59 ntpdate[150]: step time server X.X.73.241 offset 283.697217 sec
> 
> Q: Where does the initial time for the domU come from?

Wall-clock (WC) from Xen Hypervisor.

> Q: Where does that offset of ~283s come from?

The Hypervisor reads the HW-RTC once on boot to initialize its WC.
When the dom0 runs ntpdate, only "time_offset_seconds" get updated, so
dom0-WC is correct, but any new domU starts with "time_offset_seconds=0"
again and is off by that initial difference again.
You could reboot the Hypervisor, so it could read the corrected RTC
after calling 'hwclock --systohc'.

> Q: Is the shard_info.wc_sec supposed to be updated?

No: "wc_set" is more-or-less constant.
The current time is calculated by adding "wc_set + tsc_to_sec()",  so
"wc_set" is the offset calculated from "wc_time@boot" - tsc@boot()".

'wc_sec' can be updated through "XENPF_settime".

> 3. If I migrate the domain from the first host to the second while
> running "ntpdate" in a loop, I see the clock jumping ~257s, which
> matches the difference between the time_offset_seconds between the hosts
> (283 - 22):
...
> To me that looks like "time_offset_seconds" is migrated, but as "wc_sec"
> between the two hosts is not synchronized, the time jumps.
> 
> Q: Is that a known problem and has it been fixed in newer versions of Xen?

The Linux kernel has a hook to call XENPF_settime from its
"update-RTC-every-11-minutes" (sync_cmos_work) work queue, but the code
path is disabled when the time is *not* NTP-synchronized:

> kernel/time/ntp.c:454
> »···static void sync_cmos_clock(struct work_struct *work)
> »···»···if (!ntp_synced()) {

> Q: Is there some recommended procedure to synchronize the time of
> multiple hypervisors, like perhaps:

Run "ntpd" in dom0 instead of "ntpdate":
Then ntp_synced() returns True, x86_platform.set_wallclock() gets called
to update the Hypervisor wall-time clock using XENPF_settime{32,64},
then every new domU will get a reasonably correct wall-time-clock on
domain creation, ...

You can check the status of "time_status" calling "ntptime":
> # ntptime 
...
> ntp_adjtime() returns code 5 (ERROR)
...
>   status 0x2041 (PLL,UNSYNC,NANO),

It's a bit-filed; if UNSYNC=0x40 is set, Xens WC will not get updated.


@David: Thank you for looking into that and sending the list of commits.
It helped me to get a better understanding of the interaction between
the Linux dom0 and Xen Hypervisor.


Thank you for the adventure; I hope that info helps others to get it
right from the beginning.


Philipp Hahn

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel