Clock jumping during live migrations

Clock jumping during live migrations

[Nathan March]

Hi everyone,

Having an issue where the clock on a VM will jump ahead by the timezone 
offset but only certain directions between certain machines.

For example, if I run a "sleep 1; date" loop on the VM and migrate it 
from xen4 to xen7:

Wed Oct 26 21:14:46 PDT 2011
Thu Oct 27 04:19:03 PDT 2011

Trying multiple times with the same dom0's it seems to be consistently 7 
hours (PST offset maybe?) 4 minutes  and 17 seconds:

Thu Oct 27 04:23:27 PDT 2011
Thu Oct 27 11:27:45 PDT 2011

Thu Oct 27 11:43:01 PDT 2011
Thu Oct 27 18:47:18 PDT 2011

If I keep migrating back and forth it keeps going ahead each time, it 
never snaps back. Those were both migrating from xen4 -> xen7, migrating 
the VM back to xen4 does not result in a clock change of any noticeable 
amount.

I've tried using hwclock to make sure the RTC's are in sync with the 
system clock but it didn't seem to make a difference (although they were 
wrong initially, unless xen requires a reboot to pick up a change here).

If I try from xen8 to xen7 it does it, but by a much smaller amount... 
again, going the reverse direction causes no change.

Thu Oct 27 18:50:08 PDT 2011
Thu Oct 27 18:51:57 PDT 2011

and again the time between multiple runs seems to be a consistent 1:50:

Fri Oct 28 16:16:27 PDT 2011
Fri Oct 28 16:18:17 PDT 2011

All the dom0's are identical as far as I'm aware except that some have 
hyperthreading on and some have it off. I've seen this issue between two 
hyperthreading enabled hosts so doesn't seem to be specific to moving 
between them.

I've tested this with the VM using both the xen and the tsc clocksource 
but it made no difference.

Other info for xen4 / xen7:

xen4 ~ # uname -a
Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R) 
Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen4 ~ # uptime
  14:48:33 up 1 day,  6:03,  2 users,  load average: 0.00, 0.01, 0.05
xen4 ~ # xm info
host                   : xen4
release                : 3.0.3
version                : #3 SMP Thu Sep 1 23:39:43 PDT 2011
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 87822
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Thu Sep  1 23:34:06 PDT 2011
xend_config_format     : 4
xen4 ~ # cat /etc/timezone
PST8PDT
xen4 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

xen7 ~ # uname -a
Linux xen7 3.0.3 #2 SMP Tue Aug 23 12:58:39 PDT 2011 x86_64 Intel(R) 
Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen7 ~ # uptime
  14:48:34 up 42 days, 23:38,  3 users,  load average: 0.04, 0.08, 0.13
xen7 ~ # xm info
host                   : xen7
release                : 3.0.3
version                : #2 SMP Tue Aug 23 12:58:39 PDT 2011
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 24329
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Mon Aug 22 11:28:50 PDT 2011
xend_config_format     : 4
xen7 ~ # cat /etc/timezone
PST8PDT
xen7 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

and the test VM i've been using:

nathanxen1 ~ # uname -a
Linux nathanxen1 2.6.32.27-grsec-gt-r2 #1 SMP Wed Aug 10 17:13:56 PDT 
2011 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
nathanxen1 ~ # cat 
/sys/devices/system/clocksource/clocksource0/available_clocksource
xen tsc
nathanxen1 ~ # cat 
/sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Any suggestions?

Unfortunately I don't know when this bug appeared, it used to work fine 
so not sure what I upgraded to give me an idea of when it was introduced.

Thanks,
Nathan

-- 
Nathan March<nathan@gt.net>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806

Re: Clock jumping during live migrations

[Tim Deegan]

Hi, 

At 15:31 -0700 on 26 Oct (1319643098), Nathan March wrote:
> Having an issue where the clock on a VM will jump ahead by the timezone 
> offset but only certain directions between certain machines.
> 
> For example, if I run a "sleep 1; date" loop on the VM and migrate it 
> from xen4 to xen7:
> 
> Wed Oct 26 21:14:46 PDT 2011
> Thu Oct 27 04:19:03 PDT 2011

...

> xen4 ~ # uname -a
> Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R) 

I think maybe you need these patches on your dom0 kernel:

http://lists.xensource.com/archives/html/xen-devel/2011-10/msg01762.html

Tim.

Re: Clock jumping during live migrations

[Nathan March]

On 10/27/2011 4:07 AM, Tim Deegan wrote:
> At 15:31 -0700 on 26 Oct (1319643098), Nathan March wrote:
>> Having an issue where the clock on a VM will jump ahead by the timezone
>> offset but only certain directions between certain machines.
>>
>> For example, if I run a "sleep 1; date" loop on the VM and migrate it
>> from xen4 to xen7:
>>
>> Wed Oct 26 21:14:46 PDT 2011
>> Thu Oct 27 04:19:03 PDT 2011
>> xen4 ~ # uname -a
>> Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R)
> I think maybe you need these patches on your dom0 kernel:
>
> http://lists.xensource.com/archives/html/xen-devel/2011-10/msg01762.html

Unfortunately I'm still running into this problem, migrating VM's 
between particular dom0's will result in a jump forward of anywhere from 
minutes to hours. This has persisted through dom0 and xen upgrades (now 
on dom0 3.2.7 / xen 4.1.2) so maybe we're missing something in the domu 
kernel (2.6.32.27)? Reviewing kernel change logs doesn't indicate 
anything's changed in regard to clocksources though.

Examples (just running date on the domU after each migration)

Initially: Mon Jun 18 13:53:48 PDT 2012
Xen2 -> Xen8: Mon Jun 18 13:55:05 PDT 2012 (No shift)
Xen8 -> Xen2: Mon Jun 18 14:27:31 PDT 2012 (+30 approx minutes)
Xen2 -> Xen8: Mon Jun 18 14:28:38 PDT 2012 (No shift)
Xen8 -> Xen2: Mon Jun 18 15:01:21 PDT 2012 (+30 minutes again)

This is now running with a domu kernel of 2.6.32.27 (w/ grsec) and a 
dom0 kernel of 3.2.7 on both dom0's (they should be identical). I've 
tried both the "tsc" and "xen" clocksources in the domU with no effect 
(no others are listed in available_clocksources).

Not sure if this is useful, but while on xen8:
(XEN) dom27: mode=0,ofs=0xffc56b8eee259ecf,khz=2266790,inc=13,vtsc 
count: 37835 kernel, 750 user

After moving to xen2:
(XEN) dom7: mode=0,ofs=0xffc51d4984ee8f42,khz=2266790,inc=14,vtsc count: 
17628 kernel, 71 user

Anyone have any suggestions here?

Thanks,
Nathan

-- 
Nathan March<nathan@gt.net>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806