Results from the Xen 4.4-rc2 test day

Results from the Xen 4.4-rc2 test day

[Andrew Cooper]

Hello,

I participated in (a rather extended version of) the 4.4-rc2 test day,
and rc2 got a full XenRT nightly run in the XenServer testing system.

For the setup, the comparison is against XenServer trunk, which is
currently Xen-4.3-staging based (plus patch queue), Linux 3.10.y dom0
kernel, CentOS 6.4 based dom0 userspace.

The tested version had Xen 4.4 (staging, as I needed the ABI fix) in
place of Xen-4.3, but identical dom0 kernel, dom0 userspace, qemu,
toolstack and windows PV drivers.


The major issue identified is with Windows 8/8.1 and Server 2012/2012r2,
which have problems on live migrate.  Some source of time is
unexpectedly jumping forwards by two days, from the correct time to 2
days in the future.  The observed result is that it looses its DHCP
lease, drops its IP address and networking ceases to work (It appears
that windows will not attempt to renew the lease itself).

I am currently investigating which source of time is jumping forwards,
but this does appear to be a regression directly attributable to Xen 4.4.

~Andrew

Re: Results from the Xen 4.4-rc2 test day

[Andrew Cooper]

On 21/01/14 10:28, Andrew Cooper wrote:
> Hello,
>
> I participated in (a rather extended version of) the 4.4-rc2 test day,
> and rc2 got a full XenRT nightly run in the XenServer testing system.
>
> For the setup, the comparison is against XenServer trunk, which is
> currently Xen-4.3-staging based (plus patch queue), Linux 3.10.y dom0
> kernel, CentOS 6.4 based dom0 userspace.
>
> The tested version had Xen 4.4 (staging, as I needed the ABI fix) in
> place of Xen-4.3, but identical dom0 kernel, dom0 userspace, qemu,
> toolstack and windows PV drivers.
>
>
> The major issue identified is with Windows 8/8.1 and Server 2012/2012r2,
> which have problems on live migrate.  Some source of time is
> unexpectedly jumping forwards by two days, from the correct time to 2
> days in the future.  The observed result is that it looses its DHCP
> lease, drops its IP address and networking ceases to work (It appears
> that windows will not attempt to renew the lease itself).
>

This is caused by commit e36cd2cdc9674a7a4855d21fb7b3e6e17c4bb33b
"x86/viridian: Time Reference Count MSR"

After double checking with the specification, it does appear to be
implemented as required (subject to a potential issue with multiple vcpu
guests).

I am currently experimenting to see whether hvm_get_guest_time() is
returning unexpected values, or whether it is returning expected values
and Windows is interpreting them differently.

At this point in the 4.4 release cycle, reverting the patch should be
seriously considered, although I would like to see whether it is
possible to work out why it is wrong and whether there is an obvious fix
first.

~Andrew

Re: Results from the Xen 4.4-rc2 test day

[Jan Beulich]

>>> On 23.01.14 at 12:54, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 21/01/14 10:28, Andrew Cooper wrote:
>> The major issue identified is with Windows 8/8.1 and Server 2012/2012r2,
>> which have problems on live migrate.  Some source of time is
>> unexpectedly jumping forwards by two days, from the correct time to 2
>> days in the future.  The observed result is that it looses its DHCP
>> lease, drops its IP address and networking ceases to work (It appears
>> that windows will not attempt to renew the lease itself).
>>
> 
> This is caused by commit e36cd2cdc9674a7a4855d21fb7b3e6e17c4bb33b
> "x86/viridian: Time Reference Count MSR"
> 
> After double checking with the specification, it does appear to be
> implemented as required (subject to a potential issue with multiple vcpu
> guests).
> 
> I am currently experimenting to see whether hvm_get_guest_time() is
> returning unexpected values, or whether it is returning expected values
> and Windows is interpreting them differently.
> 
> At this point in the 4.4 release cycle, reverting the patch should be
> seriously considered, although I would like to see whether it is
> possible to work out why it is wrong and whether there is an obvious fix
> first.

I suppose you and/or Paul will let us know either way.

Jan

Re: Results from the Xen 4.4-rc2 test day

[Andrew Cooper]

On 23/01/14 12:53, Jan Beulich wrote:
>>>> On 23.01.14 at 12:54, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 21/01/14 10:28, Andrew Cooper wrote:
>>> The major issue identified is with Windows 8/8.1 and Server 2012/2012r2,
>>> which have problems on live migrate.  Some source of time is
>>> unexpectedly jumping forwards by two days, from the correct time to 2
>>> days in the future.  The observed result is that it looses its DHCP
>>> lease, drops its IP address and networking ceases to work (It appears
>>> that windows will not attempt to renew the lease itself).
>>>
>> This is caused by commit e36cd2cdc9674a7a4855d21fb7b3e6e17c4bb33b
>> "x86/viridian: Time Reference Count MSR"
>>
>> After double checking with the specification, it does appear to be
>> implemented as required (subject to a potential issue with multiple vcpu
>> guests).
>>
>> I am currently experimenting to see whether hvm_get_guest_time() is
>> returning unexpected values, or whether it is returning expected values
>> and Windows is interpreting them differently.
>>
>> At this point in the 4.4 release cycle, reverting the patch should be
>> seriously considered, although I would like to see whether it is
>> possible to work out why it is wrong and whether there is an obvious fix
>> first.
> I suppose you and/or Paul will let us know either way.
>
> Jan
>

The value of time read from hvm_get_guest_time() resets with a new
domid, making it an inappropriate source of time for the described
function of the MSR.

I suspect Windows 8 only notices at first on migration as I believe that
it is the first case where the generation ID is supposed to change and
signal a reset of state.  The detection of the failure is actually
further complicated as there appears to be a race condition between the
guest tools reseting the clock back to the correct value, and the DHCP
lease being flushed.  XenRT only notices the failure if the DHCP lease
is actually lost (thus XenRT can't communicate with it's xmlrpc daemon
inside the VM), and doesn't directly notice the foward/backward stepping
in time.

Anyway - please revert the patch - it will be a non-trivial change to
expose an appropriate source of time to be consumed by this MSR.

~Andrew

Re: Results from the Xen 4.4-rc2 test day

[Jan Beulich]

>>> On 23.01.14 at 18:17, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> The value of time read from hvm_get_guest_time() resets with a new
> domid, making it an inappropriate source of time for the described
> function of the MSR.
> 
> I suspect Windows 8 only notices at first on migration as I believe that
> it is the first case where the generation ID is supposed to change and
> signal a reset of state.  The detection of the failure is actually
> further complicated as there appears to be a race condition between the
> guest tools reseting the clock back to the correct value, and the DHCP
> lease being flushed.  XenRT only notices the failure if the DHCP lease
> is actually lost (thus XenRT can't communicate with it's xmlrpc daemon
> inside the VM), and doesn't directly notice the foward/backward stepping
> in time.
> 
> Anyway - please revert the patch - it will be a non-trivial change to
> expose an appropriate source of time to be consumed by this MSR.

Done, albeit not completely - I left the #define-s in place.

Jan

Re: Results from the Xen 4.4-rc2 test day

[Andrew Cooper]

On 24/01/14 09:23, Jan Beulich wrote:
>>>> On 23.01.14 at 18:17, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> The value of time read from hvm_get_guest_time() resets with a new
>> domid, making it an inappropriate source of time for the described
>> function of the MSR.
>>
>> I suspect Windows 8 only notices at first on migration as I believe that
>> it is the first case where the generation ID is supposed to change and
>> signal a reset of state.  The detection of the failure is actually
>> further complicated as there appears to be a race condition between the
>> guest tools reseting the clock back to the correct value, and the DHCP
>> lease being flushed.  XenRT only notices the failure if the DHCP lease
>> is actually lost (thus XenRT can't communicate with it's xmlrpc daemon
>> inside the VM), and doesn't directly notice the foward/backward stepping
>> in time.
>>
>> Anyway - please revert the patch - it will be a non-trivial change to
>> expose an appropriate source of time to be consumed by this MSR.
> Done, albeit not completely - I left the #define-s in place.
>
> Jan
>

Thanks - I have pulled XenServer's 4.4-rc2 branch forward to current
staging, and the w2k12 vmlifecycle tests are now working without error.

I shall organise another full nightly regression test for some time in
the next few days.

~Andrew

Re: Results from the Xen 4.4-rc2 test day

[George Dunlap]

On 01/24/2014 11:37 AM, Andrew Cooper wrote:
> On 24/01/14 09:23, Jan Beulich wrote:
>>>>> On 23.01.14 at 18:17, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>> The value of time read from hvm_get_guest_time() resets with a new
>>> domid, making it an inappropriate source of time for the described
>>> function of the MSR.
>>>
>>> I suspect Windows 8 only notices at first on migration as I believe that
>>> it is the first case where the generation ID is supposed to change and
>>> signal a reset of state.  The detection of the failure is actually
>>> further complicated as there appears to be a race condition between the
>>> guest tools reseting the clock back to the correct value, and the DHCP
>>> lease being flushed.  XenRT only notices the failure if the DHCP lease
>>> is actually lost (thus XenRT can't communicate with it's xmlrpc daemon
>>> inside the VM), and doesn't directly notice the foward/backward stepping
>>> in time.
>>>
>>> Anyway - please revert the patch - it will be a non-trivial change to
>>> expose an appropriate source of time to be consumed by this MSR.
>> Done, albeit not completely - I left the #define-s in place.
>>
>> Jan
>>
> Thanks - I have pulled XenServer's 4.4-rc2 branch forward to current
> staging, and the w2k12 vmlifecycle tests are now working without error.
>
> I shall organise another full nightly regression test for some time in
> the next few days.

Overall, excellent news.  Thanks.

  -George