Re: question about migration

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 25/12/2015 00:55, Wen Congyang wrote:
> On 12/24/2015 08:36 PM, Andrew Cooper wrote:
>> On 24/12/15 02:29, Wen Congyang wrote:
>>> Hi Andrew Cooper:
>>>
>>> I rebase the COLO codes to the newest upstream xen, and test it. I found
>>> a problem in the test, and I can reproduce this problem via the migration.
>>>
>>> How to reproduce:
>>> 1. xl cr -p hvm_nopv
>>> 2. xl migrate hvm_nopv 192.168.3.1
>> You are the very first person to try a usecase like this.
>>
>> It works as much as it does because of your changes to the uncooperative HVM domain logic.  I have said repeatedly during review, this is not necessarily a safe change to make without an in-depth analysis of the knock-on effects; it looks as if you have found the first knock-on effect.
>>
>>> The migration successes, but the vm doesn't run in the target machine.
>>> You can get the reason from 'xl dmesg':
>>> (XEN) HVM2 restore: VMCE_VCPU 1
>>> (XEN) HVM2 restore: TSC_ADJUST 0
>>> (XEN) HVM2 restore: TSC_ADJUST 1
>>> (d2) HVM Loader
>>> (d2) Detected Xen v4.7-unstable
>>> (d2) Get guest memory maps[128] failed. (-38)
>>> (d2) *** HVMLoader bug at e820.c:39
>>> (d2) *** HVMLoader crashed.
>>>
>>> The reason is that:
>>> We don't call xc_domain_set_memory_map() in the target machine.
>>> When we create a hvm domain:
>>> libxl__domain_build()
>>>       libxl__build_hvm()
>>>           libxl__arch_domain_construct_memmap()
>>>               xc_domain_set_memory_map()
>>>
>>> Should we migrate the guest memory from source machine to target machine?
>> This bug specifically is because HVMLoader is expected to have run and turned the hypercall information in an E820 table in the guest before a migration occurs.
>>
>> Unfortunately, the current codebase is riddled with such assumption and expectations (e.g. the HVM save code assumed that FPU context is valid when it is saving register state) which is a direct side effect of how it was developed.
> Does FPU context have the similar problem?

Yes, although it is far harder to spot, and no software will likely 
crash as a result.

> IIRC, I have tested colo befroe 4.6 is released. It works. In my test, I always
> use the option '-p' to start the HVM guest.

If the FPU wasn't initialised, the save code memset()'s the x87 register 
block to 0.  On the restore side, this is taken an loaded back.

The problem is that a block of zeroes is valid for the x87, and not the 
default which the vcpu would expect to observe, given no resetting 
itself.  However, the first thing any real software will do is reset the 
values properly.

>
>>
>> Having said all of the above, I agree that your example is a usecase which should work.  It is the ultimate test of whether the migration stream contains enough information to faithfully reproduce the domain on the far side.  Clearly at the moment, this is not the case.
> I think it should work. But the user doesn't use the migration like this.
> So it is not a serious problem.

It is still worth identifying as an issue.

~Andrew