Re: xen/stable-2.6.32.x xen-4.1.1 live migration fails with kernels 2.6.39, 3.0.3 and 3.1-rc2.. between different physical machines and CPUs.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> This does not neccessarily have sth to todo with the amount of memory.
> I do see this on hosts where both have the same amount of ram but
> are a different hardware platform.

<nods> Let me modify the subject a bit to reflect this.

> >I think the problem you are running into is that you are migrating between
> >different CPU families... Is the /proc/cpuinfo drastically different between
> >the boxes?
> diff:
> < model		: 26
> < model name	: Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
> < stepping	: 5
> < cpu MHz		: 2261.074
> < cache size	: 8192 KB
> ---
> > model		: 44
> > model name	: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz
> > stepping	: 2
> > cpu MHz		: 2660.050
> > cache size	: 12288 KB
> 13,14c13,14
> < flags		: fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat
> clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc
> rep_good nonstop_tsc aperfmperf pni est ssse3 cx16 sse4_1 sse4_2
> popcnt hypervisor lahf_lm ida
> < bogomips	: 4522.14
> ---
> > flags		: fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat
> clflush acpi mmx fxsr sse sse2 ss ht syscall lm constant_tsc
> rep_good nonstop_tsc aperfmperf pni pclmulqdq est ssse3 cx16 sse4_1
> sse4_2 popcnt aes hypervisor lahf_lm ida arat
> > bogomips	: 5320.10
> 
> diffrent flags are: nx and aes

On the Linux command line, try using 'noexec=off' - that should
take care of the 'nx' bit.

The aes.. the 'xl' command has a bit easier syntax for setting the CPUID:

cpuid='host,family=15,model=26,stepping=5,aes=s'

That ought to take care of that. I don't really understand how
the old 'cpuid=['...']' syntax worked (the one that 'xm' used).
It looks quite arcane - so I think doing some Google search is the
only way to figure that out.

But co-workers of mine remind me that CPUID instructions is
trapped by the hypervisor (both HVM and PV - PV via a special
opcode - look in arch/x86/include/asm/xen/interface.h for details) for
the kernel _only_. There is no such guarantee for applications. Meaning that
if the application uses the 'cpuid' to figure out if 'aes' is available
instead of using /proc/cpuinfo, it _will_ get the 'aes' on one machine.

This application using CPUID and getting and not getting the right
filtered value is not present with HVM guests - as the CPUID instruction
is trapped there irregardless of whether it is running in the kernel or
user-land.