[Bug 43501] New: all 32bit binaries produce "Illegal Instruction" after KVM migration from AMD -> Intel host

[bugzilla-daemon@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=43501

           Summary: all 32bit binaries produce "Illegal Instruction" after
                    KVM migration from AMD -> Intel host
           Product: Virtualization
           Version: unspecified
    Kernel Version: 2.6.32-220.17.1.el6.x86_64
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: kvm
        AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
        ReportedBy: pzimdars@gmail.com
        Regression: No


We recently added an AMD KVM host (AMD Opteron 6276) to our existing Intel
(E5630) only KVM host pool. We migrated multiple VMs (Linux CentOS 5.8 / 6.1 /
6.2) guests from the Intel hosts to the AMD host without issue. We rebooted
multiple VM guests (5.8 / 6.2) on the AMD host and migrated them back to the
Intel hosts. We then noticed that certain programs stopped working. We isolated
the issue to all 32bit binaries failing to run with "Illegal Instruction"
errors and we also receive "trap invalid opcode" on the VM guest console:

glibc_post_upgr[1812] trap invalid opcode ip:ae1423 sp:ff9e8d28 error:0
pango-querymodu[1825] trap invalid opcode ip:93c423 sp:ffe19604 error:0
java[1877] trap invalid opcode ip:f77a2423 sp:fffecb18 error:0
[..snip..]
(all 32bit applications)

If we migrate the VM guest back to the AMD host we no longer encounter any
issues. If we reboot the failing VM guest(s) on the Intel host (after migrating
from the AMD host) the 32bit binaries work. We attempted to remove CPU features
that are presented to the KVM guests but still encounter the problem. Here is
the flags we have enabled on the guest:

flags           : fpu de tsc msr pae cx8 apic cmov clflush fxsr sse sse2 lm
unfair_spinlock hypervisor lahf_lm

This is reproducible every time. We attempted to use different OS installs
(CentOS 5.8, RedHat 6.2, SL 6.2) but the issue is present on all of them
(including their respective kernels). We also attempted to use different
<models> and we custom created our own model. Nothing made a difference.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.