Qemu-kvm migration problems between two different amd processors.

Qemu-kvm migration problems between two different amd processors.

[Magnus]

Hello,

I am having migration problems and would appreciate very much help in 
troubleshooting this further.

I have installed two wheezy hosts (qemu-1.1.2) using glusterfs as 
backend storage and are running about 12 debian wheezy guests.

Node A has newer hardware (AMD FX(tm)-8320 Eight-Core Processor) and 
node B is older (AMD Phenom(tm) II X6 1055T Processor).

I am using libvirt and when migrating hosts from Node A to node B the 
always freeze indicating high cpu utilization in virt-manager for 
example. Migrations from Node B to Node A always works. I can start all 
gests on both hosts without problems.

To rule out glusterfs as the problem I have also used nfs as backend 
storage with the same results.
I have installed a freebsd-9.1 guest to rule out the guests linux kernel 
as the likely problem. The guests also freeze if they are in grub or the 
freebsd bootloader while being migrated.

I have tried using this kernel parameter processor.max_cstate=1 and 
using changing clocksource to acpi_pm (defaults to kvm-clock). Neither 
of these had any effect on my problem.

I have tried disabling the balloon driver and that didn't help either.

Both hosts and all guests use ntp so the time is synced.

At one point I reinstalled both machines as ubuntu 13.04 to get newer 
versions of qemu (1.4) and libvirt but the exact same problem was there 
also, as I prefer debian I went back to wheezy.

-no-kvm-irqchip and -no-kvm-pit doesn't help either.

Using -no-kvm the migration works both ways.

My gut feeling is that it has something to do with the different cpu's, 
but I read that it was possible to migrate between intel and amd I guess 
this shouldn't be a problem.

Anyone have any ide what could be the problem here?

Best Regards
Magnus

Diff between cpuinfo of linux guest when booted on each host so no cpu 
flags differ in the guest. I have tried with Opteron_G2, qemu64 and 
kvm64 cpus all causes the guest to freeze.

root@lazy:~# diff -Nru iron1-cpuinfo.txt iron2-cpuinfo.txt
--- iron1-cpuinfo.txt   2013-09-19 20:59:24.407659093 +0200
+++ iron2-cpuinfo.txt   2013-09-19 21:01:03.326022357 +0200
@@ -5,14 +5,14 @@
  model name     : QEMU Virtual CPU version 1.1.2
  stepping       : 3
  microcode      : 0x1000065
-cpu MHz                : 3511.722
+cpu MHz                : 2812.460
  cache size     : 512 KB
  fpu            : yes
  fpu_exception  : yes
  cpuid level    : 4
  wp             : yes
  flags          : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up nopl pni cx16 
popcnt hypervisor lahf_lm svm abm sse4a
-bogomips       : 7023.44
+bogomips       : 5624.92
  TLB size       : 1024 4K pages
  clflush size   : 64
  cache_alignment        : 64

Node A cpuinfo
processor       : 7
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 2
model name      : AMD FX(tm)-8320 Eight-Core Processor
stepping        : 0
microcode       : 0x600081f
cpu MHz         : 1400.000
cache size      : 2048 KB
physical id     : 0
siblings        : 8
core id         : 7
cpu cores       : 4
apicid          : 23
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc 
extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 
sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic 
cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt 
lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb arat cpb 
hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid 
decodeassists pausefilter pfthreshold bmi1
bogomips        : 7023.38
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

Node B cpuinfo
processor       : 5
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 10
model name      : AMD Phenom(tm) II X6 1055T Processor
stepping        : 0
microcode       : 0x10000dc
cpu MHz         : 800.000
cache size      : 512 KB
physical id     : 0
siblings        : 6
core id         : 5
cpu cores       : 6
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl 
nonstop_tsc extd_apicid aperfmperf pni monitor cx16 popcnt lahf_lm 
cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch 
osvw ibs skinit wdt cpb hw_pstate npt lbrv svm_lock nrip_save pausefilter
bogomips        : 5624.50
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate cpb

Libvirt starts the guests like this:
LC_ALL=C 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 
QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -enable-kvm -m 512 -smp 
2,sockets=2,cores=1,threads=1 -name jefster -uuid 
f94a85d4-629e-bdba-b585-282638526800 -no-user-config -nodefaults 
-chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/jefster.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc 
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/vmimages/gv0/jefster.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=29 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:61:9b:f3,bus=pci.0,addr=0x3 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 
-vnc 127.0.0.1:5 -vga cirrus -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Hosts kernel:
Linux iron1 3.10-0.bpo.2-amd64 #1 SMP Debian 3.10.5-1~bpo70+1 
(2013-08-11) x86_64 GNU/Linux