Live migration bug introduced in 2.6.32.16?

[Nathan March <nathan@gt.net>]

Hi All,

It looks like a live migration bug may have been introduced in 2.6.32.16...

I've been experiencing issues where upon live migration, the domU simply 
hangs once it gets resumed on the target dom0. I've been unable to get 
any crash information out of the domU, nothing comes up in xm dmesg. 
There could be a kernel panic happening but since I can't connect to the 
console during the migration I haven't been able to get anything useful. 
Comparing a successful migration to a failed one in the xend.log and 
xen-debug.log, nothing stands out as being different.

Testing a wide variety of VM's to see why some worked and some didn't, 
I've narrowed it down to the domU kernel version and down to 2.6.32.16 
specifically by trying these versions:
2.6.32.8    good
2.6.32.15    good
2.6.32.16    bad
2.6.32.17    bad
2.6.32.20    bad
2.6.32.24    bad
2.6.32.28    bad
2.6.37        bad

All are the stock kernel off kernel.org.

Note that this isn't consistent at all, I've got 6 dom0's and this only 
happens when migrating certain directions between certain dom0's:
xen1->xen2    crash
xen1->xen5    crash
xen1->xen6    crash
xen2->xen5    crash
xen2->xen1    works
xen3->xen1    works
xen5->xen2    works
xen5->xen6    works
xen6->xen1    works
xen6->xen5    works

Previously, xen6->5 worked but xen5->6 didn't work. After a few reboots 
(of the dom0) however the problem between them resolved itself and now I 
can go xen5->6 and back all day on 2.6.32.16 without issues. If i then 
migrate it to xen1 it's fine, but back to xen5 and it locks up on resume.

All 6 xen dom0's are identical:

xen5 ~ # xm info
host                   : xen5
release                : 2.6.31.13
version                : #11 SMP Wed Jan 26 10:55:28 PST 2011
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00001f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 40950
free_memory            : 38380
node_to_cpu            : node0:0-5
                          node1:6-11
node_to_memory         : node0:23388
                          node1:14991
node_to_dma32_mem      : node0:2994
                          node1:0
max_node_id            : 1
xen_major              : 4
xen_minor              : 0
xen_extra              : .1-rc6-pre
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      :
cc_compile_date        : Tue Jan 25 17:05:03 PST 2011
xend_config_format     : 4

I've tried updating to a newer dom0 release but ran into linking issues 
due to as-needed so I haven't managed to get them up yet.

Looking at the changelog for 2.6.32.16 
(http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.32.16) there 
were two xen patches made, both involving resuming.

Diffs for the two patches:
http://git.kernel.org/?p=linux/kernel/git/longterm/linux-2.6.32.y.git;a=commitdiff;h=0f58db21025d979e38db691861985ebc931551b1
http://git.kernel.org/?p=linux/kernel/git/longterm/linux-2.6.32.y.git;a=commitdiff;h=b6d1fd29840e29d1a87d0ab15ee1ccc90ea15ec4

I've tried reversing them together and 1 at a time, yet the problem 
still happens. I then took 2.6.32.15 and applied those patches and it's 
completely stable. So whatever is causing this was apparently not a 
xen-related patch?

Anyone have any ideas on what might be going on here or how I can debug 
it further? I'm completely stumped at this point, don't want to just try 
applying every patch in 2.6.32.16 to see which one is doing it. 
Compiling + testing all these kernels is time consuming =)

Thanks,
Nathan