Migration problems

Migration problems

[Chris Lalancette]

All,
     I've been doing some migration testing of KVM guests, and have been running
into some problems.  Let me describe the setup and what I've tried, and maybe
somebody has some ideas about what might be going on here.

Setup:
     2 identical Intel SDV boxes,  Intel(R) Core(TM)2 Duo CPU     E6850  @ 3.00GHz
     Both machines are running F-8 kernel, 2.6.23.8-63.fc8 x86_64, with updated
KVM kernel modules from git.
     On one machine, I have a /kvm directory that holds my guest disk image;
this directory is exported via NFS and mounted as /kvm on the secondary machine.

The guest in question is also an F-8 x86_64 guest, running the same kernel as
the hosts.  I start it up with the following command-line:

qemu-system-x86_64 -hda /kvm/f8x86_64.dsk -boot c -m 385 -net
nic,vlan=0,macaddr=00:13:6e:12:34:56 -net tap,vlan=0,script=/etc/kvm-ifup
-monitor stdio

The guest starts up just fine.  On the secondary machine, I use the following
command-line:

qemu-system-x86_64 -hda /kvm/f8x86_64.dsk -boot c -m 385 -net
nic,vlan=0,macaddr=00:13:6e:12:34:56 -net tap,vlan=0,script=/etc/kvm-ifup
-monitor stdio -incoming tcp://0:4444

(i.e. exactly the same, but I add the -incoming parameter).  When I try to do
live migration this way, things seem like they work, and it even seems like a
few instructions get executed on the destination side.  However, fairly quickly
I'll get "Disabling IRQ #11" on the console of the guest at the destination, and
the qemu process will just spin at 100%, with no interaction possible.  IRQ #11,
incidentally, is the IRQ associated with the emulated rtl8139 card.

This led me to suspect the in-kernel PIC/APIC emulation, and, sure enough, if I
pass -no-kvm-irqchip (on both the source and destination), things are better; I
can at least migrate from the host to the destination without the "Disabling IRQ
#11" message.  However, if I put any sort of load on the guest while doing
migration, I still get a hang-up, even with -no-kvm-irqchip.

Has anyone else seen this, or have ideas where I can start debugging it?

Thanks,
Chris Lalancette

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Migration problems

[Avi Kivity]

Chris Lalancette wrote:
> All,
>      I've been doing some migration testing of KVM guests, and have been running
> into some problems.  Let me describe the setup and what I've tried, and maybe
> somebody has some ideas about what might be going on here.
>
> Setup:
>      2 identical Intel SDV boxes,  Intel(R) Core(TM)2 Duo CPU     E6850  @ 3.00GHz
>      Both machines are running F-8 kernel, 2.6.23.8-63.fc8 x86_64, with updated
> KVM kernel modules from git.
>      On one machine, I have a /kvm directory that holds my guest disk image;
> this directory is exported via NFS and mounted as /kvm on the secondary machine.
>
> The guest in question is also an F-8 x86_64 guest, running the same kernel as
> the hosts.  I start it up with the following command-line:
>
> qemu-system-x86_64 -hda /kvm/f8x86_64.dsk -boot c -m 385 -net
> nic,vlan=0,macaddr=00:13:6e:12:34:56 -net tap,vlan=0,script=/etc/kvm-ifup
> -monitor stdio
>
> The guest starts up just fine.  On the secondary machine, I use the following
> command-line:
>
> qemu-system-x86_64 -hda /kvm/f8x86_64.dsk -boot c -m 385 -net
> nic,vlan=0,macaddr=00:13:6e:12:34:56 -net tap,vlan=0,script=/etc/kvm-ifup
> -monitor stdio -incoming tcp://0:4444
>
> (i.e. exactly the same, but I add the -incoming parameter).  When I try to do
> live migration this way, things seem like they work, and it even seems like a
> few instructions get executed on the destination side.  However, fairly quickly
> I'll get "Disabling IRQ #11" on the console of the guest at the destination, and
> the qemu process will just spin at 100%, with no interaction possible.  IRQ #11,
> incidentally, is the IRQ associated with the emulated rtl8139 card.
>
> This led me to suspect the in-kernel PIC/APIC emulation, and, sure enough, if I
> pass -no-kvm-irqchip (on both the source and destination), things are better; I
> can at least migrate from the host to the destination without the "Disabling IRQ
> #11" message.  However, if I put any sort of load on the guest while doing
> migration, I still get a hang-up, even with -no-kvm-irqchip.
>
> Has anyone else seen this, or have ideas where I can start debugging it?
>
>   

[copying Uri, who is also chasing migration bugs]

- check the guest kernel without an ioapic
- if that works, check the ioapic load/save paths
- I'd also suspect 3ead9ca0bd2214af63ea2ebf84573576b38e004e or 
71ab66c92f1ecd3f1aabed0bfa2e356fb6bbfebc


-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Migration problems

[Uri Lublin]

Chris Lalancette wrote:
> All,
>      I've been doing some migration testing of KVM guests, and have been running
> into some problems.  Let me describe the setup and what I've tried, and maybe
> somebody has some ideas about what might be going on here.
>
> When I try to do
> live migration this way, things seem like they work, and it even seems like a
> few instructions get executed on the destination side.  However, fairly quickly
> I'll get "Disabling IRQ #11" on the console of the guest at the destination, and
> the qemu process will just spin at 100%, with no interaction possible.  IRQ #11,
> incidentally, is the IRQ associated with the emulated rtl8139 card.
>
> This led me to suspect the in-kernel PIC/APIC emulation, and, sure enough, if I
> pass -no-kvm-irqchip (on both the source and destination), things are better; I
> can at least migrate from the host to the destination without the "Disabling IRQ
> #11" message.  However, if I put any sort of load on the guest while doing
> migration, I still get a hang-up, even with -no-kvm-irqchip.
>
> Has anyone else seen this, or have ideas where I can start debugging it?
>
> Thanks,
> Chris Lalancette
>   
Hi Chris,

I am chasing migration bug(s) too.
I think the problem lies in userspace and not in the kernel modules.
It seems that for kvm-56 migration works but for kvm-57 it does not.
The problem seems to be related to the qemu-cvs merge: 
25a2e04e5689af63d104de2dea755f30bf643afd
If you run it with -no-kvm migration seems to be working.

The test case I'm using is savevm/loadvm.
After a loadvm my FC6 guest gets double-fault (<qemu-cmdline> -loadvm <id>).
Adding '-no-kvm-irqchip' does not help.
But if I start qemu in a stopped mode (<qemu-cmdline> -S) and 'loadvm 
<id>' in qemu-monitor, it works.
Sometimes after lodavm guest network does work better with '-no-kvm-irqchip'
I am trying to use bisection to find the problematic patch.

Thanks,
Uri.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Migration problems

[Guido Guenther]

[-- Attachment #1.1: Type: text/plain, Size: 575 bytes --]

On Thu, Jan 24, 2008 at 01:52:16PM +0200, Uri Lublin wrote:
> Sometimes after lodavm guest network does work better with '-no-kvm-irqchip'
> I am trying to use bisection to find the problematic patch.
Another migration problem that pops up with kvm 60, doing: 
migrate "exec:dd of=/tmp/bla" 
works in the qemu monitor however it doesn't, when you start kvm with
"-monitor pty" and pipe the same command into /dev/pts/X (which is what
libvirt does). Kvm segfaults immediately. I'll debug this further but
maybe someone else has seen this already?
Cheers,
 -- Guido

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Migration problems

[Guido Guenther]

On Fri, Jan 25, 2008 at 05:08:41PM +0100, Guido Guenther wrote:
> On Thu, Jan 24, 2008 at 01:52:16PM +0200, Uri Lublin wrote:
> > Sometimes after lodavm guest network does work better with '-no-kvm-irqchip'
> > I am trying to use bisection to find the problematic patch.
> Another migration problem that pops up with kvm 60, doing: 
> migrate "exec:dd of=/tmp/bla" 
> works in the qemu monitor however it doesn't, when you start kvm with
> "-monitor pty" and pipe the same command into /dev/pts/X (which is what
> libvirt does). Kvm segfaults immediately. I'll debug this further but
> maybe someone else has seen this already?
Actually the above is true vor kvm 58, with kvm 60 the migration starts
but doesn't finish, qemu becomes unresponsive and every access to
/proc/<kvmpid>/cmdline hangs forever (on 2.6.23.11). Attaching gdb to
the process doesn't work either. CPU is a Dual Core Opteron 2220.
Things work fine if I pass -no-kvm. Any ideas?
 -- Guido

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/