[Qemu-devel] KVM versions, machine types and failed migrations

[Qemu-devel] KVM versions, machine types and failed migrations

[Vangelis Koukis]

[-- Attachment #1: Type: text/plain, Size: 3042 bytes --]

Hello,

I'd like to ask a few questions about the way migrations work in KVM
among different emulated machine types and different versions of the
qemu-kvm package. I am sending to both the kvm@ and qemu-devel@ lists,
please redirect me if I was wrong in doing so.

In a nutshell: while trying to live-migrate a VM on ~okeanos [1], we
see VM migrations fail silently if going from kvm 1.0 to kvm 1.1.
The source VM is frozen, "info migrate" on the source monitor reports
success, but the VM is dead upon arrival on the destination process.
Please see [3] for the exact package versions for qemu-kvm we have
tested with.

Migration works if the destination kvm has been started with the same
machine type as the source VM, e.g., using "-M pc-1.0" specifically on
the destination, when migrating a pc-1.0 machine from kvm 1.0 to
kvm 1.1.

How does the machine type specified with -M work in the case of
migrations? Are migrations expected to fail if the machine type is
different between source and destination process? If yes, shouldn't KVM be
able to detect this and abort the migration instead of failing silently?

For every (src, dst) pair of "package version / machine type",
we saw the following:

     dst:       1.0/pc-0.12      1.0/pc-1.0      1.1/pc-1.1
src:
1.0/pc-0.12     ok               ok              fails silently
1.0/pc-1.0      fails silently   ok              fails silently
1.1/pc-1.1      fails silently   fails silently  ok

Machine types pc-0.12 and pc-1.0 were run with qemu-kvm package version
1.0, machine type pc-1.1 with qemu-kvm package version 1.1.

Also, the migration 1.0/pc-0.12 to 1.0/pc-1.0 seems to work... How can
it work or even be allowed, given that the guest finds itself running on
a different hardware configuration after the migration?

Regarding different package versions of qemu-kvm, it seems migrations do
not work from source 0.12.5 to any other version *even* if -M pc-0.12 is
specified at the incoming KVM process. For versions >= 1.0 everything
works provided the machine type on the destination is the same as on the
source.

Our goal is to patch Ganeti [2] so that it sets the destination machine
type to that of the source specifically, ensuring migrations work
seamlessly after a KVM upgrade. Is there a way to retrieve the machine
type of a running KVM process through a monitor command?

Thank you,
Vangelis.

[1] ~okeanos IaaS: http://okeanos.io
[2] Ganeti: https://code.google.com/p/ganeti/
[3] We tested on Debian Squeeze with package versions:

kvm 0.12: qemu-kvm_0.12.5+dfsg-5+squeeze8_amd64.deb
kvm 1.0: qemu-kvm_1.0+dfsg-8~bpo60+1_amd64.deb
kvm 1.1: qemu-kvm_1.1.2+dfsg-2~bpo60+1_amd64.deb

-- 
Vangelis Koukis
vkoukis@grnet.gr
OpenPGP public key ID:
pub  1024D/1D038E97 2003-07-13 Vangelis Koukis <vkoukis@cslab.ece.ntua.gr>
     Key fingerprint = C5CD E02E 2C78 7C10 8A00  53D8 FBFC 3799 1D03 8E97

Only those who will risk going too far
can possibly find out how far one can go.
        -- T.S. Eliot

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] KVM versions, machine types and failed migrations

[Daniel P. Berrange]

On Wed, Jan 09, 2013 at 02:23:50PM +0200, Vangelis Koukis wrote:
> Hello,
> 
> I'd like to ask a few questions about the way migrations work in KVM
> among different emulated machine types and different versions of the
> qemu-kvm package. I am sending to both the kvm@ and qemu-devel@ lists,
> please redirect me if I was wrong in doing so.
> 
> In a nutshell: while trying to live-migrate a VM on ~okeanos [1], we
> see VM migrations fail silently if going from kvm 1.0 to kvm 1.1.
> The source VM is frozen, "info migrate" on the source monitor reports
> success, but the VM is dead upon arrival on the destination process.
> Please see [3] for the exact package versions for qemu-kvm we have
> tested with.
> 
> Migration works if the destination kvm has been started with the same
> machine type as the source VM, e.g., using "-M pc-1.0" specifically on
> the destination, when migrating a pc-1.0 machine from kvm 1.0 to
> kvm 1.1.
> 
> How does the machine type specified with -M work in the case of
> migrations? Are migrations expected to fail if the machine type is
> different between source and destination process? If yes, shouldn't KVM be
> able to detect this and abort the migration instead of failing silently?

When doing migration, the fundamental requirement is that the guest
OS visible machine ABI must not change. Thus there are three key
things to take care of when launching QEMU on the migration target
host.

 - The device PCI/USB addresses must be identical to the source
 - The machine type must be identical to the source
 - The CPU model must be identical to the source

If you don't follow those requirements, either QEMU or the guest OS
or both will crash & burn during migration & you get to keep both
pieces :-)

> Regarding different package versions of qemu-kvm, it seems migrations do
> not work from source 0.12.5 to any other version *even* if -M pc-0.12 is
> specified at the incoming KVM process. For versions >= 1.0 everything
> works provided the machine type on the destination is the same as on the
> source.

Some older versions of QEMU were buggy causing the machine type to
not correctly preserve ABI.

> Our goal is to patch Ganeti [2] so that it sets the destination machine
> type to that of the source specifically, ensuring migrations work
> seamlessly after a KVM upgrade. Is there a way to retrieve the machine
> type of a running KVM process through a monitor command?

IIRC there is not a monitor command for this. The general approach
to dealing with migration stability should be to launch QEMU with a
canonical hardware configuration. This means explicitly setting a machine
type, CPU model and PCI/USB devices addresses upfront. NB you should not
use 'pc' as a machine type - if you query the list of machine types from
QEMU, it will tell you what 'pc' corresponds to (pc-1.2) and then use the
versioned type so you have a known machine type.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [okeanos-dev] Re: KVM versions, machine types and failed migrations

[Vangelis Koukis]

[-- Attachment #1: Type: text/plain, Size: 3229 bytes --]

On Wed, Jan 09, 2013 at 01:10:45pm +0000, Daniel P. Berrange wrote:
> When doing migration, the fundamental requirement is that the guest
> OS visible machine ABI must not change. Thus there are three key
> things to take care of when launching QEMU on the migration target
> host.
> 
>  - The device PCI/USB addresses must be identical to the source
>  - The machine type must be identical to the source
>  - The CPU model must be identical to the source
> 

Thanks for the detailed list of requirements, we'll take it into account
for the relevant Ganeti patch.

> If you don't follow those requirements, either QEMU or the guest OS
> or both will crash & burn during migration & you get to keep both
> pieces :-)
> 

My point is, are these requirements left up to the caller of "kvm
-incoming" to satisfy? Since the migration will most probably break,
wouldn't it be best for QEMU to detect this and complain loudly, instead
of continuing with the migration, failing silently and destroying the
VM?

Sure there could be some "yes, do it, I know it is going to break"
option, which will make QEMU proceed with the migration. However, in 99%
of the cases this is just user error, e.g. the user has upgraded the
version on the other end and has not specified -M explicitly. It would
be best if QEMU was able to detect and warn the user about what is going
to happen, because it does lead to the VM dying.

> > Regarding different package versions of qemu-kvm, it seems migrations do
> > not work from source 0.12.5 to any other version *even* if -M pc-0.12 is
> > specified at the incoming KVM process. For versions >= 1.0 everything
> > works provided the machine type on the destination is the same as on the
> > source.
> 
> Some older versions of QEMU were buggy causing the machine type to
> not correctly preserve ABI.
> 
> > Our goal is to patch Ganeti [2] so that it sets the destination machine
> > type to that of the source specifically, ensuring migrations work
> > seamlessly after a KVM upgrade. Is there a way to retrieve the machine
> > type of a running KVM process through a monitor command?
> 
> IIRC there is not a monitor command for this.>

> The general approach
> to dealing with migration stability should be to launch QEMU with a
> canonical hardware configuration. This means explicitly setting a machine
> type, CPU model and PCI/USB devices addresses upfront. NB you should not
> use 'pc' as a machine type - if you query the list of machine types from
> QEMU, it will tell you what 'pc' corresponds to (pc-1.2) and then use the
> versioned type so you have a known machine type.
> 

This is exactly what we're trying to do: specify "-M" explicitly in the
kvm command line, instead of letting the default "pc" machine type
change arbitrarily whenever the qemu-kvm package gets upgraded.

Thanks again,
Vangelis.

-- 
Vangelis Koukis
vkoukis@grnet.gr
OpenPGP public key ID:
pub  1024D/1D038E97 2003-07-13 Vangelis Koukis <vkoukis@cslab.ece.ntua.gr>
     Key fingerprint = C5CD E02E 2C78 7C10 8A00  53D8 FBFC 3799 1D03 8E97

Only those who will risk going too far
can possibly find out how far one can go.
        -- T.S. Eliot

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] [okeanos-dev] Re: KVM versions, machine types and failed migrations

[Daniel P. Berrange]

On Wed, Jan 09, 2013 at 03:27:53PM +0200, Vangelis Koukis wrote:
> On Wed, Jan 09, 2013 at 01:10:45pm +0000, Daniel P. Berrange wrote:
> > When doing migration, the fundamental requirement is that the guest
> > OS visible machine ABI must not change. Thus there are three key
> > things to take care of when launching QEMU on the migration target
> > host.
> > 
> >  - The device PCI/USB addresses must be identical to the source
> >  - The machine type must be identical to the source
> >  - The CPU model must be identical to the source
> > 
> 
> Thanks for the detailed list of requirements, we'll take it into account
> for the relevant Ganeti patch.
> 
> > If you don't follow those requirements, either QEMU or the guest OS
> > or both will crash & burn during migration & you get to keep both
> > pieces :-)
> > 
> 
> My point is, are these requirements left up to the caller of "kvm
> -incoming" to satisfy? Since the migration will most probably break,
> wouldn't it be best for QEMU to detect this and complain loudly, instead
> of continuing with the migration, failing silently and destroying the
> VM?
> 
> Sure there could be some "yes, do it, I know it is going to break"
> option, which will make QEMU proceed with the migration. However, in 99%
> of the cases this is just user error, e.g. the user has upgraded the
> version on the other end and has not specified -M explicitly. It would
> be best if QEMU was able to detect and warn the user about what is going
> to happen, because it does lead to the VM dying.

What you describe is certainly desirable, but it is quite hard to achieve
with current QEMU. Much of the work with moving to the new QEMU object
model & configuration descriptions has been motivated by a desire to
enable improvements migration handling. As you suggest, the goal is that
the source QEMU be able to send a complete & reliable hardware description
to the destination QEMU during migration.It is getting closer, but we're
not there yet.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|