[Qemu-devel] [RFC] Migration capability negotation

[Qemu-devel] [RFC] Migration capability negotation

[Peter Lieven]

Hi,

I was thinking that it would be great to have the source and destination during migration negoatiate
migration capabilities e.g. something like this:

User wants to use a feature e.g. 'zero_blocks'. He switches it to 'on' or maybe a new state 'auto' on the source VM.

If the migration is started the source hypervisor sends a set of all desired features. The destination hypervisor
answers with a subset of all features it supports and automatically enables them on its side. Depending on the returned
subset the source disables all features the destination does not support.

This would also allow us also to introduce new features which we would like to enable by default, but we cannot
because we do not know if the destination will support it.

Is there any way to add this without breaking backwards compability?

Comments welcome.

Peter

Re: [Qemu-devel] [RFC] Migration capability negotation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

On 10/24/2013 07:39 PM, Peter Lieven wrote:
> Hi,
> 
> I was thinking that it would be great to have the source and destination during migration negoatiate
> migration capabilities e.g. something like this:
> 
> User wants to use a feature e.g. 'zero_blocks'. He switches it to 'on' or maybe a new state 'auto' on the source VM.
> 
> If the migration is started the source hypervisor sends a set of all desired features. The destination hypervisor
> answers with a subset of all features it supports and automatically enables them on its side. Depending on the returned
> subset the source disables all features the destination does not support.
> 
> This would also allow us also to introduce new features which we would like to enable by default, but we cannot
> because we do not know if the destination will support it.
> 
> Is there any way to add this without breaking backwards compability?

It's already been added: QMP has 'migrate-set-capabilities' and
'query-migrate-capabilities', which the management uses to negotiate the
capabilities between source and destination before starting the migration.

Other than that, you MUST remember that migration is one-directional
(the source does NOT get any input from the destination, other than what
the management app such as libvirt provides via the setup it does
outside of qemu before telling qemu to start migration).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [RFC] Migration capability negotation

[Juan Quintela]

Peter Lieven <pl@kamp.de> wrote:
> Hi,
>
> I was thinking that it would be great to have the source and
> destination during migration negoatiate
> migration capabilities e.g. something like this:
>
> User wants to use a feature e.g. 'zero_blocks'. He switches it to 'on'
> or maybe a new state 'auto' on the source VM.
>
> If the migration is started the source hypervisor sends a set of all
> desired features. The destination hypervisor
> answers with a subset of all features it supports and automatically
> enables them on its side. Depending on the returned
> subset the source disables all features the destination does not support.
>
> This would also allow us also to introduce new features which we would
> like to enable by default, but we cannot
> because we do not know if the destination will support it.
>
> Is there any way to add this without breaking backwards compability?
>
> Comments welcome.

As said by Eric,  comunications goes only in one direction (think that
we are migrating to a file,  A.K.A. savevm).  Anthony basically forbides
them.  You can do the equivalent thing from the management application.

Re: [Qemu-devel] [RFC] Migration capability negotation

[Peter Lieven]

Am 25.10.2013 um 01:37 schrieb Juan Quintela <quintela@redhat.com>:

> Peter Lieven <pl@kamp.de> wrote:
>> Hi,
>> 
>> I was thinking that it would be great to have the source and
>> destination during migration negoatiate
>> migration capabilities e.g. something like this:
>> 
>> User wants to use a feature e.g. 'zero_blocks'. He switches it to 'on'
>> or maybe a new state 'auto' on the source VM.
>> 
>> If the migration is started the source hypervisor sends a set of all
>> desired features. The destination hypervisor
>> answers with a subset of all features it supports and automatically
>> enables them on its side. Depending on the returned
>> subset the source disables all features the destination does not support.
>> 
>> This would also allow us also to introduce new features which we would
>> like to enable by default, but we cannot
>> because we do not know if the destination will support it.
>> 
>> Is there any way to add this without breaking backwards compability?
>> 
>> Comments welcome.
> 
> As said by Eric,  comunications goes only in one direction (think that
> we are migrating to a file,  A.K.A. savevm).  Anthony basically forbides
> them.  You can do the equivalent thing from the management application.

Ok, one way direction - i forgot about this paradigm.

2 thoughts:

a) a send-capabilities capability that "stores" the capabilities that where
used when savevm was used. I would implement a special segment
right at the beginning of the data stream that has all capabilities listed that
where set and that ultimately must be supported to import a saved state
under any circumstances. capabilities that are only have a meaning at
the source VM should not be set. if there is an unsupported capability
set the import can be aborted right at the beginning.

b) an extension the the qmp-migrate-capabilities or a new command that give
the controlling process (e.g. libvirt) a hint which features are a good thing to turn on
if they are supported on both sides (e.g. zero-blocks in block-migration).

Peter

Re: [Qemu-devel] [RFC] Migration capability negotation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

On 10/25/2013 04:27 AM, Peter Lieven wrote:

> Ok, one way direction - i forgot about this paradigm.
> 
> 2 thoughts:
> 
> a) a send-capabilities capability that "stores" the capabilities that where
> used when savevm was used. I would implement a special segment
> right at the beginning of the data stream that has all capabilities listed that
> where set and that ultimately must be supported to import a saved state
> under any circumstances. capabilities that are only have a meaning at
> the source VM should not be set. if there is an unsupported capability
> set the import can be aborted right at the beginning.

Sounds reasonable; but ideally, it would either have to be in such a way
that doesn't break back-compat with older qemu, or else you have
invented a new file format; and once you invent a new file format, we
might as well make the file format sane by being FULLY self-describing
(see also Alexander Graf's work from KVM forum on adding a migrate-debug
device).  That is, don't just require the same capabilities, but also
require all other command line arguments to be sane in comparison to
what the savevm image was using.

> 
> b) an extension the the qmp-migrate-capabilities or a new command that give
> the controlling process (e.g. libvirt) a hint which features are a good thing to turn on
> if they are supported on both sides (e.g. zero-blocks in block-migration).

Not really needed.  New capabilities must be off by default (back-compat
reasons), so the only time they will be turned on at the source is if
the management (such as libvirt) is smart enough to know what the
capability does; once you can assume that, you can also assume the
management knows how to set up the destination properly.  Which is why
what we have already works (making management do all the negotiation
correctly).  Yes, maybe qemu could make it easier or more foolproof, but
since management already has to handle the job (particularly because
management might be dealing with older qemu that doesn't have the
ease-of-use additions), I'm not sure the effort of extra code in qemu is
worth the effort.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [RFC] Migration capability negotation

[Peter Lieven]

On 25.10.2013 07:42, Eric Blake wrote:
> On 10/25/2013 04:27 AM, Peter Lieven wrote:
>
>> Ok, one way direction - i forgot about this paradigm.
>>
>> 2 thoughts:
>>
>> a) a send-capabilities capability that "stores" the capabilities that where
>> used when savevm was used. I would implement a special segment
>> right at the beginning of the data stream that has all capabilities listed that
>> where set and that ultimately must be supported to import a saved state
>> under any circumstances. capabilities that are only have a meaning at
>> the source VM should not be set. if there is an unsupported capability
>> set the import can be aborted right at the beginning.
> Sounds reasonable; but ideally, it would either have to be in such a way
> that doesn't break back-compat with older qemu, or else you have
> invented a new file format; and once you invent a new file format, we
> might as well make the file format sane by being FULLY self-describing
> (see also Alexander Graf's work from KVM forum on adding a migrate-debug
> device).  That is, don't just require the same capabilities, but also
> require all other command line arguments to be sane in comparison to
> what the savevm image was using.
what I had in mind when I thought about this was my old idea to skip
zero pages in ram migration. it turned out that there is a problem because
the target vm memory is not clean as expected but contains things like roms,
bios etc so we had to remove this.

when there is a way by ensuring that the target vm memory is clean either
by avoiding to load data into the ram or the hacky way to erase contents at
beginning of ram_load it must be somewhere in the datastream that the
zero blocks where left out. i thought to have this either by a capability
info block or by a special ram block with a flag that means erase all memory.
the latter would ensure that the memory is clean regardless if the capability
is set in the destination and secondly that the migration load fails if the
flag is unknown.

>
>> b) an extension the the qmp-migrate-capabilities or a new command that give
>> the controlling process (e.g. libvirt) a hint which features are a good thing to turn on
>> if they are supported on both sides (e.g. zero-blocks in block-migration).
> Not really needed.  New capabilities must be off by default (back-compat
> reasons), so the only time they will be turned on at the source is if
> the management (such as libvirt) is smart enough to know what the
> capability does; once you can assume that, you can also assume the
> management knows how to set up the destination properly.  Which is why
> what we have already works (making management do all the negotiation
> correctly).  Yes, maybe qemu could make it easier or more foolproof, but
> since management already has to handle the job (particularly because
> management might be dealing with older qemu that doesn't have the
> ease-of-use additions), I'm not sure the effort of extra code in qemu is
> worth the effort.
>
I understand. I am ok with saying its the management apps task
to negotiate capabilities.

Peter

Peter