[Qemu-devel] [RFC] More robust migration

[Qemu-devel] [RFC] More robust migration

[Andre Przywara]

Hi,

after fiddling around with migration (and the data dumped into the 
stream) I found the current concept possesses some shortcomings. I am 
interested in your opinions whether it is worth to implement a new 
improved format.
Issues I would like to address:
1. Transfer configuration data. Currently there is no VM configuration 
data transferred with the stream. One has to start QEMU/KVM with the 
_exact_ same parameters on the other side to allow migration. If there 
would be a pseudo-device (transferred first) holding these parameters 
(and other runtime dependent stuff like kvm_enabled()) this would ease 
migration a lot.
2. Introduce a length field to the header of each device. This would 
allow to skip unknown (or unwanted) devices. I know this imposes a bit 
of a challenge, because the length is not always known in advance, but 
one could overcome this (by using the buffer to patch in the length 
later for instance).
3. Make the device versioning really bulletproof. Currently some devices 
dump different data depending on runtime (or better time-of-creation) 
state (for instance hw/i8254.c: if (s->irq_timer)...). Another example 
is the (x86?) CPU state, which differs with KVM en/disabled. Some 
devices even dump host system dependent structures (like struct vecio in 
virtio-blk.c).
Also one could create some kind of (limited) upward compatibility, so 
older QEMU versions ignore additional, but optional fields in a device 
state (similar to the ext2 compatibility scheme). Maybe this could be 
done by an external converter program.
4. Allow optional devices. Some devices are always started (like HPET), 
although they don't need to be used by the OS. If one migrates such a 
guest from say KVM-83 to KVM-81, it will fail, because KVM-81 does not 
support HPET. One could migrate the device only if it has been used.

In general I would like to know whether QEMU migration is intended to be 
used in such a flexible manner or whether the requirement of the exact 
same software version on both side is not a limitation in everyday use.

Awaiting your comments!

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-4917
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Jochen Polster; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: [Qemu-devel] [RFC] More robust migration

[Anthony Liguori]

Hi Andre,

Andre Przywara wrote:
> Hi,
>
> after fiddling around with migration (and the data dumped into the 
> stream) I found the current concept possesses some shortcomings.

Yikes :-)  FWIW, I focused a lot on robustness in the implementation so 
hopefully a lot of what you mention below were conscious decisions with 
very specific reasoning.

> I am interested in your opinions whether it is worth to implement a 
> new improved format.

FWIW, the format is sufficiently versioned that it isn't necessary to 
completely change it (not that I think it needs changing).

> Issues I would like to address:
> 1. Transfer configuration data. Currently there is no VM configuration 
> data transferred with the stream.

Yes, the difficulty here is that we need to transfer the machine 
configuration but not the host configuration.  Management tools should 
decide how to configure the host on the target side but we should be 
passing the machine configuration.

If you've been following the config file threads, I've mentioned this as 
a use case for the current design a number of times.  We would pass a 
flattened device tree as another savevm section with a well known name 
(like "machine").  Given the semantics of the current migration 
protocol, this would ensure that the machine generated on the remote 
node was exactly the same as the source node.

> One has to start QEMU/KVM with the _exact_ same parameters on the 
> other side to allow migration. If there would be a pseudo-device 
> (transferred first) holding these parameters (and other runtime 
> dependent stuff like kvm_enabled()) this would ease migration a lot.

FWIW, there's nothing preventing migrating from TCG -> KVM.

I think one can debate about whether host config should be migrated 
too.  I'd argue that in the core migration protocol, host config should 
not be present.  I think you can have an easier to use migration 
protocol (like the old ssh protocol) that also transferred host config.  
But in the general case, you want management tools to be able to 
manipulate host config upon migration.

> 2. Introduce a length field to the header of each device.

IMHO, this would reduce robustness.  It's also difficult because of the 
way savevm registration works.  You don't know how large a section is 
until it's written and migration streams are not seekable.

> This would allow to skip unknown (or unwanted) devices.

No good can come from this.  If you have an unknown section, you must 
throw and error and stop the migration.  What if this is for a device 
that the guest is interacting with?  The device just disappears after 
migration?   All savevm state is state that affects the functionality of 
a guest.  Throwing away this state will change the functionality of the 
VM and migration should not affect guest functionality.

> I know this imposes a bit of a challenge, because the length is not 
> always known in advance, but one could overcome this (by using the 
> buffer to patch in the length later for instance).

What are the use cases where you think this would be beneficial?  I 
really see the change in semantics from the old way (throwing away 
unknown sections) to the new way (requiring strict versioning and 
validating all sections) as being a huge step toward robustness.

>
> 3. Make the device versioning really bulletproof. Currently some 
> devices dump different data depending on runtime (or better 
> time-of-creation) state (for instance hw/i8254.c: if (s->irq_timer)...).

If you look carefully, s->irq_timer will always be set.  The checks are 
unnecessary.

> Another example is the (x86?) CPU state, which differs with KVM 
> en/disabled.

Not in upstream QEMU...

> Some devices even dump host system dependent structures (like struct 
> vecio in virtio-blk.c).

That is awful and needs to be fixed.  It should have never been 
committed like that.

>
> Also one could create some kind of (limited) upward compatibility, so 
> older QEMU versions ignore additional, but optional fields in a device 
> state (similar to the ext2 compatibility scheme). Maybe this could be 
> done by an external converter program.

To me, ignoring is always a bad thing.  It's almost always going to be 
unsafe.  Doesn't this decrease robustness by being less conservative?

> 4. Allow optional devices. Some devices are always started (like 
> HPET), although they don't need to be used by the OS. If one migrates 
> such a guest from say KVM-83 to KVM-81, it will fail, because KVM-81 
> does not support HPET. One could migrate the device only if it has 
> been used.

There's no way you can migrate from KVM-83 to KVM-81 if you've enabled 
the HPET.  It cannot be made to work.

There is a -no-hpet option though.  If you are a management tool that 
needs to support migration from multiple versions, you should use 
-no-hpet.  Also, if you need to migrate from KVM-81 to KVM-83, you 
should use -no-hpet with KVM-83 to avoid changing the guest visible state.

In the long run, the machine configuration file will address this in a 
more thorough manner.  FWIW, -no-hpet was added specifically to deal 
with migration.

> In general I would like to know whether QEMU migration is intended to 
> be used in such a flexible manner or whether the requirement of the 
> exact same software version on both side is not a limitation in 
> everyday use.

My primary goal for migration is robustness.  I do not think it's a good 
idea to support any circumstances that could introduce changes in guest 
visible state during a live migration.

Live migration is a critical feature for many production environments.  
To be useful IMHO, it has to be bullet-proof.

Regards,

Anthony Liguori

> Awaiting your comments!
>
> Regards,
> Andre.
>

Re: [Qemu-devel] [RFC] More robust migration

[Paul Brook]

> > In general I would like to know whether QEMU migration is intended to
> > be used in such a flexible manner or whether the requirement of the
> > exact same software version on both side is not a limitation in
> > everyday use.
>
> My primary goal for migration is robustness.  I do not think it's a good
> idea to support any circumstances that could introduce changes in guest
> visible state during a live migration.
>
> Live migration is a critical feature for many production environments.
> To be useful IMHO, it has to be bullet-proof.

I agree.

I suspect that in practice live migration of a VM between different qemu 
versions ends up comparable to in-place live kernel upgrades. i.e. it takes 
an awful lot of work and care to make it happen, and in practice isn't going 
to happen for any particularly useful span of versions.

Paul

Re: [Qemu-devel] [RFC] More robust migration

[Jamie Lokier]

Paul Brook wrote:
> I suspect that in practice live migration of a VM between different qemu 
> versions ends up comparable to in-place live kernel upgrades. i.e. it takes 
> an awful lot of work and care to make it happen, and in practice isn't going 
> to happen for any particularly useful span of versions.

On the other hand, stored snapshots of guests which are ready to run
in a particular state are quite useful.  I'm under the impression the
migration code is/will be useful for storing snapshots of system (not
disk) state too.  Is that right?

-- Jamie

Re: [Qemu-devel] [RFC] More robust migration

[Paul Brook]

On Friday 20 February 2009, Jamie Lokier wrote:
> Paul Brook wrote:
> > I suspect that in practice live migration of a VM between different qemu
> > versions ends up comparable to in-place live kernel upgrades. i.e. it
> > takes an awful lot of work and care to make it happen, and in practice
> > isn't going to happen for any particularly useful span of versions.
>
> On the other hand, stored snapshots of guests which are ready to run
> in a particular state are quite useful.  I'm under the impression the
> migration code is/will be useful for storing snapshots of system (not
> disk) state too.  Is that right?

I don't think there's any real difference between snapshotting and migration 
in this case. It's basically the same code.

I'm not saying it's a useless feature, just that it's very extremely to do 
reliably, and for that reason unlikely to happen. An unreliable 
implementation (i.e. one that claims to migrate/snapshot, then breaks some of 
the time) is IMHO worse than nothing.

Paul

Re: [Qemu-devel] [RFC] More robust migration

[Jamie Lokier]

Paul Brook wrote (about resuming snapshots on a different QEMU):
> I'm not saying it's a useless feature, just that it's very extremely
> to do reliably, and for that reason unlikely to happen. An
> unreliable implementation (i.e. one that claims to migrate/snapshot,
> then breaks some of the time) is IMHO worse than nothing.

Well, one which cannot resume from a snapshot at all unless you have
the original host and original QEMU/KVM around is pretty bad.

I've already been bitten by that, and had to throw a useful guest
snapshot away because of it.

But I've also been bitten by it resuming in a faulty manner across
QEMU versions, so I appreciate both points of view.

-- Jamie

Re: [Qemu-devel] [RFC] More robust migration

[Paul Brook]

On Monday 23 February 2009, Jamie Lokier wrote:
> Paul Brook wrote (about resuming snapshots on a different QEMU):
> > I'm not saying it's a useless feature, just that it's very extremely
> > to do reliably, and for that reason unlikely to happen. An
> > unreliable implementation (i.e. one that claims to migrate/snapshot,
> > then breaks some of the time) is IMHO worse than nothing.
>
> Well, one which cannot resume from a snapshot at all unless you have
> the original host and original QEMU/KVM around is pretty bad.

I never said you need the same host. All the save/restore code should be host 
independent. It should be possible to save state on (say) i386 and restore on 
ppc64. Anything that prevents this is IMO a bug.

For KVM you're likely to need a cpu with at least as many features as the old 
one, but that's the price you pay for using host hardware features.

However different versions of qemu (i.e. qemu built from different sources) 
are a different question altogether. If you can't reproduce your old qemu 
binaries you have much larger problems.

Paul

Re: [Qemu-devel] [RFC] More robust migration

[Jamie Lokier]

Paul Brook wrote:
> I never said you need the same host. All the save/restore code
> should be host independent. It should be possible to save state on
> (say) i386 and restore on ppc64. Anything that prevents this is IMO
> a bug.
>
> For KVM you're likely to need a cpu with at least as many features
> as the old one, but that's the price you pay for using host hardware
> features.

I'd prefer the "host hardware features" to be an acceleration
mechanism, than something which makes a VM dependent on the specific
host it's running on.

Can't KVM invoke QEMU's emulation capabilities for those things it
cannot provide itself because of missing host abilities?

-- Jamie

Re: [Qemu-devel] [RFC] More robust migration

[Paul Brook]

> Can't KVM invoke QEMU's emulation capabilities for those things it
> cannot provide itself because of missing host abilities?

You'd have to ask the KVM folks, but I suspect the answer is no. Certainly if 
you care about performance then you really don't want to be emulating stuff.  
If you know you're going to want to do this this then the solution is to 
expose a lowest-common-denominator set of features from the start.

Paul

Re: [Qemu-devel] [RFC] More robust migration

[Anthony Liguori]

Jamie Lokier wrote:
> Paul Brook wrote:
>   
>> I never said you need the same host. All the save/restore code
>> should be host independent. It should be possible to save state on
>> (say) i386 and restore on ppc64. Anything that prevents this is IMO
>> a bug.
>>
>> For KVM you're likely to need a cpu with at least as many features
>> as the old one, but that's the price you pay for using host hardware
>> features.
>>     
>
> I'd prefer the "host hardware features" to be an acceleration
> mechanism, than something which makes a VM dependent on the specific
> host it's running on.
>
> Can't KVM invoke QEMU's emulation capabilities for those things it
> cannot provide itself because of missing host abilities?
>   

In theory, software can do anything :-)  In practice, it's pretty nasty 
to mix TCG execution with direct execution.  TCG doesn't guarantee 
atomicity of translated instructions so if you had TCG running one VCPU 
and bare metal running the other badness could occur.  Plus, TCG has 
hidden state that needs to be synchronized between the two.

Yeah, you could halt all VCPUs and just run one in TCG, or some other 
equally hackish thing.  But the point is, it's not easy and since it's 
not a tremendously popular thing to do, noone's working on it.  We have 
much more important low-hanging fruit (like making qcow2 not suck).

Regards,

Anthony Liguori

> -- Jamie
>
>
>   

Re: [Qemu-devel] [RFC] More robust migration

[Avi Kivity]

Jamie Lokier wrote:
> Paul Brook wrote:
>   
>> I never said you need the same host. All the save/restore code
>> should be host independent. It should be possible to save state on
>> (say) i386 and restore on ppc64. Anything that prevents this is IMO
>> a bug.
>>
>> For KVM you're likely to need a cpu with at least as many features
>> as the old one, but that's the price you pay for using host hardware
>> features.
>>     
>
> I'd prefer the "host hardware features" to be an acceleration
> mechanism, than something which makes a VM dependent on the specific
> host it's running on.
>
> Can't KVM invoke QEMU's emulation capabilities for those things it
> cannot provide itself because of missing host abilities?

Some host hardware features are invisible to the guest (so they are just 
acceleration mechanisms).  Others are visible to the guest (like 
instruction set extensions) but they can be turned off using cpuid so 
that mixed-feature migration pools are possible.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [RFC] More robust migration

[Jamie Lokier]

Anthony Liguori wrote:
> >2. Introduce a length field to the header of each device.
> 
> IMHO, this would reduce robustness.  It's also difficult because of the 
> way savevm registration works.  You don't know how large a section is 
> until it's written and migration streams are not seekable.

The way HTTP deals with not knowing the size in advance is is to split
data into chunks, each chunk the size of a small write buffer, and a
chunk size is written in front of each one.  This allows storing
sections of binary data whose size isn't known in advance, but still
safely skip them.

> >This would allow to skip unknown (or unwanted) devices.
> 
> No good can come from this.  If you have an unknown section, you must 
> throw and error and stop the migration.  What if this is for a device 
> that the guest is interacting with?  The device just disappears after 
> migration?   All savevm state is state that affects the functionality of 
> a guest.  Throwing away this state will change the functionality of the 
> VM and migration should not affect guest functionality.

What if you're migrating from a snapshot made on a host with some
pass-through USB device to another host which cannot provide the same
device.  In that case I'd like the option for the guest to see the
device has disappeared.  Maybe it's stopped working (HPET), or maybe
it's unplugged (anything hot unpluggable).

That's preferable to not being able to use the snapshot at all,
effectively having to trash it.

For live migration this isn't much of an issue because you can unplug
the device before migrating, and you probably would like to be warned
of this before migrating anyway.

> >I know this imposes a bit of a challenge, because the length is not 
> >always known in advance, but one could overcome this (by using the 
> >buffer to patch in the length later for instance).
> 
> What are the use cases where you think this would be beneficial?  I 
> really see the change in semantics from the old way (throwing away 
> unknown sections) to the new way (requiring strict versioning and 
> validating all sections) as being a huge step toward robustness.

I've been upset at a "savevm" which I wrote with some past version of
QEMU that I couldn't load in a later version.  It wasn't obvious why,
just that it refused. And I didn't have the old version, or even know
which the old version was.  And even if I could have reconstructed the
old QEMU - I wanted to migrate to a newer version.  It's no fun having
to reconstruct a carefully primed guest snapshot test state from its
reboot, if that can be avoided.

I do think what you've done to make migration type-checking more
strict is very good.  Much better than running the wrong thing :-)

But I think there's a case for well-defined types of migration
flexibility too: migrating KVM to TCG and back (if you're moving
between hosts of different CPU features), migrating to new versions of
KVM and QEMU especially, changes of host device availability.

> >Also one could create some kind of (limited) upward compatibility, so 
> >older QEMU versions ignore additional, but optional fields in a device 
> >state (similar to the ext2 compatibility scheme). Maybe this could be 
> >done by an external converter program.
> 
> To me, ignoring is always a bad thing.  It's almost always going to be 
> unsafe.  Doesn't this decrease robustness by being less conservative?

I wonder if there's a use for something like ext2/3/4's capability
bits.  There are "required" capabilities and "backward compatible"
capabilities.  A backward compatible capability is only used when it's
explicitly designed in later versions to be compatible with older
versions.

I don't see a use-case immediately, it wouldn't be surprising though.

> My primary goal for migration is robustness.  I do not think it's a good 
> idea to support any circumstances that could introduce changes in guest 
> visible state during a live migration.

What about safe hotpluggable devices?

> Live migration is a critical feature for many production environments.  
> To be useful IMHO, it has to be bullet-proof.

I agree, and I think all that I've said is primarily about snapshots
rather than live migrations.

-- Jamie

Re: [Qemu-devel] [RFC] More robust migration

[Anthony Liguori]

Jamie Lokier wrote:
> Anthony Liguori wrote:
>   
>>> 2. Introduce a length field to the header of each device.
>>>       
>> IMHO, this would reduce robustness.  It's also difficult because of the 
>> way savevm registration works.  You don't know how large a section is 
>> until it's written and migration streams are not seekable.
>>     
>
> The way HTTP deals with not knowing the size in advance is is to split
> data into chunks, each chunk the size of a small write buffer, and a
> chunk size is written in front of each one.  This allows storing
> sections of binary data whose size isn't known in advance, but still
> safely skip them.
>
>   
>>> This would allow to skip unknown (or unwanted) devices.
>>>       
>> No good can come from this.  If you have an unknown section, you must 
>> throw and error and stop the migration.  What if this is for a device 
>> that the guest is interacting with?  The device just disappears after 
>> migration?   All savevm state is state that affects the functionality of 
>> a guest.  Throwing away this state will change the functionality of the 
>> VM and migration should not affect guest functionality.
>>     
>
> What if you're migrating from a snapshot made on a host with some
> pass-through USB device to another host which cannot provide the same
> device.  In that case I'd like the option for the guest to see the
> device has disappeared.  Maybe it's stopped working (HPET), or maybe
> it's unplugged (anything hot unpluggable).
>   

Stop working is IMHO unacceptable.  Devices that support hot plugging, 
you can hot unplug and *then* perform the migration.

In general, hot unplugging requires guest cooperation FWIW.  Bad things 
will often happen if you just yank a USB cable out of your computer.

> That's preferable to not being able to use the snapshot at all,
> effectively having to trash it.
>   

I disagree.  Something that is broken in an unknown way is not better 
than having something gracefully fail.  If you do hardware pass through, 
forget about snapshotting/migration/etc.

>> What are the use cases where you think this would be beneficial?  I 
>> really see the change in semantics from the old way (throwing away 
>> unknown sections) to the new way (requiring strict versioning and 
>> validating all sections) as being a huge step toward robustness.
>>     
>
> I've been upset at a "savevm" which I wrote with some past version of
> QEMU that I couldn't load in a later version.  It wasn't obvious why,
> just that it refused. And I didn't have the old version, or even know
> which the old version was.  And even if I could have reconstructed the
> old QEMU - I wanted to migrate to a newer version.  It's no fun having
> to reconstruct a carefully primed guest snapshot test state from its
> reboot, if that can be avoided.
>   

Device configuration files will go a long way to upgrading.  Sometimes 
you have to blacklist older versions of devices because there were bugs 
in the save/restore functions.  In that case, there's really nothing we 
can do.  Your snapshot was invalid.

>> My primary goal for migration is robustness.  I do not think it's a good 
>> idea to support any circumstances that could introduce changes in guest 
>> visible state during a live migration.
>>     
>
> What about safe hotpluggable devices?
>   

Make your changes in the guest to allow safe unplug, then unplug, then 
migrate.

Regards,

Anthony Liguori

[Qemu-devel] Re: [RFC] More robust migration

[Charles Duffy]

Andre Przywara wrote:
> 1. Transfer configuration data. Currently there is no VM configuration 
> data transferred with the stream. One has to start QEMU/KVM with the 
> _exact_ same parameters on the other side to allow migration. If there 
> would be a pseudo-device (transferred first) holding these parameters 
> (and other runtime dependent stuff like kvm_enabled()) this would ease 
> migration a lot.

I presently have code which *relies* on the ability to migrate from one 
VM to a target with different different backing stores for their disk 
images (albeit sharing the same contents).

If this ability were to be removed, that would be unfortunate.

Re: [Qemu-devel] Re: [RFC] More robust migration

[Jamie Lokier]

Charles Duffy wrote:
> Andre Przywara wrote:
> >1. Transfer configuration data. Currently there is no VM configuration 
> >data transferred with the stream. One has to start QEMU/KVM with the 
> >_exact_ same parameters on the other side to allow migration. If there 
> >would be a pseudo-device (transferred first) holding these parameters 
> >(and other runtime dependent stuff like kvm_enabled()) this would ease 
> >migration a lot.
> 
> I presently have code which *relies* on the ability to migrate from one 
> VM to a target with different different backing stores for their disk 
> images (albeit sharing the same contents).
> 
> If this ability were to be removed, that would be unfortunate.

Also, how else are we to migrate running systems from qcow2 - which
should never be used but this wasn't obvious before - to raw? :-)

-- Jamie