Re: [Qemu-devel] Re: Live migration protocol, device features, ABIs and other beasts

[Anthony Liguori <anthony@codemonkey.ws>]

Eduardo Habkost wrote:
> On Mon, Nov 23, 2009 at 12:28:16PM -0600, Anthony Liguori wrote:
>   
>> Eduardo Habkost wrote:
>>     
>>> That may be good enough for upstream Qemu, but IMO for RHEL it is not a
>>> realistic policy. If the definition of "guest visible state" is buggy on
>>> the current implementation, we can't drop entirely the possibility of
>>> fixing it on our stable branch.
>>>   
>>>       
>> After mulling over it a bit, here's what I'd suggest:
>>
>> 1) Integrate VMstate with qdev
>> 2) Introduce a bitmap blacklist for unsupported VMstate versions
>> 3) Expose that bitmap as a qdev property for each device.
>> 4) By default, upstream qemu will always set the bitmap to be 100% correct.
>>
>> This provides a mechanism for informed users and downstreams to reduce  
>> correctness in favor of migration compatibility on a case-by-case basis.
>>     
>
> Is this for backward migration?

It's for migrating from an older qemu to a newer one.  Normally, newer 
qemu will happily support older formats but in this case, we broke 
something and we need to blacklist the old format.  This lets you 
override that black list.

> If so, even with this bitmap, how would the migration source process
> know which version it should use when generating the savevm data?
>   

To properly support this in -M pc-0.11, we'll need to be able to set the 
version to migrate for each qdev device.  Again, this is something that 
could be overridden as a qdev property.  The effect would be that we 
force a newer qemu to generate the older savevm format.

> (considering that the migration stream is unidirectional, today) We have
> been considering using a "set-savevm-version" monitor command that would
> be used by management if backward migration is forced by the user.
>   

qdev property is the right approach I think.  It's really a per-device 
setting.  It needs to get tied to machine type too and that's a 
convenient way to do that.

> BTW, we still have the "machine type" suggestion, that would still keep
> guest-visible state correctness and allow backward migration when it is
> 100% correct and safe. With such mechanism, VMs created with the x.y.1
> machine type could be safely migrated from x.y.2 to x.y.1. (Althought
> the bitmap suggestion could have some use even on this case, if the user
> really wants to force migration of a x.y.2 machine to x.y.1).
>   

In theory, a user can manually specify everything in a machine type.

>> This takes qemu out of the business of creating these sort of policies  
>> but allows RHEL to make decisions about what default policy it uses.  It  
>> also lets well informed users of RHEL to override those policy decisions  
>> when they deem it to be appropriate.
>>
>> This would make me happy both from an upstream qemu perspective but also  
>> as a consumer of RHEL.
>>     
>
> What about the suggestion of using multiple sections per device, every
> time a new feature is added, instead of just increasing the version
> numbers linearly? It allows us to keep the savevm version info
> consistent on the multiple downstream trees.
>   

It doesn't because it's just as likely to get clashes in subsection 
names.  For instance, RHEL5.4 may call the pvclock msr subsection 
"pvclock-msrs" and then upstream may call it "pvclock-msrs" and flip the 
order of the fields.

To support downstreams effectively, we need vendor specific versioning 
so that we can separate the upstream qemu namespace from each of the 
downstreams.

> Suppose we have the following scenario:
>
> 1) Device Foo has features A, B, C on "foo" section, sets version to 1
> 2) Downstream tree (e.g. RHEL) is branched off upstream
> 3) Device Foo adds support to feature D, version change to 2
> 5) Device Foo adds support to feature E, version changed to 3
> 6) Feature E is backported to a downstream tree. Now it supports
>    features A,B,C,E, and its versioning scheme will be incompatible with
>    upstream.
>   

Downstream adds a "RHEL" subversion.  This allows downstream to add a 
subversion to each device if it modifies it.  When it backports E, it 
bumps the downstream version from 0->1.

As long as the backported features aren't enabled, the migration will be 
compatible to upstream.  Once one of these backported features is 
enabled, migration will fail gracefully.

> What I suggest is something like:
>
> 1) Device Foo has features A,B,C, on "foo" section (or maybe on "foo.a",
>    "foo.b", and "foo.c" sections, depending if they make sense
>    individually)
> 2) Device Foo adds support to feature D, adds "foo.d" section
> 3) Device Foo adds support to feature E, adds "foo.e" section
>   

The combinations blow up quickly.  Just because A,B,C,E works for a 
given downstream, doesn't mean that it would work with the upstream code 
base.  Features are rarely so independent of one another.

It also doesn't address things like QXL which aren't just a simple 
matter of a backported upstream feature.

Regards,

Anthony Liguori