Re: [Qemu-devel] [PATCH RFC 1/8] virtio: add subsections to the migration stream

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, May 15, 2014 at 02:00:15PM +0200, Andreas Färber wrote:
> Am 15.05.2014 12:16, schrieb Michael S. Tsirkin:
> > On Thu, May 15, 2014 at 12:11:12PM +0200, Andreas Färber wrote:
> >> Am 15.05.2014 12:03, schrieb Michael S. Tsirkin:
> >>> On Thu, May 15, 2014 at 11:58:25AM +0200, Andreas Färber wrote:
> >>>> Am 15.05.2014 11:52, schrieb Michael S. Tsirkin:
> >>>>> On Thu, May 15, 2014 at 11:20:18AM +0200, Andreas Färber wrote:
> >>>>>> Am 15.05.2014 09:04, schrieb Greg Kurz:
> >>>>>>> On Thu, 15 May 2014 12:16:35 +0530
> >>>>>>> Amit Shah <amit.shah@redhat.com> wrote:
> >>>>>>>> On (Thu) 15 May 2014 [09:23:51], Michael S. Tsirkin wrote:
> >>>>>>>>> On Thu, May 15, 2014 at 11:34:25AM +0530, Amit Shah wrote:
> >>>>>>>>>> On (Wed) 14 May 2014 [17:41:38], Greg Kurz wrote:
> >>>>>>>>>>> Since each virtio device is streamed in its own section, the idea is to
> >>>>>>>>>>> stream subsections between the end of the device section and the start
> >>>>>>>>>>> of the next sections. This allows an older QEMU to complain and exit
> >>>>>>>>>>> when fed with subsections:
> >>>>>>>>>>>
> >>>>>>>>>>> Unknown savevm section type 5
> >>>>>>>>>>> Error -22 while loading VM state
> >>>>>>>>>>
> >>>>>>>>>> Please make this configurable -- either via configure or device
> >>>>>>>>>> properties.  That avoids having to break existing configurations that
> >>>>>>>>>> work without this patch.
> >>>>>>
> >>>>>> Since backwards migration is not supported upstream, wouldn't it be
> >>>>>> easiest to just add support for the subsection marker and skipping to
> >>>>>> the end of section in that downstream?
> >>>>>
> >>>>> Backwards and forwards migration need to be supported,
> >>>>> customers told us repeatedly. So some downstreams support this
> >>>>> and not supporting it upstream just means downstreams need
> >>>>> to do their own thing.
> >>>>>
> >>>>> As importantly, ping-pong migration is the only
> >>>>> reliable way to stress migration.
> >>>>>
> >>>>> So if we want to test cross-version we need it to work
> >>>>> both way.
> >>>>>
> >>>>> Finally, the real issue and difficulty with cross-version migration is
> >>>>> making VM behave in a backwards compatible way.  Serializing in a
> >>>>> compatible way is a trivial problem, or would be if the code wasn't a
> >>>>> mess :) Once you do the hard part, breaking migration because of the
> >>>>> trivial serialization issue is just silly.  And special-casing forward
> >>>>> migration does not make code simpler, it really only leads to
> >>>>> proliferation of hacks and lack of symmetry.
> >>>>>
> >>>>> So yes it's a useful feature, and no not supporting it does
> >>>>> not help anyway.
> >>>>
> >>>> It seems you misunderstand. I was not saying it's not useful.
> >>>>
> >>>> My point is that VMStateSubsections added in newer versions (and thus
> >>>> not existing in older versions) need to be handled for any
> >>>> VMState-converted devices. So why can't we make that work for virtio too?
> >>>
> >>> Sure.
> >>> AFAIK the way to this works is by adding an "field_exists" callback,
> >>> and not sending the section when we are in a compat mode.
> >>
> >> OK, but upstream always sends the latest version today.
> >> So isn't that
> >> just two ifs that you would need to add in save and load functions added
> >> here for the downstream? x86_64 is unaffected from ppc's endianness
> >> changes and you still do ppc64 BE, so there's no real functional problem
> >> for RHEL, is there?
> > 
> > I'm sorry I don't understand what you are saying here.
> > Simply put, if you specify a compatible -M then your
> > device should behave, and migrate, exactly like and old
> > qemu did.
> 
> Whatever the version_id fields are for, upstream QEMU *always* saves the
> newest version_id format that it knows. There is no mechanism that I'm
> aware of in upstream QEMU for migrating device fields dependent on -M.
> So a device in QEMU only migrates exactly like an old QEMU did if
> neither fields nor subsections were added.

Ah you are talking about version ids.
Yes, they can not support cross-version migration.
For this reason these should only be incremented as the last resort,
normally new fields (possibly subsections) should be added,
with field_exists callback checking for compatibility mode.


> Subsections are usually migrated based on whether the state differs from
> the default state (didn't check whether the final patch does this
> correctly here, but doesn't matter for 1/8 concept).

Yes, this would be fine, but it's not what's implemented IIUC.

> So for x86 the
> subsection should *never* get written and thus not be a problem for you.
> For ppc64 it should not get written either, unless you care about
> migration from SLES12 or upstream ppc64le to RHEL ppc64.
> As I understood the IRC discussion Alex and me had with Greg about this,
> this is copying the exact code VMState uses to write its subsections, so
> there would be no forward migration problems at all once we convert to
> VMState and VMStateSubsections. The only question remaining is how does
> your downstream react when it encounters a subsection marker for
> subsections it doesn't know - which is not something upstream can solve
> for you unless you contribute backwards migration support upstream.
> 
> Don't forget that Greg is introducing subsection support *because of*
> your backwards migration wish, so telling him not to add subsections
> because of backwards migration is really weird!

I don't really care whether it's a subsection or just plain
qemu_put_byte, but I'm not saying not to use subsection
either.
I am merely saying this should have "exists" callback checking
for compability mode for old machine types.

> Either subsections work
> with your downstream or they don't; if they don't, then you have much
> larger problems than can be solved by blocking this one section for ppc.
> Therefore I was saying if your downstream forgot to handle the case of a
> non-VMState device having subsections, then it may be easier to fix
> exactly that than make Greg jump through more hoops here for a
> theoretical problem you are unlikely to encounter on your downstream.
> 
> Andreas

I only mentioned downstream because it's a requirement coming from
users most of whom never interact with upstream.
There shouldn't be any difference and people building clusters
using upstream qemu will have same needs: clusters can't be
completely brought down for version upgrades, so need to be able to run
a mix of versions, and be able to migrate in any direction
while in the process.

> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg