Re: [Qemu-devel] [PATCH v2 43/43] Start documenting how postcopy works.

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Eric Blake (eblake@redhat.com) wrote:
> On 08/11/2014 08:29 AM, Dr. David Alan Gilbert (git) wrote:
> > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > 
> > Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > ---
> >  docs/migration.txt | 150 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 150 insertions(+)
> 
> I personally like docs early in the series (see if they make sense in
> isolation, then whether the implementation matched the docs; rather than
> being tainted by implementation and overlooking design flaws).

No problem, I can flip that around.

> > diff --git a/docs/migration.txt b/docs/migration.txt
> > index 0492a45..fec2d46 100644
> > --- a/docs/migration.txt
> > +++ b/docs/migration.txt
> > @@ -294,3 +294,153 @@ save/send this state when we are in the middle of a pio operation
> >  (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
> >  not enabled, the values on that fields are garbage and don't need to
> >  be sent.
> > +
> > += Return path =
> > +
> > +In most migration scenarios there is only a single data path that runs
> > +from the source VM to the destination, typically along a single fd (although
> > +possibly with another fd or similar for some fast way of throwing pages across).
> > +
> > +However, some uses need two way comms; in particular the Postcopy destination
> 
> s/comms/communication/

Thanks, all typos/etc applied.

> > +needs to be able to request pages on demand from the source.
> > +
> > +For these scenarios there is a 'return path' from the destination to the source;
> > +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
> > +path.
> > +
> > +  Source side
> > +     Forward path - written by migration thread
> > +     Return path  - opened by main thread, read by fd_handler on main thread
> > +
> > +  Destination side
> > +     Forward path - read by main thread
> > +     Return path  - opened by main thread, written by main thread AND postcopy
> > +                    thread (protected by rp_mutex)
> > +
> > +Opening the return path generally sets the fd to be non-blocking so that a
> > +failed destination can't block the source; and since the non-blockingness seems
> > +to follow both directions it does alter the semantics of the forward path.
> 
> Once you've opened a return path, haven't you already committed to the
> migration?  That is, since we already document that postcopy loses the
> VM if something goes wrong, what point is there in trying to protect
> from a failed destination?  Once the source knows that the destination
> has opened the return path, the source should never run the VM again.

The point at which the source can't run again is quite late; plenty of things
can fail before that point and be recovered from; as long as the destination
hasn't started its CPU (and hence started chatting on the network or
changing shared storage) recovery is possible in this implementation.

Opening the return path is done early on in migration if the postcopy capability is on,
before the migrate_start_postcopy is issued.  The entire migration might be
completed in precopy with the return path opened but unused.

(The return-path blocking-ness is up for discussion anyway with the changes that
Paolo suggested in the 1st series that I've not done yet).

> > +
> > += Postcopy =
> > +'Postcopy' migration is a way to deal with migrations that refuse to converge;
> > +it's plus side is that there is an upper bound on the amount of migration traffic
> 
> s/it's/its/
> 
> > +and time it takes, the down side is that during the postcopy phase, a failure of
> > +*either* side or the network connection causes the guest to be lost.
> > +
> ...
> 
> > +=== Postcopy states ===
> > +Postcopy moves through a series of states (see postcopy_ram_state)
> > +from ADVISE->LISTEN->RUNNING->END
> > +
> > +  Advise: Set at the start of migration if postcopy is enabled, even
> > +          if it hasn't passed the start-time threshold; here the destination
> > +          checks it's OS has the support needed for postcopy, and performs
> 
> s/it's/that its/
> 
> > +          setup to ensure the RAM mappings are suitable for later postcopy.
> > +          (Triggered by reception of POSTCOPY_RAM_ADVISE command)
> > +
> > +Normal precopy now carries on as normal, until the point that the source
> > +hits the start-time threshold and transitions to postcopy.  The source
> > +stops it's CPUs and transmits a 'discard bitmap' indicating pages that
> 
> s/it's/its/
> 
> (now that you've made the same mistake multiple times: remember, "it's"
> is correct ONLY if you can say "it is" (or "it has") in the same
> context; possession uses "its").
> 
> > +have been previously sent but are now dirty again and hence are out of
> > +date on the destination.
> > +
> > +The migration stream now contains a 'package' containing it's own chunk
> 
> s/it's/its/
> 
> > +of migration stream, followed by a return to a normal stream containing
> > +page data.  The package (sent as CMD_PACKAGED) contains the commands to
> > +cycle the states on the destination, followed by all of the device
> > +state excluding RAM.  This lets the destination request pages from the
> > +source in parallel with loading device state, this is required since
> > +some devices (virtio) access guest memory during device initialisation.
> 
> I don't know if we have a strong preference for US (initialization) or
> UK (initialisation) spelling in our docs.

grep suggests US.

> ...
> > +
> > +Note that once in postcopy mode, the sent map is still updated, however it's
> 
> s/it's/its/
> 
> > +contents are not-consistent as a local view of what's been sent since it's
> 
> This use of "it's" is correct, but sounds awkward.
> 
> > +only got the masked result.
> 
> Maybe:
> 
> Note that once in postcopy mode, the sent map is still updated; however,
> its contents are not necessarily consistent with the pages already sent
> due to the masking with the migration bitmap.

Thanks.

Dave
> -- 
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
> 


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK