Re: [PATCH 4/4] migration: Introduce POSTCOPY_DEVICE state

[Peter Xu <peterx@redhat.com>]

On Thu, Sep 25, 2025 at 01:54:40PM +0200, Jiří Denemark wrote:
> On Mon, Sep 15, 2025 at 13:59:15 +0200, Juraj Marcin wrote:
> > From: Juraj Marcin <jmarcin@redhat.com>
> > 
> > Currently, when postcopy starts, the source VM starts switchover and
> > sends a package containing the state of all non-postcopiable devices.
> > When the destination loads this package, the switchover is complete and
> > the destination VM starts. However, if the device state load fails or
> > the destination side crashes, the source side is already in
> > POSTCOPY_ACTIVE state and cannot be recovered, even when it has the most
> > up-to-date machine state as the destination has not yet started.
> > 
> > This patch introduces a new POSTCOPY_DEVICE state which is active
> > while the destination machine is loading the device state, is not yet
> > running, and the source side can be resumed in case of a migration
> > failure.
> > 
> > To transition from POSTCOPY_DEVICE to POSTCOPY_ACTIVE, the source
> > side uses a PONG message that is a response to a PING message processed
> > just before the POSTCOPY_RUN command that starts the destination VM.
> > Thus, this change does not require any changes on the destination side
> > and is effective even with older destination versions.
> 
> Thanks, this will help libvirt as we think that the migration can be
> safely aborted unless we successfully called "cont" and thus we just
> kill QEMU on the destination. But since QEMU on the source already
> entered postcopy-active, we can't cancel the migration and the result is
> a paused VM with no way of recovering it.
> 
> This series will make the situation better as the source will stay in
> postcopy-device until the destination successfully loads device data.
> There's still room for some enhancement though. Depending on how fast
> this loading is libvirt may issue cont before device data is loaded (the
> destination is already in postcopy-active at this point), which always
> succeeds as it only marks the domain to be autostarted, but the actual
> start may fail later. When discussing this with Juraj we agreed on
> introducing the new postcopy-device state on the destination as well to

I used to think and define postcopy-active be the state we should never be
able to cancel it anymore, implying that the real postcopy process is in
progress, and also implying the state where we need to start assume the
latest VM pages are spread on both sides, not one anymore.  Cancellation or
killing either side means crashing VM then.

It could be a good thing indeed to have postcopy-device on dest too from
that regard, because having postcopy-device on dest can mark out the small
initial window when dest qemu hasn't yet start to generate new data but
only applying old data (device data, which src also owns a copy).  From
that POV, that indeed does not belong to the point if we define
postcopy-active as above.

IOW, also with such definition, setting postcopy-active on dest QEMU right
at the entry of ram load thread (what we do right now..) is too early.

> make sure libvirt will only call cont once device data was successfully
> loaded so that we always get a proper result when running cont. But it

Do we know an estimate of how much extra downtime this would introduce?

We should have discussed this in a previous thread, the issue is if we cont
only after device loaded, then dest QEMU may need to wait a while until it
receives the cont from libvirt, that will contribute to the downtime.  It
would best be avoided, or if it's negligible then it's fine too but I'm not
sure whether it's guaranteed to be negligible..

If the goal is to make sure libvirt knows what is happening, can it still
relies on the event emitted, in this case, RESUME?  We can also reorg how
postcopy-device and postcopy-active states will be reported on dest, then
they'll help if RESUME is too coarse grained.

So far, dest QEMU will try to resume the VM after getting RUN command, that
is what loadvm_postcopy_handle_run_bh() does, and it will (when autostart=1
set): (1) firstly try to activate all block devices, iff it succeeded, (2)
do vm_start(), at the end of which RESUME event will be generated.  So
RESUME currently implies both disk activation success, and vm start worked.

> may still fail when locking disks fails (not sure if this is the only
> way cont may fail). In this case we cannot cancel the migration on the

Is there any known issue with locking disks that dest would fail?  This
really sound like we should have the admin taking a look.

I recall Juraj mentioned off list that drive lock sometimes will stop
working.  Is that relevant here?

> source as it is already in postcopy-active and we can't recover
> migration either as the CPUs are not running on the destination. Ideally
> we'd have a way of canceling the migration in postocpy-active if we are
> sure CPUs were not started yet. Alternatively a possibility to recover
> migration would work as well.

Thanks,

-- 
Peter Xu