Re: [PATCH v4 07/14] migration: Make switchover-ack re-usable

[Peter Xu <peterx@redhat.com>]

On Mon, Jun 29, 2026 at 06:53:48PM +0300, Avihai Horon wrote:
> 
> On 6/29/2026 5:57 PM, Cédric Le Goater wrote:
> > External email: Use caution opening links or attachments
> > 
> > 
> > Avihai
> > 
> > On 6/28/26 18:29, Cédric Le Goater wrote:
> > > On 6/28/26 12:03, Avihai Horon wrote:
> > > > 
> > > > On 6/28/2026 12:00 PM, Cédric Le Goater wrote:
> > > > > External email: Use caution opening links or attachments
> > > > > 
> > > > > 
> > > > > On 6/28/26 10:16, Avihai Horon wrote:
> > > > > > 
> > > > > > On 6/26/2026 10:54 AM, Markus Armbruster wrote:
> > > > > > > External email: Use caution opening links or attachments
> > > > > > > 
> > > > > > > 
> > > > > > > Just spelling nitpickery this time.
> > > > > > > 
> > > > > > > Avihai Horon <avihaih@nvidia.com> writes:
> > > > > > > 
> > > > > > > > Switchover-ack is a mechanism to synchronize between source and
> > > > > > > > destination QEMU during migration to prevent the
> > > > > > > > source from switching
> > > > > > > > over prematurely.
> > > > > > > > 
> > > > > > > > VFIO uses switchover-ack to ensure switchover happens only after
> > > > > > > > destination side has loaded the precopy initial
> > > > > > > > bytes. This is important
> > > > > > > > for VFIO, as otherwise downtime could be impacted and be higher.
> > > > > > > > 
> > > > > > > > In its current state, switchover-ack is a
> > > > > > > > one-time mechanism, meaning
> > > > > > > > that switchover is acked only once and past that
> > > > > > > > another ACK cannot be
> > > > > > > > requested again. This was sufficient until now,
> > > > > > > > as VFIO precopy initial
> > > > > > > > bytes was defined to be monotonically
> > > > > > > > decreasing. Thus, when precopy
> > > > > > > > initial bytes reached zero for all VFIO devices,
> > > > > > > > a single ACK would be
> > > > > > > > sent and its validity would hold.
> > > > > > > > 
> > > > > > > > However, now the new VFIO_PRECOPY_INFO_REINIT
> > > > > > > > feature allows precopy
> > > > > > > > initial bytes to be re-initialized during precopy. Specifically, it
> > > > > > > > means that initial bytes can grow after reaching zero, which would
> > > > > > > > invalidate a previously sent switchover ACK.
> > > > > > > > 
> > > > > > > > To solve this, make switchover-ack reusable and
> > > > > > > > allow devices to request
> > > > > > > > switchover ACKs when needed via the
> > > > > > > > save_query_pending SaveVMHandler.
> > > > > > > > 
> > > > > > > > Since now switchover ACK can be requested for a
> > > > > > > > specific device and in
> > > > > > > > different times, make switchover ACK per-device
> > > > > > > > (instead of a single ACK
> > > > > > > > for all devices) and let source side do the
> > > > > > > > pending ACKs accounting.
> > > > > > > > 
> > > > > > > > Keep the legacy switchover-ack mechanism for
> > > > > > > > backward compatibility and
> > > > > > > > turn it on by a compatibility property for older
> > > > > > > > machines. Enable the
> > > > > > > > property until VFIO implements the new switchover-ack.
> > > > > > > > 
> > > > > > > > Acked-by: Markus Armbruster <armbru@redhat.com>
> > > > > > > > Reviewed-by: Peter Xu <peterx@redhat.com>
> > > > > > > > Signed-off-by: Avihai Horon <avihaih@nvidia.com>
> > > > > > > > ---
> > > > > > > >   qapi/migration.json                | 14 ++++----
> > > > > > > >   include/migration/client-options.h |  1 +
> > > > > > > >   include/migration/register.h       |  2 ++
> > > > > > > >   migration/migration.h              | 32 ++++++++++++++++--
> > > > > > > >   migration/savevm.h                 |  6 ++--
> > > > > > > >   migration/migration.c              | 37 ++++++++++++++-------
> > > > > > > >   migration/options.c                | 10 ++++++
> > > > > > > >   migration/savevm.c                 | 53
> > > > > > > > +++++++++++++++++++++++-------
> > > > > > > >   migration/trace-events             |  5 +--
> > > > > > > >   9 files changed, 123 insertions(+), 37 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git a/qapi/migration.json b/qapi/migration.json
> > > > > > > > index 27a7970556..9b3070e494 100644
> > > > > > > > --- a/qapi/migration.json
> > > > > > > > +++ b/qapi/migration.json
> > > > > > > > @@ -508,14 +508,12 @@
> > > > > > > >   #     (since 7.1)
> > > > > > > >   #
> > > > > > > >   # @switchover-ack: If enabled, migration will
> > > > > > > > not stop the source VM
> > > > > > > > -#     and complete the migration until an ACK is received from the
> > > > > > > > -#     destination that it's OK to do so.  Exactly when this ACK is
> > > > > > > > -#     sent depends on the migrated devices that
> > > > > > > > use this feature. For
> > > > > > > > -#     example, a device can use it to make sure
> > > > > > > > some of its data is
> > > > > > > > -#     sent and loaded in the destination before doing switchover.
> > > > > > > Here, "switchover" is used as noun.  My dictionary
> > > > > > > is happy with that.
> > > > > > > 
> > > > > > > > -#     This can reduce downtime if devices that
> > > > > > > > support this capability
> > > > > > > > -#     are present.  'return-path' capability
> > > > > > > > must be enabled to use
> > > > > > > > -#     it.  (since 8.1)
> > > > > > > > +#     and complete the migration until the destination has
> > > > > > > > +#     acknowledged that it is OK to
> > > > > > > > switchover.  The acknowledgement
> > > > > > > 
> > > > > > > Here, it's used as verb.  I think it needs to be spelled "to switch
> > > > > > > over".
> > > > > > > 
> > > > > > > Same for code comments etc.
> > > > > > 
> > > > > > Sure, I can send a patch to fix those.
> > > > > 
> > > > > Can you please send the update as a follow up patch ?
> > > > 
> > > > I was planning to send it along with the bool return value
> > > > refactor in patch #1. Would that be fine?
> > > 
> > > OK. Nothing merged yet. Let's have a v5.
> > 
> > FYI, I plan to send vfio-next before the end the week. Initial plan was
> > tomorrow.
> 
> Hi Cedric,
> 
> Actually, I have plenty of things on my plate right now and I'm not sure if
> I'll be able to send it this week.
> 
> Would it be OK if you merge the series as is and I send those two patches
> separately early next week?
> (They are actually more related to migration core, so this question should
> be addressed to Peter/Fabiano I guess...)

I'm OK either way.

The simplest to me is you can provide a small fixup for Cédric to pick up
and squash into this patch (if it's only wording changes to doc and
comments, IMHO no re-test needed), as long as Cédric is comfortable with
it, then you can send a separate patch (or patches) if that'll be on
migration.

Thanks,

-- 
Peter Xu