Re: [PATCH v2 06/13] migration: Make switchover-ack re-usable

[Markus Armbruster <armbru@redhat.com>]

Avihai Horon <avihaih@nvidia.com> writes:

> On 6/3/2026 10:17 AM, Markus Armbruster wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> Avihai Horon <avihaih@nvidia.com> writes:
>>
>>> Switchover-ack is a mechanism to synchronize between source and
>>> destination QEMU during migration to prevent the source from switching
>>> over prematurely.
>>>
>>> VFIO uses switchover-ack to ensure switchover happens only after
>>> destination side has loaded the precopy initial bytes. This is important
>>> for VFIO, as otherwise downtime could be impacted and be higher.
>>>
>>> In its current state, switchover-ack is a one-time mechanism, meaning
>>> that switchover is acked only once and past that another ACK cannot be
>>> requested again. This was sufficient until now, as VFIO precopy initial
>>> bytes was defined to be monotonically decreasing. Thus, when precopy
>>> initial bytes reached zero for all VFIO devices, a single ACK would be
>>> sent and its validity would hold.
>>>
>>> However, now the new VFIO_PRECOPY_INFO_REINIT feature allows precopy
>>> initial bytes to be re-initialized during precopy. Specifically, it
>>> means that initial bytes can grow after reaching zero, which would
>>> invalidate a previously sent switchover ACK.
>>>
>>> To solve this, make switchover-ack reusable and allow devices to request
>>> switchover ACKs when needed via the save_query_pending SaveVMHandler.
>>>
>>> Since now switchover ACK can be requested for a specific device and in
>>> different times, make switchover ACK per-device (instead of a single ACK
>>> for all devices) and let source side do the pending ACKs accounting.
>>>
>>> Keep the legacy switchover-ack mechanism for backward compatibility and
>>> turn it on by a compatibility property for older machines. Enable the
>>> property until VFIO implements the new switchover-ack.
>>
>> I figure this is about the transmission of ACKs via the return path.  If
>> both source and destination understand the new per-device ACK, they use
>> it.  Else, they use old one.  Correct?
>
> Correct.
>
>>
>> This is not visible to the management application.  It may use migration
>> capability @switchover-ack to enable just as before.  Correct?
>
> Correct.
>
>>
>> Can you think of a reason why management applications might need to know
>> whether a certain qemu-system-FOO supports per-device ACKs?
>
> Not that I can think of. It should be an implementation detail internal to QEMU.

Thank you.

>>> Signed-off-by: Avihai Horon <avihaih@nvidia.com>
>>> ---
>>>  qapi/migration.json                | 16 ++++-----
>>>  include/migration/client-options.h |  1 +
>>>  include/migration/register.h       |  2 ++
>>>  migration/migration.h              | 32 ++++++++++++++++--
>>>  migration/savevm.h                 |  6 ++--
>>>  hw/core/machine.c                  |  1 +
>>>  migration/migration.c              | 37 ++++++++++++++-------
>>>  migration/options.c                | 10 ++++++
>>>  migration/savevm.c                 | 53 +++++++++++++++++++++++-------
>>>  migration/trace-events             |  5 +--
>>>  10 files changed, 125 insertions(+), 38 deletions(-)
>>>
>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>> index 27a7970556..550cb77762 100644
>>> --- a/qapi/migration.json
>>> +++ b/qapi/migration.json
>>> @@ -507,15 +507,13 @@
>>>  #     and should not affect the correctness of postcopy migration.
>>>  #     (since 7.1)
>>>  #
>>> -# @switchover-ack: If enabled, migration will not stop the source VM
>>> -#     and complete the migration until an ACK is received from the
>>> -#     destination that it's OK to do so.  Exactly when this ACK is
>>> -#     sent depends on the migrated devices that use this feature.  For
>>> -#     example, a device can use it to make sure some of its data is
>>> -#     sent and loaded in the destination before doing switchover.
>>> -#     This can reduce downtime if devices that support this capability
>>> -#     are present.  'return-path' capability must be enabled to use
>>> -#     it.  (since 8.1)
>>> +# @switchover-ack: If enabled, migration will rely on destination side
>>> +#     to acknowledge the source's switchover decision.  The
>>> +#     acknowledgement may depend, for example, on some device's data
>>> +#     being loaded in the destination before doing switchover.  This
>>> +#     can reduce downtime if devices that support this capability are
>>> +#     present.  'return-path' capability must be enabled to use it.
>>
>> Please use the opportunity to fix markup: @return-path.
>> docs/devel/qapi-code-gen.rst:
>>
>>      Use @foo to reference a member description within the current
>>      definition.  This is an rST extension.  It is currently rendered the
>>      same way as ````foo````, but carries additional meaning.
>>
>> Suggest "Capability @return-path must be ..."
>
> Sure, will change it.
>
>>
>> The old text is concrete: "will not stop the source VM ... until".  The
>> new text is vague "will rely on destination side to acknowledge".  What
>> does that mean exactly?  How does it impact behavior the user /
>> management application cares about?
>
> How about the suggestion below? It's both concrete and doesn't mention the number of ACKs (as requested by Peter).
>
> # @switchover-ack: If enabled, migration will not stop the source VM
> #     and complete the migration until the destination has acknowledged
> #     that switchover is safe.  The acknowledgement may depend...
>
> Thanks.

Better!

With these improvements, QAPI schema
Acked-by: Markus Armbruster <armbru@redhat.com>


>>
>>> +#     (since 8.1)
>>>  #
>>>  # @dirty-limit: If enabled, migration will throttle vCPUs as needed to
>>>  #     keep their dirty page rate within @vcpu-dirty-limit.  This can
>> [...]
>>