Re: [PATCH v4 07/14] migration: Make switchover-ack re-usable

[Avihai Horon <avihaih@nvidia.com>]

On 6/29/2026 5:57 PM, Cédric Le Goater wrote:
> External email: Use caution opening links or attachments
>
>
> Avihai
>
> On 6/28/26 18:29, Cédric Le Goater wrote:
>> On 6/28/26 12:03, Avihai Horon wrote:
>>>
>>> On 6/28/2026 12:00 PM, Cédric Le Goater wrote:
>>>> External email: Use caution opening links or attachments
>>>>
>>>>
>>>> On 6/28/26 10:16, Avihai Horon wrote:
>>>>>
>>>>> On 6/26/2026 10:54 AM, Markus Armbruster wrote:
>>>>>> External email: Use caution opening links or attachments
>>>>>>
>>>>>>
>>>>>> Just spelling nitpickery this time.
>>>>>>
>>>>>> Avihai Horon <avihaih@nvidia.com> writes:
>>>>>>
>>>>>>> Switchover-ack is a mechanism to synchronize between source and
>>>>>>> destination QEMU during migration to prevent the source from 
>>>>>>> switching
>>>>>>> over prematurely.
>>>>>>>
>>>>>>> VFIO uses switchover-ack to ensure switchover happens only after
>>>>>>> destination side has loaded the precopy initial bytes. This is 
>>>>>>> important
>>>>>>> for VFIO, as otherwise downtime could be impacted and be higher.
>>>>>>>
>>>>>>> In its current state, switchover-ack is a one-time mechanism, 
>>>>>>> meaning
>>>>>>> that switchover is acked only once and past that another ACK 
>>>>>>> cannot be
>>>>>>> requested again. This was sufficient until now, as VFIO precopy 
>>>>>>> initial
>>>>>>> bytes was defined to be monotonically decreasing. Thus, when 
>>>>>>> precopy
>>>>>>> initial bytes reached zero for all VFIO devices, a single ACK 
>>>>>>> would be
>>>>>>> sent and its validity would hold.
>>>>>>>
>>>>>>> However, now the new VFIO_PRECOPY_INFO_REINIT feature allows 
>>>>>>> precopy
>>>>>>> initial bytes to be re-initialized during precopy. Specifically, it
>>>>>>> means that initial bytes can grow after reaching zero, which would
>>>>>>> invalidate a previously sent switchover ACK.
>>>>>>>
>>>>>>> To solve this, make switchover-ack reusable and allow devices to 
>>>>>>> request
>>>>>>> switchover ACKs when needed via the save_query_pending 
>>>>>>> SaveVMHandler.
>>>>>>>
>>>>>>> Since now switchover ACK can be requested for a specific device 
>>>>>>> and in
>>>>>>> different times, make switchover ACK per-device (instead of a 
>>>>>>> single ACK
>>>>>>> for all devices) and let source side do the pending ACKs 
>>>>>>> accounting.
>>>>>>>
>>>>>>> Keep the legacy switchover-ack mechanism for backward 
>>>>>>> compatibility and
>>>>>>> turn it on by a compatibility property for older machines. 
>>>>>>> Enable the
>>>>>>> property until VFIO implements the new switchover-ack.
>>>>>>>
>>>>>>> Acked-by: Markus Armbruster <armbru@redhat.com>
>>>>>>> Reviewed-by: Peter Xu <peterx@redhat.com>
>>>>>>> Signed-off-by: Avihai Horon <avihaih@nvidia.com>
>>>>>>> ---
>>>>>>>   qapi/migration.json                | 14 ++++----
>>>>>>>   include/migration/client-options.h |  1 +
>>>>>>>   include/migration/register.h       |  2 ++
>>>>>>>   migration/migration.h              | 32 ++++++++++++++++--
>>>>>>>   migration/savevm.h                 |  6 ++--
>>>>>>>   migration/migration.c              | 37 ++++++++++++++-------
>>>>>>>   migration/options.c                | 10 ++++++
>>>>>>>   migration/savevm.c                 | 53 
>>>>>>> +++++++++++++++++++++++-------
>>>>>>>   migration/trace-events             |  5 +--
>>>>>>>   9 files changed, 123 insertions(+), 37 deletions(-)
>>>>>>>
>>>>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>>>>> index 27a7970556..9b3070e494 100644
>>>>>>> --- a/qapi/migration.json
>>>>>>> +++ b/qapi/migration.json
>>>>>>> @@ -508,14 +508,12 @@
>>>>>>>   #     (since 7.1)
>>>>>>>   #
>>>>>>>   # @switchover-ack: If enabled, migration will not stop the 
>>>>>>> source VM
>>>>>>> -#     and complete the migration until an ACK is received from the
>>>>>>> -#     destination that it's OK to do so.  Exactly when this ACK is
>>>>>>> -#     sent depends on the migrated devices that use this 
>>>>>>> feature. For
>>>>>>> -#     example, a device can use it to make sure some of its 
>>>>>>> data is
>>>>>>> -#     sent and loaded in the destination before doing switchover.
>>>>>> Here, "switchover" is used as noun.  My dictionary is happy with 
>>>>>> that.
>>>>>>
>>>>>>> -#     This can reduce downtime if devices that support this 
>>>>>>> capability
>>>>>>> -#     are present.  'return-path' capability must be enabled to 
>>>>>>> use
>>>>>>> -#     it.  (since 8.1)
>>>>>>> +#     and complete the migration until the destination has
>>>>>>> +#     acknowledged that it is OK to switchover.  The 
>>>>>>> acknowledgement
>>>>>>
>>>>>> Here, it's used as verb.  I think it needs to be spelled "to switch
>>>>>> over".
>>>>>>
>>>>>> Same for code comments etc.
>>>>>
>>>>> Sure, I can send a patch to fix those.
>>>>
>>>> Can you please send the update as a follow up patch ?
>>>
>>> I was planning to send it along with the bool return value refactor 
>>> in patch #1. Would that be fine?
>>
>> OK. Nothing merged yet. Let's have a v5.
>
> FYI, I plan to send vfio-next before the end the week. Initial plan was
> tomorrow.

Hi Cedric,

Actually, I have plenty of things on my plate right now and I'm not sure 
if I'll be able to send it this week.

Would it be OK if you merge the series as is and I send those two 
patches separately early next week?
(They are actually more related to migration core, so this question 
should be addressed to Peter/Fabiano I guess...)

Thanks.