Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Dongli Zhang <dongli.zhang@oracle.com>]

On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
> Dongli Zhang <dongli.zhang@oracle.com> writes:
> 
>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>> 
>>>> Add a memfd migration transport to the MigrationAddress QAPI schema. The
>>>> new transport has no user-supplied arguments because QEMU creates and
>>>> passes the backing memfd internally.
>>>>
>>>> The initial user is cpr-exec, where the main VM/device migration stream can
>>>> be stored in an inherited memfd instead of an external file.
>>>>
>>>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>>>> ---
>>>>  qapi/migration.json | 32 +++++++++++++++++++++++++++-----
>>>>  1 file changed, 27 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>> index b1eaf7b054..1505c201ca 100644
>>>> --- a/qapi/migration.json
>>>> +++ b/qapi/migration.json
>>>> @@ -702,9 +702,11 @@
>>>>  # @cpr-exec: The migrate command stops the VM, saves state to the
>>>>  #     migration channel, directly exec's a new version of QEMU on the
>>>>  #     same host, replacing the original process while retaining its
>>>> -#     PID, and loads state from the channel.  Guest RAM is preserved
>>>> -#     in place.  Devices and their pinned pages are also preserved for
>>>> -#     VFIO and IOMMUFD.
>>>> +#     PID, and loads state from the channel.  With the @memfd
>>>> +#     migration transport, QEMU saves VM/device state to an internal
>>>> +#     memfd instead of an external main migration channel.  Guest RAM
>>>> +#     is preserved in place.  Devices and their pinned pages are also
>>>> +#     preserved for VFIO and IOMMUFD.
>>>>  #
>>>>  #     Old QEMU starts new QEMU by exec'ing the command specified by
>>>>  #     the @cpr-exec-command parameter.  The command may be a direct
>>>> @@ -716,6 +718,13 @@
>>>>  #     as a file, that accepts all data before old QEMU exits.
>>>>  #     Otherwise, old QEMU may quietly block writing to the channel.
>>>>  #
>>>> +#     Alternatively, use the @memfd migration transport to save
>>>> +#     VM/device state to an internal memfd inherited by new QEMU.
>>>> +#
>>> 
>>> Any guidance on when to use @memfd?
>>
>> My objective is to avoid using disks, or running QEMU with writable storage.
>>
>> Both guest I/O and networking can go through VFIO, and everything remains in memory.
>>
>> When I was testing these CPR features, my favorite was cpr-exec because it does
>> not require creating an additional QEMU instance.
>>
>> Based on my understanding, the best approach to live update a running binary is
>> to replace only the code section while keeping everything else in memory, i.e.
>> guest memory, device fds, and device state.
>>
>> Unfortunately, device state cannot simply remain in memory across exec(), so we
>> need to migrate it from the old instance to the new instance. I was thinking:
>> why not use memory instead of storage?
>>
>> Initially, I added the change only for cpr-exec. Later, I reworked the change to
>> make memfd a new migration transport. As a side benefit for me, I also practiced
>> how to add a new transport to the QEMU migration code :)
> 
> This is why you wrote the feature.  Please work it into the commit
> message.

Sure.

> 
> We should also help users decide whether to use the feature.  I gather
> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
> even work with other migration modes?  If yes, which ones, and what
> happens when you try with another one?

So far I only added support for cpr-exec. I have not tried othe modes.

In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
sure it is necessary.

For cpr-transfer, the source and destination are separate QEMU processes. If we
want to pass a memfd between them, we still need an IPC mechanism that can
transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
reduces the value of using memfd as the migration channel, because we need at
least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.

If a Unix domain socket is already required, the main benefit of memfd would
likely be performance, especially when the device state is large.

cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
be preserved across exec by clearing CLOEXEC and passing the fd number through
an environment variable. This allows us to add a memfd-backed migration channel
without needing an external file or socket for the device state.

One potential future use case is LUO/KHO support for QEMU. In that case, device
state could be preserved in memory as a memfd and potentially survive across a
KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
internal tests using memfd to preserve device state across kexec although using
different implementation, and it worked.

For other live migration modes, or even local migration, I do not think this is
useful. For example, saving to and loading from a file won't use memfd, as the
QEMU processes are terminated during the this process.

Thank you very much!

Dongli Zhang

> 
>>>> +#     With @memfd, the new QEMU command may include '-incoming memfd:'
>>>> +#     to load the migration stream automatically, or '-incoming defer'
>>>> +#     to load it later with `migrate-incoming`.
>>>> +#
>>> 
>>> Isn't that the same for any transport?  I.e. old and new QEMU must use a
>>> common transport, and you configure it for the new one with -incoming
>>> right away, or later with migrate-incoming.
>>
>> Yes, it is the same for all transports. I added the explanation only because the
>> memfd transport does not need additional data or a filename, for example, the
>> URI is just "memfd:".
>>
>> Will remove it.
>>
>>> 
>>>>  #     Memory-backend objects must have the share=on attribute, but
>>>>  #     memory-backend-epc is not supported.  The VM must be started
>>>>  #     with the '-machine aux-ram-share=on' option.
> 
> [...]
>