Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Markus Armbruster <armbru@redhat.com>]

Dongli Zhang <dongli.zhang@oracle.com> writes:

> On Sun, Aug 23, 2026 11:29:53PM -0700, Markus Armbruster wrote:
> Dongli Zhang <dongli.zhang@oracle.com> writes:
>> 
>>> On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>>> 
>>>>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>
> [snip]
>
>>>> message.
>>>
>>> Sure.
>>>
>>>> 
>>>> We should also help users decide whether to use the feature.  I gather
>>>> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
>>>> even work with other migration modes?  If yes, which ones, and what
>>>> happens when you try with another one?
>>>
>>> So far I only added support for cpr-exec. I have not tried othe modes.
>> 
>> What happens when you try to use it with other migration modes?
>
> If I use memfd transport without cpr-exec mode, it will be denied.
>
> The code checks that only cpr-exec mode is supported.
>
> (qemu) migrate -d memfd:
> Error: memfd transport is only supported with cpr-exec

Good.

>>> In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
>>> sure it is necessary.
>>>
>>> For cpr-transfer, the source and destination are separate QEMU processes. If we
>>> want to pass a memfd between them, we still need an IPC mechanism that can
>>> transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
>>> reduces the value of using memfd as the migration channel, because we need at
>>> least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.
>>>
>>> If a Unix domain socket is already required, the main benefit of memfd would
>>> likely be performance, especially when the device state is large.
>> 
>> We'd want to measure the performance impact to see whether it's worth
>> complicating configuration and code.
>
> I may create a VM with many virtio devices to maximize the storage overhead of
> the VM device state. However, I don't think the goal is to improve performance.

As long as that's the case, enabling memfd for cpr-transfer makes no
sense.

>>> cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
>>> be preserved across exec by clearing CLOEXEC and passing the fd number through
>>> an environment variable. This allows us to add a memfd-backed migration channel
>>> without needing an external file or socket for the device state.
>> 
>> What are the actual user-visible benefits?
>
> The goal is to avoid the VM state file used during live update with cpr-exec.
> So, the user-visible benefit is that the file is no longer required. Live update
> with cpr-exec no longer relies on a file on disk.

Got it.

>> Performance?  Provide measurements, please.
>
> Not performance.

If we accept memfd for its other benefits, performance measurements
might be interesting.  Not now.

>> Simpler configuration?  Not sure that would be worth the special case.
>
> Yes. No file required.
>
> "migrate -d memfd:" vs. "migrate -d file:vm.state".
>
> "migrate_incoming memfd:" vs. "migrate_incoming file:vm.state".

Got it.

>> Anything else?
>
> No. Writable storage is no longer required to cache the VM device state file.
> That's the only explicit benefit so far.

Benefits of the new MigrationAddressType @memfd: we gain a zero-config
migration channel for MigMode cpr-exec.

Costs: we gain yet another kind of migration channel, useful only with
cpr-exec.  Feels niche.

Are the benefits worth the costs?  Migration maintainers decide, not me.

In my (relatively ignorant) opinion, migration already suffers from too
many features.  Hard to understand, painful test matrix, expensive to
maintain.

>>> One potential future use case is LUO/KHO support for QEMU. In that case, device
>>> state could be preserved in memory as a memfd and potentially survive across a
>>> KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
>>> internal tests using memfd to preserve device state across kexec although using
>>> different implementation, and it worked.
>>>
>>> For other live migration modes, or even local migration, I do not think this is
>>> useful. For example, saving to and loading from a file won't use memfd, as the
>>> QEMU processes are terminated during the this process.
>> 
>> User documentation should spell out when memfd can be used.

It does: "This transport is currently only valid with cpr-exec migration
mode."

>>                                                              I recommend
>> to make it usable only when it's known to improve things in at least
>> some scenario that matters.  Code should cleanly reject attempts to use
>> memfd when it's not usable.

You showed above that it does.

>> User documentation should help users decide whether to use it.

It does not so far.

>> The commit message(s) may need to provide additional detail on how
>> things improve.
>> 
> Thank you very much for feedback!
>
> Dongli Zhang

You're welcome!


Tangent, pardon my ignorance...  We have MigMode @cpr-exec and we have
MigrationAddressType @exec.  Both involve starting a new instance of
QEMU.  How do their use cases differ?