[PATCH 0/5] Support diskless memfd transport for cpr-exec

[PATCH 0/5] Support diskless memfd transport for cpr-exec

[Dongli Zhang]

Currently, cpr-exec requires a file to save and load VM/device state.

Introduce a diskless approach so cpr-exec no longer relies on disk storage
for this state. The main idea is to add a new memfd-based migration
transport: instead of using a file, VM/device state is saved to and loaded
from an inherited memfd.

I tested this with virtio-blk, NVMe VFIO, and virtio-net.

/home/opc/ext4/qemu/build/qemu-system-x86_64 \
-machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
-accel kvm -cpu host \
-name debug-threads=on \
-m 8G -smp 4 \
-object memory-backend-memfd,id=ram0,size=8G,share=on \
-device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
-drive file=ol89.qcow2,if=none,id=drive0 \
-object iommufd,id=iommufd0 \
-device vfio-pci,host=0000:82:00.0,iommufd=iommufd0,id=vfio0,x-no-mmap=on \
-device virtio-net-pci,netdev=tapnet,mq=true,vectors=9,mac=52:54:11:22:14:49 \
-netdev tap,id=tapnet,script=qemu-ifup,downscript=no,queues=2,vhost=off \
-vnc :8 -monitor stdio

(qemu) migrate_set_parameter mode cpr-exec

(qemu) migrate_set_parameter cpr-exec-command /home/opc/ext4/qemu/build/qemu-system-x86_64 -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on -accel kvm -cpu host -name debug-threads=on -m 8G -smp 4 -object memory-backend-memfd,id=ram0,size=8G,share=on -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 -drive file=ol89.qcow2,if=none,id=drive0 -object iommufd,id=iommufd0 -device vfio-pci,host=0000:82:00.0,iommufd=iommufd0,id=vfio0,x-no-mmap=on -device virtio-net-pci,netdev=tapnet,mq=true,vectors=9,mac=52:54:11:22:14:49 -netdev tap,id=tapnet,script=qemu-ifup,downscript=no,queues=2,vhost=off -vnc :8 -monitor stdio -incoming defer

(qemu) migrate -d memfd:
QEMU 11.1.50 monitor - type 'help' for more information

(qemu) info status
VM status: paused (inmigrate)

(qemu) migrate_incoming memfd:

(qemu) info status
VM status: running


Dongli Zhang (5):
  migration/qapi: Add memfd migration transport
  migration: Add generic memfd channel helpers
  migration/cpr-exec: Add memfd helpers
  migration/cpr-exec: Support diskless memfd transport
  migration: cpr-exec memfd transport documentation

 docs/devel/migration/CPR.rst | 24 +++++++++++
 migration/channel.c          |  7 +++
 migration/cpr-exec-memfd.c   | 90 +++++++++++++++++++++++++++++++++++++++
 migration/cpr-exec-memfd.h   | 18 ++++++++
 migration/cpr-exec.c         |  9 ++++
 migration/memfd.c            | 90 +++++++++++++++++++++++++++++++++++++++
 migration/memfd.h            | 19 +++++++++
 migration/meson.build        |  2 +
 migration/migration.c        |  9 ++++
 qapi/migration.json          | 32 +++++++++++---
 10 files changed, 295 insertions(+), 5 deletions(-)

base-commit: e8d693e12af9cbb89d724baadfcc08559669e279

Thank you very much!

Dongli Zhang

[PATCH 1/5] migration/qapi: Add memfd migration transport

[Dongli Zhang]

Add a memfd migration transport to the MigrationAddress QAPI schema. The
new transport has no user-supplied arguments because QEMU creates and
passes the backing memfd internally.

The initial user is cpr-exec, where the main VM/device migration stream can
be stored in an inherited memfd instead of an external file.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 qapi/migration.json | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)

diff --git a/qapi/migration.json b/qapi/migration.json
index b1eaf7b054..1505c201ca 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -702,9 +702,11 @@
 # @cpr-exec: The migrate command stops the VM, saves state to the
 #     migration channel, directly exec's a new version of QEMU on the
 #     same host, replacing the original process while retaining its
-#     PID, and loads state from the channel.  Guest RAM is preserved
-#     in place.  Devices and their pinned pages are also preserved for
-#     VFIO and IOMMUFD.
+#     PID, and loads state from the channel.  With the @memfd
+#     migration transport, QEMU saves VM/device state to an internal
+#     memfd instead of an external main migration channel.  Guest RAM
+#     is preserved in place.  Devices and their pinned pages are also
+#     preserved for VFIO and IOMMUFD.
 #
 #     Old QEMU starts new QEMU by exec'ing the command specified by
 #     the @cpr-exec-command parameter.  The command may be a direct
@@ -716,6 +718,13 @@
 #     as a file, that accepts all data before old QEMU exits.
 #     Otherwise, old QEMU may quietly block writing to the channel.
 #
+#     Alternatively, use the @memfd migration transport to save
+#     VM/device state to an internal memfd inherited by new QEMU.
+#
+#     With @memfd, the new QEMU command may include '-incoming memfd:'
+#     to load the migration stream automatically, or '-incoming defer'
+#     to load it later with `migrate-incoming`.
+#
 #     Memory-backend objects must have the share=on attribute, but
 #     memory-backend-epc is not supported.  The VM must be started
 #     with the '-machine aux-ram-share=on' option.
@@ -1309,10 +1318,14 @@
 #
 # @file: Direct the migration stream to a file.
 #
+# @memfd: Direct the migration stream to an internal memfd.  This
+#     transport is currently only valid with cpr-exec migration mode.
+#     (Since 11.2)
+#
 # Since: 8.2
 ##
 { 'enum': 'MigrationAddressType',
-  'data': [ 'socket', 'exec', 'rdma', 'file' ] }
+  'data': [ 'socket', 'exec', 'rdma', 'file', 'memfd' ] }
 
 ##
 # @FileMigrationArgs:
@@ -1337,6 +1350,14 @@
 { 'struct': 'MigrationExecCommand',
   'data': {'args': [ 'str' ] } }
 
+##
+# @MemfdMigrationArgs:
+#
+# Since: 11.2
+##
+{ 'struct': 'MemfdMigrationArgs',
+  'data': { } }
+
 ##
 # @MigrationAddress:
 #
@@ -1353,7 +1374,8 @@
     'socket': 'SocketAddress',
     'exec': 'MigrationExecCommand',
     'rdma': 'InetSocketAddress',
-    'file': 'FileMigrationArgs' } }
+    'file': 'FileMigrationArgs',
+    'memfd': 'MemfdMigrationArgs' } }
 
 ##
 # @MigrationChannelType:
-- 
2.43.5

[PATCH 2/5] migration: Add generic memfd channel helpers

[Dongli Zhang]

Add helpers for creating an outgoing memfd-backed migration channel and for
wrapping an existing memfd fd as an incoming migration channel. The
outgoing helper can optionally duplicate the memfd fd for the caller.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 migration/memfd.c     | 67 +++++++++++++++++++++++++++++++++++++++++++
 migration/memfd.h     | 17 +++++++++++
 migration/meson.build |  1 +
 3 files changed, 85 insertions(+)
 create mode 100644 migration/memfd.c
 create mode 100644 migration/memfd.h

diff --git a/migration/memfd.c b/migration/memfd.c
new file mode 100644
index 0000000000..0b5ef0234f
--- /dev/null
+++ b/migration/memfd.c
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * QEMU live migration via memfd
+ *
+ * Copyright (c) 2026 Oracle and/or its affiliates.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/cutils.h"
+#include "qemu/memfd.h"
+#include "memfd.h"
+
+QIOChannel *memfd_create_outgoing(const char *name, int *dup_fdp,
+                                  Error **errp)
+{
+    QIOChannelFile *fioc;
+    QIOChannel *ioc;
+    int fd;
+
+#ifdef CONFIG_LINUX
+    fd = qemu_memfd_create(name, 0, false, 0, 0, errp);
+#else
+    fd = -1;
+    error_setg(errp, "memfd migration is only supported on Linux");
+#endif
+
+    if (fd < 0) {
+        return NULL;
+    }
+
+    if (dup_fdp) {
+        int dup_fd = dup(fd);
+
+        if (dup_fd < 0) {
+            error_setg_errno(errp, errno,
+                             "failed to duplicate memfd migration fd");
+            close(fd);
+            return NULL;
+        }
+
+        qemu_set_cloexec(dup_fd);
+        *dup_fdp = dup_fd;
+    }
+
+    fioc = qio_channel_file_new_fd(fd);
+    ioc = QIO_CHANNEL(fioc);
+    qio_channel_set_name(ioc, name);
+    return ioc;
+}
+
+QIOChannel *memfd_open_incoming(int fd, const char *name, Error **errp)
+{
+    QIOChannelFile *fioc;
+    QIOChannel *ioc;
+
+    if (lseek(fd, 0, SEEK_SET) < 0) {
+        error_setg_errno(errp, errno, "failed to rewind memfd migration fd");
+        close(fd);
+        return NULL;
+    }
+
+    fioc = qio_channel_file_new_fd(fd);
+    ioc = QIO_CHANNEL(fioc);
+    qio_channel_set_name(ioc, name);
+    return ioc;
+}
diff --git a/migration/memfd.h b/migration/memfd.h
new file mode 100644
index 0000000000..cb6afe3eeb
--- /dev/null
+++ b/migration/memfd.h
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * QEMU live migration via memfd
+ *
+ * Copyright (c) 2026 Oracle and/or its affiliates.
+ */
+
+#ifndef QEMU_MIGRATION_MEMFD_H
+#define QEMU_MIGRATION_MEMFD_H
+
+#include "io/channel-file.h"
+
+QIOChannel *memfd_create_outgoing(const char *name, int *dup_fdp,
+                                  Error **errp);
+QIOChannel *memfd_open_incoming(int fd, const char *name, Error **errp);
+
+#endif
diff --git a/migration/meson.build b/migration/meson.build
index 0a9a5d0d37..1d852b8c6d 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -23,6 +23,7 @@ system_ss.add(files(
   'fd.c',
   'file.c',
   'global_state.c',
+  'memfd.c',
   'migration.c',
   'multifd.c',
   'multifd-device-state.c',
-- 
2.43.5

[PATCH 3/5] migration/cpr-exec: Add memfd helpers

[Dongli Zhang]

Add cpr-exec specific helpers for using a memfd as the main migration state
channel. The helpers create the outgoing memfd channel, preserve the
duplicated fd across exec via an environment variable, claim the fd on the
incoming side, and clean up the saved fd on failure paths.

The generic memfd helper remains unchanged. The cpr-exec specific helpers
use it for channel creation and keep exec inheritance local to cpr-exec.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 migration/cpr-exec-memfd.c | 90 ++++++++++++++++++++++++++++++++++++++
 migration/cpr-exec-memfd.h | 18 ++++++++
 migration/meson.build      |  1 +
 3 files changed, 109 insertions(+)
 create mode 100644 migration/cpr-exec-memfd.c
 create mode 100644 migration/cpr-exec-memfd.h

diff --git a/migration/cpr-exec-memfd.c b/migration/cpr-exec-memfd.c
new file mode 100644
index 0000000000..2bc15f644e
--- /dev/null
+++ b/migration/cpr-exec-memfd.c
@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * CPR exec migration via memfd
+ *
+ * Copyright (c) 2026 Oracle and/or its affiliates.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/cutils.h"
+#include "channel.h"
+#include "cpr-exec-memfd.h"
+#include "memfd.h"
+
+#define CPR_EXEC_MEMFD_NAME "qemu-cpr-exec-migration"
+#define CPR_EXEC_MEMFD_ENV "QEMU_CPR_EXEC_MIGRATION_FD"
+
+static int cpr_exec_memfd_fd = -1;
+
+static int cpr_exec_memfd_claim_fd(Error **errp)
+{
+    const char *val = g_getenv(CPR_EXEC_MEMFD_ENV);
+    int fd;
+
+    if (!val) {
+        error_setg(errp, "missing %s environment variable",
+                   CPR_EXEC_MEMFD_ENV);
+        return -1;
+    }
+
+    if (qemu_strtoi(val, NULL, 10, &fd) < 0) {
+        error_setg(errp, "bad %s value %s", CPR_EXEC_MEMFD_ENV, val);
+        return -1;
+    }
+
+    g_unsetenv(CPR_EXEC_MEMFD_ENV);
+    return fd;
+}
+
+QIOChannel *cpr_exec_memfd_connect_outgoing(Error **errp)
+{
+    return memfd_create_outgoing(CPR_EXEC_MEMFD_NAME, &cpr_exec_memfd_fd,
+                                 errp);
+}
+
+void cpr_exec_memfd_connect_incoming(Error **errp)
+{
+    g_autoptr(QIOChannel) ioc = NULL;
+    int fd = cpr_exec_memfd_claim_fd(errp);
+
+    if (fd < 0) {
+        return;
+    }
+
+    ioc = memfd_open_incoming(fd, CPR_EXEC_MEMFD_NAME, errp);
+    if (!ioc) {
+        return;
+    }
+
+    migration_channel_process_incoming(ioc);
+}
+
+bool cpr_exec_memfd_preserve_fd(Error **errp)
+{
+    char val[16];
+
+    if (cpr_exec_memfd_fd < 0) {
+        return true;
+    }
+
+    qemu_clear_cloexec(cpr_exec_memfd_fd);
+    snprintf(val, sizeof(val), "%d", cpr_exec_memfd_fd);
+    if (!g_setenv(CPR_EXEC_MEMFD_ENV, val, 1)) {
+        error_setg(errp, "Setting env %s = %s failed",
+                   CPR_EXEC_MEMFD_ENV, val);
+        return false;
+    }
+
+    return true;
+}
+
+void cpr_exec_memfd_cleanup(void)
+{
+    g_unsetenv(CPR_EXEC_MEMFD_ENV);
+
+    if (cpr_exec_memfd_fd >= 0) {
+        close(cpr_exec_memfd_fd);
+        cpr_exec_memfd_fd = -1;
+    }
+}
diff --git a/migration/cpr-exec-memfd.h b/migration/cpr-exec-memfd.h
new file mode 100644
index 0000000000..1c74c7bfde
--- /dev/null
+++ b/migration/cpr-exec-memfd.h
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * CPR exec migration via memfd
+ *
+ * Copyright (c) 2026 Oracle and/or its affiliates.
+ */
+
+#ifndef QEMU_MIGRATION_CPR_EXEC_MEMFD_H
+#define QEMU_MIGRATION_CPR_EXEC_MEMFD_H
+
+#include "io/channel-file.h"
+
+QIOChannel *cpr_exec_memfd_connect_outgoing(Error **errp);
+void cpr_exec_memfd_connect_incoming(Error **errp);
+bool cpr_exec_memfd_preserve_fd(Error **errp);
+void cpr_exec_memfd_cleanup(void);
+
+#endif
diff --git a/migration/meson.build b/migration/meson.build
index 1d852b8c6d..2e8a89637e 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -17,6 +17,7 @@ system_ss.add(files(
   'cpr.c',
   'cpr-transfer.c',
   'cpr-exec.c',
+  'cpr-exec-memfd.c',
   'cpu-throttle.c',
   'dirtyrate.c',
   'exec.c',
-- 
2.43.5

[PATCH 4/5] migration/cpr-exec: Support diskless memfd transport

[Dongli Zhang]

Currently cpr-exec uses a file to pass the VM and device state between the
old and new QEMU processes. Support using memfd as a migration transport so
this state can be passed without creating a state file.

Here is an example of using the new memfd transport with cpr-exec.

(qemu) migrate_set_parameter mode cpr-exec
(qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming defer
(qemu) migrate -d memfd:
QEMU 11.1.50 monitor - type 'help' for more information

(qemu) info status
VM status: paused (inmigrate)
(qemu) migrate_incoming memfd:
(qemu) info status
VM status: running

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 migration/channel.c   |  7 +++++++
 migration/cpr-exec.c  |  9 +++++++++
 migration/memfd.c     | 23 +++++++++++++++++++++++
 migration/memfd.h     |  2 ++
 migration/migration.c |  9 +++++++++
 5 files changed, 50 insertions(+)

diff --git a/migration/channel.c b/migration/channel.c
index 1e2935f926..96de40c108 100644
--- a/migration/channel.c
+++ b/migration/channel.c
@@ -18,6 +18,7 @@
 #include "file.h"
 #include "io/channel-socket.h"
 #include "io/channel-tls.h"
+#include "memfd.h"
 #include "migration.h"
 #include "multifd.h"
 #include "options.h"
@@ -62,6 +63,8 @@ void migration_connect_outgoing(MigrationState *s, MigrationAddress *addr,
         ioc = exec_connect_outgoing(s, addr->u.exec.args, errp);
     } else if (addr->transport == MIGRATION_ADDRESS_TYPE_FILE) {
         ioc = file_connect_outgoing(s, &addr->u.file, errp);
+    } else if (addr->transport == MIGRATION_ADDRESS_TYPE_MEMFD) {
+        ioc = memfd_connect_outgoing(errp);
     } else {
         error_setg(errp, "uri is not a valid migration protocol");
     }
@@ -92,6 +95,8 @@ void migration_connect_incoming(MigrationAddress *addr, Error **errp)
         exec_connect_incoming(addr->u.exec.args, errp);
     } else if (addr->transport == MIGRATION_ADDRESS_TYPE_FILE) {
         file_connect_incoming(&addr->u.file, errp);
+    } else if (addr->transport == MIGRATION_ADDRESS_TYPE_MEMFD) {
+        memfd_connect_incoming(errp);
     } else {
         error_setg(errp, "unknown migration protocol");
     }
@@ -394,6 +399,8 @@ bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
                               errp)) {
             return false;
         }
+    } else if (g_str_equal(uri, "memfd:")) {
+        addr->transport = MIGRATION_ADDRESS_TYPE_MEMFD;
     } else {
         error_setg(errp, "unknown migration protocol: %s", uri);
         return false;
diff --git a/migration/cpr-exec.c b/migration/cpr-exec.c
index daa50916d2..388cfba80d 100644
--- a/migration/cpr-exec.c
+++ b/migration/cpr-exec.c
@@ -21,6 +21,7 @@
 #include "migration/vmstate.h"
 #include "system/runstate.h"
 #include "trace.h"
+#include "cpr-exec-memfd.h"
 
 #define CPR_EXEC_STATE_NAME "QEMU_CPR_EXEC_STATE"
 
@@ -143,6 +144,11 @@ static void cpr_exec_cb(void *opaque)
      * earlier because they should not persist across miscellaneous fork and
      * exec calls that are performed during normal operation.
      */
+    if (!cpr_exec_memfd_preserve_fd(&err)) {
+        g_clear_pointer(&argv, g_strfreev);
+        cpr_exec_memfd_cleanup();
+        goto fail;
+    }
     cpr_exec_preserve_fds();
 
     trace_cpr_exec();
@@ -155,7 +161,9 @@ static void cpr_exec_cb(void *opaque)
     error_setg_errno(&err, errno, "execvp %s failed", argv[0]);
     g_clear_pointer(&argv, g_strfreev);
     cpr_exec_unpreserve_fds();
+    cpr_exec_memfd_cleanup();
 
+fail:
     error_report_err(error_copy(err));
     migrate_set_state(&s->state, s->state, MIGRATION_STATUS_FAILED);
 
@@ -189,6 +197,7 @@ static int cpr_exec_notifier(NotifierWithReturn *notifier, MigrationEvent *e,
         qemu_notify_event();
     } else if (e->type == MIG_EVENT_FAILED) {
         cpr_exec_unpersist_state();
+        cpr_exec_memfd_cleanup();
     }
     return 0;
 }
diff --git a/migration/memfd.c b/migration/memfd.c
index 0b5ef0234f..4ed185d844 100644
--- a/migration/memfd.c
+++ b/migration/memfd.c
@@ -10,6 +10,9 @@
 #include "qemu/cutils.h"
 #include "qemu/memfd.h"
 #include "memfd.h"
+#include "cpr-exec-memfd.h"
+#include "migration/cpr.h"
+#include "options.h"
 
 QIOChannel *memfd_create_outgoing(const char *name, int *dup_fdp,
                                   Error **errp)
@@ -65,3 +68,23 @@ QIOChannel *memfd_open_incoming(int fd, const char *name, Error **errp)
     qio_channel_set_name(ioc, name);
     return ioc;
 }
+
+QIOChannel *memfd_connect_outgoing(Error **errp)
+{
+    if (migrate_mode() == MIG_MODE_CPR_EXEC) {
+        return cpr_exec_memfd_connect_outgoing(errp);
+    }
+
+    error_setg(errp, "memfd transport is only supported with cpr-exec");
+    return NULL;
+}
+
+void memfd_connect_incoming(Error **errp)
+{
+    if (cpr_get_incoming_mode() == MIG_MODE_CPR_EXEC) {
+        cpr_exec_memfd_connect_incoming(errp);
+        return;
+    }
+
+    error_setg(errp, "memfd transport is only supported with cpr-exec");
+}
diff --git a/migration/memfd.h b/migration/memfd.h
index cb6afe3eeb..9c50c84ad5 100644
--- a/migration/memfd.h
+++ b/migration/memfd.h
@@ -13,5 +13,7 @@
 QIOChannel *memfd_create_outgoing(const char *name, int *dup_fdp,
                                   Error **errp);
 QIOChannel *memfd_open_incoming(int fd, const char *name, Error **errp);
+QIOChannel *memfd_connect_outgoing(Error **errp);
+void memfd_connect_incoming(Error **errp);
 
 #endif
diff --git a/migration/migration.c b/migration/migration.c
index 9df6da131c..9f3afe5a3a 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -248,6 +248,15 @@ migration_channels_and_transport_compatible(MigrationAddress *addr,
         return false;
     }
 
+    if (addr->transport == MIGRATION_ADDRESS_TYPE_MEMFD) {
+        if (migrate_mode() != MIG_MODE_CPR_EXEC &&
+            cpr_get_incoming_mode() != MIG_MODE_CPR_EXEC) {
+            error_setg(errp,
+                       "memfd transport is only supported with cpr-exec");
+            return false;
+        }
+    }
+
     return true;
 }
 
-- 
2.43.5

[PATCH 5/5] migration: cpr-exec memfd transport documentation

[Dongli Zhang]

Document the cpr-exec flow using the memfd migration transport. The example
keeps migrate_incoming explicit so the command sequence mirrors the
existing deferred incoming flow.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 docs/devel/migration/CPR.rst | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
index b6178568a8..8afc83a08d 100644
--- a/docs/devel/migration/CPR.rst
+++ b/docs/devel/migration/CPR.rst
@@ -369,6 +369,10 @@ Outgoing:
     a ``file`` type, but one can use other types such as ``exec``,
     provided the command captures all the data from the outgoing side,
     and provides all the data to the incoming side.
+  * Alternatively, use the ``memfd:`` URI to save VM/device state
+    to an internal memfd inherited by the new QEMU.  The new QEMU may
+    use ``-incoming memfd:`` to load it automatically, or ``-incoming
+    defer`` to load it later with ``migrate_incoming memfd:``.
 
 Incoming:
   * You do not need to explicitly start new QEMU.  It is started as
@@ -424,6 +428,26 @@ Example 2: incoming defer
   (qemu) info status
   VM status: running
 
+Example 3: main state memfd
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  # qemu-kvm -monitor stdio
+  -machine q35,memory-backend=ram0,aux-ram-share=on
+  -object memory-backend-memfd,id=ram0,size=8G,share=on
+  ... ...
+
+  QEMU 11.1.50 monitor - type 'help' for more information
+  (qemu) migrate_set_parameter mode cpr-exec
+  (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming defer
+  (qemu) migrate -d memfd:
+  QEMU 11.1.50 monitor - type 'help' for more information
+  (qemu) info status
+  VM status: paused (inmigrate)
+  (qemu) migrate_incoming memfd:
+  (qemu) info status
+  VM status: running
+
 Caveats
 ^^^^^^^
 
-- 
2.43.5

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Markus Armbruster]

Dongli Zhang <dongli.zhang@oracle.com> writes:

> Add a memfd migration transport to the MigrationAddress QAPI schema. The
> new transport has no user-supplied arguments because QEMU creates and
> passes the backing memfd internally.
>
> The initial user is cpr-exec, where the main VM/device migration stream can
> be stored in an inherited memfd instead of an external file.
>
> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
> ---
>  qapi/migration.json | 32 +++++++++++++++++++++++++++-----
>  1 file changed, 27 insertions(+), 5 deletions(-)
>
> diff --git a/qapi/migration.json b/qapi/migration.json
> index b1eaf7b054..1505c201ca 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -702,9 +702,11 @@
>  # @cpr-exec: The migrate command stops the VM, saves state to the
>  #     migration channel, directly exec's a new version of QEMU on the
>  #     same host, replacing the original process while retaining its
> -#     PID, and loads state from the channel.  Guest RAM is preserved
> -#     in place.  Devices and their pinned pages are also preserved for
> -#     VFIO and IOMMUFD.
> +#     PID, and loads state from the channel.  With the @memfd
> +#     migration transport, QEMU saves VM/device state to an internal
> +#     memfd instead of an external main migration channel.  Guest RAM
> +#     is preserved in place.  Devices and their pinned pages are also
> +#     preserved for VFIO and IOMMUFD.
>  #
>  #     Old QEMU starts new QEMU by exec'ing the command specified by
>  #     the @cpr-exec-command parameter.  The command may be a direct
> @@ -716,6 +718,13 @@
>  #     as a file, that accepts all data before old QEMU exits.
>  #     Otherwise, old QEMU may quietly block writing to the channel.
>  #
> +#     Alternatively, use the @memfd migration transport to save
> +#     VM/device state to an internal memfd inherited by new QEMU.
> +#

Any guidance on when to use @memfd?

> +#     With @memfd, the new QEMU command may include '-incoming memfd:'
> +#     to load the migration stream automatically, or '-incoming defer'
> +#     to load it later with `migrate-incoming`.
> +#

Isn't that the same for any transport?  I.e. old and new QEMU must use a
common transport, and you configure it for the new one with -incoming
right away, or later with migrate-incoming.

>  #     Memory-backend objects must have the share=on attribute, but
>  #     memory-backend-epc is not supported.  The VM must be started
>  #     with the '-machine aux-ram-share=on' option.
> @@ -1309,10 +1318,14 @@
>  #
>  # @file: Direct the migration stream to a file.
>  #
> +# @memfd: Direct the migration stream to an internal memfd.  This
> +#     transport is currently only valid with cpr-exec migration mode.
> +#     (Since 11.2)
> +#
>  # Since: 8.2
>  ##
>  { 'enum': 'MigrationAddressType',
> -  'data': [ 'socket', 'exec', 'rdma', 'file' ] }
> +  'data': [ 'socket', 'exec', 'rdma', 'file', 'memfd' ] }
>  
>  ##
>  # @FileMigrationArgs:
> @@ -1337,6 +1350,14 @@
>  { 'struct': 'MigrationExecCommand',
>    'data': {'args': [ 'str' ] } }
>  
> +##
> +# @MemfdMigrationArgs:
> +#
> +# Since: 11.2
> +##
> +{ 'struct': 'MemfdMigrationArgs',
> +  'data': { } }
> +
>  ##
>  # @MigrationAddress:
>  #
> @@ -1353,7 +1374,8 @@
>      'socket': 'SocketAddress',
>      'exec': 'MigrationExecCommand',
>      'rdma': 'InetSocketAddress',
> -    'file': 'FileMigrationArgs' } }
> +    'file': 'FileMigrationArgs',
> +    'memfd': 'MemfdMigrationArgs' } }

Simpler: omit branch @memfd, no need for struct MemfdMigrationArgs.

>  ##
>  # @MigrationChannelType:

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Dongli Zhang]

On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
> Dongli Zhang <dongli.zhang@oracle.com> writes:
> 
>> Add a memfd migration transport to the MigrationAddress QAPI schema. The
>> new transport has no user-supplied arguments because QEMU creates and
>> passes the backing memfd internally.
>>
>> The initial user is cpr-exec, where the main VM/device migration stream can
>> be stored in an inherited memfd instead of an external file.
>>
>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>> ---
>>  qapi/migration.json | 32 +++++++++++++++++++++++++++-----
>>  1 file changed, 27 insertions(+), 5 deletions(-)
>>
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index b1eaf7b054..1505c201ca 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -702,9 +702,11 @@
>>  # @cpr-exec: The migrate command stops the VM, saves state to the
>>  #     migration channel, directly exec's a new version of QEMU on the
>>  #     same host, replacing the original process while retaining its
>> -#     PID, and loads state from the channel.  Guest RAM is preserved
>> -#     in place.  Devices and their pinned pages are also preserved for
>> -#     VFIO and IOMMUFD.
>> +#     PID, and loads state from the channel.  With the @memfd
>> +#     migration transport, QEMU saves VM/device state to an internal
>> +#     memfd instead of an external main migration channel.  Guest RAM
>> +#     is preserved in place.  Devices and their pinned pages are also
>> +#     preserved for VFIO and IOMMUFD.
>>  #
>>  #     Old QEMU starts new QEMU by exec'ing the command specified by
>>  #     the @cpr-exec-command parameter.  The command may be a direct
>> @@ -716,6 +718,13 @@
>>  #     as a file, that accepts all data before old QEMU exits.
>>  #     Otherwise, old QEMU may quietly block writing to the channel.
>>  #
>> +#     Alternatively, use the @memfd migration transport to save
>> +#     VM/device state to an internal memfd inherited by new QEMU.
>> +#
> 
> Any guidance on when to use @memfd?

My objective is to avoid using disks, or running QEMU with writable storage.

Both guest I/O and networking can go through VFIO, and everything remains in memory.

When I was testing these CPR features, my favorite was cpr-exec because it does
not require creating an additional QEMU instance.

Based on my understanding, the best approach to live update a running binary is
to replace only the code section while keeping everything else in memory, i.e.
guest memory, device fds, and device state.

Unfortunately, device state cannot simply remain in memory across exec(), so we
need to migrate it from the old instance to the new instance. I was thinking:
why not use memory instead of storage?

Initially, I added the change only for cpr-exec. Later, I reworked the change to
make memfd a new migration transport. As a side benefit for me, I also practiced
how to add a new transport to the QEMU migration code :)

> 
>> +#     With @memfd, the new QEMU command may include '-incoming memfd:'
>> +#     to load the migration stream automatically, or '-incoming defer'
>> +#     to load it later with `migrate-incoming`.
>> +#
> 
> Isn't that the same for any transport?  I.e. old and new QEMU must use a
> common transport, and you configure it for the new one with -incoming
> right away, or later with migrate-incoming.

Yes, it is the same for all transports. I added the explanation only because the
memfd transport does not need additional data or a filename, for example, the
URI is just "memfd:".

Will remove it.

> 
>>  #     Memory-backend objects must have the share=on attribute, but
>>  #     memory-backend-epc is not supported.  The VM must be started
>>  #     with the '-machine aux-ram-share=on' option.
>> @@ -1309,10 +1318,14 @@
>>  #
>>  # @file: Direct the migration stream to a file.
>>  #
>> +# @memfd: Direct the migration stream to an internal memfd.  This
>> +#     transport is currently only valid with cpr-exec migration mode.
>> +#     (Since 11.2)
>> +#
>>  # Since: 8.2
>>  ##
>>  { 'enum': 'MigrationAddressType',
>> -  'data': [ 'socket', 'exec', 'rdma', 'file' ] }
>> +  'data': [ 'socket', 'exec', 'rdma', 'file', 'memfd' ] }
>>  
>>  ##
>>  # @FileMigrationArgs:
>> @@ -1337,6 +1350,14 @@
>>  { 'struct': 'MigrationExecCommand',
>>    'data': {'args': [ 'str' ] } }
>>  
>> +##
>> +# @MemfdMigrationArgs:
>> +#
>> +# Since: 11.2
>> +##
>> +{ 'struct': 'MemfdMigrationArgs',
>> +  'data': { } }
>> +
>>  ##
>>  # @MigrationAddress:
>>  #
>> @@ -1353,7 +1374,8 @@
>>      'socket': 'SocketAddress',
>>      'exec': 'MigrationExecCommand',
>>      'rdma': 'InetSocketAddress',
>> -    'file': 'FileMigrationArgs' } }
>> +    'file': 'FileMigrationArgs',
>> +    'memfd': 'MemfdMigrationArgs' } }
> 
> Simpler: omit branch @memfd, no need for struct MemfdMigrationArgs.

Just to keep the schema shape aligned with the other transports. Will remove it.

Thank you very much!

Dongli Zhang

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Markus Armbruster]

Dongli Zhang <dongli.zhang@oracle.com> writes:

> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>> 
>>> Add a memfd migration transport to the MigrationAddress QAPI schema. The
>>> new transport has no user-supplied arguments because QEMU creates and
>>> passes the backing memfd internally.
>>>
>>> The initial user is cpr-exec, where the main VM/device migration stream can
>>> be stored in an inherited memfd instead of an external file.
>>>
>>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>>> ---
>>>  qapi/migration.json | 32 +++++++++++++++++++++++++++-----
>>>  1 file changed, 27 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>> index b1eaf7b054..1505c201ca 100644
>>> --- a/qapi/migration.json
>>> +++ b/qapi/migration.json
>>> @@ -702,9 +702,11 @@
>>>  # @cpr-exec: The migrate command stops the VM, saves state to the
>>>  #     migration channel, directly exec's a new version of QEMU on the
>>>  #     same host, replacing the original process while retaining its
>>> -#     PID, and loads state from the channel.  Guest RAM is preserved
>>> -#     in place.  Devices and their pinned pages are also preserved for
>>> -#     VFIO and IOMMUFD.
>>> +#     PID, and loads state from the channel.  With the @memfd
>>> +#     migration transport, QEMU saves VM/device state to an internal
>>> +#     memfd instead of an external main migration channel.  Guest RAM
>>> +#     is preserved in place.  Devices and their pinned pages are also
>>> +#     preserved for VFIO and IOMMUFD.
>>>  #
>>>  #     Old QEMU starts new QEMU by exec'ing the command specified by
>>>  #     the @cpr-exec-command parameter.  The command may be a direct
>>> @@ -716,6 +718,13 @@
>>>  #     as a file, that accepts all data before old QEMU exits.
>>>  #     Otherwise, old QEMU may quietly block writing to the channel.
>>>  #
>>> +#     Alternatively, use the @memfd migration transport to save
>>> +#     VM/device state to an internal memfd inherited by new QEMU.
>>> +#
>> 
>> Any guidance on when to use @memfd?
>
> My objective is to avoid using disks, or running QEMU with writable storage.
>
> Both guest I/O and networking can go through VFIO, and everything remains in memory.
>
> When I was testing these CPR features, my favorite was cpr-exec because it does
> not require creating an additional QEMU instance.
>
> Based on my understanding, the best approach to live update a running binary is
> to replace only the code section while keeping everything else in memory, i.e.
> guest memory, device fds, and device state.
>
> Unfortunately, device state cannot simply remain in memory across exec(), so we
> need to migrate it from the old instance to the new instance. I was thinking:
> why not use memory instead of storage?
>
> Initially, I added the change only for cpr-exec. Later, I reworked the change to
> make memfd a new migration transport. As a side benefit for me, I also practiced
> how to add a new transport to the QEMU migration code :)

This is why you wrote the feature.  Please work it into the commit
message.

We should also help users decide whether to use the feature.  I gather
it's useful with migration mode cpr-exec.  Any other use cases?  Does it
even work with other migration modes?  If yes, which ones, and what
happens when you try with another one?

>>> +#     With @memfd, the new QEMU command may include '-incoming memfd:'
>>> +#     to load the migration stream automatically, or '-incoming defer'
>>> +#     to load it later with `migrate-incoming`.
>>> +#
>> 
>> Isn't that the same for any transport?  I.e. old and new QEMU must use a
>> common transport, and you configure it for the new one with -incoming
>> right away, or later with migrate-incoming.
>
> Yes, it is the same for all transports. I added the explanation only because the
> memfd transport does not need additional data or a filename, for example, the
> URI is just "memfd:".
>
> Will remove it.
>
>> 
>>>  #     Memory-backend objects must have the share=on attribute, but
>>>  #     memory-backend-epc is not supported.  The VM must be started
>>>  #     with the '-machine aux-ram-share=on' option.

[...]

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Dongli Zhang]

On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
> Dongli Zhang <dongli.zhang@oracle.com> writes:
> 
>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>> 
>>>> Add a memfd migration transport to the MigrationAddress QAPI schema. The
>>>> new transport has no user-supplied arguments because QEMU creates and
>>>> passes the backing memfd internally.
>>>>
>>>> The initial user is cpr-exec, where the main VM/device migration stream can
>>>> be stored in an inherited memfd instead of an external file.
>>>>
>>>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>>>> ---
>>>>  qapi/migration.json | 32 +++++++++++++++++++++++++++-----
>>>>  1 file changed, 27 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>> index b1eaf7b054..1505c201ca 100644
>>>> --- a/qapi/migration.json
>>>> +++ b/qapi/migration.json
>>>> @@ -702,9 +702,11 @@
>>>>  # @cpr-exec: The migrate command stops the VM, saves state to the
>>>>  #     migration channel, directly exec's a new version of QEMU on the
>>>>  #     same host, replacing the original process while retaining its
>>>> -#     PID, and loads state from the channel.  Guest RAM is preserved
>>>> -#     in place.  Devices and their pinned pages are also preserved for
>>>> -#     VFIO and IOMMUFD.
>>>> +#     PID, and loads state from the channel.  With the @memfd
>>>> +#     migration transport, QEMU saves VM/device state to an internal
>>>> +#     memfd instead of an external main migration channel.  Guest RAM
>>>> +#     is preserved in place.  Devices and their pinned pages are also
>>>> +#     preserved for VFIO and IOMMUFD.
>>>>  #
>>>>  #     Old QEMU starts new QEMU by exec'ing the command specified by
>>>>  #     the @cpr-exec-command parameter.  The command may be a direct
>>>> @@ -716,6 +718,13 @@
>>>>  #     as a file, that accepts all data before old QEMU exits.
>>>>  #     Otherwise, old QEMU may quietly block writing to the channel.
>>>>  #
>>>> +#     Alternatively, use the @memfd migration transport to save
>>>> +#     VM/device state to an internal memfd inherited by new QEMU.
>>>> +#
>>> 
>>> Any guidance on when to use @memfd?
>>
>> My objective is to avoid using disks, or running QEMU with writable storage.
>>
>> Both guest I/O and networking can go through VFIO, and everything remains in memory.
>>
>> When I was testing these CPR features, my favorite was cpr-exec because it does
>> not require creating an additional QEMU instance.
>>
>> Based on my understanding, the best approach to live update a running binary is
>> to replace only the code section while keeping everything else in memory, i.e.
>> guest memory, device fds, and device state.
>>
>> Unfortunately, device state cannot simply remain in memory across exec(), so we
>> need to migrate it from the old instance to the new instance. I was thinking:
>> why not use memory instead of storage?
>>
>> Initially, I added the change only for cpr-exec. Later, I reworked the change to
>> make memfd a new migration transport. As a side benefit for me, I also practiced
>> how to add a new transport to the QEMU migration code :)
> 
> This is why you wrote the feature.  Please work it into the commit
> message.

Sure.

> 
> We should also help users decide whether to use the feature.  I gather
> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
> even work with other migration modes?  If yes, which ones, and what
> happens when you try with another one?

So far I only added support for cpr-exec. I have not tried othe modes.

In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
sure it is necessary.

For cpr-transfer, the source and destination are separate QEMU processes. If we
want to pass a memfd between them, we still need an IPC mechanism that can
transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
reduces the value of using memfd as the migration channel, because we need at
least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.

If a Unix domain socket is already required, the main benefit of memfd would
likely be performance, especially when the device state is large.

cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
be preserved across exec by clearing CLOEXEC and passing the fd number through
an environment variable. This allows us to add a memfd-backed migration channel
without needing an external file or socket for the device state.

One potential future use case is LUO/KHO support for QEMU. In that case, device
state could be preserved in memory as a memfd and potentially survive across a
KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
internal tests using memfd to preserve device state across kexec although using
different implementation, and it worked.

For other live migration modes, or even local migration, I do not think this is
useful. For example, saving to and loading from a file won't use memfd, as the
QEMU processes are terminated during the this process.

Thank you very much!

Dongli Zhang

> 
>>>> +#     With @memfd, the new QEMU command may include '-incoming memfd:'
>>>> +#     to load the migration stream automatically, or '-incoming defer'
>>>> +#     to load it later with `migrate-incoming`.
>>>> +#
>>> 
>>> Isn't that the same for any transport?  I.e. old and new QEMU must use a
>>> common transport, and you configure it for the new one with -incoming
>>> right away, or later with migrate-incoming.
>>
>> Yes, it is the same for all transports. I added the explanation only because the
>> memfd transport does not need additional data or a filename, for example, the
>> URI is just "memfd:".
>>
>> Will remove it.
>>
>>> 
>>>>  #     Memory-backend objects must have the share=on attribute, but
>>>>  #     memory-backend-epc is not supported.  The VM must be started
>>>>  #     with the '-machine aux-ram-share=on' option.
> 
> [...]
> 

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Markus Armbruster]

Dongli Zhang <dongli.zhang@oracle.com> writes:

> On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>> 
>>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>>> 
>>>>> Add a memfd migration transport to the MigrationAddress QAPI schema. The
>>>>> new transport has no user-supplied arguments because QEMU creates and
>>>>> passes the backing memfd internally.
>>>>>
>>>>> The initial user is cpr-exec, where the main VM/device migration stream can
>>>>> be stored in an inherited memfd instead of an external file.
>>>>>
>>>>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>>>>> ---
>>>>>  qapi/migration.json | 32 +++++++++++++++++++++++++++-----
>>>>>  1 file changed, 27 insertions(+), 5 deletions(-)
>>>>>
>>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>>> index b1eaf7b054..1505c201ca 100644
>>>>> --- a/qapi/migration.json
>>>>> +++ b/qapi/migration.json
>>>>> @@ -702,9 +702,11 @@
>>>>>  # @cpr-exec: The migrate command stops the VM, saves state to the
>>>>>  #     migration channel, directly exec's a new version of QEMU on the
>>>>>  #     same host, replacing the original process while retaining its
>>>>> -#     PID, and loads state from the channel.  Guest RAM is preserved
>>>>> -#     in place.  Devices and their pinned pages are also preserved for
>>>>> -#     VFIO and IOMMUFD.
>>>>> +#     PID, and loads state from the channel.  With the @memfd
>>>>> +#     migration transport, QEMU saves VM/device state to an internal
>>>>> +#     memfd instead of an external main migration channel.  Guest RAM
>>>>> +#     is preserved in place.  Devices and their pinned pages are also
>>>>> +#     preserved for VFIO and IOMMUFD.
>>>>>  #
>>>>>  #     Old QEMU starts new QEMU by exec'ing the command specified by
>>>>>  #     the @cpr-exec-command parameter.  The command may be a direct
>>>>> @@ -716,6 +718,13 @@
>>>>>  #     as a file, that accepts all data before old QEMU exits.
>>>>>  #     Otherwise, old QEMU may quietly block writing to the channel.
>>>>>  #
>>>>> +#     Alternatively, use the @memfd migration transport to save
>>>>> +#     VM/device state to an internal memfd inherited by new QEMU.
>>>>> +#
>>>> 
>>>> Any guidance on when to use @memfd?
>>>
>>> My objective is to avoid using disks, or running QEMU with writable storage.
>>>
>>> Both guest I/O and networking can go through VFIO, and everything remains in memory.
>>>
>>> When I was testing these CPR features, my favorite was cpr-exec because it does
>>> not require creating an additional QEMU instance.
>>>
>>> Based on my understanding, the best approach to live update a running binary is
>>> to replace only the code section while keeping everything else in memory, i.e.
>>> guest memory, device fds, and device state.
>>>
>>> Unfortunately, device state cannot simply remain in memory across exec(), so we
>>> need to migrate it from the old instance to the new instance. I was thinking:
>>> why not use memory instead of storage?
>>>
>>> Initially, I added the change only for cpr-exec. Later, I reworked the change to
>>> make memfd a new migration transport. As a side benefit for me, I also practiced
>>> how to add a new transport to the QEMU migration code :)
>> 
>> This is why you wrote the feature.  Please work it into the commit
>> message.
>
> Sure.
>
>> 
>> We should also help users decide whether to use the feature.  I gather
>> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
>> even work with other migration modes?  If yes, which ones, and what
>> happens when you try with another one?
>
> So far I only added support for cpr-exec. I have not tried othe modes.

What happens when you try to use it with other migration modes?

> In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
> sure it is necessary.
>
> For cpr-transfer, the source and destination are separate QEMU processes. If we
> want to pass a memfd between them, we still need an IPC mechanism that can
> transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
> reduces the value of using memfd as the migration channel, because we need at
> least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.
>
> If a Unix domain socket is already required, the main benefit of memfd would
> likely be performance, especially when the device state is large.

We'd want to measure the performance impact to see whether it's worth
complicating configuration and code.

> cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
> be preserved across exec by clearing CLOEXEC and passing the fd number through
> an environment variable. This allows us to add a memfd-backed migration channel
> without needing an external file or socket for the device state.

What are the actual user-visible benefits?

Performance?  Provide measurements, please.

Simpler configuration?  Not sure that would be worth the special case.

Anything else?

> One potential future use case is LUO/KHO support for QEMU. In that case, device
> state could be preserved in memory as a memfd and potentially survive across a
> KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
> internal tests using memfd to preserve device state across kexec although using
> different implementation, and it worked.
>
> For other live migration modes, or even local migration, I do not think this is
> useful. For example, saving to and loading from a file won't use memfd, as the
> QEMU processes are terminated during the this process.

User documentation should spell out when memfd can be used.  I recommend
to make it usable only when it's known to improve things in at least
some scenario that matters.  Code should cleanly reject attempts to use
memfd when it's not usable.

User documentation should help users decide whether to use it.

The commit message(s) may need to provide additional detail on how
things improve.

> Thank you very much!
>
> Dongli Zhang

You're welcome!

[...]

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Dongli Zhang]

On Sun, Aug 23, 2026 11:29:53PM -0700, Markus Armbruster wrote:
Dongli Zhang <dongli.zhang@oracle.com> writes:
> 
>> On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>> 
>>>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:

[snip]

>>> message.
>>
>> Sure.
>>
>>> 
>>> We should also help users decide whether to use the feature.  I gather
>>> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
>>> even work with other migration modes?  If yes, which ones, and what
>>> happens when you try with another one?
>>
>> So far I only added support for cpr-exec. I have not tried othe modes.
> 
> What happens when you try to use it with other migration modes?

If I use memfd transport without cpr-exec mode, it will be denied.

The code checks that only cpr-exec mode is supported.

(qemu) migrate -d memfd:
Error: memfd transport is only supported with cpr-exec

> 
>> In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
>> sure it is necessary.
>>
>> For cpr-transfer, the source and destination are separate QEMU processes. If we
>> want to pass a memfd between them, we still need an IPC mechanism that can
>> transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
>> reduces the value of using memfd as the migration channel, because we need at
>> least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.
>>
>> If a Unix domain socket is already required, the main benefit of memfd would
>> likely be performance, especially when the device state is large.
> 
> We'd want to measure the performance impact to see whether it's worth
> complicating configuration and code.

I may create a VM with many virtio devices to maximize the storage overhead of
the VM device state. However, I don't think the goal is to improve performance.

> 
>> cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
>> be preserved across exec by clearing CLOEXEC and passing the fd number through
>> an environment variable. This allows us to add a memfd-backed migration channel
>> without needing an external file or socket for the device state.
> 
> What are the actual user-visible benefits?

The goal is to avoid the VM state file used during live update with cpr-exec.
So, the user-visible benefit is that the file is no longer required. Live update
with cpr-exec no longer relies on a file on disk.

> 
> Performance?  Provide measurements, please.

Not performance.

> 
> Simpler configuration?  Not sure that would be worth the special case.

Yes. No file required.

"migrate -d memfd:" vs. "migrate -d file:vm.state".

"migrate_incoming memfd:" vs. "migrate_incoming file:vm.state".

> 
> Anything else?

No. Writable storage is no longer required to cache the VM device state file.
That's the only explicit benefit so far.
> 
>> One potential future use case is LUO/KHO support for QEMU. In that case, device
>> state could be preserved in memory as a memfd and potentially survive across a
>> KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
>> internal tests using memfd to preserve device state across kexec although using
>> different implementation, and it worked.
>>
>> For other live migration modes, or even local migration, I do not think this is
>> useful. For example, saving to and loading from a file won't use memfd, as the
>> QEMU processes are terminated during the this process.
> 
> User documentation should spell out when memfd can be used.  I recommend
> to make it usable only when it's known to improve things in at least
> some scenario that matters.  Code should cleanly reject attempts to use
> memfd when it's not usable.
> 
> User documentation should help users decide whether to use it.
> 
> The commit message(s) may need to provide additional detail on how
> things improve.
> 
Thank you very much for feedback!

Dongli Zhang

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Markus Armbruster]

Dongli Zhang <dongli.zhang@oracle.com> writes:

> On Sun, Aug 23, 2026 11:29:53PM -0700, Markus Armbruster wrote:
> Dongli Zhang <dongli.zhang@oracle.com> writes:
>> 
>>> On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>>> 
>>>>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>
> [snip]
>
>>>> message.
>>>
>>> Sure.
>>>
>>>> 
>>>> We should also help users decide whether to use the feature.  I gather
>>>> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
>>>> even work with other migration modes?  If yes, which ones, and what
>>>> happens when you try with another one?
>>>
>>> So far I only added support for cpr-exec. I have not tried othe modes.
>> 
>> What happens when you try to use it with other migration modes?
>
> If I use memfd transport without cpr-exec mode, it will be denied.
>
> The code checks that only cpr-exec mode is supported.
>
> (qemu) migrate -d memfd:
> Error: memfd transport is only supported with cpr-exec

Good.

>>> In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
>>> sure it is necessary.
>>>
>>> For cpr-transfer, the source and destination are separate QEMU processes. If we
>>> want to pass a memfd between them, we still need an IPC mechanism that can
>>> transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
>>> reduces the value of using memfd as the migration channel, because we need at
>>> least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.
>>>
>>> If a Unix domain socket is already required, the main benefit of memfd would
>>> likely be performance, especially when the device state is large.
>> 
>> We'd want to measure the performance impact to see whether it's worth
>> complicating configuration and code.
>
> I may create a VM with many virtio devices to maximize the storage overhead of
> the VM device state. However, I don't think the goal is to improve performance.

As long as that's the case, enabling memfd for cpr-transfer makes no
sense.

>>> cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
>>> be preserved across exec by clearing CLOEXEC and passing the fd number through
>>> an environment variable. This allows us to add a memfd-backed migration channel
>>> without needing an external file or socket for the device state.
>> 
>> What are the actual user-visible benefits?
>
> The goal is to avoid the VM state file used during live update with cpr-exec.
> So, the user-visible benefit is that the file is no longer required. Live update
> with cpr-exec no longer relies on a file on disk.

Got it.

>> Performance?  Provide measurements, please.
>
> Not performance.

If we accept memfd for its other benefits, performance measurements
might be interesting.  Not now.

>> Simpler configuration?  Not sure that would be worth the special case.
>
> Yes. No file required.
>
> "migrate -d memfd:" vs. "migrate -d file:vm.state".
>
> "migrate_incoming memfd:" vs. "migrate_incoming file:vm.state".

Got it.

>> Anything else?
>
> No. Writable storage is no longer required to cache the VM device state file.
> That's the only explicit benefit so far.

Benefits of the new MigrationAddressType @memfd: we gain a zero-config
migration channel for MigMode cpr-exec.

Costs: we gain yet another kind of migration channel, useful only with
cpr-exec.  Feels niche.

Are the benefits worth the costs?  Migration maintainers decide, not me.

In my (relatively ignorant) opinion, migration already suffers from too
many features.  Hard to understand, painful test matrix, expensive to
maintain.

>>> One potential future use case is LUO/KHO support for QEMU. In that case, device
>>> state could be preserved in memory as a memfd and potentially survive across a
>>> KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
>>> internal tests using memfd to preserve device state across kexec although using
>>> different implementation, and it worked.
>>>
>>> For other live migration modes, or even local migration, I do not think this is
>>> useful. For example, saving to and loading from a file won't use memfd, as the
>>> QEMU processes are terminated during the this process.
>> 
>> User documentation should spell out when memfd can be used.

It does: "This transport is currently only valid with cpr-exec migration
mode."

>>                                                              I recommend
>> to make it usable only when it's known to improve things in at least
>> some scenario that matters.  Code should cleanly reject attempts to use
>> memfd when it's not usable.

You showed above that it does.

>> User documentation should help users decide whether to use it.

It does not so far.

>> The commit message(s) may need to provide additional detail on how
>> things improve.
>> 
> Thank you very much for feedback!
>
> Dongli Zhang

You're welcome!


Tangent, pardon my ignorance...  We have MigMode @cpr-exec and we have
MigrationAddressType @exec.  Both involve starting a new instance of
QEMU.  How do their use cases differ?

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Dongli Zhang]

On Thu, Aug 27, 2026 1:23:44AM -0700, Markus Armbruster wrote:
> 
> Dongli Zhang <dongli.zhang@oracle.com> writes:
> 
>> On Sun, Aug 23, 2026 11:29:53PM -0700, Markus Armbruster wrote:
>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>> 
>>>> On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote:
>>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>>>> 
>>>>>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote:
>>>>>>> Dongli Zhang <dongli.zhang@oracle.com> writes:
>>
>> [snip]
>>
>>>>> message.
>>>>
>>>> Sure.
>>>>
>>>>> 
>>>>> We should also help users decide whether to use the feature.  I gather
>>>>> it's useful with migration mode cpr-exec.  Any other use cases?  Does it
>>>>> even work with other migration modes?  If yes, which ones, and what
>>>>> happens when you try with another one?
>>>>
>>>> So far I only added support for cpr-exec. I have not tried othe modes.
>>> 
>>> What happens when you try to use it with other migration modes?
>>
>> If I use memfd transport without cpr-exec mode, it will be denied.
>>
>> The code checks that only cpr-exec mode is supported.
>>
>> (qemu) migrate -d memfd:
>> Error: memfd transport is only supported with cpr-exec
> 
> Good.
> 
>>>> In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
>>>> sure it is necessary.
>>>>
>>>> For cpr-transfer, the source and destination are separate QEMU processes. If we
>>>> want to pass a memfd between them, we still need an IPC mechanism that can
>>>> transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
>>>> reduces the value of using memfd as the migration channel, because we need at
>>>> least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.
>>>>
>>>> If a Unix domain socket is already required, the main benefit of memfd would
>>>> likely be performance, especially when the device state is large.
>>> 
>>> We'd want to measure the performance impact to see whether it's worth
>>> complicating configuration and code.
>>
>> I may create a VM with many virtio devices to maximize the storage overhead of
>> the VM device state. However, I don't think the goal is to improve performance.
> 
> As long as that's the case, enabling memfd for cpr-transfer makes no
> sense.
> 
>>>> cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
>>>> be preserved across exec by clearing CLOEXEC and passing the fd number through
>>>> an environment variable. This allows us to add a memfd-backed migration channel
>>>> without needing an external file or socket for the device state.
>>> 
>>> What are the actual user-visible benefits?
>>
>> The goal is to avoid the VM state file used during live update with cpr-exec.
>> So, the user-visible benefit is that the file is no longer required. Live update
>> with cpr-exec no longer relies on a file on disk.
> 
> Got it.
> 
>>> Performance?  Provide measurements, please.
>>
>> Not performance.
> 
> If we accept memfd for its other benefits, performance measurements
> might be interesting.  Not now.
> 
>>> Simpler configuration?  Not sure that would be worth the special case.
>>
>> Yes. No file required.
>>
>> "migrate -d memfd:" vs. "migrate -d file:vm.state".
>>
>> "migrate_incoming memfd:" vs. "migrate_incoming file:vm.state".
> 
> Got it.
> 
>>> Anything else?
>>
>> No. Writable storage is no longer required to cache the VM device state file.
>> That's the only explicit benefit so far.
> 
> Benefits of the new MigrationAddressType @memfd: we gain a zero-config
> migration channel for MigMode cpr-exec.
> 
> Costs: we gain yet another kind of migration channel, useful only with
> cpr-exec.  Feels niche.
> 
> Are the benefits worth the costs?  Migration maintainers decide, not me.
> 
> In my (relatively ignorant) opinion, migration already suffers from too
> many features.  Hard to understand, painful test matrix, expensive to
> maintain.

It will depend on the maintainer's decision.

Thank you very much for helping clarify the benefits, costs, and use cases!

Dongli Zhang

> 
>>>> One potential future use case is LUO/KHO support for QEMU. In that case, device
>>>> state could be preserved in memory as a memfd and potentially survive across a
>>>> KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
>>>> internal tests using memfd to preserve device state across kexec although using
>>>> different implementation, and it worked.
>>>>
>>>> For other live migration modes, or even local migration, I do not think this is
>>>> useful. For example, saving to and loading from a file won't use memfd, as the
>>>> QEMU processes are terminated during the this process.
>>> 
>>> User documentation should spell out when memfd can be used.
> 
> It does: "This transport is currently only valid with cpr-exec migration
> mode."
> 
>>>                                                              I recommend
>>> to make it usable only when it's known to improve things in at least
>>> some scenario that matters.  Code should cleanly reject attempts to use
>>> memfd when it's not usable.
> 
> You showed above that it does.
> 
>>> User documentation should help users decide whether to use it.
> 
> It does not so far.
> 
>>> The commit message(s) may need to provide additional detail on how
>>> things improve.
>>> 
>> Thank you very much for feedback!
>>
>> Dongli Zhang
> 
> You're welcome!
> 
> 
> Tangent, pardon my ignorance...  We have MigMode @cpr-exec and we have
> MigrationAddressType @exec.  Both involve starting a new instance of
> QEMU.  How do their use cases differ?
> 

Re: [PATCH 1/5] migration/qapi: Add memfd migration transport

[Peter Xu]

On Fri, Aug 21, 2026 at 01:11:23PM -0700, Dongli Zhang wrote:
> So far I only added support for cpr-exec. I have not tried othe modes.
> 
> In my opinion, cpr-transfer could also use a memfd-backed channel, but I am not
> sure it is necessary.
> 
> For cpr-transfer, the source and destination are separate QEMU processes. If we
> want to pass a memfd between them, we still need an IPC mechanism that can
> transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That
> reduces the value of using memfd as the migration channel, because we need at
> least one Unix domain socket anyway to pass the memfd fd and coordinate the handoff.
> 
> If a Unix domain socket is already required, the main benefit of memfd would
> likely be performance, especially when the device state is large.

I'm actually not sure dumping everything into one fd, then share it via the
other unix socket will be faster: the destination side can already start
parsing concurrently while reading things out, rather than waiting for
everything to land the page cache inside the memfd.

So I actually don't know if this will ever be useful by anything except
cpr-exec.

> 
> cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can
> be preserved across exec by clearing CLOEXEC and passing the fd number through
> an environment variable. This allows us to add a memfd-backed migration channel
> without needing an external file or socket for the device state.
> 
> One potential future use case is LUO/KHO support for QEMU. In that case, device
> state could be preserved in memory as a memfd and potentially survive across a
> KHO kexec. We won't need to preserve anything on storage. Indeed, I did some
> internal tests using memfd to preserve device state across kexec although using
> different implementation, and it worked.

IIUC we can already do it: migrate file:/dev/shm/something, then make that
file handle persistent.

Then the question is, can we stick with that too for your feature?  Instead
of introducing the "memfd:" type, just use file?  Note that non-huge memfd
should literally be shmem at least on Linux.

Thanks,

-- 
Peter Xu