[PATCH 0/3] CPR: shared RAM with /dev/fdset for LUO kexec reboot

[PATCH 0/3] CPR: shared RAM with /dev/fdset for LUO kexec reboot

[Li Chen]

Hi,

This series improves a CPR workflow for large guests where guest RAM is
shared/external and preserved in place. With shared RAM and x-ignore-shared
enabled, the migration stream skips guest RAM pages and transfers only
non-RAM VM state (vmstate). This avoids copying guest RAM to the on-disk
migration URI, which can significantly reduce checkpoint/restore downtime for
multi-GB guests.

In the LUO/KHO update flow [1], a LUO agent coordinates a host kexec reboot
while keeping VM RAM content intact. LUO creates the guest RAM backing as a
memfd and passes it to QEMU via -add-fd on the initial launch, so that
memory-backend-file can use it as the shared RAM backing. On update, LUO
checkpoints QEMU, reboots the host kernel via kexec, and then re-launches QEMU
to restore vmstate while reusing the same preserved memfd-backed RAM FD.
Today LUO only supports handing off guest RAM via memfd [2].

To re-attach that preserved RAM backing without reopening non-persistent
paths, QEMU needs to let memory-backend-file consume the pre-opened FD using
mem-path=/dev/fdset/<id>. However, memory-backend-file currently uses
open()/creat() directly, so /dev/fdset/<id> cannot be resolved through the
fdset mechanism, making this workflow impossible.

This series allows /dev/fdset/<id> for file-backed RAM, documents the setup,
and adds qtests to validate that x-ignore-shared keeps RAM transfer minimal
in the cpr-reboot path.

[1]: https://docs.kernel.org/next/core-api/liveupdate.html
[2]: https://docs.kernel.org/mm/memfd_preservation.html

Li Chen (3):
  system/physmem: allow /dev/fdset for file-backed RAM
  docs: CPR: document shared RAM with x-ignore-shared
  tests/qtest: cpr-reboot: check ignore-shared transfer

 docs/devel/migration/CPR.rst      | 17 ++++++++--
 system/physmem.c                  | 17 ++++++++--
 tests/qtest/migration/cpr-tests.c | 56 ++++++++++++++++++++++++++++++-
 3 files changed, 84 insertions(+), 6 deletions(-)

-- 
2.52.0

[PATCH 1/3] system/physmem: allow /dev/fdset for file-backed RAM

[Li Chen]

CPR with x-ignore-shared can skip migration of shared RAM pages, so only
vmstate is transferred. This relies on guest RAM being externally managed
and re-used in place.

Allow users to pass a pre-opened RAM backing FD via -add-fd and attach it
with memory-backend-file mem-path=/dev/fdset/<id>. File-backed RAM used
plain open()/creat(), so /dev/fdset/<id> could not be resolved through the
fdset mechanism. Switch to qemu_open()/qemu_create(), return -EINVAL when
creation is attempted, and add a hint for missing -add-fd. Add a qtest that
creates a memory-backend-file object from /dev/fdset/<id>.

Signed-off-by: Li Chen <me@linux.beauty>
---
 system/physmem.c                  | 17 ++++++++++++--
 tests/qtest/migration/cpr-tests.c | 38 ++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 3 deletions(-)

diff --git a/system/physmem.c b/system/physmem.c
index c9869e4049..7717de5e7b 100644
--- a/system/physmem.c
+++ b/system/physmem.c
@@ -1650,10 +1650,14 @@ static int file_ram_open(const char *path,
     char *sanitized_name;
     char *c;
     int fd = -1;
+    const char *tmp;
+    const bool is_fdset = strstart(path, "/dev/fdset/", &tmp);
 
     *created = false;
     for (;;) {
-        fd = open(path, readonly ? O_RDONLY : O_RDWR);
+        g_autoptr(Error) local_err = NULL;
+
+        fd = qemu_open(path, readonly ? O_RDONLY : O_RDWR, &local_err);
         if (fd >= 0) {
             /*
              * open(O_RDONLY) won't fail with EISDIR. Check manually if we
@@ -1682,8 +1686,13 @@ static int file_ram_open(const char *path,
                 /* Refuse to create new, readonly files. */
                 return -ENOENT;
             }
+            if (is_fdset) {
+                /* /dev/fdset/N is a QEMU fdset handle and cannot be created. */
+                return -EINVAL;
+            }
             /* @path names a file that doesn't exist, create it */
-            fd = open(path, O_RDWR | O_CREAT | O_EXCL, 0644);
+            local_err = NULL;
+            fd = qemu_create(path, O_RDWR | O_EXCL, 0644, &local_err);
             if (fd >= 0) {
                 *created = true;
                 break;
@@ -2410,6 +2419,10 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,
     if (fd < 0) {
         error_setg_errno(errp, -fd, "can't open backing store %s for guest RAM",
                          mem_path);
+        if (g_str_has_prefix(mem_path, "/dev/fdset/")) {
+            error_append_hint(errp, "Did you forget to pass the backing FD via"
+                              " '-add-fd fd=<n>,set=<id>'?\n");
+        }
         if (!(ram_flags & RAM_READONLY_FD) && !(ram_flags & RAM_SHARED) &&
             fd == -EACCES) {
             /*
diff --git a/tests/qtest/migration/cpr-tests.c b/tests/qtest/migration/cpr-tests.c
index 2a186c6f35..5eafc4d678 100644
--- a/tests/qtest/migration/cpr-tests.c
+++ b/tests/qtest/migration/cpr-tests.c
@@ -19,6 +19,39 @@
 
 static char *tmpfs;
 
+#ifndef _WIN32
+static void test_mem_backend_file_fdset(void)
+{
+    const uint64_t size = 8 * 1024 * 1024;
+    g_autofree char *file = g_strdup_printf("%s/fdset-ram.bin", tmpfs);
+    QTestState *qts;
+    int fd;
+
+    fd = open(file, O_RDWR | O_CREAT | O_TRUNC, 0600);
+    g_assert_cmpint(fd, >=, 0);
+    g_assert_cmpint(ftruncate(fd, size), ==, 0);
+
+    qts = qtest_init("-machine none -nodefaults");
+    qtest_qmp_fds_assert_success(qts, &fd, 1,
+                                 "{'execute': 'add-fd',"
+                                 " 'arguments': { 'fdset-id': 1 } }");
+    close(fd);
+
+    qtest_qmp_assert_success(qts, "{ 'execute': 'object-add',"
+                             "  'arguments': {"
+                             "    'qom-type': 'memory-backend-file',"
+                             "    'id': 'ram0',"
+                             "    'mem-path': '/dev/fdset/1',"
+                             "    'size': %" PRIu64 " } }",
+                             size);
+    qtest_qmp_assert_success(qts, "{ 'execute': 'object-del',"
+                             "  'arguments': { 'id': 'ram0' } }");
+    qtest_quit(qts);
+
+    unlink(file);
+}
+#endif
+
 static void *migrate_hook_start_mode_reboot(QTestState *from, QTestState *to)
 {
     migrate_set_parameter_str(from, "mode", "cpr-reboot");
@@ -247,7 +280,10 @@ void migration_test_add_cpr(MigrationTestEnv *env)
 {
     tmpfs = env->tmpfs;
 
-    /* no tests in the smoke set for now */
+#ifndef _WIN32
+    migration_test_add("/migration/fdset/mem-backend-file",
+                       test_mem_backend_file_fdset);
+#endif
 
     if (!env->full_set) {
         return;
-- 
2.52.0

[PATCH 2/3] docs: CPR: document shared RAM with x-ignore-shared

[Li Chen]

Document a CPR setup where guest RAM is shared/external and preserved in
place.

With shared RAM and x-ignore-shared enabled, the migration stream skips
guest RAM pages and transfers only non-RAM VM state (vmstate).

Note that a memfd-backed RAM file is not persistent across a reboot by
itself. For a host kexec reboot, use persistent shared memory (for
example a DAX device), or use an external manager (for example Linux Live
Update Orchestrator (LUO)) to preserve the RAM backing file and re-attach
it to the new QEMU instance.

Signed-off-by: Li Chen <me@linux.beauty>
---
 docs/devel/migration/CPR.rst | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
index b6178568a8..e60ae31b45 100644
--- a/docs/devel/migration/CPR.rst
+++ b/docs/devel/migration/CPR.rst
@@ -46,9 +46,20 @@ Usage
 
 It is recommended that guest RAM be backed with some type of shared
 memory, such as ``memory-backend-file,share=on``, and that the
-``x-ignore-shared`` capability be set.  This combination allows memory
-to be saved in place.  Otherwise, after QEMU stops the VM, all guest
-RAM is copied to the migration URI.
+``x-ignore-shared`` capability be set.  With shared RAM and
+``x-ignore-shared``, the migration stream skips guest RAM pages and
+transfers only non-RAM VM state (vmstate), while guest RAM is preserved
+in place.  Otherwise, after QEMU stops the VM, all guest RAM is copied
+to the migration URI.
+
+Note that a memfd-backed RAM file is not persistent across a reboot by
+itself.  If you want to reboot to a new host kernel while keeping RAM in
+place, use persistent shared memory (for example a DAX device), or use
+an external manager (for example Linux Live Update Orchestrator (LUO))
+that preserves the RAM backing file and re-attaches it to the new QEMU
+instance (for example with ``-add-fd`` and a ``memory-backend-file``
+pointing at ``/dev/fdset/<id>``).  QEMU does not manage that
+orchestration.
 
 Outgoing:
   * Set the migration mode parameter to ``cpr-reboot``.
-- 
2.52.0

[PATCH 3/3] tests/qtest: cpr-reboot: check ignore-shared transfer

[Li Chen]

Add a cpr-reboot qtest that enables x-ignore-shared and asserts the amount
of RAM transferred stays far below the guest RAM size.

This covers the shared RAM behavior: device state is saved while shared
RAM is preserved in place.

Signed-off-by: Li Chen <me@linux.beauty>
---
 tests/qtest/migration/cpr-tests.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tests/qtest/migration/cpr-tests.c b/tests/qtest/migration/cpr-tests.c
index 5eafc4d678..6fa09b54ee 100644
--- a/tests/qtest/migration/cpr-tests.c
+++ b/tests/qtest/migration/cpr-tests.c
@@ -52,6 +52,23 @@ static void test_mem_backend_file_fdset(void)
 }
 #endif
 
+static void test_ignore_shared_end(QTestState *from, QTestState *to,
+                                   void *opaque)
+{
+    int64_t transferred;
+    int64_t total;
+    int64_t limit;
+
+    transferred = read_ram_property_int(from, "transferred");
+    total = read_ram_property_int(from, "total");
+    limit = total / 32;
+    limit = MIN(limit, 8 * 1024 * 1024);
+    limit = MAX(limit, 2 * 1024 * 1024);
+    g_test_message("ram.transferred=%" PRId64 " ram.total=%" PRId64
+                   " limit=%" PRId64, transferred, total, limit);
+    g_assert_cmpint(transferred, <, limit);
+}
+
 static void *migrate_hook_start_mode_reboot(QTestState *from, QTestState *to)
 {
     migrate_set_parameter_str(from, "mode", "cpr-reboot");
@@ -69,6 +86,7 @@ static void test_mode_reboot(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_mode_reboot,
+        .end_hook = test_ignore_shared_end,
         .start = {
             .caps[MIGRATION_CAPABILITY_X_IGNORE_SHARED] = true,
         },
-- 
2.52.0

Re: [PATCH 0/3] CPR: shared RAM with /dev/fdset for LUO kexec reboot

[Pasha Tatashin]

Hi Li,

Thanks for the series. I have a few comments regarding the interaction
with the LUO agent and the broader architecture.

> In the LUO/KHO update flow [1], a LUO agent coordinates a host kexec reboot
> while keeping VM RAM content intact. LUO creates the guest RAM backing as a
> memfd and passes it to QEMU via -add-fd on the initial launch, so that

I am currently working on the LUO agent (LUOD) documentation, and the
intended behavior differs slightly from this description.

Rather than the agent simply passing FDs, the design involves LUOD
opening /dev/liveupdate and creating a UDS socket (e.g., at
/run/luod/liveupdate.sock). Live-update-capable clients connect to
this socket, create a LUO session, and preserve their resources into
that session. Once all clients notify LUOD that they are ready, LUOD
performs the kexec reboot while keeping /dev/liveupdate open, ensuring
the LUO sessions survive.

After the reboot, the restarted clients reconnect to LUOD via UDS and
request their preserved sessions. The sessions are passed as FDs back
to the clients. From those sessions, the clients retrieve the
resources, restore them, and resume suspended operations. Finally, the
clients "finish" their sessions, fully reclaiming ownership of the
preserved resources from LUO back to userspace.

> memory-backend-file can use it as the shared RAM backing. On update, LUO
> checkpoints QEMU, reboots the host kernel via kexec, and then re-launches QEMU
> to restore vmstate while reusing the same preserved memfd-backed RAM FD.
> Today LUO only supports handing off guest RAM via memfd [2].
>
> To re-attach that preserved RAM backing without reopening non-persistent
> paths, QEMU needs to let memory-backend-file consume the pre-opened FD using
> mem-path=/dev/fdset/<id>. However, memory-backend-file currently uses
> open()/creat() directly, so /dev/fdset/<id> cannot be resolved through the
> fdset mechanism, making this workflow impossible.
>
> This series allows /dev/fdset/<id> for file-backed RAM, documents the setup,
> and adds qtests to validate that x-ignore-shared keeps RAM transfer minimal
> in the cpr-reboot path.

Sessions become particularly relevant when we look beyond guest RAM.
For complex resources like VFIO or IOMMU, simply passing the FD is
insufficient. These resources require specific ioctls to be issued by
clients after the FD is retrieved, but before vCPUs are resumed or the
session is "finished".

Pasha

Re: [PATCH 0/3] CPR: shared RAM with /dev/fdset for LUO kexec reboot

[Li Chen]

Hi Pasha,

Thanks for the detailed clarification, that's very helpful.

 ---- On Mon, 29 Dec 2025 23:50:35 +0800  Pasha Tatashin <pasha.tatashin@soleen.com> wrote --- 
 > Hi Li,
 > 
 > Thanks for the series. I have a few comments regarding the interaction
 > with the LUO agent and the broader architecture.
 > 
 > > In the LUO/KHO update flow [1], a LUO agent coordinates a host kexec reboot
 > > while keeping VM RAM content intact. LUO creates the guest RAM backing as a
 > > memfd and passes it to QEMU via -add-fd on the initial launch, so that
 > 
 > I am currently working on the LUO agent (LUOD) documentation, and the
 > intended behavior differs slightly from this description.
 > 
 > Rather than the agent simply passing FDs, the design involves LUOD
 > opening /dev/liveupdate and creating a UDS socket (e.g., at
 > /run/luod/liveupdate.sock). Live-update-capable clients connect to
 > this socket, create a LUO session, and preserve their resources into
 > that session. Once all clients notify LUOD that they are ready, LUOD
 > performs the kexec reboot while keeping /dev/liveupdate open, ensuring
 > the LUO sessions survive.
 > 
 > After the reboot, the restarted clients reconnect to LUOD via UDS and
 > request their preserved sessions. The sessions are passed as FDs back
 > to the clients. From those sessions, the clients retrieve the
 > resources, restore them, and resume suspended operations. Finally, the
 > clients "finish" their sessions, fully reclaiming ownership of the
 > preserved resources from LUO back to userspace.

Ack, and sorry for my cover letter text over-simplified the design. 

 > > memory-backend-file can use it as the shared RAM backing. On update, LUO
 > > checkpoints QEMU, reboots the host kernel via kexec, and then re-launches QEMU
 > > to restore vmstate while reusing the same preserved memfd-backed RAM FD.
 > > Today LUO only supports handing off guest RAM via memfd [2].
 > >
 > > To re-attach that preserved RAM backing without reopening non-persistent
 > > paths, QEMU needs to let memory-backend-file consume the pre-opened FD using
 > > mem-path=/dev/fdset/<id>. However, memory-backend-file currently uses
 > > open()/creat() directly, so /dev/fdset/<id> cannot be resolved through the
 > > fdset mechanism, making this workflow impossible.
 > >
 > > This series allows /dev/fdset/<id> for file-backed RAM, documents the setup,
 > > and adds qtests to validate that x-ignore-shared keeps RAM transfer minimal
 > > in the cpr-reboot path.
 > 
 > Sessions become particularly relevant when we look beyond guest RAM.
 > For complex resources like VFIO or IOMMU, simply passing the FD is
 > insufficient. These resources require specific ioctls to be issued by
 > clients after the FD is retrieved, but before vCPUs are resumed or the
 > session is "finished".

One question: in the intended LUOD/session architecture, do you still expect
clients (or a wrapper) to inject the retrieved guest RAM memfd FD into QEMU
via -add-fd + /dev/fdset/<id> at QEMU startup? (as this patchset does)

Or is the longer-term plan that QEMU itself connects to LUOD over UDS, retrieves
the session/resource FD directly, and consumes it as the RAM backing without
going through -add-fd? 

If the latter, I think we'd likely need a QEMU-side interface to
consume an inherited/pre-opened FD for RAM backing during early machine init.

Regards,

Li​

Re: [PATCH 0/3] CPR: shared RAM with /dev/fdset for LUO kexec reboot

[Pasha Tatashin]

On Tue, Dec 30, 2025 at 3:30 AM Li Chen <me@linux.beauty> wrote:
>
> Hi Pasha,
>
> Thanks for the detailed clarification, that's very helpful.
>
>  ---- On Mon, 29 Dec 2025 23:50:35 +0800  Pasha Tatashin <pasha.tatashin@soleen.com> wrote ---
>  > Hi Li,
>  >
>  > Thanks for the series. I have a few comments regarding the interaction
>  > with the LUO agent and the broader architecture.
>  >
>  > > In the LUO/KHO update flow [1], a LUO agent coordinates a host kexec reboot
>  > > while keeping VM RAM content intact. LUO creates the guest RAM backing as a
>  > > memfd and passes it to QEMU via -add-fd on the initial launch, so that
>  >
>  > I am currently working on the LUO agent (LUOD) documentation, and the
>  > intended behavior differs slightly from this description.
>  >
>  > Rather than the agent simply passing FDs, the design involves LUOD
>  > opening /dev/liveupdate and creating a UDS socket (e.g., at
>  > /run/luod/liveupdate.sock). Live-update-capable clients connect to
>  > this socket, create a LUO session, and preserve their resources into
>  > that session. Once all clients notify LUOD that they are ready, LUOD
>  > performs the kexec reboot while keeping /dev/liveupdate open, ensuring
>  > the LUO sessions survive.
>  >
>  > After the reboot, the restarted clients reconnect to LUOD via UDS and
>  > request their preserved sessions. The sessions are passed as FDs back
>  > to the clients. From those sessions, the clients retrieve the
>  > resources, restore them, and resume suspended operations. Finally, the
>  > clients "finish" their sessions, fully reclaiming ownership of the
>  > preserved resources from LUO back to userspace.
>
> Ack, and sorry for my cover letter text over-simplified the design.
>
>  > > memory-backend-file can use it as the shared RAM backing. On update, LUO
>  > > checkpoints QEMU, reboots the host kernel via kexec, and then re-launches QEMU
>  > > to restore vmstate while reusing the same preserved memfd-backed RAM FD.
>  > > Today LUO only supports handing off guest RAM via memfd [2].
>  > >
>  > > To re-attach that preserved RAM backing without reopening non-persistent
>  > > paths, QEMU needs to let memory-backend-file consume the pre-opened FD using
>  > > mem-path=/dev/fdset/<id>. However, memory-backend-file currently uses
>  > > open()/creat() directly, so /dev/fdset/<id> cannot be resolved through the
>  > > fdset mechanism, making this workflow impossible.
>  > >
>  > > This series allows /dev/fdset/<id> for file-backed RAM, documents the setup,
>  > > and adds qtests to validate that x-ignore-shared keeps RAM transfer minimal
>  > > in the cpr-reboot path.
>  >
>  > Sessions become particularly relevant when we look beyond guest RAM.
>  > For complex resources like VFIO or IOMMU, simply passing the FD is
>  > insufficient. These resources require specific ioctls to be issued by
>  > clients after the FD is retrieved, but before vCPUs are resumed or the
>  > session is "finished".
>
> One question: in the intended LUOD/session architecture, do you still expect
> clients (or a wrapper) to inject the retrieved guest RAM memfd FD into QEMU
> via -add-fd + /dev/fdset/<id> at QEMU startup? (as this patchset does)
>
> Or is the longer-term plan that QEMU itself connects to LUOD over UDS, retrieves
> the session/resource FD directly, and consumes it as the RAM backing without
> going through -add-fd?

The latter, I expect QEMU and other VMMs use their LUO session FDs
that they receive from LUOD to retrieve and restore the preserved
resources: memfd+iommufd+vfiofd and resume the suspended VMs.

>
> If the latter, I think we'd likely need a QEMU-side interface to
> consume an inherited/pre-opened FD for RAM backing during early machine init.
>
> Regards,
>
> Li
>

Re: [PATCH 0/3] CPR: shared RAM with /dev/fdset for LUO kexec reboot

[Jason Gunthorpe]

On Tue, Dec 30, 2025 at 11:42:09AM -0500, Pasha Tatashin wrote:
> > One question: in the intended LUOD/session architecture, do you still expect
> > clients (or a wrapper) to inject the retrieved guest RAM memfd FD into QEMU
> > via -add-fd + /dev/fdset/<id> at QEMU startup? (as this patchset does)
> >
> > Or is the longer-term plan that QEMU itself connects to LUOD over UDS, retrieves
> > the session/resource FD directly, and consumes it as the RAM backing without
> > going through -add-fd?
> 
> The latter, I expect QEMU and other VMMs use their LUO session FDs
> that they receive from LUOD to retrieve and restore the preserved
> resources: memfd+iommufd+vfiofd and resume the suspended VMs.

Right, and we shouldn't have something like /dev/fset/ID at all.

The VMM has to know it is doing a LUO restoration and squence
restoration of all the stuff the preserving VMM put into LUO.

The general case is far more complex than just opening a memfd.

Jason