[PATCH 00/21] migration/cpr: introduce cpr-luo mode to support KHO/LUO

[PATCH 00/21] migration/cpr: introduce cpr-luo mode to support KHO/LUO

[Dongli Zhang]

This series introduces cpr-luo, a CPR migration mode that uses the Linux
Live Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host
kernel updates through kexec.

On the outgoing side, QEMU saves CPR state in a memfd and preserves that fd
together with the RAMBlock backing fds in LUO. KHO carries the preserved LUO
state into the new kernel. The incoming QEMU retrieves the CPR state and
RAMBlock backing fds from LUO after kexec.

The regular migration stream carries device vmstate.

Although VFIO is currently blocked, I have an offline implementation that
skips cpr-luo mode in cpr_is_incoming(), skips precopy, and carefully handles
monitor_fd_param() although I never use it, so that VFIO is always
reconstructed. It works with both iommufd and legacy for mlx5_vfio_pci by
saving the VFIO and vendor specific state in vmstate as well.

The result is that guest RAM remains available across a kexec-based host
kernel update without copying its contents through the migration stream.

Here is the main idea of the patchset:

1. Introduce the new cpr-luo mode.
2. Use a UUID to identify a LUO session.
3. Introduce APIs to create, preserve, and retrieve sessions and fds.
4. As LUO uses the name, id, and token instead of an fd, create an
independent VMStateDescription rather than reuse the one for cpr-exec and
cpr-transfer.
5. Compute the token when transforming entries from the hash table into the
list.
6. Preserve fds on the source. Follow the approach used in cpr-exec and
cpr-transfer.
7. Retrieve fds on the target. Follow the approach used in cpr-exec and
cpr-transfer. The tricky part is where to set the cpr incoming mode.
8. There is no need to modify save_fd() or load_fd(), as we only need to
migrate the corresponding name and token, not the fds.
9. Handle corner-case errors.
10. Block VFIO.

I may need to rebase the current patchset on top of the patches below,
from which I learned quite a lot in each iteration. While working on this
patchset, I identified some cleanup and code issues in cpr-exec and
cpr-transfer. I may send fixes for them separately.

[PATCH v16 00/22] migration/cpr: support vhost-vsock devices
https://lore.kernel.org/all/20261012-up-vsock-vhost-v5-v16-0-36c38bd15ffc@virtuozzo.com/

Example:

Add "kho=on liveupdate=on" to host kernel cmdline.

Create QEMU instance with memfd.

qemu-system-x86_64 \
-uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \
-machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
-accel kvm -cpu host \
-name debug-threads=on \
-m 8G -smp 4 \
-object memory-backend-memfd,id=ram0,size=8G,share=on \
-device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
-drive file=boot.qcow2,if=none,id=drive0 \
-netdev user,id=net0,hostfwd=tcp::5028-:22 \
-device virtio-net-pci,netdev=net0 \
-vnc :8 -monitor stdio

Don't terminate QEMU process after cpr-luo migration.

(qemu) migrate_set_parameter mode cpr-luo
(qemu) migrate -d file:cpr-luo.state
(qemu) info status
VM status: paused (postmigrate)

Use kexec to reboot the KVM host.

host# kexec -s -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --reuse-cmdline
host# sync
host# kexec -e

Resume QEMU instance after kexec reboot.

qemu-system-x86_64 \
-uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \
-machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
-accel kvm -cpu host \
-name debug-threads=on \
-m 8G -smp 4 \
-object memory-backend-memfd,id=ram0,size=8G,share=on \
-device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
-drive file=boot.qcow2,if=none,id=drive0 \
-netdev user,id=net0,hostfwd=tcp::5028-:22 \
-device virtio-net-pci,netdev=net0 \
-vnc :8 -monitor stdio \
-incoming cpr-luo:defer

(qemu) migrate_incoming file:cpr-luo.state
(qemu) info status
VM status: running


Dongli Zhang (21):
  qapi/migration: add cpr-luo migration mode
  migration/cpr: move CprFd definition to header file
  scripts/update-linux-headers: Add liveupdate.h
  linux-headers: import liveupdate UAPI header
  migration/cpr-luo: require a non-null machine UUID
  migration/cpr-luo: add LUO session management
  migration/cpr-luo: add LUO fd management helpers
  migration/cpr-luo: track LUO fds
  migration/cpr-luo: add LUO fd VMStateDescription
  migration/cpr-luo: add tokens for LUO fds
  migration/cpr: move QEMUFile fd helpers to common code
  migration/cpr-luo: add state fd helpers
  migration/cpr: add more arguments for cpr_walk_fd()
  migration/cpr-luo: add helpers to preserve and retrieve all LUO fds
  migration/cpr-luo: enable cpr-luo mode for source side preservation
  migration/cpr-luo: enable cpr-luo mode for target side retrieval
  migration/cpr-luo: abort session when migration fails
  system/physmem: add cpr-luo blocker
  monitor/qmp: close LUO session when resuming source QEMU
  vfio: block cpr-luo migration
  migration: cpr-luo documentation

 backends/hostmem-memfd.c         |   1 +
 backends/hostmem-shm.c           |   1 +
 docs/devel/migration/CPR.rst     |  64 +++++++
 hw/vfio/cpr-iommufd.c            |  17 +-
 hw/vfio/cpr-legacy.c             |   8 +
 include/hw/vfio/vfio-cpr.h       |   1 +
 include/migration/cpr.h          |  37 +++-
 include/system/iommufd.h         |   1 +
 include/system/system.h          |   2 +
 linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++
 migration/cpr-exec.c             |  32 +---
 migration/cpr-luo-stub.c         |  78 ++++++++
 migration/cpr-luo.c              | 328 ++++++++++++++++++++++++++++++++++
 migration/cpr.c                  | 126 +++++++++++--
 migration/meson.build            |   6 +
 migration/migration.c            |  12 +-
 migration/options.c              |   6 +
 migration/ram.c                  |   1 +
 monitor/qmp-cmds.c               |   7 +
 qapi/migration.json              |  11 +-
 scripts/update-linux-headers.sh  |   2 +-
 system/globals.c                 |   5 +
 system/physmem.c                 |   9 +-
 system/vl.c                      |   9 +
 24 files changed, 950 insertions(+), 53 deletions(-)

base-commit: 3377670ee96b5a68361016680a95b9013709d4b7

Thank you very much!

Dongli Zhang

[PATCH 01/21] qapi/migration: add cpr-luo migration mode

[Dongli Zhang]

Add the cpr-luo migration mode for CPR through the Linux Live Update
Orchestrator (LUO). Subsequent changes preserve CPR state and RAMBlock
backing file descriptors in an LUO session across a host kernel update.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 qapi/migration.json | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/qapi/migration.json b/qapi/migration.json
index 889a9b89c2..ab117c3897 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -712,9 +712,18 @@
 #     with the '-machine aux-ram-share=on' option.
 #
 #     (since 10.2)
+#
+# @cpr-luo: A CPR mode that preserves guest RAM backing file
+#     descriptors and CPR state through the Linux Live Update
+#     Orchestrator (LUO) across a host kernel update.
+#
+#     This mode requires Linux with LUO support.
+#
+#     (since 11.2)
 ##
 { 'enum': 'MigMode',
-  'data': [ 'normal', 'cpr-reboot', 'cpr-transfer', 'cpr-exec' ] }
+  'data': [ 'normal', 'cpr-reboot', 'cpr-transfer', 'cpr-exec',
+            'cpr-luo' ] }
 
 ##
 # @ZeroPageDetection:
-- 
2.43.5

[PATCH 02/21] migration/cpr: move CprFd definition to header file

[Dongli Zhang]

Move the definition of struct CprFd to a header file so that it can be
referenced from more source files.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h | 8 ++++++++
 migration/cpr.c         | 8 --------
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 56fb67e6b4..e33eaf588b 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -18,6 +18,14 @@
 #define QEMU_CPR_FILE_VERSION   0x00000001
 #define CPR_STATE "CprState"
 
+typedef struct CprFd {
+    char *name;
+    unsigned int namelen;
+    int id;
+    int fd;
+    QLIST_ENTRY(CprFd) next;
+} CprFd;
+
 typedef QLIST_HEAD(CprFdList, CprFd) CprFdList;
 typedef QLIST_HEAD(CprVFIODeviceList, CprVFIODevice) CprVFIODeviceList;
 
diff --git a/migration/cpr.c b/migration/cpr.c
index bca43e9bf3..33ac74c1f3 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -28,14 +28,6 @@ static GHashTable *cpr_fds_hash;
 
 /****************************************************************************/
 
-typedef struct CprFd {
-    char *name;
-    unsigned int namelen;
-    int id;
-    int fd;
-    QLIST_ENTRY(CprFd) next;
-} CprFd;
-
 static const VMStateDescription vmstate_cpr_fd = {
     .name = "cpr fd",
     .version_id = 1,
-- 
2.43.5

[PATCH 03/21] scripts/update-linux-headers: Add liveupdate.h

[Dongli Zhang]

Update the Linux header import script to include liveupdate.h. QEMU uses
this header to issue the ioctls for session and file descriptor
preservation and retrieval.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 scripts/update-linux-headers.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/update-linux-headers.sh b/scripts/update-linux-headers.sh
index 3c3a0e9394..aeec7a9666 100755
--- a/scripts/update-linux-headers.sh
+++ b/scripts/update-linux-headers.sh
@@ -197,7 +197,7 @@ rm -rf "$output/linux-headers/linux"
 mkdir -p "$output/linux-headers/linux"
 for header in const.h stddef.h kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \
               psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h \
-              vduse.h iommufd.h bits.h mshv.h; do
+              vduse.h iommufd.h bits.h mshv.h liveupdate.h; do
     cp "$hdrdir/include/linux/$header" "$output/linux-headers/linux"
 done
 
-- 
2.43.5

[PATCH 04/21] linux-headers: import liveupdate UAPI header

[Dongli Zhang]

Import the Linux liveupdate UAPI header for cpr-luo support.
QEMU uses this header to issue the ioctls for session and file descriptor
preservation and retrieval.

It is from Linux kernel include/uapi/linux/liveupdate.h.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++++++++
 1 file changed, 239 insertions(+)
 create mode 100644 linux-headers/linux/liveupdate.h

diff --git a/linux-headers/linux/liveupdate.h b/linux-headers/linux/liveupdate.h
new file mode 100644
index 0000000000..4043d40387
--- /dev/null
+++ b/linux-headers/linux/liveupdate.h
@@ -0,0 +1,239 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+
+/*
+ * Userspace interface for /dev/liveupdate
+ * Live Update Orchestrator
+ *
+ * Copyright (c) 2025, Google LLC.
+ * Pasha Tatashin <pasha.tatashin@soleen.com>
+ */
+
+#ifndef _UAPI_LIVEUPDATE_H
+#define _UAPI_LIVEUPDATE_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ *  - ENOTTY: The IOCTL number itself is not supported at all
+ *  - E2BIG: The IOCTL number is supported, but the provided structure has
+ *    non-zero in a part the kernel does not understand.
+ *  - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ *    understood, however a known field has a value the kernel does not
+ *    understand or support.
+ *  - EINVAL: Everything about the IOCTL was understood, but a field is not
+ *    correct.
+ *  - ENOENT: A provided token does not exist.
+ *  - ENOMEM: Out of memory.
+ *  - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+
+/* The ioctl type, documented in ioctl-number.rst */
+#define LIVEUPDATE_IOCTL_TYPE		0xBA
+
+/* The maximum length of session name including null termination */
+#define LIVEUPDATE_SESSION_NAME_LENGTH 64
+
+/* The /dev/liveupdate ioctl commands */
+enum {
+	LIVEUPDATE_CMD_BASE = 0x00,
+	LIVEUPDATE_CMD_CREATE_SESSION = LIVEUPDATE_CMD_BASE,
+	LIVEUPDATE_CMD_RETRIEVE_SESSION = 0x01,
+};
+
+/* ioctl commands for session file descriptors */
+enum {
+	LIVEUPDATE_CMD_SESSION_BASE = 0x40,
+	LIVEUPDATE_CMD_SESSION_PRESERVE_FD = LIVEUPDATE_CMD_SESSION_BASE,
+	LIVEUPDATE_CMD_SESSION_RETRIEVE_FD = 0x41,
+	LIVEUPDATE_CMD_SESSION_FINISH = 0x42,
+	LIVEUPDATE_CMD_SESSION_GET_NAME = 0x43,
+};
+
+/**
+ * struct liveupdate_ioctl_create_session - ioctl(LIVEUPDATE_IOCTL_CREATE_SESSION)
+ * @size:	Input; sizeof(struct liveupdate_ioctl_create_session)
+ * @fd:		Output; The new file descriptor for the created session.
+ * @name:	Input; A null-terminated string for the session name, max
+ *		length %LIVEUPDATE_SESSION_NAME_LENGTH including termination
+ *		character.
+ *
+ * Creates a new live update session for managing preserved resources.
+ * This ioctl can only be called on the main /dev/liveupdate device.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+struct liveupdate_ioctl_create_session {
+	__u32		size;
+	__s32		fd;
+	__u8		name[LIVEUPDATE_SESSION_NAME_LENGTH];
+};
+
+#define LIVEUPDATE_IOCTL_CREATE_SESSION					\
+	_IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_CREATE_SESSION)
+
+/**
+ * struct liveupdate_ioctl_retrieve_session - ioctl(LIVEUPDATE_IOCTL_RETRIEVE_SESSION)
+ * @size:    Input; sizeof(struct liveupdate_ioctl_retrieve_session)
+ * @fd:      Output; The new file descriptor for the retrieved session.
+ * @name:    Input; A null-terminated string identifying the session to retrieve.
+ *           The name must exactly match the name used when the session was
+ *           created in the previous kernel.
+ *
+ * Retrieves a handle (a new file descriptor) for a preserved session by its
+ * name. This is the primary mechanism for a userspace agent to regain control
+ * of its preserved resources after a live update.
+ *
+ * The userspace application provides the null-terminated `name` of a session
+ * it created before the live update. If a preserved session with a matching
+ * name is found, the kernel instantiates it and returns a new file descriptor
+ * in the `fd` field. This new session FD can then be used for all file-specific
+ * operations, such as restoring individual file descriptors with
+ * LIVEUPDATE_SESSION_RETRIEVE_FD.
+ *
+ * It is the responsibility of the userspace application to know the names of
+ * the sessions it needs to retrieve. If no session with the given name is
+ * found, the ioctl will fail with -ENOENT.
+ *
+ * This ioctl can only be called on the main /dev/liveupdate device when the
+ * system is in the LIVEUPDATE_STATE_UPDATED state.
+ */
+struct liveupdate_ioctl_retrieve_session {
+	__u32		size;
+	__s32		fd;
+	__u8		name[LIVEUPDATE_SESSION_NAME_LENGTH];
+};
+
+#define LIVEUPDATE_IOCTL_RETRIEVE_SESSION \
+	_IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_RETRIEVE_SESSION)
+
+/* Session specific IOCTLs */
+
+/**
+ * struct liveupdate_session_preserve_fd - ioctl(LIVEUPDATE_SESSION_PRESERVE_FD)
+ * @size:  Input; sizeof(struct liveupdate_session_preserve_fd)
+ * @fd:    Input; The user-space file descriptor to be preserved.
+ * @token: Input; An opaque, unique token for preserved resource.
+ *
+ * Holds parameters for preserving a file descriptor.
+ *
+ * User sets the @fd field identifying the file descriptor to preserve
+ * (e.g., memfd, kvm, iommufd, VFIO). The kernel validates if this FD type
+ * and its dependencies are supported for preservation. If validation passes,
+ * the kernel marks the FD internally and *initiates the process* of preparing
+ * its state for saving. The actual snapshotting of the state typically occurs
+ * during the subsequent %LIVEUPDATE_IOCTL_PREPARE execution phase, though
+ * some finalization might occur during freeze.
+ * On successful validation and initiation, the kernel uses the @token
+ * field with an opaque identifier representing the resource being preserved.
+ * This token confirms the FD is targeted for preservation and is required for
+ * the subsequent %LIVEUPDATE_SESSION_RETRIEVE_FD call after the live update.
+ *
+ * Return: 0 on success (validation passed, preservation initiated), negative
+ * error code on failure (e.g., unsupported FD type, dependency issue,
+ * validation failed).
+ */
+struct liveupdate_session_preserve_fd {
+	__u32		size;
+	__s32		fd;
+	__aligned_u64	token;
+};
+
+#define LIVEUPDATE_SESSION_PRESERVE_FD					\
+	_IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_PRESERVE_FD)
+
+/**
+ * struct liveupdate_session_retrieve_fd - ioctl(LIVEUPDATE_SESSION_RETRIEVE_FD)
+ * @size:  Input; sizeof(struct liveupdate_session_retrieve_fd)
+ * @fd:    Output; The new file descriptor representing the fully restored
+ *         kernel resource.
+ * @token: Input; An opaque, token that was used to preserve the resource.
+ *
+ * Retrieve a previously preserved file descriptor.
+ *
+ * User sets the @token field to the value obtained from a successful
+ * %LIVEUPDATE_IOCTL_FD_PRESERVE call before the live update. On success,
+ * the kernel restores the state (saved during the PREPARE/FREEZE phases)
+ * associated with the token and populates the @fd field with a new file
+ * descriptor referencing the restored resource in the current (new) kernel.
+ * This operation must be performed *before* signaling completion via
+ * %LIVEUPDATE_IOCTL_FINISH. If a retrieve of a token fails, subsequent
+ * attempts to retrieve the token fail with the same error code. Failed
+ * retrieves are not retried.
+ *
+ * Return: 0 on success, negative error code on failure (e.g., invalid token).
+ */
+struct liveupdate_session_retrieve_fd {
+	__u32		size;
+	__s32		fd;
+	__aligned_u64	token;
+};
+
+#define LIVEUPDATE_SESSION_RETRIEVE_FD					\
+	_IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_RETRIEVE_FD)
+
+/**
+ * struct liveupdate_session_finish - ioctl(LIVEUPDATE_SESSION_FINISH)
+ * @size:     Input; sizeof(struct liveupdate_session_finish)
+ * @reserved: Input; Must be zero. Reserved for future use.
+ *
+ * Signals the completion of the restoration process for a retrieved session.
+ * This is the final operation that should be performed on a session file
+ * descriptor after a live update.
+ *
+ * This ioctl must be called once all required file descriptors for the session
+ * have been successfully retrieved (using %LIVEUPDATE_SESSION_RETRIEVE_FD) and
+ * are fully restored from the userspace and kernel perspective.
+ *
+ * Upon success, the kernel releases its ownership of the preserved resources
+ * associated with this session. This allows internal resources to be freed,
+ * typically by decrementing reference counts on the underlying preserved
+ * objects.
+ *
+ * If this operation fails, the resources remain preserved in memory. Userspace
+ * may attempt to call finish again. The resources will otherwise be reset
+ * during the next live update cycle.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+struct liveupdate_session_finish {
+	__u32		size;
+	__u32		reserved;
+};
+
+#define LIVEUPDATE_SESSION_FINISH					\
+	_IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_FINISH)
+
+/**
+ * struct liveupdate_session_get_name - ioctl(LIVEUPDATE_SESSION_GET_NAME)
+ * @size:  Input; sizeof(struct liveupdate_session_get_name)
+ * @reserved: Input; Must be zero. Reserved for future use.
+ * @name:  Output; A null-terminated string with the full session name.
+ *
+ * Retrieves the full name of the session associated with this file descriptor.
+ * This is useful because the kernel may truncate the name shown in /proc.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+struct liveupdate_session_get_name {
+	__u32		size;
+	__u32		reserved;
+	__u8		name[LIVEUPDATE_SESSION_NAME_LENGTH];
+};
+
+#define LIVEUPDATE_SESSION_GET_NAME					\
+	_IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_GET_NAME)
+
+#endif /* _UAPI_LIVEUPDATE_H */
-- 
2.43.5

Re: [PATCH 04/21] linux-headers: import liveupdate UAPI header

[Cornelia Huck]

On Mon, Oct 12 2026, Dongli Zhang <dongli.zhang@oracle.com> wrote:

> Import the Linux liveupdate UAPI header for cpr-luo support.
> QEMU uses this header to issue the ioctls for session and file descriptor
> preservation and retrieval.
>
> It is from Linux kernel include/uapi/linux/liveupdate.h.
>
> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
> ---
>  linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++++++++
>  1 file changed, 239 insertions(+)
>  create mode 100644 linux-headers/linux/liveupdate.h

Please make this a full headers update, instead of just adding one file.

Re: [PATCH 04/21] linux-headers: import liveupdate UAPI header

[Dongli Zhang]

On Mon, Oct 12, 2026 11:35:43PM -0700, Cornelia Huck wrote:
> On Mon, Oct 12 2026, Dongli Zhang <dongli.zhang@oracle.com> wrote:
> 
>> Import the Linux liveupdate UAPI header for cpr-luo support.
>> QEMU uses this header to issue the ioctls for session and file descriptor
>> preservation and retrieval.
>>
>> It is from Linux kernel include/uapi/linux/liveupdate.h.
>>
>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>> ---
>>  linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++++++++
>>  1 file changed, 239 insertions(+)
>>  create mode 100644 linux-headers/linux/liveupdate.h
> 
> Please make this a full headers update, instead of just adding one file.
> 
> 

I only edited scripts/update-linux-headers.sh in patch 0003 and expected
maintainers to update the header files in the future. That's why I copied just
one file.

I will make this a full header update using scripts/update-linux-headers.sh with
the most recent Linux kernel tag.

Thank you very much!

Dongli Zhang

[PATCH 05/21] migration/cpr-luo: require a non-null machine UUID

[Dongli Zhang]

As we are going to use the QEMU machine UUID to identify a LUO session,
reject cpr-luo when the machine UUID is not explicitly set or is the nil
UUID.

Validate this when cpr-luo mode is selected and also immediately before
starting migration.

(qemu) migrate_set_parameter mode cpr-luo
Error: cpr-luo requires a non-null machine UUID

So far, this commit has no effect because migrate_mode_is_cpr() does not
include cpr-luo. Add cpr-luo to migrate_mode_is_cpr() in a later commit,
when cpr-luo is fully supported.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
There is a duplicate check for cpr-exec mode. I may remove it and send the
change as an independent patchset.

 include/system/system.h | 2 ++
 migration/migration.c   | 6 ++++++
 migration/options.c     | 6 ++++++
 system/globals.c        | 5 +++++
 4 files changed, 19 insertions(+)

diff --git a/include/system/system.h b/include/system/system.h
index 03a2d0e900..bd3fd24cbc 100644
--- a/include/system/system.h
+++ b/include/system/system.h
@@ -13,6 +13,8 @@ extern const char *qemu_name;
 extern QemuUUID qemu_uuid;
 extern bool qemu_uuid_set;
 
+bool qemu_machine_uuid_is_set(void);
+
 const char *qemu_get_vm_name(void);
 
 /* Exit notifiers will run with BQL held. */
diff --git a/migration/migration.c b/migration/migration.c
index fab7949fe1..6a51f212ca 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2090,6 +2090,12 @@ static bool migrate_prepare(MigrationState *s, bool resume, Error **errp)
             error_setg(errp, "Parameter 'cpr-exec-command' required for cpr-exec");
             return false;
         }
+
+        if (migrate_mode() == MIG_MODE_CPR_LUO &&
+            !qemu_machine_uuid_is_set()) {
+            error_setg(errp, "cpr-luo requires a non-null machine UUID");
+            return false;
+        }
     }
 
     if (migrate_init(s, errp)) {
diff --git a/migration/options.c b/migration/options.c
index 46d040fc3f..7d3504a100 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -27,6 +27,7 @@
 #include "qobject/qdict.h"
 #include "qobject/qnull.h"
 #include "system/runstate.h"
+#include "system/system.h"
 #include "migration/colo.h"
 #include "migration/cpr.h"
 #include "migration/misc.h"
@@ -1402,6 +1403,11 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
         return false;
     }
 
+    if (params->mode == MIG_MODE_CPR_LUO && !qemu_machine_uuid_is_set()) {
+        error_setg(errp, "cpr-luo requires a non-null machine UUID");
+        return false;
+    }
+
     return true;
 }
 
diff --git a/system/globals.c b/system/globals.c
index 34fd3ce9c7..d57011d168 100644
--- a/system/globals.c
+++ b/system/globals.c
@@ -67,6 +67,11 @@ int only_migratable; /* turn it off unless user states otherwise */
 QemuUUID qemu_uuid;
 bool qemu_uuid_set;
 
+bool qemu_machine_uuid_is_set(void)
+{
+    return qemu_uuid_set && !qemu_uuid_is_null(&qemu_uuid);
+}
+
 uint32_t xen_domid;
 enum xen_mode xen_mode = XEN_DISABLED;
 bool xen_domid_restrict;
-- 
2.43.5

[PATCH 06/21] migration/cpr-luo: add LUO session management

[Dongli Zhang]

Add helpers to create and retrieve Live Update Orchestrator (LUO)
sessions.

The LUO API is available only on Linux. Provide error stubs on other
hosts.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h  |  3 ++
 migration/cpr-luo-stub.c | 26 ++++++++++++++
 migration/cpr-luo.c      | 74 ++++++++++++++++++++++++++++++++++++++++
 migration/meson.build    |  6 ++++
 4 files changed, 109 insertions(+)
 create mode 100644 migration/cpr-luo-stub.c
 create mode 100644 migration/cpr-luo.c

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index e33eaf588b..f8b161b3d6 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -74,4 +74,7 @@ bool cpr_exec_persist_state(QEMUFile *f, Error **errp);
 bool cpr_exec_has_state(void);
 void cpr_exec_unpersist_state(void);
 void cpr_exec_unpreserve_fds(void);
+
+int cpr_luo_create_session(const char *name, Error **errp);
+int cpr_luo_retrieve_session(const char *name, Error **errp);
 #endif
diff --git a/migration/cpr-luo-stub.c b/migration/cpr-luo-stub.c
new file mode 100644
index 0000000000..ca060ae8c8
--- /dev/null
+++ b/migration/cpr-luo-stub.c
@@ -0,0 +1,26 @@
+/*
+ * Copyright (c) 2026 Oracle and/or its affiliates.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+
+#include "migration/cpr.h"
+#include "qapi/error.h"
+
+static int cpr_luo_unsupported(Error **errp)
+{
+    error_setg(errp, "cpr-luo requires Linux");
+    return -1;
+}
+
+int cpr_luo_create_session(const char *name, Error **errp)
+{
+    return cpr_luo_unsupported(errp);
+}
+
+int cpr_luo_retrieve_session(const char *name, Error **errp)
+{
+    return cpr_luo_unsupported(errp);
+}
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
new file mode 100644
index 0000000000..48288ed5f9
--- /dev/null
+++ b/migration/cpr-luo.c
@@ -0,0 +1,74 @@
+/*
+ * Copyright (c) 2026 Oracle and/or its affiliates.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+
+#include <sys/ioctl.h>
+
+#include "linux-headers/linux/liveupdate.h"
+#include "migration/cpr.h"
+#include "qapi/error.h"
+
+static int luo_open(Error **errp)
+{
+    return qemu_open("/dev/liveupdate", O_RDWR, errp);
+}
+
+int cpr_luo_create_session(const char *name, Error **errp)
+{
+    struct liveupdate_ioctl_create_session create = { 0 };
+    int fd;
+
+    if (g_strlcpy((char *)create.name, name, sizeof(create.name)) >=
+        sizeof(create.name)) {
+        error_setg(errp, "LUO session name is too long");
+        return -1;
+    }
+
+    fd = luo_open(errp);
+    if (fd < 0) {
+        return -1;
+    }
+
+    create.size = sizeof(create);
+    if (ioctl(fd, LIVEUPDATE_IOCTL_CREATE_SESSION, &create) < 0) {
+        error_setg_errno(errp, errno, "failed to create LUO session '%s'",
+                         (char *)create.name);
+        close(fd);
+        return -1;
+    }
+
+    close(fd);
+    return create.fd;
+}
+
+int cpr_luo_retrieve_session(const char *name, Error **errp)
+{
+    struct liveupdate_ioctl_retrieve_session retrieve = { 0 };
+    int fd;
+
+    if (g_strlcpy((char *)retrieve.name, name, sizeof(retrieve.name)) >=
+        sizeof(retrieve.name)) {
+        error_setg(errp, "LUO session name is too long");
+        return -1;
+    }
+
+    fd = luo_open(errp);
+    if (fd < 0) {
+        return -1;
+    }
+
+    retrieve.size = sizeof(retrieve);
+    if (ioctl(fd, LIVEUPDATE_IOCTL_RETRIEVE_SESSION, &retrieve) < 0) {
+        error_setg_errno(errp, errno, "failed to retrieve LUO session '%s'",
+                         (char *)retrieve.name);
+        close(fd);
+        return -1;
+    }
+
+    close(fd);
+    return retrieve.fd;
+}
diff --git a/migration/meson.build b/migration/meson.build
index 4ab9d99cea..a6363c2bf7 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -54,3 +54,9 @@ system_ss.add(when: qatzip, if_true: files('multifd-qatzip.c'))
 system_ss.add(when: 'CONFIG_VFIO',
               if_true: files('vfio.c'))
 stub_ss.add(files('vfio-stub.c'))
+
+if host_os == 'linux'
+  system_ss.add(files('cpr-luo.c'))
+else
+  system_ss.add(files('cpr-luo-stub.c'))
+endif
-- 
2.43.5

[PATCH 07/21] migration/cpr-luo: add LUO fd management helpers

[Dongli Zhang]

Add helpers to preserve and retrieve LUO fds by calling
LIVEUPDATE_SESSION_PRESERVE_FD or LIVEUPDATE_SESSION_RETRIEVE_FD.

The LUO API is available only on Linux. Provide error stubs on other OSes.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h  |  3 +++
 migration/cpr-luo-stub.c | 11 +++++++++++
 migration/cpr-luo.c      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 47 insertions(+)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index f8b161b3d6..5f30df7ec8 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -77,4 +77,7 @@ void cpr_exec_unpreserve_fds(void);
 
 int cpr_luo_create_session(const char *name, Error **errp);
 int cpr_luo_retrieve_session(const char *name, Error **errp);
+int cpr_luo_preserve_fd(int session_fd, int fd, uint64_t token,
+                        Error **errp);
+int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp);
 #endif
diff --git a/migration/cpr-luo-stub.c b/migration/cpr-luo-stub.c
index ca060ae8c8..ab64e1f99b 100644
--- a/migration/cpr-luo-stub.c
+++ b/migration/cpr-luo-stub.c
@@ -24,3 +24,14 @@ int cpr_luo_retrieve_session(const char *name, Error **errp)
 {
     return cpr_luo_unsupported(errp);
 }
+
+int cpr_luo_preserve_fd(int session_fd, int fd, uint64_t token,
+                        Error **errp)
+{
+    return cpr_luo_unsupported(errp);
+}
+
+int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp)
+{
+    return cpr_luo_unsupported(errp);
+}
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
index 48288ed5f9..f6e0e23657 100644
--- a/migration/cpr-luo.c
+++ b/migration/cpr-luo.c
@@ -72,3 +72,36 @@ int cpr_luo_retrieve_session(const char *name, Error **errp)
     close(fd);
     return retrieve.fd;
 }
+
+int cpr_luo_preserve_fd(int session_fd, int fd, uint64_t token,
+                        Error **errp)
+{
+    struct liveupdate_session_preserve_fd preserve = {
+        .size = sizeof(preserve),
+        .fd = fd,
+        .token = token,
+    };
+
+    if (ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &preserve) < 0) {
+        error_setg_errno(errp, errno, "failed to preserve fd %d with LUO",
+                         fd);
+        return -1;
+    }
+
+    return 0;
+}
+
+int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp)
+{
+    struct liveupdate_session_retrieve_fd retrieve = {
+        .size = sizeof(retrieve),
+        .token = token,
+    };
+
+    if (ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &retrieve) < 0) {
+        error_setg_errno(errp, errno, "failed to retrieve LUO fd");
+        return -1;
+    }
+
+    return retrieve.fd;
+}
-- 
2.43.5

[PATCH 08/21] migration/cpr-luo: track LUO fds

[Dongli Zhang]

Track whether saved CPR file descriptors can be used for LUO. Currently,
only RAMBlock fds (RAM or device/firmware related memory) are used for LUO.

To find the locations to mark a CprFd as LUO, I simply searched for all
direct and indirect callers of cpr_save_fd(). Most of them are for VFIO
and can be ignored.

Another option is to introduce additional argument to existing
cpr_save_fd(). Just add additional function for simplicity.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 backends/hostmem-memfd.c | 1 +
 backends/hostmem-shm.c   | 1 +
 include/migration/cpr.h  | 2 ++
 migration/cpr.c          | 8 ++++++++
 system/physmem.c         | 1 +
 5 files changed, 13 insertions(+)

diff --git a/backends/hostmem-memfd.c b/backends/hostmem-memfd.c
index 666593ff6c..c47bb09ba2 100644
--- a/backends/hostmem-memfd.c
+++ b/backends/hostmem-memfd.c
@@ -85,6 +85,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
         return false;
     }
     cpr_save_fd(name, 0, fd);
+    cpr_mark_fd_luo(name, 0);
 
 have_fd:
     backend->aligned = true;
diff --git a/backends/hostmem-shm.c b/backends/hostmem-shm.c
index 4766db6aad..77ec329f70 100644
--- a/backends/hostmem-shm.c
+++ b/backends/hostmem-shm.c
@@ -49,6 +49,7 @@ shm_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
         return false;
     }
     cpr_save_fd(backend_name, 0, fd);
+    cpr_mark_fd_luo(backend_name, 0);
 
 have_fd:
     /* Let's do the same as memory-backend-ram,share=on would do. */
diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 5f30df7ec8..425f6ee956 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -23,6 +23,7 @@ typedef struct CprFd {
     unsigned int namelen;
     int id;
     int fd;
+    bool luo;
     QLIST_ENTRY(CprFd) next;
 } CprFd;
 
@@ -37,6 +38,7 @@ typedef struct CprState {
 extern CprState cpr_state;
 
 void cpr_save_fd(const char *name, int id, int fd);
+void cpr_mark_fd_luo(const char *name, int id);
 void cpr_delete_fd(const char *name, int id);
 int cpr_find_fd(const char *name, int id);
 void cpr_resave_fd(const char *name, int id, int fd);
diff --git a/migration/cpr.c b/migration/cpr.c
index 33ac74c1f3..9b6b85cedd 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -141,6 +141,14 @@ static CprFd *find_fd(const char *name, int id)
     return g_hash_table_lookup(get_cpr_fds_hash(), &key);
 }
 
+void cpr_mark_fd_luo(const char *name, int id)
+{
+    CprFd *elem = find_fd(name, id);
+
+    g_assert(elem);
+    elem->luo = true;
+}
+
 void cpr_delete_fd(const char *name, int id)
 {
     CprFd key = {
diff --git a/system/physmem.c b/system/physmem.c
index db3dda563d..263acb5727 100644
--- a/system/physmem.c
+++ b/system/physmem.c
@@ -2458,6 +2458,7 @@ static int qemu_ram_get_shared_fd(const char *name, bool *reused, Error **errp)
 
     if (fd >= 0) {
         cpr_save_fd(name, 0, fd);
+        cpr_mark_fd_luo(name, 0);
     }
     *reused = false;
     return fd;
-- 
2.43.5

[PATCH 09/21] migration/cpr-luo: add LUO fd VMStateDescription

[Dongli Zhang]

Instead of reusing vmstate_cpr_fd and vmstate_cpr_state, add a new cpr-luo
specific VMStateDescription for LUO file descriptors.

In this way, we won't need to worry about version and compatibility issues.

As only LUO file descriptors are migrated to the new instance, we don't
need to save or load CprFd->luo.

When cpr-exec or cpr-transfer is used, the original vmstate_cpr_fd is used.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h |  1 +
 migration/cpr.c         | 46 +++++++++++++++++++++++++++++++++++++++--
 2 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 425f6ee956..3bfe91f10f 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -17,6 +17,7 @@
 #define QEMU_CPR_FILE_MAGIC     0x51435052
 #define QEMU_CPR_FILE_VERSION   0x00000001
 #define CPR_STATE "CprState"
+#define CPR_LUO_STATE "CprLuoState"
 
 typedef struct CprFd {
     char *name;
diff --git a/migration/cpr.c b/migration/cpr.c
index 9b6b85cedd..eaaca6d47d 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -41,6 +41,18 @@ static const VMStateDescription vmstate_cpr_fd = {
     }
 };
 
+static const VMStateDescription vmstate_cpr_luo_fd = {
+    .name = "cpr luo fd",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_UINT32(namelen, CprFd),
+        VMSTATE_VBUFFER_ALLOC_UINT32(name, CprFd, 0, NULL, namelen),
+        VMSTATE_INT32(id, CprFd),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static guint cpr_fd_hash(gconstpointer v)
 {
     const CprFd *elem = v;
@@ -85,11 +97,16 @@ static int cpr_fd_pre_save(void *opaque)
     CprState *state = (CprState *)opaque;
     GHashTableIter iter;
     CprFd *elem;
+    MigMode mode = migrate_mode();
 
     QLIST_INIT(&state->fds);
 
     g_hash_table_iter_init(&iter, get_cpr_fds_hash());
     while (g_hash_table_iter_next(&iter, (gpointer *)&elem, NULL)) {
+        if (mode == MIG_MODE_CPR_LUO && !elem->luo) {
+            continue;
+        }
+
         QLIST_INSERT_HEAD(&state->fds, elem, next);
     }
 
@@ -104,6 +121,10 @@ static int cpr_fd_post_load(void *opaque, int version_id)
     while ((elem = QLIST_FIRST(&state->fds))) {
         QLIST_REMOVE(elem, next);
 
+        if (cpr_get_incoming_mode() == MIG_MODE_CPR_LUO) {
+            elem->luo = true;
+        }
+
         /*
          * Preserve legacy QLIST lookup semantics if duplicate keys exist in
          * the incoming stream: the first matching entry wins.
@@ -214,6 +235,18 @@ bool cpr_walk_fd(cpr_walk_fd_cb cb)
     return true;
 }
 
+static const VMStateDescription vmstate_cpr_luo_state = {
+    .name = CPR_LUO_STATE,
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .pre_save = cpr_fd_pre_save,
+    .post_load = cpr_fd_post_load,
+    .fields = (VMStateField[]) {
+        VMSTATE_QLIST_V(fds, CprState, 1, vmstate_cpr_luo_fd, CprFd, next),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 /*************************************************************************/
 static const VMStateDescription vmstate_cpr_state = {
     .name = CPR_STATE,
@@ -256,6 +289,15 @@ bool cpr_is_incoming(void)
     return incoming_mode != MIG_MODE_NONE;
 }
 
+static const VMStateDescription *cpr_state_vmsd(MigMode mode)
+{
+    if (mode == MIG_MODE_CPR_LUO) {
+        return &vmstate_cpr_luo_state;
+    }
+
+    return &vmstate_cpr_state;
+}
+
 bool cpr_state_save(MigrationChannel *channel, Error **errp)
 {
     QEMUFile *f;
@@ -278,7 +320,7 @@ bool cpr_state_save(MigrationChannel *channel, Error **errp)
     qemu_put_be32(f, QEMU_CPR_FILE_MAGIC);
     qemu_put_be32(f, QEMU_CPR_FILE_VERSION);
 
-    if (!vmstate_save_vmsd(f, &vmstate_cpr_state, &cpr_state, 0, errp)) {
+    if (!vmstate_save_vmsd(f, cpr_state_vmsd(mode), &cpr_state, 0, errp)) {
         qemu_fclose(f);
         return false;
     }
@@ -340,7 +382,7 @@ bool cpr_state_load(MigrationChannel *channel, Error **errp)
         return false;
     }
 
-    if (!vmstate_load_vmsd(f, &vmstate_cpr_state, &cpr_state, 1, errp)) {
+    if (!vmstate_load_vmsd(f, cpr_state_vmsd(mode), &cpr_state, 1, errp)) {
         qemu_fclose(f);
         return false;
     }
-- 
2.43.5

[PATCH 10/21] migration/cpr-luo: add tokens for LUO fds

[Dongli Zhang]

LUO tokens are required as keys to preserve and retrieve LUO fds.

Instead of assigning LUO tokens during fd creation, assign monotonically
increasing LUO tokens to each CprFd only when serializing the CprFd.

We reserve token 1 for the CPR LUO state for later use. Therefore, regular
LUO fd tokens start at 2.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h | 2 ++
 migration/cpr.c         | 7 +++++++
 2 files changed, 9 insertions(+)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 3bfe91f10f..77e753a278 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -18,6 +18,7 @@
 #define QEMU_CPR_FILE_VERSION   0x00000001
 #define CPR_STATE "CprState"
 #define CPR_LUO_STATE "CprLuoState"
+#define CPR_LUO_STATE_TOKEN 1
 
 typedef struct CprFd {
     char *name;
@@ -25,6 +26,7 @@ typedef struct CprFd {
     int id;
     int fd;
     bool luo;
+    uint64_t luo_token;
     QLIST_ENTRY(CprFd) next;
 } CprFd;
 
diff --git a/migration/cpr.c b/migration/cpr.c
index eaaca6d47d..700e833003 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -49,6 +49,7 @@ static const VMStateDescription vmstate_cpr_luo_fd = {
         VMSTATE_UINT32(namelen, CprFd),
         VMSTATE_VBUFFER_ALLOC_UINT32(name, CprFd, 0, NULL, namelen),
         VMSTATE_INT32(id, CprFd),
+        VMSTATE_UINT64(luo_token, CprFd),
         VMSTATE_END_OF_LIST()
     }
 };
@@ -98,6 +99,7 @@ static int cpr_fd_pre_save(void *opaque)
     GHashTableIter iter;
     CprFd *elem;
     MigMode mode = migrate_mode();
+    uint64_t luo_token = CPR_LUO_STATE_TOKEN + 1;
 
     QLIST_INIT(&state->fds);
 
@@ -107,6 +109,10 @@ static int cpr_fd_pre_save(void *opaque)
             continue;
         }
 
+        if (mode == MIG_MODE_CPR_LUO) {
+            elem->luo_token = luo_token++;
+        }
+
         QLIST_INSERT_HEAD(&state->fds, elem, next);
     }
 
@@ -149,6 +155,7 @@ void cpr_save_fd(const char *name, int id, int fd)
     elem->namelen = strlen(name) + 1;
     elem->id = id;
     elem->fd = fd;
+    elem->luo_token = 0;
     cpr_fd_hash_insert(elem);
 }
 
-- 
2.43.5

[PATCH 11/21] migration/cpr: move QEMUFile fd helpers to common code

[Dongli Zhang]

Move the helpers qemu_file_new_fd_input() and qemu_file_new_fd_output(),
that create input and output QEMUFiles backed by existing file descriptors
from cpr-exec to common CPR code. This allows cpr-luo to reuse them.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h |  3 +++
 migration/cpr-exec.c    | 20 ++------------------
 migration/cpr.c         | 17 +++++++++++++++++
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 77e753a278..23751fc015 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -60,6 +60,9 @@ bool cpr_state_load(MigrationChannel *channel, Error **errp);
 void cpr_state_close(void);
 struct QIOChannel *cpr_state_ioc(void);
 
+QEMUFile *cpr_file_new_fd_input(int fd, const char *name);
+QEMUFile *cpr_file_new_fd_output(int fd, const char *name);
+
 bool cpr_incoming_needed(void *opaque);
 int cpr_get_fd_param(const char *name, const char *fdname, int index,
                      Error **errp);
diff --git a/migration/cpr-exec.c b/migration/cpr-exec.c
index daa50916d2..b31ec3e59f 100644
--- a/migration/cpr-exec.c
+++ b/migration/cpr-exec.c
@@ -24,22 +24,6 @@
 
 #define CPR_EXEC_STATE_NAME "QEMU_CPR_EXEC_STATE"
 
-static QEMUFile *qemu_file_new_fd_input(int fd, const char *name)
-{
-    g_autoptr(QIOChannelFile) fioc = qio_channel_file_new_fd(fd);
-    QIOChannel *ioc = QIO_CHANNEL(fioc);
-    qio_channel_set_name(ioc, name);
-    return qemu_file_new_input(ioc);
-}
-
-static QEMUFile *qemu_file_new_fd_output(int fd, const char *name)
-{
-    g_autoptr(QIOChannelFile) fioc = qio_channel_file_new_fd(fd);
-    QIOChannel *ioc = QIO_CHANNEL(fioc);
-    qio_channel_set_name(ioc, name);
-    return qemu_file_new_output(ioc);
-}
-
 bool cpr_exec_persist_state(QEMUFile *f, Error **errp)
 {
     QIOChannelFile *fioc = QIO_CHANNEL_FILE(qemu_file_get_ioc(f));
@@ -99,7 +83,7 @@ QEMUFile *cpr_exec_output(Error **errp)
         return NULL;
     }
 
-    return qemu_file_new_fd_output(mfd, CPR_EXEC_STATE_NAME);
+    return cpr_file_new_fd_output(mfd, CPR_EXEC_STATE_NAME);
 }
 
 QEMUFile *cpr_exec_input(Error **errp)
@@ -107,7 +91,7 @@ QEMUFile *cpr_exec_input(Error **errp)
     int mfd = cpr_exec_find_state();
 
     lseek(mfd, 0, SEEK_SET);
-    return qemu_file_new_fd_input(mfd, CPR_EXEC_STATE_NAME);
+    return cpr_file_new_fd_input(mfd, CPR_EXEC_STATE_NAME);
 }
 
 static bool preserve_fd(int fd)
diff --git a/migration/cpr.c b/migration/cpr.c
index 700e833003..3e8e94e5fe 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -10,6 +10,7 @@
 #include "qapi/error.h"
 #include "qemu/error-report.h"
 #include "hw/vfio/vfio-cpr.h"
+#include "io/channel-file.h"
 #include "migration/cpr.h"
 #include "migration/misc.h"
 #include "migration/options.h"
@@ -242,6 +243,22 @@ bool cpr_walk_fd(cpr_walk_fd_cb cb)
     return true;
 }
 
+QEMUFile *cpr_file_new_fd_input(int fd, const char *name)
+{
+    g_autoptr(QIOChannelFile) fioc = qio_channel_file_new_fd(fd);
+    QIOChannel *ioc = QIO_CHANNEL(fioc);
+    qio_channel_set_name(ioc, name);
+    return qemu_file_new_input(ioc);
+}
+
+QEMUFile *cpr_file_new_fd_output(int fd, const char *name)
+{
+    g_autoptr(QIOChannelFile) fioc = qio_channel_file_new_fd(fd);
+    QIOChannel *ioc = QIO_CHANNEL(fioc);
+    qio_channel_set_name(ioc, name);
+    return qemu_file_new_output(ioc);
+}
+
 static const VMStateDescription vmstate_cpr_luo_state = {
     .name = CPR_LUO_STATE,
     .version_id = 1,
-- 
2.43.5

[PATCH 12/21] migration/cpr-luo: add state fd helpers

[Dongli Zhang]

It is necessary to migrate fd information to the target QEMU instance.
cpr-exec preserves fd information in a memfd and uses an environment
variable to share the memfd with the target QEMU instance. cpr-transfer
uses a UNIX domain socket to pass fd information.

For cpr-luo, we save the fd information (i.e., names and tokens) in a
memfd and preserve the memfd using dedicated token 1.

Add helpers to create, preserve, and retrieve the cpr-luo state memfd.

Once the state memfd is preserved through LUO, it remains alive even if we
explicitly close it, until we explicitly close the session fd or terminate
the source QEMU instance.

The global session fd is stored at CprState.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h  |  5 ++++
 migration/cpr-luo-stub.c | 22 ++++++++++++++++
 migration/cpr-luo.c      | 57 ++++++++++++++++++++++++++++++++++++++++
 migration/cpr.c          |  4 ++-
 4 files changed, 87 insertions(+), 1 deletion(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 23751fc015..4897d302f1 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -36,6 +36,7 @@ typedef QLIST_HEAD(CprVFIODeviceList, CprVFIODevice) CprVFIODeviceList;
 typedef struct CprState {
     CprFdList fds;
     CprVFIODeviceList vfio_devices;
+    int luo_session_fd;
 } CprState;
 
 extern CprState cpr_state;
@@ -88,4 +89,8 @@ int cpr_luo_retrieve_session(const char *name, Error **errp);
 int cpr_luo_preserve_fd(int session_fd, int fd, uint64_t token,
                         Error **errp);
 int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp);
+QEMUFile *cpr_luo_output(Error **errp);
+QEMUFile *cpr_luo_input(int session_fd, Error **errp);
+bool cpr_luo_persist_state(int session_fd, QEMUFile *f, Error **errp);
+void cpr_luo_abort(void);
 #endif
diff --git a/migration/cpr-luo-stub.c b/migration/cpr-luo-stub.c
index ab64e1f99b..fb2e649534 100644
--- a/migration/cpr-luo-stub.c
+++ b/migration/cpr-luo-stub.c
@@ -35,3 +35,25 @@ int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp)
 {
     return cpr_luo_unsupported(errp);
 }
+
+QEMUFile *cpr_luo_output(Error **errp)
+{
+    cpr_luo_unsupported(errp);
+    return NULL;
+}
+
+QEMUFile *cpr_luo_input(int fd, Error **errp)
+{
+    cpr_luo_unsupported(errp);
+    return NULL;
+}
+
+bool cpr_luo_persist_state(int fd, QEMUFile *f, Error **errp)
+{
+    cpr_luo_unsupported(errp);
+    return false;
+}
+
+void cpr_luo_abort(void)
+{
+}
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
index f6e0e23657..5b6bd6e191 100644
--- a/migration/cpr-luo.c
+++ b/migration/cpr-luo.c
@@ -8,9 +8,15 @@
 
 #include <sys/ioctl.h>
 
+#include "io/channel-file.h"
 #include "linux-headers/linux/liveupdate.h"
 #include "migration/cpr.h"
+#include "migration/qemu-file.h"
 #include "qapi/error.h"
+#include "qemu/error-report.h"
+#include "qemu/memfd.h"
+
+#define CPR_LUO_STATE_NAME "QEMU_CPR_LUO_STATE"
 
 static int luo_open(Error **errp)
 {
@@ -105,3 +111,54 @@ int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp)
 
     return retrieve.fd;
 }
+
+QEMUFile *cpr_luo_output(Error **errp)
+{
+    int fd = qemu_memfd_create(CPR_LUO_STATE_NAME, 0, false, 0, 0, errp);
+
+    if (fd < 0) {
+        return NULL;
+    }
+
+    return cpr_file_new_fd_output(fd, CPR_LUO_STATE_NAME);
+}
+
+QEMUFile *cpr_luo_input(int session_fd, Error **errp)
+{
+    int fd = cpr_luo_retrieve_fd(session_fd, CPR_LUO_STATE_TOKEN, errp);
+
+    if (fd < 0) {
+        return NULL;
+    }
+
+    lseek(fd, 0, SEEK_SET);
+    return cpr_file_new_fd_input(fd, CPR_LUO_STATE_NAME);
+}
+
+bool cpr_luo_persist_state(int session_fd, QEMUFile *f, Error **errp)
+{
+    QIOChannelFile *fioc = QIO_CHANNEL_FILE(qemu_file_get_ioc(f));
+
+    /*
+     * The file size (the memfd in this case) cannot be changed after it
+     * is preserved. Flush all data before preserving the state fd.
+     */
+    if (qemu_fflush(f) < 0) {
+        error_setg(errp, "failed to flush cpr-luo state");
+        return false;
+    }
+
+    return cpr_luo_preserve_fd(session_fd, fioc->fd,
+                               CPR_LUO_STATE_TOKEN, errp) == 0;
+}
+
+void cpr_luo_abort(void)
+{
+    int session_fd = cpr_state.luo_session_fd;
+
+    if (session_fd >= 0) {
+        close(session_fd);
+    }
+
+    cpr_state.luo_session_fd = -1;
+}
diff --git a/migration/cpr.c b/migration/cpr.c
index 3e8e94e5fe..2253fddb42 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -24,7 +24,9 @@
 /*************************************************************************/
 /* cpr state container for all information to be saved. */
 
-CprState cpr_state;
+CprState cpr_state = {
+    .luo_session_fd = -1,
+};
 static GHashTable *cpr_fds_hash;
 
 /****************************************************************************/
-- 
2.43.5

[PATCH 13/21] migration/cpr: add more arguments for cpr_walk_fd()

[Dongli Zhang]

cpr-luo requires walking through each LUO-supported fd to preserve it.
Unlike cpr-exec or cpr-transfer, which only require the fd value, cpr-luo
requires additional information, such as a token, to preserve an fd.

In addition, we may want to return more values, such as a retrieved fd
based on a session fd and token, and pass errp when an error occurs.

Therefore, rework cpr_walk_fd() by adding more arguments for
flexibility.

Remove the assertion on elem->fd, since cpr-luo does not save or load the
fd itself, only its name, ID, and token.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h |  4 ++--
 migration/cpr-exec.c    | 12 ++++++------
 migration/cpr.c         |  5 ++---
 3 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 4897d302f1..775f584368 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -49,8 +49,8 @@ void cpr_resave_fd(const char *name, int id, int fd);
 int cpr_open_fd(const char *path, int flags, const char *name, int id,
                 Error **errp);
 
-typedef bool (*cpr_walk_fd_cb)(int fd);
-bool cpr_walk_fd(cpr_walk_fd_cb cb);
+typedef bool (*cpr_walk_fd_cb)(CprFd *fd, Error **errp);
+bool cpr_walk_fd(cpr_walk_fd_cb cb, Error **errp);
 
 MigMode cpr_get_incoming_mode(void);
 void cpr_set_incoming_mode(MigMode mode);
diff --git a/migration/cpr-exec.c b/migration/cpr-exec.c
index b31ec3e59f..06cbced7e0 100644
--- a/migration/cpr-exec.c
+++ b/migration/cpr-exec.c
@@ -94,26 +94,26 @@ QEMUFile *cpr_exec_input(Error **errp)
     return cpr_file_new_fd_input(mfd, CPR_EXEC_STATE_NAME);
 }
 
-static bool preserve_fd(int fd)
+static bool preserve_fd(CprFd *elem, Error **errp)
 {
-    qemu_clear_cloexec(fd);
+    qemu_clear_cloexec(elem->fd);
     return true;
 }
 
-static bool unpreserve_fd(int fd)
+static bool unpreserve_fd(CprFd *elem, Error **errp)
 {
-    qemu_set_cloexec(fd);
+    qemu_set_cloexec(elem->fd);
     return true;
 }
 
 static void cpr_exec_preserve_fds(void)
 {
-    cpr_walk_fd(preserve_fd);
+    cpr_walk_fd(preserve_fd, NULL);
 }
 
 void cpr_exec_unpreserve_fds(void)
 {
-    cpr_walk_fd(unpreserve_fd);
+    cpr_walk_fd(unpreserve_fd, NULL);
 }
 
 static void cpr_exec_cb(void *opaque)
diff --git a/migration/cpr.c b/migration/cpr.c
index 2253fddb42..bc4d1e979d 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -230,15 +230,14 @@ int cpr_open_fd(const char *path, int flags, const char *name, int id,
     return fd;
 }
 
-bool cpr_walk_fd(cpr_walk_fd_cb cb)
+bool cpr_walk_fd(cpr_walk_fd_cb cb, Error **errp)
 {
     GHashTableIter iter;
     CprFd *elem;
 
     g_hash_table_iter_init(&iter, get_cpr_fds_hash());
     while (g_hash_table_iter_next(&iter, (gpointer *)&elem, NULL)) {
-        g_assert(elem->fd >= 0);
-        if (!cb(elem->fd)) {
+        if (!cb(elem, errp)) {
             return false;
         }
     }
-- 
2.43.5

[PATCH 14/21] migration/cpr-luo: add helpers to preserve and retrieve all LUO fds

[Dongli Zhang]

Add helpers to walk through saved CPR fds to preserve or retrieve
LUO-supported fds. Ignore fds that are not LUO-supported.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h |  2 ++
 migration/cpr-luo.c     | 59 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 775f584368..bfb2bb0b50 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -92,5 +92,7 @@ int cpr_luo_retrieve_fd(int session_fd, uint64_t token, Error **errp);
 QEMUFile *cpr_luo_output(Error **errp);
 QEMUFile *cpr_luo_input(int session_fd, Error **errp);
 bool cpr_luo_persist_state(int session_fd, QEMUFile *f, Error **errp);
+bool cpr_luo_preserve_fds(Error **errp);
+bool cpr_luo_retrieve_fds(Error **errp);
 void cpr_luo_abort(void);
 #endif
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
index 5b6bd6e191..f20ba1f30c 100644
--- a/migration/cpr-luo.c
+++ b/migration/cpr-luo.c
@@ -152,6 +152,65 @@ bool cpr_luo_persist_state(int session_fd, QEMUFile *f, Error **errp)
                                CPR_LUO_STATE_TOKEN, errp) == 0;
 }
 
+/*
+ * This function must be called only after cpr_fd_pre_save() has assigned
+ * LUO tokens.
+ */
+static bool cpr_luo_preserve_one(CprFd *elem, Error **errp)
+{
+    if (!elem->luo) {
+        return true;
+    }
+
+    if (!elem->luo_token) {
+        error_setg(errp, "missing LUO token for CprFd %s id %d",
+                   elem->name, elem->id);
+        return false;
+    }
+
+    return cpr_luo_preserve_fd(cpr_state.luo_session_fd, elem->fd,
+                               elem->luo_token, errp) == 0;
+}
+
+bool cpr_luo_preserve_fds(Error **errp)
+{
+    return cpr_walk_fd(cpr_luo_preserve_one, errp);
+}
+
+/*
+ * This function must be called only after cpr_fd_post_load() has
+ * initialized luo and received luo_token.
+ */
+static bool cpr_luo_retrieve_one(CprFd *elem, Error **errp)
+{
+    int retrieved_fd;
+
+    if (!elem->luo) {
+        return true;
+    }
+
+    if (!elem->luo_token) {
+        error_setg(errp, "missing LUO token for CprFd %s id %d",
+                   elem->name, elem->id);
+        return false;
+    }
+
+    retrieved_fd = cpr_luo_retrieve_fd(cpr_state.luo_session_fd,
+                                       elem->luo_token, errp);
+
+    if (retrieved_fd < 0) {
+        return false;
+    }
+
+    elem->fd = retrieved_fd;
+    return true;
+}
+
+bool cpr_luo_retrieve_fds(Error **errp)
+{
+    return cpr_walk_fd(cpr_luo_retrieve_one, errp);
+}
+
 void cpr_luo_abort(void)
 {
     int session_fd = cpr_state.luo_session_fd;
-- 
2.43.5

[PATCH 15/21] migration/cpr-luo: enable cpr-luo mode for source side preservation

[Dongli Zhang]

As with cpr-exec and cpr-transfer, call the cpr-luo APIs from
cpr_state_save() to preserve both the state memfd and all LUO-supported
fds.

Use the explicit machine UUID to form the LUO session name.

Close and clean up the session if an error occurs.

A migration notifier will be added in a future commit to clean up the
session if migration later fails.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
A cleanup of existing code can help remove an unnecessary
migrate_mode().

 include/migration/cpr.h  |  1 +
 migration/cpr-luo-stub.c |  5 ++++
 migration/cpr-luo.c      | 50 ++++++++++++++++++++++++++++++++++++++++
 migration/cpr.c          |  7 ++++++
 migration/migration.c    |  3 ++-
 migration/ram.c          |  1 +
 6 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index bfb2bb0b50..62426bd3e0 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -94,5 +94,6 @@ QEMUFile *cpr_luo_input(int session_fd, Error **errp);
 bool cpr_luo_persist_state(int session_fd, QEMUFile *f, Error **errp);
 bool cpr_luo_preserve_fds(Error **errp);
 bool cpr_luo_retrieve_fds(Error **errp);
+bool cpr_luo_save(QEMUFile *f, Error **errp);
 void cpr_luo_abort(void);
 #endif
diff --git a/migration/cpr-luo-stub.c b/migration/cpr-luo-stub.c
index fb2e649534..04a5bd816a 100644
--- a/migration/cpr-luo-stub.c
+++ b/migration/cpr-luo-stub.c
@@ -54,6 +54,11 @@ bool cpr_luo_persist_state(int fd, QEMUFile *f, Error **errp)
     return false;
 }
 
+bool cpr_luo_save(QEMUFile *f, Error **errp)
+{
+    return cpr_luo_unsupported(errp);
+}
+
 void cpr_luo_abort(void)
 {
 }
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
index f20ba1f30c..1d112e7865 100644
--- a/migration/cpr-luo.c
+++ b/migration/cpr-luo.c
@@ -15,8 +15,10 @@
 #include "qapi/error.h"
 #include "qemu/error-report.h"
 #include "qemu/memfd.h"
+#include "system/system.h"
 
 #define CPR_LUO_STATE_NAME "QEMU_CPR_LUO_STATE"
+#define CPR_LUO_SESSION_PREFIX "qemu-cpr-luo-"
 
 static int luo_open(Error **errp)
 {
@@ -211,6 +213,54 @@ bool cpr_luo_retrieve_fds(Error **errp)
     return cpr_walk_fd(cpr_luo_retrieve_one, errp);
 }
 
+static bool cpr_luo_session_name(char name[LIVEUPDATE_SESSION_NAME_LENGTH],
+                                 Error **errp)
+{
+    char uuid[UUID_STR_LEN];
+
+    if (!qemu_machine_uuid_is_set()) {
+        error_setg(errp, "cpr-luo requires a non-null machine UUID");
+        return false;
+    }
+
+    qemu_uuid_unparse(&qemu_uuid, uuid);
+    snprintf(name, LIVEUPDATE_SESSION_NAME_LENGTH, "%s%s",
+             CPR_LUO_SESSION_PREFIX, uuid);
+
+    return true;
+}
+
+bool cpr_luo_save(QEMUFile *f, Error **errp)
+{
+    char name[LIVEUPDATE_SESSION_NAME_LENGTH];
+    int session_fd;
+
+    if (!cpr_luo_session_name(name, errp)) {
+        return false;
+    }
+
+    session_fd = cpr_luo_create_session(name, errp);
+    if (session_fd < 0) {
+        return false;
+    }
+
+    cpr_state.luo_session_fd = session_fd;
+
+    if (!cpr_luo_persist_state(session_fd, f, errp)) {
+        goto fail;
+    }
+
+    if (!cpr_luo_preserve_fds(errp)) {
+        goto fail;
+    }
+
+    return true;
+
+fail:
+    cpr_luo_abort();
+    return false;
+}
+
 void cpr_luo_abort(void)
 {
     int session_fd = cpr_state.luo_session_fd;
diff --git a/migration/cpr.c b/migration/cpr.c
index bc4d1e979d..b91f2fdcd7 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -335,6 +335,8 @@ bool cpr_state_save(MigrationChannel *channel, Error **errp)
         f = cpr_transfer_output(channel, errp);
     } else if (mode == MIG_MODE_CPR_EXEC) {
         f = cpr_exec_output(errp);
+    } else if (mode == MIG_MODE_CPR_LUO) {
+        f = cpr_luo_output(errp);
     } else {
         return true;
     }
@@ -355,6 +357,11 @@ bool cpr_state_save(MigrationChannel *channel, Error **errp)
             qemu_fclose(f);
             return false;
         }
+    } else if (mode == MIG_MODE_CPR_LUO) {
+        if (!cpr_luo_save(f, errp)) {
+            qemu_fclose(f);
+            return false;
+        }
     }
 
     /*
diff --git a/migration/migration.c b/migration/migration.c
index 6a51f212ca..c8055a1d98 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1698,7 +1698,8 @@ bool migrate_mode_is_cpr(void)
     MigMode mode = migrate_mode();
     return mode == MIG_MODE_CPR_REBOOT ||
            mode == MIG_MODE_CPR_TRANSFER ||
-           mode == MIG_MODE_CPR_EXEC;
+           mode == MIG_MODE_CPR_EXEC ||
+           mode == MIG_MODE_CPR_LUO;
 }
 
 int migrate_init(MigrationState *s, Error **errp)
diff --git a/migration/ram.c b/migration/ram.c
index e6db3bf523..34e327a9de 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -230,6 +230,7 @@ bool migrate_ram_is_ignored(RAMBlock *block)
     return !qemu_ram_is_migratable(block) ||
            mode == MIG_MODE_CPR_TRANSFER ||
            mode == MIG_MODE_CPR_EXEC ||
+           mode == MIG_MODE_CPR_LUO ||
            (migrate_ignore_shared() && qemu_ram_is_shared(block)
                                     && qemu_ram_is_named_file(block));
 }
-- 
2.43.5

[PATCH 16/21] migration/cpr-luo: enable cpr-luo mode for target side retrieval

[Dongli Zhang]

As with cpr-exec and cpr-transfer, call the cpr-luo APIs to retrieve the
LUO session and its state memfd before device creation. After loading the
LUO CPR state, retrieve the preserved file descriptors by token.

Finish the LUO session after the incoming migration successfully loads.

Accept "-incoming cpr-luo:defer" for deferred incoming migration and
"-incoming cpr-luo:<URI>" to start an incoming migration immediately.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
The mode management in existing code may need cleanup. There are
duplicate cpr_set_incoming_mode(). There are unnecessary call of
migrate_mode().

 include/migration/cpr.h  |  2 ++
 migration/cpr-luo-stub.c | 10 ++++++++++
 migration/cpr-luo.c      | 35 +++++++++++++++++++++++++++++++++++
 migration/cpr.c          | 24 ++++++++++++++++++++----
 migration/migration.c    |  2 ++
 system/vl.c              |  9 +++++++++
 6 files changed, 78 insertions(+), 4 deletions(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 62426bd3e0..96529ab3a1 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -95,5 +95,7 @@ bool cpr_luo_persist_state(int session_fd, QEMUFile *f, Error **errp);
 bool cpr_luo_preserve_fds(Error **errp);
 bool cpr_luo_retrieve_fds(Error **errp);
 bool cpr_luo_save(QEMUFile *f, Error **errp);
+QEMUFile *cpr_luo_load(Error **errp);
+void cpr_luo_incoming_finish(void);
 void cpr_luo_abort(void);
 #endif
diff --git a/migration/cpr-luo-stub.c b/migration/cpr-luo-stub.c
index 04a5bd816a..22a7037c20 100644
--- a/migration/cpr-luo-stub.c
+++ b/migration/cpr-luo-stub.c
@@ -59,6 +59,16 @@ bool cpr_luo_save(QEMUFile *f, Error **errp)
     return cpr_luo_unsupported(errp);
 }
 
+QEMUFile *cpr_luo_load(Error **errp)
+{
+    cpr_luo_unsupported(errp);
+    return NULL;
+}
+
+void cpr_luo_incoming_finish(void)
+{
+}
+
 void cpr_luo_abort(void)
 {
 }
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
index 1d112e7865..d9aba3bfc5 100644
--- a/migration/cpr-luo.c
+++ b/migration/cpr-luo.c
@@ -261,6 +261,41 @@ fail:
     return false;
 }
 
+QEMUFile *cpr_luo_load(Error **errp)
+{
+    char name[LIVEUPDATE_SESSION_NAME_LENGTH];
+    int session_fd;
+
+    if (!cpr_luo_session_name(name, errp)) {
+        return NULL;
+    }
+
+    session_fd = cpr_luo_retrieve_session(name, errp);
+    if (session_fd < 0) {
+        return NULL;
+    }
+
+    cpr_state.luo_session_fd = session_fd;
+    return cpr_luo_input(session_fd, errp);
+}
+
+void cpr_luo_incoming_finish(void)
+{
+    struct liveupdate_session_finish finish = { .size = sizeof(finish) };
+    int session_fd = cpr_state.luo_session_fd;
+
+    if (cpr_get_incoming_mode() != MIG_MODE_CPR_LUO || session_fd < 0) {
+        return;
+    }
+
+    if (ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, &finish) < 0) {
+        warn_report("failed to finish LUO session: %s", strerror(errno));
+    }
+
+    close(session_fd);
+    cpr_state.luo_session_fd = -1;
+}
+
 void cpr_luo_abort(void)
 {
     int session_fd = cpr_state.luo_session_fd;
diff --git a/migration/cpr.c b/migration/cpr.c
index b91f2fdcd7..fdbb684798 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -391,11 +391,15 @@ bool cpr_state_load(MigrationChannel *channel, Error **errp)
         mode = MIG_MODE_CPR_TRANSFER;
         cpr_set_incoming_mode(mode);
         f = cpr_transfer_input(channel, errp);
+    } else if (cpr_get_incoming_mode() == MIG_MODE_CPR_LUO) {
+        mode = MIG_MODE_CPR_LUO;
+        f = cpr_luo_load(errp);
     } else {
         return true;
     }
+
     if (!f) {
-        return false;
+        goto fail;
     }
 
     trace_cpr_state_load(MigMode_str(mode));
@@ -405,23 +409,28 @@ bool cpr_state_load(MigrationChannel *channel, Error **errp)
     if (v != QEMU_CPR_FILE_MAGIC) {
         error_setg(errp, "Not a migration stream (bad magic %x)", v);
         qemu_fclose(f);
-        return false;
+        goto fail;
     }
     v = qemu_get_be32(f);
     if (v != QEMU_CPR_FILE_VERSION) {
         error_setg(errp, "Unsupported migration stream version %d", v);
         qemu_fclose(f);
-        return false;
+        goto fail;
     }
 
     if (!vmstate_load_vmsd(f, cpr_state_vmsd(mode), &cpr_state, 1, errp)) {
         qemu_fclose(f);
-        return false;
+        goto fail;
     }
 
     if (migrate_mode() == MIG_MODE_CPR_EXEC) {
         /* Set cloexec to prevent fd leaks from fork until the next cpr-exec */
         cpr_exec_unpreserve_fds();
+    } else if (mode == MIG_MODE_CPR_LUO) {
+        if (!cpr_luo_retrieve_fds(errp)) {
+            qemu_fclose(f);
+            goto fail;
+        }
     }
 
     /*
@@ -431,6 +440,13 @@ bool cpr_state_load(MigrationChannel *channel, Error **errp)
     cpr_state_file = f;
 
     return true;
+
+fail:
+    if (mode == MIG_MODE_CPR_LUO) {
+        cpr_luo_abort();
+    }
+
+    return false;
 }
 
 void cpr_state_close(void)
diff --git a/migration/migration.c b/migration/migration.c
index c8055a1d98..7c2bb982a5 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -687,6 +687,8 @@ static void process_incoming_migration_bh(void *opaque)
 
     dirty_bitmap_mig_before_vm_start();
 
+    cpr_luo_incoming_finish();
+
     if (runstate_is_live(migration_get_target_runstate())) {
         if (autostart) {
             /*
diff --git a/system/vl.c b/system/vl.c
index 468a9fc247..b40f4803e1 100644
--- a/system/vl.c
+++ b/system/vl.c
@@ -1840,12 +1840,21 @@ static void incoming_option_parse(const char *str)
 {
     MigrationChannelType type = MIGRATION_CHANNEL_TYPE_MAIN;
     MigrationChannel *channel;
+    const char *uri;
     Visitor *v;
 
     if (!strcmp(str, "defer")) {
         channel = NULL;
     } else if (migrate_is_uri(str)) {
         migrate_uri_parse(str, &channel, &error_fatal);
+    } else if (strstart(str, "cpr-luo:", &uri)) {
+        cpr_set_incoming_mode(MIG_MODE_CPR_LUO);
+        if (!strcmp(uri, "defer")) {
+            channel = NULL;
+            str = "defer";
+        } else {
+            migrate_uri_parse(uri, &channel, &error_fatal);
+        }
     } else {
         v = qobject_input_visitor_new_str(str, "channel-type", &error_fatal);
         visit_type_MigrationChannel(v, NULL, &channel, &error_fatal);
-- 
2.43.5

[PATCH 17/21] migration/cpr-luo: abort session when migration fails

[Dongli Zhang]

Register a cpr-luo migration notifier that closes an active source LUO
session after migration failure.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 include/migration/cpr.h  |  1 +
 migration/cpr-luo-stub.c |  4 ++++
 migration/cpr-luo.c      | 20 ++++++++++++++++++++
 migration/migration.c    |  1 +
 4 files changed, 26 insertions(+)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 96529ab3a1..239cddf352 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -97,5 +97,6 @@ bool cpr_luo_retrieve_fds(Error **errp);
 bool cpr_luo_save(QEMUFile *f, Error **errp);
 QEMUFile *cpr_luo_load(Error **errp);
 void cpr_luo_incoming_finish(void);
+void cpr_luo_init(void);
 void cpr_luo_abort(void);
 #endif
diff --git a/migration/cpr-luo-stub.c b/migration/cpr-luo-stub.c
index 22a7037c20..4eb14ccbbc 100644
--- a/migration/cpr-luo-stub.c
+++ b/migration/cpr-luo-stub.c
@@ -69,6 +69,10 @@ void cpr_luo_incoming_finish(void)
 {
 }
 
+void cpr_luo_init(void)
+{
+}
+
 void cpr_luo_abort(void)
 {
 }
diff --git a/migration/cpr-luo.c b/migration/cpr-luo.c
index d9aba3bfc5..02ca947507 100644
--- a/migration/cpr-luo.c
+++ b/migration/cpr-luo.c
@@ -11,6 +11,7 @@
 #include "io/channel-file.h"
 #include "linux-headers/linux/liveupdate.h"
 #include "migration/cpr.h"
+#include "migration/migration.h"
 #include "migration/qemu-file.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
@@ -296,6 +297,25 @@ void cpr_luo_incoming_finish(void)
     cpr_state.luo_session_fd = -1;
 }
 
+static int cpr_luo_notifier(NotifierWithReturn *notifier,
+                            MigrationEvent *e, Error **errp)
+{
+
+    if (e->type == MIG_EVENT_FAILED && cpr_state.luo_session_fd >= 0) {
+        cpr_luo_abort();
+    }
+
+    return 0;
+}
+
+void cpr_luo_init(void)
+{
+    static NotifierWithReturn luo_notifier;
+
+    migration_add_notifier_mode(&luo_notifier, cpr_luo_notifier,
+                                MIG_MODE_CPR_LUO);
+}
+
 void cpr_luo_abort(void)
 {
     int session_fd = cpr_state.luo_session_fd;
diff --git a/migration/migration.c b/migration/migration.c
index 7c2bb982a5..a9503607d5 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -340,6 +340,7 @@ void migration_object_init(void)
     ram_mig_init();
     dirty_bitmap_mig_init();
     cpr_exec_init();
+    cpr_luo_init();
 
     /* Initialize cpu throttle timers */
     cpu_throttle_init();
-- 
2.43.5

[PATCH 18/21] system/physmem: add cpr-luo blocker

[Dongli Zhang]

Add a cpr-luo migration blocker for anonymous memory. cpr-luo requires
shared memory, and therefore cannot use memory-backend-ram,
memory-backend-memfd, or memory-backend-shm when "share=on" or
"aux-ram-share=on" is unavailable.

Here is an example if I use regular "-m 8G".

(qemu) migrate_set_parameter mode cpr-luo
(qemu) migrate file:cpr-luo.state
Error: Memory region pc.ram is not compatible with CPR. share=on is
  required for memory-backend objects, and aux-ram-share=on is required.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
MIG_MODE_CPR_EXEC is missing from some existing blockers found with cscope
"cs find s MIG_MODE_CPR_TRANSFER". I may fix them and send the changes as
an independent patchset.

 system/physmem.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/system/physmem.c b/system/physmem.c
index 263acb5727..fcf40ca3f8 100644
--- a/system/physmem.c
+++ b/system/physmem.c
@@ -2246,7 +2246,8 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
                        "which is not supported with CPR.",
                        memory_region_name(new_block->mr));
             migrate_add_blocker_modes(&new_block->cpr_blocker,
-                                      BIT(MIG_MODE_CPR_TRANSFER), errp);
+                    BIT(MIG_MODE_CPR_TRANSFER) | BIT(MIG_MODE_CPR_LUO),
+                    errp);
         }
     }
 
@@ -4539,8 +4540,9 @@ void ram_block_add_cpr_blocker(RAMBlock *rb, Error **errp)
                "Memory region %s is not compatible with CPR. share=on is "
                "required for memory-backend objects, and aux-ram-share=on is "
                "required.", memory_region_name(rb->mr));
-    migrate_add_blocker_modes(&rb->cpr_blocker, BIT(MIG_MODE_CPR_TRANSFER),
-                              errp);
+    migrate_add_blocker_modes(&rb->cpr_blocker,
+            BIT(MIG_MODE_CPR_TRANSFER) | BIT(MIG_MODE_CPR_LUO),
+            errp);
 }
 
 void ram_block_del_cpr_blocker(RAMBlock *rb)
-- 
2.43.5

[PATCH 19/21] monitor/qmp: close LUO session when resuming source QEMU

[Dongli Zhang]

After a successful cpr-luo migration, the source QEMU remains paused and
keeps the LUO session open for the upcoming kexec. If the user instead
resumes the source QEMU with the cont command, close the LUO session
before restarting the VM.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 monitor/qmp-cmds.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index 7525a88e04..fc66618c47 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -32,6 +32,7 @@
 #include "qapi/type-helpers.h"
 #include "hw/mem/memory-device.h"
 #include "hw/intc/intc.h"
+#include "migration/cpr.h"
 #include "migration/misc.h"
 
 NameInfo *qmp_query_name(Error **errp)
@@ -113,6 +114,12 @@ void qmp_cont(Error **errp)
             error_propagate(errp, local_err);
             return;
         }
+        /*
+         * If the user resumes the source QEMU after a completed cpr-luo
+         * migration, close the LUO session fd. This is a nop for other
+         * migration modes or regular operation.
+         */
+        cpr_luo_abort();
         vm_start();
     }
 }
-- 
2.43.5

[PATCH 20/21] vfio: block cpr-luo migration

[Dongli Zhang]

So far, cpr-luo does not support preserving and retrieving VFIO resources
across a kexec reboot. In the future, this may be supported by treating
VFIO as a live-migratable device when applicable, that is, by saving and
loading VFIO and vendor-specific state across kexec, as mlx5_vfio_pci does.
In addition, the upstream linux kernel work is ongoing to preserve and
retrieve PCI, IOMMU, and VFIO state across kexec.

Block cpr-luo migration whenever a VFIO is involved, just like cpr-exec and
cpr-transfer.

(qemu) migrate_set_parameter mode cpr-luo
(qemu) migrate -d file:cpr-luo.state
Error: VFIO: cpr-luo is not supported

This does not affect the existing cpr-transfer and cpr-exec support.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 hw/vfio/cpr-iommufd.c      | 17 ++++++++++++++---
 hw/vfio/cpr-legacy.c       |  8 ++++++++
 include/hw/vfio/vfio-cpr.h |  1 +
 include/system/iommufd.h   |  1 +
 4 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c
index 6758c73b6b..803fc0cd51 100644
--- a/hw/vfio/cpr-iommufd.c
+++ b/hw/vfio/cpr-iommufd.c
@@ -157,10 +157,20 @@ bool vfio_iommufd_cpr_register_iommufd(IOMMUFDBackend *be, Error **errp)
 {
     Error **cpr_blocker = &be->cpr_blocker;
 
+    error_setg(&be->luo_blocker, "VFIO: cpr-luo is not supported");
+    if (migrate_add_blocker_modes(&be->luo_blocker,
+                                  BIT(MIG_MODE_CPR_LUO), errp) < 0) {
+        return false;
+    }
+
     if (!vfio_cpr_supported(be, cpr_blocker)) {
-        return migrate_add_blocker_modes(cpr_blocker,
-                    BIT(MIG_MODE_CPR_TRANSFER) | BIT(MIG_MODE_CPR_EXEC),
-                    errp) == 0;
+        if (migrate_add_blocker_modes(cpr_blocker,
+            BIT(MIG_MODE_CPR_TRANSFER) | BIT(MIG_MODE_CPR_EXEC), errp) < 0) {
+            migrate_del_blocker(&be->luo_blocker);
+            return false;
+        }
+
+        return true;
     }
 
     vmstate_register(NULL, -1, &iommufd_cpr_vmstate, be);
@@ -172,6 +182,7 @@ void vfio_iommufd_cpr_unregister_iommufd(IOMMUFDBackend *be)
 {
     vmstate_unregister(NULL, &iommufd_cpr_vmstate, be);
     migrate_del_blocker(&be->cpr_blocker);
+    migrate_del_blocker(&be->luo_blocker);
 }
 
 bool vfio_iommufd_cpr_register_container(VFIOIOMMUFDContainer *container,
diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c
index 2d40d8baea..9160e95f27 100644
--- a/hw/vfio/cpr-legacy.c
+++ b/hw/vfio/cpr-legacy.c
@@ -171,6 +171,13 @@ bool vfio_legacy_cpr_register_container(VFIOLegacyContainer *container,
     VFIOContainer *bcontainer = VFIO_IOMMU(container);
     Error **cpr_blocker = &container->cpr.blocker;
 
+    error_setg(&container->cpr.luo_blocker,
+               "VFIO: cpr-luo is not supported");
+    if (migrate_add_blocker_modes(&container->cpr.luo_blocker,
+                                  BIT(MIG_MODE_CPR_LUO), errp) < 0) {
+        return false;
+    }
+
     migration_add_notifier_mode(&bcontainer->cpr_reboot_notifier,
                                 vfio_cpr_reboot_notifier,
                                 MIG_MODE_CPR_REBOOT);
@@ -199,6 +206,7 @@ void vfio_legacy_cpr_unregister_container(VFIOLegacyContainer *container)
     migrate_del_blocker(&container->cpr.blocker);
     vmstate_unregister(NULL, &vfio_container_vmstate, container);
     migration_remove_notifier(&container->cpr.transfer_notifier);
+    migrate_del_blocker(&container->cpr.luo_blocker);
 }
 
 /*
diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h
index ecabe0c747..8d885fe404 100644
--- a/include/hw/vfio/vfio-cpr.h
+++ b/include/hw/vfio/vfio-cpr.h
@@ -26,6 +26,7 @@ typedef int (*dma_map_fn)(const struct VFIOContainer *bcontainer,
 
 typedef struct VFIOContainerCPR {
     Error *blocker;
+    Error *luo_blocker;
     bool vaddr_unmapped;
     NotifierWithReturn transfer_notifier;
     MemoryListener remap_listener;
diff --git a/include/system/iommufd.h b/include/system/iommufd.h
index 17bf1a3727..182a71df77 100644
--- a/include/system/iommufd.h
+++ b/include/system/iommufd.h
@@ -33,6 +33,7 @@ struct IOMMUFDBackend {
     int fd;            /* /dev/iommu file descriptor */
     bool owned;        /* is the /dev/iommu opened internally */
     Error *cpr_blocker;/* set if be does not support CPR */
+    Error *luo_blocker;
     uint32_t users;
 
     /*< public >*/
-- 
2.43.5

[PATCH 21/21] migration: cpr-luo documentation

[Dongli Zhang]

Document the cpr-luo migration mode, its LUO and Kexec Handover
requirements, and its outgoing and incoming usage.

Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 docs/devel/migration/CPR.rst | 64 ++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
index b6178568a8..40baf3fc0f 100644
--- a/docs/devel/migration/CPR.rst
+++ b/docs/devel/migration/CPR.rst
@@ -434,3 +434,67 @@ cpr-exec mode requires permission to use the exec system call, which
 is denied by certain sandbox options, such as spawn.
 
 The guest pause time increases for large guest RAM backed by small pages.
+
+cpr-luo mode
+------------
+
+This mode uses the Linux Live Update Orchestrator (LUO) to preserve QEMU's CPR
+LUO state and the file descriptors backing guest RAM across a kexec handover.
+QEMU writes the remaining migration state to the migration URI.  The outgoing
+QEMU must remain alive in postmigrate until the host performs kexec: closing
+its LUO session releases the preserved file descriptors.
+
+Both the old and new host kernels must support Kexec Handover (KHO) and LUO.
+KHO must be enabled, for example with ``kho=on`` when it is not enabled by
+default, and LUO must be enabled with the ``liveupdate=on`` kernel command
+line parameter.
+
+Only RAMBlock backing file descriptors are currently preserved in LUO.  The
+target must rebuild any non-RAM file descriptors separately. VFIO is blocked.
+
+Usage
+^^^^^
+
+Memory backend objects must use ``share=on`` and the machine must use
+``aux-ram-share=on``. KVM host kernel cmdline must have
+``kho=on liveupdate=on``.
+
+Outgoing:
+  * Set the migration mode parameter to ``cpr-luo``.
+  * ``uuid`` is mandatory.
+  * Issue the ``migrate`` command.  The URI must remain available after
+    kexec, such as a file on local storage.
+  * When QEMU reaches postmigrate, perform host kexec without quitting QEMU.
+
+Incoming:
+  * Start QEMU with the same machine and memory backend configuration.
+  * ``uuid`` is mandatory to match the LUO session.
+  * Pass the UUID and migration URI with ``-incoming cpr-luo:<URI>``.
+  * If ``-incoming cpr-luo:defer`` is used, ``migrate_incoming <URI>`` will
+    be required later.
+
+Example
+^^^^^^^
+::
+
+  # qemu-kvm -monitor stdio
+      -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234
+      -object memory-backend-memfd,id=ram0,size=4G,share=on
+      -machine memory-backend=ram0,aux-ram-share=on ...
+
+  (qemu) migrate_set_parameter mode cpr-luo
+  (qemu) migrate -d file:vm.state
+  (qemu) info status
+  VM status: paused (postmigrate)
+
+  # kexec -s -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --reuse-cmdline
+  # kexec -e
+
+  # qemu-kvm -monitor stdio
+      -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234
+      -object memory-backend-memfd,id=ram0,size=4G,share=on
+      -machine memory-backend=ram0,aux-ram-share=on
+      -incoming cpr-luo:file:vm.state ...
+
+If ``-incoming cpr-luo:defer`` is configured, explicitly complete the
+migration with ``migrate_incoming file:vm.state``.
-- 
2.43.5