[RFC PATCH v2 0/4] Add KVM API for confidential guest live migration

[RFC PATCH v2 0/4] Add KVM API for confidential guest live migration

[Tony Lindgren]

Hi all,

As discussed in a recent PUCK call, Sean suggested we post what Intel is
using for the KVM live migration API for TDX as an example to see if we
can come up with APIs that are not vendor specific.

The goal of this patch series is to start a discussion about live migration
kernel APIs for CoCo guests.

Tom, since you mentioned that AMD SEV-SNP and Intel TDX live migration
sound similar, can you please take a look how the API might work for
SEV-SNP?

For CoCo VMs, the guest memory and vCPU states are not accessible to the
userspace or KVM for live migration. The memory and vCPU states need to be
extracted into encrypted blobs on the source, and decrypted on the
destination. Before live migration, an encryption key needs to be
negotiated between the source and destination.

The layer handling the encryption for live migration is implementation
specific. It can be the TDX module or Coconut-SVSM for example.

For CoCo VMs, the KVM_MEMORY_ENCRYPT_OP ioctl() has been used with vendor
specific sub-commands. Adding more vendor specific sub-commands is an
option also for live migration. However, depending on how similar the KVM
needs are, it may be possible to have a common API.

For TDX, we're using a group of ioctl()s that might be possible to adapt
also for other CoCo implementations.

Artem has put together a brief description below of the example API and the
migration flow:

Example API
===========

- KVM_CAP_LIVE_MIGRATION - if a VM supports live migration through this
  uAPI.
- KVM_MIGRATE_CMD - the main ioctl that drives the migration phases. Each
  command takes vendor-specific flags and a buffer for the blob that travels
  between the hosts.
  - KVM_MIGRATE_SETUP - establish the migration session and transfer the
    immutable VM state.
  - KVM_MIGRATE_ITERATION - close a memory copy round.
  - KVM_MIGRATE_STOP_AND_COPY - pause the VM and transfer the remaining VM
    state.
  - KVM_MIGRATE_END - complete the migration, or abort it.
- KVM_EXPORT_MEMORY - export memory pages on the source host.
- KVM_IMPORT_MEMORY - import memory pages on the destination host.
- KVM_EXPORT_VCPU - export vCPU state on the source host.
- KVM_IMPORT_VCPU - import vCPU state on the destination host.

Dirty page tracking does not add a new uAPI. Userspace keeps using
KVM_GET_DIRTY_LOG and KVM_CLEAR_DIRTY_LOG.

Migration flow
==============

 Source host                                      Destination host
 ===========                                      ================

 CMD(SETUP/SESSION)          <--- setup msgs ---> CMD(SETUP/SESSION)
       |    (repeated)                                  |
 CMD(SETUP/IMMUTABLE_STATE)  - immutable state -> CMD(SETUP/IMMUTABLE_STATE)
       |                                                |
 KVM_GET_DIRTY_LOG                                      |
 KVM_EXPORT_MEMORY           --- memory data ---> KVM_IMPORT_MEMORY
 CMD(ITERATION)              --- epoch token ---> CMD(ITERATION)
       |    (repeat until convergence)                  |
 CMD(STOP_AND_COPY/PAUSE)                               |
 CMD(STOP_AND_COPY/TD_STATE) --- VM state ------> CMD(STOP_AND_COPY/TD_STATE)
 KVM_EXPORT_VCPU             --- vCPU state ----> KVM_IMPORT_VCPU
 KVM_EXPORT_MEMORY           -- final memory ---> KVM_IMPORT_MEMORY
 CMD(ITERATION/DONE)         --- start token ---> CMD(ITERATION)
       |                                                |
 CMD(END)                                         CMD(END)

For the TDX implementation, the above map to the TDX module SEAMCALLs.

Regards,

Tony

Changes since v1 at [0] below:

- Drop KVM_MIGRATE_CMD sub-command PREPARE, SETUP sub-command has been
  enough for TDX at least

- Rename KVM_MIGRATE_CMD sub-command KVM_MIGRATE_TOKEN to
  KVM_MIGRATE_ITERATION

- Rename KVM_MIGRATE_CMD sub-command KVM_MIGRATE_SOURCE_BLACKOUT to
  KVM_MIGRATE_STOP_AND_COPY

- Add x86 ioctl handling

[0] https://lore.kernel.org/kvm/20251006113524.1573116-1-tony.lindgren@linux.intel.com/


Tony Lindgren (4):
  Documentation: KVM: Add live migration API for confidential guests
  KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD
  KVM: x86: Add optional KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY
  KVM: x86: Add optional KVM_EXPORT_VCPU and KVM_IMPORT_VCPU

 Documentation/virt/kvm/api.rst     | 205 +++++++++++++++++++++++++++++
 arch/x86/include/asm/kvm-x86-ops.h |   6 +
 arch/x86/include/asm/kvm_host.h    |   6 +
 arch/x86/kvm/x86.c                 | 102 ++++++++++++++
 include/uapi/linux/kvm.h           |  43 ++++++
 5 files changed, 362 insertions(+)


base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482
-- 
2.43.0

[RFC PATCH v2 1/4] Documentation: KVM: Add live migration API for confidential guests

[Tony Lindgren]

For CoCo VMs, the guest memory and vCPU states are not accessible to the
userspace or KVM for live migration. The memory and vCPU states need to be
extracted into encrypted blobs on the source, and decrypted on the
destination. Before live migration, an encryption key needs to be
negotiated between the source and destination.

KVM help is needed to talk to the layer exporting and importing the
encrypted state. Document the KVM live migration API for confidential
guests.

Co-developed-by: Kishen Maloor <kishen.maloor@intel.com>
Signed-off-by: Kishen Maloor <kishen.maloor@intel.com>
Signed-off-by: Tony Lindgren <tony.lindgren@linux.intel.com>
---
 Documentation/virt/kvm/api.rst | 205 +++++++++++++++++++++++++++++++++
 1 file changed, 205 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index a5f9ee92f43e8..9d546d288af5f 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6566,6 +6566,200 @@ KVM_S390_KEYOP_SSKE
   Sets the storage key for the guest address ``guest_addr`` to the key
   specified in ``key``, returning the previous value in ``key``.
 
+.. _KVM_MIGRATE_CMD:
+
+4.145 KVM_MIGRATE_CMD
+---------------------
+
+:Capability: KVM_CAP_LIVE_MIGRATION
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_migrate_cmd (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to send live migration related commands to KVM for vendor
+specific handling.
+
+For confidential computing, live migration related commands may be needed.
+The commands typically use encrypted data that needs to be passed between the
+source and destination hosts. The hosts may also require specific coordination
+steps during migration that must be triggered at precise points in the
+migration process.
+
+The vendor specific implementation handles locking and checks the valid flags
+bits. If KVM_CAP_LIVE_MIGRATION is not available for the VM, -ENOTTY is
+returned.
+
+The KVM_MIGRATE_CMD subcommand passed in struct kvm_migrate_cmd is one of::
+
+  #define KVM_MIGRATE_SETUP		0
+  #define KVM_MIGRATE_ITERATION		1
+  #define KVM_MIGRATE_STOP_AND_COPY	2
+  #define KVM_MIGRATE_ABORT		3
+  #define KVM_MIGRATE_END		4
+
+The kvm_transfer_buffer is::
+
+  /**
+   * @address: Userspace buffer address
+   * @size: Size of the userspace buffer
+   * @reserved: Reserved for future use
+   */
+  struct kvm_transfer_buffer {
+	__u64 address;
+	__u32 size;
+	__u32 reserved;
+  };
+
+The kvm_migrate_cmd is::
+
+  /**
+   * @command: One of the defined KVM_MIGRATE commands
+   * @flags: Hardware specific flags
+   * @reserved: Reserved for future use
+   * @buf: Userspace buffer for hardware specific data
+   */
+  struct kvm_migrate_cmd {
+	__u16 command;
+	__u16 flags;
+	__u32 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+.. _KVM_EXPORT_MEMORY:
+
+4.146 KVM_EXPORT_MEMORY
+-----------------------
+
+:Capability: KVM_CAP_LIVE_MIGRATION
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_memory_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the host to export an array of memory pages to a
+userspace buffer.
+
+The private memory may not be accessible to KVM because of encryption. For
+confidential computing, the guest memory is encrypted and only accessible to
+the guest.
+
+If KVM_CAP_LIVE_MIGRATION is not available for the VM, -ENOTTY is returned.
+
+The vendor specific ID is used at least for TDX for the migration thread
+index.
+
+The kvm_memory_transfer is::
+
+  /**
+   * @gfns: Userspace address of an array of nr_gfns __u64 GFNs to export
+   * @nr_gfns: Number of GFNs in the @gfns array
+   * @id: Optional vendor specific transfer ID
+   * @flags: Vendor specific flags
+   * @reserved: Reserved for future use
+   * @buf: Userspace buffer to export memory to
+   */
+  struct kvm_memory_transfer {
+       __u64 gfns;
+       __u32 nr_gfns;
+       __u16 id;
+       __u16 flags;
+       __u64 reserved;
+       struct kvm_transfer_buffer buf;
+  };
+
+The transfer buffer size is vendor specific.
+
+For the transfer buffer, seeo :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.
+
+For memory import, see also :ref:`KVM_IMPORT_MEMORY <KVM_IMPORT_MEMORY>`.
+
+
+.. _KVM_IMPORT_MEMORY:
+
+4.147 KVM_IMPORT_MEMORY
+-----------------------
+
+:Capability: KVM_CAP_LIVE_MIGRATION
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_memory_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the host to import an array of memory pages from a
+userspace buffer.
+
+The private memory may not be accessible to KVM because of encryption. For
+confidential computing, the guest memory is encrypted and only accessible to
+the guest.
+
+If KVM_CAP_LIVE_MIGRATION is not available for the VM, -ENOTTY is returned.
+
+The vendor specific ID is used at least for TDX for the migration thread
+index.
+
+The transfer buffer size is vendor specific.
+
+For kvm_memory_transfer, see :ref:`KVM_EXPORT_MEMORY <KVM_EXPORT_MEMORY>`.
+
+For the transfer buffer, seeo :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.
+
+.. _KVM_EXPORT_VCPU:
+
+4.149 KVM_EXPORT_VCPU
+---------------------
+:Capability: KVM_CAP_LIVE_MIGRATION
+:Architectures: arm64, x86
+:Type: vcpu ioctl
+:Parameters: struct kvm_vcpu_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the host to export a VCPU state to a userspace
+buffer.
+
+The VCPU state may not be directly accessible to KVM because of encryption. For
+confidential computing, the VCPU state is encrypted and only accessible to the
+guest.
+
+The vcpu_transfer is::
+
+  /**
+   * @flags: Hardware specific flags
+   * @reserved: Reserved for future use
+   * @buf: Userspace buffer to export VCPU state to
+   */
+  struct kvm_vcpu_transfer {
+        __u32 flags;
+        __u32 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+For the transfer buffer, see :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.
+
+For vCPU import, see also :ref:`KVM_IMPORT_VCPU <KVM_IMPORT_VCPU>`.
+
+.. _KVM_IMPORT_VCPU:
+
+4.148 KVM_IMPORT_VCPU
+---------------------
+
+:Capability: KVM_CAP_LIVE_MIGRATION
+:Architectures: arm64, x86
+:Type: vcpu ioctl
+:Parameters: struct kvm_vcpu_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the host to import a VCPU state from a userspace
+buffer.
+
+The VCPU state may not be directly accessible to KVM because of encryption. For
+confidential computing, the VCPU state is encrypted and only accessible to the
+guest.
+
+For vcpu_transfer and vCPU import, see :ref:`KVM_IMPORT_VCPU <KVM_IMPORT_VCPU>`.
+
+For the transfer buffer, see also :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.
+
 .. _kvm_run:
 
 5. The kvm_run structure
@@ -9493,6 +9687,17 @@ take care to differentiate between these cases.
 The presence of this capability indicates that the nested KVM guest can
 start in ESA mode.
 
+8.48 KVM_CAP_LIVE_MIGRATION
+---------------------------
+
+:Architectures: x86
+:Target: VM
+:Parameters: None
+
+Indicates that the VM needs to use KVM calls for live migration, and that the
+KVM_MIGRATE_CMD ioctl and the KVM_EXPORT_MEMORY, KVM_IMPORT_MEMORY,
+KVM_EXPORT_VCPU and KVM_IMPORT_VCPU ioctls are available.
+
 9. Known KVM API problems
 =========================
 
-- 
2.43.0

[RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

Live migration of confidential guests needs the help of KVM at least for
TDX.

Add KVM_CAP_LIVE_MIGRATION for when hardware specific live migration
functions must be used.

Add KVM_MIGRATE_CMD to configure the hardware for live migration.

Assisted-by: Claude-Code:claude-opus-5 checkpatch
[ used AI to review and simplify the code ]
Signed-off-by: Tony Lindgren <tony.lindgren@linux.intel.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  2 ++
 arch/x86/include/asm/kvm_host.h    |  2 ++
 arch/x86/kvm/x86.c                 | 25 +++++++++++++++++++++++++
 include/uapi/linux/kvm.h           | 22 ++++++++++++++++++++++
 4 files changed, 51 insertions(+)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 83dc5086138b3..ac080b556b0c8 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -148,6 +148,8 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
 KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
 KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)
 KVM_X86_OP_OPTIONAL(gmem_invalidate)
+KVM_X86_OP_OPTIONAL_RET0(cap_live_migration)
+KVM_X86_OP_OPTIONAL(migrate_cmd)
 #endif
 
 #undef KVM_X86_OP
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 5f6c1ce9673b7..d9291a8a97bb1 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2010,6 +2010,8 @@ struct kvm_x86_ops {
 	int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 	int (*gmem_max_mapping_level)(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
+	bool (*cap_live_migration)(struct kvm *kvm);
+	int (*migrate_cmd)(struct kvm *kvm, struct kvm_migrate_cmd *cmd);
 };
 
 struct kvm_x86_nested_ops {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index afcac1042947a..7064fd709e56d 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4973,6 +4973,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_READONLY_MEM:
 		r = kvm ? kvm_arch_has_readonly_mem(kvm) : 1;
 		break;
+	case KVM_CAP_LIVE_MIGRATION:
+		r = kvm ? kvm_x86_call(cap_live_migration)(kvm) : 0;
+		break;
 	default:
 		break;
 	}
@@ -7614,6 +7617,28 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 		r = kvm_vm_ioctl_set_msr_filter(kvm, &filter);
 		break;
 	}
+	case KVM_MIGRATE_CMD: {
+		struct kvm_migrate_cmd cmd;
+
+		if (!kvm_x86_ops.migrate_cmd ||
+		    !kvm_x86_call(cap_live_migration)(kvm))
+			return -ENOTTY;
+
+		if (copy_from_user(&cmd, argp, sizeof(cmd)))
+			return -EFAULT;
+
+		if (cmd.reserved || cmd.buf.reserved)
+			return -EINVAL;
+
+		r = kvm_x86_call(migrate_cmd)(kvm, &cmd);
+		if (r > 0)
+			r = -EIO;
+
+		/* Copy back also on an error to report a partially done command */
+		if (copy_to_user(argp, &cmd, sizeof(cmd)))
+			return -EFAULT;
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 419011097fa8e..e5b227d3a8e40 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -997,6 +997,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_KEYOP 247
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_S390_HPAGE_2G 249
+#define KVM_CAP_LIVE_MIGRATION 250
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1350,6 +1351,8 @@ struct kvm_s390_keyop {
 #define KVM_GET_DEVICE_ATTR	  _IOW(KVMIO,  0xe2, struct kvm_device_attr)
 #define KVM_HAS_DEVICE_ATTR	  _IOW(KVMIO,  0xe3, struct kvm_device_attr)
 
+#define KVM_MIGRATE_CMD		  _IOWR(KVMIO, 0xe4, struct kvm_migrate_cmd)
+
 /*
  * ioctls for vcpu fds
  */
@@ -1670,4 +1673,23 @@ struct kvm_pre_fault_memory {
 	__u64 padding[5];
 };
 
+#define KVM_MIGRATE_SETUP		0
+#define KVM_MIGRATE_ITERATION		1
+#define KVM_MIGRATE_STOP_AND_COPY	2
+#define KVM_MIGRATE_ABORT		3
+#define KVM_MIGRATE_END			4
+
+struct kvm_transfer_buffer {
+	__u64 address;
+	__u32 size;
+	__u32 reserved;
+};
+
+struct kvm_migrate_cmd {
+	__u16 command;
+	__u16 flags;
+	__u32 reserved;
+	struct kvm_transfer_buffer buf;
+};
+
 #endif /* __LINUX_KVM_H */
-- 
2.43.0

[RFC PATCH v2 3/4] KVM: x86: Add optional KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY

[Tony Lindgren]

Add support to export and import KVM memory for cases where the memory
is only accessible to the guest. Live migration of confidential computing
needs help of KVM for the vendor specific calls at least for TDX.
Introduce optional KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY.

Based on earlier code by Wei Wang <wei.w.wang@intel.com>.

Co-developed-by: Kishen Maloor <kishen.maloor@intel.com>
Signed-off-by: Kishen Maloor <kishen.maloor@intel.com>
Assisted-by: Claude-Code:claude-opus-5 checkpatch
[ used AI to review and simplify the code ]
Signed-off-by: Tony Lindgren <tony.lindgren@linux.intel.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  2 ++
 arch/x86/include/asm/kvm_host.h    |  2 ++
 arch/x86/kvm/x86.c                 | 37 ++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h           | 13 +++++++++++
 4 files changed, 54 insertions(+)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index ac080b556b0c8..173d0c4f1115e 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -150,6 +150,8 @@ KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)
 KVM_X86_OP_OPTIONAL(gmem_invalidate)
 KVM_X86_OP_OPTIONAL_RET0(cap_live_migration)
 KVM_X86_OP_OPTIONAL(migrate_cmd)
+KVM_X86_OP_OPTIONAL(export_memory)
+KVM_X86_OP_OPTIONAL(import_memory)
 #endif
 
 #undef KVM_X86_OP
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d9291a8a97bb1..9a517bfc2f3a6 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2012,6 +2012,8 @@ struct kvm_x86_ops {
 	int (*gmem_max_mapping_level)(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 	bool (*cap_live_migration)(struct kvm *kvm);
 	int (*migrate_cmd)(struct kvm *kvm, struct kvm_migrate_cmd *cmd);
+	int (*export_memory)(struct kvm *kvm, struct kvm_memory_transfer *mem);
+	int (*import_memory)(struct kvm *kvm, struct kvm_memory_transfer *mem);
 };
 
 struct kvm_x86_nested_ops {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 7064fd709e56d..8a99c665008a3 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7258,6 +7258,37 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
 	return -ENOIOCTLCMD;
 }
 
+static int kvm_vm_ioctl_transfer_memory(struct kvm *kvm, bool import,
+					void __user *argp)
+{
+	struct kvm_memory_transfer mem;
+	int r;
+
+	if (!kvm_x86_call(cap_live_migration)(kvm) ||
+	    (import && !kvm_x86_ops.import_memory) ||
+	    (!import && !kvm_x86_ops.export_memory))
+		return -ENOTTY;
+
+	if (copy_from_user(&mem, argp, sizeof(mem)))
+		return -EFAULT;
+
+	if (mem.reserved || mem.buf.reserved || !mem.nr_gfns)
+		return -EINVAL;
+
+	if (import)
+		r = kvm_x86_call(import_memory)(kvm, &mem);
+	else
+		r = kvm_x86_call(export_memory)(kvm, &mem);
+	if (r > 0)
+		r = -EIO;
+
+	/* Copy back also on an error to report a partially done transfer */
+	if (copy_to_user(argp, &mem, sizeof(mem)))
+		return -EFAULT;
+
+	return r;
+}
+
 int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 {
 	struct kvm *kvm = filp->private_data;
@@ -7639,6 +7670,12 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 			return -EFAULT;
 		break;
 	}
+	case KVM_EXPORT_MEMORY:
+		r = kvm_vm_ioctl_transfer_memory(kvm, false, argp);
+		break;
+	case KVM_IMPORT_MEMORY:
+		r = kvm_vm_ioctl_transfer_memory(kvm, true, argp);
+		break;
 	default:
 		r = -ENOTTY;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e5b227d3a8e40..666bbdf220d65 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1493,6 +1493,10 @@ struct kvm_enc_region {
 #define KVM_GET_SREGS2             _IOR(KVMIO,  0xcc, struct kvm_sregs2)
 #define KVM_SET_SREGS2             _IOW(KVMIO,  0xcd, struct kvm_sregs2)
 
+/* Available with KVM_CAP_LIVE_MIGRATION */
+#define KVM_EXPORT_MEMORY	  _IOWR(KVMIO, 0xe5, struct kvm_memory_transfer)
+#define KVM_IMPORT_MEMORY	  _IOWR(KVMIO, 0xe6, struct kvm_memory_transfer)
+
 #define KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE    (1 << 0)
 #define KVM_DIRTY_LOG_INITIALLY_SET            (1 << 1)
 
@@ -1692,4 +1696,13 @@ struct kvm_migrate_cmd {
 	struct kvm_transfer_buffer buf;
 };
 
+struct kvm_memory_transfer {
+	__u64 gfns;
+	__u32 nr_gfns;
+	__u16 id;
+	__u16 flags;
+	__u64 reserved;
+	struct kvm_transfer_buffer buf;
+};
+
 #endif /* __LINUX_KVM_H */
-- 
2.43.0

[RFC PATCH v2 4/4] KVM: x86: Add optional KVM_EXPORT_VCPU and KVM_IMPORT_VCPU

[Tony Lindgren]

Add support to export and import VCPU for cases where the VCPU state is
only accessible to the guest. Live migration of confidential computing
needs help of KVM for the firmware specific calls at least for TDX.

Introduce optional KVM_EXPORT_VCPU and KVM_IMPORT_VCPU.

Based on earlier code by Wei Wang <wei.w.wang@intel.com>.

Co-developed-by: Kishen Maloor <kishen.maloor@intel.com>
Signed-off-by: Kishen Maloor <kishen.maloor@intel.com>
Signed-off-by: Tony Lindgren <tony.lindgren@linux.intel.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  2 ++
 arch/x86/include/asm/kvm_host.h    |  2 ++
 arch/x86/kvm/x86.c                 | 40 ++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h           |  8 ++++++
 4 files changed, 52 insertions(+)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 173d0c4f1115e..7f110f80d6f82 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -152,6 +152,8 @@ KVM_X86_OP_OPTIONAL_RET0(cap_live_migration)
 KVM_X86_OP_OPTIONAL(migrate_cmd)
 KVM_X86_OP_OPTIONAL(export_memory)
 KVM_X86_OP_OPTIONAL(import_memory)
+KVM_X86_OP_OPTIONAL(export_vcpu)
+KVM_X86_OP_OPTIONAL(import_vcpu)
 #endif
 
 #undef KVM_X86_OP
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 9a517bfc2f3a6..b6362408dab80 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2014,6 +2014,8 @@ struct kvm_x86_ops {
 	int (*migrate_cmd)(struct kvm *kvm, struct kvm_migrate_cmd *cmd);
 	int (*export_memory)(struct kvm *kvm, struct kvm_memory_transfer *mem);
 	int (*import_memory)(struct kvm *kvm, struct kvm_memory_transfer *mem);
+	int (*export_vcpu)(struct kvm_vcpu *vcpu, struct kvm_vcpu_transfer *vcpu_state);
+	int (*import_vcpu)(struct kvm_vcpu *vcpu, struct kvm_vcpu_transfer *vcpu_state);
 };
 
 struct kvm_x86_nested_ops {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8a99c665008a3..e8385326894b1 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6189,6 +6189,38 @@ static int kvm_get_reg_list(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static int kvm_vcpu_ioctl_transfer_vcpu(struct kvm_vcpu *vcpu, bool import,
+					void __user *argp)
+{
+	struct kvm_vcpu_transfer vcpu_state;
+	struct kvm *kvm = vcpu->kvm;
+	int r;
+
+	if (!kvm_x86_call(cap_live_migration)(kvm) ||
+	    (import && !kvm_x86_ops.import_vcpu) ||
+	    (!import && !kvm_x86_ops.export_vcpu))
+		return -ENOTTY;
+
+	if (copy_from_user(&vcpu_state, argp, sizeof(vcpu_state)))
+		return -EFAULT;
+
+	if (vcpu_state.reserved || vcpu_state.buf.reserved)
+		return -EINVAL;
+
+	if (import)
+		r = kvm_x86_call(import_vcpu)(vcpu, &vcpu_state);
+	else
+		r = kvm_x86_call(export_vcpu)(vcpu, &vcpu_state);
+	if (r > 0)
+		r = -EIO;
+
+	/* Copy back also on an error to report a partially done transfer */
+	if (copy_to_user(argp, &vcpu_state, sizeof(vcpu_state)))
+		r = -EFAULT;
+
+	return r;
+}
+
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
@@ -6659,6 +6691,14 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 			goto out;
 		r = kvm_x86_ops.vcpu_mem_enc_ioctl(vcpu, argp);
 		break;
+	case KVM_EXPORT_VCPU: {
+		r = kvm_vcpu_ioctl_transfer_vcpu(vcpu, false, argp);
+		break;
+	}
+	case KVM_IMPORT_VCPU: {
+		r = kvm_vcpu_ioctl_transfer_vcpu(vcpu, true, argp);
+		break;
+	}
 	default:
 		r = -EINVAL;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 666bbdf220d65..0a9aa126daadb 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1496,6 +1496,8 @@ struct kvm_enc_region {
 /* Available with KVM_CAP_LIVE_MIGRATION */
 #define KVM_EXPORT_MEMORY	  _IOWR(KVMIO, 0xe5, struct kvm_memory_transfer)
 #define KVM_IMPORT_MEMORY	  _IOWR(KVMIO, 0xe6, struct kvm_memory_transfer)
+#define KVM_EXPORT_VCPU		  _IOWR(KVMIO, 0xe7, struct kvm_vcpu_transfer)
+#define KVM_IMPORT_VCPU		  _IOWR(KVMIO, 0xe8, struct kvm_vcpu_transfer)
 
 #define KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE    (1 << 0)
 #define KVM_DIRTY_LOG_INITIALLY_SET            (1 << 1)
@@ -1705,4 +1707,10 @@ struct kvm_memory_transfer {
 	struct kvm_transfer_buffer buf;
 };
 
+struct kvm_vcpu_transfer {
+	__u32 flags;
+	__u32 reserved;
+	struct kvm_transfer_buffer buf;
+};
+
 #endif /* __LINUX_KVM_H */
-- 
2.43.0

Re: [RFC PATCH v2 1/4] Documentation: KVM: Add live migration API for confidential guests

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The newly added documentation for the KVM live migration API contains architectural contradictions, backwards numbering, self-referencing links, and typographical errors.
--

commit 3a6c0c9c47ef3277e964b009e8dd06e7aeab932d
Author: Tony Lindgren <tony.lindgren@linux.intel.com>

Documentation: KVM: Add live migration API for confidential guests

For CoCo VMs, the guest memory and vCPU states are not accessible to the
userspace or KVM for live migration. The memory and vCPU states need to be
extracted into encrypted blobs on the source, and decrypted on the
destination. Before live migration, an encryption key needs to be
negotiated between the source and destination.

> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> index a5f9ee92f43e8..9d546d288af5f 100644
> --- a/Documentation/virt/kvm/api.rst
> +++ b/Documentation/virt/kvm/api.rst
[ ... ]
> +The transfer buffer size is vendor specific.
> +
> +For the transfer buffer, seeo :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.

[Severity: Low]
Is this "seeo" a typo for "see"?

> +
> +For memory import, see also :ref:`KVM_IMPORT_MEMORY <KVM_IMPORT_MEMORY>`.
> +
> +
> +.. _KVM_IMPORT_MEMORY:
> +
> +4.147 KVM_IMPORT_MEMORY
> +-----------------------
[ ... ]
> +The transfer buffer size is vendor specific.
> +
> +For kvm_memory_transfer, see :ref:`KVM_EXPORT_MEMORY <KVM_EXPORT_MEMORY>`.
> +
> +For the transfer buffer, seeo :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.

[Severity: Low]
Is there another "seeo" typo here?

> +
> +.. _KVM_EXPORT_VCPU:
> +
> +4.149 KVM_EXPORT_VCPU
> +---------------------
> +:Capability: KVM_CAP_LIVE_MIGRATION
> +:Architectures: arm64, x86

[Severity: Low]
Is it intentional that KVM_EXPORT_VCPU is numbered 4.149 before
KVM_IMPORT_VCPU which is numbered 4.148?

Also, KVM_EXPORT_VCPU lists arm64 as a supported architecture, but relies on
KVM_CAP_LIVE_MIGRATION which is documented as x86 only below. Should arm64 be
removed here, or added to the capability documentation?

> +:Type: vcpu ioctl
> +:Parameters: struct kvm_vcpu_transfer (in/out)
> +:Returns: 0 on success, < 0 on error
[ ... ]
> +.. _KVM_IMPORT_VCPU:
> +
> +4.148 KVM_IMPORT_VCPU
> +---------------------
> +
> +:Capability: KVM_CAP_LIVE_MIGRATION
> +:Architectures: arm64, x86
> +:Type: vcpu ioctl
> +:Parameters: struct kvm_vcpu_transfer (in/out)
> +:Returns: 0 on success, < 0 on error
> +
> +Allows userspace to request the host to import a VCPU state from a userspace
> +buffer.
> +
> +The VCPU state may not be directly accessible to KVM because of encryption. For
> +confidential computing, the VCPU state is encrypted and only accessible to the
> +guest.
> +
> +For vcpu_transfer and vCPU import, see :ref:`KVM_IMPORT_VCPU <KVM_IMPORT_VCPU>`.

[Severity: Low]
Does this circular reference point to itself? Should this point to
KVM_EXPORT_VCPU instead to help the reader find the struct kvm_vcpu_transfer
definition?

> +
> +For the transfer buffer, see also :ref:`KVM_MIGRATE_CMD <KVM_MIGRATE_CMD>`.
> +
>  .. _kvm_run:
>  
>  5. The kvm_run structure
> @@ -9493,6 +9687,17 @@ take care to differentiate between these cases.
>  The presence of this capability indicates that the nested KVM guest can
>  start in ESA mode.
>  
> +8.48 KVM_CAP_LIVE_MIGRATION
> +---------------------------
> +
> +:Architectures: x86
> +:Target: VM
> +:Parameters: None

[Severity: Low]
The KVM_CAP_LIVE_MIGRATION capability is documented here as x86 only, but
KVM_EXPORT_VCPU and KVM_IMPORT_VCPU state they are for arm64 and x86. Is
this capability intended to support arm64 as well?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831071304.762939-1-tony.lindgren@linux.intel.com?part=1

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The 'flags' field in the new 'struct kvm_migrate_cmd' UAPI is not validated to be zero.
--

commit b50bc10fe65ad9a54c9b979c445505e5baf76592
Author: Tony Lindgren <tony.lindgren@linux.intel.com>

KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

This patch introduces the KVM_CAP_LIVE_MIGRATION capability and the
KVM_MIGRATE_CMD ioctl. These additions allow KVM to configure hardware-specific
live migration functions, which is necessary for confidential guests like TDX.

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index afcac1042947a..7064fd709e56d 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c

[ ... ]

> @@ -7614,6 +7617,28 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
>  		r = kvm_vm_ioctl_set_msr_filter(kvm, &filter);
>  		break;
>  	}
> +	case KVM_MIGRATE_CMD: {
> +		struct kvm_migrate_cmd cmd;
> +
> +		if (!kvm_x86_ops.migrate_cmd ||
> +		    !kvm_x86_call(cap_live_migration)(kvm))
> +			return -ENOTTY;
> +
> +		if (copy_from_user(&cmd, argp, sizeof(cmd)))
> +			return -EFAULT;
> +
> +		if (cmd.reserved || cmd.buf.reserved)
> +			return -EINVAL;

[Severity: High]
Does this code also need to validate that the new flags field is zero?

If uninitialized memory or garbage is passed by userspace in cmd.flags, it
will currently be silently ignored. When a future kernel introduces a new
flag, old userspace applications passing uninitialized flags will inadvertently
enable it, which could permanently break backwards compatibility.

> +
> +		r = kvm_x86_call(migrate_cmd)(kvm, &cmd);
> +		if (r > 0)
> +			r = -EIO;
> +
> +		/* Copy back also on an error to report a partially done command */
> +		if (copy_to_user(argp, &cmd, sizeof(cmd)))
> +			return -EFAULT;
> +		break;
> +	}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831071304.762939-1-tony.lindgren@linux.intel.com?part=2

Re: [RFC PATCH v2 3/4] KVM: x86: Add optional KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unvalidated UAPI fields `flags` and `id` in `struct kvm_memory_transfer` can lead to ABI breakage.
--

commit b6f49e80a58eebaaa2a6d123950fd36709972a10
Author: Tony Lindgren <tony.lindgren@linux.intel.com>

KVM: x86: Add optional KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY

This commit adds support to export and import KVM memory for cases where
the memory is only accessible to the guest. It introduces the optional
KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY ioctls to assist with live
migration of confidential computing.

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 7064fd709e56d..8a99c665008a3 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
[ ... ]
> @@ -7258,6 +7258,37 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
>  	return -ENOIOCTLCMD;
>  }
>  
> +static int kvm_vm_ioctl_transfer_memory(struct kvm *kvm, bool import,
> +					void __user *argp)
> +{
> +	struct kvm_memory_transfer mem;
> +	int r;
> +
> +	if (!kvm_x86_call(cap_live_migration)(kvm) ||
> +	    (import && !kvm_x86_ops.import_memory) ||
> +	    (!import && !kvm_x86_ops.export_memory))
> +		return -ENOTTY;
> +
> +	if (copy_from_user(&mem, argp, sizeof(mem)))
> +		return -EFAULT;
> +
> +	if (mem.reserved || mem.buf.reserved || !mem.nr_gfns)
> +		return -EINVAL;

[Severity: Medium]
Does this input validation for the kvm_memory_transfer UAPI struct miss
checking mem.flags == 0 and mem.id == 0?

Failing to validate that unused UAPI fields are zero allows userspace to pass
uninitialized garbage. If KVM later assigns specific meaning to these fields,
older applications that unintentionally passed non-zero values could break,
leading to an ABI compatibility issue.

Standard Linux UAPI design usually requires enforcing these unused or reserved
fields to be zero for future extensibility.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831071304.762939-1-tony.lindgren@linux.intel.com?part=3

Re: [RFC PATCH v2 4/4] KVM: x86: Add optional KVM_EXPORT_VCPU and KVM_IMPORT_VCPU

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing validation of the flags field in UAPI struct kvm_vcpu_transfer breaks forward compatibility.
--

commit 4d7dccde0750ca7fd34158829ad4120d967ac2c3
Author: Tony Lindgren <tony.lindgren@linux.intel.com>

KVM: x86: Add optional KVM_EXPORT_VCPU and KVM_IMPORT_VCPU

This commit adds support to export and import a VCPU for cases where the
VCPU state is only accessible to the guest. It introduces the new
KVM_EXPORT_VCPU and KVM_IMPORT_VCPU ioctls, along with the required
kvm_vcpu_transfer structure.

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 8a99c665008a3..e8385326894b1 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c

[ ... ]

> @@ -6189,6 +6189,38 @@ static int kvm_get_reg_list(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +static int kvm_vcpu_ioctl_transfer_vcpu(struct kvm_vcpu *vcpu, bool import,
> +					void __user *argp)
> +{
> +	struct kvm_vcpu_transfer vcpu_state;
> +	struct kvm *kvm = vcpu->kvm;
> +	int r;

[ ... ]

> +	if (copy_from_user(&vcpu_state, argp, sizeof(vcpu_state)))
> +		return -EFAULT;
> +
> +	if (vcpu_state.reserved || vcpu_state.buf.reserved)
> +		return -EINVAL;

[Severity: High]
Should we also validate the flags field here to ensure forward compatibility?

The new kvm_vcpu_transfer structure introduces a flags field, but this
validation step only explicitly rejects non-zero reserved fields.

If the kernel silently ignores non-zero flags, userspace might
inadvertently pass uninitialized or arbitrary values without receiving
an error. If KVM later assigns meaning to these flags, old userspace
programs that have been unknowingly passing garbage could unexpectedly
trigger new behaviors or break.

Would it be appropriate to require that vcpu_state.flags is zero for now?

> +
> +	if (import)
> +		r = kvm_x86_call(import_vcpu)(vcpu, &vcpu_state);

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260831071304.762939-1-tony.lindgren@linux.intel.com?part=4

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Mon, Aug 31, 2026 at 07:23:11AM +0000, sashiko-bot@kernel.org wrote:
> Author: Tony Lindgren <tony.lindgren@linux.intel.com>
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -7614,6 +7617,28 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
> >  		r = kvm_vm_ioctl_set_msr_filter(kvm, &filter);
> >  		break;
> >  	}
> > +	case KVM_MIGRATE_CMD: {
> > +		struct kvm_migrate_cmd cmd;
> > +
> > +		if (!kvm_x86_ops.migrate_cmd ||
> > +		    !kvm_x86_call(cap_live_migration)(kvm))
> > +			return -ENOTTY;
> > +
> > +		if (copy_from_user(&cmd, argp, sizeof(cmd)))
> > +			return -EFAULT;
> > +
> > +		if (cmd.reserved || cmd.buf.reserved)
> > +			return -EINVAL;
> 
> [Severity: High]
> Does this code also need to validate that the new flags field is zero?
> 
> If uninitialized memory or garbage is passed by userspace in cmd.flags, it
> will currently be silently ignored. When a future kernel introduces a new
> flag, old userspace applications passing uninitialized flags will inadvertently
> enable it, which could permanently break backwards compatibility.

Could be zero, or we could check for the known flags.

We are currently using the following flags for KVM_MIGRATE_CMD
sub-commands for TDX:

1. KVM_MIGRATE_SETUP
====================

TDX_MIGRATE_SETUP_SESSION       exchange the migration keys
TDX_MIGRATE_IMMUTABLE_STATE     transfer the TDX immutable state

2. KVM_MIGRATE_ITERATION
========================

TDX_MIGRATE_IN_ORDER_DONE       end the in-order migration

3. KVM_MIGRATE_STOP_AND_COPY
============================

TDX_MIGRATE_STOP_COPY_PAUSE     pause the TD
TDX_MIGRATE_STOP_COPY_TD_STATE  transfer the mutable TD state

4. KVM_MIGRATE_END
==================

TDX_MIGRATE_ABORT               abort migration

Maybe all the above TDX specific flags could be turned into generic
flags for the sub-commands.

Re: [RFC PATCH v2 3/4] KVM: x86: Add optional KVM_EXPORT_MEMORY and KVM_IMPORT_MEMORY

[Tony Lindgren]

On Mon, Aug 31, 2026 at 07:23:47AM +0000, sashiko-bot@kernel.org wrote:
> commit b6f49e80a58eebaaa2a6d123950fd36709972a10
> Author: Tony Lindgren <tony.lindgren@linux.intel.com>
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> [ ... ]
> > @@ -7258,6 +7258,37 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
> >  	return -ENOIOCTLCMD;
> >  }
> >  
> > +static int kvm_vm_ioctl_transfer_memory(struct kvm *kvm, bool import,
> > +					void __user *argp)
> > +{
> > +	struct kvm_memory_transfer mem;
> > +	int r;
> > +
> > +	if (!kvm_x86_call(cap_live_migration)(kvm) ||
> > +	    (import && !kvm_x86_ops.import_memory) ||
> > +	    (!import && !kvm_x86_ops.export_memory))
> > +		return -ENOTTY;
> > +
> > +	if (copy_from_user(&mem, argp, sizeof(mem)))
> > +		return -EFAULT;
> > +
> > +	if (mem.reserved || mem.buf.reserved || !mem.nr_gfns)
> > +		return -EINVAL;
> 
> [Severity: Medium]
> Does this input validation for the kvm_memory_transfer UAPI struct miss
> checking mem.flags == 0 and mem.id == 0?

Yes flags is unused for memory transfers at least for TDX. The id TDX can
eventually use for the thread index for QEMU multifd transfers.

Re: [RFC PATCH v2 4/4] KVM: x86: Add optional KVM_EXPORT_VCPU and KVM_IMPORT_VCPU

[Tony Lindgren]

On Mon, Aug 31, 2026 at 07:23:53AM +0000, sashiko-bot@kernel.org wrote:
> commit 4d7dccde0750ca7fd34158829ad4120d967ac2c3
> Author: Tony Lindgren <tony.lindgren@linux.intel.com>
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> 
> [ ... ]
> 
> > @@ -6189,6 +6189,38 @@ static int kvm_get_reg_list(struct kvm_vcpu *vcpu,
> >  	return 0;
> >  }
> >  
> > +static int kvm_vcpu_ioctl_transfer_vcpu(struct kvm_vcpu *vcpu, bool import,
> > +					void __user *argp)
> > +{
> > +	struct kvm_vcpu_transfer vcpu_state;
> > +	struct kvm *kvm = vcpu->kvm;
> > +	int r;
> 
> [ ... ]
> 
> > +	if (copy_from_user(&vcpu_state, argp, sizeof(vcpu_state)))
> > +		return -EFAULT;
> > +
> > +	if (vcpu_state.reserved || vcpu_state.buf.reserved)
> > +		return -EINVAL;
> 
> [Severity: High]
> Should we also validate the flags field here to ensure forward compatibility?
> 
> The new kvm_vcpu_transfer structure introduces a flags field, but this
> validation step only explicitly rejects non-zero reserved fields.
> 
> If the kernel silently ignores non-zero flags, userspace might
> inadvertently pass uninitialized or arbitrary values without receiving
> an error. If KVM later assigns meaning to these flags, old userspace
> programs that have been unknowingly passing garbage could unexpectedly
> trigger new behaviors or break.
> 
> Would it be appropriate to require that vcpu_state.flags is zero for now?

Yes flags is unused for vCPU transfers at least for TDX and can be zero
for now.

Re: [RFC PATCH v2 0/4] Add KVM API for confidential guest live migration

[Artem Bityutskiy]

On Mon, 2026-08-31 at 10:13 +0300, Tony Lindgren wrote:
> Tom, since you mentioned that AMD SEV-SNP and Intel TDX live migration
> sound similar, can you please take a look how the API might work for
> SEV-SNP?

The presented example uAPIs were designed to fit the TDX live migration
flow, with the intent that they could also be used by other CoCo VMs.

It would be super great if someone could commend on how suitable these
uAPIs are for AMD and ARM flows.

AFAIU, what is generic in the presented uAPI is more of usage pattern.

- The order in which QEMU calls them.
- The idea that QEMU/KVM is a transport for opaque blobs.

The proposed container - 'struct kvm_transfer_buffer' - only has address
and a size. The vendor defines the layout of the data in it.

Some example high-level topics that would be nice to get feedback on:

- Can we come up with a single set of generic migration uAPIs for different
  CoCo models?
- Or should some uAPIs be generic while others are vendor-specific?
- Or should each CoCo model have its own vendor-specific set of migration
  uAPIs?
- Should the same uAPIs also support traditional VMs? But the only use-case
  I imagine here is "for testing purposes".

> Artem has put together a brief description below of the example API and the
> migration flow:

.. snip ...

> Migration flow
> ==============
> 
>  Source host                                      Destination host
>  ===========                                      ================
> 
>  CMD(SETUP/SESSION)          <--- setup msgs ---> CMD(SETUP/SESSION)
>        |    (repeated)                                  |
>  CMD(SETUP/IMMUTABLE_STATE)  - immutable state -> CMD(SETUP/IMMUTABLE_STATE)
>        |                                                |
>  KVM_GET_DIRTY_LOG                                      |
>  KVM_EXPORT_MEMORY           --- memory data ---> KVM_IMPORT_MEMORY
>  CMD(ITERATION)              --- epoch token ---> CMD(ITERATION)
>        |    (repeat until convergence)                  |
>  CMD(STOP_AND_COPY/PAUSE)                               |
>  CMD(STOP_AND_COPY/TD_STATE) --- VM state ------> CMD(STOP_AND_COPY/TD_STATE)
>  KVM_EXPORT_VCPU             --- vCPU state ----> KVM_IMPORT_VCPU
>  KVM_EXPORT_MEMORY           -- final memory ---> KVM_IMPORT_MEMORY
>  CMD(ITERATION/DONE)         --- start token ---> CMD(ITERATION)
>        |                                                |
>  CMD(END)                                         CMD(END)

... snip ...


As I mentioned, the example uAPI is modeled around the TDX migration flow.
In case it helps the reader, here is a summary of that flow that was
presented in PUCK.

It describes what the TDX module offers today and focuses on pre-copy
migration. This is our interpretation of the TDX specifications, not a
specification itself, and may contain errors or omissions. Please refer to
the official TDX specifications for authoritative information.

Migration Overview
------------------

In TDX, the TDX module implements the migration logic. The VMM drives
migration by issuing seamcalls to the source and destination TDX modules and
transports the resulting encrypted blobs between them. The VMM does not need
to know the contents of those blobs.

The TDX security model enforces two hard rules:

- Only one instance of the TD may run at a time. Either the source or the
  destination may run, but never both. In other words, cloning a TD is not
  allowed.
- When migration completes, the destination must have the same memory and
  vCPU state as the source. It must not end up with a partial or mixed
  state.

Simple Overview
---------------

Run the migration setup session
  |
  v
Transfer immutable TD state
  |
  v
Copy dirty memory pages while the TD runs <-------+
  |                                               |
  +-- iterate until convergence criteria is met --+
  |
  v
Pause the source TD and copy the remaining state
  |
  v
Start the destination TD

The migration process starts with a setup session. During the setup session,
the source and destination TDX modules exchange encrypted blobs and
establish migration encryption keys. For example, these keys protect memory
contents transferred during migration.

Next, the source transfers the immutable TD state to the destination and
initializes the destination TD. This state includes the read-only VM data,
such as its vCPU count and topology.

The source and destination then perform iterative memory-copy rounds. In
each round, the source scans for memory pages that need to be migrated and
exports them in encrypted form. The destination imports the pages. The
rounds continue until the number of pages that change between rounds is
small enough to meet the convergence criteria.

The source TD is then paused. The VMM transfers the remaining TD state, such
as vCPU state, and the final dirty memory pages. Finally, the destination TD
is started and the migration completes.

Notice that the VMM's role in this process is to issue the required
seamcalls and send encrypted blobs between the source and destination. The
VMM does not need to know the contents of those blobs.

More Detailed Overview
----------------------

The following diagram shows the TDX seamcalls used by the source and
destination:

 Source host                                          Destination host
 ===========                                          ================

 TDH.MIG.SETUP        -- crypto keys, attestation ->  TDH.MIG.SETUP
       |                                                    |
 TDH.EXPORT.STATE.IMMUTABLE -- read-only TD state ->
TDH.IMPORT.STATE.IMMUTABLE
       |                                                    |
 TDH.MEM.SCAN.RANGE                                         |
 TDH.MEM.TRACK + IPIs                                       |
 TDH.EXPORT.MEM        ------ memory data --------->  TDH.IMPORT.MEM
 TDH.EXPORT.TRACK      ------ epoch token --------->  TDH.IMPORT.TRACK
       |                                                    |
 TDH.EXPORT.PAUSE                                           |
       |                                                    |
 TDH.EXPORT.STATE.TD   ------ global TD state ----->  TDH.IMPORT.STATE.TD
       |                                                    |
 TDH.EXPORT.STATE.VP   ------ vCPU state ---------->  TDH.IMPORT.STATE.VP
       |                                                    |
 TDH.MEM.SCAN.COMP                                          |
 TDH.MEM.TRACK + IPIs                                       |
 TDH.EXPORT.MEM        ------ final memory data --->  TDH.IMPORT.MEM
 TDH.EXPORT.TRACK      ------ start token --------->  TDH.IMPORT.TRACK
                                                            |
                                                      TDH.IMPORT.END

The source and destination go through the following stages.

Setup
-----

The VMM issues TDH.MIG.SETUP on the source and destination TDX modules
iteratively to perform the setup session. The modules return status and may
also return an encrypted blob, which the VMM passes between the two sides.
In other words, the migration protocol is between the two TDX modules, while
the VMM is simply the transport mechanism. The setup session performs mutual
attestation, establishes trust between the modules, establishes migration
encryption keys, and loads the migration policy. It completes when the TDX
module returns success.

Immutable State Transfer
------------------------

The source calls TDH.EXPORT.STATE.IMMUTABLE to export the immutable TD
state. The destination VMM creates the destination TD skeleton and
configures migration streams, then calls TDH.IMPORT.STATE.IMMUTABLE, which
finalizes the destination TD initialization.

Iterative Memory Copy
---------------------

While the source TD continues running, the source and destination run
iterative memory copy rounds:

- The source calls TDH.MEM.SCAN.RANGE to find migration candidate pages.
- The source calls TDH.MEM.TRACK and sends IPIs to the TD's vCPUs in order
  to ensure the dirty page scanning algorithm correctness.
- The source calls TDH.EXPORT.MEM to export private memory pages, which the
  VMM sends to the destination.
- The destination calls TDH.IMPORT.MEM to import the received memory pages.
- The source calls TDH.EXPORT.TRACK to generate an epoch token. The VMM
  sends it to the destination, which calls TDH.IMPORT.TRACK to consume the
  token. This verifies that all data exported from the source was imported
  on the destination.

The rounds continue until the number of pages that change between rounds is
small enough to meet the convergence criteria.

Stop and Copy
-------------

The VMM pauses the source TD, which begins the downtime period, then calls
TDH.EXPORT.PAUSE to start the TDX-enforced blackout period. The source then
calls TDH.EXPORT.STATE.TD to export mutable TD-scope state and
TDH.EXPORT.STATE.VP to export mutable state for each vCPU. It then performs
the final dirty page scan, runs TDH.MEM.TRACK and sends IPIs, and calls
TDH.EXPORT.MEM to export the final dirty pages as encrypted blobs.

The destination calls TDH.IMPORT.STATE.TD to import mutable TD-scope state
and TDH.IMPORT.STATE.VP to import the mutable state of each vCPU. It calls
TDH.IMPORT.MEM to import the final dirty pages.

After exporting the final dirty pages, the source's last migration seamcall
is TDH.EXPORT.TRACK with IN_ORDER_DONE=1, which generates the start token.
The VMM sends the start token to the destination. The destination calls
TDH.IMPORT.TRACK with this token. This verifies that the mutable TD state
has been imported and allows the destination to start the TD. Finally, the
destination calls TDH.IMPORT.END, which ends the migration.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Mon, Aug 31, 2026 at 10:13:02AM +0300, Tony Lindgren wrote:
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1670,4 +1673,23 @@ struct kvm_pre_fault_memory {
>  	__u64 padding[5];
>  };
>  
> +#define KVM_MIGRATE_SETUP		0
> +#define KVM_MIGRATE_ITERATION		1
> +#define KVM_MIGRATE_STOP_AND_COPY	2
> +#define KVM_MIGRATE_ABORT		3
> +#define KVM_MIGRATE_END			4
> +
> +struct kvm_transfer_buffer {
> +	__u64 address;
> +	__u32 size;
> +	__u32 reserved;
> +};
> +
> +struct kvm_migrate_cmd {
> +	__u16 command;
> +	__u16 flags;
> +	__u32 reserved;
> +	struct kvm_transfer_buffer buf;
> +};

For the common flags, KVM_MIGRATE_CMD probably should have migration
direction. Or maybe we could have KVM_EXPORT_CMD and KVM_IMPORT_CMD.

The migration direction is needed early for TDX. We currently pass a flag
for delayed init for the migration destination in KVM_TDX_INIT_VM. This
is to prevent the TD and vCPU init SEAMCALLs on the destination. The TD
and vCPU are initialized only later on with the import calls.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Jörg Rödel]

On Mon, Sep 07, 2026 at 02:53:05PM +0300, Tony Lindgren wrote:
> On Mon, Aug 31, 2026 at 10:13:02AM +0300, Tony Lindgren wrote:
> > --- a/include/uapi/linux/kvm.h
> > +++ b/include/uapi/linux/kvm.h
> > @@ -1670,4 +1673,23 @@ struct kvm_pre_fault_memory {
> >  	__u64 padding[5];
> >  };
> >  
> > +#define KVM_MIGRATE_SETUP		0
> > +#define KVM_MIGRATE_ITERATION		1
> > +#define KVM_MIGRATE_STOP_AND_COPY	2
> > +#define KVM_MIGRATE_ABORT		3
> > +#define KVM_MIGRATE_END			4
> > +
> > +struct kvm_transfer_buffer {
> > +	__u64 address;
> > +	__u32 size;
> > +	__u32 reserved;
> > +};
> > +
> > +struct kvm_migrate_cmd {
> > +	__u16 command;
> > +	__u16 flags;
> > +	__u32 reserved;
> > +	struct kvm_transfer_buffer buf;
> > +};
> 
> For the common flags, KVM_MIGRATE_CMD probably should have migration
> direction. Or maybe we could have KVM_EXPORT_CMD and KVM_IMPORT_CMD.

The direction is always the same over a single live migration session, right?
So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
and KVM_IMPORT_CMD seems to be a cleaner ABI.


-Joerg

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Artem Bityutskiy]

On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
> The direction is always the same over a single live migration session, right?
> So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
> and KVM_IMPORT_CMD seems to be a cleaner ABI.

Yes, the source stays the source, and the destination stays the destination
for the entire session.

This is a special case of a broader question I keep coming back to:

** How much state should KVM keep about a migration session? **

For direction specifically, we could pass it to KVM on every call and let
KVM stay stateless about it, or KVM could record it once and remember it for
the rest of the session.

But in general. And this is addressed not just to Jörg, but community.

Traditional VM migration is driven by QEMU. KVM provides building blocks
such as dirty page tracking and vCPU state get/set APIs, but it does not
track the overall migration session. The migration session state lives in
QEMU.

Our TDX live migration prototype keeps some per-migration state in KVM,
for example the direction, the migration phase (setup done, started,
paused, and so on). This lets use validate inputs and issue the correct TDX
module seamcalls from KVM.

A different uAPI could shift this balance either way.

In the **extreme** case, we could expose a uAPI for each migration
seamcall. KVM would then just pass inputs and outputs between the TDX
module and QEMU, staying a thin layer with no per-session state. We have
not tried this, but it illustrates the trade-off.

So where is the right boundary for migration state in KVM?

Should KVM manage none of it, or is some state acceptable?

Would be interesting to know what KVM community thinks on this.

Thanks,
Artem.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
> Our TDX live migration prototype keeps some per-migration state in KVM,
> for example the direction, the migration phase (setup done, started,
> paused, and so on). This lets use validate inputs and issue the correct TDX
> module seamcalls from KVM.

Yeah for TDX, we currently keep track of some of the TDX module state for
migration. For most part it can be done with the existing kvm_tdx->state.
The paused state is additional TD_STATE_PAUSED.

Some states are trickier though, the setup done state means the TDX module
has migration keys configured. If the keys are not installed, the
migration SEAMCALLs return errors. Does the kernel need to keep track of
this? With proper errors returned, maybe not.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
> On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
> > The direction is always the same over a single live migration session, right?
> > So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
> > and KVM_IMPORT_CMD seems to be a cleaner ABI.

OK
 
> Yes, the source stays the source, and the destination stays the destination
> for the entire session.
> 
> This is a special case of a broader question I keep coming back to:
> 
> ** How much state should KVM keep about a migration session? **
> 
> For direction specifically, we could pass it to KVM on every call and let
> KVM stay stateless about it, or KVM could record it once and remember it for
> the rest of the session.

Yes the "record and remember" is another option, it could be a sub-command
something like KVM_MIGRATE_DIRECTION.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Kishen Maloor]

On 9/7/26 9:43 PM, Tony Lindgren wrote:
> On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
>> On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
>>> The direction is always the same over a single live migration session, right?
>>> So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
>>> and KVM_IMPORT_CMD seems to be a cleaner ABI.
> 
> OK


Just sharing an alternate point of view:

The roles are fixed over a migration session. A split ABI is certainly more 
self-describing, but it restates that invariant on every call, and therefore 
also permits it to be contradicted -- a failure mode that does not otherwise
exist. With a single KVM_MIGRATE_CMD and a per-session role recorded once (more 
on that below), there is no need for per-call policing: the role could be 
checked once when the session is established.

Along these lines: it raises a question of whether the MEMORY and VCPU
calls should be coalesced as well into KVM_MIGRATE_MEMORY and KVM_MIGRATE_VCPU.
As posted, direction is implicit for KVM_MIGRATE_CMD but encoded in the ioctl
number for those transfers, so collapsing them would at least make the uAPI
consistent about where direction comes from, and free two ioctls.

>  
>> ...
>>
>> For direction specifically, we could pass it to KVM on every call and let
>> KVM stay stateless about it, or KVM could record it once and remember it for
>> the rest of the session.
> 
> Yes the "record and remember" is another option, it could be a sub-command
> something like KVM_MIGRATE_DIRECTION.


Agreed on record-and-remember, with a refinement on scope.

A VM that was migrated in can later be migrated out, so the role is not a
property of the VM -- it has to be recorded per session. SETUP is the call
that starts a migration and runs ahead of all other migration calls, so its
arguments look like the natural place for userspace to state the role; a
separate sub-command would need its own scope rules relative to SETUP.
KVM can then ask the vendor layer whether the requested role is permitted for
this VM -- only it knows the confidential-VM state -- and on success record it
in generic KVM state, where it then selects the export or import callbacks.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Tue, Sep 08, 2026 at 05:22:30PM -0700, Kishen Maloor wrote:
> On 9/7/26 9:43 PM, Tony Lindgren wrote:
> > On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
> >> On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
> >>> The direction is always the same over a single live migration session, right?
> >>> So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
> >>> and KVM_IMPORT_CMD seems to be a cleaner ABI.
> > 
> > OK
> 
> 
> Just sharing an alternate point of view:
> 
> The roles are fixed over a migration session. A split ABI is certainly more 
> self-describing, but it restates that invariant on every call, and therefore 
> also permits it to be contradicted -- a failure mode that does not otherwise
> exist. With a single KVM_MIGRATE_CMD and a per-session role recorded once (more 
> on that below), there is no need for per-call policing: the role could be 
> checked once when the session is established.

There are two occasions the role is set or changed. On starting the
destination the incoming role needs to configured at least for TDX.
And then after the migration, the role changes if re-migrated.

I don't think there are other cases for role change, maybe cancelled
migration could require that for some hardware possibly.
 
> Along these lines: it raises a question of whether the MEMORY and VCPU
> calls should be coalesced as well into KVM_MIGRATE_MEMORY and KVM_MIGRATE_VCPU.
> As posted, direction is implicit for KVM_MIGRATE_CMD but encoded in the ioctl
> number for those transfers, so collapsing them would at least make the uAPI
> consistent about where direction comes from, and free two ioctls.

Using naming KVM_TRANSFER_MEMORY and KVM_TRANSFER_VCPU might be more
descriptive?

Eventually these same commands could be used to save the state to disk
for power management use.

And going back to the dmaengine like analogy of what is being done..

The transfer direction flags could be KVM_TRANSFER_FROM_GUEST and
KVM_TRANSFER_TO_GUEST?
  
> >> For direction specifically, we could pass it to KVM on every call and let
> >> KVM stay stateless about it, or KVM could record it once and remember it for
> >> the rest of the session.
> > 
> > Yes the "record and remember" is another option, it could be a sub-command
> > something like KVM_MIGRATE_DIRECTION.
> 
> 
> Agreed on record-and-remember, with a refinement on scope.
> 
> A VM that was migrated in can later be migrated out, so the role is not a
> property of the VM -- it has to be recorded per session. SETUP is the call
> that starts a migration and runs ahead of all other migration calls, so its
> arguments look like the natural place for userspace to state the role; a
> separate sub-command would need its own scope rules relative to SETUP.
> KVM can then ask the vendor layer whether the requested role is permitted for
> this VM -- only it knows the confidential-VM state -- and on success record it
> in generic KVM state, where it then selects the export or import callbacks.

The role can change, but it can be VM specific for starting the migration
destination even before migration is started.

At least for TDX we need to specify direction for migration destination on
init to prevent fully initializing the TD and vCPUs.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Kishen Maloor]

On 9/8/26 11:57 PM, Tony Lindgren wrote:
> On Tue, Sep 08, 2026 at 05:22:30PM -0700, Kishen Maloor wrote:
>> On 9/7/26 9:43 PM, Tony Lindgren wrote:
>>> On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
>>>> On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
>>>>> The direction is always the same over a single live migration session, right?
>>>>> So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
>>>>> and KVM_IMPORT_CMD seems to be a cleaner ABI.
>>>
>>> OK
>>
>>
>> Just sharing an alternate point of view:
>>
>> The roles are fixed over a migration session. A split ABI is certainly more 
>> self-describing, but it restates that invariant on every call, and therefore 
>> also permits it to be contradicted -- a failure mode that does not otherwise
>> exist. With a single KVM_MIGRATE_CMD and a per-session role recorded once (more 
>> on that below), there is no need for per-call policing: the role could be 
>> checked once when the session is established.
> 
> There are two occasions the role is set or changed. On starting the
> destination the incoming role needs to configured at least for TDX.
> And then after the migration, the role changes if re-migrated.

A destination TD needs a directive to not initialize the TD and its vCPUs.
It comes from its launch parameters (e.g. QEMU cmdline) which selects the
delayed_init path. That is a construction directive though, and it applies
only to a destination -- a source needs nothing at init. A session role is
symmetric and is what the transfer calls consume, so I don't think the two
need to be the same thing.

> I don't think there are other cases for role change, maybe cancelled
> migration could require that for some hardware possibly.

Architecturally, per-session scoping of migration roles should be
straightforward with any platform: each side asserts a role at the start of every 
migration session and vendor code will either accept or reject the stated role.

>> Along these lines: it raises a question of whether the MEMORY and VCPU
>> calls should be coalesced as well into KVM_MIGRATE_MEMORY and KVM_MIGRATE_VCPU.
>> As posted, direction is implicit for KVM_MIGRATE_CMD but encoded in the ioctl
>> number for those transfers, so collapsing them would at least make the uAPI
>> consistent about where direction comes from, and free two ioctls.
> 
> Using naming KVM_TRANSFER_MEMORY and KVM_TRANSFER_VCPU might be more
> descriptive?
> 
> Eventually these same commands could be used to save the state to disk
> for power management use.

Sure, and the save-to-disk case is a good argument for a more generic name.

> And going back to the dmaengine like analogy of what is being done..
> 
> The transfer direction flags could be KVM_TRANSFER_FROM_GUEST and
> KVM_TRANSFER_TO_GUEST?

FROM_GUEST/TO_GUEST still encodes direction per call, which is the open
question above. If direction is a per-session property, then KVM_TRANSFER_MEMORY
and KVM_TRANSFER_VCPU are sufficient on their own -- no direction flag, and no
separate export/import ioctls.

And if we settle on a per-session property, then SETUP could conceivably state a
role for a non-migration transfer session as well.

>   
>>>> For direction specifically, we could pass it to KVM on every call and let
>>>> KVM stay stateless about it, or KVM could record it once and remember it for
>>>> the rest of the session.
>>>
>>> Yes the "record and remember" is another option, it could be a sub-command
>>> something like KVM_MIGRATE_DIRECTION.
>>
>>
>> Agreed on record-and-remember, with a refinement on scope.
>>
>> A VM that was migrated in can later be migrated out, so the role is not a
>> property of the VM -- it has to be recorded per session. SETUP is the call
>> that starts a migration and runs ahead of all other migration calls, so its
>> arguments look like the natural place for userspace to state the role; a
>> separate sub-command would need its own scope rules relative to SETUP.
>> KVM can then ask the vendor layer whether the requested role is permitted for
>> this VM -- only it knows the confidential-VM state -- and on success record it
>> in generic KVM state, where it then selects the export or import callbacks.
> 
> The role can change, but it can be VM specific for starting the migration
> destination even before migration is started.
> 
> At least for TDX we need to specify direction for migration destination on
> init to prevent fully initializing the TD and vCPUs.

As mentioned above, that is a destination launch time directive that we needn't
conflate with a migration/transfer session role.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Wed, Sep 09, 2026 at 06:11:11PM -0700, Kishen Maloor wrote:
> On 9/8/26 11:57 PM, Tony Lindgren wrote:
> > On Tue, Sep 08, 2026 at 05:22:30PM -0700, Kishen Maloor wrote:
> >> On 9/7/26 9:43 PM, Tony Lindgren wrote:
> >>> On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
> >>>> On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
> >>>>> The direction is always the same over a single live migration session, right?
> >>>>> So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
> >>>>> and KVM_IMPORT_CMD seems to be a cleaner ABI.
> >>>
> >>> OK
> >>
> >>
> >> Just sharing an alternate point of view:
> >>
> >> The roles are fixed over a migration session. A split ABI is certainly more 
> >> self-describing, but it restates that invariant on every call, and therefore 
> >> also permits it to be contradicted -- a failure mode that does not otherwise
> >> exist. With a single KVM_MIGRATE_CMD and a per-session role recorded once (more 
> >> on that below), there is no need for per-call policing: the role could be 
> >> checked once when the session is established.
> > 
> > There are two occasions the role is set or changed. On starting the
> > destination the incoming role needs to configured at least for TDX.
> > And then after the migration, the role changes if re-migrated.
> 
> A destination TD needs a directive to not initialize the TD and its vCPUs.
> It comes from its launch parameters (e.g. QEMU cmdline) which selects the
> delayed_init path. That is a construction directive though, and it applies
> only to a destination -- a source needs nothing at init. A session role is
> symmetric and is what the transfer calls consume, so I don't think the two
> need to be the same thing.

Yes the source vs destination role is there from the start for sure. And
changes on re-migration. Could be set in different ways.
 
> > I don't think there are other cases for role change, maybe cancelled
> > migration could require that for some hardware possibly.
> 
> Architecturally, per-session scoping of migration roles should be
> straightforward with any platform: each side asserts a role at the start of every 
> migration session and vendor code will either accept or reject the stated role.

Agreed.
 
> >> Along these lines: it raises a question of whether the MEMORY and VCPU
> >> calls should be coalesced as well into KVM_MIGRATE_MEMORY and KVM_MIGRATE_VCPU.
> >> As posted, direction is implicit for KVM_MIGRATE_CMD but encoded in the ioctl
> >> number for those transfers, so collapsing them would at least make the uAPI
> >> consistent about where direction comes from, and free two ioctls.
> > 
> > Using naming KVM_TRANSFER_MEMORY and KVM_TRANSFER_VCPU might be more
> > descriptive?
> > 
> > Eventually these same commands could be used to save the state to disk
> > for power management use.
> 
> Sure, and the save-to-disk case is a good argument for a more generic name.
> 
> > And going back to the dmaengine like analogy of what is being done..
> > 
> > The transfer direction flags could be KVM_TRANSFER_FROM_GUEST and
> > KVM_TRANSFER_TO_GUEST?
> 
> FROM_GUEST/TO_GUEST still encodes direction per call, which is the open
> question above. If direction is a per-session property, then KVM_TRANSFER_MEMORY
> and KVM_TRANSFER_VCPU are sufficient on their own -- no direction flag, and no
> separate export/import ioctls.
> 
> And if we settle on a per-session property, then SETUP could conceivably state a
> role for a non-migration transfer session as well.
> 
> >   
> >>>> For direction specifically, we could pass it to KVM on every call and let
> >>>> KVM stay stateless about it, or KVM could record it once and remember it for
> >>>> the rest of the session.
> >>>
> >>> Yes the "record and remember" is another option, it could be a sub-command
> >>> something like KVM_MIGRATE_DIRECTION.
> >>
> >>
> >> Agreed on record-and-remember, with a refinement on scope.
> >>
> >> A VM that was migrated in can later be migrated out, so the role is not a
> >> property of the VM -- it has to be recorded per session. SETUP is the call
> >> that starts a migration and runs ahead of all other migration calls, so its
> >> arguments look like the natural place for userspace to state the role; a
> >> separate sub-command would need its own scope rules relative to SETUP.
> >> KVM can then ask the vendor layer whether the requested role is permitted for
> >> this VM -- only it knows the confidential-VM state -- and on success record it
> >> in generic KVM state, where it then selects the export or import callbacks.
> > 
> > The role can change, but it can be VM specific for starting the migration
> > destination even before migration is started.
> > 
> > At least for TDX we need to specify direction for migration destination on
> > init to prevent fully initializing the TD and vCPUs.
> 
> As mentioned above, that is a destination launch time directive that we needn't
> conflate with a migration/transfer session role.

It's still the same role though. Yes we can set it on init, but would
be nice to have some generic way to do it for qemu -incoming.

Just brainstorming.. I wonder if we need two things though. A source vs
destination role. And then at some point possibly later on also a data
transfer direction enumeration similar to what Linux has in
include/linux/dma-direction.h.

We already need to make use of the QEMU return-path for the migration key
exchange. What if some hardware needs to make use of KVM_TRANSFER_MEMORY
from source to destination, and after that back from destination to source
to ack the transfer? Sure this is just speculation, I'm not aware of this
need right now.

In any case with handling the source vs destination role, the enumeration
for data direction can be added to the transfer flags later on as needed.
No need to try to stuff the data direction flag there until really needed.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Kishen Maloor]

On 9/9/26 11:33 PM, Tony Lindgren wrote:
> On Wed, Sep 09, 2026 at 06:11:11PM -0700, Kishen Maloor wrote:
> ...
>>
>> As mentioned above, that is a destination launch time directive that we needn't
>> conflate with a migration/transfer session role.
> 
> It's still the same role though. Yes we can set it on init, but would
> be nice to have some generic way to do it for qemu -incoming.

I'd separate these.

On the role: it seems we agree on recording roles per-session. My only point
then is that a VM created through the delayed_init flow doesn't additionally
need a destination role recorded for it if SETUP will assert one when the
migration is kicked off.

On generic plumbing for -incoming: is this about KVM_TDX_INIT_VM_F_DELAY_INIT?
A generic mechanism would make sense to me if the flag were consumed by generic
KVM code, but that isn't the case here. Userspace has to make a
vendor-specific VM-init call like KVM_TDX_INIT_VM anyway, with the flag passed
on that call. So I'm not sure what a generic version would add, unless you have
something else in mind.

> Just brainstorming.. I wonder if we need two things though. A source vs
> destination role. And then at some point possibly later on also a data
> transfer direction enumeration similar to what Linux has in
> include/linux/dma-direction.h.
> 
> We already need to make use of the QEMU return-path for the migration key
> exchange. What if some hardware needs to make use of KVM_TRANSFER_MEMORY
> from source to destination, and after that back from destination to source
> to ack the transfer? Sure this is just speculation, I'm not aware of this
> need right now.
> 
> In any case with handling the source vs destination role, the enumeration
> for data direction can be added to the transfer flags later on as needed.
> No need to try to stuff the data direction flag there until really needed.

Agree on deferring such an enumeration. More generally though, the session
role determines which operations are permitted, and it seems like vendor code
could be expected to handle those in context.

The TDX architecture already has a destination-to-source example:
TDH.IMPORT.ABORT emits an abort token on the destination that
TDH.EXPORT.ABORT consumes on the source. So KVM on the source would dispatch
to its export-abort path and let the vendor layer decide what to do with
the buffer. This doesn't require a direction flag.

Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren]

On Thu, Sep 10, 2026 at 06:40:04PM -0700, Kishen Maloor wrote:
> On 9/9/26 11:33 PM, Tony Lindgren wrote:
> > On Wed, Sep 09, 2026 at 06:11:11PM -0700, Kishen Maloor wrote:
> > ...
> >>
> >> As mentioned above, that is a destination launch time directive that we needn't
> >> conflate with a migration/transfer session role.
> > 
> > It's still the same role though. Yes we can set it on init, but would
> > be nice to have some generic way to do it for qemu -incoming.
> 
> I'd separate these.
> 
> On the role: it seems we agree on recording roles per-session. My only point
> then is that a VM created through the delayed_init flow doesn't additionally
> need a destination role recorded for it if SETUP will assert one when the
> migration is kicked off.
> 
> On generic plumbing for -incoming: is this about KVM_TDX_INIT_VM_F_DELAY_INIT?
> A generic mechanism would make sense to me if the flag were consumed by generic
> KVM code, but that isn't the case here. Userspace has to make a
> vendor-specific VM-init call like KVM_TDX_INIT_VM anyway, with the flag passed
> on that call. So I'm not sure what a generic version would add, unless you have
> something else in mind.

So we could add a SETUP subcommand SET_ROLE or SET_INCOMING. The
implementation could store the role at least initially. And if we want to
set the role with KVM_TDX_INIT_VM, we could recycle the role bit there.

> > Just brainstorming.. I wonder if we need two things though. A source vs
> > destination role. And then at some point possibly later on also a data
> > transfer direction enumeration similar to what Linux has in
> > include/linux/dma-direction.h.
> > 
> > We already need to make use of the QEMU return-path for the migration key
> > exchange. What if some hardware needs to make use of KVM_TRANSFER_MEMORY
> > from source to destination, and after that back from destination to source
> > to ack the transfer? Sure this is just speculation, I'm not aware of this
> > need right now.
> > 
> > In any case with handling the source vs destination role, the enumeration
> > for data direction can be added to the transfer flags later on as needed.
> > No need to try to stuff the data direction flag there until really needed.
> 
> Agree on deferring such an enumeration. More generally though, the session
> role determines which operations are permitted, and it seems like vendor code
> could be expected to handle those in context.
> 
> The TDX architecture already has a destination-to-source example:
> TDH.IMPORT.ABORT emits an abort token on the destination that
> TDH.EXPORT.ABORT consumes on the source. So KVM on the source would dispatch
> to its export-abort path and let the vendor layer decide what to do with
> the buffer. This doesn't require a direction flag.

Yes agreed with the role we can handle migration related transfers both
ways at least for TDX.