Re: [RFC PATCH v2 0/4] Add KVM API for confidential guest live migration

[Artem Bityutskiy <dedekind1@gmail.com>]

On Mon, 2026-08-31 at 10:13 +0300, Tony Lindgren wrote:
> Tom, since you mentioned that AMD SEV-SNP and Intel TDX live migration
> sound similar, can you please take a look how the API might work for
> SEV-SNP?

The presented example uAPIs were designed to fit the TDX live migration
flow, with the intent that they could also be used by other CoCo VMs.

It would be super great if someone could commend on how suitable these
uAPIs are for AMD and ARM flows.

AFAIU, what is generic in the presented uAPI is more of usage pattern.

- The order in which QEMU calls them.
- The idea that QEMU/KVM is a transport for opaque blobs.

The proposed container - 'struct kvm_transfer_buffer' - only has address
and a size. The vendor defines the layout of the data in it.

Some example high-level topics that would be nice to get feedback on:

- Can we come up with a single set of generic migration uAPIs for different
  CoCo models?
- Or should some uAPIs be generic while others are vendor-specific?
- Or should each CoCo model have its own vendor-specific set of migration
  uAPIs?
- Should the same uAPIs also support traditional VMs? But the only use-case
  I imagine here is "for testing purposes".

> Artem has put together a brief description below of the example API and the
> migration flow:

.. snip ...

> Migration flow
> ==============
> 
>  Source host                                      Destination host
>  ===========                                      ================
> 
>  CMD(SETUP/SESSION)          <--- setup msgs ---> CMD(SETUP/SESSION)
>        |    (repeated)                                  |
>  CMD(SETUP/IMMUTABLE_STATE)  - immutable state -> CMD(SETUP/IMMUTABLE_STATE)
>        |                                                |
>  KVM_GET_DIRTY_LOG                                      |
>  KVM_EXPORT_MEMORY           --- memory data ---> KVM_IMPORT_MEMORY
>  CMD(ITERATION)              --- epoch token ---> CMD(ITERATION)
>        |    (repeat until convergence)                  |
>  CMD(STOP_AND_COPY/PAUSE)                               |
>  CMD(STOP_AND_COPY/TD_STATE) --- VM state ------> CMD(STOP_AND_COPY/TD_STATE)
>  KVM_EXPORT_VCPU             --- vCPU state ----> KVM_IMPORT_VCPU
>  KVM_EXPORT_MEMORY           -- final memory ---> KVM_IMPORT_MEMORY
>  CMD(ITERATION/DONE)         --- start token ---> CMD(ITERATION)
>        |                                                |
>  CMD(END)                                         CMD(END)

... snip ...


As I mentioned, the example uAPI is modeled around the TDX migration flow.
In case it helps the reader, here is a summary of that flow that was
presented in PUCK.

It describes what the TDX module offers today and focuses on pre-copy
migration. This is our interpretation of the TDX specifications, not a
specification itself, and may contain errors or omissions. Please refer to
the official TDX specifications for authoritative information.

Migration Overview
------------------

In TDX, the TDX module implements the migration logic. The VMM drives
migration by issuing seamcalls to the source and destination TDX modules and
transports the resulting encrypted blobs between them. The VMM does not need
to know the contents of those blobs.

The TDX security model enforces two hard rules:

- Only one instance of the TD may run at a time. Either the source or the
  destination may run, but never both. In other words, cloning a TD is not
  allowed.
- When migration completes, the destination must have the same memory and
  vCPU state as the source. It must not end up with a partial or mixed
  state.

Simple Overview
---------------

Run the migration setup session
  |
  v
Transfer immutable TD state
  |
  v
Copy dirty memory pages while the TD runs <-------+
  |                                               |
  +-- iterate until convergence criteria is met --+
  |
  v
Pause the source TD and copy the remaining state
  |
  v
Start the destination TD

The migration process starts with a setup session. During the setup session,
the source and destination TDX modules exchange encrypted blobs and
establish migration encryption keys. For example, these keys protect memory
contents transferred during migration.

Next, the source transfers the immutable TD state to the destination and
initializes the destination TD. This state includes the read-only VM data,
such as its vCPU count and topology.

The source and destination then perform iterative memory-copy rounds. In
each round, the source scans for memory pages that need to be migrated and
exports them in encrypted form. The destination imports the pages. The
rounds continue until the number of pages that change between rounds is
small enough to meet the convergence criteria.

The source TD is then paused. The VMM transfers the remaining TD state, such
as vCPU state, and the final dirty memory pages. Finally, the destination TD
is started and the migration completes.

Notice that the VMM's role in this process is to issue the required
seamcalls and send encrypted blobs between the source and destination. The
VMM does not need to know the contents of those blobs.

More Detailed Overview
----------------------

The following diagram shows the TDX seamcalls used by the source and
destination:

 Source host                                          Destination host
 ===========                                          ================

 TDH.MIG.SETUP        -- crypto keys, attestation ->  TDH.MIG.SETUP
       |                                                    |
 TDH.EXPORT.STATE.IMMUTABLE -- read-only TD state ->
TDH.IMPORT.STATE.IMMUTABLE
       |                                                    |
 TDH.MEM.SCAN.RANGE                                         |
 TDH.MEM.TRACK + IPIs                                       |
 TDH.EXPORT.MEM        ------ memory data --------->  TDH.IMPORT.MEM
 TDH.EXPORT.TRACK      ------ epoch token --------->  TDH.IMPORT.TRACK
       |                                                    |
 TDH.EXPORT.PAUSE                                           |
       |                                                    |
 TDH.EXPORT.STATE.TD   ------ global TD state ----->  TDH.IMPORT.STATE.TD
       |                                                    |
 TDH.EXPORT.STATE.VP   ------ vCPU state ---------->  TDH.IMPORT.STATE.VP
       |                                                    |
 TDH.MEM.SCAN.COMP                                          |
 TDH.MEM.TRACK + IPIs                                       |
 TDH.EXPORT.MEM        ------ final memory data --->  TDH.IMPORT.MEM
 TDH.EXPORT.TRACK      ------ start token --------->  TDH.IMPORT.TRACK
                                                            |
                                                      TDH.IMPORT.END

The source and destination go through the following stages.

Setup
-----

The VMM issues TDH.MIG.SETUP on the source and destination TDX modules
iteratively to perform the setup session. The modules return status and may
also return an encrypted blob, which the VMM passes between the two sides.
In other words, the migration protocol is between the two TDX modules, while
the VMM is simply the transport mechanism. The setup session performs mutual
attestation, establishes trust between the modules, establishes migration
encryption keys, and loads the migration policy. It completes when the TDX
module returns success.

Immutable State Transfer
------------------------

The source calls TDH.EXPORT.STATE.IMMUTABLE to export the immutable TD
state. The destination VMM creates the destination TD skeleton and
configures migration streams, then calls TDH.IMPORT.STATE.IMMUTABLE, which
finalizes the destination TD initialization.

Iterative Memory Copy
---------------------

While the source TD continues running, the source and destination run
iterative memory copy rounds:

- The source calls TDH.MEM.SCAN.RANGE to find migration candidate pages.
- The source calls TDH.MEM.TRACK and sends IPIs to the TD's vCPUs in order
  to ensure the dirty page scanning algorithm correctness.
- The source calls TDH.EXPORT.MEM to export private memory pages, which the
  VMM sends to the destination.
- The destination calls TDH.IMPORT.MEM to import the received memory pages.
- The source calls TDH.EXPORT.TRACK to generate an epoch token. The VMM
  sends it to the destination, which calls TDH.IMPORT.TRACK to consume the
  token. This verifies that all data exported from the source was imported
  on the destination.

The rounds continue until the number of pages that change between rounds is
small enough to meet the convergence criteria.

Stop and Copy
-------------

The VMM pauses the source TD, which begins the downtime period, then calls
TDH.EXPORT.PAUSE to start the TDX-enforced blackout period. The source then
calls TDH.EXPORT.STATE.TD to export mutable TD-scope state and
TDH.EXPORT.STATE.VP to export mutable state for each vCPU. It then performs
the final dirty page scan, runs TDH.MEM.TRACK and sends IPIs, and calls
TDH.EXPORT.MEM to export the final dirty pages as encrypted blobs.

The destination calls TDH.IMPORT.STATE.TD to import mutable TD-scope state
and TDH.IMPORT.STATE.VP to import the mutable state of each vCPU. It calls
TDH.IMPORT.MEM to import the final dirty pages.

After exporting the final dirty pages, the source's last migration seamcall
is TDH.EXPORT.TRACK with IN_ORDER_DONE=1, which generates the start token.
The VMM sends the start token to the destination. The destination calls
TDH.IMPORT.TRACK with this token. This verifies that the mutable TD state
has been imported and allows the destination to start the TD. Finally, the
destination calls TDH.IMPORT.END, which ends the migration.