Re: [RFC PATCH v2 0/4] Add KVM API for confidential guest live migration

[Kishen Maloor <kishen.maloor@intel.com>]

Hi Peter,

Thank you for your comments. Just adding a few other details to
complement Artem's response. 

On 9/17/26 2:27 PM, Peter Xu wrote:
> On Fri, Sep 04, 2026 at 09:24:25PM +0300, Artem Bityutskiy wrote:
>> On Mon, 2026-08-31 at 10:13 +0300, Tony Lindgren wrote:
> ...
>> - Should the same uAPIs also support traditional VMs? But the only use-case
>>   I imagine here is "for testing purposes".
> 
> This is an interesting idea, I think this could be useful.  Especially, I
> wonder if you already have it done and PoC branches you can share, so that
> I can play with it.

I had once attempted this and hit a breaking case while migrating regular VMs.
I didn't dig further since the primary use of this UAPI set is with CoCo VMs.

Here's what I gathered: KVM-mediated transfer needs KVM to resolve a GFN to the
HVA where the data lands, and on x86 QEMU mutates that mapping at runtime in a
way that is opaque to KVM. For the PAM window QEMU overlays a separate region on
top of pc.ram, and KVM sees only the flattened view, the winning memslot, with
no way to address what that slot shadows. On the source, firmware reprograms PAM
and QEMU drops the overlay. On the destination the firmware never runs, so it still
has its boot-time layout and the overlay is still in place. So importing, say, GFN 0xc0
lands on the overlay's backing store, which is a read-only memslot, and the GFN->HVA
translation fails. I suppose even if it were writable, the data would land there
rather than in the pc.ram underneath where it belongs. Without mediation, QEMU is
able to write to the HVA directly, so the problem doesn't arise.

Generally, I think KVM mediation only works if KVM's view of memory is authoritative.
TDX skirts this issue entirely because QEMU doesn't create overlays for TDX VMs.

> ...
> Could you elaborate this ITERATION operation?  Is that something the
> userapp must do after full scan of a round of guest memory?

Essentially, yes. TDX migration architecture delimits such pre-copy rounds as
"migration epochs" and emits an epoch token at each round boundary that needs to be
consumed on the destination. It allows the TDX module to verify that everything from
the prior round has been received at the destination and that two versions of the
same GPA aren't sent in the same epoch. It is userspace that decides where a round
ends, but any deviation from this model and migration would fail.

> 
>>>        |    (repeat until convergence)                  |
>>>  CMD(STOP_AND_COPY/PAUSE)                               |
>>>  CMD(STOP_AND_COPY/TD_STATE) --- VM state ------> CMD(STOP_AND_COPY/TD_STATE)
>>>  KVM_EXPORT_VCPU             --- vCPU state ----> KVM_IMPORT_VCPU
>>>  KVM_EXPORT_MEMORY           -- final memory ---> KVM_IMPORT_MEMORY
> 
> When read/write encrypted memories, two questions:
> 
> - Is there an upper bound of the buffer size per-page?

Yes. For TDX a 4KB guest page produces exactly 4KB of encrypted payload plus a small
fixed amount of ancillary data. The bound can be pre-computed and userspace 
can size its buffers accordingly. The UAPI itself doesn't impose one. The vendor 
implementation decides how pages and ancillary data are laid out.

> 
> - Does this operation supports concurrency?  If it supports, how well it
>   scales per expectation (e.g. is there known big lock for that)?

Yes, these operations can support multifd and kvm_memory_transfer carries the
channel ID. The encryption/decryption work is per-channel, so it scales as multifd
does. The one serialization point is the epoch boundary, i.e. the ITERATION call,
which has to drain in-flight transfers for that round. That's inherent to the epoch 
model rather than a lock that could be dropped.

> ...
> IMHO we should really take postcopy into account when designing the API and
> state machine.  We don't need to implement it in the first version, even
> until merging, but we need to make sure postcopy will be new ioctls on top
> of existing and it should have no major loopholes that it'll need a new set
> of APIs.

Agreed. We honestly haven't looked at post-copy in depth, but supporting it thus
far appears to be additive. One item that comes to mind is a new command in KVM_MIGRATE_CMD
to mark the post-copy switchover point. On the source, vendor code could send all
previously exported and dirty pages (deferring all un-exported pages to post-copy) and
prepare to serve pages on demand. On the destination, vendor code could make the VM
runnable with incomplete memory. This is based on a preliminary assessment though.

> For example, I think we should consider KVM_EXPORT_MEMORY being usable
> after END on source, KVM_IMPORT_MEMORY while TD is in operation, etc.  We

Yes, though I think some of these details could be handled in the vendor
implementation -- e.g., KVM_EXPORT/IMPORT_MEMORY might not need to know that
they're servicing post-copy.

> should likely also need to still picture the rough process of postcopy,
> reserve those APIs since the start (but return -EINVAL or something).

Agreed. We shall attempt to sketch that flow. Your thoughts and feedback would 
be very helpful as we work through this.

> ...
> Could you elaborate what's the relations between TDH.MEM.SCAN.RANGE and the
> GET_DIRTY_LOG ioctl?  I recall above mentioned GET_DIRTY_LOG will be
> available even for CoCo, which makes sense assuming dirty information isn't
> confidential.  However then I don't understand what TDH.MEM.SCAN.RANGE
> plays the role here.

GET_DIRTY_LOG is the UAPI, but the slot dirty bitmaps need to be populated somehow and
TDH.MEM.SCAN.RANGE fills that gap for TDX. It is a SEAMCALL that scans the SEPT upon
request to return the list of currently dirty pages. We wanted to avoid inventing a new
UAPI for this and GET_DIRTY_LOG seemed to be a natural fit. In our current PoC, we've added
a barebones kvm_x86_ops hook that is plugged into kvm_arch_sync_dirty_log(). The TDX
implementation of that hook invokes the scans and records the dirty pages into KVM's slot
dirty bitmaps that GET_DIRTY_LOG serves up to userspace (as usual).