Re: [RFC PATCH v2 0/4] Add KVM API for confidential guest live migration

[Kishen Maloor <kishen.maloor@intel.com>]

Hi Peter,

Thanks! It really helps to go over details.

On 9/23/26 2:36 PM, Peter Xu wrote:
> ...
> 
> I see this one slightly differently, and I have a major question on the
> choice of GPA for KVM's memory access API.
> 
> If reusing GET_DIRTY_LOG, it means slot_id works for CoCo like before
> because that's the old interface there.

It works because we map backwards from GPAs in TDX's dirty scan results to
a KVM memslot on the source, and a bit in its dirty bitmap, in keeping with
the GET_DIRTY_LOG contract.

> But the new KVM_EXPORT_MEM (vice versa) used GPA arrays.  Could I ask why
> the change?

The migration ABI (at least in TDX, possibly others) is GPA shaped. Also,
everything downstream like SEPT entries key off GPAs. The EXPORT and IMPORT
ABIs take GPAs alongside the encrypted blob, so both the source and the
destination need it at the call site that implements the UAPI. Passing GFNs
in the UAPI is merely a convenience in that respect. If we passed in
slot+offset instead, it would just be an indirection that needs to resolve
back to the GPA anyway to issue the vendor migration call. But this is not
the blocking issue for the PAM case, as I'll explain below.

> IIUC this is the fundamental reason why you hit that PAM issue: at a
> specific GPA, QEMU/KVM can map different things, hence GPA is not yet an
> unified identifier for a physical page that guest uses.  However, (slot_id,
> slot_offset) will be.
> 
> If the migration memory access API will be using that instead of GPA, I
> think problems will be gone because then the PAM regions (likely, pc.ram
> underneath) will be migrated in form of KVM memslots, destination apply
> that to the PAM's (slot_id, slot_offset), then when switchover, that PAM
> memory region will be enabled by QEMU and it will be visible to destination
> QEMU when VM starts on destination.

I think using slot+offset won't make the problem vanish, because the PAM window
in the underlying pc.ram region is not addressable by KVM on the destination at
that time. KVM's memslots come from QEMU's FlatView at any moment and QEMU
reconfigures them on the fly. In the destination's boot-time setup, it seems
like pc.ram's memslot is split around that window, and what covers the
range instead is a separate, read-only memslot for the overlay, which is an
allocation with its own backing, not the pc.ram bytes underneath.
And since only the flattened view is ever registered, there's no slot+offset
that refers to those bytes either. Only when the source's PAM configuration
lands at the destination, which happens after memory migration concludes, do
those pc.ram bytes become visible to KVM.

In the TDX case, QEMU does not create that overlay at the start, so the
destination has a memslot corresponding to the pc.ram region that stays put.
KVM is able to do a GFN->PFN translation to hand to TDX's IMPORT ABI.

So, it appears that a UAPI change will not address this corner case.

> I'm actually not sure if such (e.g. PAM) will ever be supported at all in
> CoCo context, but it seems to me using (slot_id, slot_offset) (for each
> entry; it can also be an array) is more flexible than GPA arrays in that it
> allows GPA mapping to change on the fly if needed even during migration.
> 
> I'm not sure if that's explicitly forbidden for CoCo, though.

A TD private page lives at a fixed GPA in the SEPT, and I believe the
EXPORT/IMPORT bundle binds the GPA into the page's integrity check, so a page
has to be imported at the GPA it was exported from.

>>> Could you elaborate this ITERATION operation?  Is that something the
>>> userapp must do after full scan of a round of guest memory?
>>
>> Essentially, yes. TDX migration architecture delimits such pre-copy rounds as
>> "migration epochs" and emits an epoch token at each round boundary that needs to be
>> consumed on the destination. It allows the TDX module to verify that everything from
>> the prior round has been received at the destination and that two versions of the
>> same GPA aren't sent in the same epoch. It is userspace that decides where a round
>> ends, but any deviation from this model and migration would fail.
> 
> I'm a bit surprised that TDX will also monitor how many times the same GPA
> is updated per iteration.  What if below happens:

Yes, and maybe that is because it won't know the ordering if the same
GPA were caught at different times in the same epoch and fed to different
migration streams (which one is newer?)
On regular VMs, I believe QEMU doesn't migrate the same GPA twice in one round.
As I understand it, a migrated GPA is revisited only in the next round.
I suppose TDX just makes that behavior architectural.

>   ...
>   ITERATION sync n
>   GET_DIRTY_LOG, see page P dirty
>   migrate page P
>   GET_DIRTY_LOG, see page P dirty again
>   migrate page P again   <--------------------- [a]

QEMU shouldn't transfer P again in the same round, right?

>   ITERATION sync n+1
>   ...
> 
> Would above crash on destination TDX at step [a] applying P 2nd time?  Or

If P were migrated again, then I believe it would fail to import on the
destination, and also abort the import session, so migration would fail.
TDX records the current epoch against a GPA on every import, and I think
that's how it catches a 2nd import attempt.

> does it mean GET_DIRTY_LOG can only be invoked once per iteration?

We don't touch QEMU's stock dirty logging/transfer flows so however it
handles it currently stays.