Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren <tony.lindgren@linux.intel.com>]

On Wed, Sep 09, 2026 at 06:11:11PM -0700, Kishen Maloor wrote:
> On 9/8/26 11:57 PM, Tony Lindgren wrote:
> > On Tue, Sep 08, 2026 at 05:22:30PM -0700, Kishen Maloor wrote:
> >> On 9/7/26 9:43 PM, Tony Lindgren wrote:
> >>> On Mon, Sep 07, 2026 at 04:32:33PM +0300, Artem Bityutskiy wrote:
> >>>> On Mon, 2026-09-07 at 15:15 +0200, Jörg Rödel wrote:
> >>>>> The direction is always the same over a single live migration session, right?
> >>>>> So it could be a setup flag, on the other hand having separate KVM_EXPORT_CMD
> >>>>> and KVM_IMPORT_CMD seems to be a cleaner ABI.
> >>>
> >>> OK
> >>
> >>
> >> Just sharing an alternate point of view:
> >>
> >> The roles are fixed over a migration session. A split ABI is certainly more 
> >> self-describing, but it restates that invariant on every call, and therefore 
> >> also permits it to be contradicted -- a failure mode that does not otherwise
> >> exist. With a single KVM_MIGRATE_CMD and a per-session role recorded once (more 
> >> on that below), there is no need for per-call policing: the role could be 
> >> checked once when the session is established.
> > 
> > There are two occasions the role is set or changed. On starting the
> > destination the incoming role needs to configured at least for TDX.
> > And then after the migration, the role changes if re-migrated.
> 
> A destination TD needs a directive to not initialize the TD and its vCPUs.
> It comes from its launch parameters (e.g. QEMU cmdline) which selects the
> delayed_init path. That is a construction directive though, and it applies
> only to a destination -- a source needs nothing at init. A session role is
> symmetric and is what the transfer calls consume, so I don't think the two
> need to be the same thing.

Yes the source vs destination role is there from the start for sure. And
changes on re-migration. Could be set in different ways.
 
> > I don't think there are other cases for role change, maybe cancelled
> > migration could require that for some hardware possibly.
> 
> Architecturally, per-session scoping of migration roles should be
> straightforward with any platform: each side asserts a role at the start of every 
> migration session and vendor code will either accept or reject the stated role.

Agreed.
 
> >> Along these lines: it raises a question of whether the MEMORY and VCPU
> >> calls should be coalesced as well into KVM_MIGRATE_MEMORY and KVM_MIGRATE_VCPU.
> >> As posted, direction is implicit for KVM_MIGRATE_CMD but encoded in the ioctl
> >> number for those transfers, so collapsing them would at least make the uAPI
> >> consistent about where direction comes from, and free two ioctls.
> > 
> > Using naming KVM_TRANSFER_MEMORY and KVM_TRANSFER_VCPU might be more
> > descriptive?
> > 
> > Eventually these same commands could be used to save the state to disk
> > for power management use.
> 
> Sure, and the save-to-disk case is a good argument for a more generic name.
> 
> > And going back to the dmaengine like analogy of what is being done..
> > 
> > The transfer direction flags could be KVM_TRANSFER_FROM_GUEST and
> > KVM_TRANSFER_TO_GUEST?
> 
> FROM_GUEST/TO_GUEST still encodes direction per call, which is the open
> question above. If direction is a per-session property, then KVM_TRANSFER_MEMORY
> and KVM_TRANSFER_VCPU are sufficient on their own -- no direction flag, and no
> separate export/import ioctls.
> 
> And if we settle on a per-session property, then SETUP could conceivably state a
> role for a non-migration transfer session as well.
> 
> >   
> >>>> For direction specifically, we could pass it to KVM on every call and let
> >>>> KVM stay stateless about it, or KVM could record it once and remember it for
> >>>> the rest of the session.
> >>>
> >>> Yes the "record and remember" is another option, it could be a sub-command
> >>> something like KVM_MIGRATE_DIRECTION.
> >>
> >>
> >> Agreed on record-and-remember, with a refinement on scope.
> >>
> >> A VM that was migrated in can later be migrated out, so the role is not a
> >> property of the VM -- it has to be recorded per session. SETUP is the call
> >> that starts a migration and runs ahead of all other migration calls, so its
> >> arguments look like the natural place for userspace to state the role; a
> >> separate sub-command would need its own scope rules relative to SETUP.
> >> KVM can then ask the vendor layer whether the requested role is permitted for
> >> this VM -- only it knows the confidential-VM state -- and on success record it
> >> in generic KVM state, where it then selects the export or import callbacks.
> > 
> > The role can change, but it can be VM specific for starting the migration
> > destination even before migration is started.
> > 
> > At least for TDX we need to specify direction for migration destination on
> > init to prevent fully initializing the TD and vCPUs.
> 
> As mentioned above, that is a destination launch time directive that we needn't
> conflate with a migration/transfer session role.

It's still the same role though. Yes we can set it on init, but would
be nice to have some generic way to do it for qemu -incoming.

Just brainstorming.. I wonder if we need two things though. A source vs
destination role. And then at some point possibly later on also a data
transfer direction enumeration similar to what Linux has in
include/linux/dma-direction.h.

We already need to make use of the QEMU return-path for the migration key
exchange. What if some hardware needs to make use of KVM_TRANSFER_MEMORY
from source to destination, and after that back from destination to source
to ack the transfer? Sure this is just speculation, I'm not aware of this
need right now.

In any case with handling the source vs destination role, the enumeration
for data direction can be added to the transfer flags later on as needed.
No need to try to stuff the data direction flag there until really needed.