Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Tony Lindgren <tony.lindgren@linux.intel.com>]

On Sun, Sep 20, 2026 at 05:13:10PM -0700, Kishen Maloor wrote:
> On 9/17/26 10:58 PM, Tony Lindgren wrote:
> > On Thu, Sep 17, 2026 at 09:32:23PM -0700, Kishen Maloor wrote:
> >> On 9/16/26 11:42 PM, Tony Lindgren wrote:
> >>> On Wed, Sep 16, 2026 at 08:31:32PM -0700, Kishen Maloor wrote:
> >>>> On 9/15/26 10:09 PM, Tony Lindgren wrote:
> >>>>> So trying to summarize the common flags for the role and separate vendor
> >>>>> flags:
> >>>>>
> >>>>> struct kvm_migrate_cmd {
> >>>>> 	__u16 command;
> >>>>> 	__u16 flags;
> >>>>> 	__u16 vflags;
> >>>>> 	__u16 reserved;
> >>>>> 	__u32 reserved;
> >>>>> 	struct kvm_transfer_buffer buf;
> >>>>> };
> >>>>>
> >>>>> Is the above along the lines what you were thinking?
> >>>>
> >>>> No, I was suggesting a 'role' field carved out of the 'reserved' space,
> >>>> like this:
> >>>>
> >>>> struct kvm_migrate_cmd {
> >>>>       __u16 command;
> >>>>       __u16 flags;
> >>>>       __u8  role;     /* 0 = unset, 1 = source, 2 = destination */
> >>>>       __u8  reserved[3];
> >>>>       struct kvm_transfer_buffer buf;
> >>>> };
> >>>
> >>> OK yes thanks for clarifying, that works for me.
> >>>  
> >>>>> Ah OK, yes that would also tell "the hardware has been initialized to a
> >>>>> certain migration role". That seems like a usable common feature.
> >>>>
> >>>> Not quite. It tells us that userspace asserted a role for this VM's migration
> >>>> session. Whether a TD was created for import is a separate, vendor-level detail.
> >>>> The generic layer only needs the role to reject a session that never stated one,
> >>>> and to pick the export or import callback. That callback then knows which side
> >>>> it's on and can reject an incorrect role (e.g., if SETUP asserted dst for a src TD).
> >>>
> >>> That's a good point, the hardware role may not be set yet.
> >>>
> >>> I'm still wondering if there is a need to stash the userspace set role in
> >>> KVM though. Likely only the hardware specific code can properly track the
> >>> state of the hardware and adjust to the userspace requests. Seems just
> >>> being able to pass the role in struct kvm_migrate_cmd should be enough?
> >>
> >> Passing it in kvm_migrate_cmd is enough for SETUP itself, but the commands
> >> after SETUP like memory/vcpu transfers still have to reach the right
> >> callback. So KVM would need to remember what was asserted so that the
> >> generic layer can dispatch to the export or import facing callbacks.
> >> We've been sketching (on this thread) an alternative UAPI set
> >> (3 vs 5 ioctls) for consideration which this stored role enables:
> >>
> >> Proposed in the RFC              Alternative
> >> KVM_MIGRATE_CMD                  KVM_MIGRATE_CMD
> >> KVM_EXPORT_MEMORY                 
> >> KVM_IMPORT_MEMORY                KVM_TRANSFER_MEMORY       
> >> KVM_EXPORT_VCPU
> >> KVM_IMPORT_VCPU                  KVM_TRANSFER_VCPU
> >>
> >> It's just a record (1 byte) of what userspace asserted for the current session 
> >> at SETUP. Vendor code still owns the hardware state and remains free to reject a 
> >> role that doesn't match it. It is also what lets the generic layer reject a 
> >> command to a VM that never set up a session.
> > 
> > For the TRANSFER style operations, I would assume the direction is passed
> > for each transfer, just like the Linux does for the dmaengine. It's
> > possible that there may be transfers going both directions without the
> > role changing.
> > 
> > So looks like we have tree things to consider: userspace set migration
> > role, the hardware state, and transfer direction.
> 
> Two of those three I agree with: userspace passes the role at SETUP, and
> the vendor implementation tracks the hardware state. It's the per-transfer
> direction I don't think we need.

Ack on the userspace passing the role at SETUP and vendor implementation
tracking the hardware state.

Then for KVM tracking the role, I don't think we need it with the two
above. The role tracking can always be added if really needed. Any other
opinions on this one?

The transfer direction is there with the EXPORT/IMPORT naming. Maybe
just let's keep that naming for easier readability rather than try to 
switch to TRANSFER style naming. No transfer direction flag needed.

> > What if userspace always passes the role and transfer direction where it
> > makes sense? And then the hardware specific implementation tracks the
> > hardware state?
> 
> Unless there's another meaning, direction would indicate that an operation
> must produce or consume a blob into/from a buffer. Such an indication would be
> necessary if the layer below the API cannot know what to do with a buffer,
> which may well be the case in your dmaengine analogy. However, in this case a
> vendor implementation sits below the generic layer and could derive what it
> needs to do from the role, command, and any session state it maintains. The
> TDH.EXPORT.ABORT example I cited upthread expects a token only once the
> session has left its pre-copy phase, so what to do with the buffer follows
> from state the vendor layer already holds.

Yeah I don't think the dmaengine API ever expects to get back a blob as a
result of an outgoing transfer.. That would be a separate DMA transfer. 
 
> The gap I do see is in how the buffer itself is described: we have no way to
> express a command that takes an input and produces an output. A direction flag
> doesn't help there either, since it can only say one thing. My comments on patch 2
> suggest a new 'capacity' field alongside a reframing of 'size' in kvm_transfer_buffer,
> where capacity bounds what the kernel may write and size reports what is actually
> there. That covers the both-ways case, and it also leaves nothing for a direction
> field to convey. Maybe that addresses your concern?

Yes that's a good point, a transfer command may also return data in the
transfer buffer and the size of returned data needs to be known. Replied
to your patch #2 comments with some ideas on it.