Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Kishen Maloor <kishen.maloor@intel.com>]

On 9/17/26 10:58 PM, Tony Lindgren wrote:
> On Thu, Sep 17, 2026 at 09:32:23PM -0700, Kishen Maloor wrote:
>> On 9/16/26 11:42 PM, Tony Lindgren wrote:
>>> On Wed, Sep 16, 2026 at 08:31:32PM -0700, Kishen Maloor wrote:
>>>> On 9/15/26 10:09 PM, Tony Lindgren wrote:
>>>>> So trying to summarize the common flags for the role and separate vendor
>>>>> flags:
>>>>>
>>>>> struct kvm_migrate_cmd {
>>>>> 	__u16 command;
>>>>> 	__u16 flags;
>>>>> 	__u16 vflags;
>>>>> 	__u16 reserved;
>>>>> 	__u32 reserved;
>>>>> 	struct kvm_transfer_buffer buf;
>>>>> };
>>>>>
>>>>> Is the above along the lines what you were thinking?
>>>>
>>>> No, I was suggesting a 'role' field carved out of the 'reserved' space,
>>>> like this:
>>>>
>>>> struct kvm_migrate_cmd {
>>>>       __u16 command;
>>>>       __u16 flags;
>>>>       __u8  role;     /* 0 = unset, 1 = source, 2 = destination */
>>>>       __u8  reserved[3];
>>>>       struct kvm_transfer_buffer buf;
>>>> };
>>>
>>> OK yes thanks for clarifying, that works for me.
>>>  
>>>>> Ah OK, yes that would also tell "the hardware has been initialized to a
>>>>> certain migration role". That seems like a usable common feature.
>>>>
>>>> Not quite. It tells us that userspace asserted a role for this VM's migration
>>>> session. Whether a TD was created for import is a separate, vendor-level detail.
>>>> The generic layer only needs the role to reject a session that never stated one,
>>>> and to pick the export or import callback. That callback then knows which side
>>>> it's on and can reject an incorrect role (e.g., if SETUP asserted dst for a src TD).
>>>
>>> That's a good point, the hardware role may not be set yet.
>>>
>>> I'm still wondering if there is a need to stash the userspace set role in
>>> KVM though. Likely only the hardware specific code can properly track the
>>> state of the hardware and adjust to the userspace requests. Seems just
>>> being able to pass the role in struct kvm_migrate_cmd should be enough?
>>
>> Passing it in kvm_migrate_cmd is enough for SETUP itself, but the commands
>> after SETUP like memory/vcpu transfers still have to reach the right
>> callback. So KVM would need to remember what was asserted so that the
>> generic layer can dispatch to the export or import facing callbacks.
>> We've been sketching (on this thread) an alternative UAPI set
>> (3 vs 5 ioctls) for consideration which this stored role enables:
>>
>> Proposed in the RFC              Alternative
>> KVM_MIGRATE_CMD                  KVM_MIGRATE_CMD
>> KVM_EXPORT_MEMORY                 
>> KVM_IMPORT_MEMORY                KVM_TRANSFER_MEMORY       
>> KVM_EXPORT_VCPU
>> KVM_IMPORT_VCPU                  KVM_TRANSFER_VCPU
>>
>> It's just a record (1 byte) of what userspace asserted for the current session 
>> at SETUP. Vendor code still owns the hardware state and remains free to reject a 
>> role that doesn't match it. It is also what lets the generic layer reject a 
>> command to a VM that never set up a session.
> 
> For the TRANSFER style operations, I would assume the direction is passed
> for each transfer, just like the Linux does for the dmaengine. It's
> possible that there may be transfers going both directions without the
> role changing.
> 
> So looks like we have tree things to consider: userspace set migration
> role, the hardware state, and transfer direction.

Two of those three I agree with: userspace passes the role at SETUP, and
the vendor implementation tracks the hardware state. It's the per-transfer
direction I don't think we need.

> What if userspace always passes the role and transfer direction where it
> makes sense? And then the hardware specific implementation tracks the
> hardware state?

Unless there's another meaning, direction would indicate that an operation
must produce or consume a blob into/from a buffer. Such an indication would be
necessary if the layer below the API cannot know what to do with a buffer,
which may well be the case in your dmaengine analogy. However, in this case a
vendor implementation sits below the generic layer and could derive what it
needs to do from the role, command, and any session state it maintains. The
TDH.EXPORT.ABORT example I cited upthread expects a token only once the
session has left its pre-copy phase, so what to do with the buffer follows
from state the vendor layer already holds.

The gap I do see is in how the buffer itself is described: we have no way to
express a command that takes an input and produces an output. A direction flag
doesn't help there either, since it can only say one thing. My comments on patch 2
suggest a new 'capacity' field alongside a reframing of 'size' in kvm_transfer_buffer,
where capacity bounds what the kernel may write and size reports what is actually
there. That covers the both-ways case, and it also leaves nothing for a direction
field to convey. Maybe that addresses your concern?