Re: [RFC PATCH v2 2/4] KVM: x86: Add optional KVM_CAP_LIVE_MIGRATION and KVM_MIGRATE_CMD

[Kishen Maloor <kishen.maloor@intel.com>]

On 9/20/26 11:52 PM, Tony Lindgren wrote:
> On Sun, Sep 20, 2026 at 05:13:10PM -0700, Kishen Maloor wrote:
>> On 9/17/26 10:58 PM, Tony Lindgren wrote:
>>> On Thu, Sep 17, 2026 at 09:32:23PM -0700, Kishen Maloor wrote:
>>>> On 9/16/26 11:42 PM, Tony Lindgren wrote:
>>>>> On Wed, Sep 16, 2026 at 08:31:32PM -0700, Kishen Maloor wrote:
>>>>>> On 9/15/26 10:09 PM, Tony Lindgren wrote:
>>>>>>> So trying to summarize the common flags for the role and separate vendor
>>>>>>> flags:
>>>>>>>
>>>>>>> struct kvm_migrate_cmd {
>>>>>>> 	__u16 command;
>>>>>>> 	__u16 flags;
>>>>>>> 	__u16 vflags;
>>>>>>> 	__u16 reserved;
>>>>>>> 	__u32 reserved;
>>>>>>> 	struct kvm_transfer_buffer buf;
>>>>>>> };
>>>>>>>
>>>>>>> Is the above along the lines what you were thinking?
>>>>>>
>>>>>> No, I was suggesting a 'role' field carved out of the 'reserved' space,
>>>>>> like this:
>>>>>>
>>>>>> struct kvm_migrate_cmd {
>>>>>>       __u16 command;
>>>>>>       __u16 flags;
>>>>>>       __u8  role;     /* 0 = unset, 1 = source, 2 = destination */
>>>>>>       __u8  reserved[3];
>>>>>>       struct kvm_transfer_buffer buf;
>>>>>> };
>>>>>
>>>>> OK yes thanks for clarifying, that works for me.
>>>>>  
>>>>>>> Ah OK, yes that would also tell "the hardware has been initialized to a
>>>>>>> certain migration role". That seems like a usable common feature.
>>>>>>
>>>>>> Not quite. It tells us that userspace asserted a role for this VM's migration
>>>>>> session. Whether a TD was created for import is a separate, vendor-level detail.
>>>>>> The generic layer only needs the role to reject a session that never stated one,
>>>>>> and to pick the export or import callback. That callback then knows which side
>>>>>> it's on and can reject an incorrect role (e.g., if SETUP asserted dst for a src TD).
>>>>>
>>>>> That's a good point, the hardware role may not be set yet.
>>>>>
>>>>> I'm still wondering if there is a need to stash the userspace set role in
>>>>> KVM though. Likely only the hardware specific code can properly track the
>>>>> state of the hardware and adjust to the userspace requests. Seems just
>>>>> being able to pass the role in struct kvm_migrate_cmd should be enough?
>>>>
>>>> Passing it in kvm_migrate_cmd is enough for SETUP itself, but the commands
>>>> after SETUP like memory/vcpu transfers still have to reach the right
>>>> callback. So KVM would need to remember what was asserted so that the
>>>> generic layer can dispatch to the export or import facing callbacks.
>>>> We've been sketching (on this thread) an alternative UAPI set
>>>> (3 vs 5 ioctls) for consideration which this stored role enables:
>>>>
>>>> Proposed in the RFC              Alternative
>>>> KVM_MIGRATE_CMD                  KVM_MIGRATE_CMD
>>>> KVM_EXPORT_MEMORY                 
>>>> KVM_IMPORT_MEMORY                KVM_TRANSFER_MEMORY       
>>>> KVM_EXPORT_VCPU
>>>> KVM_IMPORT_VCPU                  KVM_TRANSFER_VCPU
>>>>
>>>> It's just a record (1 byte) of what userspace asserted for the current session 
>>>> at SETUP. Vendor code still owns the hardware state and remains free to reject a 
>>>> role that doesn't match it. It is also what lets the generic layer reject a 
>>>> command to a VM that never set up a session.
>>>
>>> For the TRANSFER style operations, I would assume the direction is passed
>>> for each transfer, just like the Linux does for the dmaengine. It's
>>> possible that there may be transfers going both directions without the
>>> role changing.
>>>
>>> So looks like we have tree things to consider: userspace set migration
>>> role, the hardware state, and transfer direction.
>>
>> Two of those three I agree with: userspace passes the role at SETUP, and
>> the vendor implementation tracks the hardware state. It's the per-transfer
>> direction I don't think we need.
> 
> Ack on the userspace passing the role at SETUP and vendor implementation
> tracking the hardware state.
> 
> Then for KVM tracking the role, I don't think we need it with the two
> above. The role tracking can always be added if really needed. Any other
> opinions on this one?

I do think it's useful for generic KVM to track this role (1 byte).
- It lets _TRANSFER_ style calls dispatch directly to import or export 
  callbacks based on the role.
- Even if we don't adopt the _TRANSFER_ style, it enables generic KVM to
  reject mismatched calls, e.g., KVM_EXPORT_MEMORY on a destination.

> 
> The transfer direction is there with the EXPORT/IMPORT naming. Maybe
> just let's keep that naming for easier readability rather than try to 
> switch to TRANSFER style naming. No transfer direction flag needed.

I can't say I have a clear preference between EXPORT/IMPORT vs _TRANSFER_.
If we keep the EXPORT/IMPORT naming, then consistency would arguably call for
splitting MIGRATE_CMD too, which makes it 6 vs 3 (or 5 vs 3 against the RFC
as posted):

EXPORT/IMPORT style              _TRANSFER_ style
KVM_EXPORT_CMD                   KVM_MIGRATE_CMD
KVM_IMPORT_CMD
KVM_EXPORT_MEMORY                KVM_TRANSFER_MEMORY
KVM_IMPORT_MEMORY
KVM_EXPORT_VCPU                  KVM_TRANSFER_VCPU
KVM_IMPORT_VCPU            

I guess the main benefit of the _TRANSFER_ style is reduced duplication. 
Each EXPORT/IMPORT pair takes the same struct and differs only by the role
that the session already knows. Merging them gives one entry point per call
type and lets userspace drive both ends from the same call site which could
be considered a win. It doesn't reduce kernel code though as the top-level
handler still branches internally on the role.