Re: [PATCH V2 13/13] migration: cpr-transfer mode

[Steven Sistare <steven.sistare@oracle.com>]

On 10/25/2024 9:55 AM, Peter Xu wrote:
> On Thu, Oct 24, 2024 at 05:12:05PM -0400, Steven Sistare wrote:
>> On 10/10/2024 5:23 PM, Peter Xu wrote:
>>> On Thu, Oct 10, 2024 at 04:06:13PM -0400, Steven Sistare wrote:
>>>> vhost requires us to stop the vm early:
>>>>     qmp_migrate
>>>>       stop vm
>>>>       migration_call_notifiers MIG_EVENT_PRECOPY_CPR_SETUP
>>>>         vhost_cpr_notifier
>>>>           vhost_reset_device - must be after stop vm
>>>>                              - and before new qemu inits devices
>>>>         cpr_state_save
>>>>           unblocks new qemu which inits devices and calls vhost_set_owner
>>>>
>>>> Thus config commands must be sent to the target during the guest pause interval :(
>>>
>>> I can understand it needs VM stopped, but it can still happen after
>>> cpr_save(), am I right (IOW, fd wont change in the notifier)?  I meant
>>> below sequence:
>>>
>>>     - src: cpr_save(), when running, NONE->SETUP_CPR, all fds synced
>>>
>>>     - [whatever happens..]
>>>
>>>     - src: finally decide to switchover, vm stop
>>>
>>>     - vhost notifier invoked. PS: it doesn't require to be named SETUP_CPR
>>>       notifiers here, but something else..
>>
>> The problem is that the first step, cpr_save, causes the dest to finish cpr_load_state
>> and proceed to initialize devices in qemu_create_late_backends -> net_init_clients.
>> This calls ioctl VHOST_SET_OWNER which fails because the device is still owned by src qemu.
>>
>> src qemu releases ownership via VHOST_RESET_OWNER in the vhost notifier.
> 
> I think the block drives have similar issue before on ownership when disk
> is shared on both sides, and that ownership was only passed over to dest
> until switchover, rather than dest qemu init.  In the CPR routines it'll be
> also during switchover rather than cpr_save().
> 
> Maybe it's just harder for vhost, as I assume vhost was never designed to
> work with using in shared mode.  Otherwise logically the net_init_clients()
> could do the rest initialization, but provide a facility to SET_OWNER at a
> later point. I'm not sure if it's possible.

net_init_clients cannot do any initialization that issues vhost ioctls,
because the dest process does not yet own the vhost device.

- Steve

> For block it could be easier, IIRC it was mostly about the file lock and
> who owns it (e.g. on a NFS share, to make sure no concurrent writters to
> corrupt the file).
> 
>>
>> Thus the guest must be paused while config commands are sent to the target.
>> We could avoid that with any of:
>>    * do not issue config commands
>>    * precreate phase
>>    * cpr-exec mode
>>    * only pause if vhost is present.  (eg no pause for vfio).
> 
> OK.  I hope precreate will work out if that can solve this too.
> 
> Thanks,
>