Re: [PATCH V2 13/13] migration: cpr-transfer mode

[Peter Xu <peterx@redhat.com>]

On Thu, Oct 24, 2024 at 05:12:05PM -0400, Steven Sistare wrote:
> On 10/10/2024 5:23 PM, Peter Xu wrote:
> > On Thu, Oct 10, 2024 at 04:06:13PM -0400, Steven Sistare wrote:
> > > vhost requires us to stop the vm early:
> > >    qmp_migrate
> > >      stop vm
> > >      migration_call_notifiers MIG_EVENT_PRECOPY_CPR_SETUP
> > >        vhost_cpr_notifier
> > >          vhost_reset_device - must be after stop vm
> > >                             - and before new qemu inits devices
> > >        cpr_state_save
> > >          unblocks new qemu which inits devices and calls vhost_set_owner
> > > 
> > > Thus config commands must be sent to the target during the guest pause interval :(
> > 
> > I can understand it needs VM stopped, but it can still happen after
> > cpr_save(), am I right (IOW, fd wont change in the notifier)?  I meant
> > below sequence:
> > 
> >    - src: cpr_save(), when running, NONE->SETUP_CPR, all fds synced
> > 
> >    - [whatever happens..]
> > 
> >    - src: finally decide to switchover, vm stop
> > 
> >    - vhost notifier invoked. PS: it doesn't require to be named SETUP_CPR
> >      notifiers here, but something else..
> 
> The problem is that the first step, cpr_save, causes the dest to finish cpr_load_state
> and proceed to initialize devices in qemu_create_late_backends -> net_init_clients.
> This calls ioctl VHOST_SET_OWNER which fails because the device is still owned by src qemu.
> 
> src qemu releases ownership via VHOST_RESET_OWNER in the vhost notifier.

I think the block drives have similar issue before on ownership when disk
is shared on both sides, and that ownership was only passed over to dest
until switchover, rather than dest qemu init.  In the CPR routines it'll be
also during switchover rather than cpr_save().

Maybe it's just harder for vhost, as I assume vhost was never designed to
work with using in shared mode.  Otherwise logically the net_init_clients()
could do the rest initialization, but provide a facility to SET_OWNER at a
later point. I'm not sure if it's possible.

For block it could be easier, IIRC it was mostly about the file lock and
who owns it (e.g. on a NFS share, to make sure no concurrent writters to
corrupt the file).

> 
> Thus the guest must be paused while config commands are sent to the target.
> We could avoid that with any of:
>   * do not issue config commands
>   * precreate phase
>   * cpr-exec mode
>   * only pause if vhost is present.  (eg no pause for vfio).

OK.  I hope precreate will work out if that can solve this too.

Thanks,

-- 
Peter Xu