Re: [PATCH V7 mlx5-next 15/15] vfio: Extend the device migration protocol with PRE_COPY

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 7 Feb 2022 19:22:16 +0200
Yishai Hadas <yishaih@nvidia.com> wrote:

> From: Jason Gunthorpe <jgg@nvidia.com>
> 
> The optional PRE_COPY states open the saving data transfer FD before
> reaching STOP_COPY and allows the device to dirty track internal state
> changes with the general idea to reduce the volume of data transferred
> in the STOP_COPY stage.
> 
> While in PRE_COPY the device remains RUNNING, but the saving FD is open.
> 
> Only if the device also supports RUNNING_P2P can it support PRE_COPY_P2P,
> which halts P2P transfers while continuing the saving FD.
> 
> PRE_COPY, with P2P support, requires the driver to implement 7 new arcs
> and exists as an optional FSM branch between RUNNING and STOP_COPY:
>     RUNNING -> PRE_COPY -> PRE_COPY_P2P -> STOP_COPY
> 
> A new ioctl VFIO_DEVICE_MIG_PRECOPY is provided to allow userspace to
> query the progress of the precopy operation in the driver with the idea it
> will judge to move to STOP_COPY at least once the initial data set is
> transferred, and possibly after the dirty size has shrunk appropriately.
> 
> We think there may also be merit in future extensions to the
> VFIO_DEVICE_MIG_PRECOPY ioctl to also command the device to throttle the
> rate it generates internal dirty state.
> 
> Compared to the v1 clarification, STOP_COPY -> PRE_COPY is made optional
> and to be defined in future. While making the whole PRE_COPY feature
> optional eliminates the concern from mlx5, this is still a complicated arc
> to implement and seems prudent to leave it closed until a proper use case
> is developed. We also split the pending_bytes report into the initial and
> sustaining values, and define the protocol to get an event via poll() for
> new dirty data during PRE_COPY.

I feel obligated to ask, is PRE_COPY support essentially RFC at this
point since we have no proposed in-kernel users?

It seems like we're winding down comments on the remainder of the
series and I feel ok with where it's headed and the options we have
available for future extensions.  Pre-copy seems like an important gap
to fill and I think this patch shows that a future extension could
allow it, but with the scrutiny not to add unused code to the kernel,
I'm not sure there's a valid justification to add it now.  Thanks,

Alex

PS - Why is this a stand-alone ioctl rather than a DEVICE_FEATURE?