Re: [PATCH V2 vfio 02/14] vfio: Extend the device migration protocol with PRE_COPY

Alex Williamson
2022-12-01T22:44:45Z

On Thu, 1 Dec 2022 17:29:19 +0200
Yishai Hadas <yishaih@nvidia.com> wrote:

> From: Jason Gunthorpe <jgg@nvidia.com>
>
> The optional PRE_COPY states open the saving data transfer FD before
> reaching STOP_COPY and allows the device to dirty track internal state
> changes with the general idea to reduce the volume of data transferred
> in the STOP_COPY stage.
>
> While in PRE_COPY the device remains RUNNING, but the saving FD is open.
>
> Only if the device also supports RUNNING_P2P can it support PRE_COPY_P2P,
> which halts P2P transfers while continuing the saving FD.
>
> PRE_COPY, with P2P support, requires the driver to implement 7 new arcs
> and exists as an optional FSM branch between RUNNING and STOP_COPY:
> RUNNING -> PRE_COPY -> PRE_COPY_P2P -> STOP_COPY
>
> A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to
> query the progress of the precopy operation in the driver with the idea it
> will judge to move to STOP_COPY at least once the initial data set is
> transferred, and possibly after the dirty size has shrunk appropriately.
>
> This ioctl is valid only in PRE_COPY states and kernel driver should
> return -EINVAL from any other migration state.
>
> Compared to the v1 clarification, STOP_COPY -> PRE_COPY is blocked
> and to be defined in future.
> We also split the pending_bytes report into the initial and sustaining
> values, e.g.: initial_bytes and dirty_bytes.
> initial_bytes: Amount of initial precopy data.
> dirty_bytes: Device state changes relative to data previously retrieved.
> These fields are not required to have any bearing to STOP_COPY phase.
>
> It is recommended to leave PRE_COPY for STOP_COPY only after the
> initial_bytes field reaches zero. Leaving PRE_COPY earlier might make
> things slower.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Shay Drory <shayd@nvidia.com>
> Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
> ---
> drivers/vfio/vfio_main.c | 74 ++++++++++++++++++++++-
> include/uapi/linux/vfio.h | 122 ++++++++++++++++++++++++++++++++++++--
> 2 files changed, 190 insertions(+), 6 deletions(-)

This looks ok to me, so if you want to provide a branch for the first
patch we can move forward with the rest through the vfio tree as was
mentioned.

Comments and reviews still welcome, particularly I expect Shameer has
already reviewed this for the hisi-acc implementation. Thanks,

Alex