Re: [PATCH vfio 0/9] Add chunk mode support for mlx5 driver

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Sep 11, 2023 at 12:38:47PM +0300, Yishai Hadas wrote:
> This series adds 'chunk mode' support for mlx5 driver upon the migration
> flow.
> 
> Before this series, we were limited to 4GB state size, as of the 4 bytes
> max value based on the device specification for the query/save/load
> commands.
> 
> Once the device supports 'chunk mode' the driver can support state size
> which is larger than 4GB.
> 
> In that case, the device has the capability to split a single image to
> multiple chunks as long as the software provides a buffer in the minimum
> size reported by the device.
> 
> The driver should query for the minimum buffer size required using
> QUERY_VHCA_MIGRATION_STATE command with the 'chunk' bit set in its
> input, in that case, the output will include both the minimum buffer
> size and also the remaining total size to be reported/used where it will
> be applicable.
> 
> Upon chunk mode, there may be multiple images that will be read from the
> device upon STOP_COPY. The driver will read ahead from the firmware the
> full state in small/optimized chunks while letting QEMU/user space read
> in parallel the available data.
> 
> The chunk buffer size is picked up based on the minimum size that
> firmware requires, the total full size and some max value in the driver
> code which was set to 8MB to achieve some optimized downtime in the
> general case.
> 
> With that series in place, we could migrate successfully a device state
> with a larger size than 4GB, while even improving the downtime in some
> scenarios.
> 
> Note:
> As the first patch should go to net/mlx5 we may need to send it as a
> pull request format to VFIO to avoid conflicts before acceptance.
> 
> Yishai
> 
> Yishai Hadas (9):
>   net/mlx5: Introduce ifc bits for migration in a chunk mode
>   vfio/mlx5: Wake up the reader post of disabling the SAVING migration
>     file
>   vfio/mlx5: Refactor the SAVE callback to activate a work only upon an
>     error
>   vfio/mlx5: Enable querying state size which is > 4GB
>   vfio/mlx5: Rename some stuff to match chunk mode
>   vfio/mlx5: Pre-allocate chunks for the STOP_COPY phase
>   vfio/mlx5: Add support for SAVING in chunk mode
>   vfio/mlx5: Add support for READING in chunk mode
>   vfio/mlx5: Activate the chunk mode functionality

I didn't check in great depth but this looks OK to me

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

I think this is a good design to start motivating more qmeu
improvements, eg using io_uring as we could go further in the driver
to optimize with that kind of support.

Jason