Re: [PATCH v6 09/10] hisi_acc_vfio_pci: Add support for VFIO live migration

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Feb 28, 2022 at 09:01:20AM +0000, Shameer Kolothum wrote:

> +static int hisi_acc_vf_stop_copy(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +				 struct hisi_acc_vf_migration_file *migf)
> +{
> +	struct acc_vf_data *vf_data = &migf->vf_data;

This now needs to hold the migf->lock

> +
> +	if ((cur == VFIO_DEVICE_STATE_STOP || cur == VFIO_DEVICE_STATE_PRE_COPY) &&
> +	    new == VFIO_DEVICE_STATE_RUNNING) {
> +		hisi_acc_vf_start_device(hisi_acc_vdev);

This should be two stanzas STOP->RUNNING should do start_device

And PRE_COPY->RUNNING should do disable_fds, and presumably nothing
else - the device was never stopped.


> +	} else if (cmd == VFIO_DEVICE_MIG_PRECOPY) {
> +		struct vfio_device_mig_precopy precopy;
> +		enum vfio_device_mig_state curr_state;
> +		unsigned long minsz;
> +		int ret;
> +
> +		minsz = offsetofend(struct vfio_device_mig_precopy, dirty_bytes);
> +
> +		if (copy_from_user(&precopy, (void __user *)arg, minsz))
> +			return -EFAULT;
> +		if (precopy.argsz < minsz)
> +			return -EINVAL;
> +
> +		ret = hisi_acc_vfio_pci_get_device_state(core_vdev, &curr_state);
> +		if (!ret && curr_state == VFIO_DEVICE_STATE_PRE_COPY) {
> +			precopy.initial_bytes = QM_MATCH_SIZE;
> +			precopy.dirty_bytes = QM_MATCH_SIZE;

dirty_bytes should be 0

initial_bytes should be calculated based on the current file
descriptor offset.

The use of curr_state should be eliminated

This ioctl should be on the saving file_operations, not here

+ * This ioctl is used on the migration data FD in the precopy phase of the
+ * migration data transfer. It returns an estimate of the current data sizes

I see there is a bug in the qemu version:

@@ -215,12 +218,13 @@ static void vfio_save_precopy_pending(QEMUFile *f, void *>
                                       uint64_t *res_postcopy_only)
 {
     VFIODevice *vbasedev = opaque;
+    VFIOMigration *migration = vbasedev->migration;
     struct vfio_device_mig_precopy precopy = {
         .argsz = sizeof(precopy),
     };
     int ret;
 
-    ret = ioctl(vbasedev->fd, VFIO_DEVICE_MIG_PRECOPY, &precopy);
+    ret = ioctl(migration->data_fd, VFIO_DEVICE_MIG_PRECOPY, &precopy);
     if (ret) {
         return;
     }

I'll update my github.

Jason