RE: [PATCH v7 09/10] hisi_acc_vfio_pci: Add support for VFIO live migration

Shameerali Kolothum Thodi
2022-03-03T12:57:41Z

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> Sent: 03 March 2022 00:22
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org; linux-pci@vger.kernel.org;
> alex.williamson@redhat.com; cohuck@redhat.com; mgurtovoy@nvidia.com;
> yishaih@nvidia.com; Linuxarm <linuxarm@huawei.com>; liulongfang
> <liulongfang@huawei.com>; Zengtao (B) <prime.zeng@hisilicon.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>
> Subject: Re: [PATCH v7 09/10] hisi_acc_vfio_pci: Add support for VFIO live
> migration
>
> On Wed, Mar 02, 2022 at 05:29:02PM +0000, Shameer Kolothum wrote:
> > +static long hisi_acc_vf_save_unl_ioctl(struct file *filp,
> > + unsigned int cmd, unsigned long arg)
> > +{
> > + struct hisi_acc_vf_migration_file *migf = filp->private_data;
> > + struct hisi_acc_vf_core_device *hisi_acc_vdev = container_of(migf,
> > + struct hisi_acc_vf_core_device, saving_migf);
> > + loff_t *pos = &filp->f_pos;
> > + struct vfio_precopy_info info;
> > + unsigned long minsz;
> > + int ret;
> > +
> > + if (cmd != VFIO_MIG_GET_PRECOPY_INFO)
> > + return -ENOTTY;
> > +
> > + minsz = offsetofend(struct vfio_precopy_info, dirty_bytes);
> > +
> > + if (copy_from_user(&info, (void __user *)arg, minsz))
> > + return -EFAULT;
> > + if (info.argsz < minsz)
> > + return -EINVAL;
> > +
> > + mutex_lock(&hisi_acc_vdev->state_mutex);
> > + if (hisi_acc_vdev->mig_state != VFIO_DEVICE_STATE_PRE_COPY) {
> > + mutex_unlock(&hisi_acc_vdev->state_mutex);
> > + return -EINVAL;
> > + }
>
> IMHO it is easier just to check the total_length and not grab this
> other lock

The problem with checking the total_length here is that it is possible that
in STOP_COPY the dev is not ready and there are no more data to be transferred
and the total_length remains at QM_MATCH_SIZE.

This just reminded me that the -ENOMSG setting logic in save_read() is
wrong now as it uses only the total_length to determine the PRE_COPY state.

I think either we need to get the curr state info at both places or in STOP_COPY,
if there are no additional data, set the total_length = 0 and handle it in save_read().

Looks like setting the total_length = 0 in STOP_COPY is a better solution(If there are
no other issues with that) as it will avoid grabbing the state_mutex as you
mentioned above.

> > +struct acc_vf_data {
> > +#define QM_MATCH_SIZE 32L
>
> This should be
>
> #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)

Ok.

> > + /* QM match information */
>
> You should probably put an 8 byte random magic number here just to
> make the compatibility more unique.

Ok. Will add one.

> > + u32 qp_num;
> > + u32 dev_id;
> > + u32 que_iso_cfg;
> > + u32 qp_base;
> > + /* QM reserved match information */
> > + u32 qm_rsv_state[4];
> > +
> > + /* QM RW regs */
> > + u32 aeq_int_mask;
> > + u32 eq_int_mask;
> > + u32 ifc_int_source;
> > + u32 ifc_int_mask;
> > + u32 ifc_int_set;
> > + u32 page_size;
> > +
> > + /* QM_EQC_DW has 7 regs */
> > + u32 qm_eqc_dw[7];
> > +
> > + /* QM_AEQC_DW has 7 regs */
> > + u32 qm_aeqc_dw[7];
> > +
> > + /* QM reserved 5 regs */
> > + u32 qm_rsv_regs[5];
> > +
> > + /* qm memory init information */
> > + u64 eqe_dma;
>
> Am I counting wrong or is there a padding before this? 7+7+5 is not a multiple
> of 2. Be explicit about padding in a structure like this.

That's right. It needs padding before 'eqe_dma'.

Thanks,
Shameer