RE: [PATCH v3 6/6] hisi_acc_vfio_pci: Add support for VFIO live migration

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> Sent: 15 September 2021 14:08
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-crypto@vger.kernel.org; alex.williamson@redhat.com;
> mgurtovoy@nvidia.com; Linuxarm <linuxarm@huawei.com>; liulongfang
> <liulongfang@huawei.com>; Zengtao (B) <prime.zeng@hisilicon.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>
> Subject: Re: [PATCH v3 6/6] hisi_acc_vfio_pci: Add support for VFIO live
> migration
> 
> On Wed, Sep 15, 2021 at 10:50:37AM +0100, Shameer Kolothum wrote:
> > +/*
> > + * HiSilicon ACC VF dev MMIO space contains both the functional register
> > + * space and the migration control register space. We hide the migration
> > + * control space from the Guest. But to successfully complete the live
> > + * migration, we still need access to the functional MMIO space assigned
> > + * to the Guest. To avoid any potential security issues, we need to be
> > + * careful not to access this region while the Guest vCPUs are running.
> > + *
> > + * Hence check the device state before we map the region.
> > + */
> 
> The prior patch prevents mapping this area into the guest at all,
> right?

That’s right. It will prevent Guest from mapping this area.

> So why the comment and logic? If the MMIO area isn't mapped then there
> is nothing to do, right?
> 
> The only risk is P2P transactions from devices in the same IOMMU
> group, and you might do well to mitigate that by asserting that the
> device is in a singleton IOMMU group?

This was added as an extra protection. I will add the singleton check instead.

> > +static int hisi_acc_vfio_pci_init(struct vfio_pci_core_device *vdev)
> > +{
> > +	struct acc_vf_migration *acc_vf_dev;
> > +	struct pci_dev *pdev = vdev->pdev;
> > +	struct pci_dev *pf_dev, *vf_dev;
> > +	struct hisi_qm *pf_qm;
> > +	int vf_id, ret;
> > +
> > +	pf_dev = pdev->physfn;
> > +	vf_dev = pdev;
> > +
> > +	pf_qm = pci_get_drvdata(pf_dev);
> > +	if (!pf_qm) {
> > +		pr_err("HiSi ACC qm driver not loaded\n");
> > +		return -EINVAL;
> > +	}
> 
> Nope, this is locked wrong and has no lifetime management.

Ok. Holding the device_lock() sufficient here?

> 
> > +	if (pf_qm->ver < QM_HW_V3) {
> > +		dev_err(&pdev->dev,
> > +			"Migration not supported, hw version: 0x%x\n",
> > +			 pf_qm->ver);
> > +		return -ENODEV;
> > +	}
> > +
> > +	vf_id = PCI_FUNC(vf_dev->devfn);
> > +	acc_vf_dev = kzalloc(sizeof(*acc_vf_dev), GFP_KERNEL);
> > +	if (!acc_vf_dev)
> > +		return -ENOMEM;
> 
> Don't do the memory like this, the entire driver should have a global
> struct, not one that is allocated/freed around open/close_device
> 
> struct hisi_acc_vfio_device {
>       struct vfio_pci_core_device core_device;
>       [put acc_vf_migration here]
>       [put required state from mig_ctl here, don't allocate again]
>       struct acc_vf_data mig_data; // Don't use wonky pointer maths
> }
> 
> Then leave the releae function on the reg ops NULL and consistently
> pass the hisi_acc_vfio_device everywhere instead of
> acc_vf_migration. This way all the functions get all the needed
> information, eg if they want to log or something.
> 
> The mlx5 driver that should be posted soon will show how to structure
> most of this well and include several more patches you'll want to be
> using here.

Ok. Thanks for taking a look. I will take a closer look at the mlx5 driver and
rework based on it.

Thanks,
Shameer