[PATCH v5 0/3] update live migration configuration region

[PATCH v5 0/3] update live migration configuration region

[Longfang Liu]

On the new hardware platform, the configuration register space
of the live migration function is set on the PF, while on the
old platform, this part is placed on the VF.

Change v4 -> v5
	Remove BAR length alignment

Change v3 -> v4
	Rebase on kernel 6.15

Change v2 -> v3
	Put the changes of Pre_Copy into another bugfix patchset.

Change v1 -> v2
	Delete the vf_qm_state read operation in Pre_Copy

Longfang Liu (3):
  migration: update BAR space size
  migration: qm updates BAR configuration
  migration: adapt to new migration configuration

 drivers/crypto/hisilicon/qm.c                 |  29 +++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 202 ++++++++++++------
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |   7 +
 3 files changed, 176 insertions(+), 62 deletions(-)

-- 
2.24.0

[PATCH v5 1/3] migration: update BAR space size

[Longfang Liu]

On new platforms greater than QM_HW_V3, the live migration configuration
region is moved from VF to PF. The VF's own configuration space is
restored to the complete 64KB, and there is no need to divide the
size of the BAR configuration space equally.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 36 ++++++++++++++-----
 1 file changed, 27 insertions(+), 9 deletions(-)

diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 2149f49aeec7..1ddc9dbadb70 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -1250,6 +1250,28 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct pci_dev *pdev)
 	return !IS_ERR(pf_qm) ? pf_qm : NULL;
 }
 
+static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,
+					unsigned int index)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev =
+			hisi_acc_drvdata(vdev->pdev);
+
+	/*
+	 * On the old QM_HW_V3 device, the ACC VF device BAR2
+	 * region encompasses both functional register space
+	 * and migration control register space.
+	 * only the functional region should be report to Guest.
+	 *
+	 * On the new HW device, the migration control register
+	 * has been moved to the PF device BAR2 region.
+	 * The VF device BAR2 is entirely functional register space.
+	 */
+	if (hisi_acc_vdev->pf_qm->ver == QM_HW_V3)
+		return (pci_resource_len(vdev->pdev, index) >> 1);
+
+	return pci_resource_len(vdev->pdev, index);
+}
+
 static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
 					size_t count, loff_t *ppos,
 					size_t *new_count)
@@ -1260,8 +1282,9 @@ static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
 
 	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
 		loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
-		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
+		resource_size_t end;
 
+		end = hisi_acc_get_resource_len(vdev, index);
 		/* Check if access is for migration control region */
 		if (pos >= end)
 			return -EINVAL;
@@ -1282,8 +1305,9 @@ static int hisi_acc_vfio_pci_mmap(struct vfio_device *core_vdev,
 	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
 	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
 		u64 req_len, pgoff, req_start;
-		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
+		resource_size_t end;
 
+		end = hisi_acc_get_resource_len(vdev, index);
 		req_len = vma->vm_end - vma->vm_start;
 		pgoff = vma->vm_pgoff &
 			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
@@ -1330,7 +1354,6 @@ static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int
 	if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
 		struct vfio_pci_core_device *vdev =
 			container_of(core_vdev, struct vfio_pci_core_device, vdev);
-		struct pci_dev *pdev = vdev->pdev;
 		struct vfio_region_info info;
 		unsigned long minsz;
 
@@ -1345,12 +1368,7 @@ static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int
 		if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
 			info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
 
-			/*
-			 * ACC VF dev BAR2 region consists of both functional
-			 * register space and migration control register space.
-			 * Report only the functional region to Guest.
-			 */
-			info.size = pci_resource_len(pdev, info.index) / 2;
+			info.size = hisi_acc_get_resource_len(vdev, info.index);
 
 			info.flags = VFIO_REGION_INFO_FLAG_READ |
 					VFIO_REGION_INFO_FLAG_WRITE |
-- 
2.24.0

RE: [PATCH v5 1/3] migration: update BAR space size

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: liulongfang <liulongfang@huawei.com>
> Sent: Monday, June 30, 2025 9:54 AM
> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
> Subject: [PATCH v5 1/3] migration: update BAR space size
> 
> On new platforms greater than QM_HW_V3, the live migration
> configuration
> region is moved from VF to PF. The VF's own configuration space is
> restored to the complete 64KB, and there is no need to divide the
> size of the BAR configuration space equally.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> ---

See one minor comment below.

LGTM:
Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 36 ++++++++++++++-----
>  1 file changed, 27 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> index 2149f49aeec7..1ddc9dbadb70 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> @@ -1250,6 +1250,28 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct
> pci_dev *pdev)
>  	return !IS_ERR(pf_qm) ? pf_qm : NULL;
>  }
> 
> +static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,
> +					unsigned int index)
> +{
> +	struct hisi_acc_vf_core_device *hisi_acc_vdev =
> +			hisi_acc_drvdata(vdev->pdev);
> +
> +	/*
> +	 * On the old QM_HW_V3 device, the ACC VF device BAR2
> +	 * region encompasses both functional register space
> +	 * and migration control register space.
> +	 * only the functional region should be report to Guest.
> +	 *
> +	 * On the new HW device, the migration control register
> +	 * has been moved to the PF device BAR2 region.
> +	 * The VF device BAR2 is entirely functional register space.
> +	 */

Nit: May be move the above comment about new HW to the below
check  is better.

> +	if (hisi_acc_vdev->pf_qm->ver == QM_HW_V3)
> +		return (pci_resource_len(vdev->pdev, index) >> 1);
> +
> +	return pci_resource_len(vdev->pdev, index);
> +}
> +
>  static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
>  					size_t count, loff_t *ppos,
>  					size_t *new_count)
> @@ -1260,8 +1282,9 @@ static int hisi_acc_pci_rw_access_check(struct
> vfio_device *core_vdev,
> 
>  	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
>  		loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
> -		resource_size_t end = pci_resource_len(vdev->pdev, index) /
> 2;
> +		resource_size_t end;
> 
> +		end = hisi_acc_get_resource_len(vdev, index);
>  		/* Check if access is for migration control region */
>  		if (pos >= end)
>  			return -EINVAL;
> @@ -1282,8 +1305,9 @@ static int hisi_acc_vfio_pci_mmap(struct
> vfio_device *core_vdev,
>  	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
>  	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
>  		u64 req_len, pgoff, req_start;
> -		resource_size_t end = pci_resource_len(vdev->pdev, index) /
> 2;
> +		resource_size_t end;
> 
> +		end = hisi_acc_get_resource_len(vdev, index);
>  		req_len = vma->vm_end - vma->vm_start;
>  		pgoff = vma->vm_pgoff &
>  			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
> @@ -1330,7 +1354,6 @@ static long hisi_acc_vfio_pci_ioctl(struct
> vfio_device *core_vdev, unsigned int
>  	if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
>  		struct vfio_pci_core_device *vdev =
>  			container_of(core_vdev, struct vfio_pci_core_device,
> vdev);
> -		struct pci_dev *pdev = vdev->pdev;
>  		struct vfio_region_info info;
>  		unsigned long minsz;
> 
> @@ -1345,12 +1368,7 @@ static long hisi_acc_vfio_pci_ioctl(struct
> vfio_device *core_vdev, unsigned int
>  		if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
>  			info.offset =
> VFIO_PCI_INDEX_TO_OFFSET(info.index);
> 
> -			/*
> -			 * ACC VF dev BAR2 region consists of both
> functional
> -			 * register space and migration control register
> space.
> -			 * Report only the functional region to Guest.
> -			 */
> -			info.size = pci_resource_len(pdev, info.index) / 2;
> +			info.size = hisi_acc_get_resource_len(vdev,
> info.index);
> 
>  			info.flags = VFIO_REGION_INFO_FLAG_READ |
>  					VFIO_REGION_INFO_FLAG_WRITE |
> --
> 2.24.0

Re: [PATCH v5 1/3] migration: update BAR space size

[liulongfang]

On 2025/7/8 16:02, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: liulongfang <liulongfang@huawei.com>
>> Sent: Monday, June 30, 2025 9:54 AM
>> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
>> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
>> <jonathan.cameron@huawei.com>
>> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
>> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
>> Subject: [PATCH v5 1/3] migration: update BAR space size
>>
>> On new platforms greater than QM_HW_V3, the live migration
>> configuration
>> region is moved from VF to PF. The VF's own configuration space is
>> restored to the complete 64KB, and there is no need to divide the
>> size of the BAR configuration space equally.
>>
>> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
>> ---
> 
> See one minor comment below.
> 
> LGTM:
> Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> 
>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 36 ++++++++++++++-----
>>  1 file changed, 27 insertions(+), 9 deletions(-)
>>
>> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> index 2149f49aeec7..1ddc9dbadb70 100644
>> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> @@ -1250,6 +1250,28 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct
>> pci_dev *pdev)
>>  	return !IS_ERR(pf_qm) ? pf_qm : NULL;
>>  }
>>
>> +static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,
>> +					unsigned int index)
>> +{
>> +	struct hisi_acc_vf_core_device *hisi_acc_vdev =
>> +			hisi_acc_drvdata(vdev->pdev);
>> +
>> +	/*
>> +	 * On the old QM_HW_V3 device, the ACC VF device BAR2
>> +	 * region encompasses both functional register space
>> +	 * and migration control register space.
>> +	 * only the functional region should be report to Guest.
>> +	 *
>> +	 * On the new HW device, the migration control register
>> +	 * has been moved to the PF device BAR2 region.
>> +	 * The VF device BAR2 is entirely functional register space.
>> +	 */
> 
> Nit: May be move the above comment about new HW to the below
> check  is better.
>

OK.

Thanks.
Longfang.

>> +	if (hisi_acc_vdev->pf_qm->ver == QM_HW_V3)
>> +		return (pci_resource_len(vdev->pdev, index) >> 1);
>> +
>> +	return pci_resource_len(vdev->pdev, index);
>> +}
>> +
>>  static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
>>  					size_t count, loff_t *ppos,
>>  					size_t *new_count)
>> @@ -1260,8 +1282,9 @@ static int hisi_acc_pci_rw_access_check(struct
>> vfio_device *core_vdev,
>>
>>  	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
>>  		loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
>> -		resource_size_t end = pci_resource_len(vdev->pdev, index) /
>> 2;
>> +		resource_size_t end;
>>
>> +		end = hisi_acc_get_resource_len(vdev, index);
>>  		/* Check if access is for migration control region */
>>  		if (pos >= end)
>>  			return -EINVAL;
>> @@ -1282,8 +1305,9 @@ static int hisi_acc_vfio_pci_mmap(struct
>> vfio_device *core_vdev,
>>  	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
>>  	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
>>  		u64 req_len, pgoff, req_start;
>> -		resource_size_t end = pci_resource_len(vdev->pdev, index) /
>> 2;
>> +		resource_size_t end;
>>
>> +		end = hisi_acc_get_resource_len(vdev, index);
>>  		req_len = vma->vm_end - vma->vm_start;
>>  		pgoff = vma->vm_pgoff &
>>  			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
>> @@ -1330,7 +1354,6 @@ static long hisi_acc_vfio_pci_ioctl(struct
>> vfio_device *core_vdev, unsigned int
>>  	if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
>>  		struct vfio_pci_core_device *vdev =
>>  			container_of(core_vdev, struct vfio_pci_core_device,
>> vdev);
>> -		struct pci_dev *pdev = vdev->pdev;
>>  		struct vfio_region_info info;
>>  		unsigned long minsz;
>>
>> @@ -1345,12 +1368,7 @@ static long hisi_acc_vfio_pci_ioctl(struct
>> vfio_device *core_vdev, unsigned int
>>  		if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
>>  			info.offset =
>> VFIO_PCI_INDEX_TO_OFFSET(info.index);
>>
>> -			/*
>> -			 * ACC VF dev BAR2 region consists of both
>> functional
>> -			 * register space and migration control register
>> space.
>> -			 * Report only the functional region to Guest.
>> -			 */
>> -			info.size = pci_resource_len(pdev, info.index) / 2;
>> +			info.size = hisi_acc_get_resource_len(vdev,
>> info.index);
>>
>>  			info.flags = VFIO_REGION_INFO_FLAG_READ |
>>  					VFIO_REGION_INFO_FLAG_WRITE |
>> --
>> 2.24.0
> 
> .
> 

[PATCH v5 2/3] migration: qm updates BAR configuration

[Longfang Liu]

On new platforms greater than QM_HW_V3, the configuration region for the
live migration function of the accelerator device is no longer
placed in the VF, but is instead placed in the PF.

Therefore, the configuration region of the live migration function
needs to be opened when the QM driver is loaded. When the QM driver
is uninstalled, the driver needs to clear this configuration.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
---
 drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index d3f5d108b898..0a8888304e15 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -242,6 +242,9 @@
 #define QM_QOS_MAX_CIR_U		6
 #define QM_AUTOSUSPEND_DELAY		3000
 
+#define QM_MIG_REGION_SEL		0x100198
+#define QM_MIG_REGION_EN		0x1
+
  /* abnormal status value for stopping queue */
 #define QM_STOP_QUEUE_FAIL		1
 #define	QM_DUMP_SQC_FAIL		3
@@ -3004,11 +3007,36 @@ static void qm_put_pci_res(struct hisi_qm *qm)
 	pci_release_mem_regions(pdev);
 }
 
+static void hisi_mig_region_clear(struct hisi_qm *qm)
+{
+	u32 val;
+
+	/* Clear migration region set of PF */
+	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
+		val = readl(qm->io_base + QM_MIG_REGION_SEL);
+		val &= ~BIT(0);
+		writel(val, qm->io_base + QM_MIG_REGION_SEL);
+	}
+}
+
+static void hisi_mig_region_enable(struct hisi_qm *qm)
+{
+	u32 val;
+
+	/* Select migration region of PF */
+	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
+		val = readl(qm->io_base + QM_MIG_REGION_SEL);
+		val |= QM_MIG_REGION_EN;
+		writel(val, qm->io_base + QM_MIG_REGION_SEL);
+	}
+}
+
 static void hisi_qm_pci_uninit(struct hisi_qm *qm)
 {
 	struct pci_dev *pdev = qm->pdev;
 
 	pci_free_irq_vectors(pdev);
+	hisi_mig_region_clear(qm);
 	qm_put_pci_res(qm);
 	pci_disable_device(pdev);
 }
@@ -5630,6 +5658,7 @@ int hisi_qm_init(struct hisi_qm *qm)
 		goto err_free_qm_memory;
 
 	qm_cmd_init(qm);
+	hisi_mig_region_enable(qm);
 
 	return 0;
 
-- 
2.24.0

RE: [PATCH v5 2/3] migration: qm updates BAR configuration

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: liulongfang <liulongfang@huawei.com>
> Sent: Monday, June 30, 2025 9:54 AM
> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
> Subject: [PATCH v5 2/3] migration: qm updates BAR configuration
> 
> On new platforms greater than QM_HW_V3, the configuration region for
> the
> live migration function of the accelerator device is no longer
> placed in the VF, but is instead placed in the PF.
> 
> Therefore, the configuration region of the live migration function
> needs to be opened when the QM driver is loaded. When the QM driver
> is uninstalled, the driver needs to clear this configuration.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> ---
>  drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index d3f5d108b898..0a8888304e15 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -242,6 +242,9 @@
>  #define QM_QOS_MAX_CIR_U		6
>  #define QM_AUTOSUSPEND_DELAY		3000
> 
> +#define QM_MIG_REGION_SEL		0x100198
> +#define QM_MIG_REGION_EN		0x1
> +
>   /* abnormal status value for stopping queue */
>  #define QM_STOP_QUEUE_FAIL		1
>  #define	QM_DUMP_SQC_FAIL		3
> @@ -3004,11 +3007,36 @@ static void qm_put_pci_res(struct hisi_qm *qm)
>  	pci_release_mem_regions(pdev);
>  }
> 
> +static void hisi_mig_region_clear(struct hisi_qm *qm)
> +{
> +	u32 val;
> +
> +	/* Clear migration region set of PF */
> +	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
> +		val = readl(qm->io_base + QM_MIG_REGION_SEL);
> +		val &= ~BIT(0);
> +		writel(val, qm->io_base + QM_MIG_REGION_SEL);
> +	}
> +}
> +
> +static void hisi_mig_region_enable(struct hisi_qm *qm)
> +{
> +	u32 val;
> +
> +	/* Select migration region of PF */
> +	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
> +		val = readl(qm->io_base + QM_MIG_REGION_SEL);
> +		val |= QM_MIG_REGION_EN;
> +		writel(val, qm->io_base + QM_MIG_REGION_SEL);
> +	}
> +}

May adding a  comment for above functions  will be helpful.

> +
>  static void hisi_qm_pci_uninit(struct hisi_qm *qm)
>  {
>  	struct pci_dev *pdev = qm->pdev;
> 
>  	pci_free_irq_vectors(pdev);
> +	hisi_mig_region_clear(qm);
>  	qm_put_pci_res(qm);
>  	pci_disable_device(pdev);
>  }
> @@ -5630,6 +5658,7 @@ int hisi_qm_init(struct hisi_qm *qm)
>  		goto err_free_qm_memory;
> 
>  	qm_cmd_init(qm);
> +	hisi_mig_region_enable(qm);
> 
>  	return 0;

Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>

You need to CC QM driver maintainers for this.

Thanks,
SHameer

Re: [PATCH v5 2/3] migration: qm updates BAR configuration

[liulongfang]

On 2025/7/8 16:18, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: liulongfang <liulongfang@huawei.com>
>> Sent: Monday, June 30, 2025 9:54 AM
>> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
>> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
>> <jonathan.cameron@huawei.com>
>> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
>> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
>> Subject: [PATCH v5 2/3] migration: qm updates BAR configuration
>>
>> On new platforms greater than QM_HW_V3, the configuration region for
>> the
>> live migration function of the accelerator device is no longer
>> placed in the VF, but is instead placed in the PF.
>>
>> Therefore, the configuration region of the live migration function
>> needs to be opened when the QM driver is loaded. When the QM driver
>> is uninstalled, the driver needs to clear this configuration.
>>
>> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
>> ---
>>  drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++
>>  1 file changed, 29 insertions(+)
>>
>> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
>> index d3f5d108b898..0a8888304e15 100644
>> --- a/drivers/crypto/hisilicon/qm.c
>> +++ b/drivers/crypto/hisilicon/qm.c
>> @@ -242,6 +242,9 @@
>>  #define QM_QOS_MAX_CIR_U		6
>>  #define QM_AUTOSUSPEND_DELAY		3000
>>
>> +#define QM_MIG_REGION_SEL		0x100198
>> +#define QM_MIG_REGION_EN		0x1
>> +
>>   /* abnormal status value for stopping queue */
>>  #define QM_STOP_QUEUE_FAIL		1
>>  #define	QM_DUMP_SQC_FAIL		3
>> @@ -3004,11 +3007,36 @@ static void qm_put_pci_res(struct hisi_qm *qm)
>>  	pci_release_mem_regions(pdev);
>>  }
>>
>> +static void hisi_mig_region_clear(struct hisi_qm *qm)
>> +{
>> +	u32 val;
>> +
>> +	/* Clear migration region set of PF */
>> +	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
>> +		val = readl(qm->io_base + QM_MIG_REGION_SEL);
>> +		val &= ~BIT(0);
>> +		writel(val, qm->io_base + QM_MIG_REGION_SEL);
>> +	}
>> +}
>> +
>> +static void hisi_mig_region_enable(struct hisi_qm *qm)
>> +{
>> +	u32 val;
>> +
>> +	/* Select migration region of PF */
>> +	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
>> +		val = readl(qm->io_base + QM_MIG_REGION_SEL);
>> +		val |= QM_MIG_REGION_EN;
>> +		writel(val, qm->io_base + QM_MIG_REGION_SEL);
>> +	}
>> +}
> 
> May adding a  comment for above functions  will be helpful.
> 
>> +
>>  static void hisi_qm_pci_uninit(struct hisi_qm *qm)
>>  {
>>  	struct pci_dev *pdev = qm->pdev;
>>
>>  	pci_free_irq_vectors(pdev);
>> +	hisi_mig_region_clear(qm);
>>  	qm_put_pci_res(qm);
>>  	pci_disable_device(pdev);
>>  }
>> @@ -5630,6 +5658,7 @@ int hisi_qm_init(struct hisi_qm *qm)
>>  		goto err_free_qm_memory;
>>
>>  	qm_cmd_init(qm);
>> +	hisi_mig_region_enable(qm);
>>
>>  	return 0;
> 
> Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> 
> You need to CC QM driver maintainers for this.
>

OK, thanks.
Longfang.

> Thanks,
> SHameer
> .
> 

Re: [PATCH v5 2/3] migration: qm updates BAR configuration

[liulongfang]

On 2025/6/30 16:54, Longfang Liu wrote:
> On new platforms greater than QM_HW_V3, the configuration region for the
> live migration function of the accelerator device is no longer
> placed in the VF, but is instead placed in the PF.
> 
> Therefore, the configuration region of the live migration function
> needs to be opened when the QM driver is loaded. When the QM driver
> is uninstalled, the driver needs to clear this configuration.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> ---
>  drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
>

Hello, Herbert. There is a patch in this patchset that modifies the crypto subsystem.
Could you help review it?

Thanks.
Longfang.

> diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
> index d3f5d108b898..0a8888304e15 100644
> --- a/drivers/crypto/hisilicon/qm.c
> +++ b/drivers/crypto/hisilicon/qm.c
> @@ -242,6 +242,9 @@
>  #define QM_QOS_MAX_CIR_U		6
>  #define QM_AUTOSUSPEND_DELAY		3000
>  
> +#define QM_MIG_REGION_SEL		0x100198
> +#define QM_MIG_REGION_EN		0x1
> +
>   /* abnormal status value for stopping queue */
>  #define QM_STOP_QUEUE_FAIL		1
>  #define	QM_DUMP_SQC_FAIL		3
> @@ -3004,11 +3007,36 @@ static void qm_put_pci_res(struct hisi_qm *qm)
>  	pci_release_mem_regions(pdev);
>  }
>  
> +static void hisi_mig_region_clear(struct hisi_qm *qm)
> +{
> +	u32 val;
> +
> +	/* Clear migration region set of PF */
> +	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
> +		val = readl(qm->io_base + QM_MIG_REGION_SEL);
> +		val &= ~BIT(0);
> +		writel(val, qm->io_base + QM_MIG_REGION_SEL);
> +	}
> +}
> +
> +static void hisi_mig_region_enable(struct hisi_qm *qm)
> +{
> +	u32 val;
> +
> +	/* Select migration region of PF */
> +	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
> +		val = readl(qm->io_base + QM_MIG_REGION_SEL);
> +		val |= QM_MIG_REGION_EN;
> +		writel(val, qm->io_base + QM_MIG_REGION_SEL);
> +	}
> +}
> +
>  static void hisi_qm_pci_uninit(struct hisi_qm *qm)
>  {
>  	struct pci_dev *pdev = qm->pdev;
>  
>  	pci_free_irq_vectors(pdev);
> +	hisi_mig_region_clear(qm);
>  	qm_put_pci_res(qm);
>  	pci_disable_device(pdev);
>  }
> @@ -5630,6 +5658,7 @@ int hisi_qm_init(struct hisi_qm *qm)
>  		goto err_free_qm_memory;
>  
>  	qm_cmd_init(qm);
> +	hisi_mig_region_enable(qm);
>  
>  	return 0;
>  
> 

Re: [PATCH v5 2/3] migration: qm updates BAR configuration

[Herbert Xu]

On Tue, Jul 15, 2025 at 03:23:09PM +0800, liulongfang wrote:
> On 2025/6/30 16:54, Longfang Liu wrote:
> > On new platforms greater than QM_HW_V3, the configuration region for the
> > live migration function of the accelerator device is no longer
> > placed in the VF, but is instead placed in the PF.
> > 
> > Therefore, the configuration region of the live migration function
> > needs to be opened when the QM driver is loaded. When the QM driver
> > is uninstalled, the driver needs to clear this configuration.
> > 
> > Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> > ---
> >  drivers/crypto/hisilicon/qm.c | 29 +++++++++++++++++++++++++++++
> >  1 file changed, 29 insertions(+)
> >
> 
> Hello, Herbert. There is a patch in this patchset that modifies the crypto subsystem.
> Could you help review it?

Acked-by: Herbert Xu <herbert@gondor.apana.org.au>

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[PATCH v5 3/3] migration: adapt to new migration configuration

[Longfang Liu]

On new platforms greater than QM_HW_V3, the migration region has been
relocated from the VF to the PF. The driver must also be modified
accordingly to adapt to the new hardware device.

Utilize the PF's I/O base directly on the new hardware platform,
and no mmap operation is required. If it is on an old platform,
the driver needs to be compatible with the old solution.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 166 ++++++++++++------
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |   7 +
 2 files changed, 120 insertions(+), 53 deletions(-)

diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 1ddc9dbadb70..3aec3b92787f 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 *addr)
 	return 0;
 }
 
+static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			   struct acc_vf_data *vf_data)
+{
+	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
+	struct device *dev = &qm->pdev->dev;
+	u32 eqc_addr, aeqc_addr;
+	int ret;
+
+	if (qm->ver == QM_HW_V3) {
+		eqc_addr = QM_EQC_DW0;
+		aeqc_addr = QM_AEQC_DW0;
+	} else {
+		eqc_addr = QM_EQC_PF_DW0;
+		aeqc_addr = QM_AEQC_PF_DW0;
+	}
+
+	/* QM_EQC_DW has 7 regs */
+	ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to read QM_EQC_DW\n");
+		return ret;
+	}
+
+	/* QM_AEQC_DW has 7 regs */
+	ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to read QM_AEQC_DW\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
+			   struct acc_vf_data *vf_data)
+{
+	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
+	struct device *dev = &qm->pdev->dev;
+	u32 eqc_addr, aeqc_addr;
+	int ret;
+
+	if (qm->ver == QM_HW_V3) {
+		eqc_addr = QM_EQC_DW0;
+		aeqc_addr = QM_AEQC_DW0;
+	} else {
+		eqc_addr = QM_EQC_PF_DW0;
+		aeqc_addr = QM_AEQC_PF_DW0;
+	}
+
+	/* QM_EQC_DW has 7 regs */
+	ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to write QM_EQC_DW\n");
+		return ret;
+	}
+
+	/* QM_AEQC_DW has 7 regs */
+	ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
+	if (ret) {
+		dev_err(dev, "failed to write QM_AEQC_DW\n");
+		return ret;
+	}
+
+	return 0;
+}
+
 static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
 {
 	struct device *dev = &qm->pdev->dev;
@@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
 		return ret;
 	}
 
-	/* QM_EQC_DW has 7 regs */
-	ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
-	if (ret) {
-		dev_err(dev, "failed to read QM_EQC_DW\n");
-		return ret;
-	}
-
-	/* QM_AEQC_DW has 7 regs */
-	ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
-	if (ret) {
-		dev_err(dev, "failed to read QM_AEQC_DW\n");
-		return ret;
-	}
-
 	return 0;
 }
 
@@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
 		return ret;
 	}
 
-	/* QM_EQC_DW has 7 regs */
-	ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
-	if (ret) {
-		dev_err(dev, "failed to write QM_EQC_DW\n");
-		return ret;
-	}
-
-	/* QM_AEQC_DW has 7 regs */
-	ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
-	if (ret) {
-		dev_err(dev, "failed to write QM_AEQC_DW\n");
-		return ret;
-	}
-
 	return 0;
 }
 
@@ -522,6 +560,10 @@ static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev,
 		return ret;
 	}
 
+	ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data);
+	if (ret)
+		return ret;
+
 	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
 	if (ret) {
 		dev_err(dev, "set sqc failed\n");
@@ -589,6 +631,10 @@ static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev,
 	vf_data->vf_qm_state = QM_READY;
 	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
 
+	ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data);
+	if (ret)
+		return ret;
+
 	ret = vf_qm_read_data(vf_qm, vf_data);
 	if (ret)
 		return ret;
@@ -1186,34 +1232,47 @@ static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
 {
 	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
 	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
 	struct pci_dev *vf_dev = vdev->pdev;
 
-	/*
-	 * ACC VF dev BAR2 region consists of both functional register space
-	 * and migration control register space. For migration to work, we
-	 * need access to both. Hence, we map the entire BAR2 region here.
-	 * But unnecessarily exposing the migration BAR region to the Guest
-	 * has the potential to prevent/corrupt the Guest migration. Hence,
-	 * we restrict access to the migration control space from
-	 * Guest(Please see mmap/ioctl/read/write override functions).
-	 *
-	 * Please note that it is OK to expose the entire VF BAR if migration
-	 * is not supported or required as this cannot affect the ACC PF
-	 * configurations.
-	 *
-	 * Also the HiSilicon ACC VF devices supported by this driver on
-	 * HiSilicon hardware platforms are integrated end point devices
-	 * and the platform lacks the capability to perform any PCIe P2P
-	 * between these devices.
-	 */
+	if (pf_qm->ver == QM_HW_V3) {
+		/*
+		 * ACC VF dev BAR2 region consists of both functional register space
+		 * and migration control register space. For migration to work, we
+		 * need access to both. Hence, we map the entire BAR2 region here.
+		 * But unnecessarily exposing the migration BAR region to the Guest
+		 * has the potential to prevent/corrupt the Guest migration. Hence,
+		 * we restrict access to the migration control space from
+		 * Guest(Please see mmap/ioctl/read/write override functions).
+		 *
+		 * Please note that it is OK to expose the entire VF BAR if migration
+		 * is not supported or required as this cannot affect the ACC PF
+		 * configurations.
+		 *
+		 * Also the HiSilicon ACC VF devices supported by this driver on
+		 * HiSilicon hardware platforms are integrated end point devices
+		 * and the platform lacks the capability to perform any PCIe P2P
+		 * between these devices.
+		 */
 
-	vf_qm->io_base =
-		ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
-			pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
-	if (!vf_qm->io_base)
-		return -EIO;
+		vf_qm->io_base =
+			ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
+				pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
+		if (!vf_qm->io_base)
+			return -EIO;
 
-	vf_qm->fun_type = QM_HW_VF;
+		vf_qm->fun_type = QM_HW_VF;
+		vf_qm->ver = pf_qm->ver;
+	} else {
+		/*
+		 * On hardware platforms greater than QM_HW_V3, the migration function
+		 * register is placed in the BAR2 configuration region of the PF,
+		 * and each VF device occupies 8KB of configuration space.
+		 */
+		vf_qm->io_base = pf_qm->io_base + QM_MIG_REGION_OFFSET +
+				 hisi_acc_vdev->vf_id * QM_MIG_REGION_SIZE;
+		vf_qm->fun_type = QM_HW_PF;
+	}
 	vf_qm->pdev = vf_dev;
 	mutex_init(&vf_qm->mailbox_lock);
 
@@ -1539,7 +1598,8 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)
 	hisi_acc_vf_disable_fds(hisi_acc_vdev);
 	mutex_lock(&hisi_acc_vdev->open_mutex);
 	hisi_acc_vdev->dev_opened = false;
-	iounmap(vf_qm->io_base);
+	if (vf_qm->ver == QM_HW_V3)
+		iounmap(vf_qm->io_base);
 	mutex_unlock(&hisi_acc_vdev->open_mutex);
 	vfio_pci_core_close_device(core_vdev);
 }
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
index 91002ceeebc1..348f8bb5b42c 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
@@ -59,6 +59,13 @@
 #define ACC_DEV_MAGIC_V1	0XCDCDCDCDFEEDAACC
 #define ACC_DEV_MAGIC_V2	0xAACCFEEDDECADEDE
 
+#define QM_MIG_REGION_OFFSET		0x180000
+#define QM_MIG_REGION_SIZE		0x2000
+
+#define QM_SUB_VERSION_ID		0x100210
+#define QM_EQC_PF_DW0			0x1c00
+#define QM_AEQC_PF_DW0			0x1c20
+
 struct acc_vf_data {
 #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
 	/* QM match information */
-- 
2.24.0

RE: [PATCH v5 3/3] migration: adapt to new migration configuration

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: liulongfang <liulongfang@huawei.com>
> Sent: Monday, June 30, 2025 9:54 AM
> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>
> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
> Subject: [PATCH v5 3/3] migration: adapt to new migration configuration
> 
> On new platforms greater than QM_HW_V3, the migration region has been
> relocated from the VF to the PF. The driver must also be modified
> accordingly to adapt to the new hardware device.
> 
> Utilize the PF's I/O base directly on the new hardware platform,
> and no mmap operation is required. If it is on an old platform,
> the driver needs to be compatible with the old solution.
> 
> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
> ---
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 166 ++++++++++++------
>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |   7 +
>  2 files changed, 120 insertions(+), 53 deletions(-)
> 
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> index 1ddc9dbadb70..3aec3b92787f 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64
> *addr)
>  	return 0;
>  }
> 
> +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			   struct acc_vf_data *vf_data)
> +{
> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
> +	struct device *dev = &qm->pdev->dev;
> +	u32 eqc_addr, aeqc_addr;
> +	int ret;
> +
> +	if (qm->ver == QM_HW_V3) {
> +		eqc_addr = QM_EQC_DW0;
> +		aeqc_addr = QM_AEQC_DW0;
> +	} else {
> +		eqc_addr = QM_EQC_PF_DW0;
> +		aeqc_addr = QM_AEQC_PF_DW0;
> +	}
> +
> +	/* QM_EQC_DW has 7 regs */
> +	ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
> +	if (ret) {
> +		dev_err(dev, "failed to read QM_EQC_DW\n");
> +		return ret;
> +	}
> +
> +	/* QM_AEQC_DW has 7 regs */
> +	ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
> +	if (ret) {
> +		dev_err(dev, "failed to read QM_AEQC_DW\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
> +			   struct acc_vf_data *vf_data)
> +{
> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
> +	struct device *dev = &qm->pdev->dev;
> +	u32 eqc_addr, aeqc_addr;
> +	int ret;
> +
> +	if (qm->ver == QM_HW_V3) {
> +		eqc_addr = QM_EQC_DW0;
> +		aeqc_addr = QM_AEQC_DW0;
> +	} else {
> +		eqc_addr = QM_EQC_PF_DW0;
> +		aeqc_addr = QM_AEQC_PF_DW0;
> +	}
> +
> +	/* QM_EQC_DW has 7 regs */
> +	ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
> +	if (ret) {
> +		dev_err(dev, "failed to write QM_EQC_DW\n");
> +		return ret;
> +	}
> +
> +	/* QM_AEQC_DW has 7 regs */
> +	ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
> +	if (ret) {
> +		dev_err(dev, "failed to write QM_AEQC_DW\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
>  static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
>  {
>  	struct device *dev = &qm->pdev->dev;
> @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct
> acc_vf_data *vf_data)
>  		return ret;
>  	}
> 
> -	/* QM_EQC_DW has 7 regs */
> -	ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
> -	if (ret) {
> -		dev_err(dev, "failed to read QM_EQC_DW\n");
> -		return ret;
> -	}
> -
> -	/* QM_AEQC_DW has 7 regs */
> -	ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw,
> 7);
> -	if (ret) {
> -		dev_err(dev, "failed to read QM_AEQC_DW\n");
> -		return ret;
> -	}
> -
>  	return 0;
>  }
> 
> @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct
> acc_vf_data *vf_data)
>  		return ret;
>  	}
> 
> -	/* QM_EQC_DW has 7 regs */
> -	ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
> -	if (ret) {
> -		dev_err(dev, "failed to write QM_EQC_DW\n");
> -		return ret;
> -	}
> -
> -	/* QM_AEQC_DW has 7 regs */
> -	ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw,
> 7);
> -	if (ret) {
> -		dev_err(dev, "failed to write QM_AEQC_DW\n");
> -		return ret;
> -	}
> -
>  	return 0;
>  }
> 
> @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct
> hisi_acc_vf_core_device *hisi_acc_vdev,
>  		return ret;
>  	}
> 
> +	ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data);
> +	if (ret)
> +		return ret;
> +
>  	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
>  	if (ret) {
>  		dev_err(dev, "set sqc failed\n");
> @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct
> hisi_acc_vf_core_device *hisi_acc_vdev,
>  	vf_data->vf_qm_state = QM_READY;
>  	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
> 
> +	ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data);
> +	if (ret)
> +		return ret;
> +
>  	ret = vf_qm_read_data(vf_qm, vf_data);
>  	if (ret)
>  		return ret;
> @@ -1186,34 +1232,47 @@ static int hisi_acc_vf_qm_init(struct
> hisi_acc_vf_core_device *hisi_acc_vdev)
>  {
>  	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
>  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
> +	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
>  	struct pci_dev *vf_dev = vdev->pdev;
> 
> -	/*
> -	 * ACC VF dev BAR2 region consists of both functional register space
> -	 * and migration control register space. For migration to work, we
> -	 * need access to both. Hence, we map the entire BAR2 region here.
> -	 * But unnecessarily exposing the migration BAR region to the Guest
> -	 * has the potential to prevent/corrupt the Guest migration. Hence,
> -	 * we restrict access to the migration control space from
> -	 * Guest(Please see mmap/ioctl/read/write override functions).
> -	 *
> -	 * Please note that it is OK to expose the entire VF BAR if migration
> -	 * is not supported or required as this cannot affect the ACC PF
> -	 * configurations.
> -	 *
> -	 * Also the HiSilicon ACC VF devices supported by this driver on
> -	 * HiSilicon hardware platforms are integrated end point devices
> -	 * and the platform lacks the capability to perform any PCIe P2P
> -	 * between these devices.
> -	 */
> +	if (pf_qm->ver == QM_HW_V3) {
> +		/*
> +		 * ACC VF dev BAR2 region consists of both functional
> register space
> +		 * and migration control register space. For migration to
> work, we
> +		 * need access to both. Hence, we map the entire BAR2
> region here.
> +		 * But unnecessarily exposing the migration BAR region to
> the Guest
> +		 * has the potential to prevent/corrupt the Guest migration.
> Hence,
> +		 * we restrict access to the migration control space from
> +		 * Guest(Please see mmap/ioctl/read/write override
> functions).
> +		 *
> +		 * Please note that it is OK to expose the entire VF BAR if
> migration
> +		 * is not supported or required as this cannot affect the ACC
> PF
> +		 * configurations.
> +		 *
> +		 * Also the HiSilicon ACC VF devices supported by this driver
> on
> +		 * HiSilicon hardware platforms are integrated end point
> devices
> +		 * and the platform lacks the capability to perform any PCIe
> P2P
> +		 * between these devices.
> +		 */
> 
> -	vf_qm->io_base =
> -		ioremap(pci_resource_start(vf_dev,
> VFIO_PCI_BAR2_REGION_INDEX),
> -			pci_resource_len(vf_dev,
> VFIO_PCI_BAR2_REGION_INDEX));
> -	if (!vf_qm->io_base)
> -		return -EIO;
> +		vf_qm->io_base =
> +			ioremap(pci_resource_start(vf_dev,
> VFIO_PCI_BAR2_REGION_INDEX),
> +				pci_resource_len(vf_dev,
> VFIO_PCI_BAR2_REGION_INDEX));
> +		if (!vf_qm->io_base)
> +			return -EIO;
> 
> -	vf_qm->fun_type = QM_HW_VF;
> +		vf_qm->fun_type = QM_HW_VF;
> +		vf_qm->ver = pf_qm->ver;
> +	} else {
> +		/*
> +		 * On hardware platforms greater than QM_HW_V3, the
> migration function
> +		 * register is placed in the BAR2 configuration region of the
> PF,
> +		 * and each VF device occupies 8KB of configuration space.
> +		 */
> +		vf_qm->io_base = pf_qm->io_base +
> QM_MIG_REGION_OFFSET +
> +				 hisi_acc_vdev->vf_id *
> QM_MIG_REGION_SIZE;
> +		vf_qm->fun_type = QM_HW_PF;

I don't think you can use the QM fun_type to distinguish this, because it is still a
VF dev. Better to have another field to detect this.

Also I have another question. In the hisi_acc_vfio_pci_probe()  we currently
look for pf_qm-> >= QM_HW_V3. This means current driver can get loaded
on this new hardware, right? If so, I think we need to prevent that. And also
we need to block any migration  attempt from existing host kernels to new
ones.

Thanks,
Shameer

> +	}
>  	vf_qm->pdev = vf_dev;
>  	mutex_init(&vf_qm->mailbox_lock);
> 
> @@ -1539,7 +1598,8 @@ static void hisi_acc_vfio_pci_close_device(struct
> vfio_device *core_vdev)
>  	hisi_acc_vf_disable_fds(hisi_acc_vdev);
>  	mutex_lock(&hisi_acc_vdev->open_mutex);
>  	hisi_acc_vdev->dev_opened = false;
> -	iounmap(vf_qm->io_base);
> +	if (vf_qm->ver == QM_HW_V3)
> +		iounmap(vf_qm->io_base);
>  	mutex_unlock(&hisi_acc_vdev->open_mutex);
>  	vfio_pci_core_close_device(core_vdev);
>  }
> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> index 91002ceeebc1..348f8bb5b42c 100644
> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> @@ -59,6 +59,13 @@
>  #define ACC_DEV_MAGIC_V1	0XCDCDCDCDFEEDAACC
>  #define ACC_DEV_MAGIC_V2	0xAACCFEEDDECADEDE
> 
> +#define QM_MIG_REGION_OFFSET		0x180000
> +#define QM_MIG_REGION_SIZE		0x2000
> +
> +#define QM_SUB_VERSION_ID		0x100210
> +#define QM_EQC_PF_DW0			0x1c00
> +#define QM_AEQC_PF_DW0			0x1c20
> +
>  struct acc_vf_data {
>  #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
>  	/* QM match information */
> --
> 2.24.0

Re: [PATCH v5 3/3] migration: adapt to new migration configuration

[liulongfang]

On 2025/7/8 16:28, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: liulongfang <liulongfang@huawei.com>
>> Sent: Monday, June 30, 2025 9:54 AM
>> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
>> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
>> <jonathan.cameron@huawei.com>
>> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
>> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
>> Subject: [PATCH v5 3/3] migration: adapt to new migration configuration
>>
>> On new platforms greater than QM_HW_V3, the migration region has been
>> relocated from the VF to the PF. The driver must also be modified
>> accordingly to adapt to the new hardware device.
>>
>> Utilize the PF's I/O base directly on the new hardware platform,
>> and no mmap operation is required. If it is on an old platform,
>> the driver needs to be compatible with the old solution.
>>
>> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
>> ---
>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 166 ++++++++++++------
>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |   7 +
>>  2 files changed, 120 insertions(+), 53 deletions(-)
>>
>> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> index 1ddc9dbadb70..3aec3b92787f 100644
>> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>> @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64
>> *addr)
>>  	return 0;
>>  }
>>
>> +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>> +			   struct acc_vf_data *vf_data)
>> +{
>> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
>> +	struct device *dev = &qm->pdev->dev;
>> +	u32 eqc_addr, aeqc_addr;
>> +	int ret;
>> +
>> +	if (qm->ver == QM_HW_V3) {
>> +		eqc_addr = QM_EQC_DW0;
>> +		aeqc_addr = QM_AEQC_DW0;
>> +	} else {
>> +		eqc_addr = QM_EQC_PF_DW0;
>> +		aeqc_addr = QM_AEQC_PF_DW0;
>> +	}
>> +
>> +	/* QM_EQC_DW has 7 regs */
>> +	ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
>> +	if (ret) {
>> +		dev_err(dev, "failed to read QM_EQC_DW\n");
>> +		return ret;
>> +	}
>> +
>> +	/* QM_AEQC_DW has 7 regs */
>> +	ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
>> +	if (ret) {
>> +		dev_err(dev, "failed to read QM_AEQC_DW\n");
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>> +			   struct acc_vf_data *vf_data)
>> +{
>> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
>> +	struct device *dev = &qm->pdev->dev;
>> +	u32 eqc_addr, aeqc_addr;
>> +	int ret;
>> +
>> +	if (qm->ver == QM_HW_V3) {
>> +		eqc_addr = QM_EQC_DW0;
>> +		aeqc_addr = QM_AEQC_DW0;
>> +	} else {
>> +		eqc_addr = QM_EQC_PF_DW0;
>> +		aeqc_addr = QM_AEQC_PF_DW0;
>> +	}
>> +
>> +	/* QM_EQC_DW has 7 regs */
>> +	ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
>> +	if (ret) {
>> +		dev_err(dev, "failed to write QM_EQC_DW\n");
>> +		return ret;
>> +	}
>> +
>> +	/* QM_AEQC_DW has 7 regs */
>> +	ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
>> +	if (ret) {
>> +		dev_err(dev, "failed to write QM_AEQC_DW\n");
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>>  static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
>>  {
>>  	struct device *dev = &qm->pdev->dev;
>> @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct
>> acc_vf_data *vf_data)
>>  		return ret;
>>  	}
>>
>> -	/* QM_EQC_DW has 7 regs */
>> -	ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
>> -	if (ret) {
>> -		dev_err(dev, "failed to read QM_EQC_DW\n");
>> -		return ret;
>> -	}
>> -
>> -	/* QM_AEQC_DW has 7 regs */
>> -	ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw,
>> 7);
>> -	if (ret) {
>> -		dev_err(dev, "failed to read QM_AEQC_DW\n");
>> -		return ret;
>> -	}
>> -
>>  	return 0;
>>  }
>>
>> @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct
>> acc_vf_data *vf_data)
>>  		return ret;
>>  	}
>>
>> -	/* QM_EQC_DW has 7 regs */
>> -	ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
>> -	if (ret) {
>> -		dev_err(dev, "failed to write QM_EQC_DW\n");
>> -		return ret;
>> -	}
>> -
>> -	/* QM_AEQC_DW has 7 regs */
>> -	ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw,
>> 7);
>> -	if (ret) {
>> -		dev_err(dev, "failed to write QM_AEQC_DW\n");
>> -		return ret;
>> -	}
>> -
>>  	return 0;
>>  }
>>
>> @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct
>> hisi_acc_vf_core_device *hisi_acc_vdev,
>>  		return ret;
>>  	}
>>
>> +	ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data);
>> +	if (ret)
>> +		return ret;
>> +
>>  	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
>>  	if (ret) {
>>  		dev_err(dev, "set sqc failed\n");
>> @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct
>> hisi_acc_vf_core_device *hisi_acc_vdev,
>>  	vf_data->vf_qm_state = QM_READY;
>>  	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
>>
>> +	ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data);
>> +	if (ret)
>> +		return ret;
>> +
>>  	ret = vf_qm_read_data(vf_qm, vf_data);
>>  	if (ret)
>>  		return ret;
>> @@ -1186,34 +1232,47 @@ static int hisi_acc_vf_qm_init(struct
>> hisi_acc_vf_core_device *hisi_acc_vdev)
>>  {
>>  	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
>>  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
>> +	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
>>  	struct pci_dev *vf_dev = vdev->pdev;
>>
>> -	/*
>> -	 * ACC VF dev BAR2 region consists of both functional register space
>> -	 * and migration control register space. For migration to work, we
>> -	 * need access to both. Hence, we map the entire BAR2 region here.
>> -	 * But unnecessarily exposing the migration BAR region to the Guest
>> -	 * has the potential to prevent/corrupt the Guest migration. Hence,
>> -	 * we restrict access to the migration control space from
>> -	 * Guest(Please see mmap/ioctl/read/write override functions).
>> -	 *
>> -	 * Please note that it is OK to expose the entire VF BAR if migration
>> -	 * is not supported or required as this cannot affect the ACC PF
>> -	 * configurations.
>> -	 *
>> -	 * Also the HiSilicon ACC VF devices supported by this driver on
>> -	 * HiSilicon hardware platforms are integrated end point devices
>> -	 * and the platform lacks the capability to perform any PCIe P2P
>> -	 * between these devices.
>> -	 */
>> +	if (pf_qm->ver == QM_HW_V3) {
>> +		/*
>> +		 * ACC VF dev BAR2 region consists of both functional
>> register space
>> +		 * and migration control register space. For migration to
>> work, we
>> +		 * need access to both. Hence, we map the entire BAR2
>> region here.
>> +		 * But unnecessarily exposing the migration BAR region to
>> the Guest
>> +		 * has the potential to prevent/corrupt the Guest migration.
>> Hence,
>> +		 * we restrict access to the migration control space from
>> +		 * Guest(Please see mmap/ioctl/read/write override
>> functions).
>> +		 *
>> +		 * Please note that it is OK to expose the entire VF BAR if
>> migration
>> +		 * is not supported or required as this cannot affect the ACC
>> PF
>> +		 * configurations.
>> +		 *
>> +		 * Also the HiSilicon ACC VF devices supported by this driver
>> on
>> +		 * HiSilicon hardware platforms are integrated end point
>> devices
>> +		 * and the platform lacks the capability to perform any PCIe
>> P2P
>> +		 * between these devices.
>> +		 */
>>
>> -	vf_qm->io_base =
>> -		ioremap(pci_resource_start(vf_dev,
>> VFIO_PCI_BAR2_REGION_INDEX),
>> -			pci_resource_len(vf_dev,
>> VFIO_PCI_BAR2_REGION_INDEX));
>> -	if (!vf_qm->io_base)
>> -		return -EIO;
>> +		vf_qm->io_base =
>> +			ioremap(pci_resource_start(vf_dev,
>> VFIO_PCI_BAR2_REGION_INDEX),
>> +				pci_resource_len(vf_dev,
>> VFIO_PCI_BAR2_REGION_INDEX));
>> +		if (!vf_qm->io_base)
>> +			return -EIO;
>>
>> -	vf_qm->fun_type = QM_HW_VF;
>> +		vf_qm->fun_type = QM_HW_VF;
>> +		vf_qm->ver = pf_qm->ver;
>> +	} else {
>> +		/*
>> +		 * On hardware platforms greater than QM_HW_V3, the
>> migration function
>> +		 * register is placed in the BAR2 configuration region of the
>> PF,
>> +		 * and each VF device occupies 8KB of configuration space.
>> +		 */
>> +		vf_qm->io_base = pf_qm->io_base +
>> QM_MIG_REGION_OFFSET +
>> +				 hisi_acc_vdev->vf_id *
>> QM_MIG_REGION_SIZE;
>> +		vf_qm->fun_type = QM_HW_PF;
> 
> I don't think you can use the QM fun_type to distinguish this, because it is still a
> VF dev. Better to have another field to detect this.
>

The devices that explicitly support live migration have already been identified during probe,
and only versions >= QM_HW_V3 are supported.
Among these, only QM_HW_V3 uses the VF's BAR2 configuration space. The others use the PF's
configuration space. This difference is already distinguishable through the QM fun_type,
and both functional verification and testing are OK.

> Also I have another question. In the hisi_acc_vfio_pci_probe()  we currently
> look for pf_qm-> >= QM_HW_V3. This means current driver can get loaded
> on this new hardware, right? If so, I think we need to prevent that. And also
> we need to block any migration  attempt from existing host kernels to new
> ones.
> 

The current driver can be loaded on both old QM_HW_V3 hardware and new hardware platforms.
This is because they only differ in the io_base, while other functionalities are the same.
Additionally, the compatibility issue for live migration between old and new devices is handled.
Compatibility checks are performed during vf_qm_check_match --> vf_qm_version_check to
ensure normal functionality after migration.

Thanks,
Longfang.

> Thanks,
> Shameer
> 
>> +	}
>>  	vf_qm->pdev = vf_dev;
>>  	mutex_init(&vf_qm->mailbox_lock);
>>
>> @@ -1539,7 +1598,8 @@ static void hisi_acc_vfio_pci_close_device(struct
>> vfio_device *core_vdev)
>>  	hisi_acc_vf_disable_fds(hisi_acc_vdev);
>>  	mutex_lock(&hisi_acc_vdev->open_mutex);
>>  	hisi_acc_vdev->dev_opened = false;
>> -	iounmap(vf_qm->io_base);
>> +	if (vf_qm->ver == QM_HW_V3)
>> +		iounmap(vf_qm->io_base);
>>  	mutex_unlock(&hisi_acc_vdev->open_mutex);
>>  	vfio_pci_core_close_device(core_vdev);
>>  }
>> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>> index 91002ceeebc1..348f8bb5b42c 100644
>> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>> @@ -59,6 +59,13 @@
>>  #define ACC_DEV_MAGIC_V1	0XCDCDCDCDFEEDAACC
>>  #define ACC_DEV_MAGIC_V2	0xAACCFEEDDECADEDE
>>
>> +#define QM_MIG_REGION_OFFSET		0x180000
>> +#define QM_MIG_REGION_SIZE		0x2000
>> +
>> +#define QM_SUB_VERSION_ID		0x100210
>> +#define QM_EQC_PF_DW0			0x1c00
>> +#define QM_AEQC_PF_DW0			0x1c20
>> +
>>  struct acc_vf_data {
>>  #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
>>  	/* QM match information */
>> --
>> 2.24.0
> 
> .
> 

Re: [PATCH v5 3/3] migration: adapt to new migration configuration

[liulongfang]

On 2025/7/15 15:15, liulongfang wrote:
> On 2025/7/8 16:28, Shameerali Kolothum Thodi wrote:
>>
>>
>>> -----Original Message-----
>>> From: liulongfang <liulongfang@huawei.com>
>>> Sent: Monday, June 30, 2025 9:54 AM
>>> To: alex.williamson@redhat.com; jgg@nvidia.com; Shameerali Kolothum
>>> Thodi <shameerali.kolothum.thodi@huawei.com>; Jonathan Cameron
>>> <jonathan.cameron@huawei.com>
>>> Cc: kvm@vger.kernel.org; linux-kernel@vger.kernel.org;
>>> linuxarm@openeuler.org; liulongfang <liulongfang@huawei.com>
>>> Subject: [PATCH v5 3/3] migration: adapt to new migration configuration
>>>
>>> On new platforms greater than QM_HW_V3, the migration region has been
>>> relocated from the VF to the PF. The driver must also be modified
>>> accordingly to adapt to the new hardware device.
>>>
>>> Utilize the PF's I/O base directly on the new hardware platform,
>>> and no mmap operation is required. If it is on an old platform,
>>> the driver needs to be compatible with the old solution.
>>>
>>> Signed-off-by: Longfang Liu <liulongfang@huawei.com>
>>> ---
>>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    | 166 ++++++++++++------
>>>  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h    |   7 +
>>>  2 files changed, 120 insertions(+), 53 deletions(-)
>>>
>>> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>> index 1ddc9dbadb70..3aec3b92787f 100644
>>> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
>>> @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64
>>> *addr)
>>>  	return 0;
>>>  }
>>>
>>> +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>>> +			   struct acc_vf_data *vf_data)
>>> +{
>>> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
>>> +	struct device *dev = &qm->pdev->dev;
>>> +	u32 eqc_addr, aeqc_addr;
>>> +	int ret;
>>> +
>>> +	if (qm->ver == QM_HW_V3) {
>>> +		eqc_addr = QM_EQC_DW0;
>>> +		aeqc_addr = QM_AEQC_DW0;
>>> +	} else {
>>> +		eqc_addr = QM_EQC_PF_DW0;
>>> +		aeqc_addr = QM_AEQC_PF_DW0;
>>> +	}
>>> +
>>> +	/* QM_EQC_DW has 7 regs */
>>> +	ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
>>> +	if (ret) {
>>> +		dev_err(dev, "failed to read QM_EQC_DW\n");
>>> +		return ret;
>>> +	}
>>> +
>>> +	/* QM_AEQC_DW has 7 regs */
>>> +	ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
>>> +	if (ret) {
>>> +		dev_err(dev, "failed to read QM_AEQC_DW\n");
>>> +		return ret;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev,
>>> +			   struct acc_vf_data *vf_data)
>>> +{
>>> +	struct hisi_qm *qm = &hisi_acc_vdev->vf_qm;
>>> +	struct device *dev = &qm->pdev->dev;
>>> +	u32 eqc_addr, aeqc_addr;
>>> +	int ret;
>>> +
>>> +	if (qm->ver == QM_HW_V3) {
>>> +		eqc_addr = QM_EQC_DW0;
>>> +		aeqc_addr = QM_AEQC_DW0;
>>> +	} else {
>>> +		eqc_addr = QM_EQC_PF_DW0;
>>> +		aeqc_addr = QM_AEQC_PF_DW0;
>>> +	}
>>> +
>>> +	/* QM_EQC_DW has 7 regs */
>>> +	ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
>>> +	if (ret) {
>>> +		dev_err(dev, "failed to write QM_EQC_DW\n");
>>> +		return ret;
>>> +	}
>>> +
>>> +	/* QM_AEQC_DW has 7 regs */
>>> +	ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
>>> +	if (ret) {
>>> +		dev_err(dev, "failed to write QM_AEQC_DW\n");
>>> +		return ret;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>>  static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
>>>  {
>>>  	struct device *dev = &qm->pdev->dev;
>>> @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct
>>> acc_vf_data *vf_data)
>>>  		return ret;
>>>  	}
>>>
>>> -	/* QM_EQC_DW has 7 regs */
>>> -	ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
>>> -	if (ret) {
>>> -		dev_err(dev, "failed to read QM_EQC_DW\n");
>>> -		return ret;
>>> -	}
>>> -
>>> -	/* QM_AEQC_DW has 7 regs */
>>> -	ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw,
>>> 7);
>>> -	if (ret) {
>>> -		dev_err(dev, "failed to read QM_AEQC_DW\n");
>>> -		return ret;
>>> -	}
>>> -
>>>  	return 0;
>>>  }
>>>
>>> @@ -239,20 +291,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct
>>> acc_vf_data *vf_data)
>>>  		return ret;
>>>  	}
>>>
>>> -	/* QM_EQC_DW has 7 regs */
>>> -	ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
>>> -	if (ret) {
>>> -		dev_err(dev, "failed to write QM_EQC_DW\n");
>>> -		return ret;
>>> -	}
>>> -
>>> -	/* QM_AEQC_DW has 7 regs */
>>> -	ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw,
>>> 7);
>>> -	if (ret) {
>>> -		dev_err(dev, "failed to write QM_AEQC_DW\n");
>>> -		return ret;
>>> -	}
>>> -
>>>  	return 0;
>>>  }
>>>
>>> @@ -522,6 +560,10 @@ static int vf_qm_load_data(struct
>>> hisi_acc_vf_core_device *hisi_acc_vdev,
>>>  		return ret;
>>>  	}
>>>
>>> +	ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>>  	ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
>>>  	if (ret) {
>>>  		dev_err(dev, "set sqc failed\n");
>>> @@ -589,6 +631,10 @@ static int vf_qm_state_save(struct
>>> hisi_acc_vf_core_device *hisi_acc_vdev,
>>>  	vf_data->vf_qm_state = QM_READY;
>>>  	hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
>>>
>>> +	ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>>  	ret = vf_qm_read_data(vf_qm, vf_data);
>>>  	if (ret)
>>>  		return ret;
>>> @@ -1186,34 +1232,47 @@ static int hisi_acc_vf_qm_init(struct
>>> hisi_acc_vf_core_device *hisi_acc_vdev)
>>>  {
>>>  	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
>>>  	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
>>> +	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
>>>  	struct pci_dev *vf_dev = vdev->pdev;
>>>
>>> -	/*
>>> -	 * ACC VF dev BAR2 region consists of both functional register space
>>> -	 * and migration control register space. For migration to work, we
>>> -	 * need access to both. Hence, we map the entire BAR2 region here.
>>> -	 * But unnecessarily exposing the migration BAR region to the Guest
>>> -	 * has the potential to prevent/corrupt the Guest migration. Hence,
>>> -	 * we restrict access to the migration control space from
>>> -	 * Guest(Please see mmap/ioctl/read/write override functions).
>>> -	 *
>>> -	 * Please note that it is OK to expose the entire VF BAR if migration
>>> -	 * is not supported or required as this cannot affect the ACC PF
>>> -	 * configurations.
>>> -	 *
>>> -	 * Also the HiSilicon ACC VF devices supported by this driver on
>>> -	 * HiSilicon hardware platforms are integrated end point devices
>>> -	 * and the platform lacks the capability to perform any PCIe P2P
>>> -	 * between these devices.
>>> -	 */
>>> +	if (pf_qm->ver == QM_HW_V3) {
>>> +		/*
>>> +		 * ACC VF dev BAR2 region consists of both functional
>>> register space
>>> +		 * and migration control register space. For migration to
>>> work, we
>>> +		 * need access to both. Hence, we map the entire BAR2
>>> region here.
>>> +		 * But unnecessarily exposing the migration BAR region to
>>> the Guest
>>> +		 * has the potential to prevent/corrupt the Guest migration.
>>> Hence,
>>> +		 * we restrict access to the migration control space from
>>> +		 * Guest(Please see mmap/ioctl/read/write override
>>> functions).
>>> +		 *
>>> +		 * Please note that it is OK to expose the entire VF BAR if
>>> migration
>>> +		 * is not supported or required as this cannot affect the ACC
>>> PF
>>> +		 * configurations.
>>> +		 *
>>> +		 * Also the HiSilicon ACC VF devices supported by this driver
>>> on
>>> +		 * HiSilicon hardware platforms are integrated end point
>>> devices
>>> +		 * and the platform lacks the capability to perform any PCIe
>>> P2P
>>> +		 * between these devices.
>>> +		 */
>>>
>>> -	vf_qm->io_base =
>>> -		ioremap(pci_resource_start(vf_dev,
>>> VFIO_PCI_BAR2_REGION_INDEX),
>>> -			pci_resource_len(vf_dev,
>>> VFIO_PCI_BAR2_REGION_INDEX));
>>> -	if (!vf_qm->io_base)
>>> -		return -EIO;
>>> +		vf_qm->io_base =
>>> +			ioremap(pci_resource_start(vf_dev,
>>> VFIO_PCI_BAR2_REGION_INDEX),
>>> +				pci_resource_len(vf_dev,
>>> VFIO_PCI_BAR2_REGION_INDEX));
>>> +		if (!vf_qm->io_base)
>>> +			return -EIO;
>>>
>>> -	vf_qm->fun_type = QM_HW_VF;
>>> +		vf_qm->fun_type = QM_HW_VF;
>>> +		vf_qm->ver = pf_qm->ver;
>>> +	} else {
>>> +		/*
>>> +		 * On hardware platforms greater than QM_HW_V3, the
>>> migration function
>>> +		 * register is placed in the BAR2 configuration region of the
>>> PF,
>>> +		 * and each VF device occupies 8KB of configuration space.
>>> +		 */
>>> +		vf_qm->io_base = pf_qm->io_base +
>>> QM_MIG_REGION_OFFSET +
>>> +				 hisi_acc_vdev->vf_id *
>>> QM_MIG_REGION_SIZE;
>>> +		vf_qm->fun_type = QM_HW_PF;
>>
>> I don't think you can use the QM fun_type to distinguish this, because it is still a
>> VF dev. Better to have another field to detect this.
>>
>

Yes, I understand your point. The device is still a VF device, even though its configuration domain
is in the PF. I will modify it accordingly and assign it as QM_HW_VF.

Thanks.
Longfang.

> The devices that explicitly support live migration have already been identified during probe,
> and only versions >= QM_HW_V3 are supported.
> Among these, only QM_HW_V3 uses the VF's BAR2 configuration space. The others use the PF's
> configuration space. This difference is already distinguishable through the QM fun_type,
> and both functional verification and testing are OK.
> 
>> Also I have another question. In the hisi_acc_vfio_pci_probe()  we currently
>> look for pf_qm-> >= QM_HW_V3. This means current driver can get loaded
>> on this new hardware, right? If so, I think we need to prevent that. And also
>> we need to block any migration  attempt from existing host kernels to new
>> ones.
>>
> 
> The current driver can be loaded on both old QM_HW_V3 hardware and new hardware platforms.
> This is because they only differ in the io_base, while other functionalities are the same.
> Additionally, the compatibility issue for live migration between old and new devices is handled.
> Compatibility checks are performed during vf_qm_check_match --> vf_qm_version_check to
> ensure normal functionality after migration.
> 
> Thanks,
> Longfang.
> 
>> Thanks,
>> Shameer
>>
>>> +	}
>>>  	vf_qm->pdev = vf_dev;
>>>  	mutex_init(&vf_qm->mailbox_lock);
>>>
>>> @@ -1539,7 +1598,8 @@ static void hisi_acc_vfio_pci_close_device(struct
>>> vfio_device *core_vdev)
>>>  	hisi_acc_vf_disable_fds(hisi_acc_vdev);
>>>  	mutex_lock(&hisi_acc_vdev->open_mutex);
>>>  	hisi_acc_vdev->dev_opened = false;
>>> -	iounmap(vf_qm->io_base);
>>> +	if (vf_qm->ver == QM_HW_V3)
>>> +		iounmap(vf_qm->io_base);
>>>  	mutex_unlock(&hisi_acc_vdev->open_mutex);
>>>  	vfio_pci_core_close_device(core_vdev);
>>>  }
>>> diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>>> b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>>> index 91002ceeebc1..348f8bb5b42c 100644
>>> --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>>> +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
>>> @@ -59,6 +59,13 @@
>>>  #define ACC_DEV_MAGIC_V1	0XCDCDCDCDFEEDAACC
>>>  #define ACC_DEV_MAGIC_V2	0xAACCFEEDDECADEDE
>>>
>>> +#define QM_MIG_REGION_OFFSET		0x180000
>>> +#define QM_MIG_REGION_SIZE		0x2000
>>> +
>>> +#define QM_SUB_VERSION_ID		0x100210
>>> +#define QM_EQC_PF_DW0			0x1c00
>>> +#define QM_AEQC_PF_DW0			0x1c20
>>> +
>>>  struct acc_vf_data {
>>>  #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
>>>  	/* QM match information */
>>> --
>>> 2.24.0
>>
>> .
>>
> 
> .
>