[PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

This patch series implements live guest migration of a guest to which AP
devices have been passed through. To better comprehend this design, one has
to understand that VFIO AP mediated device is not used to provide userspace
with direct access to a device as is the case with other devices that use
the VFIO framework to pass them through to a guest. The sole purpose of the
VFIO AP mediated device is to manage an AP configuration for a guest. An AP
configuration is comprised of the AP adapter IDs (APID), AP queue 
indexes (APQI) and domain numbers of the control domains to which a guest
will be granted access. Once the VFIO AP mediated device is attached to a
guest, its AP configuration is set by the vfio_ap device driver. Once set,
all access to the AP devices is handled by the s390 Interpretive Execution
facility; in other words, the vfio_ap device driver plays no role in
providing direct access to the AP devices in the guest's AP configuration.

The only role that the vfio_ap device driver plays in the migration
process is to verify that the AP configuration for the source guest is
compatible with the AP configuration of the destination guest.
Incompatibility will result in a live guest migration failure.
In order to be compatible, the following requirements must be met:

1. The destination guest will be started with the same QEMU command line
   as the source guest, so the mediated device supplying the AP
   configuration on both guests must have the same name (UUID).

2. The AP configuration assigned via the VFIO AP mediated device on both
   guests must be compatible. As such, each AP configuration must meet
   the following requirements:

   * Both guests must have the same number of APQNs

   * Each APQN assigned to the source guest must also be assigned to the 
     destination guest

   * Each APQN assigned to both guests must reference an AP queue with the
     same hardware capabilities
     
Note: There is a forthcoming consumer of this series which will be a QEMU 
      patch series is entitled: 
      'hw/vfio/ap: Implement live guest migration of guests using AP 
      devices'

This design also adds a use case for enabling and disabling 
migration of guests to which AP devices have been passed through. To
facilitate this, a new read/write sysfs 'migratable' attribute is added to
the mediated device. This attribute specifies whether the vfio device is
migratable (1) or not (0). When the value of this attribute is changed, the
vfio_ap device driver will signal an eventfd to userspace. It is up to
userspace to respond to the change by enabling or disabling migration of
the guest to which the mediated device is attached. The operation will be
rejected with a 'Device or resource busy' message if a migration is in
progress.

Userspace must also have a means for retrieving the value of the sysfs
'migratable' attribute when the guest is started to initialize whether it
can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The 
struct vfio_device_info object passed to the ioctl will be extended with a
capability specifying the vfio device attributes. One of the attributes 
will contain the value of the mediated device's 'migratable' attribute.

Anthony Krowiak (23):
  s390/vfio-ap: Store queue hardware info when probed
  s390/vfio-ap: Provide access to queue objects and related info
  s390/vfio-ap: Add header file for xfer of vfio device caps to
    userspace
  MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
    maintainers
  s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
    migration of guest
  s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
  s390/vfio-ap: Signal event to enable/disable live guest migration
  s390/vfio-ap: Return value of sysfs migratable attribute from
    VFIO_DEVICE_GET_INFO ioctl
  s390/vfio-ap: Data structures for facilitating vfio device migration
  s390/vfio-ap: Initialize/release vfio device migration data
  s390-vfio-ap: Callback to set vfio device mig state during guest
    migration
  s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
  s390/vfio-ap: File ops called to save the vfio device migration state
  s390/vfio-ap: Transition device migration state from STOP to RESUMING
  s390/vfio-ap: File ops called to resume the vfio device migration
  s390/vfio-ap: Transition device migration state from RESUMING to STOP
  s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
  s390/vfio-ap: Transition device migration state from STOP to RUNNING
    and vice versa
  s390-vfio-ap: Callback to get the current vfio device migration state
  s390/vfio-ap: Callback to get the size of data to be migrated during
    guest migration
  s390/vfio-ap: Provide API to query whether migration is in progress
  s390/vfio-ap: Disallow blocking migration in progress
  s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst

 Documentation/arch/s390/vfio-ap.rst     |  339 +++++--
 MAINTAINERS                             |    1 +
 drivers/s390/crypto/Makefile            |    2 +-
 drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
 drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_ops.c       |  263 +++++-
 drivers/s390/crypto/vfio_ap_private.h   |   20 +
 include/uapi/linux/vfio.h               |    2 +
 include/uapi/linux/vfio_ap.h            |   34 +
 9 files changed, 1685 insertions(+), 111 deletions(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
 create mode 100644 include/uapi/linux/vfio_ap.h

-- 
2.52.0

[PATCH v1 01/24] fixup! KVM: s390: Remove non-atomic dat_crstep_xchg()

[Anthony Krowiak]

From: Claudio Imbrenda <imbrenda@linux.ibm.com>

Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
 arch/s390/kvm/gmap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/s390/kvm/gmap.h b/arch/s390/kvm/gmap.h
index 967a280b3235..19379d162777 100644
--- a/arch/s390/kvm/gmap.h
+++ b/arch/s390/kvm/gmap.h
@@ -198,7 +198,7 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio
 							 union crste oldcrste, union crste newcrste,
 							 gfn_t gfn, bool needs_lock)
 {
-	unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11);
+	unsigned long align = is_pmd(*crstep) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES;
 
 	lockdep_assert_held(&gmap->kvm->mmu_lock);
 	if (!needs_lock)
-- 
2.52.0

Re: [PATCH v1 01/24] fixup! KVM: s390: Remove non-atomic dat_crstep_xchg()

[Matthew Rosato]

On 3/25/26 5:00 PM, Anthony Krowiak wrote:
> From: Claudio Imbrenda <imbrenda@linux.ibm.com>
> 
> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>

Tony/Claudio, I suspect this was included on accident and will be
handled separately and dropped from this series?  I don't even see it
mentioned in the cover letter.

If intentional and intended to be part of this series, then please add a
proper commit message.

> ---
>  arch/s390/kvm/gmap.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/s390/kvm/gmap.h b/arch/s390/kvm/gmap.h
> index 967a280b3235..19379d162777 100644
> --- a/arch/s390/kvm/gmap.h
> +++ b/arch/s390/kvm/gmap.h
> @@ -198,7 +198,7 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio
>  							 union crste oldcrste, union crste newcrste,
>  							 gfn_t gfn, bool needs_lock)
>  {
> -	unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11);
> +	unsigned long align = is_pmd(*crstep) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES;
>  
>  	lockdep_assert_held(&gmap->kvm->mmu_lock);
>  	if (!needs_lock)

Re: [PATCH v1 01/24] fixup! KVM: s390: Remove non-atomic dat_crstep_xchg()

[Anthony Krowiak]

On 3/25/26 5:30 PM, Matthew Rosato wrote:
> On 3/25/26 5:00 PM, Anthony Krowiak wrote:
>> From: Claudio Imbrenda <imbrenda@linux.ibm.com>
>>
>> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> Tony/Claudio, I suspect this was included on accident and will be
> handled separately and dropped from this series?  I don't even see it
> mentioned in the cover letter.
>
> If intentional and intended to be part of this series, then please add a
> proper commit message.

This definitely does not belong in this series. I accidentally included it
when building the patch series.

>
>> ---
>>   arch/s390/kvm/gmap.h | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/kvm/gmap.h b/arch/s390/kvm/gmap.h
>> index 967a280b3235..19379d162777 100644
>> --- a/arch/s390/kvm/gmap.h
>> +++ b/arch/s390/kvm/gmap.h
>> @@ -198,7 +198,7 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio
>>   							 union crste oldcrste, union crste newcrste,
>>   							 gfn_t gfn, bool needs_lock)
>>   {
>> -	unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11);
>> +	unsigned long align = is_pmd(*crstep) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES;
>>   
>>   	lockdep_assert_held(&gmap->kvm->mmu_lock);
>>   	if (!needs_lock)

[PATCH v1 02/24] s390/vfio-ap: Store queue hardware info when probed

[Anthony Krowiak]

Store the output of the PQAP(TAPQ) command with the struct vfio_ap_queue
object when a queue device is probed. This data can be retrieved from the
object passed to the probe callback.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c     | 5 ++++-
 drivers/s390/crypto/vfio_ap_private.h | 1 +
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 44b3a1dcc1b3..ef3101f6865e 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -2419,6 +2419,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 	struct vfio_ap_queue *q;
 	DECLARE_BITMAP(apm_filtered, AP_DEVICES);
 	struct ap_matrix_mdev *matrix_mdev;
+	struct ap_queue *ap_queue;
 
 	ret = sysfs_create_group(&apdev->device.kobj, &vfio_queue_attr_group);
 	if (ret)
@@ -2430,8 +2431,10 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 		goto err_remove_group;
 	}
 
-	q->apqn = to_ap_queue(&apdev->device)->qid;
+	ap_queue = to_ap_queue(&apdev->device);
+	q->apqn = ap_queue->qid;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
+	memcpy(&q->hwinfo, &ap_queue->card->hwinfo, sizeof(q->hwinfo));
 	memset(&q->reset_status, 0, sizeof(q->reset_status));
 	INIT_WORK(&q->reset_work, apq_reset_check);
 	matrix_mdev = get_update_locks_by_apqn(q->apqn);
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 9bff666b0b35..39c85bab05c3 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -150,6 +150,7 @@ struct vfio_ap_queue {
 	struct list_head reset_qnode;
 	struct ap_queue_status reset_status;
 	struct work_struct reset_work;
+	struct ap_tapq_hwinfo hwinfo;
 };
 
 int vfio_ap_mdev_register(void);
-- 
2.52.0

[PATCH v1 03/24] s390/vfio-ap: Provide access to queue objects and related info

[Anthony Krowiak]

Provide the following:

1. A function that returns a struct vfio_ap_queue object representing a
   queue that is bound to the vfio_ap device driver and is assigned to an
   mdev

2. A function that returns the number of queues that are or will be passed
   through to a guest when the mdev is attached to a guest.

There is already a function for #1; however, it is a static function in
drivers/s390/crypto/vfio_ap_ops.c. The function will be defined in the
drivers/s390/crypto/vfio_ap_private.h file and made non-static. Note that
it would probably make sense to ensure that the mdevs_lock is held for
this function, so a call to lockdep_assert_held(&matrix_dev->mdevs_lock)
will be added to the function implementation.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c     | 20 +++++++++++++++++---
 drivers/s390/crypto/vfio_ap_private.h |  4 ++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index ef3101f6865e..cd85866b81a0 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -203,12 +203,12 @@ static inline void get_update_locks_for_queue(struct vfio_ap_queue *q)
  * Return: the pointer to the vfio_ap_queue struct representing the queue or
  *	   NULL if the queue is not assigned to @matrix_mdev
  */
-static struct vfio_ap_queue *vfio_ap_mdev_get_queue(
-					struct ap_matrix_mdev *matrix_mdev,
-					int apqn)
+struct vfio_ap_queue *vfio_ap_mdev_get_queue(struct ap_matrix_mdev *matrix_mdev, int apqn)
 {
 	struct vfio_ap_queue *q;
 
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
 	hash_for_each_possible(matrix_mdev->qtable.queues, q, mdev_qnode,
 			       apqn) {
 		if (q && q->apqn == apqn)
@@ -2098,6 +2098,20 @@ static void vfio_ap_mdev_request(struct vfio_device *vdev, unsigned int count)
 	release_update_locks_for_mdev(matrix_mdev);
 }
 
+int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix)
+{
+	unsigned long apid, apqi;
+	int num_queues = 0;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	for_each_set_bit_inv(apid, ap_matrix->apm, AP_DEVICES)
+		for_each_set_bit_inv(apqi, ap_matrix->aqm, AP_DOMAINS)
+			num_queues++;
+
+	return num_queues;
+}
+
 static int vfio_ap_mdev_get_device_info(unsigned long arg)
 {
 	unsigned long minsz;
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 39c85bab05c3..d66586825ef4 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -153,6 +153,10 @@ struct vfio_ap_queue {
 	struct ap_tapq_hwinfo hwinfo;
 };
 
+struct vfio_ap_queue *vfio_ap_mdev_get_queue(struct ap_matrix_mdev *matrix_mdev, int apqn);
+
+int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix);
+
 int vfio_ap_mdev_register(void);
 void vfio_ap_mdev_unregister(void);
 
-- 
2.52.0

[PATCH v1 04/24] s390/vfio-ap: Add header file for xfer of vfio device caps to userspace

[Anthony Krowiak]

Adds the vfio_ap.h file used to transfer vfio device capabilities to
userspace in response to the VFIO_DEVICE_GET_INFO ioctl. The device
capabilities includes an object specifying the vfio device attributes.
This object contains a flag indicating whether the vfio device is
migratable.

Note: The migratable flag could have been returned as a capability
      unto itself, but returning it as one attribute of a set of
      attributes allows for this capability to easily be extended
      to include additional attributes in the future if need be.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 include/uapi/linux/vfio_ap.h | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 include/uapi/linux/vfio_ap.h

diff --git a/include/uapi/linux/vfio_ap.h b/include/uapi/linux/vfio_ap.h
new file mode 100644
index 000000000000..9ee14f8649ed
--- /dev/null
+++ b/include/uapi/linux/vfio_ap.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Interfaces for vfio-ap
+ *
+ * Copyright IBM Corp. 2025
+ *
+ * Author(s): Anthony Krowiak <akrowiak@linux.ibm.com>
+ */
+
+#ifndef INCLUDE_UAPI_LINUX_VFIO_AP_H_
+#define INCLUDE_UAPI_LINUX_VFIO_AP_H_
+
+#include <linux/types.h>
+#include <linux/vfio.h>
+
+#define VFIO_DEVINFO_CAP_AP_ATTRS_VERSION	1
+#define VFIO_DEVINFO_CAP_AP_ATTRS_ID		1
+
+/**
+ * struct vfio_device_info_cap_ap_attrs - vfio device capability object used
+ *					  supply the device attributes to
+ *					  userspace.
+ *
+ * @header:	the header used to extend the struct vfio_device_info object
+ *		to provide the vfio device attributes to userspace
+ * @migratable:	indicates whether the vfio device is migratable (1) or
+ *		not (0).
+ */
+struct vfio_device_info_cap_ap_attrs {
+	struct	vfio_info_cap_header	header;
+	int				migratable;
+};
+
+#endif /* INCLUDE_UAPI_LINUX_VFIO_AP_H_ */
-- 
2.52.0

[PATCH v1 05/24] MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER maintainers

[Anthony Krowiak]

Add new header file include/uapi/linux/vfio_ap.h for the S390 VFIO AP
DRIVER maintainers.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7d10988cbc62..b84896648860 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -23339,6 +23339,7 @@ L:	linux-s390@vger.kernel.org
 S:	Supported
 F:	Documentation/arch/s390/vfio-ap*
 F:	drivers/s390/crypto/vfio_ap*
+F:	include/uapi/linux/vfio_ap.h
 
 S390 VFIO-CCW DRIVER
 M:	Eric Farman <farman@linux.ibm.com>
-- 
2.52.0

[PATCH v1 06/24] s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable migration of guest

[Anthony Krowiak]

This patch creates a new R/W sysfs attribute of the vfio-ap mediated device
that will ultimately be used to enable and disable migration of the guest
to which the mediated device is attached. The value for this attribute is
an int to be stored with the mediated device. A value of 1 indicates the
vfio device can be migrated, and a value of 0 indicates migration is
blocked. The initial value will be set to 0.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c     | 46 +++++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_private.h |  3 ++
 2 files changed, 49 insertions(+)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index cd85866b81a0..97ec1f2fdbd1 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -32,6 +32,8 @@
 
 #define AP_RESET_INTERVAL		20	/* Reset sleep interval (20ms)		*/
 
+#define VFIO_AP_DEVICE_MIGRATABLE_DEFAULT 0
+
 static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev);
 static int vfio_ap_mdev_reset_qlist(struct list_head *qlist);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
@@ -802,6 +804,7 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 		goto err_put_vdev;
 	matrix_mdev->req_trigger = NULL;
 	matrix_mdev->cfg_chg_trigger = NULL;
+	matrix_mdev->migratable = VFIO_AP_DEVICE_MIGRATABLE_DEFAULT;
 	dev_set_drvdata(&mdev->dev, matrix_mdev);
 	mutex_lock(&matrix_dev->mdevs_lock);
 	list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
@@ -1785,6 +1788,48 @@ static ssize_t ap_config_store(struct device *dev, struct device_attribute *attr
 }
 static DEVICE_ATTR_RW(ap_config);
 
+static ssize_t migratable_show(struct device *dev,
+			       struct device_attribute *attr, char *buf)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	size_t nchars;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = dev_get_drvdata(dev);
+	nchars = sysfs_emit(buf, "%d\n", matrix_mdev->migratable);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return nchars;
+}
+
+static ssize_t migratable_store(struct device *dev, struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	int migratable;
+	int ret;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = dev_get_drvdata(dev);
+
+	ret = kstrtoint(buf, 10, &migratable);
+	if (ret)
+		goto out_unlock;
+
+	if (migratable != 0 && migratable != 1) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	matrix_mdev->migratable = migratable;
+	ret = count;
+out_unlock:
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return ret;
+}
+static DEVICE_ATTR_RW(migratable);
+
 static struct attribute *vfio_ap_mdev_attrs[] = {
 	&dev_attr_assign_adapter.attr,
 	&dev_attr_unassign_adapter.attr,
@@ -1796,6 +1841,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = {
 	&dev_attr_control_domains.attr,
 	&dev_attr_matrix.attr,
 	&dev_attr_guest_matrix.attr,
+	&dev_attr_migratable.attr,
 	NULL,
 };
 
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index d66586825ef4..b75299c9c1d0 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -104,6 +104,8 @@ struct ap_queue_table {
  *		PQAP(AQIC) instruction.
  * @mdev:	the mediated device
  * @qtable:	table of queues (struct vfio_ap_queue) assigned to the mdev
+ * @migratable:	an int value indicating whether the vfio device can be
+ *		migrated (1) or not (0)
  * @req_trigger eventfd ctx for signaling userspace to return a device
  * @cfg_chg_trigger eventfd ctx to signal AP config changed to userspace
  * @apm_add:	bitmap of APIDs added to the host's AP configuration
@@ -120,6 +122,7 @@ struct ap_matrix_mdev {
 	crypto_hook pqap_hook;
 	struct mdev_device *mdev;
 	struct ap_queue_table qtable;
+	int    migratable;
 	struct eventfd_ctx *req_trigger;
 	struct eventfd_ctx *cfg_chg_trigger;
 	DECLARE_BITMAP(apm_add, AP_DEVICES);
-- 
2.52.0

[PATCH v1 07/24] s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute

[Anthony Krowiak]

Add the 'migratable' feature to the features provided by the sysfs
/sys/devices/vfio_ap/matrix/feature attribute to indicate that migration
of vfio devices is supported.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_drv.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index fd7394d81880..d2fc3598cd24 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -26,9 +26,11 @@ MODULE_LICENSE("GPL v2");
 struct ap_matrix_dev *matrix_dev;
 debug_info_t *vfio_ap_dbf_info;
 
+#define FEATURES "guest_matrix hotplug ap_config migratable"
+
 static ssize_t features_show(struct device *dev, struct device_attribute *attr, char *buf)
 {
-	return sysfs_emit(buf, "guest_matrix hotplug ap_config\n");
+	return sysfs_emit(buf, "%s\n", FEATURES);
 }
 static DEVICE_ATTR_RO(features);
 
-- 
2.52.0

[PATCH v1 08/24] s390/vfio-ap: Signal event to enable/disable live guest migration

[Anthony Krowiak]

Defines two new eventfd contexts for signaling userspace to enable and
disable vfio device migration. The IRQ indexes defined for these events are
defined in an enum in the include/uapi/linux/vfio.h header file:

VFIO_AP_ENABLE_MIG_IRQ_INDEX:  enable vfio device migration
VFIO_AP_DISABLE_MIG_IRQ_INDEX: disable vfio device migration

Information about those IRQs can be retrieved from userspace via the
VFIO_DEVICE_GET_IRQ_INFO ioctl. These events are initialized by the vfio_ap
device driver via the in response to the VFIO_DEVICE_SET_IRQ_INFO ioctl.
The eventfd for these IRQs will be signaled whenever the sysfs migratable
attribute of the mediated device is changed.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c     | 70 +++++++++++++++++++++++++--
 drivers/s390/crypto/vfio_ap_private.h |  6 +++
 include/uapi/linux/vfio.h             |  2 +
 3 files changed, 75 insertions(+), 3 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 97ec1f2fdbd1..213832263dc9 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -658,6 +658,18 @@ static void signal_guest_ap_cfg_changed(struct ap_matrix_mdev *matrix_mdev)
 		eventfd_signal(matrix_mdev->cfg_chg_trigger);
 }
 
+static void signal_enable_migration(struct ap_matrix_mdev *matrix_mdev)
+{
+	if (matrix_mdev->enable_mig_trigger)
+		eventfd_signal(matrix_mdev->enable_mig_trigger);
+}
+
+static void signal_disable_migration(struct ap_matrix_mdev *matrix_mdev)
+{
+	if (matrix_mdev->disable_mig_trigger)
+		eventfd_signal(matrix_mdev->disable_mig_trigger);
+}
+
 static void vfio_ap_mdev_update_guest_apcb(struct ap_matrix_mdev *matrix_mdev)
 {
 	if (matrix_mdev->kvm) {
@@ -805,6 +817,8 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 	matrix_mdev->req_trigger = NULL;
 	matrix_mdev->cfg_chg_trigger = NULL;
 	matrix_mdev->migratable = VFIO_AP_DEVICE_MIGRATABLE_DEFAULT;
+	matrix_mdev->enable_mig_trigger = NULL;
+	matrix_mdev->disable_mig_trigger = NULL;
 	dev_set_drvdata(&mdev->dev, matrix_mdev);
 	mutex_lock(&matrix_dev->mdevs_lock);
 	list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
@@ -1821,7 +1835,18 @@ static ssize_t migratable_store(struct device *dev, struct device_attribute *att
 		goto out_unlock;
 	}
 
+	if (matrix_mdev->migratable == migratable) {
+		ret = count;
+		goto out_unlock;
+	}
+
 	matrix_mdev->migratable = migratable;
+
+	if (matrix_mdev->migratable)
+		signal_enable_migration(matrix_mdev);
+	else
+		signal_disable_migration(matrix_mdev);
+
 	ret = count;
 out_unlock:
 	mutex_unlock(&matrix_dev->mdevs_lock);
@@ -2193,10 +2218,9 @@ static ssize_t vfio_ap_get_irq_info(unsigned long arg)
 
 	switch (info.index) {
 	case VFIO_AP_REQ_IRQ_INDEX:
-		info.count = 1;
-		info.flags = VFIO_IRQ_INFO_EVENTFD;
-		break;
 	case VFIO_AP_CFG_CHG_IRQ_INDEX:
+	case VFIO_AP_ENABLE_MIG_IRQ_INDEX:
+	case VFIO_AP_DISABLE_MIG_IRQ_INDEX:
 		info.count = 1;
 		info.flags = VFIO_IRQ_INFO_EVENTFD;
 		break;
@@ -2296,6 +2320,40 @@ static int vfio_ap_set_cfg_change_irq(struct ap_matrix_mdev *matrix_mdev, unsign
 	return 0;
 }
 
+static int vfio_ap_set_mig_irq(struct eventfd_ctx **mig_trigger,
+			       unsigned long arg)
+{
+	s32 fd;
+	void __user *data;
+	unsigned long minsz;
+	struct eventfd_ctx *mig_ctx;
+
+	minsz = offsetofend(struct vfio_irq_set, count);
+	data = (void __user *)(arg + minsz);
+
+	if (get_user(fd, (s32 __user *)data))
+		return -EFAULT;
+
+	if (fd == -1) {
+		if (*mig_trigger)
+			eventfd_ctx_put(*mig_trigger);
+		*mig_trigger = NULL;
+	} else if (fd >= 0) {
+		mig_ctx = eventfd_ctx_fdget(fd);
+		if (IS_ERR(mig_ctx))
+			return PTR_ERR(mig_ctx);
+
+		if (*mig_trigger)
+			eventfd_ctx_put(*mig_trigger);
+
+		*mig_trigger = mig_ctx;
+	} else {
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static int vfio_ap_set_irqs(struct ap_matrix_mdev *matrix_mdev,
 			    unsigned long arg)
 {
@@ -2313,6 +2371,12 @@ static int vfio_ap_set_irqs(struct ap_matrix_mdev *matrix_mdev,
 			return vfio_ap_set_request_irq(matrix_mdev, arg);
 		case VFIO_AP_CFG_CHG_IRQ_INDEX:
 			return vfio_ap_set_cfg_change_irq(matrix_mdev, arg);
+		case VFIO_AP_ENABLE_MIG_IRQ_INDEX:
+			return vfio_ap_set_mig_irq(&matrix_mdev->enable_mig_trigger,
+						   arg);
+		case VFIO_AP_DISABLE_MIG_IRQ_INDEX:
+			return vfio_ap_set_mig_irq(&matrix_mdev->disable_mig_trigger,
+						   arg);
 		default:
 			return -EINVAL;
 		}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index b75299c9c1d0..514de4a73678 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -108,6 +108,10 @@ struct ap_queue_table {
  *		migrated (1) or not (0)
  * @req_trigger eventfd ctx for signaling userspace to return a device
  * @cfg_chg_trigger eventfd ctx to signal AP config changed to userspace
+ * @enable_mig_trigger:  eventfd ctx to signal userspace to enable vfio device
+ *			 migration
+ * @disable_mig_trigger: eventfd ctx to signal userspace to disable vfio device
+ *			 migration
  * @apm_add:	bitmap of APIDs added to the host's AP configuration
  * @aqm_add:	bitmap of APQIs added to the host's AP configuration
  * @adm_add:	bitmap of control domain numbers added to the host's AP
@@ -125,6 +129,8 @@ struct ap_matrix_mdev {
 	int    migratable;
 	struct eventfd_ctx *req_trigger;
 	struct eventfd_ctx *cfg_chg_trigger;
+	struct eventfd_ctx *enable_mig_trigger;
+	struct eventfd_ctx *disable_mig_trigger;
 	DECLARE_BITMAP(apm_add, AP_DEVICES);
 	DECLARE_BITMAP(aqm_add, AP_DOMAINS);
 	DECLARE_BITMAP(adm_add, AP_DOMAINS);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index bb7b89330d35..c661b8b22ead 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -673,6 +673,8 @@ enum {
 enum {
 	VFIO_AP_REQ_IRQ_INDEX,
 	VFIO_AP_CFG_CHG_IRQ_INDEX,
+	VFIO_AP_ENABLE_MIG_IRQ_INDEX,
+	VFIO_AP_DISABLE_MIG_IRQ_INDEX,
 	VFIO_AP_NUM_IRQS
 };
 
-- 
2.52.0

[PATCH v1 09/24] s390/vfio-ap: Return value of sysfs migratable attribute from VFIO_DEVICE_GET_INFO ioctl

[Anthony Krowiak]

Extends the 'vfio_device_info' object passed to the VFIO_DEVICE_GET_INFO
ioctl to include a vfio device capability specifying the value of the
mdev's sysfs 'migratable' attribute. This value may be used to enable or
block live guest migration when the vfio device is realized in userspace.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c | 75 ++++++++++++++++++++++++++++++-
 1 file changed, 73 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 213832263dc9..d487628027c7 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -19,6 +19,7 @@
 #include <linux/uuid.h>
 #include <asm/kvm.h>
 #include <asm/zcrypt.h>
+#include <uapi/linux/vfio_ap.h>
 
 #include "vfio_ap_private.h"
 #include "vfio_ap_debug.h"
@@ -2183,11 +2184,75 @@ int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix)
 	return num_queues;
 }
 
-static int vfio_ap_mdev_get_device_info(unsigned long arg)
+static int vfio_ap_add_caps_to_dev_info(struct vfio_device_info *dev_info,
+					unsigned long arg,
+					struct vfio_info_cap *caps)
 {
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	if (caps->size) {
+		dev_info->flags |= VFIO_DEVICE_FLAGS_CAPS;
+		if (dev_info->argsz < sizeof(*dev_info) + caps->size) {
+			dev_info->argsz = sizeof(*dev_info) + caps->size;
+		} else {
+			vfio_info_cap_shift(caps, sizeof(*dev_info));
+
+			if (copy_to_user((void __user *)arg + sizeof(*dev_info),
+					 caps->buf, caps->size)) {
+				kfree(caps->buf);
+				return -EFAULT;
+			}
+
+			dev_info->cap_offset = sizeof(*dev_info);
+		}
+
+		kfree(caps->buf);
+	}
+
+	return 0;
+}
+
+static int vfio_ap_add_attrs_cap_to_caps(struct ap_matrix_mdev *matrix_mdev,
+					 struct vfio_info_cap *caps)
+{
+	struct vfio_device_info_cap_ap_attrs cap;
+
+	lockdep_assert_held(matrix_dev->mdevs_lock);
+	cap.migratable = matrix_mdev->migratable;
+	cap.header.id = VFIO_DEVINFO_CAP_AP_ATTRS_ID;
+	cap.header.version = VFIO_DEVINFO_CAP_AP_ATTRS_VERSION;
+
+	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
+}
+
+static int vfio_ap_add_caps(struct ap_matrix_mdev *matrix_mdev,
+			    unsigned long arg,
+			    struct vfio_device_info *dev_info)
+{
+	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
+	int ret = 0;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	ret = vfio_ap_add_attrs_cap_to_caps(matrix_mdev, &caps);
+	if (ret)
+		return ret;
+
+	ret = vfio_ap_add_caps_to_dev_info(dev_info, arg, &caps);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int vfio_ap_mdev_get_device_info(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long arg)
+{
+	int ret;
 	unsigned long minsz;
 	struct vfio_device_info info;
 
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
 	minsz = offsetofend(struct vfio_device_info, num_irqs);
 
 	if (copy_from_user(&info, (void __user *)arg, minsz))
@@ -2200,6 +2265,12 @@ static int vfio_ap_mdev_get_device_info(unsigned long arg)
 	info.num_regions = 0;
 	info.num_irqs = VFIO_AP_NUM_IRQS;
 
+	ret = vfio_ap_add_caps(matrix_mdev, arg, &info);
+	if (ret)
+		return ret;
+
+	minsz = min_t(size_t, info.argsz, sizeof(info));
+
 	return copy_to_user((void __user *)arg, &info, minsz) ? -EFAULT : 0;
 }
 
@@ -2395,7 +2466,7 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,
 	mutex_lock(&matrix_dev->mdevs_lock);
 	switch (cmd) {
 	case VFIO_DEVICE_GET_INFO:
-		ret = vfio_ap_mdev_get_device_info(arg);
+		ret = vfio_ap_mdev_get_device_info(matrix_mdev, arg);
 		break;
 	case VFIO_DEVICE_RESET:
 		ret = vfio_ap_mdev_reset_queues(matrix_mdev);
-- 
2.52.0

[PATCH v1 10/24] s390/vfio-ap: Data structures for facilitating vfio device migration

[Anthony Krowiak]

Creates the data structures used to facilitate state transitions during
vfio device migration.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/Makefile            |  2 +-
 drivers/s390/crypto/vfio_ap_migration.c | 68 +++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_private.h   |  2 +
 3 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_migration.c

diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index e83c6603c858..20f29184825a 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -34,5 +34,5 @@ pkey-uv-objs := pkey_uv.o
 obj-$(CONFIG_PKEY_UV) += pkey-uv.o
 
 # adjunct processor matrix
-vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o
+vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o vfio_ap_migration.o
 obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
new file mode 100644
index 000000000000..2c5978dde61d
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Drives vfio_ap mdev migration.
+ *
+ * Copyright IBM Corp. 2025
+ */
+#include "uapi/linux/vfio_ap.h"
+
+#include "vfio_ap_private.h"
+
+/**
+ * vfio_ap_migration_data - the data needed to migrate a guest with pass-through
+ *			    access to AP devices
+ *
+ * @mig_state:		the current migration state
+ * @resuming_migf:	the object used to resume the target guest
+ * @saving_migf:	the object used to save the state of the source guest
+ */
+struct vfio_ap_migration_data {
+	enum vfio_device_mig_state	mig_state;
+	struct vfio_ap_migration_file	*resuming_migf;
+	struct vfio_ap_migration_file	*saving_migf;
+};
+
+/**
+ * vfio_ap_queue_info - the information for an AP queue
+ *
+ * @data: contains the queue information returned in GR2 from the PQAP(TAPQ)
+ *	  command
+ * @apqn: the APQN of the queue
+ */
+struct vfio_ap_queue_info {
+	u64 data;
+	u16 apqn;
+};
+
+/**
+ * vfio_ap_config - the guest's AP configuration
+ *
+ * @num_queues:	the number of queues passed through to the guest
+ * @qinfo:	an array of vfio_ap_queue_info objects, each specifying the
+ *		queue information for a queue passed through to the guest
+ */
+struct vfio_ap_config {
+	unsigned int			num_queues;
+	struct vfio_ap_queue_info	qinfo[];
+};
+
+/**
+ * vfio_ap_migration_file - object used to facilitate migration of a guest with
+ *			    pass-through access to AP devices
+ *
+ * @matrix_mdev: the mediated device attached to the guest being migrated
+ * @filp:	 the file used to facilitate communication between userspace
+ *		 and the vfio_ap device driver during a particular phase of
+ *		 the migration
+ * @disabled:	 boolean value indicating whether this object is disabled (true)
+ *		 or not (false)
+ * @ap_config:	 the information for each queue passed through to a guest
+ * @config_sz:	 the size of @ap_config when filled with queue information
+ */
+struct vfio_ap_migration_file {
+	struct ap_matrix_mdev	*matrix_mdev;
+	struct file		*filp;
+	bool			disabled;
+	struct vfio_ap_config	*ap_config;
+	size_t			config_sz;
+};
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 514de4a73678..24db7d32e41c 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -116,6 +116,7 @@ struct ap_queue_table {
  * @aqm_add:	bitmap of APQIs added to the host's AP configuration
  * @adm_add:	bitmap of control domain numbers added to the host's AP
  *		configuration
+ * @mig_data:  vfio device migration data
  */
 struct ap_matrix_mdev {
 	struct vfio_device vdev;
@@ -134,6 +135,7 @@ struct ap_matrix_mdev {
 	DECLARE_BITMAP(apm_add, AP_DEVICES);
 	DECLARE_BITMAP(aqm_add, AP_DOMAINS);
 	DECLARE_BITMAP(adm_add, AP_DOMAINS);
+	void *mig_data;
 };
 
 /**
-- 
2.52.0

[PATCH v1 11/24] s390/vfio-ap: Initialize/release vfio device migration data

[Anthony Krowiak]

Provides the functions that initialize and release the data structures
used during live guest migration. These functions should
be called when the mdev fd is opened and closed respectively. The function
to release the data structures shall also be invoked when the mdev
is released.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 94 +++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_ops.c       | 41 +++++++++--
 drivers/s390/crypto/vfio_ap_private.h   |  3 +
 3 files changed, 131 insertions(+), 7 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 2c5978dde61d..76cbbe1adb7a 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -4,6 +4,7 @@
  *
  * Copyright IBM Corp. 2025
  */
+#include <linux/file.h>
 #include "uapi/linux/vfio_ap.h"
 
 #include "vfio_ap_private.h"
@@ -66,3 +67,96 @@ struct vfio_ap_migration_file {
 	struct vfio_ap_config	*ap_config;
 	size_t			config_sz;
 };
+
+static void vfio_ap_disable_file(struct vfio_ap_migration_file *migf)
+{
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	migf->matrix_mdev = NULL;
+	migf->disabled = true;
+	migf->filp->f_pos = 0;
+}
+
+static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	if (mig_data->resuming_migf) {
+		vfio_ap_disable_file(mig_data->resuming_migf);
+		fput(mig_data->resuming_migf->filp);
+		mig_data->resuming_migf = NULL;
+	}
+
+	if (mig_data->saving_migf) {
+		vfio_ap_disable_file(mig_data->saving_migf);
+		fput(mig_data->saving_migf->filp);
+		mig_data->saving_migf = NULL;
+	}
+}
+
+static struct file *vfio_ap_set_state(struct vfio_device *vdev,
+				      enum vfio_device_mig_state  new_state)
+{
+	return NULL;
+}
+
+static int vfio_ap_get_state(struct vfio_device *vdev,
+			     enum vfio_device_mig_state  *current_state)
+{
+	return -EOPNOTSUPP;
+}
+
+static int vfio_ap_get_data_size(struct vfio_device *vdev,
+				 unsigned long *stop_copy_length)
+{
+	return -EOPNOTSUPP;
+}
+
+static const struct vfio_migration_ops vfio_ap_migration_ops = {
+	.migration_set_state = vfio_ap_set_state,
+	.migration_get_state = vfio_ap_get_state,
+	.migration_get_data_size = vfio_ap_get_data_size,
+};
+
+/**
+ * vfio_ap_init_migration_data - initialize migration data and functions
+ *
+ * @matrix_mdev: pointer to object containing the mdev state
+ *
+ * Return: zero if initialization is successful; otherwise, returns a error.
+ */
+int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	mig_data = kzalloc_obj(struct vfio_ap_migration_data, GFP_KERNEL);
+	if (!mig_data)
+		return -ENOMEM;
+
+	mig_data->mig_state = VFIO_DEVICE_STATE_STOP;
+	matrix_mdev->vdev.migration_flags = VFIO_MIGRATION_STOP_COPY;
+	matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops;
+	matrix_mdev->mig_data = mig_data;
+
+	return 0;
+}
+
+/**
+ * vfio_ap_release_migration_data: reclaim private migration data
+ *
+ * @vdev: pointer to the mdev
+ */
+void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev)
+{
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	if (!matrix_mdev->mig_data)
+		return;
+
+	vfio_ap_release_mig_files(matrix_mdev);
+	kfree(matrix_mdev->mig_data);
+	matrix_mdev->mig_data = NULL;
+}
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index d487628027c7..1ff9994bd7ec 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -790,18 +790,30 @@ static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
 
 static int vfio_ap_mdev_init_dev(struct vfio_device *vdev)
 {
-	struct ap_matrix_mdev *matrix_mdev =
-		container_of(vdev, struct ap_matrix_mdev, vdev);
+	struct ap_matrix_mdev *matrix_mdev;
 
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
 	matrix_mdev->mdev = to_mdev_device(vdev->dev);
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
 	matrix_mdev->pqap_hook = handle_pqap;
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
 	hash_init(matrix_mdev->qtable.queues);
+	mutex_unlock(&matrix_dev->mdevs_lock);
 
 	return 0;
 }
 
+static void vfio_ap_mdev_release_dev(struct vfio_device *vdev)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	vfio_ap_release_migration_data(matrix_mdev);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+}
+
 static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 {
 	struct ap_matrix_mdev *matrix_mdev;
@@ -2124,19 +2136,33 @@ static int vfio_ap_mdev_reset_qlist(struct list_head *qlist)
 
 static int vfio_ap_mdev_open_device(struct vfio_device *vdev)
 {
-	struct ap_matrix_mdev *matrix_mdev =
-		container_of(vdev, struct ap_matrix_mdev, vdev);
+	struct ap_matrix_mdev *matrix_mdev;
+	int ret;
 
 	if (!vdev->kvm)
 		return -EINVAL;
 
-	return vfio_ap_mdev_set_kvm(matrix_mdev, vdev->kvm);
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	ret = vfio_ap_init_migration_data(matrix_mdev);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	if (ret)
+		return ret;
+
+	ret = vfio_ap_mdev_set_kvm(matrix_mdev, vdev->kvm);
+
+	return ret;
 }
 
 static void vfio_ap_mdev_close_device(struct vfio_device *vdev)
 {
-	struct ap_matrix_mdev *matrix_mdev =
-		container_of(vdev, struct ap_matrix_mdev, vdev);
+	struct ap_matrix_mdev *matrix_mdev;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	vfio_ap_release_migration_data(matrix_mdev);
+	mutex_unlock(&matrix_dev->mdevs_lock);
 
 	vfio_ap_mdev_unset_kvm(matrix_mdev);
 }
@@ -2555,6 +2581,7 @@ static const struct attribute_group vfio_queue_attr_group = {
 
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops = {
 	.init = vfio_ap_mdev_init_dev,
+	.release = vfio_ap_mdev_release_dev,
 	.open_device = vfio_ap_mdev_open_device,
 	.close_device = vfio_ap_mdev_close_device,
 	.ioctl = vfio_ap_mdev_ioctl,
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 24db7d32e41c..cad49b509f7a 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -181,4 +181,7 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
 void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
 			      struct ap_config_info *old_config_info);
 
+int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev);
+void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev);
+
 #endif /* _VFIO_AP_PRIVATE_H_ */
-- 
2.52.0

[PATCH v1 12/24] s390-vfio-ap: Callback to set vfio device mig state during guest migration

[Anthony Krowiak]

Implements the callback that sets a new migration state of a vfio device
during live migration of guests with pass-through access to AP devices.
This callback is mandatory for VFIO_DEVICE_FEATURE_MIGRATION
support.

The function pointer for this callback is specified via the
'migration_set_state' field of the 'vfio_migration_ops' structure
which is stored with the VFIO device when the 'vfio_device'
structure representing the mediated device is initialized.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 87 ++++++++++++++++++++++++-
 1 file changed, 86 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 76cbbe1adb7a..a8b109bc5cf9 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -96,10 +96,95 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
 	}
 }
 
+static struct file *
+vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
+			    enum vfio_device_mig_state new_state)
+{
+	struct vfio_ap_migration_data *mig_data;
+	enum vfio_device_mig_state cur_state;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+	cur_state = mig_data->mig_state;
+	dev_dbg(matrix_mdev->vdev.dev, "%s: %d -> %d\n", __func__, cur_state,
+		new_state);
+
+	if (cur_state == VFIO_DEVICE_STATE_STOP &&
+	    new_state == VFIO_DEVICE_STATE_STOP_COPY) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if (cur_state == VFIO_DEVICE_STATE_STOP &&
+	    new_state == VFIO_DEVICE_STATE_RESUMING) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
+	    new_state == VFIO_DEVICE_STATE_STOP) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
+	    new_state == VFIO_DEVICE_STATE_STOP) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if ((cur_state == VFIO_DEVICE_STATE_STOP &&
+	     new_state == VFIO_DEVICE_STATE_RUNNING) ||
+	    (cur_state == VFIO_DEVICE_STATE_RUNNING &&
+	     new_state == VFIO_DEVICE_STATE_STOP)) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	/* vfio_mig_get_next_state() does not use arcs other than the above */
+	WARN_ON(true);
+
+	return ERR_PTR(-EINVAL);
+}
+
 static struct file *vfio_ap_set_state(struct vfio_device *vdev,
 				      enum vfio_device_mig_state  new_state)
 {
-	return NULL;
+	int ret;
+	struct file *filp = NULL;
+	struct ap_matrix_mdev *matrix_mdev;
+	enum vfio_device_mig_state next_state;
+	struct vfio_ap_migration_data *mig_data;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	mig_data = matrix_mdev->mig_data;
+	dev_dbg(vdev->dev, "%s -> %d\n", __func__, new_state);
+
+	while (mig_data->mig_state != new_state) {
+		ret = vfio_mig_get_next_state(vdev, mig_data->mig_state,
+					      new_state, &next_state);
+		if (ret) {
+			filp = ERR_PTR(ret);
+			break;
+		}
+
+		filp = vfio_ap_transition_to_state(matrix_mdev, next_state);
+		if (IS_ERR(filp))
+			break;
+
+		mig_data->mig_state = next_state;
+
+		if (WARN_ON(filp && new_state != next_state)) {
+			fput(filp);
+			filp = ERR_PTR(-EINVAL);
+			break;
+		}
+	}
+
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return filp;
 }
 
 static int vfio_ap_get_state(struct vfio_device *vdev,
-- 
2.52.0

[PATCH v1 13/24] s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY

[Anthony Krowiak]

The transition of the guest migration state from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_STOP_COPY begins the process of saving the
vfio device state. The vfio device state is comprised of the guest's
AP configuration which specifies the adapters, domains and control
domains to which the guest will be given access.

To begin the process of saving the vfio device state, the VFIO AP device
driver will:

1. Allocate an object in which to store the guest's AP configuration
   information comprised of the APQN and hardware information for each
   queue passed through to the source guest.

2. Opens a file stream that will be used to read the AP configuration
   stored with the object created in #1. The AP configuration will be read
   from this file stream when the guest is resumed on the target system to
   verify that the AP configuration of the source and target guests are
   compatible.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 166 +++++++++++++++++++++++-
 1 file changed, 164 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index a8b109bc5cf9..ecc9f0f6819e 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -4,6 +4,7 @@
  *
  * Copyright IBM Corp. 2025
  */
+#include <linux/anon_inodes.h>
 #include <linux/file.h>
 #include "uapi/linux/vfio_ap.h"
 
@@ -96,12 +97,160 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
 	}
 }
 
+static ssize_t vfio_ap_save_read(struct file *, char __user *, size_t, loff_t *)
+{
+	/* TODO */
+	return -EOPNOTSUPP;
+}
+
+static int vfio_ap_release_migf(struct inode *, struct file *)
+{
+	/* TODO */
+	return -EOPNOTSUPP;
+}
+
+static const struct file_operations vfio_ap_save_fops = {
+	.owner = THIS_MODULE,
+	.read = vfio_ap_save_read,
+	.compat_ioctl = compat_ptr_ioctl,
+	.release = vfio_ap_release_migf,
+};
+
+static struct vfio_ap_config
+*vfio_ap_allocate_config(struct ap_matrix_mdev *matrix_mdev, size_t *config_sz)
+{
+	struct vfio_ap_config *ap_config;
+	size_t qinfo_size;
+	int num_queues;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	num_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
+	qinfo_size = num_queues * sizeof(struct vfio_ap_queue_info);
+	*config_sz = qinfo_size + sizeof(struct vfio_ap_config);
+	ap_config = kzalloc(*config_sz, GFP_KERNEL_ACCOUNT);
+
+	if (!ap_config)
+		return ERR_PTR(-ENOMEM);
+
+	ap_config->num_queues = num_queues;
+
+	return ap_config;
+}
+
+static void vfio_ap_store_queue_info(struct vfio_ap_migration_file *migf)
+{
+	unsigned long *apm, *aqm, num_queues, apid, apqi, apqn, data;
+	struct ap_matrix_mdev *matrix_mdev;
+	struct vfio_ap_queue *q;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	matrix_mdev = migf->matrix_mdev;
+	apm = matrix_mdev->shadow_apcb.apm;
+	aqm = matrix_mdev->shadow_apcb.aqm;
+	num_queues = 0;
+
+	for_each_set_bit_inv(apid, apm, AP_DEVICES) {
+		for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) {
+			apqn = AP_MKQID(apid, apqi);
+			q = vfio_ap_mdev_get_queue(matrix_mdev, apqn);
+
+			if (!q)
+				continue;
+
+			migf->ap_config->qinfo[num_queues].apqn = apqn;
+			data = q->hwinfo.value;
+			migf->ap_config->qinfo[num_queues].data = data;
+			num_queues += 1;
+			dev_dbg(matrix_mdev->vdev.dev,
+				"%s (%d): qinfo: apqn=%04lx data=%016lx\n",
+				__func__, __LINE__, apqn, data);
+		}
+	}
+}
+
+static struct vfio_ap_migration_file
+*vfio_ap_allocate_migf(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_file *migf;
+	struct vfio_ap_config *ap_config;
+	size_t config_size;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	migf = kzalloc_obj(struct vfio_ap_migration_file, GFP_KERNEL_ACCOUNT);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	ap_config = vfio_ap_allocate_config(matrix_mdev, &config_size);
+	if (IS_ERR(ap_config)) {
+		kfree(migf);
+		return ERR_CAST(ap_config);
+	}
+
+	migf->ap_config = ap_config;
+	migf->config_sz = config_size;
+	migf->matrix_mdev = matrix_mdev;
+
+	return migf;
+}
+
+static void vfio_ap_deallocate_migf(struct vfio_ap_migration_file *migf)
+{
+	kfree(migf->ap_config);
+	kfree(migf);
+}
+
+static struct file *vfio_ap_open_file_stream(struct vfio_ap_migration_file *migf,
+					     const struct file_operations *fops,
+					     int flags)
+{
+	struct file *filp;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	filp = anon_inode_getfile("vfio_ap_migf", fops, migf, flags);
+	if (IS_ERR(filp))
+		return ERR_CAST(filp);
+
+	stream_open(filp->f_inode, filp);
+
+	return filp;
+}
+
+static struct vfio_ap_migration_file *
+vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+	struct vfio_ap_migration_file *migf;
+	struct file *filp;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	migf = vfio_ap_allocate_migf(matrix_mdev);
+	if (IS_ERR(migf))
+		return ERR_CAST(migf);
+
+	filp = vfio_ap_open_file_stream(migf, &vfio_ap_save_fops, O_RDONLY);
+	if (IS_ERR(filp)) {
+		vfio_ap_deallocate_migf(migf);
+		return ERR_CAST(filp);
+	}
+
+	migf->filp = filp;
+	mig_data->saving_migf = migf;
+	vfio_ap_store_queue_info(mig_data->saving_migf);
+
+	return mig_data->saving_migf;
+}
+
 static struct file *
 vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 			    enum vfio_device_mig_state new_state)
 {
 	struct vfio_ap_migration_data *mig_data;
 	enum vfio_device_mig_state cur_state;
+	struct vfio_ap_migration_file *migf;
 
 	lockdep_assert_held(&matrix_dev->mdevs_lock);
 	mig_data = matrix_mdev->mig_data;
@@ -109,10 +258,23 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 	dev_dbg(matrix_mdev->vdev.dev, "%s: %d -> %d\n", __func__, cur_state,
 		new_state);
 
+	/*
+	 * Begins the process of saving the vfio device state by creating and
+	 * returning a streaming data_fd to be used to read out the internal
+	 * state of the vfio-ap device on the source host.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_STOP &&
 	    new_state == VFIO_DEVICE_STATE_STOP_COPY) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		migf = vfio_ap_save_mdev_state(matrix_mdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+
+		if (migf) {
+			get_file(migf->filp);
+			return migf->filp;
+		}
+
+		return NULL;
 	}
 
 	if (cur_state == VFIO_DEVICE_STATE_STOP &&
-- 
2.52.0

[PATCH v1 14/24] s390/vfio-ap: File ops called to save the vfio device migration state

[Anthony Krowiak]

Implements two callback functions that were added to the 'file_operations'
structure for the file created to save the state of the vfio device
when the migration state transitioned from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_STOP_COPY:

* Read callback

  This function copies the guest's AP configuration information to
  userspace. The information copied is comprised of the APQN of each queue
  device passed through to the guest along with its hardware information.
  This data will be used to verify the source and target guests have
  compatible AP configurations.

* Release callback

  This function deallocates the object used to save the state of the
  vfio device.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 78 +++++++++++++++++++++++--
 1 file changed, 72 insertions(+), 6 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index ecc9f0f6819e..a340dca1426c 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -97,16 +97,82 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
 	}
 }
 
-static ssize_t vfio_ap_save_read(struct file *, char __user *, size_t, loff_t *)
+static ssize_t validate_save_read_parms(struct vfio_ap_migration_file *migf,
+					loff_t *pos, size_t len)
 {
-	/* TODO */
-	return -EOPNOTSUPP;
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	if (migf->disabled) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"%s (%d): migration file is disabled\n",
+			__func__, __LINE__);
+		return -ENODEV;
+	}
+
+	if (*pos > migf->config_sz) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"%s (%d): file pos (%llu) exceeds migf->config size (%zu)\n",
+			__func__, __LINE__, *pos, migf->config_sz);
+		return -EINVAL;
+	}
+
+	return 0;
 }
 
-static int vfio_ap_release_migf(struct inode *, struct file *)
+static ssize_t vfio_ap_save_read(struct file *filp, char __user *buf,
+				 size_t len, loff_t *pos)
 {
-	/* TODO */
-	return -EOPNOTSUPP;
+	struct vfio_ap_migration_file *migf;
+	ssize_t ret = 0;
+
+	if (pos)
+		return -ESPIPE;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	pos = &filp->f_pos;
+	migf = filp->private_data;
+
+	ret = validate_save_read_parms(migf, pos, len);
+	if (ret)
+		goto out_unlock;
+
+	len = min_t(size_t, migf->config_sz - *pos, len);
+	if (len) {
+		if (copy_to_user(buf, (void *)migf->ap_config + *pos, len)) {
+			ret = -EFAULT;
+			dev_err(migf->matrix_mdev->vdev.dev,
+				"%s (%d): failed to copy config data to user\n",
+				__func__, __LINE__);
+			goto out_unlock;
+		}
+
+		*pos += len;
+		ret = len;
+	}
+
+	dev_dbg(migf->matrix_mdev->vdev.dev,
+		"%s (%d): copied %zu bytes of AP config data to user\n",
+		__func__, __LINE__, len);
+
+out_unlock:
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return ret;
+}
+
+static void vfio_ap_deallocate_migf(struct vfio_ap_migration_file *migf);
+
+static int vfio_ap_release_migf(struct inode *inode, struct file *filp)
+{
+	struct vfio_ap_migration_file *migf;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	migf = filp->private_data;
+	vfio_ap_deallocate_migf(migf);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return 0;
 }
 
 static const struct file_operations vfio_ap_save_fops = {
-- 
2.52.0

[PATCH v1 15/24] s390/vfio-ap: Transition device migration state from STOP to RESUMING

[Anthony Krowiak]

Transitioning from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING
starts a process of restoring the device state of the vfio device on the
target system. To prepare for restoring the device state, open a file
stream to receive the guest's AP configuration saved when the device state
on the source system transitioned from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_STOP_COPY.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 55 ++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index a340dca1426c..a243ea134794 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -310,6 +310,46 @@ vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)
 	return mig_data->saving_migf;
 }
 
+static ssize_t vfio_ap_resume_write(struct file *filp, const char __user *buf,
+				    size_t len, loff_t *pos)
+{
+	/* TODO */
+	return -EOPNOTSUPP;
+}
+
+static const struct file_operations vfio_ap_resume_fops = {
+	.owner = THIS_MODULE,
+	.write = vfio_ap_resume_write,
+	.release = vfio_ap_release_migf,
+};
+
+static struct vfio_ap_migration_file *
+vfio_ap_resume_mdev_state(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+	struct vfio_ap_migration_file *migf;
+	struct file *filp;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	migf = vfio_ap_allocate_migf(matrix_mdev);
+	if (IS_ERR(migf))
+		return ERR_CAST(migf);
+
+	filp = vfio_ap_open_file_stream(migf, &vfio_ap_resume_fops, O_WRONLY);
+	if (IS_ERR(filp)) {
+		vfio_ap_deallocate_migf(migf);
+		return ERR_CAST(filp);
+	}
+
+	migf->matrix_mdev = matrix_mdev;
+	migf->filp = filp;
+	mig_data->resuming_migf = migf;
+
+	return migf;
+}
+
 static struct file *
 vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 			    enum vfio_device_mig_state new_state)
@@ -343,10 +383,21 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return NULL;
 	}
 
+	/*
+	 * Starts the process of restoring the state of the vfio-ap device
+	 * on the target host by creating a filestream to be used to transfer
+	 * the internal state of the vfio-ap device on the source host that
+	 * was saved during the STOP_COPY phase of the migration.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_STOP &&
 	    new_state == VFIO_DEVICE_STATE_RESUMING) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		migf = vfio_ap_resume_mdev_state(matrix_mdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+
+		get_file(migf->filp);
+
+		return migf->filp;
 	}
 
 	if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
-- 
2.52.0

[PATCH v1 16/24] s390/vfio-ap: File ops called to resume the vfio device migration

[Anthony Krowiak]

Implements two callback functions that were added to the 'file_operations'
structure for the file created to restore the state of the vfio device
when the migration state transitioned from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_RESUMING:

* Write callback

  The write callback retrieves the vfio device migration state saved to the
  file stream created when the vfio device state was transitioned from
  VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY. The saved state
  contains the guest's AP configuration information. Since the guest's AP
  configuration on the target system is supplied by the mediated device
  residing thereon, the migration state passed into this callback will not
  be used to restore the vfio device state, but to verify that the source
  and target AP configurations are compatible since the mediated devices on
  the source and target systems may differ.

  The verification rules are:

  o The source and target AP configurations must have the same number of
    APQNs

  o Each APQN in the source guest's AP configuration must also be in the
    target guest's configuration

  o Each APQN in the source and target guests' AP configurations must
    reference a queue device with compatible hardware:

    - The source and target queues must have the same facilities installed
  	- APSC facility
  	- APQKM facility
  	- AP4KC facility

    - The source and target queues must have the same mode
  	- Coprocessor-mode
  	- Accelerator-mode
  	- XCP-mode

    - The source and target queues must have the same APXA facility setting
   	- If the APXA facility is installed on target queue, it must also
   	  be installed on the source queue
   	- If the APXA facility is not installed on target queue, it must
   	  also not be installed on the source queue

    - The source and target queues must have the same classification
   	- Full native card function
   	- Stateless functions

    - The binding and associated state
   	~ At the time of migration, the queue on the target guest will
   	  neither be bound nor associated, so the binding and associated
   	  state for both the source and target must indicate that the
   	  queue is usable for all messages (i.e., BS bits equal to 00).

    - The AP type of the target queue must be the same or newer than the
      source queue

* Release callback

This function deallocates the object used to restore the state of the
vfio device when the file is released.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 525 +++++++++++++++++++++++-
 1 file changed, 523 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index a243ea134794..d9b6dba53181 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -10,6 +10,36 @@
 
 #include "vfio_ap_private.h"
 
+/*
+ * Masks the fields of the queue information returned from the PQAP(TAPQ)
+ * command. In order to migrate a guest, it's AP configuration must be
+ * compatible with AP configuration assigned to the target guest's mdev.
+ * This mask is used to verify that the queue information for each source and
+ * target queue is compatible (i.e., the masked fields are equivalent).
+ *
+ * The relevant fields covered by this mask are:
+ * S bit 0: APSC  facility installed
+ * M bit 1: APQKM facility installed
+ * C bit 2: AP4KC facility installed
+ * Mode bits 3-5:
+ *     D bit 3: CCA-mode facility
+ *     A bit 4: accelerator-mode facility
+ *     X bit 5: XCP-mode facility
+ * N  bit 6: APXA facility installed
+ * SL bit 7: SLCF facility installed
+ * Classification (functional capabilities) bits 8-16
+ *     bit 8: Native card function
+ *     bit 9: Only stateless functions
+ * AP Type bits 32-40:
+ */
+#define QINFO_DATA_MASK		0xffffc000ff000000
+
+/*
+ * Masks the bit that indicates whether full native card function is available
+ * from the 8 bits specifying the functional capabilities of a queue
+ */
+#define CLASSIFICATION_NATIVE_FCN_MASK		0x80
+
 /**
  * vfio_ap_migration_data - the data needed to migrate a guest with pass-through
  *			    access to AP devices
@@ -234,6 +264,53 @@ static void vfio_ap_store_queue_info(struct vfio_ap_migration_file *migf)
 	}
 }
 
+static int validate_resume_write_parms(struct vfio_ap_migration_file *migf,
+				       size_t len, loff_t *pos)
+{
+	loff_t total_len;
+	int ret = -EIO;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	if (!migf->matrix_mdev) {
+		pr_err("migration failed: matrix_mdev object not linked to migration file");
+		goto done;
+	}
+
+	if (*pos < 0) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: invalid migration file position  (%lli) for write\n",
+			*pos);
+		goto done;
+	}
+
+	if (check_add_overflow((loff_t)len, *pos, &total_len)) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: pos (%llu) plus len (%zu) operation overflowed loff_t precision\n",
+			*pos, len);
+		goto done;
+	}
+
+	if (total_len > migf->config_sz) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: source guest's AP config size (%llu) larger than target's (%lu)",
+			total_len, migf->config_sz);
+		goto done;
+	}
+
+	if (migf->disabled) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: migration file is disabled");
+		goto done;
+	}
+
+	dev_dbg(migf->matrix_mdev->vdev.dev, "resume write parameters validated\n");
+	ret = 0;
+
+done:
+	return ret;
+}
+
 static struct vfio_ap_migration_file
 *vfio_ap_allocate_migf(struct ap_matrix_mdev *matrix_mdev)
 {
@@ -310,11 +387,455 @@ vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)
 	return mig_data->saving_migf;
 }
 
+static void report_facilities_compatibility(struct ap_matrix_mdev *matrix_mdev,
+					    unsigned long apqn,
+					    struct ap_tapq_hwinfo *src_hwinfo,
+					    struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->apsc != target_hwinfo->apsc) {
+		if (src_hwinfo->apsc) {
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->mex4k != target_hwinfo->mex4k) {
+		if (src_hwinfo->mex4k) {
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->crt4k != target_hwinfo->crt4k) {
+		if (src_hwinfo->crt4k) {
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_mode_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqn,
+				      struct ap_tapq_hwinfo *src_hwinfo,
+				      struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->cca != target_hwinfo->cca) {
+		if (src_hwinfo->cca) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode  facility not installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode  facility installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->accel != target_hwinfo->accel) {
+		if (src_hwinfo->accel) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility not installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility not installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->ep11 != target_hwinfo->ep11) {
+		if (src_hwinfo->ep11) {
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility not installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility not installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_apxa_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqn,
+				      struct ap_tapq_hwinfo *src_hwinfo,
+				      struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->apxa != target_hwinfo->apxa) {
+		if (src_hwinfo->apxa) {
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_slcf_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqn,
+				      struct ap_tapq_hwinfo *src_hwinfo,
+				      struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->slcf != target_hwinfo->slcf) {
+		if (src_hwinfo->slcf) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) available in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) not available in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) not available in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) available in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_class_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				       unsigned long apqn,
+				       struct ap_tapq_hwinfo *src_hwinfo,
+				       struct ap_tapq_hwinfo *target_hwinfo)
+{
+	unsigned long src_native, target_native;
+
+	src_native = src_hwinfo->class & CLASSIFICATION_NATIVE_FCN_MASK;
+	target_native = target_hwinfo->class & CLASSIFICATION_NATIVE_FCN_MASK;
+
+	if (src_native != target_native) {
+		/*
+		 * If the source queue has full native card function and the
+		 * target queue has only stateless functions available, then
+		 * there may be instructions that will not execute on the
+		 * target queue.
+		 *
+		 * If the source queue has only stateless card functions and the
+		 * target queue has full native card function available, then
+		 * we are okay because the target queue can run all card
+		 * functions.
+		 */
+		if (src_native) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Full native card function available on source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Only stateless functions available on target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_bs_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				    unsigned long apqn,
+				    struct ap_tapq_hwinfo *src_hwinfo,
+				    struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->bs || target_hwinfo->bs) {
+		dev_err(matrix_mdev->vdev.dev,
+			"Bind/associate state for source (%01x) and target (%01x) queue %02lx.%04lx are not compatible\n",
+			src_hwinfo->bs, target_hwinfo->bs,
+			AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+	}
+}
+
+static void report_aptype_compatibility(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long apqn,
+					struct ap_tapq_hwinfo *src_hwinfo,
+					struct ap_tapq_hwinfo *target_hwinfo)
+{
+	dev_err(matrix_mdev->vdev.dev,
+		"AP type of source (%02x) not compatible with target (%02x)\n",
+		src_hwinfo->at, target_hwinfo->at);
+}
+
+/*
+ * Log a device error reporting that migration failed due to queue
+ * incompatibilities followed by a device error for each incompatible feature.
+ */
+static void report_qinfo_incompatibilities(struct ap_matrix_mdev *matrix_mdev,
+					   unsigned long apqn,
+					   struct ap_tapq_hwinfo *src_hwinfo,
+					   struct ap_tapq_hwinfo *target_hwinfo)
+{
+	dev_err(matrix_mdev->vdev.dev,
+		"Migration failed: Source and target queue (%02lx.%04lx) not compatible\n",
+		AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+	report_facilities_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_mode_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_apxa_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_slcf_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_class_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_aptype_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_bs_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+}
+
+static bool qinfo_compatible(struct ap_matrix_mdev *matrix_mdev,
+			     unsigned long apqn,
+			     struct ap_tapq_hwinfo *src_hwinfo,
+			     struct ap_tapq_hwinfo *target_hwinfo)
+{
+	unsigned long src_bits, target_bits;
+
+	src_bits = src_hwinfo->value & QINFO_DATA_MASK;
+	target_bits = target_hwinfo->value & QINFO_DATA_MASK;
+
+	/*
+	 * If all relevant bits are the same, or only the AP type of the source
+	 * and target queue differ but the source type is older than the target
+	 * type, then no incompatibilities will be reported. The AP types are
+	 * considered compatible even if they differ as long as the source type
+	 * is older than the target type since AP devices are backwards
+	 * compatible.
+	 */
+	if (src_bits == target_bits ||
+	    (src_hwinfo->fac == target_hwinfo->fac &&
+	     src_hwinfo->at <= target_hwinfo->at)) {
+		return true;
+	}
+
+	report_qinfo_incompatibilities(matrix_mdev, apqn, src_hwinfo,
+				       target_hwinfo);
+
+	return false;
+}
+
+static int matrixes_compatible(struct vfio_ap_migration_file *migf)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	struct ap_tapq_hwinfo src_hwinfo;
+	struct vfio_ap_queue *q;
+	unsigned long apqn;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	matrix_mdev = migf->matrix_mdev;
+
+	for (int i = 0; i < migf->ap_config->num_queues; i++) {
+		apqn = migf->ap_config->qinfo[i].apqn;
+		q = vfio_ap_mdev_get_queue(matrix_mdev, apqn);
+		memcpy(&src_hwinfo, &migf->ap_config->qinfo[i].data,
+		       sizeof(src_hwinfo));
+
+		if (!qinfo_compatible(matrix_mdev, apqn, &src_hwinfo, &q->hwinfo))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+static bool apqns_match(struct vfio_ap_migration_file *migf)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	unsigned long apid, apqi, apqn;
+	bool ret = true;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	matrix_mdev = migf->matrix_mdev;
+
+	for (int i = 0; i < migf->ap_config->num_queues; i++) {
+		apqn = migf->ap_config->qinfo[i].apqn;
+		apid = AP_QID_CARD(apqn);
+		apqi = AP_QID_QUEUE(apqn);
+
+		if (!test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) ||
+		    !test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
+			dev_err(matrix_mdev->vdev.dev,
+				"migration failed: queue %02lx.%04lx not assigned to guest matrix\n",
+				apid, apqi);
+			ret = false;
+		}
+	}
+
+	return ret;
+}
+
+static int vfio_ap_validate_num_queues(struct vfio_ap_migration_file *migf)
+{
+	int num_migf_queues, num_mdev_queues;
+	struct ap_matrix_mdev *matrix_mdev;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	matrix_mdev = migf->matrix_mdev;
+	num_mdev_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
+	num_migf_queues = migf->ap_config->num_queues;
+
+	if (num_mdev_queues != num_migf_queues) {
+		dev_err(matrix_mdev->vdev.dev,
+			"migration failed: number of queues on source (%d) and target (%d) guests differ\n",
+			num_migf_queues, num_mdev_queues);
+		return (num_mdev_queues > num_migf_queues) ? -ENODEV : -E2BIG;
+	}
+
+	return 0;
+}
+
+static int do_post_copy_validation(struct vfio_ap_migration_file *migf, loff_t pos)
+{
+	unsigned long nqueues_offset;
+	int ret;
+
+	nqueues_offset = offsetofend(struct vfio_ap_config, num_queues);
+	if (pos >= nqueues_offset) {
+		ret = vfio_ap_validate_num_queues(migf);
+		if (ret)
+			return ret;
+
+		if (pos == migf->config_sz) {
+			if (!apqns_match(migf))
+				return -ENODEV;
+			ret = matrixes_compatible(migf);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * vfio_ap_resume_write - store the AP configuration information sent from the
+ *			  source guest into the migration file.
+ * @filp: the file used to send the AP configuration information from the source
+ *	  guest.
+ * @buf:  buffer containing the AP configuration information sent from the
+ *	  source guest
+ * @len:  the length of the AP configuration information contained in @buf
+ * *pos:  a pointer to store the file position after retrieving the AP config
+ *	  information from @buf
+ */
 static ssize_t vfio_ap_resume_write(struct file *filp, const char __user *buf,
 				    size_t len, loff_t *pos)
 {
-	/* TODO */
-	return -EOPNOTSUPP;
+	struct vfio_ap_migration_data *mig_data;
+	struct vfio_ap_migration_file *migf;
+	ssize_t ret = 0;
+
+	if (pos)
+		return -ESPIPE;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	pos = &filp->f_pos;
+	migf = filp->private_data;
+
+	if (ap_is_se_guest()) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"Migration failed: Target is a secure execution guest\n");
+		return -EOPNOTSUPP;
+	}
+
+	mig_data = migf->matrix_mdev->mig_data;
+
+	ret = validate_resume_write_parms(migf, len, pos);
+	if (ret)
+		goto out_unlock;
+
+	if (copy_from_user((void *)migf->ap_config + *pos, buf, len)) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"%s (%d): failed to copy queue information from userspace",
+			__func__, __LINE__);
+		ret = -EFAULT;
+		goto out_unlock;
+	}
+
+	*pos += len;
+
+	ret = do_post_copy_validation(migf, *pos);
+	if (ret)
+		goto out_unlock;
+
+	ret = len;
+
+	dev_dbg(migf->matrix_mdev->vdev.dev,
+		"%s (%d): %zu bytes of queue information stored in the migration file",
+		__func__, __LINE__, len);
+
+out_unlock:
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return ret;
 }
 
 static const struct file_operations vfio_ap_resume_fops = {
-- 
2.52.0

[PATCH v1 17/24] s390/vfio-ap: Transition device migration state from RESUMING to STOP

[Anthony Krowiak]

Leaving the RESUMING state terminates a data transfer of the internal
state of the vfio-ap device on the source host to the vfio-ap device on the
target host. This state transition indicates the vfio_ap driver should
complete the incorporation of data written to the data transfer FD into the
into the vfio-ap device. Since a vfio-ap device does not virtualize a
physical device, there is no internal device state to
incorporate; so, the only thing left to do is release the migration files
used to help facilitate the migration process.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index d9b6dba53181..71cae26eab96 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -921,10 +921,20 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return migf->filp;
 	}
 
+	/*
+	 * Terminates the data transfer session of the vfio-ap device state
+	 * between the source and target hosts. Since the vfio-ap device does
+	 * not virtualize a DMA device, there is no internal device state to
+	 * incorporate into the vfio-ap device on the target; so, the only
+	 * thing left to do is release the migration files used to process
+	 * the vfio device migration. Note that this state transition is for
+	 * the vfio-ap device on the target host.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
 	    new_state == VFIO_DEVICE_STATE_STOP) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		vfio_ap_release_mig_files(matrix_mdev);
+
+		return NULL;
 	}
 
 	if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
-- 
2.52.0

[PATCH v1 18/24] s390/vfio-ap: Transition device migration state from STOP_COPY to STOP

[Anthony Krowiak]

Typically at this point, interrupt processing for the device, DMA and other
changes to the device's external state must be prohibited. For VFIO AP,
there is no physical DMA device virtualized by the vfio-ap device . It
merely manages the guest's AP configuration that identifies the devices
passed through to the guest, but does not have access to any of them. Those
AP devices are passed through and controlled exclusively by the SIE program
used to start the guest. The only thing to do at this point is release all
migration objects used to manage migration of the guest.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 71cae26eab96..c6e01294be7f 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -937,10 +937,18 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return NULL;
 	}
 
+	/*
+	 * Stop the operation of the vfio-ap device. Since the vfio-ap device
+	 * does not virtualize a DMA device, there is no physical device to
+	 * stop; so, the only thing left to do is release the migration files
+	 * used to process the vfio device migration. Note that this state
+	 * transition is for the vfio-ap device on the source host.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
 	    new_state == VFIO_DEVICE_STATE_STOP) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		vfio_ap_release_mig_files(matrix_mdev);
+
+		return NULL;
 	}
 
 	if ((cur_state == VFIO_DEVICE_STATE_STOP &&
-- 
2.52.0

[PATCH v1 19/24] s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa

[Anthony Krowiak]

The transition from STOP to RUNNING and vice versa indicates that
the vfio device is operating normally and migration is not
currently taking place. In this case, just set the new state and
if there are any migrations files used to help facilitate migration
hanging around, release them.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index c6e01294be7f..750299fb1a84 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -951,12 +951,18 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return NULL;
 	}
 
+	/*
+	 * These states indicates migration has either not been initiated or
+	 * has completed and the vfio-ap device is operating normally; so
+	 * just set the new migration state. Just in case, release the
+	 * migration files used to facilitate migration if any are hanging
+	 * around.
+	 */
 	if ((cur_state == VFIO_DEVICE_STATE_STOP &&
 	     new_state == VFIO_DEVICE_STATE_RUNNING) ||
 	    (cur_state == VFIO_DEVICE_STATE_RUNNING &&
 	     new_state == VFIO_DEVICE_STATE_STOP)) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		return NULL;
 	}
 
 	/* vfio_mig_get_next_state() does not use arcs other than the above */
-- 
2.52.0

[PATCH v1 20/24] s390-vfio-ap: Callback to get the current vfio device migration state

[Anthony Krowiak]

Implements the callback that returns the current vfio device migration
state during live migration of guests with pass-through access to
AP devices.

The function pointer for this callback is specified via the
'migration_get_state' field of the 'vfio_migration_ops' structure
which is stored with the VFIO device when the 'vfio_device'
structure representing the mediated device is initialized.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 750299fb1a84..305fd3221be1 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -1014,7 +1014,18 @@ static struct file *vfio_ap_set_state(struct vfio_device *vdev,
 static int vfio_ap_get_state(struct vfio_device *vdev,
 			     enum vfio_device_mig_state  *current_state)
 {
-	return -EOPNOTSUPP;
+	struct ap_matrix_mdev *matrix_mdev;
+	struct vfio_ap_migration_data *mig_data;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	mig_data =  matrix_mdev->mig_data;
+	*current_state = mig_data->mig_state;
+
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return 0;
 }
 
 static int vfio_ap_get_data_size(struct vfio_device *vdev,
-- 
2.52.0

[PATCH v1 21/24] s390/vfio-ap: Callback to get the size of data to be migrated during guest migration

[Anthony Krowiak]

Callback to get the estimated data length of the data that will be required
to complete reading of the vfio device's internal state. It's mandatory for
VFIO_DEVICE_FEATURE_MIGRATION migration support.

The function pointer for this callback is specified via the
'migration_get_data_size' field of the 'vfio_migration_ops' structure
which is stored with the VFIO device when the 'vfio_device'
structure representing the mediated device is initialized.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 305fd3221be1..fa163b129c66 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -1031,7 +1031,18 @@ static int vfio_ap_get_state(struct vfio_device *vdev,
 static int vfio_ap_get_data_size(struct vfio_device *vdev,
 				 unsigned long *stop_copy_length)
 {
-	return -EOPNOTSUPP;
+	struct ap_matrix_mdev *matrix_mdev;
+	size_t qinfo_sz;
+	int num_queues;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	num_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
+	qinfo_sz = num_queues * sizeof(struct vfio_ap_queue_info);
+	*stop_copy_length = qinfo_sz + sizeof(struct vfio_ap_config);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return 0;
 }
 
 static const struct vfio_migration_ops vfio_ap_migration_ops = {
-- 
2.52.0

[PATCH v1 22/24] s390/vfio-ap: Provide API to query whether migration is in progress

[Anthony Krowiak]

Provides an AP to query whether a live guest migration is in progress. This
can be called to determine whether an action should be allowed during
migration; for example, allowing a system administrator to disable
migration while it is in progress. Other uses may be to prevent changing
a guest's AP configuration while a migration is in progress.

The only means the vfio_ap device driver has to determine that a migration
is in progress is via the callback functions specified in the
'struct vfio_migration_ops' object associated with the vfio device.
The very first callback invoked during migration is the callback specified
in the 'migration_get_data_size' field of the structure. This callback will
set a flag indicating that migration is in progress. The flag will be
cleared when the vfio device migration state transitions from
VFIO_DEVICE_STATE_STOP_COPY to VFIO_DEVICE_STATE_STOP. At this point, the
internal state of the vfio device on the source has been copied and will
be restored on the target.

Note that the 'migration_get_data_size' callback occurs after migration has
been initiated, so there is a short period of time during which the API
will erroneously indicate migration is not in progress. This,
unfortunately, is unavoidable given the only means of determining that a
migration is in progress is via these callbacks.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 40 ++++++++++++++++++++++++-
 drivers/s390/crypto/vfio_ap_private.h   |  1 +
 2 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index fa163b129c66..8a7628fcf285 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -47,11 +47,14 @@
  * @mig_state:		the current migration state
  * @resuming_migf:	the object used to resume the target guest
  * @saving_migf:	the object used to save the state of the source guest
+ * @mig_in_progress	flag indicating whether migration is in progress (true)
+ *			or not (false)
  */
 struct vfio_ap_migration_data {
 	enum vfio_device_mig_state	mig_state;
 	struct vfio_ap_migration_file	*resuming_migf;
 	struct vfio_ap_migration_file	*saving_migf;
+	bool				mig_in_progress;
 };
 
 /**
@@ -971,18 +974,30 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 	return ERR_PTR(-EINVAL);
 }
 
+static void vfio_ap_set_mig_in_progress(struct vfio_ap_migration_data *mig_data,
+					enum vfio_device_mig_state prev_state,
+					enum vfio_device_mig_state new_state)
+{
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	if (prev_state == VFIO_DEVICE_STATE_STOP_COPY &&
+	    new_state == VFIO_DEVICE_STATE_STOP) {
+		mig_data->mig_in_progress = false;
+	}
+}
+
 static struct file *vfio_ap_set_state(struct vfio_device *vdev,
 				      enum vfio_device_mig_state  new_state)
 {
 	int ret;
 	struct file *filp = NULL;
 	struct ap_matrix_mdev *matrix_mdev;
-	enum vfio_device_mig_state next_state;
+	enum vfio_device_mig_state prev_state, next_state;
 	struct vfio_ap_migration_data *mig_data;
 
 	mutex_lock(&matrix_dev->mdevs_lock);
 	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
 	mig_data = matrix_mdev->mig_data;
+	prev_state = mig_data->mig_state;
 	dev_dbg(vdev->dev, "%s -> %d\n", __func__, new_state);
 
 	while (mig_data->mig_state != new_state) {
@@ -1006,6 +1021,7 @@ static struct file *vfio_ap_set_state(struct vfio_device *vdev,
 		}
 	}
 
+	vfio_ap_set_mig_in_progress(mig_data, prev_state, new_state);
 	mutex_unlock(&matrix_dev->mdevs_lock);
 
 	return filp;
@@ -1031,12 +1047,15 @@ static int vfio_ap_get_state(struct vfio_device *vdev,
 static int vfio_ap_get_data_size(struct vfio_device *vdev,
 				 unsigned long *stop_copy_length)
 {
+	struct vfio_ap_migration_data *mig_data;
 	struct ap_matrix_mdev *matrix_mdev;
 	size_t qinfo_sz;
 	int num_queues;
 
 	mutex_lock(&matrix_dev->mdevs_lock);
 	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	mig_data = matrix_mdev->mig_data;
+	mig_data->mig_in_progress = true;
 	num_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
 	qinfo_sz = num_queues * sizeof(struct vfio_ap_queue_info);
 	*stop_copy_length = qinfo_sz + sizeof(struct vfio_ap_config);
@@ -1051,6 +1070,24 @@ static const struct vfio_migration_ops vfio_ap_migration_ops = {
 	.migration_get_data_size = vfio_ap_get_data_size,
 };
 
+/**
+ * vfio_ap_migration_in_progress: Indicates whether a live guest migration is in
+ *				  progress or not.
+ *
+ * @matrix_mdev: pointer to object maintaining the vfio device state
+ *
+ * Return: returns true if a migration is in progress; otherwise, returns false.
+ */
+bool vfio_ap_migration_in_progress(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	return mig_data->mig_in_progress;
+}
+
 /**
  * vfio_ap_init_migration_data - initialize migration data and functions
  *
@@ -1069,6 +1106,7 @@ int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev)
 		return -ENOMEM;
 
 	mig_data->mig_state = VFIO_DEVICE_STATE_STOP;
+	mig_data->mig_in_progress = false;
 	matrix_mdev->vdev.migration_flags = VFIO_MIGRATION_STOP_COPY;
 	matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops;
 	matrix_mdev->mig_data = mig_data;
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index cad49b509f7a..e04644c335fe 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -183,5 +183,6 @@ void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
 
 int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev);
 void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev);
+bool vfio_ap_migration_in_progress(struct ap_matrix_mdev *matrix_mdev);
 
 #endif /* _VFIO_AP_PRIVATE_H_ */
-- 
2.52.0

[PATCH v1 23/24] s390/vfio-ap: Disallow blocking migration in progress

[Anthony Krowiak]

The sysfs 'migratable' attribute allows a system administrator to enable or
block live guest migration. If an attempt to block migration is attempted
while a migration is in already in progress, the migration will complete
leaving the system administrator believing that it was blocked. To prevent
this scenario, the operation will fail with the
"Device or resource busy" (-EBUSY) error if a migration is in progress.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 1ff9994bd7ec..c8d454e52665 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1853,6 +1853,12 @@ static ssize_t migratable_store(struct device *dev, struct device_attribute *att
 		goto out_unlock;
 	}
 
+	if (!migratable && matrix_mdev->migratable &&
+	    vfio_ap_migration_in_progress(matrix_mdev)) {
+		ret = -EBUSY;
+		goto out_unlock;
+	}
+
 	matrix_mdev->migratable = migratable;
 
 	if (matrix_mdev->migratable)
-- 
2.52.0

[PATCH v1 24/24] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst

[Anthony Krowiak]

Add a 'Live Guest Migration' chapter to the
Documentation/arch/s390/vfio-ap.rst document to describe the details for
initiating live guest migration for a guest to which AP adapters, domains
and control domains have been passed through.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 Documentation/arch/s390/vfio-ap.rst | 339 ++++++++++++++++++++--------
 1 file changed, 246 insertions(+), 93 deletions(-)

diff --git a/Documentation/arch/s390/vfio-ap.rst b/Documentation/arch/s390/vfio-ap.rst
index eba1991fbdba..9df7614fd62b 100644
--- a/Documentation/arch/s390/vfio-ap.rst
+++ b/Documentation/arch/s390/vfio-ap.rst
@@ -1016,7 +1016,7 @@ guest_matrix dyn ap_config
 
 the following features are advertised:
 
----------------+---------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
 | Flag         | Description                                                   |
 +==============+===============================================================+
 | guest_matrix | guest_matrix attribute exists. It reports the matrix of       |
@@ -1025,105 +1025,258 @@ the following features are advertised:
 +--------------+---------------------------------------------------------------+
 | dyn          | Indicates hot plug/unplug of AP adapters, domains and control |
 |              | domains for a guest to which the mdev is attached.            |
-+------------+-----------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
 | ap_config    | ap_config interface for one-shot modifications to mdev config |
 +--------------+---------------------------------------------------------------+
+| migratable   | Indicates that live guest migration is supported for guests   |
+|              | to which crypto devices are passed through                    |
++--------------+---------------------------------------------------------------+
 
-Limitations
-===========
-Live guest migration is not supported for guests using AP devices without
-intervention by a system administrator. Before a KVM guest can be migrated,
-the vfio_ap mediated device must be removed. Unfortunately, it can not be
-removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
-the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
-its mdev can be hot unplugged from the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
-   the following commands:
-
-      virsh detach-device <guestname> <path-to-device-xml>
-
-      For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
-      the guest named 'my-guest':
-
-         virsh detach-device my-guest ~/config/my-guest-hostdev.xml
-
-            The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
-            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
-              <source>
-                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
-              </source>
-            </hostdev>
-
-
-      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
-
-      For example, to hot unplug the vfio_ap mediated device identified on the
-      qemu command line with 'id=hostdev0' from the guest named 'my-guest':
-
-.. code-block:: sh
-
-         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
-
-2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
-   to the guest and using the following qemu monitor command:
-
-      (QEMU) device-del id=<device-id>
-
-      For example, to hot unplug the vfio_ap mediated device that was specified
-      on the qemu command line with 'id=hostdev0' when the guest was started:
-
-         (QEMU) device-del id=hostdev0
-
-After live migration of the KVM guest completes, an AP configuration can be
-restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
-system into the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
-   device into the guest via the following virsh commands:
-
-   virsh attach-device <guestname> <path-to-device-xml>
-
-      For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
-      the guest named 'my-guest':
-
-         virsh attach-device my-guest ~/config/my-guest-hostdev.xml
-
-            The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
-            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
-              <source>
-                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
-              </source>
-            </hostdev>
-
+Live Guest Migration
+====================
+A guest to which crypto devices (i.e., AP adapters, domains and control domains)
+have been passed through can be migrated to another system if both the source
+and destination guests meet the following requirements:
 
-   virsh qemu-monitor-command <guest-name> --hmp \
-   "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+* The destination guest will be started with the same QEMU command line as the
+  source guest, so the mediated device supplying the AP configuration on both
+  the source and destination guests must have the same name (UUID).
 
-      For example, to hot plug the vfio_ap mediated device
-      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
-      device-id hostdev0:
+* The AP configuration passed through to both the source and destination guests
+  must be compatible. As such, each AP configuration:
 
-      virsh qemu-monitor-command my-guest --hmp \
-      "device_add vfio-ap,\
-      sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
-      id=hostdev0"
+  * Must have the same number of APQNs
 
-2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
-   to the guest and using the following qemu monitor command:
+  * Each APQN assigned to the source guest must also be assigned to the
+    destination guest
 
-      (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+  * Each APQN assigned to the source guest and destination guest must reference
+    a queue with compatible hardware capabilities:
 
-      For example, to plug the vfio_ap mediated device
-      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
-      hostdev0:
++--------------+---------------------------------------------------------------+
+| Hardware     | Description                                                   |
+| Capabilities |                                                               |
++==============+===============================================================+
+| facilities   | * AP special command facility (APSC)                          |
+|              | * AP 4096-bit ME PKU commands facility (AP4KM)                |
+|              | * AP 4096-bit CRT PKU commands (AP4KC)                        |
++--------------+---------------------------------------------------------------+
+| mode         | * CCA-mode                                                    |
+|              | * Accelerator-mode                                            |
+|              | * XCP-mode (EP11)                                             |
++--------------+---------------------------------------------------------------+
+| AP extended  | APXA installed                                                |
+| addressing   |                                                               |
++--------------+---------------------------------------------------------------+
+| command      | Command filtering available                                   |
+| filtering    |                                                               |
++--------------+---------------------------------------------------------------+
+| functional   | * Full native card function                                   |
+| capabilities | * Only stateless functions                                    |
++--------------+---------------------------------------------------------------+
+| secure       | The guest running on the source host can not have any queues  |
+| execution    | bound or associated with it                                   |
++--------------+---------------------------------------------------------------+
+| AP type      | * No AP type                                                  |
+|              | * PCICC (Leeds-2)                                             |
+|              | * PCICA (Leeds-2 Lite)                                        |
+|              | * PCIXCC                                                      |
+|              | * CEX2A                                                       |
+|              | * CEX2C                                                       |
+|              | * CEX3A                                                       |
+|              | * CEX3C                                                       |
+|              | * CEX4S                                                       |
+|              | * CEX5S                                                       |
+|              | * CEX6S                                                       |
+|              | * CEX7S                                                       |
+|              | * CEX8S                                                       |
+|              |                                                               |
+|              | Note: The AP type on the source and destination guests can    |
+|              |       differ if the queue passed through to the target guest  |
+|              |       is a newer model (backwards compatible)                 |
++--------------+---------------------------------------------------------------+
 
-         (QEMU) device-add "vfio-ap,\
-         sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
-         id=hostdev0"
+Enabling and disabling live guest migration
+-------------------------------------------
+By default, live guest migration will be disabled for guests that use crypto
+devices. A r/w sysfs 'migratable' attribute for each mediated device is provided
+to enable/disable live guest migration for the guest to which the mediated
+device is attached. The valid values for this attribute are 1 (enable) and
+0 (disable).
+
+To enable live migration:
+
+	echo 1 /sys/bus/mdev/devices/$UUID/migratable
+
+To disable live migration:
+
+	echo 0 /sys/bus/mdev/devices/$UUID/migratable
+
+The 'migratable' attribute allows migration to be dynamically enabled or
+disabled for the guest to which the mediated device is attached; however,
+if migration of the vfio-ap device is already in progress, the operation will
+be rejected with a 'Device or resource busy' message.
+
+Live guest migration failures due to AP configuration errors
+------------------------------------------------------------
+The VFIO AP mediated device is not used to provide userspace with direct access
+to a device as is the case with other devices that use the VFIO framework to
+pass them through to a guest. The sole purpose of the VFIO AP mediated device
+is to manage an AP configuration for a guest. An AP configuration is comprised
+of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the
+control domains to which a guest will be granted access. Once the VFIO AP
+mediated device is attached to the guest, its AP configuration is set by the
+vfio_ap device driver. Once set, all access to the AP devices is handled
+by the s390 Interpretive Execution facility; in other words, the vfio_ap device
+driver plays no role in providing direct access to these AP devices.
+
+The only role that the vfio_ap device driver plays in the migration process is
+to verify that the AP configuration for the source guest is compatible with the
+AP configuration of the destination guest. Incompatibility will result in a live
+guest migration failure. This section describes how to debug this type of
+failure.
+
+The destination host is missing the mediated device with the same name as the
+mdev attached to the source guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: device not found: mediated device '$UUID' not found*       |
++--------------+--------------------------------------------------------------------+
+
+The source guest is not enabled for migration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: Requested operation is not valid:                          |
+|              | cannot migrate domain: $UUID: Migration is disabled for VFIO       |
+|              | device*                                                            |
++--------------+--------------------------------------------------------------------+
+
+
+The AP configuration of the source and destination guests are not compatible
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: operation failed: job 'migration in' failed: load of       |
+|              | migration failed: Bad address*                                     |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: Sibling indicated error 1*                     |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | *vfio_ap_mdev $UUID: Migration failed: Source and target queues    |
+|              | ($APQN) are not compatible*                                        |
+|              |                                                                    |
+|              | The message above will be followed by one or more messages         |
+|              | enumerating the incompatible features; for example:                |
+|              |                                                                    |
+|              | *vfio_ap_mdev $UUID: APSC facility installed in source queue $APQN*|
+|              |                                                                    |
+|              | *vfio_ap_mdev $UUID: APSC facility not installed in target queue   |
+|              | $APQN*                                                             |
+|              |                                                                    |
+|              | *AP type of source ($APTYPE) not compatible with target ($APTYPE)* |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: error while loading state section id ...*      |
+|              |                                                                    |
+|              | *shutting down, reason=failed*                                     |
+|              |                                                                    |
+|              | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)*     |
++--------------+--------------------------------------------------------------------+
+
+The AP configuration of the source guest has more APQNS than the destination guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: operation f1e166ee77e6failed: job 'migration in' failed: load of       |
+|              | migration failed: Input/output error*                              |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: Sibling indicated error 1*                     |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | *vfio_ap_mdev $UUID:                                               |
+|              | migration failed: source guest's AP config size (xx) larger than   |
+|              | target's (yy)*                                                     |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: error while loading state section id ...*      |
+|              |                                                                    |
+|              | *shutting down, reason=failed*                                     |
+|              |                                                                    |
+|              | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)*     |
++--------------+--------------------------------------------------------------------+
+
+The AP configuration of the source guest has fewer APQNS than the destination guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: operation failed: job 'migration in' failed: load of       |
+|              | migration failed: No such device*                                  |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: Sibling indicated error 1*                     |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | *vfio_ap_mdev $UUID:                                               |
+|              | migration failed: number of queues on source (x) and target (y)    |
+|              | guests differ*                                                     |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: error while loading state section id ...*      |
+|              |                                                                    |
+|              | *shutting down, reason=failed*                                     |
+|              |                                                                    |
+|              | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)*     |
++--------------+--------------------------------------------------------------------+
-- 
2.52.0

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Alex Williamson]

On Wed, 25 Mar 2026 17:00:47 -0400
Anthony Krowiak <akrowiak@linux.ibm.com> wrote:

> This patch series implements live guest migration of a guest to which AP
> devices have been passed through. To better comprehend this design, one has
> to understand that VFIO AP mediated device is not used to provide userspace
> with direct access to a device as is the case with other devices that use
> the VFIO framework to pass them through to a guest. The sole purpose of the
> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
> configuration is comprised of the AP adapter IDs (APID), AP queue 
> indexes (APQI) and domain numbers of the control domains to which a guest
> will be granted access. Once the VFIO AP mediated device is attached to a
> guest, its AP configuration is set by the vfio_ap device driver. Once set,
> all access to the AP devices is handled by the s390 Interpretive Execution
> facility; in other words, the vfio_ap device driver plays no role in
> providing direct access to the AP devices in the guest's AP configuration.
> 
> The only role that the vfio_ap device driver plays in the migration
> process is to verify that the AP configuration for the source guest is
> compatible with the AP configuration of the destination guest.
> Incompatibility will result in a live guest migration failure.
> In order to be compatible, the following requirements must be met:
> 
> 1. The destination guest will be started with the same QEMU command line
>    as the source guest, so the mediated device supplying the AP
>    configuration on both guests must have the same name (UUID).

AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
supporting migration.  You're really concerned more with the
configuration/composition of the mdev device, so requiring the same
UUID seems a bit arbitrary.

> 
> 2. The AP configuration assigned via the VFIO AP mediated device on both
>    guests must be compatible. As such, each AP configuration must meet
>    the following requirements:
> 
>    * Both guests must have the same number of APQNs
> 
>    * Each APQN assigned to the source guest must also be assigned to the 
>      destination guest
> 
>    * Each APQN assigned to both guests must reference an AP queue with the
>      same hardware capabilities

Why isn't this sufficient vs also requiring the same UUID?
      
> Note: There is a forthcoming consumer of this series which will be a QEMU 
>       patch series is entitled: 
>       'hw/vfio/ap: Implement live guest migration of guests using AP 
>       devices'
> 
> This design also adds a use case for enabling and disabling 
> migration of guests to which AP devices have been passed through. To
> facilitate this, a new read/write sysfs 'migratable' attribute is added to
> the mediated device. This attribute specifies whether the vfio device is
> migratable (1) or not (0). When the value of this attribute is changed, the
> vfio_ap device driver will signal an eventfd to userspace. It is up to
> userspace to respond to the change by enabling or disabling migration of
> the guest to which the mediated device is attached. The operation will be
> rejected with a 'Device or resource busy' message if a migration is in
> progress.

This seems inherently racy.  What happens if the device becomes
unmigratable while it's being migrated?

If userspace is deciding that the device is now unmigratable, why does
it need to signal this through the kernel driver rather than with the
userspace orchestration agent?  The entire path seems unnecessary.

> Userspace must also have a means for retrieving the value of the sysfs
> 'migratable' attribute when the guest is started to initialize whether it
> can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The 
> struct vfio_device_info object passed to the ioctl will be extended with a
> capability specifying the vfio device attributes. One of the attributes 
> will contain the value of the mediated device's 'migratable' attribute.

This is just broken, it's redundant to our current device feature
mechanism for exposing migration support.  If you want the capability
to create unmigratable devices statically, can't that be encompassed
within the definition of the mdev type?  Dynamic migration support just
seems like it's involving the kernel in orchestration it shouldn't be a
part of.  Thanks,

Alex
 
> Anthony Krowiak (23):
>   s390/vfio-ap: Store queue hardware info when probed
>   s390/vfio-ap: Provide access to queue objects and related info
>   s390/vfio-ap: Add header file for xfer of vfio device caps to
>     userspace
>   MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
>     maintainers
>   s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
>     migration of guest
>   s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
>   s390/vfio-ap: Signal event to enable/disable live guest migration
>   s390/vfio-ap: Return value of sysfs migratable attribute from
>     VFIO_DEVICE_GET_INFO ioctl
>   s390/vfio-ap: Data structures for facilitating vfio device migration
>   s390/vfio-ap: Initialize/release vfio device migration data
>   s390-vfio-ap: Callback to set vfio device mig state during guest
>     migration
>   s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
>   s390/vfio-ap: File ops called to save the vfio device migration state
>   s390/vfio-ap: Transition device migration state from STOP to RESUMING
>   s390/vfio-ap: File ops called to resume the vfio device migration
>   s390/vfio-ap: Transition device migration state from RESUMING to STOP
>   s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
>   s390/vfio-ap: Transition device migration state from STOP to RUNNING
>     and vice versa
>   s390-vfio-ap: Callback to get the current vfio device migration state
>   s390/vfio-ap: Callback to get the size of data to be migrated during
>     guest migration
>   s390/vfio-ap: Provide API to query whether migration is in progress
>   s390/vfio-ap: Disallow blocking migration in progress
>   s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
> 
>  Documentation/arch/s390/vfio-ap.rst     |  339 +++++--
>  MAINTAINERS                             |    1 +
>  drivers/s390/crypto/Makefile            |    2 +-
>  drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
>  drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
>  drivers/s390/crypto/vfio_ap_ops.c       |  263 +++++-
>  drivers/s390/crypto/vfio_ap_private.h   |   20 +
>  include/uapi/linux/vfio.h               |    2 +
>  include/uapi/linux/vfio_ap.h            |   34 +
>  9 files changed, 1685 insertions(+), 111 deletions(-)
>  create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>  create mode 100644 include/uapi/linux/vfio_ap.h
> 

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

On 3/30/26 12:27 PM, Alex Williamson wrote:
> On Wed, 25 Mar 2026 17:00:47 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> This patch series implements live guest migration of a guest to which AP
>> devices have been passed through. To better comprehend this design, one has
>> to understand that VFIO AP mediated device is not used to provide userspace
>> with direct access to a device as is the case with other devices that use
>> the VFIO framework to pass them through to a guest. The sole purpose of the
>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>> configuration is comprised of the AP adapter IDs (APID), AP queue
>> indexes (APQI) and domain numbers of the control domains to which a guest
>> will be granted access. Once the VFIO AP mediated device is attached to a
>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>> all access to the AP devices is handled by the s390 Interpretive Execution
>> facility; in other words, the vfio_ap device driver plays no role in
>> providing direct access to the AP devices in the guest's AP configuration.
>>
>> The only role that the vfio_ap device driver plays in the migration
>> process is to verify that the AP configuration for the source guest is
>> compatible with the AP configuration of the destination guest.
>> Incompatibility will result in a live guest migration failure.
>> In order to be compatible, the following requirements must be met:
>>
>> 1. The destination guest will be started with the same QEMU command line
>>     as the source guest, so the mediated device supplying the AP
>>     configuration on both guests must have the same name (UUID).
> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> supporting migration.  You're really concerned more with the
> configuration/composition of the mdev device, so requiring the same
> UUID seems a bit arbitrary.

As stated above, the destination guest will be started with the same
QEMU command line as the source guest. Within that command line
will be a '-device' parameter like the following:

-device 
'{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}

Note that sysfsdev is the path to the mdev named 
62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or
the migration will fail with the following error:

error: device not found: mediated device 
'62177883-f1bb-47f0-914d-32a22e3a8804' not found

>
>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>     guests must be compatible. As such, each AP configuration must meet
>>     the following requirements:
>>
>>     * Both guests must have the same number of APQNs
>>
>>     * Each APQN assigned to the source guest must also be assigned to the
>>       destination guest
>>
>>     * Each APQN assigned to both guests must reference an AP queue with the
>>       same hardware capabilities
> Why isn't this sufficient vs also requiring the same UUID?
>        
>> Note: There is a forthcoming consumer of this series which will be a QEMU
>>        patch series is entitled:
>>        'hw/vfio/ap: Implement live guest migration of guests using AP
>>        devices'
>>
>> This design also adds a use case for enabling and disabling
>> migration of guests to which AP devices have been passed through. To
>> facilitate this, a new read/write sysfs 'migratable' attribute is added to
>> the mediated device. This attribute specifies whether the vfio device is
>> migratable (1) or not (0). When the value of this attribute is changed, the
>> vfio_ap device driver will signal an eventfd to userspace. It is up to
>> userspace to respond to the change by enabling or disabling migration of
>> the guest to which the mediated device is attached. The operation will be
>> rejected with a 'Device or resource busy' message if a migration is in
>> progress.
> This seems inherently racy.  What happens if the device becomes
> unmigratable while it's being migrated?
>
> If userspace is deciding that the device is now unmigratable, why does
> it need to signal this through the kernel driver rather than with the
> userspace orchestration agent?  The entire path seems unnecessary.
>
>> Userspace must also have a means for retrieving the value of the sysfs
>> 'migratable' attribute when the guest is started to initialize whether it
>> can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
>> struct vfio_device_info object passed to the ioctl will be extended with a
>> capability specifying the vfio device attributes. One of the attributes
>> will contain the value of the mediated device's 'migratable' attribute.
> This is just broken, it's redundant to our current device feature
> mechanism for exposing migration support.  If you want the capability
> to create unmigratable devices statically, can't that be encompassed
> within the definition of the mdev type?  Dynamic migration support just
> seems like it's involving the kernel in orchestration it shouldn't be a
> part of.  Thanks,
>
> Alex
>   
>> Anthony Krowiak (23):
>>    s390/vfio-ap: Store queue hardware info when probed
>>    s390/vfio-ap: Provide access to queue objects and related info
>>    s390/vfio-ap: Add header file for xfer of vfio device caps to
>>      userspace
>>    MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
>>      maintainers
>>    s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
>>      migration of guest
>>    s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
>>    s390/vfio-ap: Signal event to enable/disable live guest migration
>>    s390/vfio-ap: Return value of sysfs migratable attribute from
>>      VFIO_DEVICE_GET_INFO ioctl
>>    s390/vfio-ap: Data structures for facilitating vfio device migration
>>    s390/vfio-ap: Initialize/release vfio device migration data
>>    s390-vfio-ap: Callback to set vfio device mig state during guest
>>      migration
>>    s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
>>    s390/vfio-ap: File ops called to save the vfio device migration state
>>    s390/vfio-ap: Transition device migration state from STOP to RESUMING
>>    s390/vfio-ap: File ops called to resume the vfio device migration
>>    s390/vfio-ap: Transition device migration state from RESUMING to STOP
>>    s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
>>    s390/vfio-ap: Transition device migration state from STOP to RUNNING
>>      and vice versa
>>    s390-vfio-ap: Callback to get the current vfio device migration state
>>    s390/vfio-ap: Callback to get the size of data to be migrated during
>>      guest migration
>>    s390/vfio-ap: Provide API to query whether migration is in progress
>>    s390/vfio-ap: Disallow blocking migration in progress
>>    s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
>>
>>   Documentation/arch/s390/vfio-ap.rst     |  339 +++++--
>>   MAINTAINERS                             |    1 +
>>   drivers/s390/crypto/Makefile            |    2 +-
>>   drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
>>   drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
>>   drivers/s390/crypto/vfio_ap_ops.c       |  263 +++++-
>>   drivers/s390/crypto/vfio_ap_private.h   |   20 +
>>   include/uapi/linux/vfio.h               |    2 +
>>   include/uapi/linux/vfio_ap.h            |   34 +
>>   9 files changed, 1685 insertions(+), 111 deletions(-)
>>   create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>>   create mode 100644 include/uapi/linux/vfio_ap.h
>>

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

On 3/30/26 12:27 PM, Alex Williamson wrote:
> On Wed, 25 Mar 2026 17:00:47 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> This patch series implements live guest migration of a guest to which AP
>> devices have been passed through. To better comprehend this design, one has
>> to understand that VFIO AP mediated device is not used to provide userspace
>> with direct access to a device as is the case with other devices that use
>> the VFIO framework to pass them through to a guest. The sole purpose of the
>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>> configuration is comprised of the AP adapter IDs (APID), AP queue
>> indexes (APQI) and domain numbers of the control domains to which a guest
>> will be granted access. Once the VFIO AP mediated device is attached to a
>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>> all access to the AP devices is handled by the s390 Interpretive Execution
>> facility; in other words, the vfio_ap device driver plays no role in
>> providing direct access to the AP devices in the guest's AP configuration.
>>
>> The only role that the vfio_ap device driver plays in the migration
>> process is to verify that the AP configuration for the source guest is
>> compatible with the AP configuration of the destination guest.
>> Incompatibility will result in a live guest migration failure.
>> In order to be compatible, the following requirements must be met:
>>
>> 1. The destination guest will be started with the same QEMU command line
>>     as the source guest, so the mediated device supplying the AP
>>     configuration on both guests must have the same name (UUID).
> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> supporting migration.  You're really concerned more with the
> configuration/composition of the mdev device, so requiring the same
> UUID seems a bit arbitrary.
>
>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>     guests must be compatible. As such, each AP configuration must meet
>>     the following requirements:
>>
>>     * Both guests must have the same number of APQNs
>>
>>     * Each APQN assigned to the source guest must also be assigned to the
>>       destination guest
>>
>>     * Each APQN assigned to both guests must reference an AP queue with the
>>       same hardware capabilities
> Why isn't this sufficient vs also requiring the same UUID?

I explained why in my previous response.

>        
>> Note: There is a forthcoming consumer of this series which will be a QEMU
>>        patch series is entitled:
>>        'hw/vfio/ap: Implement live guest migration of guests using AP
>>        devices'
>>
>> This design also adds a use case for enabling and disabling
>> migration of guests to which AP devices have been passed through. To
>> facilitate this, a new read/write sysfs 'migratable' attribute is added to
>> the mediated device. This attribute specifies whether the vfio device is
>> migratable (1) or not (0). When the value of this attribute is changed, the
>> vfio_ap device driver will signal an eventfd to userspace. It is up to
>> userspace to respond to the change by enabling or disabling migration of
>> the guest to which the mediated device is attached. The operation will be
>> rejected with a 'Device or resource busy' message if a migration is in
>> progress.
> This seems inherently racy.  What happens if the device becomes
> unmigratable while it's being migrated?
>
> If userspace is deciding that the device is now unmigratable, why does
> it need to signal this through the kernel driver rather than with the
> userspace orchestration agent?  The entire path seems unnecessary.

I am not familiar with what a userspace orchestration agent is, so
I can't address that. Can you please describe how that would work?

Maybe it would help to provide the reason for this. For certain types
of crypto operations, a master key must be configured for the crypto
card domain being used. This master key must be synchronized
between the source and destination crypto device so that in-flight
crypto operations can be completed during migration. If these master
keys must be changed, migration must be blocked until the master
key changes can be synchronized between the source and destination
system(s).

>
>> Userspace must also have a means for retrieving the value of the sysfs
>> 'migratable' attribute when the guest is started to initialize whether it
>> can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
>> struct vfio_device_info object passed to the ioctl will be extended with a
>> capability specifying the vfio device attributes. One of the attributes
>> will contain the value of the mediated device's 'migratable' attribute.
> This is just broken, it's redundant to our current device feature
> mechanism for exposing migration support.  If you want the capability
> to create unmigratable devices statically, can't that be encompassed
> within the definition of the mdev type?  Dynamic migration support just
> seems like it's involving the kernel in orchestration it shouldn't be a
> part of.

So, it appears you are suggesting the creation of a new mdev type
for unmigratable crypto devices. I don't see the value in that.
As I stated above, there is a valid reason for wanting to prevent
migration while master key synchronization is taking place.

If this feature violates the implicit rules of vfio device migration,
then so be it. Maybe we have to figure out another way to ensure
migration is not initiated during master key synchronization.

If we can't find an acceptable means to do this programmatically,
then maybe it will come down to a matter of documenting the
need to ensure migration is not initiated while master key
synchronization is taking place. This would put the onus on the
various system administrators responsible for host, guest and
master key administration to communicate out of band to
ensure they are all on the same page with regard to migration.

It would be preferable to be able to do this with a userspace
interface, so any suggestions would be greatly appreciated.

>    Thanks,
>
> Alex

Thank you for your review.

>   
>> Anthony Krowiak (23):
>>    s390/vfio-ap: Store queue hardware info when probed
>>    s390/vfio-ap: Provide access to queue objects and related info
>>    s390/vfio-ap: Add header file for xfer of vfio device caps to
>>      userspace
>>    MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
>>      maintainers
>>    s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
>>      migration of guest
>>    s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
>>    s390/vfio-ap: Signal event to enable/disable live guest migration
>>    s390/vfio-ap: Return value of sysfs migratable attribute from
>>      VFIO_DEVICE_GET_INFO ioctl
>>    s390/vfio-ap: Data structures for facilitating vfio device migration
>>    s390/vfio-ap: Initialize/release vfio device migration data
>>    s390-vfio-ap: Callback to set vfio device mig state during guest
>>      migration
>>    s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
>>    s390/vfio-ap: File ops called to save the vfio device migration state
>>    s390/vfio-ap: Transition device migration state from STOP to RESUMING
>>    s390/vfio-ap: File ops called to resume the vfio device migration
>>    s390/vfio-ap: Transition device migration state from RESUMING to STOP
>>    s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
>>    s390/vfio-ap: Transition device migration state from STOP to RUNNING
>>      and vice versa
>>    s390-vfio-ap: Callback to get the current vfio device migration state
>>    s390/vfio-ap: Callback to get the size of data to be migrated during
>>      guest migration
>>    s390/vfio-ap: Provide API to query whether migration is in progress
>>    s390/vfio-ap: Disallow blocking migration in progress
>>    s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
>>
>>   Documentation/arch/s390/vfio-ap.rst     |  339 +++++--
>>   MAINTAINERS                             |    1 +
>>   drivers/s390/crypto/Makefile            |    2 +-
>>   drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
>>   drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
>>   drivers/s390/crypto/vfio_ap_ops.c       |  263 +++++-
>>   drivers/s390/crypto/vfio_ap_private.h   |   20 +
>>   include/uapi/linux/vfio.h               |    2 +
>>   include/uapi/linux/vfio_ap.h            |   34 +
>>   9 files changed, 1685 insertions(+), 111 deletions(-)
>>   create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>>   create mode 100644 include/uapi/linux/vfio_ap.h
>>

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Alex Williamson]

On Tue, 31 Mar 2026 08:07:06 -0400
Anthony Krowiak <akrowiak@linux.ibm.com> wrote:

> On 3/30/26 12:27 PM, Alex Williamson wrote:
> > On Wed, 25 Mar 2026 17:00:47 -0400
> > Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
> >  
> >> This patch series implements live guest migration of a guest to which AP
> >> devices have been passed through. To better comprehend this design, one has
> >> to understand that VFIO AP mediated device is not used to provide userspace
> >> with direct access to a device as is the case with other devices that use
> >> the VFIO framework to pass them through to a guest. The sole purpose of the
> >> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
> >> configuration is comprised of the AP adapter IDs (APID), AP queue
> >> indexes (APQI) and domain numbers of the control domains to which a guest
> >> will be granted access. Once the VFIO AP mediated device is attached to a
> >> guest, its AP configuration is set by the vfio_ap device driver. Once set,
> >> all access to the AP devices is handled by the s390 Interpretive Execution
> >> facility; in other words, the vfio_ap device driver plays no role in
> >> providing direct access to the AP devices in the guest's AP configuration.
> >>
> >> The only role that the vfio_ap device driver plays in the migration
> >> process is to verify that the AP configuration for the source guest is
> >> compatible with the AP configuration of the destination guest.
> >> Incompatibility will result in a live guest migration failure.
> >> In order to be compatible, the following requirements must be met:
> >>
> >> 1. The destination guest will be started with the same QEMU command line
> >>     as the source guest, so the mediated device supplying the AP
> >>     configuration on both guests must have the same name (UUID).  
> > AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> > supporting migration.  You're really concerned more with the
> > configuration/composition of the mdev device, so requiring the same
> > UUID seems a bit arbitrary.

Combining replies:

On Tue, 31 Mar 2026 07:17:08 -0400
Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
> 
> As stated above, the destination guest will be started with the same
> QEMU command line as the source guest. Within that command line
> will be a '-device' parameter like the following:
> 
> -device 
> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
> 
> Note that sysfsdev is the path to the mdev named 
> 62177883-f1bb-47f0-914d-32a22e3a8804;
> therefore, the mdev with that name must exist on the destination guest or
> the migration will fail with the following error:
> 
> error: device not found: mediated device 
> '62177883-f1bb-47f0-914d-32a22e3a8804' not found

Then this is a requirement of your tooling, not a kernel requirement, not
something the kernel should care about.  QEMU matches devices by their
virtual bus path, not the sysfsdev or host attributes.  In the case of
VF migration with vfio-pci variant drivers we cannot require that the
source and target devices exist at the same bus address.  Ideally the
pre-copy data from the source device to the target will include relevant
configuration information to validate that the source and target are
compatible, regardless of the uuid.

> >> 2. The AP configuration assigned via the VFIO AP mediated device on both
> >>     guests must be compatible. As such, each AP configuration must meet
> >>     the following requirements:
> >>
> >>     * Both guests must have the same number of APQNs
> >>
> >>     * Each APQN assigned to the source guest must also be assigned to the
> >>       destination guest
> >>
> >>     * Each APQN assigned to both guests must reference an AP queue with the
> >>       same hardware capabilities  
> > Why isn't this sufficient vs also requiring the same UUID?  
> 
> I explained why in my previous response.

See above, userspace tooling requirements don't imply kernel
requirement.

> >> Note: There is a forthcoming consumer of this series which will be a QEMU
> >>        patch series is entitled:
> >>        'hw/vfio/ap: Implement live guest migration of guests using AP
> >>        devices'
> >>
> >> This design also adds a use case for enabling and disabling
> >> migration of guests to which AP devices have been passed through. To
> >> facilitate this, a new read/write sysfs 'migratable' attribute is added to
> >> the mediated device. This attribute specifies whether the vfio device is
> >> migratable (1) or not (0). When the value of this attribute is changed, the
> >> vfio_ap device driver will signal an eventfd to userspace. It is up to
> >> userspace to respond to the change by enabling or disabling migration of
> >> the guest to which the mediated device is attached. The operation will be
> >> rejected with a 'Device or resource busy' message if a migration is in
> >> progress.  
> > This seems inherently racy.  What happens if the device becomes
> > unmigratable while it's being migrated?
> >
> > If userspace is deciding that the device is now unmigratable, why does
> > it need to signal this through the kernel driver rather than with the
> > userspace orchestration agent?  The entire path seems unnecessary.  
> 
> I am not familiar with what a userspace orchestration agent is, so
> I can't address that. Can you please describe how that would work?

Something in userspace, perhaps libvirt, is managing the VM.  It needs
to coordinate with a counterpart on another host managing and
configuring the target VM to accept the migration data stream.  There's
likely also another entity that's responsible for deciding this
migration should occur and where to place the target.  All of this is
what I'm referring to as orchestration.

The path of a userspace agent writing into a sysfs attribute to mark
the device as becoming non-migratable so that the host driver can send
an interrupt to the VM to poll an INFO ioctl to block migration is an
over-engineered path at the wrong layer versus some sort of RPC to the
managing process, or even to QEMU, to block migrations.

Additionally, if the device actually itself becomes non-migratable, it
can simply fail any migration state transition other than returning to
RUNNING and can generate errors in the data stream if it needs to abort
an in-progress migration.

> Maybe it would help to provide the reason for this. For certain types
> of crypto operations, a master key must be configured for the crypto
> card domain being used. This master key must be synchronized
> between the source and destination crypto device so that in-flight
> crypto operations can be completed during migration. If these master
> keys must be changed, migration must be blocked until the master
> key changes can be synchronized between the source and destination
> system(s).

This sounds like a userspace orchestration problem, not a kernel
problem.  It might be a valid choice to use the mechanisms I outline
above to abort an in progress migration if a new master key is
configured during migration, but it's not the kernel's problem to
provide a synchronization point for this through the kernel.

> >> Userspace must also have a means for retrieving the value of the sysfs
> >> 'migratable' attribute when the guest is started to initialize whether it
> >> can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
> >> struct vfio_device_info object passed to the ioctl will be extended with a
> >> capability specifying the vfio device attributes. One of the attributes
> >> will contain the value of the mediated device's 'migratable' attribute.  
> > This is just broken, it's redundant to our current device feature
> > mechanism for exposing migration support.  If you want the capability
> > to create unmigratable devices statically, can't that be encompassed
> > within the definition of the mdev type?  Dynamic migration support just
> > seems like it's involving the kernel in orchestration it shouldn't be a
> > part of.  
> 
> So, it appears you are suggesting the creation of a new mdev type
> for unmigratable crypto devices. I don't see the value in that.
> As I stated above, there is a valid reason for wanting to prevent
> migration while master key synchronization is taking place.

Then prevent it in userspace.

> If this feature violates the implicit rules of vfio device migration,
> then so be it. Maybe we have to figure out another way to ensure
> migration is not initiated during master key synchronization.

If there's a software entity that has the ability to write to sysfs to
declare that a device is not currently available for migration, give it
the ability to notify whatever entity is coordinating and scheduling,
ie. orchestrating, the migration rather than creating a channel through
the device.  Combine that with using the existing mechanisms to abort a
migration if its already in progress.

> If we can't find an acceptable means to do this programmatically,
> then maybe it will come down to a matter of documenting the
> need to ensure migration is not initiated while master key
> synchronization is taking place. This would put the onus on the
> various system administrators responsible for host, guest and
> master key administration to communicate out of band to
> ensure they are all on the same page with regard to migration.
> 
> It would be preferable to be able to do this with a userspace
> interface, so any suggestions would be greatly appreciated.

A userspace interface can still exist, I just don't find this path
through the driver to the VM acceptable with this justification.
Mechanisms already exist for the device to refuse a state transition or
generate an error for a migration already in progress.  IMHO, it would
be acceptable for the device to block a key change if the migration is
already in progress.  If the key change cannot be represented in the
migration data stream, then it's up to the orchestration of the
migration to make sure they stay synchronized, but I don't see that
the vfio uAPI needs to be involved.  Thanks,

Alex

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

On 3/31/26 1:40 PM, Alex Williamson wrote:
> On Tue, 31 Mar 2026 08:07:06 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> On 3/30/26 12:27 PM, Alex Williamson wrote:
>>> On Wed, 25 Mar 2026 17:00:47 -0400
>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>   
>>>> This patch series implements live guest migration of a guest to which AP
>>>> devices have been passed through. To better comprehend this design, one has
>>>> to understand that VFIO AP mediated device is not used to provide userspace
>>>> with direct access to a device as is the case with other devices that use
>>>> the VFIO framework to pass them through to a guest. The sole purpose of the
>>>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>>>> configuration is comprised of the AP adapter IDs (APID), AP queue
>>>> indexes (APQI) and domain numbers of the control domains to which a guest
>>>> will be granted access. Once the VFIO AP mediated device is attached to a
>>>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>>>> all access to the AP devices is handled by the s390 Interpretive Execution
>>>> facility; in other words, the vfio_ap device driver plays no role in
>>>> providing direct access to the AP devices in the guest's AP configuration.
>>>>
>>>> The only role that the vfio_ap device driver plays in the migration
>>>> process is to verify that the AP configuration for the source guest is
>>>> compatible with the AP configuration of the destination guest.
>>>> Incompatibility will result in a live guest migration failure.
>>>> In order to be compatible, the following requirements must be met:
>>>>
>>>> 1. The destination guest will be started with the same QEMU command line
>>>>      as the source guest, so the mediated device supplying the AP
>>>>      configuration on both guests must have the same name (UUID).
>>> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
>>> supporting migration.  You're really concerned more with the
>>> configuration/composition of the mdev device, so requiring the same
>>> UUID seems a bit arbitrary.
> Combining replies:
>
> On Tue, 31 Mar 2026 07:17:08 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>> As stated above, the destination guest will be started with the same
>> QEMU command line as the source guest. Within that command line
>> will be a '-device' parameter like the following:
>>
>> -device
>> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
>>
>> Note that sysfsdev is the path to the mdev named
>> 62177883-f1bb-47f0-914d-32a22e3a8804;
>> therefore, the mdev with that name must exist on the destination guest or
>> the migration will fail with the following error:
>>
>> error: device not found: mediated device
>> '62177883-f1bb-47f0-914d-32a22e3a8804' not found
> Then this is a requirement of your tooling, not a kernel requirement, not
> something the kernel should care about.  QEMU matches devices by their
> virtual bus path, not the sysfsdev or host attributes.  In the case of
> VF migration with vfio-pci variant drivers we cannot require that the
> source and target devices exist at the same bus address.  Ideally the
> pre-copy data from the source device to the target will include relevant
> configuration information to validate that the source and target are
> compatible, regardless of the uuid.

Maybe the problem here is stating that having the same UUID is a
requirement in the patch series description. I agree this is
not a kernel requirement; however, a live guest migration will not
succeed unless the destination host has a mediated device with
a UUID that matches that of the UUID of the mediated device used
to pass through s390 crypto devices to the source guest for the
reason I stated above. Would it help if I removed item #1 as a
requirement?

>
>>>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>>>      guests must be compatible. As such, each AP configuration must meet
>>>>      the following requirements:
>>>>
>>>>      * Both guests must have the same number of APQNs
>>>>
>>>>      * Each APQN assigned to the source guest must also be assigned to the
>>>>        destination guest
>>>>
>>>>      * Each APQN assigned to both guests must reference an AP queue with the
>>>>        same hardware capabilities
>>> Why isn't this sufficient vs also requiring the same UUID?
>> I explained why in my previous response.
> See above, userspace tooling requirements don't imply kernel
> requirement.

Got it.

> A userspace interface can still exist, I just don't find this path
> through the driver to the VM acceptable with this justification.
> Mechanisms already exist for the device to refuse a state transition or
> generate an error for a migration already in progress.  IMHO, it would
> be acceptable for the device to block a key change if the migration is
> already in progress.  If the key change cannot be represented in the
> migration data stream, then it's up to the orchestration of the
> migration to make sure they stay synchronized, but I don't see that
> the vfio uAPI needs to be involved.  Thanks,

As I stated in the cover letter description, all access to the AP 
devices is handled by the s390 Interpretive Execution facility, so 
without a complete redesign of the vfio_ap device driver, there is no 
way to know that a master key change is being requested. I do get your 
point, however, so I will figure out how best to handle this in 
userspace. Thanks for you review.
>
> Alex

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Alex Williamson]

On Wed, 1 Apr 2026 09:38:59 -0400
Anthony Krowiak <akrowiak@linux.ibm.com> wrote:

> On 3/31/26 1:40 PM, Alex Williamson wrote:
> > On Tue, 31 Mar 2026 08:07:06 -0400
> > Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
> >  
> >> On 3/30/26 12:27 PM, Alex Williamson wrote:  
> >>> On Wed, 25 Mar 2026 17:00:47 -0400
> >>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
> >>>     
> >>>> This patch series implements live guest migration of a guest to which AP
> >>>> devices have been passed through. To better comprehend this design, one has
> >>>> to understand that VFIO AP mediated device is not used to provide userspace
> >>>> with direct access to a device as is the case with other devices that use
> >>>> the VFIO framework to pass them through to a guest. The sole purpose of the
> >>>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
> >>>> configuration is comprised of the AP adapter IDs (APID), AP queue
> >>>> indexes (APQI) and domain numbers of the control domains to which a guest
> >>>> will be granted access. Once the VFIO AP mediated device is attached to a
> >>>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
> >>>> all access to the AP devices is handled by the s390 Interpretive Execution
> >>>> facility; in other words, the vfio_ap device driver plays no role in
> >>>> providing direct access to the AP devices in the guest's AP configuration.
> >>>>
> >>>> The only role that the vfio_ap device driver plays in the migration
> >>>> process is to verify that the AP configuration for the source guest is
> >>>> compatible with the AP configuration of the destination guest.
> >>>> Incompatibility will result in a live guest migration failure.
> >>>> In order to be compatible, the following requirements must be met:
> >>>>
> >>>> 1. The destination guest will be started with the same QEMU command line
> >>>>      as the source guest, so the mediated device supplying the AP
> >>>>      configuration on both guests must have the same name (UUID).  
> >>> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> >>> supporting migration.  You're really concerned more with the
> >>> configuration/composition of the mdev device, so requiring the same
> >>> UUID seems a bit arbitrary.  
> > Combining replies:
> >
> > On Tue, 31 Mar 2026 07:17:08 -0400
> > Anthony Krowiak <akrowiak@linux.ibm.com> wrote:  
> >> As stated above, the destination guest will be started with the same
> >> QEMU command line as the source guest. Within that command line
> >> will be a '-device' parameter like the following:
> >>
> >> -device
> >> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
> >>
> >> Note that sysfsdev is the path to the mdev named
> >> 62177883-f1bb-47f0-914d-32a22e3a8804;
> >> therefore, the mdev with that name must exist on the destination guest or
> >> the migration will fail with the following error:
> >>
> >> error: device not found: mediated device
> >> '62177883-f1bb-47f0-914d-32a22e3a8804' not found  
> > Then this is a requirement of your tooling, not a kernel requirement, not
> > something the kernel should care about.  QEMU matches devices by their
> > virtual bus path, not the sysfsdev or host attributes.  In the case of
> > VF migration with vfio-pci variant drivers we cannot require that the
> > source and target devices exist at the same bus address.  Ideally the
> > pre-copy data from the source device to the target will include relevant
> > configuration information to validate that the source and target are
> > compatible, regardless of the uuid.  
> 
> Maybe the problem here is stating that having the same UUID is a
> requirement in the patch series description. I agree this is
> not a kernel requirement; however, a live guest migration will not
> succeed unless the destination host has a mediated device with
> a UUID that matches that of the UUID of the mediated device used
> to pass through s390 crypto devices to the source guest for the
> reason I stated above. Would it help if I removed item #1 as a
> requirement?

Yes, AIUI this is an artifact of your tooling.  The destination QEMU
can be started with any arbitrary UUID and should accept an incoming
migration, routing the migration to the correct device by the virtual
path, independent of the UUID.

> >>>> 2. The AP configuration assigned via the VFIO AP mediated device on both
> >>>>      guests must be compatible. As such, each AP configuration must meet
> >>>>      the following requirements:
> >>>>
> >>>>      * Both guests must have the same number of APQNs
> >>>>
> >>>>      * Each APQN assigned to the source guest must also be assigned to the
> >>>>        destination guest
> >>>>
> >>>>      * Each APQN assigned to both guests must reference an AP queue with the
> >>>>        same hardware capabilities  
> >>> Why isn't this sufficient vs also requiring the same UUID?  
> >> I explained why in my previous response.  
> > See above, userspace tooling requirements don't imply kernel
> > requirement.  
> 
> Got it.
> 
> > A userspace interface can still exist, I just don't find this path
> > through the driver to the VM acceptable with this justification.
> > Mechanisms already exist for the device to refuse a state transition or
> > generate an error for a migration already in progress.  IMHO, it would
> > be acceptable for the device to block a key change if the migration is
> > already in progress.  If the key change cannot be represented in the
> > migration data stream, then it's up to the orchestration of the
> > migration to make sure they stay synchronized, but I don't see that
> > the vfio uAPI needs to be involved.  Thanks,  
> 
> As I stated in the cover letter description, all access to the AP 
> devices is handled by the s390 Interpretive Execution facility, so 
> without a complete redesign of the vfio_ap device driver, there is no 
> way to know that a master key change is being requested. I do get your 
> point, however, so I will figure out how best to handle this in 
> userspace. Thanks for you review.

The cover letter has one sentence describing the Interpretive Execution
facility handling access to the device.  There are a lot of subtleties
there that are not obvious to laypersons.

It seems like the architecture really doesn't have the ability
to monitor state changes of the device, and therefore represent those
changes in the migration data stream, so you want to fallback to a
cooperative scheme with a side-band channel to interrupt or block
migration when an asynchronous configuration change happens.

I just don't see that such cooperation needs a new uAPI through the
device; either the device itself could perform the same within the
existing uAPI or the agent changing the configuration could coordinate
with the migration orchestration.  Thanks,

Alex

Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

On 4/1/26 12:57 PM, Alex Williamson wrote:
> On Wed, 1 Apr 2026 09:38:59 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> On 3/31/26 1:40 PM, Alex Williamson wrote:
>>> On Tue, 31 Mar 2026 08:07:06 -0400
>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>   
>>>> On 3/30/26 12:27 PM, Alex Williamson wrote:
>>>>> On Wed, 25 Mar 2026 17:00:47 -0400
>>>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>>>      
>>>>>> This patch series implements live guest migration of a guest to which AP
>>>>>> devices have been passed through. To better comprehend this design, one has
>>>>>> to understand that VFIO AP mediated device is not used to provide userspace
>>>>>> with direct access to a device as is the case with other devices that use
>>>>>> the VFIO framework to pass them through to a guest. The sole purpose of the
>>>>>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>>>>>> configuration is comprised of the AP adapter IDs (APID), AP queue
>>>>>> indexes (APQI) and domain numbers of the control domains to which a guest
>>>>>> will be granted access. Once the VFIO AP mediated device is attached to a
>>>>>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>>>>>> all access to the AP devices is handled by the s390 Interpretive Execution
>>>>>> facility; in other words, the vfio_ap device driver plays no role in
>>>>>> providing direct access to the AP devices in the guest's AP configuration.
>>>>>>
>>>>>> The only role that the vfio_ap device driver plays in the migration
>>>>>> process is to verify that the AP configuration for the source guest is
>>>>>> compatible with the AP configuration of the destination guest.
>>>>>> Incompatibility will result in a live guest migration failure.
>>>>>> In order to be compatible, the following requirements must be met:
>>>>>>
>>>>>> 1. The destination guest will be started with the same QEMU command line
>>>>>>       as the source guest, so the mediated device supplying the AP
>>>>>>       configuration on both guests must have the same name (UUID).
>>>>> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
>>>>> supporting migration.  You're really concerned more with the
>>>>> configuration/composition of the mdev device, so requiring the same
>>>>> UUID seems a bit arbitrary.
>>> Combining replies:
>>>
>>> On Tue, 31 Mar 2026 07:17:08 -0400
>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>> As stated above, the destination guest will be started with the same
>>>> QEMU command line as the source guest. Within that command line
>>>> will be a '-device' parameter like the following:
>>>>
>>>> -device
>>>> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
>>>>
>>>> Note that sysfsdev is the path to the mdev named
>>>> 62177883-f1bb-47f0-914d-32a22e3a8804;
>>>> therefore, the mdev with that name must exist on the destination guest or
>>>> the migration will fail with the following error:
>>>>
>>>> error: device not found: mediated device
>>>> '62177883-f1bb-47f0-914d-32a22e3a8804' not found
>>> Then this is a requirement of your tooling, not a kernel requirement, not
>>> something the kernel should care about.  QEMU matches devices by their
>>> virtual bus path, not the sysfsdev or host attributes.  In the case of
>>> VF migration with vfio-pci variant drivers we cannot require that the
>>> source and target devices exist at the same bus address.  Ideally the
>>> pre-copy data from the source device to the target will include relevant
>>> configuration information to validate that the source and target are
>>> compatible, regardless of the uuid.
>> Maybe the problem here is stating that having the same UUID is a
>> requirement in the patch series description. I agree this is
>> not a kernel requirement; however, a live guest migration will not
>> succeed unless the destination host has a mediated device with
>> a UUID that matches that of the UUID of the mediated device used
>> to pass through s390 crypto devices to the source guest for the
>> reason I stated above. Would it help if I removed item #1 as a
>> requirement?
> Yes, AIUI this is an artifact of your tooling.  The destination QEMU
> can be started with any arbitrary UUID and should accept an incoming
> migration, routing the migration to the correct device by the virtual
> path, independent of the UUID.

If by tooling you are referring to libvirt and/or quemu, then I agree;
this is an artifact of tooling. According to
https://www.linux-kvm.org/page/Migration#Requirements,
"the guest on the destination must be started the same way it was
started on the source". s390 guests are run under qemu, so the
qemu command line used to start the source guest will also be used
to start the destination guest. To pass through
AP devices to the guest, the mediated device that provides the guest's
AP configuration - i.e., the crypto devices to which the guest is granted
access - must be identified on the qemu command line. As stated
previously, the following qemu command line parameter is included
for this purpose:


-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}

When the qemu command is used to start the destination guest, the same
qemu command line will be used to start it. As such, qemu will try to open
an fd for that device. If the mdev with UUID does not exist, the migration
will fail. In other words, I don't think the guest can be started with any
arbitrary UUID. Am I wrong about this?

In any case, it is probably not relevant to mention this for this patch 
series
since it is not a KVM/kernel requirement, so I will remove it from the
cover-letter description.

>
>>>>>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>>>>>       guests must be compatible. As such, each AP configuration must meet
>>>>>>       the following requirements:
>>>>>>
>>>>>>       * Both guests must have the same number of APQNs
>>>>>>
>>>>>>       * Each APQN assigned to the source guest must also be assigned to the
>>>>>>         destination guest
>>>>>>
>>>>>>       * Each APQN assigned to both guests must reference an AP queue with the
>>>>>>         same hardware capabilities
>>>>> Why isn't this sufficient vs also requiring the same UUID?
>>>> I explained why in my previous response.
>>> See above, userspace tooling requirements don't imply kernel
>>> requirement.
>> Got it.
>>
>>> A userspace interface can still exist, I just don't find this path
>>> through the driver to the VM acceptable with this justification.
>>> Mechanisms already exist for the device to refuse a state transition or
>>> generate an error for a migration already in progress.  IMHO, it would
>>> be acceptable for the device to block a key change if the migration is
>>> already in progress.  If the key change cannot be represented in the
>>> migration data stream, then it's up to the orchestration of the
>>> migration to make sure they stay synchronized, but I don't see that
>>> the vfio uAPI needs to be involved.  Thanks,
>> As I stated in the cover letter description, all access to the AP
>> devices is handled by the s390 Interpretive Execution facility, so
>> without a complete redesign of the vfio_ap device driver, there is no
>> way to know that a master key change is being requested. I do get your
>> point, however, so I will figure out how best to handle this in
>> userspace. Thanks for you review.
> The cover letter has one sentence describing the Interpretive Execution
> facility handling access to the device.  There are a lot of subtleties
> there that are not obvious to laypersons.
>
> It seems like the architecture really doesn't have the ability
> to monitor state changes of the device, and therefore represent those
> changes in the migration data stream, so you want to fallback to a
> cooperative scheme with a side-band channel to interrupt or block
> migration when an asynchronous configuration change happens.
>
> I just don't see that such cooperation needs a new uAPI through the
> device; either the device itself could perform the same within the
> existing uAPI or the agent changing the configuration could coordinate
> with the migration orchestration.  Thanks,

I understand your view on this and have already agreed to remove
the new uAPI.

>
> Alex