[PATCH v2 00/16] s390/vfio-ap: Add live guest migration support

[PATCH v2 00/16] s390/vfio-ap: Add live guest migration support

[Anthony Krowiak]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 5159 bytes --]

This patch series implements live guest migration support for KVM guests
with s390 AP (Adjunct Processor) devices passed through via the VFIO
mediated device framework.

Background
----------

The vfio-ap device driver differs from typical VFIO device drivers in that
it does not virtualize a physical device. Instead, it manages AP
configuration metadata identifying the AP adapters, domains, and control
domains to which a guest will be granted access. These AP resources are 
configured by assigning them to a vfio-ap mediated device via its sysfs
assignment interfaces. When the fd for the VFIO device is opened by 
userspace, the vfio_ap device driver sets the guest's AP configuration
from the metadata stored with the mediated device. As such, the AP devices
are not accessed directly through the vfio_ap driver, so the driver has no
internal device state to migrate. It's sole purpose during migration is to
ensure that the AP configurations of the source and destination guests are
compatible.

Implementation Approach
-----------------------

This series implements the VFIO migration protocol using the STOP_COPY
migration flow. The key aspects are:

1. Hardware Information Capture (Patches 1-2)
   - Store AP queue hardware characteristics at probe time
   - Provide access to queue objects for validation

2. Migration Infrastructure (Patches 3-5)
   - Define migration data structures
   - Initialize/release migration data on device open/close

3. State Machine Implementation (Patches 6-13)
   - Implement required VFIO migration callbacks
   - Handle state transitions: STOP → STOP_COPY → STOP (source)
                               STOP → RESUMING → STOP (destination)
   - Use file streams for AP configuration data transfer

4. Validation and Callbacks (Patches 14-15)
   - Implement migration state and data size callbacks
   - Required for VFIO_DEVICE_FEATURE_MIGRATION support

5. Documentation (Patch 16)
   - Add live guest migration chapter to vfio-ap.rst

Compatibility Validation
------------------------

The series includes comprehensive validation to ensure source and
destination AP configurations are compatible. For each queue, the following
characteristics must match:

- AP type (target must be same or newer than source)
- Installed facilities (APSC, APQKM, AP4KC, SLCF)
- Operating mode (CCA, Accelerator, XCP)
- APXA facility setting
- Classification (native vs stateless functions)
- Queue usability (binding/associated state)

When incompatibilities are detected, migration fails with detailed error
messages identifying the specific queue and characteristic that caused
the failure.

Configuration Management
------------------------

This implementation does not prevent configuration changes during
migration. Configuration stability is an orchestration-layer
responsibility, consistent with other VFIO device types. The driver's
role is to validate configurations and provide clear diagnostics when
incompatibilities are detected, enabling orchestration tools to implement
appropriate policies.

Change log v1 to v2
-------------------

- Removed patches that attempted to block configuration changes during
  migration due to inherent race conditions and incomplete protection
- Simplified approach focuses on validation and error reporting
- Reduced series from 18 to 16 patches
- Rewrote the description in the cover letter to better describe the
  patch series, remove confusing comments as well as references to
  function provided by the patches that were removed.

Anthony Krowiak (16):
  s390/vfio-ap: Store queue hardware info when probed
  s390/vfio-ap: Provide access to queue objects and related info
  s390/vfio-ap: Data structures for facilitating vfio device migration
  s390/vfio-ap: Initialize/release vfio device migration data
  s390-vfio-ap: Callback to set vfio device mig state during guest
    migration
  s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
  s390/vfio-ap: File ops called to save the vfio device migration state
  s390/vfio-ap: Transition device migration state from STOP to RESUMING
  s390/vfio-ap: File ops called to resume the vfio device migration
  s390/vfio-ap: Transition device migration state from RESUMING to STOP
  s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
  s390/vfio-ap: Transition device migration state from STOP to RUNNING
    and vice versa
  s390-vfio-ap: Callback to get the current vfio device migration state
  s390/vfio-ap: Callback to get the size of data to be migrated during
    guest migration
  s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
  s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst

 Documentation/arch/s390/vfio-ap.rst     |  325 +++++--
 drivers/s390/crypto/Makefile            |    2 +-
 drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
 drivers/s390/crypto/vfio_ap_migration.c | 1095 +++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_ops.c       |   66 +-
 drivers/s390/crypto/vfio_ap_private.h   |   10 +
 6 files changed, 1395 insertions(+), 107 deletions(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_migration.c

-- 
2.52.0

[PATCH v2 01/16] s390/vfio-ap: Store queue hardware info when probed

[Anthony Krowiak]

Store the output of the PQAP(TAPQ) command with the struct vfio_ap_queue
object when a queue device is probed. This data can be retrieved from the
object passed to the probe callback.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c     | 5 ++++-
 drivers/s390/crypto/vfio_ap_private.h | 1 +
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 44b3a1dcc1b3..ef3101f6865e 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -2419,6 +2419,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 	struct vfio_ap_queue *q;
 	DECLARE_BITMAP(apm_filtered, AP_DEVICES);
 	struct ap_matrix_mdev *matrix_mdev;
+	struct ap_queue *ap_queue;
 
 	ret = sysfs_create_group(&apdev->device.kobj, &vfio_queue_attr_group);
 	if (ret)
@@ -2430,8 +2431,10 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 		goto err_remove_group;
 	}
 
-	q->apqn = to_ap_queue(&apdev->device)->qid;
+	ap_queue = to_ap_queue(&apdev->device);
+	q->apqn = ap_queue->qid;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
+	memcpy(&q->hwinfo, &ap_queue->card->hwinfo, sizeof(q->hwinfo));
 	memset(&q->reset_status, 0, sizeof(q->reset_status));
 	INIT_WORK(&q->reset_work, apq_reset_check);
 	matrix_mdev = get_update_locks_by_apqn(q->apqn);
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 9bff666b0b35..39c85bab05c3 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -150,6 +150,7 @@ struct vfio_ap_queue {
 	struct list_head reset_qnode;
 	struct ap_queue_status reset_status;
 	struct work_struct reset_work;
+	struct ap_tapq_hwinfo hwinfo;
 };
 
 int vfio_ap_mdev_register(void);
-- 
2.52.0

[PATCH v2 02/16] s390/vfio-ap: Provide access to queue objects and related info

[Anthony Krowiak]

Provide the following:

1. A function that returns a struct vfio_ap_queue object representing a
   queue that is bound to the vfio_ap device driver and is assigned to an
   mdev

2. A function that returns the number of queues that are or will be passed
   through to a guest when the mdev is attached to a guest.

There is already a function for #1; however, it is a static function in
drivers/s390/crypto/vfio_ap_ops.c. The function will be defined in the
drivers/s390/crypto/vfio_ap_private.h file and made non-static. Note that
it would probably make sense to ensure that the mdevs_lock is held for
this function, so a call to lockdep_assert_held(&matrix_dev->mdevs_lock)
will be added to the function implementation.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_ops.c     | 20 +++++++++++++++++---
 drivers/s390/crypto/vfio_ap_private.h |  4 ++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index ef3101f6865e..cd85866b81a0 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -203,12 +203,12 @@ static inline void get_update_locks_for_queue(struct vfio_ap_queue *q)
  * Return: the pointer to the vfio_ap_queue struct representing the queue or
  *	   NULL if the queue is not assigned to @matrix_mdev
  */
-static struct vfio_ap_queue *vfio_ap_mdev_get_queue(
-					struct ap_matrix_mdev *matrix_mdev,
-					int apqn)
+struct vfio_ap_queue *vfio_ap_mdev_get_queue(struct ap_matrix_mdev *matrix_mdev, int apqn)
 {
 	struct vfio_ap_queue *q;
 
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
 	hash_for_each_possible(matrix_mdev->qtable.queues, q, mdev_qnode,
 			       apqn) {
 		if (q && q->apqn == apqn)
@@ -2098,6 +2098,20 @@ static void vfio_ap_mdev_request(struct vfio_device *vdev, unsigned int count)
 	release_update_locks_for_mdev(matrix_mdev);
 }
 
+int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix)
+{
+	unsigned long apid, apqi;
+	int num_queues = 0;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	for_each_set_bit_inv(apid, ap_matrix->apm, AP_DEVICES)
+		for_each_set_bit_inv(apqi, ap_matrix->aqm, AP_DOMAINS)
+			num_queues++;
+
+	return num_queues;
+}
+
 static int vfio_ap_mdev_get_device_info(unsigned long arg)
 {
 	unsigned long minsz;
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 39c85bab05c3..d66586825ef4 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -153,6 +153,10 @@ struct vfio_ap_queue {
 	struct ap_tapq_hwinfo hwinfo;
 };
 
+struct vfio_ap_queue *vfio_ap_mdev_get_queue(struct ap_matrix_mdev *matrix_mdev, int apqn);
+
+int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix);
+
 int vfio_ap_mdev_register(void);
 void vfio_ap_mdev_unregister(void);
 
-- 
2.52.0

[PATCH v2 03/16] s390/vfio-ap: Data structures for facilitating vfio device migration

[Anthony Krowiak]

Creates the data structures used to facilitate state transitions during
vfio device migration.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/Makefile            |  2 +-
 drivers/s390/crypto/vfio_ap_migration.c | 68 +++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_private.h   |  2 +
 3 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_migration.c

diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index e83c6603c858..20f29184825a 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -34,5 +34,5 @@ pkey-uv-objs := pkey_uv.o
 obj-$(CONFIG_PKEY_UV) += pkey-uv.o
 
 # adjunct processor matrix
-vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o
+vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o vfio_ap_migration.o
 obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
new file mode 100644
index 000000000000..2c5978dde61d
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Drives vfio_ap mdev migration.
+ *
+ * Copyright IBM Corp. 2025
+ */
+#include "uapi/linux/vfio_ap.h"
+
+#include "vfio_ap_private.h"
+
+/**
+ * vfio_ap_migration_data - the data needed to migrate a guest with pass-through
+ *			    access to AP devices
+ *
+ * @mig_state:		the current migration state
+ * @resuming_migf:	the object used to resume the target guest
+ * @saving_migf:	the object used to save the state of the source guest
+ */
+struct vfio_ap_migration_data {
+	enum vfio_device_mig_state	mig_state;
+	struct vfio_ap_migration_file	*resuming_migf;
+	struct vfio_ap_migration_file	*saving_migf;
+};
+
+/**
+ * vfio_ap_queue_info - the information for an AP queue
+ *
+ * @data: contains the queue information returned in GR2 from the PQAP(TAPQ)
+ *	  command
+ * @apqn: the APQN of the queue
+ */
+struct vfio_ap_queue_info {
+	u64 data;
+	u16 apqn;
+};
+
+/**
+ * vfio_ap_config - the guest's AP configuration
+ *
+ * @num_queues:	the number of queues passed through to the guest
+ * @qinfo:	an array of vfio_ap_queue_info objects, each specifying the
+ *		queue information for a queue passed through to the guest
+ */
+struct vfio_ap_config {
+	unsigned int			num_queues;
+	struct vfio_ap_queue_info	qinfo[];
+};
+
+/**
+ * vfio_ap_migration_file - object used to facilitate migration of a guest with
+ *			    pass-through access to AP devices
+ *
+ * @matrix_mdev: the mediated device attached to the guest being migrated
+ * @filp:	 the file used to facilitate communication between userspace
+ *		 and the vfio_ap device driver during a particular phase of
+ *		 the migration
+ * @disabled:	 boolean value indicating whether this object is disabled (true)
+ *		 or not (false)
+ * @ap_config:	 the information for each queue passed through to a guest
+ * @config_sz:	 the size of @ap_config when filled with queue information
+ */
+struct vfio_ap_migration_file {
+	struct ap_matrix_mdev	*matrix_mdev;
+	struct file		*filp;
+	bool			disabled;
+	struct vfio_ap_config	*ap_config;
+	size_t			config_sz;
+};
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index d66586825ef4..a1bc50a59633 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -110,6 +110,7 @@ struct ap_queue_table {
  * @aqm_add:	bitmap of APQIs added to the host's AP configuration
  * @adm_add:	bitmap of control domain numbers added to the host's AP
  *		configuration
+ * @mig_data:  vfio device migration data
  */
 struct ap_matrix_mdev {
 	struct vfio_device vdev;
@@ -125,6 +126,7 @@ struct ap_matrix_mdev {
 	DECLARE_BITMAP(apm_add, AP_DEVICES);
 	DECLARE_BITMAP(aqm_add, AP_DOMAINS);
 	DECLARE_BITMAP(adm_add, AP_DOMAINS);
+	void *mig_data;
 };
 
 /**
-- 
2.52.0

[PATCH v2 04/16] s390/vfio-ap: Initialize/release vfio device migration data

[Anthony Krowiak]

Provides the functions that initialize and release the data structures
used during live guest migration. These functions should
be called when the mdev fd is opened and closed respectively. The function
to release the data structures shall also be invoked when the mdev
is released.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 94 +++++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_ops.c       | 41 +++++++++--
 drivers/s390/crypto/vfio_ap_private.h   |  3 +
 3 files changed, 131 insertions(+), 7 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 2c5978dde61d..a1ae4f1a3473 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -4,6 +4,7 @@
  *
  * Copyright IBM Corp. 2025
  */
+#include <linux/file.h>
 #include "uapi/linux/vfio_ap.h"
 
 #include "vfio_ap_private.h"
@@ -66,3 +67,96 @@ struct vfio_ap_migration_file {
 	struct vfio_ap_config	*ap_config;
 	size_t			config_sz;
 };
+
+static void vfio_ap_disable_file(struct vfio_ap_migration_file *migf)
+{
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	migf->matrix_mdev = NULL;
+	migf->disabled = true;
+	migf->filp->f_pos = 0;
+}
+
+static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	if (mig_data->resuming_migf) {
+		vfio_ap_disable_file(mig_data->resuming_migf);
+		fput(mig_data->resuming_migf->filp);
+		mig_data->resuming_migf = NULL;
+	}
+
+	if (mig_data->saving_migf) {
+		vfio_ap_disable_file(mig_data->saving_migf);
+		fput(mig_data->saving_migf->filp);
+		mig_data->saving_migf = NULL;
+	}
+}
+
+static struct file *vfio_ap_set_state(struct vfio_device *vdev,
+				      enum vfio_device_mig_state  new_state)
+{
+	return NULL;
+}
+
+static int vfio_ap_get_state(struct vfio_device *vdev,
+			     enum vfio_device_mig_state  *current_state)
+{
+	return -EOPNOTSUPP;
+}
+
+static int vfio_ap_get_data_size(struct vfio_device *vdev,
+				 unsigned long *stop_copy_length)
+{
+	return -EOPNOTSUPP;
+}
+
+static const struct vfio_migration_ops vfio_ap_migration_ops = {
+	.migration_set_state = vfio_ap_set_state,
+	.migration_get_state = vfio_ap_get_state,
+	.migration_get_data_size = vfio_ap_get_data_size,
+};
+
+/**
+ * vfio_ap_init_migration_data - initialize migration data and functions
+ *
+ * @matrix_mdev: pointer to object containing the mdev state
+ *
+ * Return: zero if initialization is successful; otherwise, returns a error.
+ */
+int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	mig_data = kzalloc_obj(struct vfio_ap_migration_data, GFP_KERNEL);
+	if (!mig_data)
+		return -ENOMEM;
+
+	mig_data->mig_state = VFIO_DEVICE_STATE_RUNNING;
+	matrix_mdev->vdev.migration_flags = VFIO_MIGRATION_STOP_COPY;
+	matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops;
+	matrix_mdev->mig_data = mig_data;
+
+	return 0;
+}
+
+/**
+ * vfio_ap_release_migration_data: reclaim private migration data
+ *
+ * @vdev: pointer to the mdev
+ */
+void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev)
+{
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	if (!matrix_mdev->mig_data)
+		return;
+
+	vfio_ap_release_mig_files(matrix_mdev);
+	kfree(matrix_mdev->mig_data);
+	matrix_mdev->mig_data = NULL;
+}
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index cd85866b81a0..aafcdc57f6e2 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -775,18 +775,30 @@ static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
 
 static int vfio_ap_mdev_init_dev(struct vfio_device *vdev)
 {
-	struct ap_matrix_mdev *matrix_mdev =
-		container_of(vdev, struct ap_matrix_mdev, vdev);
+	struct ap_matrix_mdev *matrix_mdev;
 
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
 	matrix_mdev->mdev = to_mdev_device(vdev->dev);
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
 	matrix_mdev->pqap_hook = handle_pqap;
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
 	hash_init(matrix_mdev->qtable.queues);
+	mutex_unlock(&matrix_dev->mdevs_lock);
 
 	return 0;
 }
 
+static void vfio_ap_mdev_release_dev(struct vfio_device *vdev)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	vfio_ap_release_migration_data(matrix_mdev);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+}
+
 static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 {
 	struct ap_matrix_mdev *matrix_mdev;
@@ -2052,19 +2064,33 @@ static int vfio_ap_mdev_reset_qlist(struct list_head *qlist)
 
 static int vfio_ap_mdev_open_device(struct vfio_device *vdev)
 {
-	struct ap_matrix_mdev *matrix_mdev =
-		container_of(vdev, struct ap_matrix_mdev, vdev);
+	struct ap_matrix_mdev *matrix_mdev;
+	int ret;
 
 	if (!vdev->kvm)
 		return -EINVAL;
 
-	return vfio_ap_mdev_set_kvm(matrix_mdev, vdev->kvm);
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	ret = vfio_ap_init_migration_data(matrix_mdev);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	if (ret)
+		return ret;
+
+	ret = vfio_ap_mdev_set_kvm(matrix_mdev, vdev->kvm);
+
+	return ret;
 }
 
 static void vfio_ap_mdev_close_device(struct vfio_device *vdev)
 {
-	struct ap_matrix_mdev *matrix_mdev =
-		container_of(vdev, struct ap_matrix_mdev, vdev);
+	struct ap_matrix_mdev *matrix_mdev;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	vfio_ap_release_migration_data(matrix_mdev);
+	mutex_unlock(&matrix_dev->mdevs_lock);
 
 	vfio_ap_mdev_unset_kvm(matrix_mdev);
 }
@@ -2374,6 +2400,7 @@ static const struct attribute_group vfio_queue_attr_group = {
 
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops = {
 	.init = vfio_ap_mdev_init_dev,
+	.release = vfio_ap_mdev_release_dev,
 	.open_device = vfio_ap_mdev_open_device,
 	.close_device = vfio_ap_mdev_close_device,
 	.ioctl = vfio_ap_mdev_ioctl,
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index a1bc50a59633..7e21b5b5f39e 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -172,4 +172,7 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
 void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
 			      struct ap_config_info *old_config_info);
 
+int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev);
+void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev);
+
 #endif /* _VFIO_AP_PRIVATE_H_ */
-- 
2.52.0

[PATCH v2 05/16] s390-vfio-ap: Callback to set vfio device mig state during guest migration

[Anthony Krowiak]

Implements the callback that sets a new migration state of a vfio device
during live migration of guests with pass-through access to AP devices.
This callback is mandatory for VFIO_DEVICE_FEATURE_MIGRATION
support.

The function pointer for this callback is specified via the
'migration_set_state' field of the 'vfio_migration_ops' structure
which is stored with the VFIO device when the 'vfio_device'
structure representing the mediated device is initialized.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 96 ++++++++++++++++++++++++-
 1 file changed, 95 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index a1ae4f1a3473..a4daa05822fb 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -7,6 +7,7 @@
 #include <linux/file.h>
 #include "uapi/linux/vfio_ap.h"
 
+#include "ap_bus.h"
 #include "vfio_ap_private.h"
 
 /**
@@ -96,10 +97,103 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
 	}
 }
 
+static struct file *
+vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
+			    enum vfio_device_mig_state new_state)
+{
+	struct vfio_ap_migration_data *mig_data;
+	enum vfio_device_mig_state cur_state;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+	cur_state = mig_data->mig_state;
+	dev_dbg(matrix_mdev->vdev.dev, "%s: %d -> %d\n", __func__, cur_state,
+		new_state);
+
+	if (cur_state == VFIO_DEVICE_STATE_STOP &&
+	    new_state == VFIO_DEVICE_STATE_STOP_COPY) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if (cur_state == VFIO_DEVICE_STATE_STOP &&
+	    new_state == VFIO_DEVICE_STATE_RESUMING) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
+	    new_state == VFIO_DEVICE_STATE_STOP) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
+	    new_state == VFIO_DEVICE_STATE_STOP) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	if ((cur_state == VFIO_DEVICE_STATE_STOP &&
+	     new_state == VFIO_DEVICE_STATE_RUNNING) ||
+	    (cur_state == VFIO_DEVICE_STATE_RUNNING &&
+	     new_state == VFIO_DEVICE_STATE_STOP)) {
+		/* TODO */
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	/* vfio_mig_get_next_state() does not use arcs other than the above */
+	WARN_ON(true);
+
+	return ERR_PTR(-EINVAL);
+}
+
 static struct file *vfio_ap_set_state(struct vfio_device *vdev,
 				      enum vfio_device_mig_state  new_state)
 {
-	return NULL;
+	int ret;
+	struct file *filp = NULL;
+	struct ap_matrix_mdev *matrix_mdev;
+	enum vfio_device_mig_state next_state;
+	struct vfio_ap_migration_data *mig_data;
+
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	if (ap_is_se_guest()) {
+		dev_err(matrix_mdev->vdev.dev,
+				"Migration not allowed from or to a Secure Execution guest\n");
+		mutex_unlock(&matrix_dev->mdevs_lock);
+		return ERR_PTR(-EPERM);
+	}
+
+	mig_data = matrix_mdev->mig_data;
+	dev_dbg(vdev->dev, "%s -> %d\n", __func__, new_state);
+
+	while (mig_data->mig_state != new_state) {
+		ret = vfio_mig_get_next_state(vdev, mig_data->mig_state,
+					      new_state, &next_state);
+		if (ret) {
+			filp = ERR_PTR(ret);
+			break;
+		}
+
+		filp = vfio_ap_transition_to_state(matrix_mdev, next_state);
+		if (IS_ERR(filp))
+			break;
+
+		mig_data->mig_state = next_state;
+
+		if (WARN_ON(filp && new_state != next_state)) {
+			fput(filp);
+			filp = ERR_PTR(-EINVAL);
+			break;
+		}
+	}
+
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return filp;
 }
 
 static int vfio_ap_get_state(struct vfio_device *vdev,
-- 
2.52.0

[PATCH v2 06/16] s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY

[Anthony Krowiak]

The transition of the guest migration state from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_STOP_COPY begins the process of saving the
vfio device state. The vfio device state is comprised of the guest's
AP configuration which specifies the adapters, domains and control
domains to which the guest will be given access.

To begin the process of saving the vfio device state, the VFIO AP device
driver will:

1. Allocate an object in which to store the guest's AP configuration
   information comprised of the APQN and hardware information for each
   queue passed through to the source guest.

2. Opens a file stream that will be used to read the AP configuration
   stored with the object created in #1. The AP configuration will be read
   from this file stream when the guest is resumed on the target system to
   verify that the AP configuration of the source and target guests are
   compatible.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 166 +++++++++++++++++++++++-
 1 file changed, 164 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index a4daa05822fb..21734d3ed7b7 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -4,6 +4,7 @@
  *
  * Copyright IBM Corp. 2025
  */
+#include <linux/anon_inodes.h>
 #include <linux/file.h>
 #include "uapi/linux/vfio_ap.h"
 
@@ -97,12 +98,160 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
 	}
 }
 
+static ssize_t vfio_ap_save_read(struct file *, char __user *, size_t, loff_t *)
+{
+	/* TODO */
+	return -EOPNOTSUPP;
+}
+
+static int vfio_ap_release_migf(struct inode *, struct file *)
+{
+	/* TODO */
+	return -EOPNOTSUPP;
+}
+
+static const struct file_operations vfio_ap_save_fops = {
+	.owner = THIS_MODULE,
+	.read = vfio_ap_save_read,
+	.compat_ioctl = compat_ptr_ioctl,
+	.release = vfio_ap_release_migf,
+};
+
+static struct vfio_ap_config
+*vfio_ap_allocate_config(struct ap_matrix_mdev *matrix_mdev, size_t *config_sz)
+{
+	struct vfio_ap_config *ap_config;
+	size_t qinfo_size;
+	int num_queues;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	num_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
+	qinfo_size = num_queues * sizeof(struct vfio_ap_queue_info);
+	*config_sz = qinfo_size + sizeof(struct vfio_ap_config);
+	ap_config = kzalloc(*config_sz, GFP_KERNEL_ACCOUNT);
+
+	if (!ap_config)
+		return ERR_PTR(-ENOMEM);
+
+	ap_config->num_queues = num_queues;
+
+	return ap_config;
+}
+
+static void vfio_ap_store_queue_info(struct vfio_ap_migration_file *migf)
+{
+	unsigned long *apm, *aqm, num_queues, apid, apqi, apqn, data;
+	struct ap_matrix_mdev *matrix_mdev;
+	struct vfio_ap_queue *q;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	matrix_mdev = migf->matrix_mdev;
+	apm = matrix_mdev->shadow_apcb.apm;
+	aqm = matrix_mdev->shadow_apcb.aqm;
+	num_queues = 0;
+
+	for_each_set_bit_inv(apid, apm, AP_DEVICES) {
+		for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) {
+			apqn = AP_MKQID(apid, apqi);
+			q = vfio_ap_mdev_get_queue(matrix_mdev, apqn);
+
+			if (!q)
+				continue;
+
+			migf->ap_config->qinfo[num_queues].apqn = apqn;
+			data = q->hwinfo.value;
+			migf->ap_config->qinfo[num_queues].data = data;
+			num_queues += 1;
+			dev_dbg(matrix_mdev->vdev.dev,
+				"%s (%d): qinfo: apqn=%04lx data=%016lx\n",
+				__func__, __LINE__, apqn, data);
+		}
+	}
+}
+
+static struct vfio_ap_migration_file
+*vfio_ap_allocate_migf(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_file *migf;
+	struct vfio_ap_config *ap_config;
+	size_t config_size;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	migf = kzalloc_obj(struct vfio_ap_migration_file, GFP_KERNEL_ACCOUNT);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	ap_config = vfio_ap_allocate_config(matrix_mdev, &config_size);
+	if (IS_ERR(ap_config)) {
+		kfree(migf);
+		return ERR_CAST(ap_config);
+	}
+
+	migf->ap_config = ap_config;
+	migf->config_sz = config_size;
+	migf->matrix_mdev = matrix_mdev;
+
+	return migf;
+}
+
+static void vfio_ap_deallocate_migf(struct vfio_ap_migration_file *migf)
+{
+	kfree(migf->ap_config);
+	kfree(migf);
+}
+
+static struct file *vfio_ap_open_file_stream(struct vfio_ap_migration_file *migf,
+					     const struct file_operations *fops,
+					     int flags)
+{
+	struct file *filp;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	filp = anon_inode_getfile("vfio_ap_migf", fops, migf, flags);
+	if (IS_ERR(filp))
+		return ERR_CAST(filp);
+
+	stream_open(filp->f_inode, filp);
+
+	return filp;
+}
+
+static struct vfio_ap_migration_file *
+vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+	struct vfio_ap_migration_file *migf;
+	struct file *filp;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	migf = vfio_ap_allocate_migf(matrix_mdev);
+	if (IS_ERR(migf))
+		return ERR_CAST(migf);
+
+	filp = vfio_ap_open_file_stream(migf, &vfio_ap_save_fops, O_RDONLY);
+	if (IS_ERR(filp)) {
+		vfio_ap_deallocate_migf(migf);
+		return ERR_CAST(filp);
+	}
+
+	migf->filp = filp;
+	mig_data->saving_migf = migf;
+	vfio_ap_store_queue_info(mig_data->saving_migf);
+
+	return mig_data->saving_migf;
+}
+
 static struct file *
 vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 			    enum vfio_device_mig_state new_state)
 {
 	struct vfio_ap_migration_data *mig_data;
 	enum vfio_device_mig_state cur_state;
+	struct vfio_ap_migration_file *migf;
 
 	lockdep_assert_held(&matrix_dev->mdevs_lock);
 	mig_data = matrix_mdev->mig_data;
@@ -110,10 +259,23 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 	dev_dbg(matrix_mdev->vdev.dev, "%s: %d -> %d\n", __func__, cur_state,
 		new_state);
 
+	/*
+	 * Begins the process of saving the vfio device state by creating and
+	 * returning a streaming data_fd to be used to read out the internal
+	 * state of the vfio-ap device on the source host.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_STOP &&
 	    new_state == VFIO_DEVICE_STATE_STOP_COPY) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		migf = vfio_ap_save_mdev_state(matrix_mdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+
+		if (migf) {
+			get_file(migf->filp);
+			return migf->filp;
+		}
+
+		return NULL;
 	}
 
 	if (cur_state == VFIO_DEVICE_STATE_STOP &&
-- 
2.52.0

[PATCH v2 07/16] s390/vfio-ap: File ops called to save the vfio device migration state

[Anthony Krowiak]

Implements two callback functions that were added to the 'file_operations'
structure for the file created to save the state of the vfio device
when the migration state transitioned from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_STOP_COPY:

* Read callback

  This function copies the guest's AP configuration information to
  userspace. The information copied is comprised of the APQN of each queue
  device passed through to the guest along with its hardware information.
  This data will be used to verify the source and target guests have
  compatible AP configurations.

* Release callback

  This function deallocates the object used to save the state of the
  vfio device.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 78 +++++++++++++++++++++++--
 1 file changed, 72 insertions(+), 6 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 21734d3ed7b7..aadc362edf78 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -98,16 +98,82 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev)
 	}
 }
 
-static ssize_t vfio_ap_save_read(struct file *, char __user *, size_t, loff_t *)
+static ssize_t validate_save_read_parms(struct vfio_ap_migration_file *migf,
+					loff_t *pos, size_t len)
 {
-	/* TODO */
-	return -EOPNOTSUPP;
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	if (migf->disabled) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"%s (%d): migration file is disabled\n",
+			__func__, __LINE__);
+		return -ENODEV;
+	}
+
+	if (*pos > migf->config_sz) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"%s (%d): file pos (%llu) exceeds migf->config size (%zu)\n",
+			__func__, __LINE__, *pos, migf->config_sz);
+		return -EINVAL;
+	}
+
+	return 0;
 }
 
-static int vfio_ap_release_migf(struct inode *, struct file *)
+static ssize_t vfio_ap_save_read(struct file *filp, char __user *buf,
+				 size_t len, loff_t *pos)
 {
-	/* TODO */
-	return -EOPNOTSUPP;
+	struct vfio_ap_migration_file *migf;
+	ssize_t ret = 0;
+
+	if (pos)
+		return -ESPIPE;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	pos = &filp->f_pos;
+	migf = filp->private_data;
+
+	ret = validate_save_read_parms(migf, pos, len);
+	if (ret)
+		goto out_unlock;
+
+	len = min_t(size_t, migf->config_sz - *pos, len);
+	if (len) {
+		if (copy_to_user(buf, (void *)migf->ap_config + *pos, len)) {
+			ret = -EFAULT;
+			dev_err(migf->matrix_mdev->vdev.dev,
+				"%s (%d): failed to copy config data to user\n",
+				__func__, __LINE__);
+			goto out_unlock;
+		}
+
+		*pos += len;
+		ret = len;
+	}
+
+	dev_dbg(migf->matrix_mdev->vdev.dev,
+		"%s (%d): copied %zu bytes of AP config data to user\n",
+		__func__, __LINE__, len);
+
+out_unlock:
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return ret;
+}
+
+static void vfio_ap_deallocate_migf(struct vfio_ap_migration_file *migf);
+
+static int vfio_ap_release_migf(struct inode *inode, struct file *filp)
+{
+	struct vfio_ap_migration_file *migf;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	migf = filp->private_data;
+	vfio_ap_deallocate_migf(migf);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return 0;
 }
 
 static const struct file_operations vfio_ap_save_fops = {
-- 
2.52.0

[PATCH v2 08/16] s390/vfio-ap: Transition device migration state from STOP to RESUMING

[Anthony Krowiak]

Transitioning from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING
starts a process of restoring the device state of the vfio device on the
target system. To prepare for restoring the device state, open a file
stream to receive the guest's AP configuration saved when the device state
on the source system transitioned from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_STOP_COPY.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 55 ++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index aadc362edf78..c78e24767d20 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -311,6 +311,46 @@ vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)
 	return mig_data->saving_migf;
 }
 
+static ssize_t vfio_ap_resume_write(struct file *filp, const char __user *buf,
+				    size_t len, loff_t *pos)
+{
+	/* TODO */
+	return -EOPNOTSUPP;
+}
+
+static const struct file_operations vfio_ap_resume_fops = {
+	.owner = THIS_MODULE,
+	.write = vfio_ap_resume_write,
+	.release = vfio_ap_release_migf,
+};
+
+static struct vfio_ap_migration_file *
+vfio_ap_resume_mdev_state(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_migration_data *mig_data;
+	struct vfio_ap_migration_file *migf;
+	struct file *filp;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	mig_data = matrix_mdev->mig_data;
+
+	migf = vfio_ap_allocate_migf(matrix_mdev);
+	if (IS_ERR(migf))
+		return ERR_CAST(migf);
+
+	filp = vfio_ap_open_file_stream(migf, &vfio_ap_resume_fops, O_WRONLY);
+	if (IS_ERR(filp)) {
+		vfio_ap_deallocate_migf(migf);
+		return ERR_CAST(filp);
+	}
+
+	migf->matrix_mdev = matrix_mdev;
+	migf->filp = filp;
+	mig_data->resuming_migf = migf;
+
+	return migf;
+}
+
 static struct file *
 vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 			    enum vfio_device_mig_state new_state)
@@ -344,10 +384,21 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return NULL;
 	}
 
+	/*
+	 * Starts the process of restoring the state of the vfio-ap device
+	 * on the target host by creating a filestream to be used to transfer
+	 * the internal state of the vfio-ap device on the source host that
+	 * was saved during the STOP_COPY phase of the migration.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_STOP &&
 	    new_state == VFIO_DEVICE_STATE_RESUMING) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		migf = vfio_ap_resume_mdev_state(matrix_mdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+
+		get_file(migf->filp);
+
+		return migf->filp;
 	}
 
 	if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
-- 
2.52.0

[PATCH v2 09/16] s390/vfio-ap: File ops called to resume the vfio device migration

[Anthony Krowiak]

Implements two callback functions that were added to the 'file_operations'
structure for the file created to restore the state of the vfio device
when the migration state transitioned from VFIO_DEVICE_STATE_STOP to
VFIO_DEVICE_STATE_RESUMING:

* Write callback

  The write callback retrieves the vfio device migration state saved to the
  file stream created when the vfio device state was transitioned from
  VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY. The saved state
  contains the guest's AP configuration information. Since the guest's AP
  configuration on the target system is supplied by the mediated device
  residing thereon, the migration state passed into this callback will not
  be used to restore the vfio device state, but to verify that the source
  and target AP configurations are compatible since the mediated devices on
  the source and target systems may differ.

  The verification rules are:

  o The source and target AP configurations must have the same number of
    APQNs

  o Each APQN in the source guest's AP configuration must also be in the
    target guest's configuration

  o Each APQN in the source and target guests' AP configurations must
    reference a queue device with compatible hardware:

    - The source and target queues must have the same facilities installed
  	- APSC facility
  	- APQKM facility
  	- AP4KC facility

    - The source and target queues must have the same mode
  	- Coprocessor-mode
  	- Accelerator-mode
  	- XCP-mode

    - The source and target queues must have the same APXA facility setting
   	- If the APXA facility is installed on target queue, it must also
   	  be installed on the source queue
   	- If the APXA facility is not installed on target queue, it must
   	  also not be installed on the source queue

    - The source and target queues must have the same classification
   	- Full native card function
   	- Stateless functions

    - The binding and associated state
   	~ At the time of migration, the queue on the target guest will
   	  neither be bound nor associated, so the binding and associated
   	  state for both the source and target must indicate that the
   	  queue is usable for all messages (i.e., BS bits equal to 00).

    - The AP type of the target queue must be the same or newer than the
      source queue

* Release callback

This function deallocates the object used to restore the state of the
vfio device when the file is released.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 518 +++++++++++++++++++++++-
 1 file changed, 516 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index c78e24767d20..ce16028e50a4 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -11,6 +11,36 @@
 #include "ap_bus.h"
 #include "vfio_ap_private.h"
 
+/*
+ * Masks the fields of the queue information returned from the PQAP(TAPQ)
+ * command. In order to migrate a guest, it's AP configuration must be
+ * compatible with AP configuration assigned to the target guest's mdev.
+ * This mask is used to verify that the queue information for each source and
+ * target queue is compatible (i.e., the masked fields are equivalent).
+ *
+ * The relevant fields covered by this mask are:
+ * S bit 0: APSC  facility installed
+ * M bit 1: APQKM facility installed
+ * C bit 2: AP4KC facility installed
+ * Mode bits 3-5:
+ *     D bit 3: CCA-mode facility
+ *     A bit 4: accelerator-mode facility
+ *     X bit 5: XCP-mode facility
+ * N  bit 6: APXA facility installed
+ * SL bit 7: SLCF facility installed
+ * Classification (functional capabilities) bits 8-16
+ *     bit 8: Native card function
+ *     bit 9: Only stateless functions
+ * AP Type bits 32-40:
+ */
+#define QINFO_DATA_MASK		0xffffc000ff000000
+
+/*
+ * Masks the bit that indicates whether full native card function is available
+ * from the 8 bits specifying the functional capabilities of a queue
+ */
+#define CLASSIFICATION_NATIVE_FCN_MASK		0x80
+
 /**
  * vfio_ap_migration_data - the data needed to migrate a guest with pass-through
  *			    access to AP devices
@@ -235,6 +265,53 @@ static void vfio_ap_store_queue_info(struct vfio_ap_migration_file *migf)
 	}
 }
 
+static int validate_resume_write_parms(struct vfio_ap_migration_file *migf,
+				       size_t len, loff_t *pos)
+{
+	loff_t total_len;
+	int ret = -EIO;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	if (!migf->matrix_mdev) {
+		pr_err("migration failed: matrix_mdev object not linked to migration file");
+		goto done;
+	}
+
+	if (*pos < 0) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: invalid migration file position  (%lli) for write\n",
+			*pos);
+		goto done;
+	}
+
+	if (check_add_overflow((loff_t)len, *pos, &total_len)) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: pos (%llu) plus len (%zu) operation overflowed loff_t precision\n",
+			*pos, len);
+		goto done;
+	}
+
+	if (total_len > migf->config_sz) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: source guest's AP config size (%llu) larger than target's (%lu)",
+			total_len, migf->config_sz);
+		goto done;
+	}
+
+	if (migf->disabled) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"migration failed: migration file is disabled");
+		goto done;
+	}
+
+	dev_dbg(migf->matrix_mdev->vdev.dev, "resume write parameters validated\n");
+	ret = 0;
+
+done:
+	return ret;
+}
+
 static struct vfio_ap_migration_file
 *vfio_ap_allocate_migf(struct ap_matrix_mdev *matrix_mdev)
 {
@@ -311,11 +388,448 @@ vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)
 	return mig_data->saving_migf;
 }
 
+static void report_facilities_compatibility(struct ap_matrix_mdev *matrix_mdev,
+					    unsigned long apqn,
+					    struct ap_tapq_hwinfo *src_hwinfo,
+					    struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->apsc != target_hwinfo->apsc) {
+		if (src_hwinfo->apsc) {
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"APSC facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->mex4k != target_hwinfo->mex4k) {
+		if (src_hwinfo->mex4k) {
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"mex4k facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->crt4k != target_hwinfo->crt4k) {
+		if (src_hwinfo->crt4k) {
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"crt4k facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_mode_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqn,
+				      struct ap_tapq_hwinfo *src_hwinfo,
+				      struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->cca != target_hwinfo->cca) {
+		if (src_hwinfo->cca) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode  facility not installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Coprocessor-mode  facility installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->accel != target_hwinfo->accel) {
+		if (src_hwinfo->accel) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility not installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility not installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Accelerator-mode facility installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+
+	if (src_hwinfo->ep11 != target_hwinfo->ep11) {
+		if (src_hwinfo->ep11) {
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility not installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility not installed source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"XCP-mode facility installed target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_apxa_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqn,
+				      struct ap_tapq_hwinfo *src_hwinfo,
+				      struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->apxa != target_hwinfo->apxa) {
+		if (src_hwinfo->apxa) {
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility not installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility not installed in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"AP-extended-addressing (APXA) facility installed in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_slcf_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqn,
+				      struct ap_tapq_hwinfo *src_hwinfo,
+				      struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->slcf != target_hwinfo->slcf) {
+		if (src_hwinfo->slcf) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) available in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) not available in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		} else {
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) not available in source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Stateless-command-filtering (SLCF) available in target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_class_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				       unsigned long apqn,
+				       struct ap_tapq_hwinfo *src_hwinfo,
+				       struct ap_tapq_hwinfo *target_hwinfo)
+{
+	unsigned long src_native, target_native;
+
+	src_native = src_hwinfo->class & CLASSIFICATION_NATIVE_FCN_MASK;
+	target_native = target_hwinfo->class & CLASSIFICATION_NATIVE_FCN_MASK;
+
+	if (src_native != target_native) {
+		/*
+		 * If the source queue has full native card function and the
+		 * target queue has only stateless functions available, then
+		 * there may be instructions that will not execute on the
+		 * target queue.
+		 *
+		 * If the source queue has only stateless card functions and the
+		 * target queue has full native card function available, then
+		 * we are okay because the target queue can run all card
+		 * functions.
+		 */
+		if (src_native) {
+			dev_err(matrix_mdev->vdev.dev,
+				"Full native card function available on source queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+			dev_err(matrix_mdev->vdev.dev,
+				"Only stateless functions available on target queue %02lx.%04lx\n",
+				AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+		}
+	}
+}
+
+static void report_bs_compatibility(struct ap_matrix_mdev *matrix_mdev,
+				    unsigned long apqn,
+				    struct ap_tapq_hwinfo *src_hwinfo,
+				    struct ap_tapq_hwinfo *target_hwinfo)
+{
+	if (src_hwinfo->bs || target_hwinfo->bs) {
+		dev_err(matrix_mdev->vdev.dev,
+			"Bind/associate state for source (%01x) and target (%01x) queue %02lx.%04lx are not compatible\n",
+			src_hwinfo->bs, target_hwinfo->bs,
+			AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+	}
+}
+
+static void report_aptype_compatibility(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long apqn,
+					struct ap_tapq_hwinfo *src_hwinfo,
+					struct ap_tapq_hwinfo *target_hwinfo)
+{
+	dev_err(matrix_mdev->vdev.dev,
+		"AP type of source (%02x) not compatible with target (%02x)\n",
+		src_hwinfo->at, target_hwinfo->at);
+}
+
+/*
+ * Log a device error reporting that migration failed due to queue
+ * incompatibilities followed by a device error for each incompatible feature.
+ */
+static void report_qinfo_incompatibilities(struct ap_matrix_mdev *matrix_mdev,
+					   unsigned long apqn,
+					   struct ap_tapq_hwinfo *src_hwinfo,
+					   struct ap_tapq_hwinfo *target_hwinfo)
+{
+	dev_err(matrix_mdev->vdev.dev,
+		"Migration failed: Source and target queue (%02lx.%04lx) not compatible\n",
+		AP_QID_CARD(apqn), AP_QID_QUEUE(apqn));
+
+	report_facilities_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_mode_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_apxa_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_slcf_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_class_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_aptype_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+	report_bs_compatibility(matrix_mdev, apqn, src_hwinfo, target_hwinfo);
+}
+
+static bool qinfo_compatible(struct ap_matrix_mdev *matrix_mdev,
+			     unsigned long apqn,
+			     struct ap_tapq_hwinfo *src_hwinfo,
+			     struct ap_tapq_hwinfo *target_hwinfo)
+{
+	unsigned long src_bits, target_bits;
+
+	src_bits = src_hwinfo->value & QINFO_DATA_MASK;
+	target_bits = target_hwinfo->value & QINFO_DATA_MASK;
+
+	/*
+	 * If all relevant bits are the same, or only the AP type of the source
+	 * and target queue differ but the source type is older than the target
+	 * type, then no incompatibilities will be reported. The AP types are
+	 * considered compatible even if they differ as long as the source type
+	 * is older than the target type since AP devices are backwards
+	 * compatible.
+	 */
+	if (src_bits == target_bits ||
+	    (src_hwinfo->fac == target_hwinfo->fac &&
+	     src_hwinfo->at <= target_hwinfo->at)) {
+		return true;
+	}
+
+	report_qinfo_incompatibilities(matrix_mdev, apqn, src_hwinfo,
+				       target_hwinfo);
+
+	return false;
+}
+
+static int matrixes_compatible(struct vfio_ap_migration_file *migf)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	struct ap_tapq_hwinfo src_hwinfo;
+	struct vfio_ap_queue *q;
+	unsigned long apqn;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	matrix_mdev = migf->matrix_mdev;
+
+	for (int i = 0; i < migf->ap_config->num_queues; i++) {
+		apqn = migf->ap_config->qinfo[i].apqn;
+		q = vfio_ap_mdev_get_queue(matrix_mdev, apqn);
+		memcpy(&src_hwinfo, &migf->ap_config->qinfo[i].data,
+		       sizeof(src_hwinfo));
+
+		if (!qinfo_compatible(matrix_mdev, apqn, &src_hwinfo, &q->hwinfo))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+static bool apqns_match(struct vfio_ap_migration_file *migf)
+{
+	struct ap_matrix_mdev *matrix_mdev;
+	unsigned long apid, apqi, apqn;
+	bool ret = true;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+
+	matrix_mdev = migf->matrix_mdev;
+
+	for (int i = 0; i < migf->ap_config->num_queues; i++) {
+		apqn = migf->ap_config->qinfo[i].apqn;
+		apid = AP_QID_CARD(apqn);
+		apqi = AP_QID_QUEUE(apqn);
+
+		if (!test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) ||
+		    !test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
+			dev_err(matrix_mdev->vdev.dev,
+				"migration failed: queue %02lx.%04lx not assigned to guest matrix\n",
+				apid, apqi);
+			ret = false;
+		}
+	}
+
+	return ret;
+}
+
+static int vfio_ap_validate_num_queues(struct vfio_ap_migration_file *migf)
+{
+	int num_migf_queues, num_mdev_queues;
+	struct ap_matrix_mdev *matrix_mdev;
+
+	lockdep_assert_held(&matrix_dev->mdevs_lock);
+	matrix_mdev = migf->matrix_mdev;
+	num_mdev_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
+	num_migf_queues = migf->ap_config->num_queues;
+
+	if (num_mdev_queues != num_migf_queues) {
+		dev_err(matrix_mdev->vdev.dev,
+			"migration failed: number of queues on source (%d) and target (%d) guests differ\n",
+			num_migf_queues, num_mdev_queues);
+		return (num_mdev_queues > num_migf_queues) ? -ENODEV : -E2BIG;
+	}
+
+	return 0;
+}
+
+static int do_post_copy_validation(struct vfio_ap_migration_file *migf, loff_t pos)
+{
+	unsigned long nqueues_offset;
+	int ret;
+
+	nqueues_offset = offsetofend(struct vfio_ap_config, num_queues);
+	if (pos >= nqueues_offset) {
+		ret = vfio_ap_validate_num_queues(migf);
+		if (ret)
+			return ret;
+
+		if (pos == migf->config_sz) {
+			if (!apqns_match(migf))
+				return -ENODEV;
+			ret = matrixes_compatible(migf);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * vfio_ap_resume_write - store the AP configuration information sent from the
+ *			  source guest into the migration file.
+ * @filp: the file used to send the AP configuration information from the source
+ *	  guest.
+ * @buf:  buffer containing the AP configuration information sent from the
+ *	  source guest
+ * @len:  the length of the AP configuration information contained in @buf
+ * *pos:  a pointer to store the file position after retrieving the AP config
+ *	  information from @buf
+ */
 static ssize_t vfio_ap_resume_write(struct file *filp, const char __user *buf,
 				    size_t len, loff_t *pos)
 {
-	/* TODO */
-	return -EOPNOTSUPP;
+	struct vfio_ap_migration_data *mig_data;
+	struct vfio_ap_migration_file *migf;
+	ssize_t ret = 0;
+
+	if (pos)
+		return -ESPIPE;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	pos = &filp->f_pos;
+	migf = filp->private_data;
+	mig_data = migf->matrix_mdev->mig_data;
+
+	ret = validate_resume_write_parms(migf, len, pos);
+	if (ret)
+		goto out_unlock;
+
+	if (copy_from_user((void *)migf->ap_config + *pos, buf, len)) {
+		dev_err(migf->matrix_mdev->vdev.dev,
+			"%s (%d): failed to copy queue information from userspace",
+			__func__, __LINE__);
+		ret = -EFAULT;
+		goto out_unlock;
+	}
+
+	*pos += len;
+
+	ret = do_post_copy_validation(migf, *pos);
+	if (ret)
+		goto out_unlock;
+
+	ret = len;
+
+	dev_dbg(migf->matrix_mdev->vdev.dev,
+		"%s (%d): %zu bytes of queue information stored in the migration file",
+		__func__, __LINE__, len);
+
+out_unlock:
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return ret;
 }
 
 static const struct file_operations vfio_ap_resume_fops = {
-- 
2.52.0

[PATCH v2 10/16] s390/vfio-ap: Transition device migration state from RESUMING to STOP

[Anthony Krowiak]

Leaving the RESUMING state terminates a data transfer of the internal
state of the vfio-ap device on the source host to the vfio-ap device on the
target host. This state transition indicates the vfio_ap driver should
complete the incorporation of data written to the data transfer FD into the
into the vfio-ap device. Since a vfio-ap device does not virtualize a
physical device, there is no internal device state to
incorporate; so, the only thing left to do is release the migration files
used to help facilitate the migration process.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index ce16028e50a4..ebb3b78b1e19 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -915,10 +915,20 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return migf->filp;
 	}
 
+	/*
+	 * Terminates the data transfer session of the vfio-ap device state
+	 * between the source and target hosts. Since the vfio-ap device does
+	 * not virtualize a DMA device, there is no internal device state to
+	 * incorporate into the vfio-ap device on the target; so, the only
+	 * thing left to do is release the migration files used to process
+	 * the vfio device migration. Note that this state transition is for
+	 * the vfio-ap device on the target host.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
 	    new_state == VFIO_DEVICE_STATE_STOP) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		vfio_ap_release_mig_files(matrix_mdev);
+
+		return NULL;
 	}
 
 	if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
-- 
2.52.0

[PATCH v2 11/16] s390/vfio-ap: Transition device migration state from STOP_COPY to STOP

[Anthony Krowiak]

Typically at this point, interrupt processing for the device, DMA and other
changes to the device's external state must be prohibited. For VFIO AP,
there is no physical DMA device virtualized by the vfio-ap device . It
merely manages the guest's AP configuration that identifies the devices
passed through to the guest, but does not have access to any of them. Those
AP devices are passed through and controlled exclusively by the SIE program
used to start the guest. The only thing to do at this point is release all
migration objects used to manage migration of the guest.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index ebb3b78b1e19..6074ccfb9240 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -931,10 +931,18 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return NULL;
 	}
 
+	/*
+	 * Stop the operation of the vfio-ap device. Since the vfio-ap device
+	 * does not virtualize a DMA device, there is no physical device to
+	 * stop; so, the only thing left to do is release the migration files
+	 * used to process the vfio device migration. Note that this state
+	 * transition is for the vfio-ap device on the source host.
+	 */
 	if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
 	    new_state == VFIO_DEVICE_STATE_STOP) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		vfio_ap_release_mig_files(matrix_mdev);
+
+		return NULL;
 	}
 
 	if ((cur_state == VFIO_DEVICE_STATE_STOP &&
-- 
2.52.0

[PATCH v2 12/16] s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa

[Anthony Krowiak]

The transition from STOP to RUNNING and vice versa indicates that
the vfio device is operating normally and migration is not
currently taking place. In this case, just set the new state and
if there are any migrations files used to help facilitate migration
hanging around, release them.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 6074ccfb9240..28209d8476d9 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -945,12 +945,18 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,
 		return NULL;
 	}
 
+	/*
+	 * These states indicates migration has either not been initiated or
+	 * has completed and the vfio-ap device is operating normally; so
+	 * just set the new migration state. Just in case, release the
+	 * migration files used to facilitate migration if any are hanging
+	 * around.
+	 */
 	if ((cur_state == VFIO_DEVICE_STATE_STOP &&
 	     new_state == VFIO_DEVICE_STATE_RUNNING) ||
 	    (cur_state == VFIO_DEVICE_STATE_RUNNING &&
 	     new_state == VFIO_DEVICE_STATE_STOP)) {
-		/* TODO */
-		return ERR_PTR(-EOPNOTSUPP);
+		return NULL;
 	}
 
 	/* vfio_mig_get_next_state() does not use arcs other than the above */
-- 
2.52.0

[PATCH v2 13/16] s390-vfio-ap: Callback to get the current vfio device migration state

[Anthony Krowiak]

Implements the callback that returns the current vfio device migration
state during live migration of guests with pass-through access to
AP devices.

The function pointer for this callback is specified via the
'migration_get_state' field of the 'vfio_migration_ops' structure
which is stored with the VFIO device when the 'vfio_device'
structure representing the mediated device is initialized.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index 28209d8476d9..f9b219083e0f 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -1016,7 +1016,18 @@ static struct file *vfio_ap_set_state(struct vfio_device *vdev,
 static int vfio_ap_get_state(struct vfio_device *vdev,
 			     enum vfio_device_mig_state  *current_state)
 {
-	return -EOPNOTSUPP;
+	struct ap_matrix_mdev *matrix_mdev;
+	struct vfio_ap_migration_data *mig_data;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	mig_data =  matrix_mdev->mig_data;
+	*current_state = mig_data->mig_state;
+
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return 0;
 }
 
 static int vfio_ap_get_data_size(struct vfio_device *vdev,
-- 
2.52.0

[PATCH v2 14/16] s390/vfio-ap: Callback to get the size of data to be migrated during guest migration

[Anthony Krowiak]

Callback to get the estimated data length of the data that will be required
to complete reading of the vfio device's internal state. It's mandatory for
VFIO_DEVICE_FEATURE_MIGRATION migration support.

The function pointer for this callback is specified via the
'migration_get_data_size' field of the 'vfio_migration_ops' structure
which is stored with the VFIO device when the 'vfio_device'
structure representing the mediated device is initialized.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/vfio_ap_migration.c
index f9b219083e0f..ad751df33af0 100644
--- a/drivers/s390/crypto/vfio_ap_migration.c
+++ b/drivers/s390/crypto/vfio_ap_migration.c
@@ -1033,7 +1033,18 @@ static int vfio_ap_get_state(struct vfio_device *vdev,
 static int vfio_ap_get_data_size(struct vfio_device *vdev,
 				 unsigned long *stop_copy_length)
 {
-	return -EOPNOTSUPP;
+	struct ap_matrix_mdev *matrix_mdev;
+	size_t qinfo_sz;
+	int num_queues;
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev);
+	num_queues = vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb);
+	qinfo_sz = num_queues * sizeof(struct vfio_ap_queue_info);
+	*stop_copy_length = qinfo_sz + sizeof(struct vfio_ap_config);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return 0;
 }
 
 static const struct vfio_migration_ops vfio_ap_migration_ops = {
-- 
2.52.0

[PATCH v2 15/16] s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute

[Anthony Krowiak]

Add the 'migratable' feature to the features provided by the sysfs
/sys/devices/vfio_ap/matrix/feature attribute to indicate that migration
of vfio devices is supported.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 drivers/s390/crypto/vfio_ap_drv.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index fd7394d81880..d2fc3598cd24 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -26,9 +26,11 @@ MODULE_LICENSE("GPL v2");
 struct ap_matrix_dev *matrix_dev;
 debug_info_t *vfio_ap_dbf_info;
 
+#define FEATURES "guest_matrix hotplug ap_config migratable"
+
 static ssize_t features_show(struct device *dev, struct device_attribute *attr, char *buf)
 {
-	return sysfs_emit(buf, "guest_matrix hotplug ap_config\n");
+	return sysfs_emit(buf, "%s\n", FEATURES);
 }
 static DEVICE_ATTR_RO(features);
 
-- 
2.52.0

[PATCH v2 16/16] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst

[Anthony Krowiak]

Add a 'Live Guest Migration' chapter to the
Documentation/arch/s390/vfio-ap.rst document to describe the details for
initiating live guest migration for a guest to which AP adapters, domains
and control domains have been passed through.

Signed-off-by: Anthony Krowiak <akrowiak@linux.ibm.com>
---
 Documentation/arch/s390/vfio-ap.rst | 325 ++++++++++++++++++++--------
 1 file changed, 231 insertions(+), 94 deletions(-)

diff --git a/Documentation/arch/s390/vfio-ap.rst b/Documentation/arch/s390/vfio-ap.rst
index eba1991fbdba..8684d5ff87e4 100644
--- a/Documentation/arch/s390/vfio-ap.rst
+++ b/Documentation/arch/s390/vfio-ap.rst
@@ -1016,7 +1016,7 @@ guest_matrix dyn ap_config
 
 the following features are advertised:
 
----------------+---------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
 | Flag         | Description                                                   |
 +==============+===============================================================+
 | guest_matrix | guest_matrix attribute exists. It reports the matrix of       |
@@ -1025,105 +1025,242 @@ the following features are advertised:
 +--------------+---------------------------------------------------------------+
 | dyn          | Indicates hot plug/unplug of AP adapters, domains and control |
 |              | domains for a guest to which the mdev is attached.            |
-+------------+-----------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
 | ap_config    | ap_config interface for one-shot modifications to mdev config |
 +--------------+---------------------------------------------------------------+
+| migratable   | Indicates that live guest migration is supported for guests   |
+|              | to which crypto devices are passed through                    |
++--------------+---------------------------------------------------------------+
 
-Limitations
-===========
-Live guest migration is not supported for guests using AP devices without
-intervention by a system administrator. Before a KVM guest can be migrated,
-the vfio_ap mediated device must be removed. Unfortunately, it can not be
-removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
-the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
-its mdev can be hot unplugged from the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
-   the following commands:
-
-      virsh detach-device <guestname> <path-to-device-xml>
-
-      For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
-      the guest named 'my-guest':
-
-         virsh detach-device my-guest ~/config/my-guest-hostdev.xml
-
-            The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
-            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
-              <source>
-                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
-              </source>
-            </hostdev>
-
-
-      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
-
-      For example, to hot unplug the vfio_ap mediated device identified on the
-      qemu command line with 'id=hostdev0' from the guest named 'my-guest':
-
-.. code-block:: sh
-
-         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
-
-2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
-   to the guest and using the following qemu monitor command:
-
-      (QEMU) device-del id=<device-id>
-
-      For example, to hot unplug the vfio_ap mediated device that was specified
-      on the qemu command line with 'id=hostdev0' when the guest was started:
-
-         (QEMU) device-del id=hostdev0
-
-After live migration of the KVM guest completes, an AP configuration can be
-restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
-system into the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
-   device into the guest via the following virsh commands:
-
-   virsh attach-device <guestname> <path-to-device-xml>
-
-      For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
-      the guest named 'my-guest':
-
-         virsh attach-device my-guest ~/config/my-guest-hostdev.xml
-
-            The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
-            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
-              <source>
-                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
-              </source>
-            </hostdev>
-
-
-   virsh qemu-monitor-command <guest-name> --hmp \
-   "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+Live Guest Migration
+====================
+The VFIO AP mediated device is not used to provide userspace with direct access
+to a device as is the case with other devices that use the VFIO framework to
+pass them through to a guest. Instead, it manages AP configuration metadata
+identifying the AP adapters, domains, and control domains to which a guest will
+be granted access. These AP resources are configured by assigning them to a
+vfio-ap mediated device via its sysfs assignment interfaces. When the guest is
+started, the vfio_ap device driver sets the guest's AP configuration
+from the metadata stored with the mediated device. The AP devices
+are not accessed directly through the vfio_ap driver, so the driver has no
+internal device state to migrate. It's sole purpose during migration is to
+ensure that the AP configurations of the source and destination guests are
+compatible.
 
-      For example, to hot plug the vfio_ap mediated device
-      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
-      device-id hostdev0:
+To be considered compatible, the AP configuration for both the source and
+destination guests must meet these requirements:
 
-      virsh qemu-monitor-command my-guest --hmp \
-      "device_add vfio-ap,\
-      sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
-      id=hostdev0"
+  * Must have the same number of APQNs
 
-2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
-   to the guest and using the following qemu monitor command:
+  * Each APQN assigned to the source guest must also be assigned to the
+    destination guest
 
-      (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+  * Each APQN assigned to the source guest and destination guest must reference
+    a queue with compatible hardware capabilities:
 
-      For example, to plug the vfio_ap mediated device
-      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
-      hostdev0:
++--------------+---------------------------------------------------------------+
+| Hardware     | Description                                                   |
+| Capabilities |                                                               |
++==============+===============================================================+
+| facilities   | * AP special command facility (APSC)                          |
+|              | * AP 4096-bit ME PKU commands facility (AP4KM)                |
+|              | * AP 4096-bit CRT PKU commands (AP4KC)                        |
++--------------+---------------------------------------------------------------+
+| mode         | * CCA-mode                                                    |
+|              | * Accelerator-mode                                            |
+|              | * XCP-mode (EP11)                                             |
++--------------+---------------------------------------------------------------+
+| AP extended  | APXA installed                                                |
+| addressing   |                                                               |
++--------------+---------------------------------------------------------------+
+| command      | Command filtering available                                   |
+| filtering    |                                                               |
++--------------+---------------------------------------------------------------+
+| functional   | * Full native card function                                   |
+| capabilities | * Only stateless functions                                    |
++--------------+---------------------------------------------------------------+
+| secure       | The guest running on the source host can not have any queues  |
+| execution    | bound or associated with it                                   |
++--------------+---------------------------------------------------------------+
+| AP type      | * No AP type                                                  |
+|              | * PCICC (Leeds-2)                                             |
+|              | * PCICA (Leeds-2 Lite)                                        |
+|              | * PCIXCC                                                      |
+|              | * CEX2A                                                       |
+|              | * CEX2C                                                       |
+|              | * CEX3A                                                       |
+|              | * CEX3C                                                       |
+|              | * CEX4S                                                       |
+|              | * CEX5S                                                       |
+|              | * CEX6S                                                       |
+|              | * CEX7S                                                       |
+|              | * CEX8S                                                       |
+|              |                                                               |
+|              | Note: The AP type on the source and destination guests can    |
+|              |       differ if the queue passed through to the target guest  |
+|              |       is a newer model (backwards compatible)                 |
++--------------+---------------------------------------------------------------+
 
-         (QEMU) device-add "vfio-ap,\
-         sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
-         id=hostdev0"
+Live guest migration failures due to AP configuration errors
+------------------------------------------------------------
+The destination host is missing the mediated device with the same name as the
+mdev attached to the source guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: device not found: mediated device '$UUID' not found*       |
++--------------+--------------------------------------------------------------------+
+
+The source guest is not enabled for migration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: Requested operation is not valid:                          |
+|              | cannot migrate domain: $UUID: Migration is disabled for VFIO       |
+|              | device*                                                            |
++--------------+--------------------------------------------------------------------+
+
+
+The AP configuration of the source and destination guests are not compatible
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: operation failed: job 'migration in' failed: load of       |
+|              | migration failed: Bad address*                                     |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: Sibling indicated error 1*                     |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | *vfio_ap_mdev $UUID: Migration failed: Source and target queues    |
+|              | ($APQN) are not compatible*                                        |
+|              |                                                                    |
+|              | The message above will be followed by one or more messages         |
+|              | enumerating the incompatible features; for example:                |
+|              |                                                                    |
+|              | *vfio_ap_mdev $UUID: APSC facility installed in source queue $APQN*|
+|              |                                                                    |
+|              | *vfio_ap_mdev $UUID: APSC facility not installed in target queue   |
+|              | $APQN*                                                             |
+|              |                                                                    |
+|              | *AP type of source ($APTYPE) not compatible with target ($APTYPE)* |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: error while loading state section id ...*      |
+|              |                                                                    |
+|              | *shutting down, reason=failed*                                     |
+|              |                                                                    |
+|              | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)*     |
++--------------+--------------------------------------------------------------------+
+
+The AP configuration of the source guest has more APQNS than the destination guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: operation f1e166ee77e6failed: job 'migration in' failed: load of       |
+|              | migration failed: Input/output error*                              |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: Sibling indicated error 1*                     |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | *vfio_ap_mdev $UUID:                                               |
+|              | migration failed: source guest's AP config size (xx) larger than   |
+|              | target's (yy)*                                                     |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: error while loading state section id ...*      |
+|              |                                                                    |
+|              | *shutting down, reason=failed*                                     |
+|              |                                                                    |
+|              | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)*     |
++--------------+--------------------------------------------------------------------+
+
+The AP configuration of the source guest has fewer APQNS than the destination guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | *error: operation failed: job 'migration in' failed: load of       |
+|              | migration failed: No such device*                                  |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: Sibling indicated error 1*                     |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log          | Message                                                            |
++==============+====================================================================+
+| Console log: | N/A                                                                |
++--------------+--------------------------------------------------------------------+
+| Kernel log:  | *vfio_ap_mdev $UUID:                                               |
+|              | migration failed: number of queues on source (x) and target (y)    |
+|              | guests differ*                                                     |
++--------------+--------------------------------------------------------------------+
+| QEMU log:    | *initiating migration*                                             |
+|              |                                                                    |
+|              | *qemu-system-s390x: error while loading state section id ...*      |
+|              |                                                                    |
+|              | *shutting down, reason=failed*                                     |
+|              |                                                                    |
+|              | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)*     |
++--------------+--------------------------------------------------------------------+
+
+AP Configuration Management
+---------------------------
+The AP configurations of the source and destination guests must be synchronized or
+live guest migration will likely fail due to incompatibility. In particular, it is
+imperative that such changes are not made during migration. Configuration stability
+is an orchestration-layer or system administrator responsibility, consistent with
+other VFIO device types. The vfio_ap driver's role is to validate configurations
+and provide clear diagnostics when incompatibilities are detected, enabling
+orchestration tools to implement appropriate policies.
+
+Note that s390 Common Cryptographic Architecture (CCA) master key administration
+must to be performed on both the source and destination AP devices to synchronize
+the key values prior to allowing live guest migration. If the master keys do not
+match, then crypto applications that rely on secure keys wrapped by a CCA master
+key will fail when the guest on which they are running is migrated to the
+destination host.
-- 
2.52.0