[PATCH v7 00/15] s390/vfio-ap: Add live guest migration support

[Anthony Krowiak <akrowiak@linux.ibm.com>]

This patch series implements live guest migration support for KVM guests
with s390 AP (Adjunct Processor) devices passed through via the VFIO
mediated device framework.

Background
~~~~~~~~~~

The vfio-ap device driver differs from typical VFIO device drivers in that
it does not virtualize a physical device. Instead, it manages AP
configuration metadata identifying the AP adapters, domains, and control
domains to which a guest will be granted access. These AP resources are
configured by assigning them to a vfio-ap mediated device via its sysfs
assignment interfaces. When the fd for the VFIO device is opened by
userspace, the vfio_ap device driver sets the guest's AP configuration
from the metadata stored with the mediated device. As such, the AP devices
are not accessed directly through the vfio_ap driver, so the driver has no
internal AP device state to migrate. What it does migrate is the AP
configuration metadata of the source guest.

Implementation Approach
~~~~~~~~~~~~~~~~~~~~~~~

This series implements the VFIO migration protocol using the STOP_COPY
migration flow. The key aspects are:

1. On transition of the migration state from STOP to STOP_COPY
   - The vfio_ap device driver creates a filestream for userspace to use to
     read the guest's AP configuration from the mdev

2. During the STOP_COPY phase
   - Userspace uses the filestream created in #1 to read the source guest's
     AP configuration
   - The vfio_ap device driver copies the source guest's AP configuration
     information to userspace

3. On transition of the migration state from STOP to RESUMING
   - The vfio_ap device driver creates a filestream for userspace to use to
     write the source guest's AP configuration information so it can be
     restored to the mdev on the destination host.

4. During the RESUMING phase
   - Userspace uses the filestream created in #3 to send the source guest's
     AP configuration information to the vfio_ap device driver on the
     destination host.
   - The vfio_ap device driver first verifies the source guest's AP
     configuration is compatible with the destination host's.
   - The driver restores AP configuration to the mdev on the destination
     host which automatically hot plugs the AP resources identified
     therein.

5. Documentation
   - Add live guest migration chapter to vfio-ap.rst

Compatibility Validation
~~~~~~~~~~~~~~~~~~~~~~~~

The series includes comprehensive validation to ensure source and
destination AP configurations are compatible. For each queue, the following
characteristics must match:

- AP type (target must be same or newer than source)
- Installed facilities (APSC, APQKM, AP4KC, SLCF)
- Operating mode (CCA, Accelerator, XCP)
- APXA facility setting
- Classification (native vs stateless functions)
- Queue usability (binding/associated state)

When incompatibilities are detected, migration fails with detailed error
messages identifying the specific queue and characteristic that caused
the failure.

Configuration Management
~~~~~~~~~~~~~~~~~~~~~~~~

This implementation does not prevent configuration changes during
migration. Configuration stability is an orchestration-layer
responsibility, consistent with other VFIO device types. The driver's
role is to validate configurations and provide clear diagnostics when
incompatibilities are detected, enabling orchestration tools to implement
appropriate policies.

QEMU patches exploiting this series:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
https://lore.kernel.org/qemu-devel/20260409141352.997844-1-akrowiak@linux.ibm.com/

Change log v6 => v7:
~~~~~~~~~~~~~~~~~~~
Patch 2: Data structures for facilitating vfio device migration
* Added 'magic' and 'version'fields to struct vfio_ap_config to
  handle ABI stability across host migrations.
  
Patch 6: Transition device migration state from STOP to STOP_COPY
* Refactored stop_copy file release functions

Patch 7: File ops called to save the vfio device migration state
* Drop mdevs_lock prior to calling the vfio_ap_store_queue_info
  function and re-acquire it afterward. The vfio_ap_store_queue_info
  executes the PQAP(TAPQ) instruction which could consume a 
  lot of time if there are an inordinate number of queues for which
  info needs to be retrieved. 

Patch 8: Transition device migration state from STOP to RESUMING
* Refactored resuming file release functions

Patch 9: Add method to set a new guest AP configuration
* Removed !q->matrix_mdev check in collect_queues_by_apid function
  as it is not necessary because all callers pass a matrix_mdev
  obtained from container context and will never be NULL.

Patch 10: File ops called to resume the vfio device migration
* Replaced pr_err calls with pr_err_ratelimited calls in 
  report_qinfo_incompatibilities() function along with all of the functions
  it calls to prevent the CPU stalls or a denial of service attacks.
* Added a scratch buffer that must be filled with vfio_ap_config header
  information (the new 'magic' and 'version' fields (see Patch 2 above)
  as well as the 'num_queues' field before allocating a vfio_ap_config
  object.
* Added write_in_progress flag to the vfio_ap_migration_data structure to
  reject concurrent write operations. This flag is checked when the
  vfio_ap_resuming_write() function is called and if already set, the
  function will return -EBUSY; otherwise, it will be set.

Patch 13:Callback to get the size of data to be migrated during guest
         migration
* modified vfio_ap_get_data_size function to call the vfio_ap_config_size
  function to set *stop_copy_length
* Use struct_size_t function to calculate config size

Patch 14: Add 'migratable' feature to sysfs 'features' attribute
* Do not display 'migratable' feature when the system is an SE guest;
  live guest migration is not supported for such guests.


Anthony Krowiak (15):
  s390/vfio-ap: Provide function to get the number of queues assigned to
    mdev
  s390/vfio-ap: Data structures for facilitating vfio device migration
  s390/vfio-ap: Functions to initialize/release vfio device migration
    data
  s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler
  s390/vfio-ap: Callback to get/set vfio device mig state during guest
    migration
  s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
  s390/vfio-ap: File ops called to save the vfio device migration state
  s390/vfio-ap: Transition device migration state from STOP to RESUMING
  s390/vfio-ap: Add method to set a new guest AP configuration
  s390/vfio-ap: File ops called to resume the vfio device migration
  s390/vfio-ap: Transition device migration state to STOP
  s390/vfio-ap: Transition device migration state from STOP to RUNNING
    and vice versa
  s390/vfio-ap: Callback to get the size of data to be migrated during
    guest migration
  s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
  s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst

 Documentation/arch/s390/vfio-ap.rst     |  616 ++++++--
 drivers/s390/crypto/Makefile            |    2 +-
 drivers/s390/crypto/vfio_ap_drv.c       |   14 +-
 drivers/s390/crypto/vfio_ap_migration.c | 1762 +++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_ops.c       |  297 ++--
 drivers/s390/crypto/vfio_ap_private.h   |   73 +
 6 files changed, 2557 insertions(+), 207 deletions(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_migration.c

-- 
2.53.0