Re: [PATCH v2 00/16] s390/vfio-ap: Add live guest migration support

[Anthony Krowiak <akrowiak@linux.ibm.com>]

On 4/7/26 4:50 PM, Anthony Krowiak wrote:

A patch review would be greatly appreciated.

> This patch series implements live guest migration support for KVM guests
> with s390 AP (Adjunct Processor) devices passed through via the VFIO
> mediated device framework.
>
> Background
> ----------
>
> The vfio-ap device driver differs from typical VFIO device drivers in that
> it does not virtualize a physical device. Instead, it manages AP
> configuration metadata identifying the AP adapters, domains, and control
> domains to which a guest will be granted access. These AP resources are
> configured by assigning them to a vfio-ap mediated device via its sysfs
> assignment interfaces. When the fd for the VFIO device is opened by
> userspace, the vfio_ap device driver sets the guest's AP configuration
> from the metadata stored with the mediated device. As such, the AP devices
> are not accessed directly through the vfio_ap driver, so the driver has no
> internal device state to migrate. It's sole purpose during migration is to
> ensure that the AP configurations of the source and destination guests are
> compatible.
>
> Implementation Approach
> -----------------------
>
> This series implements the VFIO migration protocol using the STOP_COPY
> migration flow. The key aspects are:
>
> 1. Hardware Information Capture (Patches 1-2)
>     - Store AP queue hardware characteristics at probe time
>     - Provide access to queue objects for validation
>
> 2. Migration Infrastructure (Patches 3-5)
>     - Define migration data structures
>     - Initialize/release migration data on device open/close
>
> 3. State Machine Implementation (Patches 6-13)
>     - Implement required VFIO migration callbacks
>     - Handle state transitions: STOP → STOP_COPY → STOP (source)
>                                 STOP → RESUMING → STOP (destination)
>     - Use file streams for AP configuration data transfer
>
> 4. Validation and Callbacks (Patches 14-15)
>     - Implement migration state and data size callbacks
>     - Required for VFIO_DEVICE_FEATURE_MIGRATION support
>
> 5. Documentation (Patch 16)
>     - Add live guest migration chapter to vfio-ap.rst
>
> Compatibility Validation
> ------------------------
>
> The series includes comprehensive validation to ensure source and
> destination AP configurations are compatible. For each queue, the following
> characteristics must match:
>
> - AP type (target must be same or newer than source)
> - Installed facilities (APSC, APQKM, AP4KC, SLCF)
> - Operating mode (CCA, Accelerator, XCP)
> - APXA facility setting
> - Classification (native vs stateless functions)
> - Queue usability (binding/associated state)
>
> When incompatibilities are detected, migration fails with detailed error
> messages identifying the specific queue and characteristic that caused
> the failure.
>
> Configuration Management
> ------------------------
>
> This implementation does not prevent configuration changes during
> migration. Configuration stability is an orchestration-layer
> responsibility, consistent with other VFIO device types. The driver's
> role is to validate configurations and provide clear diagnostics when
> incompatibilities are detected, enabling orchestration tools to implement
> appropriate policies.
>
> Change log v1 to v2
> -------------------
>
> - Removed patches that attempted to block configuration changes during
>    migration due to inherent race conditions and incomplete protection
> - Simplified approach focuses on validation and error reporting
> - Reduced series from 18 to 16 patches
> - Rewrote the description in the cover letter to better describe the
>    patch series, remove confusing comments as well as references to
>    function provided by the patches that were removed.
>
> Anthony Krowiak (16):
>    s390/vfio-ap: Store queue hardware info when probed
>    s390/vfio-ap: Provide access to queue objects and related info
>    s390/vfio-ap: Data structures for facilitating vfio device migration
>    s390/vfio-ap: Initialize/release vfio device migration data
>    s390-vfio-ap: Callback to set vfio device mig state during guest
>      migration
>    s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
>    s390/vfio-ap: File ops called to save the vfio device migration state
>    s390/vfio-ap: Transition device migration state from STOP to RESUMING
>    s390/vfio-ap: File ops called to resume the vfio device migration
>    s390/vfio-ap: Transition device migration state from RESUMING to STOP
>    s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
>    s390/vfio-ap: Transition device migration state from STOP to RUNNING
>      and vice versa
>    s390-vfio-ap: Callback to get the current vfio device migration state
>    s390/vfio-ap: Callback to get the size of data to be migrated during
>      guest migration
>    s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
>    s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
>
>   Documentation/arch/s390/vfio-ap.rst     |  325 +++++--
>   drivers/s390/crypto/Makefile            |    2 +-
>   drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
>   drivers/s390/crypto/vfio_ap_migration.c | 1095 +++++++++++++++++++++++
>   drivers/s390/crypto/vfio_ap_ops.c       |   66 +-
>   drivers/s390/crypto/vfio_ap_private.h   |   10 +
>   6 files changed, 1395 insertions(+), 107 deletions(-)
>   create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>