Re: [PATCH v2 00/16] s390/vfio-ap: Add live guest migration support

Anthony Krowiak
2026-04-21T18:34:18Z

On 4/7/26 4:50 PM, Anthony Krowiak wrote:

PING

> This patch series implements live guest migration support for KVM guests
> with s390 AP (Adjunct Processor) devices passed through via the VFIO
> mediated device framework.
>
> Background
> ----------
>
> The vfio-ap device driver differs from typical VFIO device drivers in that
> it does not virtualize a physical device. Instead, it manages AP
> configuration metadata identifying the AP adapters, domains, and control
> domains to which a guest will be granted access. These AP resources are
> configured by assigning them to a vfio-ap mediated device via its sysfs
> assignment interfaces. When the fd for the VFIO device is opened by
> userspace, the vfio_ap device driver sets the guest's AP configuration
> from the metadata stored with the mediated device. As such, the AP devices
> are not accessed directly through the vfio_ap driver, so the driver has no
> internal device state to migrate. It's sole purpose during migration is to
> ensure that the AP configurations of the source and destination guests are
> compatible.
>
> Implementation Approach
> -----------------------
>
> This series implements the VFIO migration protocol using the STOP_COPY
> migration flow. The key aspects are:
>
> 1. Hardware Information Capture (Patches 1-2)
> - Store AP queue hardware characteristics at probe time
> - Provide access to queue objects for validation
>
> 2. Migration Infrastructure (Patches 3-5)
> - Define migration data structures
> - Initialize/release migration data on device open/close
>
> 3. State Machine Implementation (Patches 6-13)
> - Implement required VFIO migration callbacks
> - Handle state transitions: STOP → STOP_COPY → STOP (source)
> STOP → RESUMING → STOP (destination)
> - Use file streams for AP configuration data transfer
>
> 4. Validation and Callbacks (Patches 14-15)
> - Implement migration state and data size callbacks
> - Required for VFIO_DEVICE_FEATURE_MIGRATION support
>
> 5. Documentation (Patch 16)
> - Add live guest migration chapter to vfio-ap.rst
>
> Compatibility Validation
> ------------------------
>
> The series includes comprehensive validation to ensure source and
> destination AP configurations are compatible. For each queue, the following
> characteristics must match:
>
> - AP type (target must be same or newer than source)
> - Installed facilities (APSC, APQKM, AP4KC, SLCF)
> - Operating mode (CCA, Accelerator, XCP)
> - APXA facility setting
> - Classification (native vs stateless functions)
> - Queue usability (binding/associated state)
>
> When incompatibilities are detected, migration fails with detailed error
> messages identifying the specific queue and characteristic that caused
> the failure.
>
> Configuration Management
> ------------------------
>
> This implementation does not prevent configuration changes during
> migration. Configuration stability is an orchestration-layer
> responsibility, consistent with other VFIO device types. The driver's
> role is to validate configurations and provide clear diagnostics when
> incompatibilities are detected, enabling orchestration tools to implement
> appropriate policies.
>
> Change log v1 to v2
> -------------------
>
> - Removed patches that attempted to block configuration changes during
> migration due to inherent race conditions and incomplete protection
> - Simplified approach focuses on validation and error reporting
> - Reduced series from 18 to 16 patches
> - Rewrote the description in the cover letter to better describe the
> patch series, remove confusing comments as well as references to
> function provided by the patches that were removed.
>
> Anthony Krowiak (16):
> s390/vfio-ap: Store queue hardware info when probed
> s390/vfio-ap: Provide access to queue objects and related info
> s390/vfio-ap: Data structures for facilitating vfio device migration
> s390/vfio-ap: Initialize/release vfio device migration data
> s390-vfio-ap: Callback to set vfio device mig state during guest
> migration
> s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
> s390/vfio-ap: File ops called to save the vfio device migration state
> s390/vfio-ap: Transition device migration state from STOP to RESUMING
> s390/vfio-ap: File ops called to resume the vfio device migration
> s390/vfio-ap: Transition device migration state from RESUMING to STOP
> s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
> s390/vfio-ap: Transition device migration state from STOP to RUNNING
> and vice versa
> s390-vfio-ap: Callback to get the current vfio device migration state
> s390/vfio-ap: Callback to get the size of data to be migrated during
> guest migration
> s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
> s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
>
> Documentation/arch/s390/vfio-ap.rst | 325 +++++--
> drivers/s390/crypto/Makefile | 2 +-
> drivers/s390/crypto/vfio_ap_drv.c | 4 +-
> drivers/s390/crypto/vfio_ap_migration.c | 1095 +++++++++++++++++++++++
> drivers/s390/crypto/vfio_ap_ops.c | 66 +-
> drivers/s390/crypto/vfio_ap_private.h | 10 +
> 6 files changed, 1395 insertions(+), 107 deletions(-)
> create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>