Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

Anthony Krowiak
2026-03-31T11:17:19Z

On 3/30/26 12:27 PM, Alex Williamson wrote:
> On Wed, 25 Mar 2026 17:00:47 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> This patch series implements live guest migration of a guest to which AP
>> devices have been passed through. To better comprehend this design, one has
>> to understand that VFIO AP mediated device is not used to provide userspace
>> with direct access to a device as is the case with other devices that use
>> the VFIO framework to pass them through to a guest. The sole purpose of the
>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>> configuration is comprised of the AP adapter IDs (APID), AP queue
>> indexes (APQI) and domain numbers of the control domains to which a guest
>> will be granted access. Once the VFIO AP mediated device is attached to a
>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>> all access to the AP devices is handled by the s390 Interpretive Execution
>> facility; in other words, the vfio_ap device driver plays no role in
>> providing direct access to the AP devices in the guest's AP configuration.
>>
>> The only role that the vfio_ap device driver plays in the migration
>> process is to verify that the AP configuration for the source guest is
>> compatible with the AP configuration of the destination guest.
>> Incompatibility will result in a live guest migration failure.
>> In order to be compatible, the following requirements must be met:
>>
>> 1. The destination guest will be started with the same QEMU command line
>> as the source guest, so the mediated device supplying the AP
>> configuration on both guests must have the same name (UUID).
> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> supporting migration. You're really concerned more with the
> configuration/composition of the mdev device, so requiring the same
> UUID seems a bit arbitrary.

As stated above, the destination guest will be started with the same
QEMU command line as the source guest. Within that command line
will be a '-device' parameter like the following:

-device
'{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}

Note that sysfsdev is the path to the mdev named
62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or
the migration will fail with the following error:

error: device not found: mediated device
'62177883-f1bb-47f0-914d-32a22e3a8804' not found

>
>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>> guests must be compatible. As such, each AP configuration must meet
>> the following requirements:
>>
>> * Both guests must have the same number of APQNs
>>
>> * Each APQN assigned to the source guest must also be assigned to the
>> destination guest
>>
>> * Each APQN assigned to both guests must reference an AP queue with the
>> same hardware capabilities
> Why isn't this sufficient vs also requiring the same UUID?
>
>> Note: There is a forthcoming consumer of this series which will be a QEMU
>> patch series is entitled:
>> 'hw/vfio/ap: Implement live guest migration of guests using AP
>> devices'
>>
>> This design also adds a use case for enabling and disabling
>> migration of guests to which AP devices have been passed through. To
>> facilitate this, a new read/write sysfs 'migratable' attribute is added to
>> the mediated device. This attribute specifies whether the vfio device is
>> migratable (1) or not (0). When the value of this attribute is changed, the
>> vfio_ap device driver will signal an eventfd to userspace. It is up to
>> userspace to respond to the change by enabling or disabling migration of
>> the guest to which the mediated device is attached. The operation will be
>> rejected with a 'Device or resource busy' message if a migration is in
>> progress.
> This seems inherently racy. What happens if the device becomes
> unmigratable while it's being migrated?
>
> If userspace is deciding that the device is now unmigratable, why does
> it need to signal this through the kernel driver rather than with the
> userspace orchestration agent? The entire path seems unnecessary.
>
>> Userspace must also have a means for retrieving the value of the sysfs
>> 'migratable' attribute when the guest is started to initialize whether it
>> can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
>> struct vfio_device_info object passed to the ioctl will be extended with a
>> capability specifying the vfio device attributes. One of the attributes
>> will contain the value of the mediated device's 'migratable' attribute.
> This is just broken, it's redundant to our current device feature
> mechanism for exposing migration support. If you want the capability
> to create unmigratable devices statically, can't that be encompassed
> within the definition of the mdev type? Dynamic migration support just
> seems like it's involving the kernel in orchestration it shouldn't be a
> part of. Thanks,
>
> Alex
>
>> Anthony Krowiak (23):
>> s390/vfio-ap: Store queue hardware info when probed
>> s390/vfio-ap: Provide access to queue objects and related info
>> s390/vfio-ap: Add header file for xfer of vfio device caps to
>> userspace
>> MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
>> maintainers
>> s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
>> migration of guest
>> s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
>> s390/vfio-ap: Signal event to enable/disable live guest migration
>> s390/vfio-ap: Return value of sysfs migratable attribute from
>> VFIO_DEVICE_GET_INFO ioctl
>> s390/vfio-ap: Data structures for facilitating vfio device migration
>> s390/vfio-ap: Initialize/release vfio device migration data
>> s390-vfio-ap: Callback to set vfio device mig state during guest
>> migration
>> s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
>> s390/vfio-ap: File ops called to save the vfio device migration state
>> s390/vfio-ap: Transition device migration state from STOP to RESUMING
>> s390/vfio-ap: File ops called to resume the vfio device migration
>> s390/vfio-ap: Transition device migration state from RESUMING to STOP
>> s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
>> s390/vfio-ap: Transition device migration state from STOP to RUNNING
>> and vice versa
>> s390-vfio-ap: Callback to get the current vfio device migration state
>> s390/vfio-ap: Callback to get the size of data to be migrated during
>> guest migration
>> s390/vfio-ap: Provide API to query whether migration is in progress
>> s390/vfio-ap: Disallow blocking migration in progress
>> s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
>>
>> Documentation/arch/s390/vfio-ap.rst | 339 +++++--
>> MAINTAINERS | 1 +
>> drivers/s390/crypto/Makefile | 2 +-
>> drivers/s390/crypto/vfio_ap_drv.c | 4 +-
>> drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
>> drivers/s390/crypto/vfio_ap_ops.c | 263 +++++-
>> drivers/s390/crypto/vfio_ap_private.h | 20 +
>> include/uapi/linux/vfio.h | 2 +
>> include/uapi/linux/vfio_ap.h | 34 +
>> 9 files changed, 1685 insertions(+), 111 deletions(-)
>> create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
>> create mode 100644 include/uapi/linux/vfio_ap.h
>>