Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak <akrowiak@linux.ibm.com>]

On 3/31/26 1:40 PM, Alex Williamson wrote:
> On Tue, 31 Mar 2026 08:07:06 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> On 3/30/26 12:27 PM, Alex Williamson wrote:
>>> On Wed, 25 Mar 2026 17:00:47 -0400
>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>   
>>>> This patch series implements live guest migration of a guest to which AP
>>>> devices have been passed through. To better comprehend this design, one has
>>>> to understand that VFIO AP mediated device is not used to provide userspace
>>>> with direct access to a device as is the case with other devices that use
>>>> the VFIO framework to pass them through to a guest. The sole purpose of the
>>>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>>>> configuration is comprised of the AP adapter IDs (APID), AP queue
>>>> indexes (APQI) and domain numbers of the control domains to which a guest
>>>> will be granted access. Once the VFIO AP mediated device is attached to a
>>>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>>>> all access to the AP devices is handled by the s390 Interpretive Execution
>>>> facility; in other words, the vfio_ap device driver plays no role in
>>>> providing direct access to the AP devices in the guest's AP configuration.
>>>>
>>>> The only role that the vfio_ap device driver plays in the migration
>>>> process is to verify that the AP configuration for the source guest is
>>>> compatible with the AP configuration of the destination guest.
>>>> Incompatibility will result in a live guest migration failure.
>>>> In order to be compatible, the following requirements must be met:
>>>>
>>>> 1. The destination guest will be started with the same QEMU command line
>>>>      as the source guest, so the mediated device supplying the AP
>>>>      configuration on both guests must have the same name (UUID).
>>> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
>>> supporting migration.  You're really concerned more with the
>>> configuration/composition of the mdev device, so requiring the same
>>> UUID seems a bit arbitrary.
> Combining replies:
>
> On Tue, 31 Mar 2026 07:17:08 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>> As stated above, the destination guest will be started with the same
>> QEMU command line as the source guest. Within that command line
>> will be a '-device' parameter like the following:
>>
>> -device
>> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
>>
>> Note that sysfsdev is the path to the mdev named
>> 62177883-f1bb-47f0-914d-32a22e3a8804;
>> therefore, the mdev with that name must exist on the destination guest or
>> the migration will fail with the following error:
>>
>> error: device not found: mediated device
>> '62177883-f1bb-47f0-914d-32a22e3a8804' not found
> Then this is a requirement of your tooling, not a kernel requirement, not
> something the kernel should care about.  QEMU matches devices by their
> virtual bus path, not the sysfsdev or host attributes.  In the case of
> VF migration with vfio-pci variant drivers we cannot require that the
> source and target devices exist at the same bus address.  Ideally the
> pre-copy data from the source device to the target will include relevant
> configuration information to validate that the source and target are
> compatible, regardless of the uuid.

Maybe the problem here is stating that having the same UUID is a
requirement in the patch series description. I agree this is
not a kernel requirement; however, a live guest migration will not
succeed unless the destination host has a mediated device with
a UUID that matches that of the UUID of the mediated device used
to pass through s390 crypto devices to the source guest for the
reason I stated above. Would it help if I removed item #1 as a
requirement?

>
>>>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>>>      guests must be compatible. As such, each AP configuration must meet
>>>>      the following requirements:
>>>>
>>>>      * Both guests must have the same number of APQNs
>>>>
>>>>      * Each APQN assigned to the source guest must also be assigned to the
>>>>        destination guest
>>>>
>>>>      * Each APQN assigned to both guests must reference an AP queue with the
>>>>        same hardware capabilities
>>> Why isn't this sufficient vs also requiring the same UUID?
>> I explained why in my previous response.
> See above, userspace tooling requirements don't imply kernel
> requirement.

Got it.

> A userspace interface can still exist, I just don't find this path
> through the driver to the VM acceptable with this justification.
> Mechanisms already exist for the device to refuse a state transition or
> generate an error for a migration already in progress.  IMHO, it would
> be acceptable for the device to block a key change if the migration is
> already in progress.  If the key change cannot be represented in the
> migration data stream, then it's up to the orchestration of the
> migration to make sure they stay synchronized, but I don't see that
> the vfio uAPI needs to be involved.  Thanks,

As I stated in the cover letter description, all access to the AP 
devices is handled by the s390 Interpretive Execution facility, so 
without a complete redesign of the vfio_ap device driver, there is no 
way to know that a master key change is being requested. I do get your 
point, however, so I will figure out how best to handle this in 
userspace. Thanks for you review.
>
> Alex