Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

Anthony Krowiak
2026-04-02T12:03:49Z

On 4/1/26 12:57 PM, Alex Williamson wrote:
> On Wed, 1 Apr 2026 09:38:59 -0400
> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> On 3/31/26 1:40 PM, Alex Williamson wrote:
>>> On Tue, 31 Mar 2026 08:07:06 -0400
>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>
>>>> On 3/30/26 12:27 PM, Alex Williamson wrote:
>>>>> On Wed, 25 Mar 2026 17:00:47 -0400
>>>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>>>
>>>>>> This patch series implements live guest migration of a guest to which AP
>>>>>> devices have been passed through. To better comprehend this design, one has
>>>>>> to understand that VFIO AP mediated device is not used to provide userspace
>>>>>> with direct access to a device as is the case with other devices that use
>>>>>> the VFIO framework to pass them through to a guest. The sole purpose of the
>>>>>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
>>>>>> configuration is comprised of the AP adapter IDs (APID), AP queue
>>>>>> indexes (APQI) and domain numbers of the control domains to which a guest
>>>>>> will be granted access. Once the VFIO AP mediated device is attached to a
>>>>>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
>>>>>> all access to the AP devices is handled by the s390 Interpretive Execution
>>>>>> facility; in other words, the vfio_ap device driver plays no role in
>>>>>> providing direct access to the AP devices in the guest's AP configuration.
>>>>>>
>>>>>> The only role that the vfio_ap device driver plays in the migration
>>>>>> process is to verify that the AP configuration for the source guest is
>>>>>> compatible with the AP configuration of the destination guest.
>>>>>> Incompatibility will result in a live guest migration failure.
>>>>>> In order to be compatible, the following requirements must be met:
>>>>>>
>>>>>> 1. The destination guest will be started with the same QEMU command line
>>>>>> as the source guest, so the mediated device supplying the AP
>>>>>> configuration on both guests must have the same name (UUID).
>>>>> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
>>>>> supporting migration. You're really concerned more with the
>>>>> configuration/composition of the mdev device, so requiring the same
>>>>> UUID seems a bit arbitrary.
>>> Combining replies:
>>>
>>> On Tue, 31 Mar 2026 07:17:08 -0400
>>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
>>>> As stated above, the destination guest will be started with the same
>>>> QEMU command line as the source guest. Within that command line
>>>> will be a '-device' parameter like the following:
>>>>
>>>> -device
>>>> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
>>>>
>>>> Note that sysfsdev is the path to the mdev named
>>>> 62177883-f1bb-47f0-914d-32a22e3a8804;
>>>> therefore, the mdev with that name must exist on the destination guest or
>>>> the migration will fail with the following error:
>>>>
>>>> error: device not found: mediated device
>>>> '62177883-f1bb-47f0-914d-32a22e3a8804' not found
>>> Then this is a requirement of your tooling, not a kernel requirement, not
>>> something the kernel should care about. QEMU matches devices by their
>>> virtual bus path, not the sysfsdev or host attributes. In the case of
>>> VF migration with vfio-pci variant drivers we cannot require that the
>>> source and target devices exist at the same bus address. Ideally the
>>> pre-copy data from the source device to the target will include relevant
>>> configuration information to validate that the source and target are
>>> compatible, regardless of the uuid.
>> Maybe the problem here is stating that having the same UUID is a
>> requirement in the patch series description. I agree this is
>> not a kernel requirement; however, a live guest migration will not
>> succeed unless the destination host has a mediated device with
>> a UUID that matches that of the UUID of the mediated device used
>> to pass through s390 crypto devices to the source guest for the
>> reason I stated above. Would it help if I removed item #1 as a
>> requirement?
> Yes, AIUI this is an artifact of your tooling. The destination QEMU
> can be started with any arbitrary UUID and should accept an incoming
> migration, routing the migration to the correct device by the virtual
> path, independent of the UUID.

If by tooling you are referring to libvirt and/or quemu, then I agree;
this is an artifact of tooling. According to
https://www.linux-kvm.org/page/Migration#Requirements,
"the guest on the destination must be started the same way it was
started on the source". s390 guests are run under qemu, so the
qemu command line used to start the source guest will also be used
to start the destination guest. To pass through
AP devices to the guest, the mediated device that provides the guest's
AP configuration - i.e., the crypto devices to which the guest is granted
access - must be identified on the qemu command line. As stated
previously, the following qemu command line parameter is included
for this purpose:

-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}

When the qemu command is used to start the destination guest, the same
qemu command line will be used to start it. As such, qemu will try to open
an fd for that device. If the mdev with UUID does not exist, the migration
will fail. In other words, I don't think the guest can be started with any
arbitrary UUID. Am I wrong about this?

In any case, it is probably not relevant to mention this for this patch
series
since it is not a KVM/kernel requirement, so I will remove it from the
cover-letter description.

>
>>>>>> 2. The AP configuration assigned via the VFIO AP mediated device on both
>>>>>> guests must be compatible. As such, each AP configuration must meet
>>>>>> the following requirements:
>>>>>>
>>>>>> * Both guests must have the same number of APQNs
>>>>>>
>>>>>> * Each APQN assigned to the source guest must also be assigned to the
>>>>>> destination guest
>>>>>>
>>>>>> * Each APQN assigned to both guests must reference an AP queue with the
>>>>>> same hardware capabilities
>>>>> Why isn't this sufficient vs also requiring the same UUID?
>>>> I explained why in my previous response.
>>> See above, userspace tooling requirements don't imply kernel
>>> requirement.
>> Got it.
>>
>>> A userspace interface can still exist, I just don't find this path
>>> through the driver to the VM acceptable with this justification.
>>> Mechanisms already exist for the device to refuse a state transition or
>>> generate an error for a migration already in progress. IMHO, it would
>>> be acceptable for the device to block a key change if the migration is
>>> already in progress. If the key change cannot be represented in the
>>> migration data stream, then it's up to the orchestration of the
>>> migration to make sure they stay synchronized, but I don't see that
>>> the vfio uAPI needs to be involved. Thanks,
>> As I stated in the cover letter description, all access to the AP
>> devices is handled by the s390 Interpretive Execution facility, so
>> without a complete redesign of the vfio_ap device driver, there is no
>> way to know that a master key change is being requested. I do get your
>> point, however, so I will figure out how best to handle this in
>> userspace. Thanks for you review.
> The cover letter has one sentence describing the Interpretive Execution
> facility handling access to the device. There are a lot of subtleties
> there that are not obvious to laypersons.
>
> It seems like the architecture really doesn't have the ability
> to monitor state changes of the device, and therefore represent those
> changes in the migration data stream, so you want to fallback to a
> cooperative scheme with a side-band channel to interrupt or block
> migration when an asynchronous configuration change happens.
>
> I just don't see that such cooperation needs a new uAPI through the
> device; either the device itself could perform the same within the
> existing uAPI or the agent changing the configuration could coordinate
> with the migration orchestration. Thanks,

I understand your view on this and have already agreed to remove
the new uAPI.

>
> Alex