Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Alex Williamson <alex@shazbot.org>]

On Wed, 1 Apr 2026 09:38:59 -0400
Anthony Krowiak <akrowiak@linux.ibm.com> wrote:

> On 3/31/26 1:40 PM, Alex Williamson wrote:
> > On Tue, 31 Mar 2026 08:07:06 -0400
> > Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
> >  
> >> On 3/30/26 12:27 PM, Alex Williamson wrote:  
> >>> On Wed, 25 Mar 2026 17:00:47 -0400
> >>> Anthony Krowiak <akrowiak@linux.ibm.com> wrote:
> >>>     
> >>>> This patch series implements live guest migration of a guest to which AP
> >>>> devices have been passed through. To better comprehend this design, one has
> >>>> to understand that VFIO AP mediated device is not used to provide userspace
> >>>> with direct access to a device as is the case with other devices that use
> >>>> the VFIO framework to pass them through to a guest. The sole purpose of the
> >>>> VFIO AP mediated device is to manage an AP configuration for a guest. An AP
> >>>> configuration is comprised of the AP adapter IDs (APID), AP queue
> >>>> indexes (APQI) and domain numbers of the control domains to which a guest
> >>>> will be granted access. Once the VFIO AP mediated device is attached to a
> >>>> guest, its AP configuration is set by the vfio_ap device driver. Once set,
> >>>> all access to the AP devices is handled by the s390 Interpretive Execution
> >>>> facility; in other words, the vfio_ap device driver plays no role in
> >>>> providing direct access to the AP devices in the guest's AP configuration.
> >>>>
> >>>> The only role that the vfio_ap device driver plays in the migration
> >>>> process is to verify that the AP configuration for the source guest is
> >>>> compatible with the AP configuration of the destination guest.
> >>>> Incompatibility will result in a live guest migration failure.
> >>>> In order to be compatible, the following requirements must be met:
> >>>>
> >>>> 1. The destination guest will be started with the same QEMU command line
> >>>>      as the source guest, so the mediated device supplying the AP
> >>>>      configuration on both guests must have the same name (UUID).  
> >>> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> >>> supporting migration.  You're really concerned more with the
> >>> configuration/composition of the mdev device, so requiring the same
> >>> UUID seems a bit arbitrary.  
> > Combining replies:
> >
> > On Tue, 31 Mar 2026 07:17:08 -0400
> > Anthony Krowiak <akrowiak@linux.ibm.com> wrote:  
> >> As stated above, the destination guest will be started with the same
> >> QEMU command line as the source guest. Within that command line
> >> will be a '-device' parameter like the following:
> >>
> >> -device
> >> '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
> >>
> >> Note that sysfsdev is the path to the mdev named
> >> 62177883-f1bb-47f0-914d-32a22e3a8804;
> >> therefore, the mdev with that name must exist on the destination guest or
> >> the migration will fail with the following error:
> >>
> >> error: device not found: mediated device
> >> '62177883-f1bb-47f0-914d-32a22e3a8804' not found  
> > Then this is a requirement of your tooling, not a kernel requirement, not
> > something the kernel should care about.  QEMU matches devices by their
> > virtual bus path, not the sysfsdev or host attributes.  In the case of
> > VF migration with vfio-pci variant drivers we cannot require that the
> > source and target devices exist at the same bus address.  Ideally the
> > pre-copy data from the source device to the target will include relevant
> > configuration information to validate that the source and target are
> > compatible, regardless of the uuid.  
> 
> Maybe the problem here is stating that having the same UUID is a
> requirement in the patch series description. I agree this is
> not a kernel requirement; however, a live guest migration will not
> succeed unless the destination host has a mediated device with
> a UUID that matches that of the UUID of the mediated device used
> to pass through s390 crypto devices to the source guest for the
> reason I stated above. Would it help if I removed item #1 as a
> requirement?

Yes, AIUI this is an artifact of your tooling.  The destination QEMU
can be started with any arbitrary UUID and should accept an incoming
migration, routing the migration to the correct device by the virtual
path, independent of the UUID.

> >>>> 2. The AP configuration assigned via the VFIO AP mediated device on both
> >>>>      guests must be compatible. As such, each AP configuration must meet
> >>>>      the following requirements:
> >>>>
> >>>>      * Both guests must have the same number of APQNs
> >>>>
> >>>>      * Each APQN assigned to the source guest must also be assigned to the
> >>>>        destination guest
> >>>>
> >>>>      * Each APQN assigned to both guests must reference an AP queue with the
> >>>>        same hardware capabilities  
> >>> Why isn't this sufficient vs also requiring the same UUID?  
> >> I explained why in my previous response.  
> > See above, userspace tooling requirements don't imply kernel
> > requirement.  
> 
> Got it.
> 
> > A userspace interface can still exist, I just don't find this path
> > through the driver to the VM acceptable with this justification.
> > Mechanisms already exist for the device to refuse a state transition or
> > generate an error for a migration already in progress.  IMHO, it would
> > be acceptable for the device to block a key change if the migration is
> > already in progress.  If the key change cannot be represented in the
> > migration data stream, then it's up to the orchestration of the
> > migration to make sure they stay synchronized, but I don't see that
> > the vfio uAPI needs to be involved.  Thanks,  
> 
> As I stated in the cover letter description, all access to the AP 
> devices is handled by the s390 Interpretive Execution facility, so 
> without a complete redesign of the vfio_ap device driver, there is no 
> way to know that a master key change is being requested. I do get your 
> point, however, so I will figure out how best to handle this in 
> userspace. Thanks for you review.

The cover letter has one sentence describing the Interpretive Execution
facility handling access to the device.  There are a lot of subtleties
there that are not obvious to laypersons.

It seems like the architecture really doesn't have the ability
to monitor state changes of the device, and therefore represent those
changes in the migration data stream, so you want to fallback to a
cooperative scheme with a side-band channel to interrupt or block
migration when an asynchronous configuration change happens.

I just don't see that such cooperation needs a new uAPI through the
device; either the device itself could perform the same within the
existing uAPI or the agent changing the configuration could coordinate
with the migration orchestration.  Thanks,

Alex