Re: [PATCH v7 14/15] s390x/pci: Implement migration for emulated devices

[Matthew Rosato <mjrosato@linux.ibm.com>]

On 8/18/26 1:25 PM, Konstantin Shkolnyy wrote:
> Implement zPCI device state migration, consequently enabling migration
> of VMs that have emulated PCI devices, whether virtio or not.
> Migration is allowed for devices whose function handle has the
> FH_SHM_EMUL bit set. For these devices QEMU will save and restore the
> state of its zPCI emulator.
> 
> This will enable emulated PCI migration starting with s390-ccw-virtio-11.2.
> 
> Passthrough devices will continue to block migration.
> 
> Reviewed-by: Farhan Ali<alifm@linux.ibm.com>
> Signed-off-by: Konstantin Shkolnyy <kshk@linux.ibm.com>
> ---
>  hw/s390x/s390-pci-bus.c         | 195 +++++++++++++++++++++++++++++++-
>  hw/s390x/s390-pci-inst.c        |   2 +-
>  hw/s390x/s390-virtio-ccw.c      |   4 +
>  include/hw/s390x/s390-pci-bus.h |   4 +
>  4 files changed, 199 insertions(+), 6 deletions(-)
> 

One high level comment is that S390pciState has some fields in it that
need consideration.

pending_sei seems like it needs to be handled for sure.

Migrating next_idx would probably only be an optimization; it should get
re-calculated the next time a device is plugged if we don't migrate it.

I think we are OK with zpci_groups because we are only migrating
emulated devices, they should get recreated as-expected on the target.
If we were to ever change the default CLP payload, it would need to be
on a qemu machine boundary.
That would need to be re-visited for vfio-pci migration.  Basically, a
guest does not expect the CLP payload to change.

And then same for next_sim_grp, this is only used for vfio-pci devices
that disable interpretation support.
Not necessary now but would need to be re-visited for vfio-pci migration.

zpci_dma_limit is going to be null without vfio-pci; if/when vfio-pci is
supported I think the list will need to be re-populated (not migrated)
as part of updating the max_dma_limit and replaying the IOMMU.

bus_no, bus, zpci_table, iommu_table and zpci_devs all need to be
reconstructed on the target + fixed up during load for zpci_devs and
zpci_table.

[...]
>  static const TypeInfo s390_pcibus_info = {
>      .name = TYPE_S390_PCI_BUS,
>      .parent = TYPE_BUS,
>      .instance_size = sizeof(S390PCIBus),
> +    /*
> +     * Implement get_dev_path() to provide each zpci device with a unique
> +     * stable UID-based bus "path". The "path" is used as part of idstr in the
> +     * migration stream, making idstr unique and instance_id always 0.
> +     * For migration to succeed, (idstr+instance_id) must match those generated
> +     * during QEMU start. Without unique idstr, QEMU will generate variable
> +     * instance_id to distinquish devices, and that instance_id can change

Typo 's/distinquish/distinguish/' (replace the Q with a G)

[...]

> +        VMSTATE_BOOL(iommu_enabled, S390PCIBusDevice),
> +        VMSTATE_UINT64(g_iota, S390PCIBusDevice),
> +        VMSTATE_UINT64(pba, S390PCIBusDevice),
> +        VMSTATE_UINT64(pal, S390PCIBusDevice),
> +        VMSTATE_UINT64(max_dma_limit, S390PCIBusDevice),

As mentioned in an earlier patch, let's not migrate this value.

Since you are only migrating emulated devices for now and fencing
vfio-pci, it will always be 0.  Set it to 0 unconditionally during load,
before the iommu replay.

If we enable migration of vfio-pci in the future, we would have to
acquire the new DMA limit on the target side prior to the iommu replay
for those devices (while emulated devices remain a 0).