Re: [PATCH 14/15] s390x/pci: Implement migration for emulated devices

[Cornelia Huck <cohuck@redhat.com>]

On Wed, Apr 08 2026, Matthew Rosato <mjrosato@linux.ibm.com> wrote:

> On 4/8/26 6:01 AM, Thomas Huth wrote:
>> On 02/04/2026 04.29, Konstantin Shkolnyy wrote:
>>> Implement zPCI device state migration, consequently enabling migration
>>> of VMs that have emulated PCI devices, whether virtio or not.
>>> Migration is allowed for devices whose function handle has the
>>> FH_SHM_EMUL bit set. For these devices QEMU will save and restore the
>>> state of its zPCI emulator.
>>>
>>> Passthrough devices will continue to block migration.
>>>
>>> Signed-off-by: Konstantin Shkolnyy <kshk@linux.ibm.com>
>>> ---
>>>   hw/s390x/s390-pci-bus.c         | 112 ++++++++++++++++++++++++++++++--
>>>   hw/s390x/s390-pci-inst.c        |   2 +-
>>>   include/hw/s390x/s390-pci-bus.h |   2 +
>>>   3 files changed, 110 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c
>>> index f339a0fd94..b3037b9a8a 100644
>>> --- a/hw/s390x/s390-pci-bus.c
>>> +++ b/hw/s390x/s390-pci-bus.c
>>> ...
>>> +static const VMStateDescription s390_pci_device_vmstate = {
>>> +    .name = TYPE_S390_PCI_DEVICE,
>>> +    .version_id = 1,
>>> +    .minimum_version_id = 1,
>>> +    .post_load = s390_pci_device_post_load,
>>> +    .fields = (const VMStateField[]) {
>>> +        VMSTATE_UINT32(state, S390PCIBusDevice),
>>> +        VMSTATE_UINT16(uid, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(idx, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(fh, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(fid, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(fid_defined, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(fmb_addr, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(fmb.format, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(fmb.sample, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(fmb.last_update, S390PCIBusDevice),
>>> +        VMSTATE_UINT64_ARRAY(fmb.counter, S390PCIBusDevice,
>>> +                ARRAY_SIZE(((S390PCIBusDevice *)0)->fmb.counter)),
>>> +        VMSTATE_UINT64(fmb.fmt0.dma_rbytes, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(fmb.fmt0.dma_wbytes, S390PCIBusDevice),
>>> +        VMSTATE_UINT8(isc, S390PCIBusDevice),
>>> +        VMSTATE_UINT16(noi, S390PCIBusDevice),
>>> +        VMSTATE_UINT8(sum, S390PCIBusDevice),
>>> +        VMSTATE_UINT8(pft, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(routes.adapter.ind_addr, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(routes.adapter.summary_addr, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(routes.adapter.ind_offset, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(routes.adapter.summary_offset, S390PCIBusDevice),
>>> +        VMSTATE_UINT32(routes.adapter.adapter_id, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(iommu_enabled, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(g_iota, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(pba, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(pal, S390PCIBusDevice),
>>> +        VMSTATE_UINT64(max_dma_limit, S390PCIBusDevice),
>>> +        VMSTATE_PTR_TO_IND_ADDR(summary_ind, S390PCIBusDevice),
>>> +        VMSTATE_PTR_TO_IND_ADDR(indicator, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(pci_unplug_request_processed, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(unplug_requested, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(interp, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(forwarding_assist, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(aif, S390PCIBusDevice),
>>> +        VMSTATE_BOOL(rtr_avail, S390PCIBusDevice),
>>> +        VMSTATE_END_OF_LIST()
>>> +    }
>>>   };
>>  Hi!
>> 
>> I assume this will likely break backward migration with old machine types?
>> 
>> Could you please try whether migrating a s390-ccw-virtio-10.2 machine
>> from an old version of QEMU to a version of QEMU that includes your
>> patches, and back from the new version to the old version works as
>> expected?
>> 
>> If not, you might need to add a ".needed = ..." line to the
>> s390_pci_device_vmstate, with a function that checks a variable from the
>> machine state whether the PCI state is allowed or not. And then enable
>> that variable only on the newest -11.1 machine type once it is available.
>
> +1, I was having the same thought but hadn't gotten around to trying it yet.
>
> We do already have a few examples for zPCI where we did this even though
> migration was not even supported yet to kind of lay the groundwork for
> backward migration ahead of time.
>
> But honestly, looking at just this series in isolation, my concern is
> things like an older machine type would not have the IOMMU replay
> functionality added in patch 1.
>
> So yes, please test to see what happens but make sure to include IOMMU
> activity in that testing.
>
> Tying this to a machine type is probably the safest bet, and given that
> virtio-pci is not the default virtio transport on s390x I don't think it
> would cause much hardship.
> 11.1 machine isn't available yet but I imagine will be soon after QEMU
> 11.0 release -- you can use Connie's series as a base:
> https://lore.kernel.org/qemu-devel/20260331140347.653404-1-cohuck@redhat.com/

I agree, that needs to be tied to the machine type.

I'll pick up the 11.1 machine types patch as soon as I prepare the first
s390 pull req for 11.1, so there should be a good base around soon.