Re: [PATCH V4 00/43] Live update: vfio and iommufd

["Cédric Le Goater" <clg@redhat.com>]

On 5/29/25 21:23, Steve Sistare wrote:
> Support vfio and iommufd devices with the cpr-transfer live migration mode.
> Devices that do not support live migration can still support cpr-transfer,
> allowing live update to a new version of QEMU on the same host, with no loss
> of guest connectivity.
> 
> No user-visible interfaces are added.
> 
> For legacy containers:
> 
> Pass vfio device descriptors to new QEMU.  In new QEMU, during vfio_realize,
> skip the ioctls that configure the device, because it is already configured.
> 
> Use VFIO_DMA_UNMAP_FLAG_VADDR to abandon the old VA's for DMA mapped
> regions, and use VFIO_DMA_MAP_FLAG_VADDR to register the new VA in new
> QEMU and update the locked memory accounting.  The physical pages remain
> pinned, because the descriptor of the device that locked them remains open,
> so DMA to those pages continues without interruption.  Mediated devices are
> not supported, however, because they require the VA to always be valid, and
> there is a brief window where no VA is registered.
> 
> Save the MSI message area as part of vfio-pci vmstate, and pass the interrupt
> and notifier eventfd's to new QEMU.  New QEMU loads the MSI data, then the
> vfio-pci post_load handler finds the eventfds in CPR state, rebuilds vector
> data structures, and attaches the interrupts to the new KVM instance.  This
> logic also applies to iommufd containers.
> 
> For iommufd containers:
> 
> Use IOMMU_IOAS_MAP_FILE to register memory regions for DMA when they are
> backed by a file (including a memfd), so DMA mappings do not depend on VA,
> which can differ after live update.  This allows mediated devices to be
> supported.
> 
> Pass the iommufd and vfio device descriptors from old to new QEMU.  In new
> QEMU, during vfio_realize, skip the ioctls that configure the device, because
> it is already configured.
> 
> In new QEMU, call ioctl(IOMMU_IOAS_CHANGE_PROCESS) to update mm ownership and
> locked memory accounting.
> 
> Patches 4 to 12 are specific to legacy containers.
> Patches 25 to 41 are specific to iommufd containers.> The remainder apply to both.

Steve,

I am considering patches 1-23 for vfio-next. This is to be able to merge
quickly a first part because we still have time ahead of us and to catch
issues early. It will also help John to rebase.

I think patch 16 can be simplified. If you agree, and Michael too, please
resend just this patch. I will update the series and send a PR.

Thanks,

C.

  




  
> Changes from previous versions:
>    * V1 of this series contains minor changes from the "Live update: vfio" and
>      "Live update: iommufd" series, mainly bug fixes and refactored patches.
> 
> Changes in V2:
>    * refactored various vfio code snippets into new cpr helpers
>    * refactored vfio struct members into cpr-specific structures
>    * refactored various small changes into their own patches
>    * split complex patches.  Notably:
>      - split "refactor for cpr" into 5 patches
>      - split "reconstruct device" into 4 patches
>    * refactored vfio_connect_container using helpers and made its
>      error recovery more robust.
>    * moved vfio pci msi/vector/intx cpr functions to cpr.c
>    * renamed "reused" to cpr_reused and cpr.reused
>    * squashed vfio_cpr_[un]register_container to their call sites
>    * simplified iommu_type setting after cpr
>    * added cpr_open_fd and cpr_is_incoming helpers
>    * removed changes from vfio_legacy_dma_map, and instead temporarily
>      override dma_map and dma_unmap ops.
>    * deleted error_report and returned Error to callers where possible.
>    * simplified the memory_get_xlat_addr interface
>    * fixed flags passed to iommufd_backend_alloc_hwpt
>    * defined MIG_PRI_UNINITIALIZED
>    * added maintainers
> 
> Changes in V3:
>    * removed cleanup patches that were already pulled
>    * rebased to latest master
> 
> Changes in V4:
>    * added SPDX-License-Identifier
>    * patch "vfio/container: preserve descriptors"
>      - rewrote search loop in vfio_container_connect
>      - do not return pfd from vfio_cpr_container_match
>      - add helper for VFIO_GROUP_GET_DEVICE_FD
>    * deleted patch "export vfio_legacy_dma_map"
>    * patch "vfio/container: restore DMA vaddr"
>      - deleted redundant error_report from vfio_legacy_cpr_dma_map
>      - save old dma_map function
>    * patch "vfio-pci: skip reset during cpr"
>      - use cpr_is_incoming instead of cpr_reused
>    * renamed err -> local_err in all new code
>    * patch "export MSI functions"
>      -  renamed with vfio_pci prefix, and defined wrappers for low level
>         routines instead of exporting them.
>    * patch "close kvm after cpr"
>      - fixed build error for !CONFIG_KVM
>    * added the cpr_resave_fd helper
>    * dropped patch "pass ramblock to vfio_container_dma_map", relying on
>      "pass MemoryRegion" from the vfio-user series instead.
>    * deleted "reused" variables, replaced with cpr_is_incoming()
>    * renamed cpr_needed_for_reuse -> cpr_incoming_needed
>    * rewrote patch "pci: skip reset during cpr"
>    * rebased to latest master
> 
>    for iommufd:
>      * deleted redundant error_report from iommufd_backend_map_file_dma
>      * added interface doc for dma_map_file
>      * check return value of cpr_open_fd
>      * deleted "export iommufd_cdev_get_info_iova_range"
>      * deleted "reconstruct device"
>      * deleted "reconstruct hw_caps"
>      * deleted "define hwpt constructors"
>      * seperated cpr registration for iommufd be and vfio container
>      * correctly attach to multiple containers per iommufd using ioas_id
>      * simplified "reconstruct hwpt" by matching against hwpt_id.
>      * added patch "add vfio_device_free_name"
> 
> 
> Steve Sistare (43):
>    MAINTAINERS: Add reviewer for CPR
>    vfio: return mr from vfio_get_xlat_addr
>    vfio/container: pass MemoryRegion to DMA operations
>    vfio/pci: vfio_pci_put_device on failure
>    migration: cpr helpers
>    migration: lower handler priority
>    vfio: vfio_find_ram_discard_listener
>    vfio: move vfio-cpr.h
>    vfio/container: register container for cpr
>    vfio/container: preserve descriptors
>    vfio/container: discard old DMA vaddr
>    vfio/container: restore DMA vaddr
>    vfio/container: mdev cpr blocker
>    vfio/container: recover from unmap-all-vaddr failure
>    pci: export msix_is_pending
>    pci: skip reset during cpr
>    vfio-pci: skip reset during cpr
>    vfio/pci: vfio_pci_vector_init
>    vfio/pci: vfio_notifier_init
>    vfio/pci: pass vector to virq functions
>    vfio/pci: vfio_notifier_init cpr parameters
>    vfio/pci: vfio_notifier_cleanup
>    vfio/pci: export MSI functions
>    vfio-pci: preserve MSI
>    vfio-pci: preserve INTx
>    migration: close kvm after cpr
>    migration: cpr_get_fd_param helper
>    backends/iommufd: iommufd_backend_map_file_dma
>    backends/iommufd: change process ioctl
>    physmem: qemu_ram_get_fd_offset
>    vfio/iommufd: use IOMMU_IOAS_MAP_FILE
>    vfio/iommufd: invariant device name
>    vfio/iommufd: add vfio_device_free_name
>    vfio/iommufd: device name blocker
>    vfio/iommufd: register container for cpr
>    migration: vfio cpr state hook
>    vfio/iommufd: cpr state
>    vfio/iommufd: preserve descriptors
>    vfio/iommufd: reconstruct device
>    vfio/iommufd: reconstruct hwpt
>    vfio/iommufd: change process
>    iommufd: preserve DMA mappings
>    vfio/container: delete old cpr register
> 
>   MAINTAINERS                           |  10 ++
>   hw/vfio/pci.h                         |  10 ++
>   hw/vfio/vfio-cpr.h                    |  15 --
>   include/exec/cpu-common.h             |   1 +
>   include/hw/pci/msix.h                 |   1 +
>   include/hw/pci/pci_device.h           |   3 +
>   include/hw/vfio/vfio-container-base.h |  38 ++++-
>   include/hw/vfio/vfio-container.h      |   2 +
>   include/hw/vfio/vfio-cpr.h            |  78 +++++++++
>   include/hw/vfio/vfio-device.h         |   5 +
>   include/migration/cpr.h               |  21 +++
>   include/migration/vmstate.h           |   6 +-
>   include/system/iommufd.h              |   6 +
>   include/system/kvm.h                  |   1 +
>   include/system/memory.h               |  19 ++-
>   accel/kvm/kvm-all.c                   |  28 ++++
>   accel/stubs/kvm-stub.c                |   5 +
>   backends/iommufd.c                    | 101 +++++++++++-
>   hw/pci/msix.c                         |   2 +-
>   hw/pci/pci.c                          |   5 +
>   hw/vfio/ap.c                          |   2 +-
>   hw/vfio/ccw.c                         |   2 +-
>   hw/vfio/container-base.c              |  13 +-
>   hw/vfio/container.c                   | 101 +++++++++---
>   hw/vfio/cpr-iommufd.c                 | 220 ++++++++++++++++++++++++++
>   hw/vfio/cpr-legacy.c                  | 288 ++++++++++++++++++++++++++++++++++
>   hw/vfio/cpr.c                         | 161 +++++++++++++++++--
>   hw/vfio/device.c                      |  40 +++--
>   hw/vfio/helpers.c                     |  10 ++
>   hw/vfio/iommufd.c                     |  86 ++++++++--
>   hw/vfio/listener.c                    |  93 +++++++----
>   hw/vfio/pci.c                         | 232 ++++++++++++++++++++-------
>   hw/vfio/platform.c                    |   2 +-
>   hw/vfio/vfio-stubs.c                  |  13 ++
>   hw/virtio/vhost-vdpa.c                |   9 +-
>   migration/cpr-transfer.c              |  18 +++
>   migration/cpr.c                       |  95 +++++++++--
>   migration/migration.c                 |   1 +
>   migration/savevm.c                    |   4 +-
>   system/memory.c                       |  32 +---
>   system/physmem.c                      |   5 +
>   backends/trace-events                 |   2 +
>   hw/vfio/meson.build                   |   4 +
>   43 files changed, 1576 insertions(+), 214 deletions(-)
>   delete mode 100644 hw/vfio/vfio-cpr.h
>   create mode 100644 include/hw/vfio/vfio-cpr.h
>   create mode 100644 hw/vfio/cpr-iommufd.c
>   create mode 100644 hw/vfio/cpr-legacy.c
>   create mode 100644 hw/vfio/vfio-stubs.c
> 
> base-commit: d2e9b78162e31b1eaf20f3a4f563da82da56908d