Re: [PATCH V5 00/38] Live update: vfio and iommufd

["Cédric Le Goater" <clg@redhat.com>]

On 6/10/25 17:39, Steve Sistare wrote:
> Support vfio and iommufd devices with the cpr-transfer live migration mode.
> Devices that do not support live migration can still support cpr-transfer,
> allowing live update to a new version of QEMU on the same host, with no loss
> of guest connectivity.
> 
> No user-visible interfaces are added.
> 
> For legacy containers:
> 
> Pass vfio device descriptors to new QEMU.  In new QEMU, during vfio_realize,
> skip the ioctls that configure the device, because it is already configured.
> 
> Use VFIO_DMA_UNMAP_FLAG_VADDR to abandon the old VA's for DMA mapped
> regions, and use VFIO_DMA_MAP_FLAG_VADDR to register the new VA in new
> QEMU and update the locked memory accounting.  The physical pages remain
> pinned, because the descriptor of the device that locked them remains open,
> so DMA to those pages continues without interruption.  Mediated devices are
> not supported, however, because they require the VA to always be valid, and
> there is a brief window where no VA is registered.
> 
> Save the MSI message area as part of vfio-pci vmstate, and pass the interrupt
> and notifier eventfd's to new QEMU.  New QEMU loads the MSI data, then the
> vfio-pci post_load handler finds the eventfds in CPR state, rebuilds vector
> data structures, and attaches the interrupts to the new KVM instance.  This
> logic also applies to iommufd containers.
> 
> For iommufd containers:
> 
> Use IOMMU_IOAS_MAP_FILE to register memory regions for DMA when they are
> backed by a file (including a memfd), so DMA mappings do not depend on VA,
> which can differ after live update.  This allows mediated devices to be
> supported.
> 
> Pass the iommufd and vfio device descriptors from old to new QEMU.  In new
> QEMU, during vfio_realize, skip the ioctls that configure the device, because
> it is already configured.
> 
> In new QEMU, call ioctl(IOMMU_IOAS_CHANGE_PROCESS) to update mm ownership and
> locked memory accounting.
> 
> Patches 3 to 8 are specific to legacy containers.
> Patches 21 to 36 are specific to iommufd containers.
> The remainder apply to both.
> 
> Changes from previous versions:
>    * V1 of this series contains minor changes from the "Live update: vfio" and
>      "Live update: iommufd" series, mainly bug fixes and refactored patches.
> 
> Changes in V2:
>    * refactored various vfio code snippets into new cpr helpers
>    * refactored vfio struct members into cpr-specific structures
>    * refactored various small changes into their own patches
>    * split complex patches.  Notably:
>      - split "refactor for cpr" into 5 patches
>      - split "reconstruct device" into 4 patches
>    * refactored vfio_connect_container using helpers and made its
>      error recovery more robust.
>    * moved vfio pci msi/vector/intx cpr functions to cpr.c
>    * renamed "reused" to cpr_reused and cpr.reused
>    * squashed vfio_cpr_[un]register_container to their call sites
>    * simplified iommu_type setting after cpr
>    * added cpr_open_fd and cpr_is_incoming helpers
>    * removed changes from vfio_legacy_dma_map, and instead temporarily
>      override dma_map and dma_unmap ops.
>    * deleted error_report and returned Error to callers where possible.
>    * simplified the memory_get_xlat_addr interface
>    * fixed flags passed to iommufd_backend_alloc_hwpt
>    * defined MIG_PRI_UNINITIALIZED
>    * added maintainers
> 
> Changes in V3:
>    * removed cleanup patches that were already pulled
>    * rebased to latest master
> 
> Changes in V4:
>    * added SPDX-License-Identifier
>    * patch "vfio/container: preserve descriptors"
>      - rewrote search loop in vfio_container_connect
>      - do not return pfd from vfio_cpr_container_match
>      - add helper for VFIO_GROUP_GET_DEVICE_FD
>    * deleted patch "export vfio_legacy_dma_map"
>    * patch "vfio/container: restore DMA vaddr"
>      - deleted redundant error_report from vfio_legacy_cpr_dma_map
>      - save old dma_map function
>    * patch "vfio-pci: skip reset during cpr"
>      - use cpr_is_incoming instead of cpr_reused
>    * renamed err -> local_err in all new code
>    * patch "export MSI functions"
>      -  renamed with vfio_pci prefix, and defined wrappers for low level
>         routines instead of exporting them.
>    * patch "close kvm after cpr"
>      - fixed build error for !CONFIG_KVM
>    * added the cpr_resave_fd helper
>    * dropped patch "pass ramblock to vfio_container_dma_map", relying on
>      "pass MemoryRegion" from the vfio-user series instead.
>    * deleted "reused" variables, replaced with cpr_is_incoming()
>    * renamed cpr_needed_for_reuse -> cpr_incoming_needed
>    * rewrote patch "pci: skip reset during cpr"
>    * rebased to latest master
> 
>    for iommufd:
>      * deleted redundant error_report from iommufd_backend_map_file_dma
>      * added interface doc for dma_map_file
>      * check return value of cpr_open_fd
>      * deleted "export iommufd_cdev_get_info_iova_range"
>      * deleted "reconstruct device"
>      * deleted "reconstruct hw_caps"
>      * deleted "define hwpt constructors"
>      * separated cpr registration for iommufd be and vfio container
>      * correctly attach to multiple containers per iommufd using ioas_id
>      * simplified "reconstruct hwpt" by matching against hwpt_id.
>      * added patch "add vfio_device_free_name"
> 
> Changes in V5:
>    * dropped: vfio/pci: vfio_pci_put_device on failure
>    * added: "vfio: doc changes for cpr"
>    * deleted unnecessary include of vfio-cpr.h
>    * fixed compilation for !CONFIG_VFIO and !CONFIG_IOMMUFD
>    * misc minor changes
>    * Added RB's, rebased to master
> 
> Steve Sistare (38):
>    migration: cpr helpers
>    migration: lower handler priority
>    vfio/container: register container for cpr
>    vfio/container: preserve descriptors
>    vfio/container: discard old DMA vaddr
>    vfio/container: restore DMA vaddr
>    vfio/container: mdev cpr blocker
>    vfio/container: recover from unmap-all-vaddr failure
>    pci: export msix_is_pending
>    pci: skip reset during cpr
>    vfio-pci: skip reset during cpr
>    vfio/pci: vfio_pci_vector_init
>    vfio/pci: vfio_notifier_init
>    vfio/pci: pass vector to virq functions
>    vfio/pci: vfio_notifier_init cpr parameters
>    vfio/pci: vfio_notifier_cleanup
>    vfio/pci: export MSI functions
>    vfio-pci: preserve MSI
>    vfio-pci: preserve INTx
>    migration: close kvm after cpr
>    migration: cpr_get_fd_param helper
>    backends/iommufd: iommufd_backend_map_file_dma
>    backends/iommufd: change process ioctl
>    physmem: qemu_ram_get_fd_offset
>    vfio/iommufd: use IOMMU_IOAS_MAP_FILE
>    vfio/iommufd: invariant device name
>    vfio/iommufd: add vfio_device_free_name
>    vfio/iommufd: device name blocker
>    vfio/iommufd: register container for cpr
>    migration: vfio cpr state hook
>    vfio/iommufd: cpr state
>    vfio/iommufd: preserve descriptors
>    vfio/iommufd: reconstruct device
>    vfio/iommufd: reconstruct hwpt
>    vfio/iommufd: change process
>    iommufd: preserve DMA mappings
>    vfio/container: delete old cpr register
>    vfio: doc changes for cpr
> 
>   docs/devel/migration/CPR.rst          |   5 +-
>   qapi/migration.json                   |   6 +-
>   hw/vfio/pci.h                         |  10 ++
>   include/exec/cpu-common.h             |   1 +
>   include/hw/pci/msix.h                 |   1 +
>   include/hw/pci/pci_device.h           |   3 +
>   include/hw/vfio/vfio-container-base.h |  18 +++
>   include/hw/vfio/vfio-container.h      |   2 +
>   include/hw/vfio/vfio-cpr.h            |  66 +++++++-
>   include/hw/vfio/vfio-device.h         |   5 +
>   include/migration/cpr.h               |  21 +++
>   include/migration/vmstate.h           |   6 +-
>   include/system/iommufd.h              |   7 +
>   include/system/kvm.h                  |   1 +
>   accel/kvm/kvm-all.c                   |  32 ++++
>   accel/stubs/kvm-stub.c                |   5 +
>   backends/iommufd.c                    | 101 +++++++++++-
>   hw/pci/msix.c                         |   2 +-
>   hw/pci/pci.c                          |   5 +
>   hw/vfio/ap.c                          |   2 +-
>   hw/vfio/ccw.c                         |   2 +-
>   hw/vfio/container-base.c              |   9 ++
>   hw/vfio/container.c                   |  97 +++++++++---
>   hw/vfio/cpr-iommufd.c                 | 220 ++++++++++++++++++++++++++
>   hw/vfio/cpr-legacy.c                  | 287 ++++++++++++++++++++++++++++++++++
>   hw/vfio/cpr.c                         | 159 +++++++++++++++++--
>   hw/vfio/device.c                      |  40 +++--
>   hw/vfio/helpers.c                     |  10 ++
>   hw/vfio/iommufd-stubs.c               |  18 +++
>   hw/vfio/iommufd.c                     |  81 ++++++++--
>   hw/vfio/listener.c                    |  19 ++-
>   hw/vfio/pci.c                         | 231 ++++++++++++++++++++-------
>   hw/vfio/platform.c                    |   2 +-
>   hw/vfio/vfio-stubs.c                  |  13 ++
>   migration/cpr-transfer.c              |  18 +++
>   migration/cpr.c                       |  95 +++++++++--
>   migration/migration.c                 |   1 +
>   migration/savevm.c                    |   4 +-
>   system/physmem.c                      |   5 +
>   backends/trace-events                 |   2 +
>   hw/vfio/meson.build                   |   5 +
>   41 files changed, 1482 insertions(+), 135 deletions(-)
>   create mode 100644 hw/vfio/cpr-iommufd.c
>   create mode 100644 hw/vfio/cpr-legacy.c
>   create mode 100644 hw/vfio/iommufd-stubs.c
>   create mode 100644 hw/vfio/vfio-stubs.c
> 
> base-commit: bc98ffdc7577e55ab8373c579c28fe24d600c40f


Steve,

For the next vfio PR, I plan to take patches 1-17 when patch 10 is
updated. The rest is for later in this cycle.

Thanks,

C.