Re: [PATCH V1 4/8] vfio-pci: cpr part 1 (fd and dma)

[Steven Sistare <steven.sistare@oracle.com>]

On 7/10/2024 4:03 PM, Alex Williamson wrote:
> On Tue,  9 Jul 2024 13:58:53 -0700
> Steve Sistare <steven.sistare@oracle.com> wrote:
> 
>> Enable vfio-pci devices to be saved and restored across a cpr-exec of qemu.
>>
>> At vfio creation time, save the value of vfio container, group, and device
>> descriptors in CPR state.
>>
>> In the container pre_save handler, suspend the use of virtual addresses
>> in DMA mappings with VFIO_DMA_UNMAP_FLAG_VADDR, because guest ram will
>> be remapped at a different VA after exec.  DMA to already-mapped pages
>> continues.  Save the msi message area as part of vfio-pci vmstate, and
>> save the interrupt and notifier eventfd's in vmstate.
>>
>> On qemu restart, vfio_realize() finds the saved descriptors, uses the
>> descriptors, and notes that the device is being reused.  Device and iommu
>> state is already configured, so operations in vfio_realize that would
>> modify the configuration are skipped for a reused device, including vfio
>> ioctl's and writes to PCI configuration space.  Vfio PCI device reset
>> is also suppressed. The result is that vfio_realize constructs qemu
>> data structures that reflect the current state of the device.  However,
>> the reconstruction is not complete until migrate_incoming is called.
>> migrate_incoming loads the msi data, the vfio post_load handler finds
>> eventfds in CPR state, rebuilds vector data structures, and attaches the
>> interrupts to the new KVM instance.  The container post_load handler then
>> invokes the main vfio listener callback, which walks the flattened ranges
>> of the vfio address space and calls VFIO_DMA_MAP_FLAG_VADDR to inform the
>> kernel of the new VA's.  Lastly, migration resumes the VM.
> 
> Hi Steve,
> 
> What's the iommufd plan for cpr?  Thanks,

I am working on vdpa and iommufd as we speak, with working prototypes for both.
I plan to submit the kernel and qemu RFC for vdpa next week, followed by vacation,
and iommfd in the weeks after that.

- Steve