[PATCH RFC v2] vfio: Documentation for the migration region

Jason Gunthorpe
2021-11-29T14:47:56Z

Provide some more complete documentation for the migration regions
behavior, specifically focusing on the device_state bits and the whole
system view from a VMM.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
Documentation/driver-api/vfio.rst | 277 +++++++++++++++++++++++++++++-
1 file changed, 276 insertions(+), 1 deletion(-)

Alex/Cornelia, here is the second draft of the requested documentation I promised

We think it includes all the feedback from hns, Intel and NVIDIA on this mechanism.

Our thinking is that NDMA would be implemented like this:

+#define VFIO_DEVICE_STATE_NDMA (1 << 3)

And a .add_capability ops will be used to signal to userspace driver support:

+#define VFIO_REGION_INFO_CAP_MIGRATION_NDMA 6

I've described DIRTY TRACKING as a seperate concept here. With the current
uAPI this would be controlled by VFIO_IOMMU_DIRTY_PAGES_FLAG_START, with our
change in direction this would be per-tracker control, but no semantic change.

Upon some agreement we'll include this patch in the next iteration of the mlx5
driver along with the NDMA bits.

v2:
- RST fixups for sphinx rendering
- Inclue the priority order for multi-bit-changes
- Add a small discussion on devices like hns with migration control inside
the same function as is being migrated.
- Language cleanups from v1, the diff says almost every line was touched in some way
v1: https://lore.kernel.org/r/0-v1-0ec87874bede+123-vfio_mig_doc_jgg@nvidia.com

diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
index c663b6f978255b..d9be47570f878c 100644
--- a/Documentation/driver-api/vfio.rst
+++ b/Documentation/driver-api/vfio.rst
@@ -242,7 +242,282 @@ group and can access them as follows::
VFIO User API
-------------------------------------------------------------------------------

-Please see include/linux/vfio.h for complete API documentation.
+Please see include/uapi/linux/vfio.h for complete API documentation.
+
+-------------------------------------------------------------------------------
+
+VFIO migration driver API
+-------------------------------------------------------------------------------
+
+VFIO drivers that support migration implement a migration control register
+called device_state in the struct vfio_device_migration_info which is in its
+VFIO_REGION_TYPE_MIGRATION region.
+
+The device_state controls both device action and continuous behavior.
+Setting/clearing bit groups triggers device action, and each bit controls a
+continuous device behavior.
+
+Along with the device_state the migration driver provides a data window which
+allows streaming migration data into or out of the device.
+
+A lot of flexibility is provided to user-space in how it operates these
+bits. What follows is a reference flow for saving device state in a live
+migration, with all features, and an illustration how other external non-VFIO
+entities (VCPU_RUNNING and DIRTY_TRACKING) the VMM controls fit in.
+
+ RUNNING, VCPU_RUNNING
+ Normal operating state
+ RUNNING, DIRTY_TRACKING, VCPU_RUNNING
+ Log DMAs
+
+ Stream all memory
+ SAVING | RUNNING, DIRTY_TRACKING, VCPU_RUNNING
+ Log internal device changes (pre-copy)
+
+ Stream device state through the migration window
+
+ While in this state repeat as desired:
+
+ Atomic Read and Clear DMA Dirty log
+
+ Stream dirty memory
+ SAVING | NDMA | RUNNING, VCPU_RUNNING
+ vIOMMU grace state
+
+ Complete all in progress IO page faults, idle the vIOMMU
+ SAVING | NDMA | RUNNING
+ Peer to Peer DMA grace state
+
+ Final snapshot of DMA dirty log (atomic not required)
+ SAVING
+ Stream final device state through the migration window
+
+ Copy final dirty data
+ 0
+ Device is halted
+
+and the reference flow for resuming:
+
+ RUNNING
+ Issue VFIO_DEVICE_RESET to clear the internal device state
+ 0
+ Device is halted
+ RESUMING
+ Push in migration data. Data captured during pre-copy should be
+ prepended to data captured during SAVING.
+ NDMA | RUNNING
+ Peer to Peer DMA grace state
+ RUNNING, VCPU_RUNNING
+ Normal operating state
+
+If the VMM has multiple VFIO devices undergoing migration then the grace
+states act as cross device synchronization points. The VMM must bring all
+devices to the grace state before advancing past it.
+
+The above reference flows are built around specific requirements on the
+migration driver for its implementation of the migration_state input.
+
+The migration_state cannot change asynchronously, upon writing the
+migration_state the driver will either keep the current state and return
+failure, return failure and go to ERROR, or succeed and go to the new state.
+
+Event triggered actions happen when user-space requests a new migration_state
+that differs from the current migration_state. Actions happen on a bit group
+basis:
+
+ SAVING | RUNNING
+ The device clears the data window and begins streaming 'pre copy' migration
+ data through the window. Devices that cannot log internal state changes
+ return a 0 length migration stream.
+
+ SAVING | !RUNNING
+ The device captures its internal state that is not covered by internal
+ logging, as well as any logged changes.
+
+ The device clears the data window and begins streaming the captured
+ migration data through the window. Devices that cannot log internal state
+ changes stream all their device state here.
+
+ RESUMING
+ The data window is cleared, opened, and can receive the migration data
+ stream.
+
+ !RESUMING
+ All the data transferred into the data window is loaded into the device's
+ internal state. The migration driver can rely on user-space issuing a
+ VFIO_DEVICE_RESET prior to starting RESUMING.
+
+ To abort a RESUMING issue a VFIO_DEVICE_RESET.
+
+ If the migration data is invalid then the ERROR state must be set.
+
+Continuous actions are in effect when migration_state bit groups are active:
+
+ RUNNING | NDMA
+ The device is not allowed to issue new DMA operations.
+
+ Whenever the kernel returns with a migration_state of NDMA there can be no
+ in progress DMAs.
+
+ !RUNNING
+ The device should not change its internal state. Further implies the NDMA
+ behavior above.
+
+ SAVING | !RUNNING
+ RESUMING | !RUNNING
+ The device may assume there are no incoming MMIO operations.
+
+ Internal state logging can stop.
+
+ RUNNING
+ The device can alter its internal state and must respond to incoming MMIO.
+
+ SAVING | RUNNING
+ The device is logging changes to the internal state.
+
+ ERROR
+ The behavior of the device is largely undefined. The device must be
+ recovered by issuing VFIO_DEVICE_RESET or closing the device file
+ descriptor.
+
+ However, devices supporting NDMA must behave as though NDMA is asserted
+ during ERROR to avoid corrupting other devices or a VM during a failed
+ migration.
+
+When multiple bits change in the migration_state they may describe multiple
+event triggered actions, and multiple changes to continuous actions. The
+migration driver must process them in a priority order:
+
+ - SAVING | RUNNING
+ - NDMA
+ - !RUNNING
+ - SAVING | !RUNNING
+ - RESUMING
+ - !RESUMING
+ - RUNNING
+ - !NDMA
+
+In general, userspace can issue a VFIO_DEVICE_RESET ioctl and recover the
+device back to device_state RUNNING. When a migration driver executes this
+ioctl it should discard the data window and set migration_state to RUNNING as
+part of resetting the device to a clean state. This must happen even if the
+migration_state has errored. A freshly opened device FD should always be in
+the RUNNING state.
+
+The migration driver has limitations on what device state it can affect. Any
+device state controlled by general kernel subsystems must not be changed
+during RESUME, and SAVING must tolerate mutation of this state. Change to
+externally controlled device state can happen at any time, asynchronously, to
+the migration (ie interrupt rebalancing).
+
+Some examples of externally controlled state:
+ - MSI-X interrupt page
+ - MSI/legacy interrupt configuration
+ - Large parts of the PCI configuration space, ie common control bits
+ - PCI power management
+ - Changes via VFIO_DEVICE_SET_IRQS
+
+During !RUNNING, especially during SAVING and RESUMING, the device may have
+limitations on what it can tolerate. An ideal device will discard/return all
+ones to all incoming MMIO, PIO, or equivalent operations (exclusive of the
+external state above) in !RUNNING. However, devices are free to have undefined
+behavior if they receive incoming operations. This includes
+corrupting/aborting the migration, dirtying pages, and segfaulting user-space.
+
+However, a device may not compromise system integrity if it is subjected to a
+MMIO. It cannot trigger an error TLP, it cannot trigger an x86 Machine Check
+or similar, and it cannot compromise device isolation.
+
+There are several edge cases that user-space should keep in mind when
+implementing migration:
+
+- Device Peer to Peer DMA. In this case devices are able issue DMAs to each
+ other's MMIO regions. The VMM can permit this if it maps the MMIO memory into
+ the IOMMU.
+
+ As Peer to Peer DMA is a MMIO touch like any other, it is important that
+ userspace suspend these accesses before entering any device_state where MMIO
+ is not permitted, such as !RUNNING. This can be accomplished with the NDMA
+ state. Userspace may also choose to remove MMIO mappings from the IOMMU if the
+ device does not support NDMA and rely on that to guarantee quiet MMIO.
+
+ The Peer to Peer Grace States exist so that all devices may reach RUNNING
+ before any device is subjected to a MMIO access.
+
+ Failure to guarantee quiet MMIO may allow a hostile VM to use P2P to violate
+ the no-MMIO restriction during SAVING or RESUMING and corrupt the migration
+ on devices that cannot protect themselves.
+
+- IOMMU Page faults handled in user-space can occur at any time. A migration
+ driver is not required to serialize in-progress page faults. It can assume
+ that all page faults are completed before entering SAVING | !RUNNING. Since
+ the guest VCPU is required to complete page faults the VMM can accomplish
+ this by asserting NDMA | VCPU_RUNNING and clearing all pending page faults
+ before clearing VCPU_RUNNING.
+
+ Device that do not support NDMA cannot be configured to generate page faults
+ that require the VCPU to complete.
+
+- pre-copy allows the device to implement a dirty log for its internal state.
+ During the SAVING | RUNNING state the data window should present the device
+ state being logged and during SAVING | !RUNNING the data window should present
+ the unlogged device state as well as the changes from the internal dirty log.
+
+ On RESUME these two data streams are concatenated together.
+
+ pre-copy is only concerned with internal device state. External DMAs are
+ covered by the separate DIRTY_TRACKING function.
+
+- Atomic Read and Clear of the DMA log is a HW feature. If the tracker
+ cannot support this, then NDMA could be used to synthesize it less
+ efficiently.
+
+- NDMA is optional. If the device does not support this then the NDMA States
+ are pushed down to the next step in the sequence and various behaviors that
+ rely on NDMA cannot be used.
+
+- Migration control registers inside the same iommu_group as the VFIO device.
+ This immediately raises a security concern as user-space can use Peer to Peer
+ DMA to manipulate these migration control registers concurrently with
+ any kernel actions.
+
+ A device driver operating such a device must ensure that kernel integrity
+ cannot be broken by hostile user space operating the migration MMIO
+ registers via peer to peer, at any point in the sequence. Notably the kernel
+ cannot use DMA to transfer any migration data.
+
+ However, as discussed above in the "Device Peer to Peer DMA" section, it can
+ assume quiet MMIO as a condition to have a successful and uncorrupted
+ migration.
+
+To elaborate details on the reference flows, they assume the following details
+about the external behaviors:
+
+ !VCPU_RUNNING
+ User-space must not generate dirty pages or issue MMIO, PIO or equivalent
+ operations to devices. For a VMM this would typically be controlled by
+ KVM.
+
+ DIRTY_TRACKING
+ Clear the DMA log and start DMA logging
+
+ DMA logs should be readable with an "atomic test and clear" to allow
+ continuous non-disruptive sampling of the log.
+
+ This is controlled by VFIO_IOMMU_DIRTY_PAGES_FLAG_START on the container
+ fd.
+
+ !DIRTY_TRACKING
+ Freeze the DMA log, stop tracking and allow user-space to read it.
+
+ If user-space is going to have any use of the dirty log it must ensure that
+ all DMA is suspended before clearing DIRTY_TRACKING, for instance by using
+ NDMA or !RUNNING on all VFIO devices.
+
+
+TDB - discoverable feature flag for NDMA
+TDB IMS xlation
+TBD PASID xlation

VFIO bus driver API
-------------------------------------------------------------------------------

base-commit: ae0351a976d1880cf152de2bc680f1dff14d9049
--
2.34.0