Re: [RFC v2] VFIO Migration

[Cornelia Huck <cohuck@redhat.com>]

On Thu, 5 Nov 2020 15:09:02 +0000
Stefan Hajnoczi <stefanha@redhat.com> wrote:

(...)

<did not fully read through the v1 thread, so apologies if I missed
something>

> VFIO/mdev Devices
> -----------------
> TODO this is a first draft, more thought needed around enumerating supported
> parameters, representing default values, etc
> 
> The following mdev type sysfs attrs are available for managing device
> instances:
> 
>   /sys/.../<parent-device>/mdev_supported_types/<type-id>/
>       create - writing a UUID to this file instantiates a device
>       migration/ - migration related files
>           model - unique device model string, e.g. vendor-a.com/my-nic
> 
> Device models supported by an mdev driver can be enumerated by reading the
> migration/model attr for each <type-id>.

IIUC, we're grouping together all users of a specific mdev_type, but
support a variety of sub-configurations? Does that include parameters
or not? If not, shouldn't we already be covered by mdev_type?

> 
> The following mdev device sysfs attrs relate to a specific device instance:
> 
>   /sys/.../<parent-device>/<uuid>/
>       mdev_type/ - symlink to mdev type sysfs attrs, e.g. to fetch migration/model
>       migration/ - migration related files
>           applied - Write "1" to apply current migration parameter values or
>                     "0" to reset migration parameter values to their defaults.
>                     Parameters can only be applied or reset while the mdev is
>                     not opened.
>           params/ - migration parameters
>               <my-param> - read/write migration parameter "my-param"
>               ...
> 
> When the device is created the migration/applied attr is "0". Migration
> parameters are accessible in migration/params/ and read 0 bytes because they
> are at their default values.  At the point opening the mdev device will fail
> because migration parameters must be applied first. Migration parameters can be
> set to the desired values or left at their defaults. "1" must be written to
> migration/applied before opening the mdev device.
> 
> If writing to a migration/params/<param> attr or setting migration/applied to
> "1" fails, then the device implementation does not support the migration
> parameters.
> 
> An open mdev device typically does not allow migration parameters to be changed
> at runtime. However, certain migration/params attrs may allow writes at
> runtime. Usually these migration parameters only affect the device state
> representation and not the hardware interface. This makes it possible to
> upgrade or downgrade the device state representation at runtime so that
> migration is possible to newer or older device implementations.
> 
> An existing mdev device instance can be reused by closing the mdev device and
> writing "0" to migration/applied. This resets parameters to their defaults so
> that a new list of migration parameters can be applied.
> 
> The migration parameter list for an mdev device that is in operation can be
> read from migration/params/. Parameters that read 0 bytes are at their default
> value.

I'm trying to figure out what that means for the mdevs I'm most
familiar with, ccw and ap. Both of them currently support a single
mdev_type.

For ccw, there are some things that I could imagine as parameters, like
the device number, or channel paths. Maybe we could include the channel
path type (FICON, ...) in the migration device model? We should not
include device numbers etc. in the device model.

For ap, we have matrices covering tuples (APQNs) derived from a
cross-product of card/domains configure via sysfs attributes. I think
later modification of these is desired. I think we also might be able
to mix-and-match different types within the same matrix, so not sure if
we can put these into any device model. In fact, I'm a bit at a loss
how the device model for ap would look like (other than simply
'matrix'). Can we deal with dynamic parameters?

(...)