Re: [RFC PATCH] vfio: Update/Clarify migration uAPI, add NDMA state

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Jan 10, 2022 at 03:14:44AM +0000, Tian, Kevin wrote:
> > An operator might need to emergency migrate a VM without the
> > possibility for failure. For instance there is something wrong with
> > the base HW. SLA ignored, migration must be done.
> 
> How is it done today when no assigned device supports migration?

That is different, the operator deliberately created a VM that is not
migratable. Operators may simply prefer to never do this.

You are talking about migration which is blockable by the guest -
outside of operator controll this is a totally different thing.

> - It's necessary to support existing HW though it may only supports
>   optional migration due to unbounded time of stopping DMA;

At a minimum a device with optional migration needs to be reported to
userspace and qemu should not blindly adopt it without some opt-in
IMHO.
 
> - We should influence IP designers to design HW to allow preempting
>   in-fly requests and stop DMA quickly (also implying the capability of
>   aborting/resuming in-fly PRI requests);

Yes, I think we need a way to suspend the device then abort its PRIs
with some error. The ATS cache is not something that is migrated so
this seems reasonable.

The only sketchy bit looks like how to resync the VM that still would
have a PRI in its queue and would still want to answer it. That answer
would have to be discarded..

> - Specific to the device state management uAPI, it should not assume
>   a specific usage and instead allow the user to set a timeout value so
>   transitioning to NDMA is failed if the operation cannot be completed
>   within the specified timeout value. If the user doesn't set it, the 
>   migration driver could conservatively use a default timeout value to
>   gate any potentially unbounded operation.

This would need to go along with the flag above, as only optional
drivers should have something like this.

Jason