Re: [question] VFIO Device Migration: The vCPU may be paused during vfio device DMA in iommu nested stage mode && vSVA

[Kirti Wankhede <kwankhede@nvidia.com>]

On 9/24/2021 12:17 PM, Tian, Kevin wrote:
>> From: Kunkun Jiang <jiangkunkun@huawei.com>
>> Sent: Friday, September 24, 2021 2:19 PM
>>
>> Hi all,
>>
>> I encountered a problem in vfio device migration test. The
>> vCPU may be paused during vfio-pci DMA in iommu nested
>> stage mode && vSVA. This may lead to migration fail and
>> other problems related to device hardware and driver
>> implementation.
>>
>> It may be a bit early to discuss this issue, after all, the iommu
>> nested stage mode and vSVA are not yet mature. But judging
>> from the current implementation, we will definitely encounter
>> this problem in the future.
> 
> Yes, this is a known limitation to support migration with vSVA.
> 
>>
>> This is the current process of vSVA processing translation fault
>> in iommu nested stage mode (take SMMU as an example):
>>
>> guest os            4.handle translation fault 5.send CMD_RESUME to vSMMU
>>
>>
>> qemu                3.inject fault into guest os 6.deliver response to
>> host os
>> (vfio/vsmmu)
>>
>>
>> host os              2.notify the qemu 7.send CMD_RESUME to SMMU
>> (vfio/smmu)
>>
>>
>> SMMU              1.address translation fault              8.retry or
>> terminate
>>
>> The order is 1--->8.
>>
>> Currently, qemu may pause vCPU at any step. It is possible to
>> pause vCPU at step 1-5, that is, in a DMA. This may lead to
>> migration fail and other problems related to device hardware
>> and driver implementation. For example, the device status
>> cannot be changed from RUNNING && SAVING to SAVING,
>> because the device DMA is not over.
>>
>> As far as i can see, vCPU should not be paused during a device
>> IO process, such as DMA. However, currently live migration
>> does not pay attention to the state of vfio device when pausing
>> the vCPU. And if the vCPU is not paused, the vfio device is
>> always running. This looks like a *deadlock*.
> 
> Basically this requires:
> 
> 1) stopping vCPU after stopping device (could selectively enable
> this sequence for vSVA);
> 

I don't think this is change is required. When vCPUs are at halt vCPU 
states are already saved, step 4 or 5 will be taken care by that. Then 
when device is transitioned in SAVING state, save qemu and host os state 
in the migration stream, i.e. state at step 2 and 3, depending on that 
take action while resuming, about step 6 or 7 to run.

Thanks,
Kirti

> 2) when stopping device, the driver should block new requests
> from vCPU (queued to a pending list) and then drain all in-fly
> requests including faults;
>      * to block this further requires switching from fast-path to
> slow trap-emulation path for the cmd portal before stopping
> the device;
> 
> 3) save the pending requests in the vm image and replay them
> after the vm is resumed;
>      * finally disable blocking by switching back to the fast-path for
> the cmd portal;
> 
>>
>> Do you have any ideas to solve this problem?
>> Looking forward to your replay.
>>
> 
> We verified above flow can work in our internal POC.
> 
> Thanks
> Kevin
>