Re: [PATCH v8 01/16] s390x/pci: implement IOMMU replay

[Matthew Rosato <mjrosato@linux.ibm.com>]

>> +                    pbdev->state = ZPCI_FS_ERROR;
>> +                    s390_pci_generate_error_event(ERR_EVENT_PERMERR,
>> pbdev->fh,
>> +                                                  pbdev->fid, 0, 0);
> 
> A permanent error will indicate to the guest that PCI device in
> unusable, in that case would the guest drive a device re-initialization?
> Would it be better to generate a ERR_EVENT_SERVAC to allow the guest to
> attempt recovery and so drive the mappings again?
> 
The problem is that if we hit this scenario, we've exhausted the vfio
DMA limit for the device, so there is no way to create more mappings
without freeing some other ones up.
In the case of a RPCIT, we can ask the guest to try and free up all
stale/invalidated mappings to make room, but in this case
1) we have no mechanism to ask the guest to do that
2) we are in the middle replaying only the valid mappings, so there are
no invalid mappings to flush anyway -- so we're really in a permanent
error case here -- the number of vfio-allowed concurrent DMA mappings is
less than what our IOMMU wishes to replay.  AFAICT this can only get
solved by forcing the guest to throw everything out and start over.

I think in reality, you would only hit this in a migration scenario with
a vfio-pci device (which we don't support with this series) where the
vfio DMA limit is lower on the target than it was on the host.
If you hit this doing IOMMU replay of a vfio-pci device without a
migration involved then the limit should be the same as it was before
replay and it should have been impossible to map more than the vfio DMA
limit (so if you managed it, a permanent error also seems reasonable
because there may be a deeper issue).

Would an 0x3a trigger a total wipe of the IOMMU contents that we were
trying to replay?  If yes, it might be something to consider if/when we
support vfio-pci migration but I'm not sure it makes sense under the
current scenario(s) where we should never hit this path.

Thanks,
Matt