Re: [PATCH v7 17/22] vfio-pci/zdev: add open/close device hooks

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, May 16, 2022 at 05:59:01PM -0400, Matthew Rosato wrote:
> On 5/16/22 2:35 PM, Jason Gunthorpe wrote:
> > On Mon, May 16, 2022 at 02:30:46PM -0400, Matthew Rosato wrote:
> > 
> > > Conceptually I think this would work for QEMU anyway (it always sets the kvm
> > > before we open the device).  I tried to test the idea quickly but couldn't
> > > get the following to apply on vfio-next or your vfio_group_locking -- but I
> > > understand what you're trying to do so I'll re-work and try it out.
> > 
> > I created it on 8c9350e9bf43de1ebab3cc8a80703671e6495ab4 which is the
> > vfio_group_locking.. I can send you a github if it helps
> > https://github.com/jgunthorpe/linux/commits/vfio_group_lockin
> > 
> Thanks -- I was able to successfully test your proposed idea (+ some changes
> to make it compile :)) on top of vfio_group_locking along with a modified
> version of my zdev series.  I also tried it out with vfio-ap successfully,
> but have nothing to test GVT with.
> 
> That said, this has caused me to realize that 'iommu:
> iommu_group_claim_dma_owner() must always assign a domain' breaks s390x
> vfio-pci :( I wonder if it is due to the way s390x PCI currently switches
> between dma ops and iommu ops.  It looks like it breaks vfio-ap mdevs too,
> but I know less about that --  I will have to investigate both more
> tomorrow.

At the very least it appears that s390 thinks that attach/detach_dev
are strictly paired and that isn't how the iommu ops are defined.

attach_dev can be called multiple times without any detach_dev.

Possibly the unbalance of zpci_register_ioat / zpci_unregister_ioat
causes the trouble?

Jason