Re: [PATCH vfio 10/11] vfio/virtio: Expose admin commands over virtio device

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Oct 11, 2023 at 07:17:25AM -0700, Christoph Hellwig wrote:
> On Wed, Oct 11, 2023 at 10:57:09AM -0300, Jason Gunthorpe wrote:
> > > Independent of my above points on the doubts on VF-controlled live
> > > migration for PCe device I absolutely agree with your that the Linux
> > > abstraction and user interface should be VF based.  Which further
> > > reinforeces my point that the VFIO driver for the controlled function
> > > (PF or VF) and the Linux driver for the controlling function (better
> > > be a PF in practice) must be very tightly integrated.  And the best
> > > way to do that is to export the vfio nodes from the Linux driver
> > > that knowns the hardware and not split out into a separate one.
> > 
> > I'm not sure how we get to "very tightly integrated". We have many
> > examples of live migration vfio drivers now and they do not seem to
> > require tight integration. The PF driver only has to provide a way to
> > execute a small number of proxied operations.
> 
> Yes.  And for that I need to know what VF it actually is dealing
> with.  Which is tight integration in my book.

Well, I see two modalities here

Simple devices with a fixed PF/VF relationship use a VF pci_device for
VFIO and pci_iov_get_pf_drvdata()/related APIs to assemble their
parts. This is very limited (and kind of hacky).

Complex devices can use an auxiliary_device for VFIO and assemble
their parts however they like.

After probe is done the VFIO code operates effectively identically
regardless of how the components were found.

Intel is going to submit their IDXD SIOV driver "soon" and I'd like to
pause there and have a real discussion about how to manage VFIO
lifecycle and dynamic "function" creation in this brave new world.

Ideally we can get a lifecycle API that works uniformly for PCI VFs
too. Then maybe this gets more resolved.

In my mind at least, definately no mdevs and that sysfs GUID junk. :(

> I really don't care about where the code lives (in the directory tree)
> either.  But as you see with virtio trying to split it out into
> an arbitrary module causes all kinds of pain.

Trying to put VFIO-only code in virtio is what causes all the
issues. If you mis-design the API boundary everything will be painful,
no matter where you put the code.

Jason