Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

Alex Williamson
2019-11-08T20:34:45Z

On Fri, 8 Nov 2019 16:12:53 -0400
Jason Gunthorpe <jgg@ziepe.ca> wrote:

> On Fri, Nov 08, 2019 at 11:12:38AM -0800, Jakub Kicinski wrote:
> > On Fri, 8 Nov 2019 15:40:22 +0000, Parav Pandit wrote:
> > > > The new intel driver has been having a very similar discussion about how to
> > > > model their 'multi function device' ie to bind RDMA and other drivers to a
> > > > shared PCI function, and I think that discussion settled on adding a new bus?
> > > >
> > > > Really these things are all very similar, it would be nice to have a clear
> > > > methodology on how to use the device core if a single PCI device is split by
> > > > software into multiple different functional units and attached to different
> > > > driver instances.
> > > >
> > > > Currently there is alot of hacking in this area.. And a consistent scheme
> > > > might resolve the ugliness with the dma_ops wrappers.
> > > >
> > > > We already have the 'mfd' stuff to support splitting platform devices, maybe
> > > > we need to create a 'pci-mfd' to support splitting PCI devices?
> > > >
> > > > I'm not really clear how mfd and mdev relate, I always thought mdev was
> > > > strongly linked to vfio.
> > > >
> > >
> > > Mdev at beginning was strongly linked to vfio, but as I mentioned
> > > above it is addressing more use case.
> > >
> > > I observed that discussion, but was not sure of extending mdev further.
> > >
> > > One way to do for Intel drivers to do is after series [9].
> > > Where PCI driver says, MDEV_CLASS_ID_I40_FOO
> > > RDMA driver mdev_register_driver(), matches on it and does the probe().
> >
> > Yup, FWIW to me the benefit of reusing mdevs for the Intel case vs
> > muddying the purpose of mdevs is not a clear trade off.
>
> IMHO, mdev has amdev_parent_ops structure clearly intended to link it
> to vfio, so using a mdev for something not related to vfio seems like
> a poor choice.

Unless there's some opposition, I'm intended to queue this for v5.5:

https://www.spinics.net/lists/kvm/msg199613.html

mdev has started out as tied to vfio, but at it's core, it's just a
device life cycle infrastructure with callbacks between bus drivers
and vendor devices. If virtio is on the wrong path with the above
series, please speak up. Thanks,

Alex

> I suppose this series is the start and we will eventually see the
> mlx5's mdev_parent_ops filled in to support vfio - but *right now*
> this looks identical to the problem most of the RDMA capable net
> drivers have splitting into a 'core' and a 'function'
>
> > IMHO MFD should be of more natural use for Intel, since it's about
> > providing different functionality rather than virtual slices of the
> > same device.
>
> I don't think the 'different functionality' should matter much.
>
> Generally these multi-function drivers are build some some common
> 'core' language like queues interrupts, BAR space, etc and then these
> common things can be specialized into netdev, rdma, scsi, etc. So we
> see a general rough design with a core layer managing the raw HW then
> drivers on top of that (including netdev) using that API.
>
> The actual layering doesn't come through in the driver model,
> generally people put all the core stuff in with the netdev and then
> try and shuffle the netdev around as the 'handle' for that core API.
>
> These SFs are pretty similar in that the core physical driver
> continues to provide some software API support to the SF children (at
> least for mlx it is a small API)
>
> For instance mdev has no generic way to learn the BAR struct
> resources, so there is some extra API around the side that does this -
> in this series it is done by hackily co-opting the drvdata to
> something owned by the struct device instead of the device_driver and
> using that to access the API surface on 'struct mlx5_sf *', which
> includes the BAR info and so forth.
>
> This is probably the main difference from MFD. At least the few
> drivers I looked at, did not try and expose an SW API from the 'core'
> to the 'part', everything was usual generic driver resource stuff.
>
> Jason