Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports

[Jason Gunthorpe <jgg@mellanox.com>]

On Mon, Mar 04, 2019 at 05:03:20PM -0800, Jakub Kicinski wrote:

> > Don't we already have devlink instances for every mlx5 physical port
> > and VF as they are unique PCI functions?
> 
> That's a very NIC-centric view of the world, though.  Equating devlink
> instances to ports, and further to PCI devices.  Its fundamentally
> different from what switches and some NICs do, where all ports are under
> single devlink instance.

I think, as a practical matter, it is a bit hard to recombine an asic
that presents multiple PCI BDFs into a single SW object. It is tricky
to give stable labels to things, to leave gaps to allow for uncertain
discovey, to co-ordinate between multiple struct pci_device drivers
probe functions, etc.

And at least with devlink, if you have a object layer that is broader
then PCI BDF, how do the devlink commands work? Are all BDFs just an
alias for this hidden super object?

Do any drivers attempt to provide single instant made up of merged
BDFs?

In other words, is a PCI BDF really the largest granularity that
devlink can address today?

At least in RDMA we have drivers doing all combinations of this:
multiple ports per BDF, one port per BDF, and one composite RDMA
device formed by combining multiple BDFs worth of ports together.

> > > You guys come from the RDMA side of the world, with which I'm less
> > > familiar, and the soft bus + spawning devices seems to be a popular
> > > design there.  Could you describe the advantages of that model for 
> > > the sake of the netdev-only folks? :)  
> > 
> > I don't think we do this in RDMA at all yet, or maybe I'm not sure
> > what you are thinking of?
> 
> Mm.. I caught an Intel patch set recently which was talking about buses
> and spawning devices.  It must have been a different kettle of fish.

That sounds like scalable iov..

Jason