Re: [RFC PATCH v3 net-next] Documentation: devlink: Add devlink-sd

[Jacob Keller <jacob.e.keller@intel.com>]

On 2/15/2024 5:19 AM, Jiri Pirko wrote:
> Fri, Feb 09, 2024 at 02:26:33AM CET, kuba@kernel.org wrote:
>> On Fri, 2 Feb 2024 08:46:56 +0100 Jiri Pirko wrote:
>>> Fri, Feb 02, 2024 at 05:00:41AM CET, kuba@kernel.org wrote:
>>>> On Thu, 1 Feb 2024 11:13:57 +0100 Jiri Pirko wrote:  
>>>>> Wait a sec.  
>>>>
>>>> No, you wait a sec ;) Why do you think this belongs to devlink?
>>>> Two months ago you were complaining bitterly when people were
>>>> considering using devlink rate to control per-queue shapers.
>>>> And now it's fine to add queues as a concept to devlink?  
>>>
>>> Do you have a better suggestion how to model common pool object for
>>> multiple netdevices? This is the reason why devlink was introduced to
>>> provide a platform for common/shared things for a device that contains
>>> multiple netdevs/ports/whatever. But I may be missing something here,
>>> for sure.
>>
>> devlink just seems like the lowest common denominator, but the moment
>> we start talking about multi-PF devices it also gets wobbly :(
> 
> You mean you see real to have a multi-PF device that allows to share the
> pools between the PFs? If, in theory, that exists, could this just be a
> limitation perhaps?
> 
> 

I don't know offhand if we have a device which can share pools
specifically, but we do have multi-PF devices which have a lot of shared
resources. However, due to the multi-PF PCIe design. I looked into ways
to get a single devlink across the devices.. but ultimately got stymied
and gave up.

This left us with accepting the limitation that each PF gets its own
devlink and can't really communicate with other PFs.

The existing solution has just been to partition the shared resources
evenly across PFs, typically via firmware. No flexibility.

I do think the best solution here would be to figure out a generic way
to tie multiple functions into a single devlink representing the device.
Then each function gets the set of devlink_port objects associated to
it. I'm not entirely sure how that would work. We could hack something
together with auxbus.. but thats pretty ugly. Some sort of orchestration
in the PCI layer that could identify when a device wants to have some
sort of "parent" driver which loads once and has ties to each of the
function drivers would be ideal.

Then this parent driver could register devlink, and each function driver
could connect to it and allocate ports and function-specific resources.

Alternatively a design which loads a single driver that maintains
references to each function could work but that requires a significant
change to the entire driver design and is unlikely to be done for
existing drivers...

This is obviously a bit more of a tangential problem to this series.