Re: [RFC PATCH v3 net-next] Documentation: devlink: Add devlink-sd

[Jiri Pirko <jiri@resnulli.us>]

Thu, Feb 01, 2024 at 12:17:26AM CET, kuba@kernel.org wrote:
>On Wed, 31 Jan 2024 15:02:58 -0800 William Tu wrote:
>> > I just did a grep on METADATA_HW_PORT_MUX and assumed bnxt, ice and nfp
>> > all do buffer sharing. You're saying you mux Tx queues but not Rx
>> > queues? Or I need to actually read the code instead of grepping? :)
>> 
>> I guess bnxt, ice, nfp are doing tx buffer sharing?
>
>I'm not familiar with ice. I'm 90% sure bnxt shares both Rx and Tx.
>I'm 99.9% sure nfp does.

Wait a sec. You refer to using the lower device (like PF) to actually
send and receive trafic of representors. That means, you share the
entire queues. Or maybe better term is not "share" but "use PF queues".

The infra William is proposing is about something else. In that
scenario, each representor has a separate independent set of queues,
as well as the PF has. Currently in mlx5, all representor queues have
descriptors only used for the individual representor. So there is
a huge waste of memory for that, as often there is only very low traffic
there and probability of hitting trafic burst on many representors at
the same time is very low.

Say you have 1 queue for a rep. 1 queue has 1k descriptors. For 1k
representors you end up with:
1k x 1k = 1m descriptors

With this API, user can configure sharing of the descriptors.
So there would be a pool (or multiple pools) of descriptors and the
descriptors could be used by many queues/representors.

So in the example above, for 1k representors you have only 1k
descriptors.

The infra allows great flexibility in terms of configuring multiple
pools of different sizes and assigning queues from representors to
different pools. So you can have multiple "classes" of representors.
For example the ones you expect heavy trafic could have a separate pool,
the rest can share another pool together, etc.


>
>It'd be great if you could do the due diligence rather than guessing
>given that you're proposing uAPI extension :(
>
>> This devlink sd is for RX queues not TX queues.
>> 
>> And devlink-sd creates a pool of shared descriptors only for RX queue.
>> 
>> The TX queues/ TX path remain unchanged.
>