Re: [net-next 10/15] net/mlx5e: Let channels be SD-aware

[Tariq Toukan <ttoukan.linux@gmail.com>]

On 10/01/2024 16:09, Gal Pressman wrote:
> On 09/01/2024 18:00, Jakub Kicinski wrote:
>> On Tue, 9 Jan 2024 16:15:50 +0200 Gal Pressman wrote:
>>>>> I'm confused, how are RX queues related to XPS?
>>>>
>>>> Separate sentence, perhaps I should be more verbose..
>>>
>>> Sorry, yes, your understanding is correct.
>>> If a packet is received on RQ 0 then it is from PF 0, RQ 1 came from PF
>>> 1, etc. Though this is all from the same wire/port.
>>>
>>> You can enable arfs for example, which will make sure that packets that
>>> are destined to a certain CPU will be received by the PF that is closer
>>> to it.
>>
>> Got it.
>>
>>>>> XPS shouldn't be affected, we just make sure that whatever queue XPS
>>>>> chose will go out through the "right" PF.
>>>>
>>>> But you said "correct" to queue 0 going to PF 0 and queue 1 to PF 1.
>>>> The queue IDs in my question refer to the queue mapping form the stacks
>>>> perspective. If user wants to send everything to queue 0 will it use
>>>> both PFs?
>>>
>>> If all traffic is transmitted through queue 0, it will go out from PF 0
>>> (the PF that is closer to CPU 0 numa).
>>

Hi,
I'm back from a long vacation. Catching up on emails...

>> Okay, but earlier you said: "whatever queue XPS chose will go out
>> through the "right" PF." - which I read as PF will be chosen based
>> on CPU locality regardless of XPS logic.
>>
>> If queue 0 => PF 0, then user has to set up XPS to make CPUs from NUMA
>> node which has PF 0 use even number queues, and PF 1 to use odd number
>> queues. Correct?

Exactly. That's the desired configuration.
Our driver has the logic to set it in default.

Here's the default XPS on my setup:

NUMA:
   NUMA node(s):          2
   NUMA node0 CPU(s):     0-11
   NUMA node1 CPU(s):     12-23

PF0 on node0, PF1 on node1.

/sys/class/net/eth2/queues/tx-0/xps_cpus:000001
/sys/class/net/eth2/queues/tx-1/xps_cpus:001000
/sys/class/net/eth2/queues/tx-2/xps_cpus:000002
/sys/class/net/eth2/queues/tx-3/xps_cpus:002000
/sys/class/net/eth2/queues/tx-4/xps_cpus:000004
/sys/class/net/eth2/queues/tx-5/xps_cpus:004000
/sys/class/net/eth2/queues/tx-6/xps_cpus:000008
/sys/class/net/eth2/queues/tx-7/xps_cpus:008000
/sys/class/net/eth2/queues/tx-8/xps_cpus:000010
/sys/class/net/eth2/queues/tx-9/xps_cpus:010000
/sys/class/net/eth2/queues/tx-10/xps_cpus:000020
/sys/class/net/eth2/queues/tx-11/xps_cpus:020000
/sys/class/net/eth2/queues/tx-12/xps_cpus:000040
/sys/class/net/eth2/queues/tx-13/xps_cpus:040000
/sys/class/net/eth2/queues/tx-14/xps_cpus:000080
/sys/class/net/eth2/queues/tx-15/xps_cpus:080000
/sys/class/net/eth2/queues/tx-16/xps_cpus:000100
/sys/class/net/eth2/queues/tx-17/xps_cpus:100000
/sys/class/net/eth2/queues/tx-18/xps_cpus:000200
/sys/class/net/eth2/queues/tx-19/xps_cpus:200000
/sys/class/net/eth2/queues/tx-20/xps_cpus:000400
/sys/class/net/eth2/queues/tx-21/xps_cpus:400000
/sys/class/net/eth2/queues/tx-22/xps_cpus:000800
/sys/class/net/eth2/queues/tx-23/xps_cpus:800000

> 
> I think it is based on the default xps configuration, but I don't want
> to get the details wrong, checking with Tariq and will reply (he's OOO).
>