Re: [RESEND PATCH v2 0/4] nvme-tcp: NIC topology aware I/O queue scaling and queue info export

[Stanislav Fomichev <sdf.kernel@gmail.com>]

On 07/31, Nilay Shroff wrote:
> Hi,
> 
> This is a resend of the previous series to include the networking maintainers
> and mailing list. There are no code or commit message changes since the
> previous posting.
> 
> This series has been updated based on the feedback received during LSFMM.
> The changelog is updated accordingly.
> 
> The NVMe/TCP host driver currently provisions I/O queues primarily based
> on CPU availability rather than the capabilities and topology of the
> underlying network interface.
> 
> On modern systems with many CPUs but fewer NIC hardware queues, this can
> lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue,
> resulting in increased lock contention, cacheline bouncing, and degraded
> throughput.
> 
> This RFC proposes a set of changes to better align NVMe/TCP I/O queues
> with NIC queue resources, and to expose queue/flow information to enable
> more effective system-level tuning.
> 
> Key ideas
> ---------
> 
> 1. Scale NVMe/TCP I/O queues based on NIC queue count
>    Instead of relying solely on CPU count, limit the number of I/O workers
>    to:
>        min(num_online_cpus, netdev->real_num_{tx,rx}_queues)
> 
> 2. Improve CPU locality
>    Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity
>    to reduce cross-CPU traffic and improve cache locality.
> 
> 3. Expose queue and flow information via debugfs
>    Export per-I/O queue information including:
>        - queue id (qid)
>        - CPU affinity
>        - TCP flow (src/dst IP and ports)

[..]

>    This enables userspace tools to configure:
>        - IRQ affinity
>        - RPS/XPS
>        - ntuple steering
>        - or any other scaling as deemed feasible

Can you expand on this a bit? What specifically helped the most for your
tuned case?