Re: [PATCH 0/3] nvme-cli: NIC topology aware I/O queue scaling

[Nilay Shroff <nilay@linux.ibm.com>]

Hi Sagi,

A gentle ping on this one.... Does the proposed solution address your concern?

Thanks,
--Nilay

On 8/31/26 10:34 AM, Nilay Shroff wrote:
> On 8/31/26 3:36 AM, Sagi Grimberg wrote:
>>
>>
>> On 24/08/2026 11:48, Nilay Shroff wrote:
>>> On 8/23/26 3:13 AM, Sagi Grimberg wrote:
>>>>
>>>>
>>>> On 21/08/2026 17:43, Nilay Shroff wrote:
>>>>> Hi,
>>>>>
>>>>> This series is a rework of the earlier patchset[1]. The main
>>>>> difference is that --nr-io-queues is now calculated in nvme-cli
>>>>> instead of in the kernel when establishing an NVMe/TCP connection.
>>>>>
>>>>> This rework is based on the feedback received[2] from the netdev
>>>>> maintainers.
>>>>>
>>>>> The original patchset determined the number of NVMe/TCP I/O queues
>>>>> based on the number of online CPUs and the number of hardware queues
>>>>> available on the NIC in kernel driver. This series moves that logic
>>>>> to nvme-cli.
>>>>>
>>>>> When --nr-io-queues is not explicitly specified, nvme-cli determines
>>>>> the egress netdev for the NVMe/TCP connection, retrieves its current
>>>>> hardware queue count, and calculates the default as:
>>>>>
>>>>>     min(nr_hw_queues, num_online_cpus)
>>>>
>>>> This looks reasonable Nilay.
>>>>
>>> Thank you...
>>>
>>>> I am wandering tho if we want to place some lower limit here.
>>>> For example, my laptop has a virtio device with 4 cpu cores and
>>>> a single combined ring:
>>>> -- 
>>>> $ lscpu | grep NUMA
>>>> NUMA node(s):                            1
>>>> NUMA node0 CPU(s):                       0-3
>>>> $ ethtool -l enp7s0
>>>> Channel parameters for enp7s0:
>>>> Pre-set maximums:
>>>> RX:        n/a
>>>> TX:        n/a
>>>> Other:        n/a
>>>> Combined:    1
>>>> Current hardware settings:
>>>> RX:        n/a
>>>> TX:        n/a
>>>> Other:        n/a
>>>> Combined:    1
>>>> -- 
>>>>
>>>> It would be kinda annoying for me to now explicitly pass the nr-io-queues...
>>>> I am wandering if some sort of threshold make sense as what you are aiming for
>>>> is reducing the amount of queues for large cpu counts...
>>>
>>> I think you're running a QEMU guest using user-mode (SLIRP) networking, so having
>>> a combined queue count of 1 is expected.
>>>
>>> I also tested this setup before posting the change. With QEMU user-mode networking,
>>> increasing --nr-io-queues beyond 1 (I tried 4 and 8 with vCPU set to match those
>>> numbers) did not improve performance. In fact, limiting --nr-io-queues to 1, which
>>> matches the netdev's single combined queue, gave slightly better performance.
>>>
>>> My understanding is that in this topology there is only a single underlying
>>> virtqueue/network queue, so creating multiple NVMe/TCP I/O queues does not provide
>>> additional network parallelism. Instead, those NVMe/TCP queues end up contending
>>> on the same virtqueue/network queue, which can add overhead without providing additional
>>> throughput.
>>
>> I don't care about performance. I care that if I am testing stuff, I want more than a single
>> queue. And it is annoying to explicitly change the queue count...
>>
>> Also, I don't know if your performance statements are correct for TLS.
> 
> Okay, in that case, if nr_hw_queues is 1 and the user hasn't explicitly specified --nr-io-queues,
> we would not limit the number of I/O queues based on nr_hw_queues. Instead, we would use
> num_online_cpus as the default.
> 
> So the policy would effectively be:
> 
> if (nr_hw_queues > 1)
>      nr_io_queues = min(nr_hw_queues, num_online_cpus);
> else
>      nr_io_queues = num_online_cpus;
> 
> This would preserve the existing behavior for single-queue devices while still using the
> NIC hardware queue count to constrain the default on multi-queue devices.
> 
> Does this look reasonable?
> 
> Thanks,
> --Nilay
>