Re: [PATCH 0/3] nvme-cli: NIC topology aware I/O queue scaling

[Nilay Shroff <nilay@linux.ibm.com>]

On 8/31/26 3:36 AM, Sagi Grimberg wrote:
> 
> 
> On 24/08/2026 11:48, Nilay Shroff wrote:
>> On 8/23/26 3:13 AM, Sagi Grimberg wrote:
>>>
>>>
>>> On 21/08/2026 17:43, Nilay Shroff wrote:
>>>> Hi,
>>>>
>>>> This series is a rework of the earlier patchset[1]. The main
>>>> difference is that --nr-io-queues is now calculated in nvme-cli
>>>> instead of in the kernel when establishing an NVMe/TCP connection.
>>>>
>>>> This rework is based on the feedback received[2] from the netdev
>>>> maintainers.
>>>>
>>>> The original patchset determined the number of NVMe/TCP I/O queues
>>>> based on the number of online CPUs and the number of hardware queues
>>>> available on the NIC in kernel driver. This series moves that logic
>>>> to nvme-cli.
>>>>
>>>> When --nr-io-queues is not explicitly specified, nvme-cli determines
>>>> the egress netdev for the NVMe/TCP connection, retrieves its current
>>>> hardware queue count, and calculates the default as:
>>>>
>>>>     min(nr_hw_queues, num_online_cpus)
>>>
>>> This looks reasonable Nilay.
>>>
>> Thank you...
>>
>>> I am wandering tho if we want to place some lower limit here.
>>> For example, my laptop has a virtio device with 4 cpu cores and
>>> a single combined ring:
>>> -- 
>>> $ lscpu | grep NUMA
>>> NUMA node(s):                            1
>>> NUMA node0 CPU(s):                       0-3
>>> $ ethtool -l enp7s0
>>> Channel parameters for enp7s0:
>>> Pre-set maximums:
>>> RX:        n/a
>>> TX:        n/a
>>> Other:        n/a
>>> Combined:    1
>>> Current hardware settings:
>>> RX:        n/a
>>> TX:        n/a
>>> Other:        n/a
>>> Combined:    1
>>> -- 
>>>
>>> It would be kinda annoying for me to now explicitly pass the nr-io-queues...
>>> I am wandering if some sort of threshold make sense as what you are aiming for
>>> is reducing the amount of queues for large cpu counts...
>>
>> I think you're running a QEMU guest using user-mode (SLIRP) networking, so having
>> a combined queue count of 1 is expected.
>>
>> I also tested this setup before posting the change. With QEMU user-mode networking,
>> increasing --nr-io-queues beyond 1 (I tried 4 and 8 with vCPU set to match those
>> numbers) did not improve performance. In fact, limiting --nr-io-queues to 1, which
>> matches the netdev's single combined queue, gave slightly better performance.
>>
>> My understanding is that in this topology there is only a single underlying
>> virtqueue/network queue, so creating multiple NVMe/TCP I/O queues does not provide
>> additional network parallelism. Instead, those NVMe/TCP queues end up contending
>> on the same virtqueue/network queue, which can add overhead without providing additional
>> throughput.
> 
> I don't care about performance. I care that if I am testing stuff, I want more than a single
> queue. And it is annoying to explicitly change the queue count...
> 
> Also, I don't know if your performance statements are correct for TLS.

Okay, in that case, if nr_hw_queues is 1 and the user hasn't explicitly specified --nr-io-queues,
we would not limit the number of I/O queues based on nr_hw_queues. Instead, we would use
num_online_cpus as the default.

So the policy would effectively be:

if (nr_hw_queues > 1)
     nr_io_queues = min(nr_hw_queues, num_online_cpus);
else
     nr_io_queues = num_online_cpus;

This would preserve the existing behavior for single-queue devices while still using the
NIC hardware queue count to constrain the default on multi-queue devices.

Does this look reasonable?

Thanks,
--Nilay