Re: [PATCH 0/3] nvme-cli: NIC topology aware I/O queue scaling

[Nilay Shroff <nilay@linux.ibm.com>]

On 8/23/26 3:13 AM, Sagi Grimberg wrote:
> 
> 
> On 21/08/2026 17:43, Nilay Shroff wrote:
>> Hi,
>>
>> This series is a rework of the earlier patchset[1]. The main
>> difference is that --nr-io-queues is now calculated in nvme-cli
>> instead of in the kernel when establishing an NVMe/TCP connection.
>>
>> This rework is based on the feedback received[2] from the netdev
>> maintainers.
>>
>> The original patchset determined the number of NVMe/TCP I/O queues
>> based on the number of online CPUs and the number of hardware queues
>> available on the NIC in kernel driver. This series moves that logic
>> to nvme-cli.
>>
>> When --nr-io-queues is not explicitly specified, nvme-cli determines
>> the egress netdev for the NVMe/TCP connection, retrieves its current
>> hardware queue count, and calculates the default as:
>>
>>     min(nr_hw_queues, num_online_cpus)
> 
> This looks reasonable Nilay.
> 
Thank you...

> I am wandering tho if we want to place some lower limit here.
> For example, my laptop has a virtio device with 4 cpu cores and
> a single combined ring:
> -- 
> $ lscpu | grep NUMA
> NUMA node(s):                            1
> NUMA node0 CPU(s):                       0-3
> $ ethtool -l enp7s0
> Channel parameters for enp7s0:
> Pre-set maximums:
> RX:        n/a
> TX:        n/a
> Other:        n/a
> Combined:    1
> Current hardware settings:
> RX:        n/a
> TX:        n/a
> Other:        n/a
> Combined:    1
> -- 
> 
> It would be kinda annoying for me to now explicitly pass the nr-io-queues...
> I am wandering if some sort of threshold make sense as what you are aiming for
> is reducing the amount of queues for large cpu counts...

I think you're running a QEMU guest using user-mode (SLIRP) networking, so having
a combined queue count of 1 is expected.

I also tested this setup before posting the change. With QEMU user-mode networking,
increasing --nr-io-queues beyond 1 (I tried 4 and 8 with vCPU set to match those
numbers) did not improve performance. In fact, limiting --nr-io-queues to 1, which
matches the netdev's single combined queue, gave slightly better performance.

My understanding is that in this topology there is only a single underlying
virtqueue/network queue, so creating multiple NVMe/TCP I/O queues does not provide
additional network parallelism. Instead, those NVMe/TCP queues end up contending
on the same virtqueue/network queue, which can add overhead without providing additional
throughput.

So I'm inclined not to impose an arbitrary lower limit on --nr-io-queues. If the
egress netdev reports only one hardware queue, min(nr_hw_queues, num_online_cpus)
naturally gives 1, which seems like the appropriate choice for this topology. If
the goal is to achieve higher throughput in a QEMU guest, using a networking
configuration with multiqueue support (for example, multiqueue virtio with
vhost-net) would be more appropriate. In that case, the number of available netdev
hardware queues can scale beyond one, and the calculated nr-io-queues can make use
of that parallelism. Alternatively, with PCI/VFIO passthrough, the guest can directly
use the queues exposed by the physical NIC, allowing --nr-io-queues to scale
accordingly.

Thanks,
--Nilay