Re: [RESEND PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count

[Nilay Shroff <nilay@linux.ibm.com>]

On 8/8/26 4:39 AM, Jakub Kicinski wrote:
> On Sat, 1 Aug 2026 19:08:05 +0530 Nilay Shroff wrote:
>>> Looks like the userspace can already pass the preferred number of queues,
>>> so in this case, why not do all this netdev resolution and queue
>>> estimation in the userspace? Presumably most or the users you care
>>> about always go through nvme-cli, right?
>>
>> Yes, userspace can already specify the preferred number of I/O queues, and nvme-cli
>> provides an option to do so when creating an NVMe/TCP connection. However, choosing
>> an appropriate value requires userspace to know both the number of online CPUs and
>> the number of active TX/RX queues on the NIC used for the connection. Determining
>> the latter also requires identifying the correct netdevice. That may involve a route
>> lookup to determine the egress interface, particularly when the NVMe/TCP host and
>> target are not on the same subnet.
>>
>> So while this could be implemented in nvme-cli, it would require userspace to duplicate
>> the logic needed to determine the actual netdevice and its current queue configuration.
>> The intent of this change is to make the default queue selection automatic and avoid
>> requiring users to determine and specify this topology manually.
> 
> In another message you said you add ntuple filters. So you _are_ doing
> what you describe here as a problem. User space will know something we
> don't know sooner or later, so you should just add the uAPI instead of
> guessing in the kernel. BTW the queue count is likely to change after
> all of user space boots, so if you run before whatever configures
> queues for the machine in userspace you'll be using wrong counts.
> 
Well, that ntuple filter configuration was done looking at the debugfs
output which is produced in patch 4/4. The debugfs generates the
enough information including queue count and per queue flow information
which is then programmed into ntuple filter.

>> An explicitly specified "nr_io_queues" would still take precedence, so userspace can
>> override the default when desired.
>>
>> Just for the note, this change also follows the general approach used by nvme-pci, where
>> the default number of I/O queues is constrained by both the number of possible CPUs and the
>> queue resources available from the controller.
> 
> Not sure that maps well to networking. For TCP at least there will be
> a protocol stack that runs between the device queues and your queues.
> I guess that will depend on the network and the details of the
> benchmark. But again, better to let the user tune to their workload
> and machine.
> 
> Consider patch 1 nacked.

My motivation here was to improve the default behavior for the common case
where nr_io_queues is not explicitly specified. So if the preference is to
keep this in userspace, would you be open to exposing the required information
through a kernel interface (if something is still missing) and implementing
the queue selection logic in nvme-cli instead? That would still allow us to
automate the default queue selection without embedding this change in the kernel.

Thanks,
--Nilay