From: Jakub Kicinski <kuba@kernel.org>
To: Nilay Shroff <nilay@linux.ibm.com>
Cc: Stanislav Fomichev <sdf.kernel@gmail.com>,
kbusch@kernel.org, hch@lst.de, hare@suse.de, sagi@grimberg.me,
chaitanyak@nvidia.com, gjoyce@linux.ibm.com, davem@davemloft.net,
edumazet@google.com, pabeni@redhat.com, horms@kernel.org,
linux-nvme@lists.infradead.org, netdev@vger.kernel.org
Subject: Re: [RESEND PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count
Date: Fri, 7 Aug 2026 16:09:04 -0700 [thread overview]
Message-ID: <20260807160904.2eb1b3d1@kernel.org> (raw)
In-Reply-To: <4d8c8d92-d39b-4721-a405-f03f01fbb95b@linux.ibm.com>
On Sat, 1 Aug 2026 19:08:05 +0530 Nilay Shroff wrote:
> > Looks like the userspace can already pass the preferred number of queues,
> > so in this case, why not do all this netdev resolution and queue
> > estimation in the userspace? Presumably most or the users you care
> > about always go through nvme-cli, right?
>
> Yes, userspace can already specify the preferred number of I/O queues, and nvme-cli
> provides an option to do so when creating an NVMe/TCP connection. However, choosing
> an appropriate value requires userspace to know both the number of online CPUs and
> the number of active TX/RX queues on the NIC used for the connection. Determining
> the latter also requires identifying the correct netdevice. That may involve a route
> lookup to determine the egress interface, particularly when the NVMe/TCP host and
> target are not on the same subnet.
>
> So while this could be implemented in nvme-cli, it would require userspace to duplicate
> the logic needed to determine the actual netdevice and its current queue configuration.
> The intent of this change is to make the default queue selection automatic and avoid
> requiring users to determine and specify this topology manually.
In another message you said you add ntuple filters. So you _are_ doing
what you describe here as a problem. User space will know something we
don't know sooner or later, so you should just add the uAPI instead of
guessing in the kernel. BTW the queue count is likely to change after
all of user space boots, so if you run before whatever configures
queues for the machine in userspace you'll be using wrong counts.
> An explicitly specified "nr_io_queues" would still take precedence, so userspace can
> override the default when desired.
>
> Just for the note, this change also follows the general approach used by nvme-pci, where
> the default number of I/O queues is constrained by both the number of possible CPUs and the
> queue resources available from the controller.
Not sure that maps well to networking. For TCP at least there will be
a protocol stack that runs between the device queues and your queues.
I guess that will depend on the network and the details of the
benchmark. But again, better to let the user tune to their workload
and machine.
Consider patch 1 nacked.
next prev parent reply other threads:[~2026-08-07 23:09 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-31 7:39 [RESEND PATCH v2 0/4] nvme-tcp: NIC topology aware I/O queue scaling and queue info export Nilay Shroff
2026-07-31 7:39 ` [RESEND PATCH v2 1/4] net: add helper for device lookup by destination address Nilay Shroff
2026-07-31 7:39 ` [RESEND PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count Nilay Shroff
2026-07-31 16:41 ` Stanislav Fomichev
2026-08-01 13:38 ` Nilay Shroff
2026-08-07 23:09 ` Jakub Kicinski [this message]
2026-08-08 10:42 ` Nilay Shroff
2026-07-31 7:39 ` [RESEND PATCH v2 3/4] nvme: add debugfs helpers for NVMe drivers Nilay Shroff
2026-07-31 8:14 ` Maurizio Lombardi
2026-08-01 13:47 ` Nilay Shroff
2026-07-31 7:39 ` [RESEND PATCH v2 4/4] nvme: expose queue information via debugfs Nilay Shroff
2026-07-31 16:39 ` [RESEND PATCH v2 0/4] nvme-tcp: NIC topology aware I/O queue scaling and queue info export Stanislav Fomichev
2026-08-01 13:45 ` Nilay Shroff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260807160904.2eb1b3d1@kernel.org \
--to=kuba@kernel.org \
--cc=chaitanyak@nvidia.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=gjoyce@linux.ibm.com \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=horms@kernel.org \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=netdev@vger.kernel.org \
--cc=nilay@linux.ibm.com \
--cc=pabeni@redhat.com \
--cc=sagi@grimberg.me \
--cc=sdf.kernel@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox