Re: Number of data and admin queues in use

[Chaitanya Kulkarni <chaitanyak@nvidia.com>]

On 7/15/25 07:39, Keith Busch wrote:
> On Tue, Jul 15, 2025 at 03:58:01AM +0200, Thomas Glanzmann wrote:
>> I one heard that the Linux kernel allocates one queue per processor (core or
>> hyperthread). I can see that using /proc/interrupts but only on physical
>> devices, but not on NVME/TCP systems.
> For PCI, the driver automatically handles the queue and interrupt setup,
> and cpu assignment.
>
> For TCP (and all fabrics transports), you have to specificy how many
> connections you want to make ("nr_io_queues=X") when you're setting up
> your initial fabrics connection.
>
> If you want to see what you've ended up with, you can consult the
> namespaces' sysfs entries:
>
> How many IO queues are there:
>
>    # ls -1 /sys/block/nvme0n1/mq/ | wc -l
>    64
>
> How large is each IO queue:
>
>    # cat /sys/block/nvme0n1/queue/nr_requests
>    1023
>

For block layer queue allocation which will happen when you issue connect
command from host to target which will create controller and populate block
and char devices see below [1].

 From what I can see you are getting number of queues for both tcp and pcie
NVMe controller, what is your question ?

Another way to dig into controller side fields or queue depth you can
read the CAP space see this from spec

Figure 36: Offset 0h: CAP – Controller Capabilities :-

"Maximum Queue Entries Supported (MQES): This field indicates the maximum
individual queue size that the controller supports. For NVMe over PCIe
implementations, this value applies to the I/O Submission Queues and I/O
Completion Queues that the host creates. For NVMe over Fabrics 
implementations,
this value applies to only the I/O Submission Queues that the host 
creates. This is
a 0’s based value. The minimum value is 1h, indicating two entries."

-ck

[1]

For fabrics transport (TCP) number are queues are calculated using
nvmf_nr_io_queue() to make sure we don't create more read/defult
queues than CPUs available same check is also applicable for write
and poll queues.
nvme_set_queue_count adjusts the queue count based on controller
capabilities which cal also clamp the queue count.

nvmf_set_io_queues() set queue count for each queue type read,
default, poll. then nvmf_map_queues() maps them into blk-mq
structure so that default/read/poll and each gets attached to
blk_mq context.

On My machine I've 48 CPUs so when I create tcp target I get :-

[ 1196.058440] nvme nvme1: creating 48 I/O queues.
[ 1196.062370] nvme nvme1: mapped 48/0/0 default/read/poll queues.

you should be able to see this into debug messages that is coming
from queue allocation helpers respectively that also has controller
device name "nvme1" :-

nvme_tcp_alloc_io_queues()
nvmf_map_queues()

Hope this helps.