Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Nilay Shroff <nilay@linux.ibm.com>]

On 8/22/26 6:19 AM, Saravanan D wrote:
> On Fri, 21 Aug 2026 21:17:49 +0530 Nilay Shroff <nilay@linux.ibm.com> wrote:
>> It seems that here multi tenants shares the same NVMe/TCP controller.
>> So if the concern is CPU isolation between tenants, why are multiple
>> tenants sharing the same NVMe/TCP controller? Wouldn't a per-tenant
>> controller/connection provide better isolation and allow each
>> controller's queues to be mapped to the tenant's CPU set?
> 
> The controllers are shared because the tenant VMs' virtio-blk devices
> are backed by namespaces under one multipath subsystem the hypervisor host
> connects to. With many VMs per host, maintaining a per-tenant controller
> is not always feasible because of the overhead on the host and risk of
> running into target connection limits. blk-mq spreads any controller's
> queues across every online CPU, so nvme_tcp_set_queue_io_cpu() picks io_cpu
> from a machine wide map whether the controller is shared or dedicated.
> Tying socket work to the submitting CPU will reduce VM steal time in
> these deployment scenarios.
> 

Yes, nvme_tcp_set_queue_io_cpu() currently spreads the I/O queues across the
online CPUs, so I understand why a shared controller can end up with its queues
mapped across CPUs belonging to different VM/tenant cpusets.

My point was if we could instead  make the queue-to-CPU mapping aware of the
tenant's CPU partition when the controller is created. For example, if the
hypervisor knows the CPU set associated with a VM, we could pass a CPU-placement
hint/cpuset as part of the fabric connection setup and then have
nvme_tcp_set_queue_io_cpu() select the queue CPUs from that set rather than from
the machine-wide blk-mq CPU map.

This would preserve a stable queue-to-CPU mapping while ensuring that the
NVMe/TCP socket work for a controller is confined to the tenant's CPU partition.
Compared with dynamically adopting the submitting CPU, I think this could have
some advantages:
- the queue-to-CPU mapping remains stable
- it would make CPU/NIC topology tuning such as XPS/RPS and ntuple steering
   more deterministic
- As queues are not moved across cpus, it may provide better cache locality
   and potentially reduce cross-CPU wakeups/IPIs associated with moving the
   socket work.

There is another aspect I'm wondering about regarding the VM steal-time
observation. The io_cpu adoption addresses the execution of nvme_tcp_io_work,
but there are still other parts of the receive path such as the NIC RX interrupt
or NAPI and subsequent network processing that can execute on CPUs outside the
VM's cpuset depending on IRQ/RPS configuration. So I'm not sure whether moving
io_cpu to the submitting CPU by itself can guarantee that all NVMe/TCP network
processing stays within the tenant's CPU partition.

If the objective is strict CPU isolation, perhaps it would be useful to consider the
CPU partition as a property of the NVMe/TCP connection and keep the queue/CPU mapping
stable within that partition, while separately configuring the NIC IRQ/RPS/XPS/steering
to maintain the same locality.

But yes my above recommendation would require creating separate controller
per tenant/VM.

Thanks,
--Nilay