[PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Saravanan D]

nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time,
before any I/O exists, as the least loaded CPU in the queue's blk-mq
map group, and all socket work then runs there for the connection's
lifetime. On hosts that partition CPUs between pinned workloads a map
group can straddle a partition boundary, so the pick can land one
workload's socket processing on CPUs owned by another. On a 384 cpu
multi tenant host with one VM driving ~1.1 GB/s of writes, 9% of
nvme_tcp_io_work executions ran outside the submitting VM's cpuset,
all on io_cpus of boundary straddling map groups, observed by the
neighbor as steal time it did not cause.

Adopt the submitting CPU as io_cpu for every command except the
fabrics Connect. The submitter is a member of the map group by
construction, and the nvme_tcp_cpu_queues accounting moves with each
adoption. Connect is the only command on an I/O queue that does not
represent the data path, since it is injected on an arbitrary CPU by
blk_mq_alloc_request_hctx(), so it is skipped and the first real read
or write decides. User passthrough is submitted from a real task on
the submitting CPU and adopts like any other command.

Queues outlive the workloads that submit through them, so adoption
re-arms after 30 seconds of queue quiet. An idle queue is reclaimed by
its next submitter, while a busy queue keeps a stable io_cpu and
cannot ping pong between two live submitters. Concurrent writers on
different CPUs serialize on a cmpxchg on io_cpu.

The behavior is opt in per controller via the io_cpu_adopt fabrics
option at connect time. wq_unbound takes precedence when set.

Signed-off-by: Saravanan D <saravanand@crusoe.ai>
---
Changes since v1 [1]:
- Special case the fabrics Connect command instead of skipping all
  passthrough commands, so user passthrough I/O adopts too.
- Make it a per-controller io_cpu_adopt fabrics option instead of a
  global wq_adopt module parameter, set once at connect time rather
  than flipped under a live connection.

Both per Christoph Hellwig's review.

[1] https://lore.kernel.org/linux-nvme/20260806023947.94680-2-saravanand@crusoe.ai/

 drivers/nvme/host/fabrics.c |  4 ++
 drivers/nvme/host/fabrics.h |  2 +
 drivers/nvme/host/tcp.c     | 75 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 80 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index fd5abd04e080..26f8703744de 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -695,6 +695,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_NR_WRITE_QUEUES,	"nr_write_queues=%d"	},
 	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
 	{ NVMF_OPT_TOS,			"tos=%d"		},
+	{ NVMF_OPT_IO_CPU_ADOPT,	"io_cpu_adopt"		},
 #ifdef CONFIG_NVME_TCP_TLS
 	{ NVMF_OPT_KEYRING,		"keyring=%d"		},
 	{ NVMF_OPT_TLS_KEY,		"tls_key=%d"		},
@@ -951,6 +952,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DATA_DIGEST:
 			opts->data_digest = true;
 			break;
+		case NVMF_OPT_IO_CPU_ADOPT:
+			opts->io_cpu_adopt = true;
+			break;
 		case NVMF_OPT_NR_WRITE_QUEUES:
 			if (match_int(args, &token)) {
 				ret = -EINVAL;
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index caf5503d0833..3ecac041a628 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -67,6 +67,7 @@ enum {
 	NVMF_OPT_KEYRING	= 1 << 26,
 	NVMF_OPT_TLS_KEY	= 1 << 27,
 	NVMF_OPT_CONCAT		= 1 << 28,
+	NVMF_OPT_IO_CPU_ADOPT	= 1 << 29,
 };
 
 /**
@@ -140,6 +141,7 @@ struct nvmf_ctrl_options {
 	unsigned int		nr_poll_queues;
 	int			tos;
 	int			fast_io_fail_tmo;
+	bool			io_cpu_adopt;
 };
 
 /*
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 87d8067f3283..530e38695257 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -92,6 +92,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_LIVE		= 1,
 	NVME_TCP_Q_POLLING	= 2,
 	NVME_TCP_Q_IO_CPU_SET	= 3,
+	NVME_TCP_Q_IO_CPU_ADOPTED = 4,
 };
 
 enum nvme_tcp_recv_state {
@@ -105,6 +106,7 @@ struct nvme_tcp_queue {
 	struct socket		*sock;
 	struct work_struct	io_work;
 	int			io_cpu;
+	unsigned long		last_data;
 
 	struct mutex		queue_lock;
 	struct mutex		send_mutex;
@@ -2783,6 +2785,74 @@ static void nvme_tcp_commit_rqs(struct blk_mq_hw_ctx *hctx)
 		queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
 }
 
+/* Re-adopt io_cpu on the first data request after this much queue idle time */
+#define NVME_TCP_READOPT_IDLE	(30 * HZ)
+
+/*
+ * Adopt the CPU of the current data submission as the queue's io_cpu.
+ *
+ * The connect time choice in nvme_tcp_set_queue_io_cpu() picks the least
+ * loaded CPU in the queue's mq_map group before any I/O exists, so it
+ * cannot know which side of the group the actual submitters live on. On
+ * hosts that partition CPUs between pinned workloads a group that
+ * straddles a partition boundary can get an io_cpu on CPUs the submitting
+ * workload does not own, and its network processing then preempts an
+ * unrelated workload. The submitting CPU is in the queue's mq_map group
+ * by construction, so adopting it preserves the spreading property while
+ * landing the work on the side that generates it.
+ *
+ * Queues belong to the controller connection and outlive the workloads
+ * that submit through them, so adoption re-arms after NVME_TCP_READOPT_IDLE
+ * of queue quiet. A successor workload reclaims an idle queue with its
+ * first data request, while a continuously busy queue keeps a stable
+ * io_cpu and cannot ping pong between two live submitters.
+ *
+ * The fabrics Connect command targets a specific queue via
+ * blk_mq_alloc_request_hctx() and so runs on an arbitrary CPU that does
+ * not represent the data path, so it is skipped and the first real read
+ * or write decides. All other commands, including user passthrough,
+ * carry a real submitting CPU and adopt.
+ *
+ * Adoption is opt in per controller via the io_cpu_adopt connect option
+ * and is bypassed when wq_unbound is set.
+ */
+static void nvme_tcp_adopt_io_cpu(struct nvme_tcp_queue *queue,
+		struct request *rq)
+{
+	struct nvme_command *cmd = nvme_req(rq)->cmd;
+	int old, new;
+
+	if (!queue->ctrl->ctrl.opts->io_cpu_adopt || wq_unbound)
+		return;
+	if (!nvme_tcp_queue_id(queue))
+		return;
+	if (nvme_is_fabrics(cmd) &&
+	    cmd->fabrics.fctype == nvme_fabrics_type_connect)
+		return;
+
+	if (test_bit(NVME_TCP_Q_IO_CPU_ADOPTED, &queue->flags) &&
+	    time_before(jiffies, READ_ONCE(queue->last_data) +
+				NVME_TCP_READOPT_IDLE)) {
+		WRITE_ONCE(queue->last_data, jiffies);
+		return;
+	}
+
+	WRITE_ONCE(queue->last_data, jiffies);
+	set_bit(NVME_TCP_Q_IO_CPU_ADOPTED, &queue->flags);
+
+	old = READ_ONCE(queue->io_cpu);
+	new = raw_smp_processor_id();
+	if (old == new || !try_cmpxchg(&queue->io_cpu, &old, new))
+		return;
+
+	if (test_bit(NVME_TCP_Q_IO_CPU_SET, &queue->flags)) {
+		atomic_dec(&nvme_tcp_cpu_queues[old]);
+		atomic_inc(&nvme_tcp_cpu_queues[new]);
+	}
+	dev_dbg(queue->ctrl->ctrl.device, "queue %d: adopted io_cpu %d\n",
+		nvme_tcp_queue_id(queue), new);
+}
+
 static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx,
 		const struct blk_mq_queue_data *bd)
 {
@@ -2802,6 +2872,8 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx,
 
 	nvme_start_request(rq);
 
+	nvme_tcp_adopt_io_cpu(queue, rq);
+
 	nvme_tcp_queue_request(req, bd->last);
 
 	return BLK_STS_OK;
@@ -3047,7 +3119,8 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
 			  NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
 			  NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES |
 			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS |
-			  NVMF_OPT_KEYRING | NVMF_OPT_TLS_KEY | NVMF_OPT_CONCAT,
+			  NVMF_OPT_KEYRING | NVMF_OPT_TLS_KEY | NVMF_OPT_CONCAT |
+			  NVMF_OPT_IO_CPU_ADOPT,
 	.create_ctrl	= nvme_tcp_create_ctrl,
 };
 

base-commit: bf881dd20062db5e951a0d0703cb476df8c9fdee
-- 
2.53.0

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Daniel Wagner]

On Thu, Aug 20, 2026 at 01:36:34AM -0700, Saravanan D wrote:
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index fd5abd04e080..26f8703744de 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -695,6 +695,7 @@ static const match_table_t opt_tokens = {
>  	{ NVMF_OPT_NR_WRITE_QUEUES,	"nr_write_queues=%d"	},
>  	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
>  	{ NVMF_OPT_TOS,			"tos=%d"		},
> +	{ NVMF_OPT_IO_CPU_ADOPT,	"io_cpu_adopt"		},
>  #ifdef CONFIG_NVME_TCP_TLS
>  	{ NVMF_OPT_KEYRING,		"keyring=%d"		},
>  	{ NVMF_OPT_TLS_KEY,		"tls_key=%d"		},
> @@ -951,6 +952,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>  		case NVMF_OPT_DATA_DIGEST:
>  			opts->data_digest = true;
>  			break;
> +		case NVMF_OPT_IO_CPU_ADOPT:
> +			opts->io_cpu_adopt = true;
> +			break;

FWIW, from past experience if this gets accepted, it is most likely not
the last policy controlling knob which is needed. I see this somewhat
orthogonal to io_policy options. Thus my question should this user
interface be just a bool or should we have something more future proof
here?

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Saravanan D]

On Thu, 20 Aug 2026 17:39:48 +0200 Daniel Wagner <dwagner@suse.de> wrote:
> FWIW, from past experience if this gets accepted, it is most likely not
> the last policy controlling knob which is needed. I see this somewhat
> orthogonal to io_policy options. Thus my question should this user
> interface be just a bool or should we have something more future proof
> here?

I agree with you, wq_unbound already exists as a boolean and this
patch introduces io_cpu_adopt as a second one, which is why I had
to introduce a precedence rule so that wq_unbound wins when both
are set. Adding more such behaviors as separate bools would only
multiply these precedence rules.

So rather than a bool, we can consider io_cpu to be its own policy
knob which is orthogonal to io_policy which primarily decides
which path the I/O takes. io_cpu=<policy> decides for a given
connection's queue which host cpu will perform the socket work,
with values such as default, adopt and unbound.

On the same theme, the one heuristic in this v2 patch is the re-arm
idle time, which could be surfaced as io_cpu_readopt_idle=<sec> to
become configurable.

I am very new to nvme-tcp and I dont have any strong opinions on
how to make the contribution future proof and reduce burden for
nvme maintainers.

Christoph asked me in v1 to move the knob from a module parameter
to a per controller connect option, so if you and other nvme
maintainers have a preference, I am willing to consider the
interface before v3.

Thanks for your review.

Saravanan D.

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Nilay Shroff]

On 8/20/26 2:06 PM, Saravanan D wrote:
> nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time,
> before any I/O exists, as the least loaded CPU in the queue's blk-mq
> map group, and all socket work then runs there for the connection's
> lifetime. On hosts that partition CPUs between pinned workloads a map
> group can straddle a partition boundary, so the pick can land one
> workload's socket processing on CPUs owned by another. On a 384 cpu
> multi tenant host with one VM driving ~1.1 GB/s of writes, 9% of
> nvme_tcp_io_work executions ran outside the submitting VM's cpuset,
> all on io_cpus of boundary straddling map groups, observed by the
> neighbor as steal time it did not cause.

It seems that here multi tenants shares the same NVMe/TCP controller.
Sp if the concern is CPU isolation between tenants, why are multiple
tenants sharing the same NVMe/TCP controller? Wouldn't a per-tenant
controller/connection provide better isolation and allow each controller's
queues to be mapped to the tenant's CPU set? If sharing a controller is
required, it would be useful to understand why.

Thanks,
--Nilay

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Saravanan D]

On Fri, 21 Aug 2026 21:17:49 +0530 Nilay Shroff <nilay@linux.ibm.com> wrote:
> It seems that here multi tenants shares the same NVMe/TCP controller.
> So if the concern is CPU isolation between tenants, why are multiple
> tenants sharing the same NVMe/TCP controller? Wouldn't a per-tenant
> controller/connection provide better isolation and allow each
> controller's queues to be mapped to the tenant's CPU set?

The controllers are shared because the tenant VMs' virtio-blk devices 
are backed by namespaces under one multipath subsystem the hypervisor host
connects to. With many VMs per host, maintaining a per-tenant controller
is not always feasible because of the overhead on the host and risk of
running into target connection limits. blk-mq spreads any controller's
queues across every online CPU, so nvme_tcp_set_queue_io_cpu() picks io_cpu
from a machine wide map whether the controller is shared or dedicated.
Tying socket work to the submitting CPU will reduce VM steal time in 
these deployment scenarios.

Saravanan D.

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Nilay Shroff]

On 8/22/26 6:19 AM, Saravanan D wrote:
> On Fri, 21 Aug 2026 21:17:49 +0530 Nilay Shroff <nilay@linux.ibm.com> wrote:
>> It seems that here multi tenants shares the same NVMe/TCP controller.
>> So if the concern is CPU isolation between tenants, why are multiple
>> tenants sharing the same NVMe/TCP controller? Wouldn't a per-tenant
>> controller/connection provide better isolation and allow each
>> controller's queues to be mapped to the tenant's CPU set?
> 
> The controllers are shared because the tenant VMs' virtio-blk devices
> are backed by namespaces under one multipath subsystem the hypervisor host
> connects to. With many VMs per host, maintaining a per-tenant controller
> is not always feasible because of the overhead on the host and risk of
> running into target connection limits. blk-mq spreads any controller's
> queues across every online CPU, so nvme_tcp_set_queue_io_cpu() picks io_cpu
> from a machine wide map whether the controller is shared or dedicated.
> Tying socket work to the submitting CPU will reduce VM steal time in
> these deployment scenarios.
> 

Yes, nvme_tcp_set_queue_io_cpu() currently spreads the I/O queues across the
online CPUs, so I understand why a shared controller can end up with its queues
mapped across CPUs belonging to different VM/tenant cpusets.

My point was if we could instead  make the queue-to-CPU mapping aware of the
tenant's CPU partition when the controller is created. For example, if the
hypervisor knows the CPU set associated with a VM, we could pass a CPU-placement
hint/cpuset as part of the fabric connection setup and then have
nvme_tcp_set_queue_io_cpu() select the queue CPUs from that set rather than from
the machine-wide blk-mq CPU map.

This would preserve a stable queue-to-CPU mapping while ensuring that the
NVMe/TCP socket work for a controller is confined to the tenant's CPU partition.
Compared with dynamically adopting the submitting CPU, I think this could have
some advantages:
- the queue-to-CPU mapping remains stable
- it would make CPU/NIC topology tuning such as XPS/RPS and ntuple steering
   more deterministic
- As queues are not moved across cpus, it may provide better cache locality
   and potentially reduce cross-CPU wakeups/IPIs associated with moving the
   socket work.

There is another aspect I'm wondering about regarding the VM steal-time
observation. The io_cpu adoption addresses the execution of nvme_tcp_io_work,
but there are still other parts of the receive path such as the NIC RX interrupt
or NAPI and subsequent network processing that can execute on CPUs outside the
VM's cpuset depending on IRQ/RPS configuration. So I'm not sure whether moving
io_cpu to the submitting CPU by itself can guarantee that all NVMe/TCP network
processing stays within the tenant's CPU partition.

If the objective is strict CPU isolation, perhaps it would be useful to consider the
CPU partition as a property of the NVMe/TCP connection and keep the queue/CPU mapping
stable within that partition, while separately configuring the NIC IRQ/RPS/XPS/steering
to maintain the same locality.

But yes my above recommendation would require creating separate controller
per tenant/VM.

Thanks,
--Nilay

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Saravanan D]

On Sat, 22 Aug 2026 17:48:17 +0530 Nilay Shroff <nilay@linux.ibm.com> wrote:
> My point was if we could instead make the queue-to-CPU mapping aware of
> the tenant's CPU partition when the controller is created [...] pass a
> CPU-placement hint/cpuset as part of the fabric connection setup [...]

Appreciate your time reviewing this patch and offering your suggestions.

You are right that a stable queue to CPU mapping has real advantages. It
is easier to reason about for NIC side steering and cache locality, and
where the CPU partition is known and fixed at connect time that is the
better design.

Our controllers do not fall under that category. They are host wide
and shared, the tenant VMs are namespaces under one multipath subsystem,
and the control plane creates, destroys and bin packs VMs continuously,
so a cpuset passed at connect would be stale as soon as the tenant mix
changes and could only be re-applied by reconnecting a controller many
live tenants depend on. Adoption needs no per tenant hint and follows that
churn on its own. I posted the concrete topology, 128 io queues over 384
CPUs with the straddling cpu_list groups and two live VM cpusets, in the
reply to Sagi.

The fleet operators who care about steal time already steer the NIC IRQs
and NAPI to a housekeeping CPU pool off the tenant cpusets, so network
processing does not land on tenant CPUs. That leaves the shared queue's
socket work, which this change pins to the submitting CPU so it stays
within the tenant's cpuset.

Thanks,
Saravanan D.

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Sagi Grimberg]

Hey Saravanan,

On 20/08/2026 11:36, Saravanan D wrote:
> nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time,
> before any I/O exists, as the least loaded CPU in the queue's blk-mq
> map group, and all socket work then runs there for the connection's
> lifetime.

That is true. It effectively takes a sane heuristics given that there are
multiple factors it is not aware of.

>   On hosts that partition CPUs between pinned workloads a map
> group can straddle a partition boundary, so the pick can land one
> workload's socket processing on CPUs owned by another. On a 384 cpu
> multi tenant host with one VM driving ~1.1 GB/s of writes, 9% of
> nvme_tcp_io_work executions ran outside the submitting VM's cpuset,
> all on io_cpus of boundary straddling map groups, observed by the
> neighbor as steal time it did not cause.

Can you share a concrete example?
What is the output of:
1. lscpu
2. grep -r ''  /sys/block/nvmeXnY/mq/*/cpu_list

Also, what would be the VMs cpusets on such a server?

In an ideal setting, the nvme controller would support $NR_CPUS worth of
io queues, and both nvme_tcp_set_queue_io_cpu() and blk_mq_map_queues() 
would
map cpu-queue with perfect alignment.

FWIW, I don't think this is an nvme-tcp specific problem, other than the 
fact that TCP
is more CPU consuming than other transports (doing stuff that is not 
offloaded by
the HW...)

>
> Adopt the submitting CPU as io_cpu for every command except the
> fabrics Connect. The submitter is a member of the map group by
> construction, and the nvme_tcp_cpu_queues accounting moves with each
> adoption. Connect is the only command on an I/O queue that does not
> represent the data path, since it is injected on an arbitrary CPU by
> blk_mq_alloc_request_hctx(), so it is skipped and the first real read
> or write decides. User passthrough is submitted from a real task on
> the submitting CPU and adopts like any other command.
>
> Queues outlive the workloads that submit through them, so adoption
> re-arms after 30 seconds of queue quiet. An idle queue is reclaimed by
> its next submitter, while a busy queue keeps a stable io_cpu and
> cannot ping pong between two live submitters. Concurrent writers on
> different CPUs serialize on a cmpxchg on io_cpu.
>
> The behavior is opt in per controller via the io_cpu_adopt fabrics
> option at connect time. wq_unbound takes precedence when set.

I think that for controllers with low queue count this approach would not
work very well. Plus the cpu<->queue mapping by the block layer is not
guaranteed to align with how workloads/VMs are scheduled on the host...

To me the problem statement sounds to me like you are looking to get a 
"private"
controller (set of IO queues) for each VM. Wouldn't a better approach 
would be to pass
a "private controller" connect-param which would for TCP mean that the 
driver would
create a dedicated UNBOUND workqueue which allows setting its cpumask?

Something like: nvme connect -t tcp -a <traddr> --queues_scope=private ?
Then you'd set its IO queues to something like:
echo [cpumask] > /sys/devices/virtual/workqueue/nvmeX-unb-wq/cpumask

Even explicit cpu-groups would be a better approach IMO. This would make
nvme-tcp to override blk_mq_map_queues() and adhere to the passed groups and
try to divide queues between them.

Something like the below example:
nvme connect -t tcp -a <traddr> --cpu-groups=0-63,64-127,128-191,192-255

In this case, nvme-tcp would try to both assign queue->io_cpu within these
ranges, and also assign the cpu<->mapping according to these groups.

Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Saravanan D]

On Sun, 23 Aug 2026 00:20:21 +0300 Sagi Grimberg <sagi@grimberg.me> wrote:
> Can you share a concrete example?
> 1. lscpu
> 2. grep -r ''  /sys/block/nvmeXnY/mq/*/cpu_list
> Also, what would be the VMs cpusets on such a server?

Very grateful for your review and suggestions.

Concrete example from one of these hosts, where there are more cpus than
the io queues NVMe-oF targets offer.

lscpu (trimmed): AMD EPYC 9654, 2 sockets, 2 threads/core, 384 CPUs
  NUMA node0 CPUs: 0-95,192-287
  NUMA node1 CPUs: 96-191,288-383

Each nvme-tcp controller exposes 128 io queues (queue_count 129) against
384 CPUs, so blk_mq_map_queues() folds three CPUs into every hctx. cpu_list for the namespace's path device, around one tenant's boundary:

  hctx67:  6, 7, 198
  hctx68:  8, 9, 200
  hctx69:  10, 11, 202
  hctx75:  22, 23, 214
  hctx76:  24, 25, 216
  hctx113: 199, 201, 203
  hctx114: 205, 207, 209
  hctx115: 211, 213, 215

Two VMs are live on this same shared controller:
  VM A cpuset: 8-23,200-215
  VM B cpuset: 104-191,296-383

VM A owns 200-215, so hctx113's group {199,201,203} has 201 and 203 inside
VM A and 199 outside it. The connect time pick can select 199 as that
queue's io_cpu, so VM A's I/O through hctx113 runs its socket work on 199,
outside its cpuset for its entire lifetime. Adoption will pull it back 
to 201 or 203 when VM A submits I/O.

> In an ideal setting, the nvme controller would support $NR_CPUS worth of
> io queues [...] map cpu-queue with perfect alignment.

Right, and that is the case we cannot reach here. The controller tops out
at 128 io queues while the host has 384 CPUs, so the three to one folding
is unavoidable and some groups end up straddling a tenant boundary.

> I think that for controllers with low queue count this approach would not
> work very well. Plus the cpu<->queue mapping by the block layer is not
> guaranteed to align with how workloads/VMs are scheduled on the host...

Agreed on both. A queue whose cpu group straddles two busy tenants is only
partially addressed, since adoption then selects which tenant hosts that
shared queue rather than removing the sharing. It fixes the common case
where cpu group sits within one VM's cpuset and the case where only one
tenant on the queue is active, and it is strictly better than the connect
time pick, which chooses that queue's io_cpu before any I/O and can land on
any member. Solving the common case also lets the control plane, which owns
the VM cpusets, affinitize each VM's virtio-blk submitter threads away from
CPUs whose mapped queue would land socket work on a neighbor.

> To me the problem statement sounds to me like you are looking to get a
> "private" controller (set of IO queues) for each VM. [...]
> nvme connect -t tcp -a <traddr> --cpu-groups=0-63,64-127,128-191,192-255

A private controller or a cpu-groups partition fits workloads where the CPU
partition is known and stable at connect time. Our production constraint 
does not afford us that luxury. The nvme-tcp controllers are host wide and
shared, the tenant VMs are namespaces under one multipath subsystem, and the
control plane creates, destroys and bin packs VMs continuously on these core
dense compute hosts. A cpu-groups mask fixed at connect would be stale as
soon as the tenant mix changes, and re-partitioning would mean reconnecting
a controller that many live tenants depend on. We also cannot give each
tenant its own controller, because the overhead on the host adds up and we
run the risk of exceeding the target's connection limits. Adopting the
submitting CPU needs no per tenant configuration and follows the tenant
churn on its own, which is why it fits a long lived shared connection whose
tenant set is not known at connect time.

Next the fleet operators who care about VM steal time already steer NIC IRQs,
NAPI to a dedicated housekeeping CPU pool outside the tenant cpusets, so the
common network processing overhead Nilay raised does not run on tenant CPUs.
With that in place the remaining contributor that still lands on a
tenant's CPU is nvme_tcp_io_work, which this patch pins to the submitting
cpu and keeps within the tenant's cpuset.

Thanks,
Saravanan D.