From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD57637646A for ; Sat, 22 Aug 2026 12:18:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787401135; cv=none; b=usfp3hPKAsXEyGY0vIHMFza+tK1VPtJn1KHy+2ZVeq5yEVnFbKGVsIQ2W4Jc1meeRowy1gTbpF00AkR//MQm807Yi5iR3FCtPqdCqLgaLJsr+wwU0LvQISPNOzgeFFBBux2RF5U5gqDgjGWsp5bm3WCQc9OmccL2k4WLiYTph1E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787401135; c=relaxed/simple; bh=X8+6+U1NQIyPcxR2oMkN6UhKnGHcppAztPCNLvq+PXo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=LvSD9tg0eG5TtGknZC1iyFr/ykfWFqm9uJVDn7FoRZFVkTieSOPt3R7P5klAEX6G2XTgew4tAFDxOdkjevZ+EhnTMFdJCIgLniGCJ+CpoPF8txi/XmWSKPH14V2eqCFCvlDFzIywGU2FR83bhnpPRJu8SMdetqH/1bD/ibbRLYE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=sPjJwsjZ; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="sPjJwsjZ" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 67MBVbHb2547654; Sat, 22 Aug 2026 12:18:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=yLAUOu WzZP2XlsWB7vmpKR01zaVGZ9B5JKjM779GxUQ=; b=sPjJwsjZsebXjfsKKSZxCr iuBF4YZs7LcXNuzpDMNuJ99w8itvoFiicNcnvlSdSOJVLMhklImDb4g7F1yfayuW FfpTA8hl2XFmD5aRe9Dqd3pVOs7tqqP2hDlF4X+tk5omPi94qAHJQYMN3mOTRm/1 FqyslRcouep18JtV6pNUtAwVeMi8gkkxRYPQZ6leSonyXqa5EhqvTPxjHRzI1u8T tIbFTPyN50ZHytBwn0CU57aB36cuLfu5So/dY6JeR4LQagUHyIGtY7yRQ357kM4m vN5spfTlGiH4qCo+Rjt2J0BtHid5oW2PeDqrCQph9R3cAwize/cSUisJXWVWTsdw == Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4g73eq9bjy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 22 Aug 2026 12:18:24 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 67MCBluY015613; Sat, 22 Aug 2026 12:18:23 GMT Received: from smtprelay01.wdc07v.mail.ibm.com ([172.16.1.68]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4g33xhsasu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 22 Aug 2026 12:18:23 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay01.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 67MCIMPb46465462 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 22 Aug 2026 12:18:23 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DADE958055; Sat, 22 Aug 2026 12:18:22 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5147F5803F; Sat, 22 Aug 2026 12:18:20 +0000 (GMT) Received: from [9.61.73.62] (unknown [9.61.73.62]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Sat, 22 Aug 2026 12:18:19 +0000 (GMT) Message-ID: Date: Sat, 22 Aug 2026 17:48:17 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu To: Saravanan D Cc: linux-nvme@lists.infradead.org, Keith Busch , Christoph Hellwig , Sagi Grimberg , Jens Axboe , linux-kernel@vger.kernel.org, Daniel Wagner References: <20260820083634.71689-1-saravanand@crusoe.ai> <20260822004932.79147-1-saravanand@crusoe.ai> Content-Language: en-US From: Nilay Shroff In-Reply-To: <20260822004932.79147-1-saravanand@crusoe.ai> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: c7IcwTDV93NIF_R3YnpEnv3h46L8Bpi_ X-Proofpoint-ORIG-GUID: c7IcwTDV93NIF_R3YnpEnv3h46L8Bpi_ X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwODIyMDA5OCBTYWx0ZWRfX8teFXZrRNGkH GZyEduytBrg3/FezYI7FOJrVxuuYZ4S+xTbeAtndSHOVNKNmrOU6sl7nuyKEHDSa3ENQHv/5fuA r9dQnrwwoczfGSCZplcEkNej9qZtCqCBBWGEcaQnxpiCZQUZGnOFq0T/Xv0gg65vqF/Ao4vTXgl Y817MyNKp/XFiCs+IHrdC+90D54dlLkKGRM6pEJoqRWgR7qSyigs3a3eh4ZSlH6qnMhV789IB1S EsHDdXkI4ranpI5N4ycxElKLKtxzM1eXpuCQzHAvsjfnoaKMyiI0nY0U5/bXDzIQUl3vl/EXyLg UGoAYFOks5TInFbMA/eEPf3+d1ppuasyPkVvV7FXmXptDiqE7/fhXhuMft9K80UJKKmfsAAdOrt BonffQkBIjbC6J/EBIp7G7TwbGpkdVsloAHPESh0kbOq+/j8jM8AgNVJ9IRQSUeDnRJv4lv3X5w fqoVnSm6bginx4Ibd0Q== X-Authority-Analysis: v=2.4 cv=QsRuG1yd c=1 sm=1 tr=0 ts=6a899390 cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=IkcTkHD0fZMA:10 a=Sv0fKeRqtYgA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=_UoOXGINWOthsTMRIeoA:9 a=QEXdDO2ut3YA:10 X-Proofpoint-Spam-Info: AW1haW4tMjYwODIyMDA5OCBTYWx0ZWRfX5SES2iKvu1+R qpKrU1Dsrg9tUiWuXTVuXO5UMMzg8fHRFwh4N9N7k/BUsZzBTNxjiMIdBloZ0e3OXzzEo+a7H6w hMvklOWFruqNL3NRyuR6AYwbwEEAxo8= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-08-22_04,2026-08-21_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 adultscore=0 suspectscore=0 priorityscore=1501 impostorscore=0 spamscore=0 lowpriorityscore=0 clxscore=1015 bulkscore=0 phishscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2608220098 On 8/22/26 6:19 AM, Saravanan D wrote: > On Fri, 21 Aug 2026 21:17:49 +0530 Nilay Shroff wrote: >> It seems that here multi tenants shares the same NVMe/TCP controller. >> So if the concern is CPU isolation between tenants, why are multiple >> tenants sharing the same NVMe/TCP controller? Wouldn't a per-tenant >> controller/connection provide better isolation and allow each >> controller's queues to be mapped to the tenant's CPU set? > > The controllers are shared because the tenant VMs' virtio-blk devices > are backed by namespaces under one multipath subsystem the hypervisor host > connects to. With many VMs per host, maintaining a per-tenant controller > is not always feasible because of the overhead on the host and risk of > running into target connection limits. blk-mq spreads any controller's > queues across every online CPU, so nvme_tcp_set_queue_io_cpu() picks io_cpu > from a machine wide map whether the controller is shared or dedicated. > Tying socket work to the submitting CPU will reduce VM steal time in > these deployment scenarios. > Yes, nvme_tcp_set_queue_io_cpu() currently spreads the I/O queues across the online CPUs, so I understand why a shared controller can end up with its queues mapped across CPUs belonging to different VM/tenant cpusets. My point was if we could instead make the queue-to-CPU mapping aware of the tenant's CPU partition when the controller is created. For example, if the hypervisor knows the CPU set associated with a VM, we could pass a CPU-placement hint/cpuset as part of the fabric connection setup and then have nvme_tcp_set_queue_io_cpu() select the queue CPUs from that set rather than from the machine-wide blk-mq CPU map. This would preserve a stable queue-to-CPU mapping while ensuring that the NVMe/TCP socket work for a controller is confined to the tenant's CPU partition. Compared with dynamically adopting the submitting CPU, I think this could have some advantages: - the queue-to-CPU mapping remains stable - it would make CPU/NIC topology tuning such as XPS/RPS and ntuple steering more deterministic - As queues are not moved across cpus, it may provide better cache locality and potentially reduce cross-CPU wakeups/IPIs associated with moving the socket work. There is another aspect I'm wondering about regarding the VM steal-time observation. The io_cpu adoption addresses the execution of nvme_tcp_io_work, but there are still other parts of the receive path such as the NIC RX interrupt or NAPI and subsequent network processing that can execute on CPUs outside the VM's cpuset depending on IRQ/RPS configuration. So I'm not sure whether moving io_cpu to the submitting CPU by itself can guarantee that all NVMe/TCP network processing stays within the tenant's CPU partition. If the objective is strict CPU isolation, perhaps it would be useful to consider the CPU partition as a property of the NVMe/TCP connection and keep the queue/CPU mapping stable within that partition, while separately configuring the NIC IRQ/RPS/XPS/steering to maintain the same locality. But yes my above recommendation would require creating separate controller per tenant/VM. Thanks, --Nilay