From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from aer-iport-3.cisco.com (aer-iport-3.cisco.com [173.38.203.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93C403B9D9A for ; Fri, 4 Sep 2026 19:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.38.203.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788549720; cv=none; b=YtUnPwv7C+Wd9qCY+4OxDzyGR6ooSiXmBTCAN9J/fhpRuzXi1Pao0PuBbBrk6Jh0tztzNHcQYARV74aEr0wMuEV9DmtAU5Xrux3jmEmOpvDm3jJGk2XvnnuUpEzdT5oPc3cLxl1aa4JIZjbPBggDlirxzs9D+7YviOLrD4/aWy0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788549720; c=relaxed/simple; bh=fEKqOy4nfItoW/ObfAXZF3dep0aULkFXeG910l8PB1k=; h=Date:Message-ID:From:To:Subject:Content-Type; b=YIDbT+vt1OhXCVKITn2fDK2WGyLgWHspnPQpXkiSh6xdsoMrV525Apnp9NHzWoxF2ZR9s5NHEsdufQtSOHy7zP2FzZ0EP0oxVadR+YQzSPO2E72nUYkbjWN2YCGI/IVyXDRvj6AxWUwEjRVibuLcLp3YeNjhxtANqXj0Bby49Og= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (2048-bit key) header.d=cisco.com header.i=@cisco.com header.b=QTErGHaS; arc=none smtp.client-ip=173.38.203.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cisco.com header.i=@cisco.com header.b="QTErGHaS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=2458; q=dns/txt; s=iport01; t=1788549717; x=1789759317; h=date:message-id:from:to:subject: content-transfer-encoding; bh=Z4lBATd5lDRBhznP4/a7SvDdca/LxsNP0UkBX6uDgd0=; b=QTErGHaSHsg5mMhE0UPdcOpNvaxWUmom8+0M7dqL4NI4Z3fSiywOaH0J V8tcCGQV8CeG2TkRz/AbLInexaEu4R2UsTZCG2tC2zhPuM/dHhvXUzEW6 X/Ucim8nq1VpuFeCZ+By9zMIWC0v5dUg50ztYHWV9qy82GXmNoGCkyIJg CFmEDaniVY4wFpCA+ICZfriVRqt65zDmJIq+RR3sKci6yvPs9lwvpL+Lo IEaGZN3G6VFb+mTAX+9X8PxRdSMbYDpy64J1M7Q/6aNac46RBc1aZRsdn uRf5uWG1tCp4wK2xBHjFCexFlYxayFqwKhEYLNDWC01AZKWW0GEMDaiNK Q==; X-CSE-ConnectionGUID: r0KmhGWUTMS3np5jsdkhtQ== X-CSE-MsgGUID: gEd8s1jgSaOHVrgUMe08Cg== X-IPAS-Result: =?us-ascii?q?A0D3AwCVGZtq/85K/pBaglmDS2BDhSCETYsFgiGEMZltg?= =?us-ascii?q?X4PAQEBD0QNBAEBgVEBgykBjg4CJjQJDgECBAMCAwEBAQEBAQEBAQEBAQoBA?= =?us-ascii?q?QUBAQECAQcFgQ4Thk8NkEYPAT02CAImAkovgnuCdAMRwzt6gTKBAYR+2UmBb?= =?us-ascii?q?QaBHy6IXgGKdCcbgUlEgRWDaYJKgUYLdYMOgmoEgQ6BFIEMgXKJB4JxhgEsg?= =?us-ascii?q?R4cA1ksAQ9GExcLBwWBZgOBBm4yHYEjPhcvWBsGBYEdgSeDPyMZNnqBCV6BK?= =?us-ascii?q?ylhEheBCYIIAoJUggECAUlDDgdFUwknSxIBCwdmPTcVGQQCATp7jkkZH4MGD?= =?us-ascii?q?A5vImMTGxoNDitkA5Jps06EKIwilUISHBeXX5MNAZkIjgqWAIU5gWg8K4EuM?= =?us-ascii?q?xoIGxWDI1IZD44rGYNggmTJN0VvAgcCBw4DC4ZGiyOBfQEB?= IronPort-Data: A9a23:ePIQV6n27Vg/ZUr9iWAf8Cjo5gw4JERdPkR7XQ2eYbSJt1+Wr1Gzt xIaXGmAO/iPazb2Ko1xatjn8k8BsZ+GyNBjHAdrpCkzRFtH+JHPbTi7wugcHM8zwunrFh8PA xA2M4GYRCwMZiaC4Errav6+/SEUOZigHtLUEPTDNj16WThqQSIgjQMLs+Mii+aEu/Dha++2k Y20+ZC31GONgWYubDpFs/7b8XuDgdyr0N8mlg1mDRx0lAe2e0k9VPo3Oay3Jn3kdYhYdsbSb /rD1ryw4lTC9B4rDN6/+p6jGqHdauePVeQmoiM+t5mK2nCulARrukoIHKZ0hXNsttm8t4sZJ OOhGnCHYVxB0qXkwIzxWvTDes10FfUuFLTveRBTvSEPpqHLWyOE/hlgMK05FZdA2eN3Ll5zy dkjczYVMxKYi+G23IvuH4GAhux7RCXqFIoSoDRkiDreF/tjGc2FSKTR7tge1zA17ixMNa+CO 4xDNGYpM0iGOUURUrsUIMpWcOOAhGX4dzlVtHqepLE85C7YywkZPL3FbYWLIILXFZ49ckCwq U/Kr3+gAB4hPfu/9yCKz1eh3r/kknauMG4VPPjinhJwu3WI3mAQIBkXTkeg5/24jFOuHd5SN SQpFjEGpKUosUjuRd7nUljg/TiPvwUXXJxbFOhSBByx95c4Kj2xXgAsJgOtovR/3CPqbVTGD mO0ou4= IronPort-HdrOrdr: A9a23:I0JeGasiCnoaD5K6gDgqjXa57skDTtV00zEX/kB9WHVpm6uj9/ xG/c576faaslwssR0b9exoWpPsfZq0z/ccirX5Vo3NYOCJggSVBbAny5f+yDv9HCC73Otc2a B8N5VaMrTLfDtHZQKQ2njcLz7mq+P3kpyVuQ== X-Talos-CUID: 9a23:kFsah29wUSjLN/LE9NSVv1RMOcUlXULg8FLreEmgVGRyRL6VVWbFrQ== X-Talos-MUID: =?us-ascii?q?9a23=3AGOQ57w5I0bDDNPq2WLbnfBBRxoxnzqj+ExkPnq4?= =?us-ascii?q?PnOm6LyNNHjrF3B+4F9o=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.25,262,1779148800"; d="scan'208";a="57626583" Received: from aer-l-core-05.cisco.com ([144.254.74.206]) by aer-iport-3.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 04 Sep 2026 19:20:45 +0000 Received: from localhost (unknown [10.189.108.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aer-l-core-05.cisco.com (Postfix) with ESMTPS id EEA4E180008C6; Fri, 4 Sep 2026 19:20:44 +0000 (GMT) Date: Fri, 04 Sep 2026 21:20:43 +0200 Message-ID: From: Jerome Tollet To: spdk@lists.linux.dev Subject: [SPDK] RFC: dynamic NVMf/TCP receive vectors for socket RX zero-copy Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Outbound-Client-TLS: ANONYMOUS;unknown [10.189.108.183];TLSv1.3;TLS_AES_256_GCM_SHA384;256 X-Outbound-SMTP-Client: 10.189.108.183, [10.189.108.183] X-Outbound-Node: aer-l-core-05.cisco.com Precedence: bulk X-Mailing-List: spdk@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Hi, I am experimenting with the SPDK socket RX zero-copy review series 27033, 27037, 27048 and 27049 using an architecture where SPDK is embedded inside VPP as an in-process plugin. The SPDK socket backend uses the VPP HostStack instead of POSIX sockets. The initial architecture and BlueField-3 results are described here: https://medium.com/fd-io-vpp/spdk-inside-vpp-accelerating-nvme-tcp-on-bluefield-3-bc0419048b1e The current work extends that integration with RX zero-copy. The VPP HostStack backend can expose a received TCP payload as many separate segments. With the existing inline limits, fragmented NVMe/TCP payloads can exceed the 33 request iovecs or 16 PDU iovecs and fall back to copying. A simple initial workaround expanded the inline arrays to a full 256-entry receive vector. However, this substantially increases the permanent size of every request and PDU object, while the current uint8_t iovec count still limits the usable vector to 255 segments. I posted three dependent changes: - 29255 fixes two correctness issues in the copied fallback: it preserves the number of bytes already consumed and allocates the materialized buffer with SPDK_MALLOC_DMA. https://review.spdk.io/c/spdk/spdk/+/29255 - 29257 lets an NVMf transport provide an extended request iovec while keeping the existing 33-entry inline array as the default. https://review.spdk.io/c/spdk/spdk/+/29257 - 29258 makes NVMf/TCP grow its request, segment-reference and PDU vectors geometrically and only when their inline capacity is exceeded. Expanded vectors are retained for object reuse, while the copied fallback is preserved for allocation failures or the uint8_t iovec-count limit. https://review.spdk.io/c/spdk/spdk/+/29258 On BlueField-3, the integration delivered almost 100% of RX payload bytes directly to SPDK. In a controlled 128 KiB write test, throughput increased from 3.104 to 4.804 GB/s compared with the copied path. A fixed-256 versus dynamic-vector comparison showed essentially identical throughput at 128 KiB and a 1.05% improvement at 1 MiB, while the dynamic implementation keeps the normal object footprint close to the original sizes. I would particularly appreciate feedback on whether the optional extended request vector belongs in the generic NVMf layer, and whether retaining expanded allocations for object reuse is the preferred lifecycle. Best regards, Jerome