[SPDK] RFC: dynamic NVMf/TCP receive vectors for socket RX zero-copy

[Jerome Tollet <jtollet@cisco.com>]

Hi,

I am experimenting with the SPDK socket RX zero-copy review series
27033, 27037, 27048 and 27049 using an architecture where SPDK is embedded
inside VPP as an in-process plugin. The SPDK socket backend uses the VPP
HostStack instead of POSIX sockets.

The initial architecture and BlueField-3 results are described here:
https://medium.com/fd-io-vpp/spdk-inside-vpp-accelerating-nvme-tcp-on-bluefield-3-bc0419048b1e

The current work extends that integration with RX zero-copy.

The VPP HostStack backend can expose a received TCP payload as many separate
segments. With the existing inline limits, fragmented NVMe/TCP payloads can
exceed the 33 request iovecs or 16 PDU iovecs and fall back to copying.

A simple initial workaround expanded the inline arrays to a full 256-entry
receive vector. However, this substantially increases the permanent size of
every request and PDU object, while the current uint8_t iovec count still
limits the usable vector to 255 segments.

I posted three dependent changes:

- 29255 fixes two correctness issues in the copied fallback: it preserves
  the number of bytes already consumed and allocates the materialized buffer
  with SPDK_MALLOC_DMA.
  https://review.spdk.io/c/spdk/spdk/+/29255

- 29257 lets an NVMf transport provide an extended request iovec while
  keeping the existing 33-entry inline array as the default.
  https://review.spdk.io/c/spdk/spdk/+/29257

- 29258 makes NVMf/TCP grow its request, segment-reference and PDU vectors
  geometrically and only when their inline capacity is exceeded. Expanded
  vectors are retained for object reuse, while the copied fallback is
  preserved for allocation failures or the uint8_t iovec-count limit.
  https://review.spdk.io/c/spdk/spdk/+/29258

On BlueField-3, the integration delivered almost 100% of RX payload bytes
directly to SPDK. In a controlled 128 KiB write test, throughput increased
from 3.104 to 4.804 GB/s compared with the copied path.

A fixed-256 versus dynamic-vector comparison showed essentially identical
throughput at 128 KiB and a 1.05% improvement at 1 MiB, while the dynamic
implementation keeps the normal object footprint close to the original
sizes.

I would particularly appreciate feedback on whether the optional extended
request vector belongs in the generic NVMf layer, and whether retaining
expanded allocations for object reuse is the preferred lifecycle.

Best regards,
Jerome