Re: [RFC PATCH net-next v6 14/15] net: add devmem TCP documentation

[Jakub Kicinski <kuba@kernel.org>]

On Mon,  4 Mar 2024 18:01:49 -0800 Mina Almasry wrote:
> +Intro
> +=====
> +
> +Device memory TCP (devmem TCP) enables receiving data directly into device
> +memory (dmabuf). The feature is currently implemented for TCP sockets.
> +
> +
> +Opportunity
> +-----------
> +
> +A large amount of data transfers have device memory as the source and/or

s/amount/number/

> +destination. Accelerators drastically increased the volume of such transfers.

s/volume/prevalence/

> +Some examples include:
> +
> +- Distributed training, where ML accelerators, such as GPUs on different hosts,
> +  exchange data among them.

s/among them//

> +- Distributed raw block storage applications transfer large amounts of data with
> +  remote SSDs, much of this data does not require host processing.
> +
> +Today, the majority of the Device-to-Device data transfers the network are

"Today" won't age well.

> +implemented as the following low level operations: Device-to-Host copy,
> +Host-to-Host network transfer, and Host-to-Device copy.
> +
> +The implementation is suboptimal, especially for bulk data transfers, and can

/The implementation/The flow involving host copies/

> +put significant strains on system resources such as host memory bandwidth and
> +PCIe bandwidth.
> +
> +Devmem TCP optimizes this use case by implementing socket APIs that enable
> +the user to receive incoming network packets directly into device memory.

> +More Info
> +---------
> +
> +  slides, video
> +    https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html
> +
> +  patchset
> +    [RFC PATCH v3 00/12] Device Memory TCP
> +    https://lore.kernel.org/lkml/20231106024413.2801438-1-almasrymina@google.com/T/

Won't age well? :)

> +Interface
> +=========
> +
> +Example
> +-------
> +
> +tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up
> +the RX path of this API.
> +
> +NIC Setup
> +---------
> +
> +Header split, flow steering, & RSS are required features for devmem TCP.
> +
> +Header split is used to split incoming packets into a header buffer in host
> +memory, and a payload buffer in device memory.
> +
> +Flow steering & RSS are used to ensure that only flows targeting devmem land on
> +RX queue bound to devmem.
> +
> +Enable header split & flow steering:
> +
> +::

You can put the :: at the end of the text, IIRC, like this:

Enable header split & flow steering::

> +
> +	# enable header split (assuming priv-flag)
> +	ethtool --set-priv-flags eth1 enable-header-split on

Olek added the "set" in commit 50d73710715d ("ethtool: add SET for
TCP_DATA_SPLIT ringparam"), no need for the priv flag any more.