Re: [RFC PATCH v3 35/35] Documentation: driver-api: ntb: Document remote eDMA transport backend

[Dave Jiang <dave.jiang@intel.com>]

On 12/17/25 8:16 AM, Koichiro Den wrote:
> Add a description of the ntb_transport backend architecture and the new
> remote eDMA backed mode introduced by CONFIG_NTB_TRANSPORT_EDMA and the
> use_remote_edma module parameter.
> 
> Signed-off-by: Koichiro Den <den@valinux.co.jp>
> ---
>  Documentation/driver-api/ntb.rst | 58 ++++++++++++++++++++++++++++++++
>  1 file changed, 58 insertions(+)
> 
> diff --git a/Documentation/driver-api/ntb.rst b/Documentation/driver-api/ntb.rst
> index a49c41383779..eb7b889d17c4 100644
> --- a/Documentation/driver-api/ntb.rst
> +++ b/Documentation/driver-api/ntb.rst
> @@ -132,6 +132,64 @@ Transport queue pair.  Network data is copied between socket buffers and the
>  Transport queue pair buffer.  The Transport client may be used for other things
>  besides Netdev, however no other applications have yet been written.
>  
> +Transport backends
> +~~~~~~~~~~~~~~~~~~
> +
> +The ``ntb_transport`` core driver implements a generic "queue pair"
> +abstraction on top of the memory windows exported by the NTB hardware. Each
> +queue pair has a TX and an RX ring and is used by client drivers such as
> +``ntb_netdev`` to exchange variable sized payloads with the peer.
> +
> +There are currently two ways for ``ntb_transport`` to move payload data
> +between the local system memory and the peer:
> +
> +* The default backend copies data between the caller buffers and the TX/RX
> +  rings in the memory windows using ``memcpy()`` on the local CPU or, when
> +  the ``use_dma`` module parameter is set, a local DMA engine via the
> +  standard dmaengine ``DMA_MEMCPY`` interface.
> +
> +* When ``CONFIG_NTB_TRANSPORT_EDMA`` is enabled in the kernel configuration
> +  and the ``use_remote_edma`` module parameter is set at run time, a second
> +  backend uses a DesignWare eDMA engine that resides on the endpoint side

I would say "embedded DMA device" instead of a specific DesignWare eDMA engine to keep the transport generic. But provide a reference or link to DesignWare eDMA engine as reference.

> +  of the NTB. In this mode the endpoint driver exposes a dedicated peer
> +  memory window that contains the eDMA register block together with a small
> +  control structure and per-channel linked-list rings only for read
> +  channels. The host ioremaps this window and configures a dmaengine
> +  device. The endpoint uses its local eDMA write channels for its TX
> +  transfer, while the host side uses the remote eDMA read channels for its
> +  TX transfer.

Can you provide some more text on the data flow from one host to the other for eDMA vs via host based DMA in the current transport? i.e. currently for a transmit, user data gets copied into an skbuff by the network stack, and then the local host copies it into the ring buffer on the remote host via DMA write (or CPU). And the remote host then copies out of the ring buffer entry to a kernel skbuff and back to user space on the receiver side. How does it now work with eDMA? Also can the mechanism used by eDMA be achieved with a host DMA setup or is the eDMA mechanism specifically tied to the DW hardware design? Would be nice to move the ASCII data flow diagram in the cover to documentation so we don't lose that.

DJ

> +
> +The ``ntb_transport`` core routes queue pair operations (enqueue,
> +completion polling, link bring-up/teardown etc.) through a small
> +backend-ops structure so that both implementations can coexist in the same
> +module without affecting the public queue pair API used by clients. From a
> +client driver's point of view (for example ``ntb_netdev``) the queue pair
> +interface is the same regardless of which backend is active.
> +
> +When ``use_remote_edma`` is not enabled, ``ntb_transport`` behaves as in
> +previous kernels before the optional ``use_remote_edma`` parameter was
> +introduced, and continues to use the shared-memory backend. Existing
> +configurations that do not select the eDMA backend therefore see no
> +behavioural change.
> +
> +In the remote eDMA mode host-to-endpoint notifications are delivered via a
> +dedicated DMA read channel located at the endpoint. In both the default
> +backend mode and the remote eDMA mode, endpoint-to-host notifications are
> +backed by native MSI support on DW EPC, even when ``use_msi=0``.  Because
> +of this, the ``use_msi`` module parameter has no effect when
> +``use_remote_edma=1`` on the host.
> +
> +At a high level, enabling the remote eDMA transport backend requires:
> +
> +* building the kernel with ``CONFIG_NTB_TRANSPORT`` and
> +  ``CONFIG_NTB_TRANSPORT_EDMA`` enabled,
> +* configuring the NTB endpoint so that it exposes a memory window containing
> +  the eDMA register block, descriptor rings and control structure expected by
> +  the helper driver, and
> +* loading ``ntb_transport`` on the host with ``use_remote_edma=1`` so that
> +  the eDMA-backed backend is selected instead of the default shared-memory
> +  backend.
> +
>  NTB Ping Pong Test Client (ntb\_pingpong)
>  -----------------------------------------
>