Re: [PATCH v4] vhost: support CPU copy for small packets

[Stephen Hemminger <stephen@networkplumber.org>]

On Mon, 29 Aug 2022 08:56:58 +0800
Wenwu Ma <wenwux.ma@intel.com> wrote:

> Offloading small packets to DMA degrades throughput 10%~20%,
> and this is because DMA offloading is not free and DMA is not
> good at processing small packets. In addition, control plane
> packets are usually small, and assign those packets to DMA will
> significantly increase latency, which may cause timeout like
> TCP handshake packets. Therefore, this patch use CPU to perform
> small copies in vhost.
> 
> Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> ---
> v4:
> * fix coding style issues
> v3:
> * compare threshold with entire packet length
> v2:
> * fix CI build error
> ---
>  lib/vhost/vhost.h      |  7 ++--
>  lib/vhost/virtio_net.c | 73 +++++++++++++++++++++++++++++++++---------
>  2 files changed, 62 insertions(+), 18 deletions(-)
> 
> diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
> index 40fac3b7c6..8a7d90f737 100644
> --- a/lib/vhost/vhost.h
> +++ b/lib/vhost/vhost.h
> @@ -142,8 +142,10 @@ struct virtqueue_stats {
>   * iovec
>   */
>  struct vhost_iovec {
> -	void *src_addr;
> -	void *dst_addr;
> +	void *src_iov_addr;
> +	void *dst_iov_addr;
> +	void *src_virt_addr;
> +	void *dst_virt_addr;
>  	size_t len;
>  };
>  
> @@ -155,6 +157,7 @@ struct vhost_iov_iter {
>  	struct vhost_iovec *iov;
>  	/** number of iovec in this iterator */
>  	unsigned long nr_segs;
> +	unsigned long nr_len;
>  };
>  
>  struct async_dma_vchan_info {
> diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
> index 35fa4670fd..cf796183a0 100644
> --- a/lib/vhost/virtio_net.c
> +++ b/lib/vhost/virtio_net.c
> @@ -26,6 +26,8 @@
>  
>  #define MAX_BATCH_LEN 256
>  

> +#define CPU_COPY_THRESHOLD_LEN 256

Good idea.
This heuristic matches what Linux network drivers do to avoid
creating large buffers from small packets.

Patch no longer applies cleanly to main branch (after 2 years).

Also the copy threshold should be configurable via dev_args to allow
for testing and CPU differences.