Re: [PATCH] skbuff: Improve the sending efficiency of __skb_send_sock

[Stanislav Fomichev <stfomichev@gmail.com>]

On 06/23, Feng Yang wrote:
> From: Feng Yang <yangfeng@kylinos.cn>
> 
> By aggregating skb data into a bvec array for transmission, when using sockmap to forward large packets,
> what previously required multiple transmissions now only needs a single transmission, which significantly enhances performance.
> For small packets, the performance remains comparable to the original level.
> 
> When using sockmap for forwarding, the average latency for different packet sizes
> after sending 10,000 packets is as follows:
> size	old(us)		new(us)
> 512	56		55
> 1472	58		58
> 1600	106		79
> 3000	145		108
> 5000	182		123
> 
> Signed-off-by: Feng Yang <yangfeng@kylinos.cn>
> ---
>  net/core/skbuff.c | 112 +++++++++++++++++++++-------------------------
>  1 file changed, 52 insertions(+), 60 deletions(-)
> 
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 85fc82f72d26..664443fc9baf 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -3235,84 +3235,75 @@ typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg);
>  static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset,
>  			   int len, sendmsg_func sendmsg, int flags)
>  {
> -	unsigned int orig_len = len;
>  	struct sk_buff *head = skb;
>  	unsigned short fragidx;
> -	int slen, ret;
> +	struct msghdr msg;
> +	struct bio_vec *bvec;
> +	int max_vecs, ret, slen;
> +	int bvec_count = 0;
> +	unsigned int copied = 0;
>  
> -do_frag_list:
> -
> -	/* Deal with head data */
> -	while (offset < skb_headlen(skb) && len) {
> -		struct kvec kv;
> -		struct msghdr msg;
> -
> -		slen = min_t(int, len, skb_headlen(skb) - offset);
> -		kv.iov_base = skb->data + offset;
> -		kv.iov_len = slen;
> -		memset(&msg, 0, sizeof(msg));
> -		msg.msg_flags = MSG_DONTWAIT | flags;
> -
> -		iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &kv, 1, slen);
> -		ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked,
> -				      sendmsg_unlocked, sk, &msg);
> -		if (ret <= 0)
> -			goto error;
> +	max_vecs = skb_shinfo(skb)->nr_frags + 1; // +1 for linear data
> +	if (skb_has_frag_list(skb)) {
> +		struct sk_buff *frag_skb = skb_shinfo(skb)->frag_list;
>  
> -		offset += ret;
> -		len -= ret;
> +		while (frag_skb) {
> +			max_vecs += skb_shinfo(frag_skb)->nr_frags + 1; // +1 for linear data
> +			frag_skb = frag_skb->next;
> +		}
>  	}
>  
> -	/* All the data was skb head? */
> -	if (!len)
> -		goto out;
> +	bvec = kcalloc(max_vecs, sizeof(struct bio_vec), GFP_KERNEL);
> +	if (!bvec)
> +		return -ENOMEM;

Not sure allocating memory here is a good idea. From what I can tell
this function is used by non-sockmap callers as well..