[PATCH net-next v7 0/3] net: A lightweight zero-copy notification

[PATCH net-next v7 0/3] net: A lightweight zero-copy notification

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

Original title is "net: socket sendmsg MSG_ZEROCOPY_UARG".

Original notification mechanism needs poll + recvmmsg which is not
easy for applcations to accommodate. And, it also incurs unignorable
overhead including extra system calls.

While making maximum reuse of the existing MSG_ZEROCOPY related code,
this patch set introduces a new zerocopy socket notification mechanism.
Users of sendmsg pass a control message as a placeholder for the incoming
notifications. Upon returning, kernel embeds notifications directly into
user arguments passed in. By doing so, we can significantly reduce the
complexity and overhead for managing notifications.

We also have the logic related to copying cmsg to the userspace in sendmsg
generic for any possible uses cases in the future.

Initially, we expect users to pass the user address of the user array
as a data in cmsg, so that the kernel can copy_to_user to this address
directly.

As Willem commented,

> The main design issue with this series is this indirection, rather
> than passing the array of notifications as cmsg.

> This trick circumvents having to deal with compat issues and having to
> figure out copy_to_user in ____sys_sendmsg (as msg_control is an
> in-kernel copy).

> This is quite hacky, from an API design PoV.

> As is passing a pointer, but expecting msg_controllen to hold the
> length not of the pointer, but of the pointed to user buffer.

> I had also hoped for more significant savings. Especially with the
> higher syscall overhead due to meltdown and spectre mitigations vs
> when MSG_ZEROCOPY was introduced and I last tried this optimization.

We solve it by supporting put_cmsg to userspace in sendmsg path starting
from v5.

Changelog:
  v1 -> v2:
    - Reuse errormsg queue in the new notification mechanism,
      users can actually use these two mechanisms in hybrid way
      if they want to do so.
    - Update case SCM_ZC_NOTIFICATION in __sock_cmsg_send
      1. Regardless of 32-bit, 64-bit program, we will always handle
      u64 type user address.
      2. The size of data to copy_to_user is precisely calculated
      in case of kernel stack leak.
    - fix (kbuild-bot)
      1. Add SCM_ZC_NOTIFICATION to arch-specific header files.
      2. header file types.h in include/uapi/linux/socket.h

  v2 -> v3:
    - 1. Users can now pass in the address of the zc_info_elem directly
      with appropriate cmsg_len instead of the ugly user interface. Plus,
      the handler is now compatible with MSG_CMSG_COMPAT and 32-bit
      pointer.
    - 2. Suggested by Willem, another strategy of getting zc info is
      briefly taking the lock of sk_error_queue and move to a private
      list, like net_rx_action. I thought sk_error_queue is protected by
      sock_lock, so that it's impossible for the handling of zc info and
      users recvmsg from the sk_error_queue at the same time.
      However, sk_error_queue is protected by its own lock. I am afraid
      that during the time it is handling the private list, users may
      fail to get other error messages in the queue via recvmsg. Thus,
      I don't implement the splice logic in this version. Any comments?

  v3 -> v4:
    - 1. Change SOCK_ZC_INFO_MAX to 64 to avoid large stack frame size.
    - 2. Fix minor typos.
    - 3. Change cfg_zerocopy from int to enum in msg_zerocopy.c

  v4 -> v5:
    - 1. Passing user address directly to kernel raises concerns about
    ABI. In this version, we support put_cmsg to userspace in TX path
    to solve this problem.

  v5 -> v6:
    - 1. Cleanly copy cmsg to user upon returning of ___sys_sendmsg

  v6 -> v7:
    - 1. Remove flag MSG_CMSG_COPY_TO_USER, use a member in msghdr instead
    - 2. Pass msg to __sock_cmsg_send.
    - 3. sendmsg_copy_cmsg_to_user should be put at the end of
    ____sys_sendmsg to make sure msg_sys->msg_control is a valid pointer.
    - 4. Add struct zc_info to contain the array of zc_info_elem, so that
    the kernel can update the zc_info->size. Another possible solution is
    updating the cmsg_len directly, but it will break for_each_cmsghdr.
    - 5. Update selftest to make cfg_notification_limit have the same
    semantics in both methods for better comparison.

* Performance

We update selftests/net/msg_zerocopy.c to accommodate the new mechanism,
cfg_notification_limit has the same semantics for both methods. Test
results are as follows, we update skb_orphan_frags_rx to the same as
skb_orphan_frags to support zerocopy in the localhost test.

cfg_notification_limit = 1, both method get notifications after 1 calling
of sendmsg. In this case, the new method has around 17% cpu savings in TCP
and 23% cpu savings in UDP.
+----------------------+---------+---------+---------+---------+
| Test Type / Protocol | TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+----------------------+---------+---------+---------+---------+
| ZCopy (MB)           | 7523    | 7706    | 7489    | 7304    |
+----------------------+---------+---------+---------+---------+
| New ZCopy (MB)       | 8834    | 8993    | 9053    | 9228    |
+----------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy    | 117.42% | 116.70% | 120.88% | 126.34% |
+----------------------+---------+---------+---------+---------+

cfg_notification_limit = 32, both get notifications after 32 calling of
sendmsg, which means more chances to coalesce notifications, and less
overhead of poll + recvmsg for the original method. In this case, the new
method has around 7% cpu savings in TCP and slightly better cpu usage in
UDP. In the env of selftest, notifications of TCP are more likely to be
out of order than UDP, it's easier to coalesce more notifications in UDP.
The original method can get one notification with range of 32 in a recvmsg
most of the time. In TCP, most notifications' range is around 2, so the
original method needs around 16 recvmsgs to get notified in one round.
That's the reason for the "New ZCopy / ZCopy" diff in TCP and UDP here.
+----------------------+---------+---------+---------+---------+
| Test Type / Protocol | TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+----------------------+---------+---------+---------+---------+
| ZCopy (MB)           | 8842    | 8735    | 10072   | 9380    |
+----------------------+---------+---------+---------+---------+
| New ZCopy (MB)       | 9366    | 9477    | 10108   | 9385    |
+----------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy    | 106.00% | 108.28% | 100.31% | 100.01% |
+----------------------+---------+---------+---------+---------+

In conclusion, when notification interval is small or notifications are
hard to be coalesced, the new mechanism is highly recommended. Otherwise,
the performance gain from the new mechanism is very limited.

Zijian Zhang (3):
  sock: support copying cmsgs to the user space in sendmsg
  sock: add MSG_ZEROCOPY notification mechanism based on msg_control
  selftests: add MSG_ZEROCOPY msg_control notification test

 arch/alpha/include/uapi/asm/socket.h        |   2 +
 arch/mips/include/uapi/asm/socket.h         |   2 +
 arch/parisc/include/uapi/asm/socket.h       |   2 +
 arch/sparc/include/uapi/asm/socket.h        |   2 +
 include/linux/socket.h                      |   6 ++
 include/net/sock.h                          |   2 +-
 include/uapi/asm-generic/socket.h           |   2 +
 include/uapi/linux/socket.h                 |  13 +++
 net/core/sock.c                             |  52 ++++++++-
 net/ipv4/ip_sockglue.c                      |   2 +-
 net/ipv6/datagram.c                         |   2 +-
 net/socket.c                                |  54 +++++++++-
 tools/testing/selftests/net/msg_zerocopy.c  | 111 ++++++++++++++++++--
 tools/testing/selftests/net/msg_zerocopy.sh |   1 +
 14 files changed, 236 insertions(+), 17 deletions(-)

-- 
2.20.1

[PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

Users can pass msg_control as a placeholder to recvmsg, and get some info
from the kernel upon returning of it, but it's not available for sendmsg.
Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
creates a kernel copy of msg_control and passes that to the callees,
put_cmsg in sendmsg path will write into this kernel buffer.

If users want to get info after returning of sendmsg, they typically have
to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
call overhead. This commit supports copying cmsg from the kernel space to
the user space upon returning of sendmsg to mitigate this overhead.

Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
---
 include/linux/socket.h |  6 +++++
 include/net/sock.h     |  2 +-
 net/core/sock.c        |  6 +++--
 net/ipv4/ip_sockglue.c |  2 +-
 net/ipv6/datagram.c    |  2 +-
 net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
 6 files changed, 62 insertions(+), 10 deletions(-)

diff --git a/include/linux/socket.h b/include/linux/socket.h
index 2a1ff91d1914..75461812a7a3 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -71,6 +71,7 @@ struct msghdr {
 		void __user	*msg_control_user;
 	};
 	bool		msg_control_is_user : 1;
+	bool		msg_control_copy_to_user : 1;
 	bool		msg_get_inq : 1;/* return INQ after receive */
 	unsigned int	msg_flags;	/* flags on received message */
 	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
@@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
 	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
 }
 
+static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
+{
+	return 0;
+}
+
 static inline size_t msg_data_left(struct msghdr *msg)
 {
 	return iov_iter_count(&msg->msg_iter);
diff --git a/include/net/sock.h b/include/net/sock.h
index cce23ac4d514..9c728287d21d 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
 	};
 }
 
-int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
+int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
 		     struct sockcm_cookie *sockc);
 int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
 		   struct sockcm_cookie *sockc);
diff --git a/net/core/sock.c b/net/core/sock.c
index 9abc4fe25953..efb30668dac3 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 }
 EXPORT_SYMBOL(sock_alloc_send_pskb);
 
-int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
+int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
 		     struct sockcm_cookie *sockc)
 {
 	u32 tsflags;
@@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
 	default:
 		return -EINVAL;
 	}
+	if (cmsg_copy_to_user(cmsg))
+		msg->msg_control_copy_to_user = true;
 	return 0;
 }
 EXPORT_SYMBOL(__sock_cmsg_send);
@@ -2881,7 +2883,7 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
 			return -EINVAL;
 		if (cmsg->cmsg_level != SOL_SOCKET)
 			continue;
-		ret = __sock_cmsg_send(sk, cmsg, sockc);
+		ret = __sock_cmsg_send(sk, msg, cmsg, sockc);
 		if (ret)
 			return ret;
 	}
diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c
index cf377377b52d..6360b8ba9c84 100644
--- a/net/ipv4/ip_sockglue.c
+++ b/net/ipv4/ip_sockglue.c
@@ -267,7 +267,7 @@ int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc,
 		}
 #endif
 		if (cmsg->cmsg_level == SOL_SOCKET) {
-			err = __sock_cmsg_send(sk, cmsg, &ipc->sockc);
+			err = __sock_cmsg_send(sk, msg, cmsg, &ipc->sockc);
 			if (err)
 				return err;
 			continue;
diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c
index fff78496803d..c9ae30acf895 100644
--- a/net/ipv6/datagram.c
+++ b/net/ipv6/datagram.c
@@ -777,7 +777,7 @@ int ip6_datagram_send_ctl(struct net *net, struct sock *sk,
 		}
 
 		if (cmsg->cmsg_level == SOL_SOCKET) {
-			err = __sock_cmsg_send(sk, cmsg, &ipc6->sockc);
+			err = __sock_cmsg_send(sk, msg, cmsg, &ipc6->sockc);
 			if (err)
 				return err;
 			continue;
diff --git a/net/socket.c b/net/socket.c
index e416920e9399..6a9c9e24d781 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -2525,8 +2525,43 @@ static int copy_msghdr_from_user(struct msghdr *kmsg,
 	return err < 0 ? err : 0;
 }
 
-static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
-			   unsigned int flags, struct used_address *used_address,
+static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
+				     struct user_msghdr __user *umsg)
+{
+	struct compat_msghdr __user *umsg_compat =
+				(struct compat_msghdr __user *)umsg;
+	unsigned int flags = msg_sys->msg_flags;
+	struct msghdr msg_user = *msg_sys;
+	unsigned long cmsg_ptr;
+	struct cmsghdr *cmsg;
+	int err;
+
+	msg_user.msg_control_is_user = true;
+	msg_user.msg_control_user = umsg->msg_control;
+	cmsg_ptr = (unsigned long)msg_user.msg_control;
+	for_each_cmsghdr(cmsg, msg_sys) {
+		if (!CMSG_OK(msg_sys, cmsg))
+			break;
+		if (cmsg_copy_to_user(cmsg))
+			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
+				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
+	}
+
+	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
+	if (err)
+		return err;
+	if (MSG_CMSG_COMPAT & flags)
+		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
+				 &umsg_compat->msg_controllen);
+	else
+		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
+				 &umsg->msg_controllen);
+	return err;
+}
+
+static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
+			   struct msghdr *msg_sys, unsigned int flags,
+			   struct used_address *used_address,
 			   unsigned int allowed_msghdr_flags)
 {
 	unsigned char ctl[sizeof(struct cmsghdr) + 20]
@@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
 	ssize_t err;
 
 	err = -ENOBUFS;
+	msg_sys->msg_control_copy_to_user = false;
 
 	if (msg_sys->msg_controllen > INT_MAX)
 		goto out;
@@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
 			       used_address->name_len);
 	}
 
+	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
+		ssize_t len = err;
+
+		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
+		if (!err)
+			err = len;
+	}
+
 out_freectl:
 	if (ctl_buf != ctl)
 		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
@@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
 	if (err < 0)
 		return err;
 
-	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
-				allowed_msghdr_flags);
+	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
+			      allowed_msghdr_flags);
 	kfree(iov);
 	return err;
 }
@@ -2648,7 +2692,7 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
 long __sys_sendmsg_sock(struct socket *sock, struct msghdr *msg,
 			unsigned int flags)
 {
-	return ____sys_sendmsg(sock, msg, flags, NULL, 0);
+	return ____sys_sendmsg(sock, NULL, msg, flags, NULL, 0);
 }
 
 long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags,
-- 
2.20.1

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Simon Horman]

+ Dave Miller, Jakub Kicinski, Paolo Abeni, David Ahern, and Jens Axboe

  Please generate the CC list Networking for patches using
  get_maintainer.pl --git-min-percent=25 this.patch
  but omitting LKML.


On Mon, Jul 08, 2024 at 09:04:03PM +0000, zijianzhang@bytedance.com wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
> 
> Users can pass msg_control as a placeholder to recvmsg, and get some info
> from the kernel upon returning of it, but it's not available for sendmsg.
> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
> creates a kernel copy of msg_control and passes that to the callees,
> put_cmsg in sendmsg path will write into this kernel buffer.
> 
> If users want to get info after returning of sendmsg, they typically have
> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> call overhead. This commit supports copying cmsg from the kernel space to
> the user space upon returning of sendmsg to mitigate this overhead.
> 
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>

...

> diff --git a/net/socket.c b/net/socket.c
> index e416920e9399..6a9c9e24d781 100644
> --- a/net/socket.c
> +++ b/net/socket.c
> @@ -2525,8 +2525,43 @@ static int copy_msghdr_from_user(struct msghdr *kmsg,
>  	return err < 0 ? err : 0;
>  }
>  
> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> -			   unsigned int flags, struct used_address *used_address,
> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
> +				     struct user_msghdr __user *umsg)
> +{
> +	struct compat_msghdr __user *umsg_compat =
> +				(struct compat_msghdr __user *)umsg;
> +	unsigned int flags = msg_sys->msg_flags;
> +	struct msghdr msg_user = *msg_sys;
> +	unsigned long cmsg_ptr;
> +	struct cmsghdr *cmsg;
> +	int err;
> +
> +	msg_user.msg_control_is_user = true;
> +	msg_user.msg_control_user = umsg->msg_control;

nit: Sparse seems unhappy about the use of a __user pointer here.

     net/socket.c:2540:37: warning: dereference of noderef expression

> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
> +	for_each_cmsghdr(cmsg, msg_sys) {
> +		if (!CMSG_OK(msg_sys, cmsg))
> +			break;
> +		if (cmsg_copy_to_user(cmsg))
> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
> +	}
> +
> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));

nit: The line above could be trivially line-wrapped so that it is
     no more than 80 columns wide, as is still preferred in Networking code.

     Flagged by: checkpatch.pl --max-line-length=80

> +	if (err)
> +		return err;
> +	if (MSG_CMSG_COMPAT & flags)
> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> +				 &umsg_compat->msg_controllen);
> +	else
> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> +				 &umsg->msg_controllen);
> +	return err;
> +}

...

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

On 7/9/24 2:14 AM, Simon Horman wrote:
> + Dave Miller, Jakub Kicinski, Paolo Abeni, David Ahern, and Jens Axboe
> 
>    Please generate the CC list Networking for patches using
>    get_maintainer.pl --git-min-percent=25 this.patch
>    but omitting LKML.
> 
> 
> On Mon, Jul 08, 2024 at 09:04:03PM +0000, zijianzhang@bytedance.com wrote:
>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>
>> Users can pass msg_control as a placeholder to recvmsg, and get some info
>> from the kernel upon returning of it, but it's not available for sendmsg.
>> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
>> creates a kernel copy of msg_control and passes that to the callees,
>> put_cmsg in sendmsg path will write into this kernel buffer.
>>
>> If users want to get info after returning of sendmsg, they typically have
>> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
>> call overhead. This commit supports copying cmsg from the kernel space to
>> the user space upon returning of sendmsg to mitigate this overhead.
>>
>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> 
> ...
> 
>> diff --git a/net/socket.c b/net/socket.c
>> index e416920e9399..6a9c9e24d781 100644
>> --- a/net/socket.c
>> +++ b/net/socket.c
>> @@ -2525,8 +2525,43 @@ static int copy_msghdr_from_user(struct msghdr *kmsg,
>>   	return err < 0 ? err : 0;
>>   }
>>   
>> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>> -			   unsigned int flags, struct used_address *used_address,
>> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
>> +				     struct user_msghdr __user *umsg)
>> +{
>> +	struct compat_msghdr __user *umsg_compat =
>> +				(struct compat_msghdr __user *)umsg;
>> +	unsigned int flags = msg_sys->msg_flags;
>> +	struct msghdr msg_user = *msg_sys;
>> +	unsigned long cmsg_ptr;
>> +	struct cmsghdr *cmsg;
>> +	int err;
>> +
>> +	msg_user.msg_control_is_user = true;
>> +	msg_user.msg_control_user = umsg->msg_control;
> 
> nit: Sparse seems unhappy about the use of a __user pointer here.
> 
>       net/socket.c:2540:37: warning: dereference of noderef expression
> 
>> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
>> +	for_each_cmsghdr(cmsg, msg_sys) {
>> +		if (!CMSG_OK(msg_sys, cmsg))
>> +			break;
>> +		if (cmsg_copy_to_user(cmsg))
>> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
>> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
>> +	}
>> +
>> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
> 
> nit: The line above could be trivially line-wrapped so that it is
>       no more than 80 columns wide, as is still preferred in Networking code.
> 
>       Flagged by: checkpatch.pl --max-line-length=80
> 
>> +	if (err)
>> +		return err;
>> +	if (MSG_CMSG_COMPAT & flags)
>> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
>> +				 &umsg_compat->msg_controllen);
>> +	else
>> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
>> +				 &umsg->msg_controllen);
>> +	return err;
>> +}
> 
> ...

Thanks for the suggestions, will update in the next version.

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Willem de Bruijn]

zijianzhang@ wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
> 
> Users can pass msg_control as a placeholder to recvmsg, and get some info
> from the kernel upon returning of it, but it's not available for sendmsg.
> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
> creates a kernel copy of msg_control and passes that to the callees,
> put_cmsg in sendmsg path will write into this kernel buffer.
> 
> If users want to get info after returning of sendmsg, they typically have
> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system

nit: error queue or MSG_ERRQUEUE

> call overhead. This commit supports copying cmsg from the kernel space to
> the user space upon returning of sendmsg to mitigate this overhead.
>
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>

Overall this approach follows what I had in mind, thanks.

Looking forward to the discussion with a wider audience at netdevconf
next week.

> ---
>  include/linux/socket.h |  6 +++++
>  include/net/sock.h     |  2 +-
>  net/core/sock.c        |  6 +++--
>  net/ipv4/ip_sockglue.c |  2 +-
>  net/ipv6/datagram.c    |  2 +-
>  net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
>  6 files changed, 62 insertions(+), 10 deletions(-)
> 
> diff --git a/include/linux/socket.h b/include/linux/socket.h
> index 2a1ff91d1914..75461812a7a3 100644
> --- a/include/linux/socket.h
> +++ b/include/linux/socket.h
> @@ -71,6 +71,7 @@ struct msghdr {
>  		void __user	*msg_control_user;
>  	};
>  	bool		msg_control_is_user : 1;
> +	bool		msg_control_copy_to_user : 1;
>  	bool		msg_get_inq : 1;/* return INQ after receive */
>  	unsigned int	msg_flags;	/* flags on received message */
>  	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
> @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
>  	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
>  }
>  
> +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
> +{
> +	return 0;
> +}
> +
>  static inline size_t msg_data_left(struct msghdr *msg)
>  {
>  	return iov_iter_count(&msg->msg_iter);
> diff --git a/include/net/sock.h b/include/net/sock.h
> index cce23ac4d514..9c728287d21d 100644
> --- a/include/net/sock.h
> +++ b/include/net/sock.h
> @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
>  	};
>  }
>  
> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>  		     struct sockcm_cookie *sockc);
>  int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
>  		   struct sockcm_cookie *sockc);
> diff --git a/net/core/sock.c b/net/core/sock.c
> index 9abc4fe25953..efb30668dac3 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
>  }
>  EXPORT_SYMBOL(sock_alloc_send_pskb);
>  
> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>  		     struct sockcm_cookie *sockc)
>  {
>  	u32 tsflags;
> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>  	default:
>  		return -EINVAL;
>  	}
> +	if (cmsg_copy_to_user(cmsg))
> +		msg->msg_control_copy_to_user = true;

This seems a bit roundabout.

Just have case SCM_ZC_NOTIFICATION set this bit directly?

>  	return 0;
>  }

> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> -			   unsigned int flags, struct used_address *used_address,
> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
> +				     struct user_msghdr __user *umsg)
> +{
> +	struct compat_msghdr __user *umsg_compat =
> +				(struct compat_msghdr __user *)umsg;
> +	unsigned int flags = msg_sys->msg_flags;
> +	struct msghdr msg_user = *msg_sys;
> +	unsigned long cmsg_ptr;
> +	struct cmsghdr *cmsg;
> +	int err;
> +
> +	msg_user.msg_control_is_user = true;
> +	msg_user.msg_control_user = umsg->msg_control;
> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
> +	for_each_cmsghdr(cmsg, msg_sys) {
> +		if (!CMSG_OK(msg_sys, cmsg))
> +			break;
> +		if (cmsg_copy_to_user(cmsg))
> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
> +	}
> +
> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
> +	if (err)
> +		return err;
> +	if (MSG_CMSG_COMPAT & flags)
> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> +				 &umsg_compat->msg_controllen);
> +	else
> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> +				 &umsg->msg_controllen);
> +	return err;
> +}
> +
> +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
> +			   struct msghdr *msg_sys, unsigned int flags,
> +			   struct used_address *used_address,
>  			   unsigned int allowed_msghdr_flags)
>  {
>  	unsigned char ctl[sizeof(struct cmsghdr) + 20]
> @@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>  	ssize_t err;
>  
>  	err = -ENOBUFS;
> +	msg_sys->msg_control_copy_to_user = false;
>  
>  	if (msg_sys->msg_controllen > INT_MAX)
>  		goto out;
> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>  			       used_address->name_len);
>  	}
>  
> +	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
> +		ssize_t len = err;
> +
> +		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
> +		if (!err)
> +			err = len;
> +	}
> +

The main issue is adding the above initialization and this branch in
the hot path, adding a minor cost to every other send call only for
this use case (and potentially tx timestamps eventually).

>  out_freectl:
>  	if (ctl_buf != ctl)
>  		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
> @@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>  	if (err < 0)
>  		return err;
>  
> -	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
> -				allowed_msghdr_flags);
> +	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
> +			      allowed_msghdr_flags);

Does it make more sense to do the copy_to_user here, so as not to have to plumb
msg down to the callee?
>  	kfree(iov);
>  	return err;
>  }
> @@ -2648,7 +2692,7 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>  long __sys_sendmsg_sock(struct socket *sock, struct msghdr *msg,
>  			unsigned int flags)
>  {
> -	return ____sys_sendmsg(sock, msg, flags, NULL, 0);
> +	return ____sys_sendmsg(sock, NULL, msg, flags, NULL, 0);
>  }
>  
>  long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags,
> -- 
> 2.20.1
> 

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

On 7/9/24 9:40 AM, Willem de Bruijn wrote:
> zijianzhang@ wrote:
>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>
>> Users can pass msg_control as a placeholder to recvmsg, and get some info
>> from the kernel upon returning of it, but it's not available for sendmsg.
>> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
>> creates a kernel copy of msg_control and passes that to the callees,
>> put_cmsg in sendmsg path will write into this kernel buffer.
>>
>> If users want to get info after returning of sendmsg, they typically have
>> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> 
> nit: error queue or MSG_ERRQUEUE
> 
>> call overhead. This commit supports copying cmsg from the kernel space to
>> the user space upon returning of sendmsg to mitigate this overhead.
>>
>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> 
> Overall this approach follows what I had in mind, thanks.
> 
> Looking forward to the discussion with a wider audience at netdevconf
> next week.
> 

No problem, see you next week ;)

>> ---
>>   include/linux/socket.h |  6 +++++
>>   include/net/sock.h     |  2 +-
>>   net/core/sock.c        |  6 +++--
>>   net/ipv4/ip_sockglue.c |  2 +-
>>   net/ipv6/datagram.c    |  2 +-
>>   net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
>>   6 files changed, 62 insertions(+), 10 deletions(-)
>>
>> diff --git a/include/linux/socket.h b/include/linux/socket.h
>> index 2a1ff91d1914..75461812a7a3 100644
>> --- a/include/linux/socket.h
>> +++ b/include/linux/socket.h
>> @@ -71,6 +71,7 @@ struct msghdr {
>>   		void __user	*msg_control_user;
>>   	};
>>   	bool		msg_control_is_user : 1;
>> +	bool		msg_control_copy_to_user : 1;
>>   	bool		msg_get_inq : 1;/* return INQ after receive */
>>   	unsigned int	msg_flags;	/* flags on received message */
>>   	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
>> @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
>>   	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
>>   }
>>   
>> +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
>> +{
>> +	return 0;
>> +}
>> +
>>   static inline size_t msg_data_left(struct msghdr *msg)
>>   {
>>   	return iov_iter_count(&msg->msg_iter);
>> diff --git a/include/net/sock.h b/include/net/sock.h
>> index cce23ac4d514..9c728287d21d 100644
>> --- a/include/net/sock.h
>> +++ b/include/net/sock.h
>> @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
>>   	};
>>   }
>>   
>> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>>   		     struct sockcm_cookie *sockc);
>>   int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
>>   		   struct sockcm_cookie *sockc);
>> diff --git a/net/core/sock.c b/net/core/sock.c
>> index 9abc4fe25953..efb30668dac3 100644
>> --- a/net/core/sock.c
>> +++ b/net/core/sock.c
>> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
>>   }
>>   EXPORT_SYMBOL(sock_alloc_send_pskb);
>>   
>> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>>   		     struct sockcm_cookie *sockc)
>>   {
>>   	u32 tsflags;
>> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>>   	default:
>>   		return -EINVAL;
>>   	}
>> +	if (cmsg_copy_to_user(cmsg))
>> +		msg->msg_control_copy_to_user = true;
> 
> This seems a bit roundabout.
> 
> Just have case SCM_ZC_NOTIFICATION set this bit directly?

If I directly set this bit in SCM_ZC_... and delete this if code block,
I may have to add "msg" argument to __sock_cmsg_send in the second
commit, because if I still keep it in this commit, there will be an
"unused argument" warning.

However, I think the change to __sock_cmsg_send function declaration is
generic, so I would like to make it in the first commit, but it is truly
a bit roundabout. Not sure which way is better?

>>   	return 0;
>>   }
> 
>> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>> -			   unsigned int flags, struct used_address *used_address,
>> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
>> +				     struct user_msghdr __user *umsg)
>> +{
>> +	struct compat_msghdr __user *umsg_compat =
>> +				(struct compat_msghdr __user *)umsg;
>> +	unsigned int flags = msg_sys->msg_flags;
>> +	struct msghdr msg_user = *msg_sys;
>> +	unsigned long cmsg_ptr;
>> +	struct cmsghdr *cmsg;
>> +	int err;
>> +
>> +	msg_user.msg_control_is_user = true;
>> +	msg_user.msg_control_user = umsg->msg_control;
>> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
>> +	for_each_cmsghdr(cmsg, msg_sys) {
>> +		if (!CMSG_OK(msg_sys, cmsg))
>> +			break;
>> +		if (cmsg_copy_to_user(cmsg))
>> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
>> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
>> +	}
>> +
>> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
>> +	if (err)
>> +		return err;
>> +	if (MSG_CMSG_COMPAT & flags)
>> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
>> +				 &umsg_compat->msg_controllen);
>> +	else
>> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
>> +				 &umsg->msg_controllen);
>> +	return err;
>> +}
>> +
>> +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>> +			   struct msghdr *msg_sys, unsigned int flags,
>> +			   struct used_address *used_address,
>>   			   unsigned int allowed_msghdr_flags)
>>   {
>>   	unsigned char ctl[sizeof(struct cmsghdr) + 20]
>> @@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>>   	ssize_t err;
>>   
>>   	err = -ENOBUFS;
>> +	msg_sys->msg_control_copy_to_user = false;
>>   
>>   	if (msg_sys->msg_controllen > INT_MAX)
>>   		goto out;
>> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>>   			       used_address->name_len);
>>   	}
>>   
>> +	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
>> +		ssize_t len = err;
>> +
>> +		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
>> +		if (!err)
>> +			err = len;
>> +	}
>> +
> 
> The main issue is adding the above initialization and this branch in
> the hot path, adding a minor cost to every other send call only for
> this use case (and potentially tx timestamps eventually).
> 
>>   out_freectl:
>>   	if (ctl_buf != ctl)
>>   		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
>> @@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>>   	if (err < 0)
>>   		return err;
>>   
>> -	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
>> -				allowed_msghdr_flags);
>> +	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
>> +			      allowed_msghdr_flags);
> 
> Does it make more sense to do the copy_to_user here, so as not to have to plumb
> msg down to the callee?

I did this in the previous patchset. The problem is that the msg_control
of msg_sys is either a stack pointer or kmalloc-ed pointer (in
____sys_sendmsg), after returning of it, the msg_control of msg_sys is
either invalid or freed. I may have to do the copy_to_user at the end of
____sys_sendmsg.

>>   	kfree(iov);
>>   	return err;
>>   }
>> @@ -2648,7 +2692,7 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>>   long __sys_sendmsg_sock(struct socket *sock, struct msghdr *msg,
>>   			unsigned int flags)
>>   {
>> -	return ____sys_sendmsg(sock, msg, flags, NULL, 0);
>> +	return ____sys_sendmsg(sock, NULL, msg, flags, NULL, 0);
>>   }
>>   
>>   long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags,
>> -- 
>> 2.20.1
>>
> 
> 

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Willem de Bruijn]

Zijian Zhang wrote:
> On 7/9/24 9:40 AM, Willem de Bruijn wrote:
> > zijianzhang@ wrote:
> >> From: Zijian Zhang <zijianzhang@bytedance.com>
> >>
> >> Users can pass msg_control as a placeholder to recvmsg, and get some info
> >> from the kernel upon returning of it, but it's not available for sendmsg.
> >> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
> >> creates a kernel copy of msg_control and passes that to the callees,
> >> put_cmsg in sendmsg path will write into this kernel buffer.
> >>
> >> If users want to get info after returning of sendmsg, they typically have
> >> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> > 
> > nit: error queue or MSG_ERRQUEUE
> > 
> >> call overhead. This commit supports copying cmsg from the kernel space to
> >> the user space upon returning of sendmsg to mitigate this overhead.
> >>
> >> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> >> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> > 
> > Overall this approach follows what I had in mind, thanks.
> > 
> > Looking forward to the discussion with a wider audience at netdevconf
> > next week.
> > 
> 
> No problem, see you next week ;)
> 
> >> ---
> >>   include/linux/socket.h |  6 +++++
> >>   include/net/sock.h     |  2 +-
> >>   net/core/sock.c        |  6 +++--
> >>   net/ipv4/ip_sockglue.c |  2 +-
> >>   net/ipv6/datagram.c    |  2 +-
> >>   net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
> >>   6 files changed, 62 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/include/linux/socket.h b/include/linux/socket.h
> >> index 2a1ff91d1914..75461812a7a3 100644
> >> --- a/include/linux/socket.h
> >> +++ b/include/linux/socket.h
> >> @@ -71,6 +71,7 @@ struct msghdr {
> >>   		void __user	*msg_control_user;
> >>   	};
> >>   	bool		msg_control_is_user : 1;
> >> +	bool		msg_control_copy_to_user : 1;
> >>   	bool		msg_get_inq : 1;/* return INQ after receive */
> >>   	unsigned int	msg_flags;	/* flags on received message */
> >>   	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
> >> @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
> >>   	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
> >>   }
> >>   
> >> +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
> >> +{
> >> +	return 0;
> >> +}
> >> +
> >>   static inline size_t msg_data_left(struct msghdr *msg)
> >>   {
> >>   	return iov_iter_count(&msg->msg_iter);
> >> diff --git a/include/net/sock.h b/include/net/sock.h
> >> index cce23ac4d514..9c728287d21d 100644
> >> --- a/include/net/sock.h
> >> +++ b/include/net/sock.h
> >> @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
> >>   	};
> >>   }
> >>   
> >> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> >> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
> >>   		     struct sockcm_cookie *sockc);
> >>   int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
> >>   		   struct sockcm_cookie *sockc);
> >> diff --git a/net/core/sock.c b/net/core/sock.c
> >> index 9abc4fe25953..efb30668dac3 100644
> >> --- a/net/core/sock.c
> >> +++ b/net/core/sock.c
> >> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
> >>   }
> >>   EXPORT_SYMBOL(sock_alloc_send_pskb);
> >>   
> >> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> >> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
> >>   		     struct sockcm_cookie *sockc)
> >>   {
> >>   	u32 tsflags;
> >> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> >>   	default:
> >>   		return -EINVAL;
> >>   	}
> >> +	if (cmsg_copy_to_user(cmsg))
> >> +		msg->msg_control_copy_to_user = true;
> > 
> > This seems a bit roundabout.
> > 
> > Just have case SCM_ZC_NOTIFICATION set this bit directly?
> 
> If I directly set this bit in SCM_ZC_... and delete this if code block,
> I may have to add "msg" argument to __sock_cmsg_send in the second
> commit, because if I still keep it in this commit, there will be an
> "unused argument" warning.
> 
> However, I think the change to __sock_cmsg_send function declaration is
> generic, so I would like to make it in the first commit, but it is truly
> a bit roundabout. Not sure which way is better?

A temporary __attribute__((unused))
 
> >>   	return 0;
> >>   }
> > 
> >> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >> -			   unsigned int flags, struct used_address *used_address,
> >> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
> >> +				     struct user_msghdr __user *umsg)
> >> +{
> >> +	struct compat_msghdr __user *umsg_compat =
> >> +				(struct compat_msghdr __user *)umsg;
> >> +	unsigned int flags = msg_sys->msg_flags;
> >> +	struct msghdr msg_user = *msg_sys;
> >> +	unsigned long cmsg_ptr;
> >> +	struct cmsghdr *cmsg;
> >> +	int err;
> >> +
> >> +	msg_user.msg_control_is_user = true;
> >> +	msg_user.msg_control_user = umsg->msg_control;
> >> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
> >> +	for_each_cmsghdr(cmsg, msg_sys) {
> >> +		if (!CMSG_OK(msg_sys, cmsg))
> >> +			break;
> >> +		if (cmsg_copy_to_user(cmsg))
> >> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
> >> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
> >> +	}
> >> +
> >> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
> >> +	if (err)
> >> +		return err;
> >> +	if (MSG_CMSG_COMPAT & flags)
> >> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> >> +				 &umsg_compat->msg_controllen);
> >> +	else
> >> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> >> +				 &umsg->msg_controllen);
> >> +	return err;
> >> +}
> >> +
> >> +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
> >> +			   struct msghdr *msg_sys, unsigned int flags,
> >> +			   struct used_address *used_address,
> >>   			   unsigned int allowed_msghdr_flags)
> >>   {
> >>   	unsigned char ctl[sizeof(struct cmsghdr) + 20]
> >> @@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >>   	ssize_t err;
> >>   
> >>   	err = -ENOBUFS;
> >> +	msg_sys->msg_control_copy_to_user = false;
> >>   
> >>   	if (msg_sys->msg_controllen > INT_MAX)
> >>   		goto out;
> >> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >>   			       used_address->name_len);
> >>   	}
> >>   
> >> +	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
> >> +		ssize_t len = err;
> >> +
> >> +		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
> >> +		if (!err)
> >> +			err = len;
> >> +	}
> >> +
> > 
> > The main issue is adding the above initialization and this branch in
> > the hot path, adding a minor cost to every other send call only for
> > this use case (and potentially tx timestamps eventually).
> > 
> >>   out_freectl:
> >>   	if (ctl_buf != ctl)
> >>   		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
> >> @@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
> >>   	if (err < 0)
> >>   		return err;
> >>   
> >> -	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
> >> -				allowed_msghdr_flags);
> >> +	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
> >> +			      allowed_msghdr_flags);
> > 
> > Does it make more sense to do the copy_to_user here, so as not to have to plumb
> > msg down to the callee?
> 
> I did this in the previous patchset. The problem is that the msg_control
> of msg_sys is either a stack pointer or kmalloc-ed pointer (in
> ____sys_sendmsg), after returning of it, the msg_control of msg_sys is
> either invalid or freed. I may have to do the copy_to_user at the end of
> ____sys_sendmsg.

I see. Ack.

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

On 7/9/24 9:40 AM, Willem de Bruijn wrote:
> zijianzhang@ wrote:
>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>
>> Users can pass msg_control as a placeholder to recvmsg, and get some info
>> from the kernel upon returning of it, but it's not available for sendmsg.
>> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
>> creates a kernel copy of msg_control and passes that to the callees,
>> put_cmsg in sendmsg path will write into this kernel buffer.
>>
>> If users want to get info after returning of sendmsg, they typically have
>> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> 
> nit: error queue or MSG_ERRQUEUE
> 
>> call overhead. This commit supports copying cmsg from the kernel space to
>> the user space upon returning of sendmsg to mitigate this overhead.
>>
>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> 
> Overall this approach follows what I had in mind, thanks.
> 
> Looking forward to the discussion with a wider audience at netdevconf
> next week.


After wider exposure to netdev, besides the comments in this email
series, I want to align the next step with you :)

Shall I also make this a config and add conditional compilation in the
hot path？

>> ---
>>   include/linux/socket.h |  6 +++++
>>   include/net/sock.h     |  2 +-
>>   net/core/sock.c        |  6 +++--
>>   net/ipv4/ip_sockglue.c |  2 +-
>>   net/ipv6/datagram.c    |  2 +-
>>   net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
>>   6 files changed, 62 insertions(+), 10 deletions(-)
>>

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Willem de Bruijn]

Zijian Zhang wrote:
> On 7/9/24 9:40 AM, Willem de Bruijn wrote:
> > zijianzhang@ wrote:
> >> From: Zijian Zhang <zijianzhang@bytedance.com>
> >>
> >> Users can pass msg_control as a placeholder to recvmsg, and get some info
> >> from the kernel upon returning of it, but it's not available for sendmsg.
> >> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
> >> creates a kernel copy of msg_control and passes that to the callees,
> >> put_cmsg in sendmsg path will write into this kernel buffer.
> >>
> >> If users want to get info after returning of sendmsg, they typically have
> >> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> > 
> > nit: error queue or MSG_ERRQUEUE
> > 
> >> call overhead. This commit supports copying cmsg from the kernel space to
> >> the user space upon returning of sendmsg to mitigate this overhead.
> >>
> >> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> >> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> > 
> > Overall this approach follows what I had in mind, thanks.
> > 
> > Looking forward to the discussion with a wider audience at netdevconf
> > next week.
> 
> 
> After wider exposure to netdev, besides the comments in this email
> series, I want to align the next step with you :)
> 
> Shall I also make this a config and add conditional compilation in the
> hot path？

At netdev there appeared to be some support for your original approach
of the application passing a user address as CMSG_DATA and the kernel
writing directly there.

That has the benefit of no modifications to net/socket.c and lower
overhead.

But there evidently hasn't been much other feedback on either approach.
Since this is an ABI change, SubmittingPatches suggests "User-space
API changes should also be copied to linux-api@vger.kernel.org." That
might give you more opinions, and is probably a good idea for
something this invasive.

If you choose to go with the current approach, a static_branch in
____sys_sendmsg would make the branch a noop in the common case.
Could be enabled on first setsockopt SO_ZEROCOPY. And never
disabled: no need for refcounting it.

Either way, no need for a CONFIG. Distros ship with defaults, so that
is what matters. And you would not want this default off.

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

On 7/24/24 8:08 PM, Willem de Bruijn wrote:
> Zijian Zhang wrote:
>> On 7/9/24 9:40 AM, Willem de Bruijn wrote:
>>> zijianzhang@ wrote:
>>>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>>>
>>>> Users can pass msg_control as a placeholder to recvmsg, and get some info
>>>> from the kernel upon returning of it, but it's not available for sendmsg.
>>>> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
>>>> creates a kernel copy of msg_control and passes that to the callees,
>>>> put_cmsg in sendmsg path will write into this kernel buffer.
>>>>
>>>> If users want to get info after returning of sendmsg, they typically have
>>>> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
>>>
>>> nit: error queue or MSG_ERRQUEUE
>>>
>>>> call overhead. This commit supports copying cmsg from the kernel space to
>>>> the user space upon returning of sendmsg to mitigate this overhead.
>>>>
>>>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>>>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
>>>
>>> Overall this approach follows what I had in mind, thanks.
>>>
>>> Looking forward to the discussion with a wider audience at netdevconf
>>> next week.
>>
>>
>> After wider exposure to netdev, besides the comments in this email
>> series, I want to align the next step with you :)
>>
>> Shall I also make this a config and add conditional compilation in the
>> hot path？
> 
> At netdev there appeared to be some support for your original approach
> of the application passing a user address as CMSG_DATA and the kernel
> writing directly there.
> 
> That has the benefit of no modifications to net/socket.c and lower
> overhead.
> 
> But there evidently hasn't been much other feedback on either approach.
> Since this is an ABI change, SubmittingPatches suggests "User-space
> API changes should also be copied to linux-api@vger.kernel.org." That
> might give you more opinions, and is probably a good idea for
> something this invasive.
> 
> If you choose to go with the current approach, a static_branch in
> ____sys_sendmsg would make the branch a noop in the common case.
> Could be enabled on first setsockopt SO_ZEROCOPY. And never
> disabled: no need for refcounting it.
> 
> Either way, no need for a CONFIG. Distros ship with defaults, so that
> is what matters. And you would not want this default off.
> 

Agree, the ABI change should be the main concern. I think I will go with
the current method firstly, and the original one as backup.

Thanks for the quick reply and clarification!

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Mina Almasry]

On Mon, Jul 08, 2024 at 09:04:03PM +0000, zijianzhang@bytedance.com wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
>
> Users can pass msg_control as a placeholder to recvmsg, and get some info
> from the kernel upon returning of it, but it's not available for sendmsg.
> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
> creates a kernel copy of msg_control and passes that to the callees,
> put_cmsg in sendmsg path will write into this kernel buffer.
>
> If users want to get info after returning of sendmsg, they typically have
> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> call overhead. This commit supports copying cmsg from the kernel space to
> the user space upon returning of sendmsg to mitigate this overhead.
>
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> ---
>  include/linux/socket.h |  6 +++++
>  include/net/sock.h     |  2 +-
>  net/core/sock.c        |  6 +++--
>  net/ipv4/ip_sockglue.c |  2 +-
>  net/ipv6/datagram.c    |  2 +-
>  net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
>  6 files changed, 62 insertions(+), 10 deletions(-)
>
> diff --git a/include/linux/socket.h b/include/linux/socket.h
> index 2a1ff91d1914..75461812a7a3 100644
> --- a/include/linux/socket.h
> +++ b/include/linux/socket.h
> @@ -71,6 +71,7 @@ struct msghdr {
>  		void __user	*msg_control_user;
>  	};
>  	bool		msg_control_is_user : 1;
> +	bool		msg_control_copy_to_user : 1;

Please add some docs explaining what this does if possible. From reading the
code, it seems if this is true then we should copy cmsg to user. Not sure where
or how it's set though.

>  	bool		msg_get_inq : 1;/* return INQ after receive */
>  	unsigned int	msg_flags;	/* flags on received message */
>  	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
> @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
>  	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
>  }
>
> +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
> +{
> +	return 0;
> +}
> +
>  static inline size_t msg_data_left(struct msghdr *msg)
>  {
>  	return iov_iter_count(&msg->msg_iter);
> diff --git a/include/net/sock.h b/include/net/sock.h
> index cce23ac4d514..9c728287d21d 100644
> --- a/include/net/sock.h
> +++ b/include/net/sock.h
> @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
>  	};
>  }
>
> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>  		     struct sockcm_cookie *sockc);
>  int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
>  		   struct sockcm_cookie *sockc);
> diff --git a/net/core/sock.c b/net/core/sock.c
> index 9abc4fe25953..efb30668dac3 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
>  }
>  EXPORT_SYMBOL(sock_alloc_send_pskb);
>
> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>  		     struct sockcm_cookie *sockc)
>  {
>  	u32 tsflags;
> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>  	default:
>  		return -EINVAL;
>  	}
> +	if (cmsg_copy_to_user(cmsg))
> +		msg->msg_control_copy_to_user = true;
>  	return 0;
>  }
>  EXPORT_SYMBOL(__sock_cmsg_send);
> @@ -2881,7 +2883,7 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
>  			return -EINVAL;
>  		if (cmsg->cmsg_level != SOL_SOCKET)
>  			continue;
> -		ret = __sock_cmsg_send(sk, cmsg, sockc);
> +		ret = __sock_cmsg_send(sk, msg, cmsg, sockc);
>  		if (ret)
>  			return ret;
>  	}
> diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c
> index cf377377b52d..6360b8ba9c84 100644
> --- a/net/ipv4/ip_sockglue.c
> +++ b/net/ipv4/ip_sockglue.c
> @@ -267,7 +267,7 @@ int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc,
>  		}
>  #endif
>  		if (cmsg->cmsg_level == SOL_SOCKET) {
> -			err = __sock_cmsg_send(sk, cmsg, &ipc->sockc);
> +			err = __sock_cmsg_send(sk, msg, cmsg, &ipc->sockc);
>  			if (err)
>  				return err;
>  			continue;
> diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c
> index fff78496803d..c9ae30acf895 100644
> --- a/net/ipv6/datagram.c
> +++ b/net/ipv6/datagram.c
> @@ -777,7 +777,7 @@ int ip6_datagram_send_ctl(struct net *net, struct sock *sk,
>  		}
>
>  		if (cmsg->cmsg_level == SOL_SOCKET) {
> -			err = __sock_cmsg_send(sk, cmsg, &ipc6->sockc);
> +			err = __sock_cmsg_send(sk, msg, cmsg, &ipc6->sockc);
>  			if (err)
>  				return err;
>  			continue;
> diff --git a/net/socket.c b/net/socket.c
> index e416920e9399..6a9c9e24d781 100644
> --- a/net/socket.c
> +++ b/net/socket.c
> @@ -2525,8 +2525,43 @@ static int copy_msghdr_from_user(struct msghdr *kmsg,
>  	return err < 0 ? err : 0;
>  }
>
> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> -			   unsigned int flags, struct used_address *used_address,
> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
> +				     struct user_msghdr __user *umsg)
> +{
> +	struct compat_msghdr __user *umsg_compat =
> +				(struct compat_msghdr __user *)umsg;
> +	unsigned int flags = msg_sys->msg_flags;
> +	struct msghdr msg_user = *msg_sys;
> +	unsigned long cmsg_ptr;
> +	struct cmsghdr *cmsg;
> +	int err;
> +
> +	msg_user.msg_control_is_user = true;
> +	msg_user.msg_control_user = umsg->msg_control;
> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
> +	for_each_cmsghdr(cmsg, msg_sys) {
> +		if (!CMSG_OK(msg_sys, cmsg))
> +			break;
> +		if (cmsg_copy_to_user(cmsg))
> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));

put_cmsg() can fail as far as I can tell. Any reason we don't have to check for
failure here?

What happens when these failures happen. Do we end up putting the ZC
notification later, or is the zc notification lost forever because we did not
detect the failure to put_cmsg() it?

> +	}
> +
> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
> +	if (err)
> +		return err;
> +	if (MSG_CMSG_COMPAT & flags)
> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> +				 &umsg_compat->msg_controllen);
> +	else
> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> +				 &umsg->msg_controllen);
> +	return err;
> +}
> +
> +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
> +			   struct msghdr *msg_sys, unsigned int flags,
> +			   struct used_address *used_address,
>  			   unsigned int allowed_msghdr_flags)
>  {
>  	unsigned char ctl[sizeof(struct cmsghdr) + 20]
> @@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>  	ssize_t err;
>
>  	err = -ENOBUFS;
> +	msg_sys->msg_control_copy_to_user = false;


This may be a lack of knowledge on my part, but i'm very confused that
msg_control_copy_to_user is set to false here, and then checked below, and it's
not touched in between. How could it evaluate to true below? Is it because something
overwrites the value in msg_sys between this set and the check?

If something is overwriting it, is the initialization to false necessary?
I don't see other fields of msg_sys initialized this way.

>
>  	if (msg_sys->msg_controllen > INT_MAX)
>  		goto out;
> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>  			       used_address->name_len);
>  	}
>
> +	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
> +		ssize_t len = err;
> +
> +		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
> +		if (!err)
> +			err = len;

I'm a bit surprised there isn't any cleanup here if copying the cmsg to user
fails. It seems that that __sock_sendmsg() is executed, then if we fail here,
we just return an error without unrolling what __sock_sendmsg() did. Why is
this ok?

Should sendmsg_copy_cmsg_to_user() be done before __sock_sendms() with a goto
out if it fails?

> +	}
> +
>  out_freectl:
>  	if (ctl_buf != ctl)
>  		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
> @@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>  	if (err < 0)
>  		return err;
>
> -	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
> -				allowed_msghdr_flags);
> +	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
> +			      allowed_msghdr_flags);
>  	kfree(iov);
>  	return err;
>  }
> @@ -2648,7 +2692,7 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>  long __sys_sendmsg_sock(struct socket *sock, struct msghdr *msg,
>  			unsigned int flags)
>  {
> -	return ____sys_sendmsg(sock, msg, flags, NULL, 0);
> +	return ____sys_sendmsg(sock, NULL, msg, flags, NULL, 0);
>  }
>
>  long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags,
> --
> 2.20.1
>

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

Firstly, thanks for your attention to this patch set :)

On 7/25/24 2:34 PM, Mina Almasry wrote:
> On Mon, Jul 08, 2024 at 09:04:03PM +0000, zijianzhang@bytedance.com wrote:
>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>
>> Users can pass msg_control as a placeholder to recvmsg, and get some info
>> from the kernel upon returning of it, but it's not available for sendmsg.
>> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
>> creates a kernel copy of msg_control and passes that to the callees,
>> put_cmsg in sendmsg path will write into this kernel buffer.
>>
>> If users want to get info after returning of sendmsg, they typically have
>> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
>> call overhead. This commit supports copying cmsg from the kernel space to
>> the user space upon returning of sendmsg to mitigate this overhead.
>>
>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
>> ---
>>   include/linux/socket.h |  6 +++++
>>   include/net/sock.h     |  2 +-
>>   net/core/sock.c        |  6 +++--
>>   net/ipv4/ip_sockglue.c |  2 +-
>>   net/ipv6/datagram.c    |  2 +-
>>   net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
>>   6 files changed, 62 insertions(+), 10 deletions(-)
>>
>> diff --git a/include/linux/socket.h b/include/linux/socket.h
>> index 2a1ff91d1914..75461812a7a3 100644
>> --- a/include/linux/socket.h
>> +++ b/include/linux/socket.h
>> @@ -71,6 +71,7 @@ struct msghdr {
>>   		void __user	*msg_control_user;
>>   	};
>>   	bool		msg_control_is_user : 1;
>> +	bool		msg_control_copy_to_user : 1;
> 
> Please add some docs explaining what this does if possible. From reading the
> code, it seems if this is true then we should copy cmsg to user. Not sure where
> or how it's set though.
> 

Agree!

>>   	bool		msg_get_inq : 1;/* return INQ after receive */
>>   	unsigned int	msg_flags;	/* flags on received message */
>>   	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
>> @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
>>   	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
>>   }
>>
>> +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
>> +{
>> +	return 0;
>> +}
>> +
>>   static inline size_t msg_data_left(struct msghdr *msg)
>>   {
>>   	return iov_iter_count(&msg->msg_iter);
>> diff --git a/include/net/sock.h b/include/net/sock.h
>> index cce23ac4d514..9c728287d21d 100644
>> --- a/include/net/sock.h
>> +++ b/include/net/sock.h
>> @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
>>   	};
>>   }
>>
>> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>>   		     struct sockcm_cookie *sockc);
>>   int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
>>   		   struct sockcm_cookie *sockc);
>> diff --git a/net/core/sock.c b/net/core/sock.c
>> index 9abc4fe25953..efb30668dac3 100644
>> --- a/net/core/sock.c
>> +++ b/net/core/sock.c
>> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
>>   }
>>   EXPORT_SYMBOL(sock_alloc_send_pskb);
>>
>> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>>   		     struct sockcm_cookie *sockc)
>>   {
>>   	u32 tsflags;
>> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>>   	default:
>>   		return -EINVAL;
>>   	}
>> +	if (cmsg_copy_to_user(cmsg))
>> +		msg->msg_control_copy_to_user = true;
>>   	return 0;
>>   }
>>   EXPORT_SYMBOL(__sock_cmsg_send);
>> @@ -2881,7 +2883,7 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
>>   			return -EINVAL;
>>   		if (cmsg->cmsg_level != SOL_SOCKET)
>>   			continue;
>> -		ret = __sock_cmsg_send(sk, cmsg, sockc);
>> +		ret = __sock_cmsg_send(sk, msg, cmsg, sockc);
>>   		if (ret)
>>   			return ret;
>>   	}
>> diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c
>> index cf377377b52d..6360b8ba9c84 100644
>> --- a/net/ipv4/ip_sockglue.c
>> +++ b/net/ipv4/ip_sockglue.c
>> @@ -267,7 +267,7 @@ int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc,
>>   		}
>>   #endif
>>   		if (cmsg->cmsg_level == SOL_SOCKET) {
>> -			err = __sock_cmsg_send(sk, cmsg, &ipc->sockc);
>> +			err = __sock_cmsg_send(sk, msg, cmsg, &ipc->sockc);
>>   			if (err)
>>   				return err;
>>   			continue;
>> diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c
>> index fff78496803d..c9ae30acf895 100644
>> --- a/net/ipv6/datagram.c
>> +++ b/net/ipv6/datagram.c
>> @@ -777,7 +777,7 @@ int ip6_datagram_send_ctl(struct net *net, struct sock *sk,
>>   		}
>>
>>   		if (cmsg->cmsg_level == SOL_SOCKET) {
>> -			err = __sock_cmsg_send(sk, cmsg, &ipc6->sockc);
>> +			err = __sock_cmsg_send(sk, msg, cmsg, &ipc6->sockc);
>>   			if (err)
>>   				return err;
>>   			continue;
>> diff --git a/net/socket.c b/net/socket.c
>> index e416920e9399..6a9c9e24d781 100644
>> --- a/net/socket.c
>> +++ b/net/socket.c
>> @@ -2525,8 +2525,43 @@ static int copy_msghdr_from_user(struct msghdr *kmsg,
>>   	return err < 0 ? err : 0;
>>   }
>>
>> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>> -			   unsigned int flags, struct used_address *used_address,
>> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
>> +				     struct user_msghdr __user *umsg)
>> +{
>> +	struct compat_msghdr __user *umsg_compat =
>> +				(struct compat_msghdr __user *)umsg;
>> +	unsigned int flags = msg_sys->msg_flags;
>> +	struct msghdr msg_user = *msg_sys;
>> +	unsigned long cmsg_ptr;
>> +	struct cmsghdr *cmsg;
>> +	int err;
>> +
>> +	msg_user.msg_control_is_user = true;
>> +	msg_user.msg_control_user = umsg->msg_control;
>> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
>> +	for_each_cmsghdr(cmsg, msg_sys) {
>> +		if (!CMSG_OK(msg_sys, cmsg))
>> +			break;
>> +		if (cmsg_copy_to_user(cmsg))
>> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
>> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
> 
> put_cmsg() can fail as far as I can tell. Any reason we don't have to check for
> failure here?
> 
> What happens when these failures happen. Do we end up putting the ZC
> notification later, or is the zc notification lost forever because we did not
> detect the failure to put_cmsg() it?
> 

That's a good question,

The reason why I don't have check here is that I refered to net/socket.c 
and sock.c. It turns out there is no failure check for put_cmsgs in
these files.

For example, in sock_recv_errqueue, it invokes put_cmsg without check,
and kfree_skb anyway. In this case, if put_cmsg fails, we will lose the
information forever. I find cases where sock_recv_errqueue is used for
TX_TIMESTAMP. Maybe loss for timestamp is okay?

However, I find that sock_recv_errqueue is also used in rds_recvmsg to
receive the zc notifications for rds socket. The zc notification could
also be lost forever in this case?

Not sure if anyone knows the reason why there is no failure check for
put_cmsg in net/socket.c and sock.c?

>> +	}
>> +
>> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
>> +	if (err)
>> +		return err;
>> +	if (MSG_CMSG_COMPAT & flags)
>> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
>> +				 &umsg_compat->msg_controllen);
>> +	else
>> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
>> +				 &umsg->msg_controllen);
>> +	return err;
>> +}
>> +
>> +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>> +			   struct msghdr *msg_sys, unsigned int flags,
>> +			   struct used_address *used_address,
>>   			   unsigned int allowed_msghdr_flags)
>>   {
>>   	unsigned char ctl[sizeof(struct cmsghdr) + 20]
>> @@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>>   	ssize_t err;
>>
>>   	err = -ENOBUFS;
>> +	msg_sys->msg_control_copy_to_user = false;
> 
> 
> This may be a lack of knowledge on my part, but i'm very confused that
> msg_control_copy_to_user is set to false here, and then checked below, and it's
> not touched in between. How could it evaluate to true below? Is it because something
> overwrites the value in msg_sys between this set and the check?
> 
> If something is overwriting it, is the initialization to false necessary?
> I don't see other fields of msg_sys initialized this way.
> 

```
msg_sys->msg_control_copy_to_user = false;
...
err = __sock_sendmsg(sock, msg_sys); -> __sock_cmsg_send
...
if (msg && msg_sys->msg_control_copy_to_user && err >= 0)
```

The msg_control_copy_to_user maybe updated by the cmsg handler in
the function __sock_cmsg_send. In patch 2/3, we have
msg_control_copy_to_user updated to true in SCM_ZC_NOTIFICATION
handler.

As for the initialization,

msg_sys is allocated from the kernel stack, if we don't initialize
it to false, it might be randomly true, even though there is no
cmsg wants to be copied back.

Why is there only one initialization here? The existing bit 
msg_control_is_user only get initialized where the following code
path will use it. msg_control_is_user is initialized in multiple
locations in net/socket.c. However, In function hidp_send_frame,
msg_control_is_user is not initialized, because the following path will
not use this bit.

We only initialize msg_control_copy_to_user in function
____sys_sendmsg, because only in this function will we check this bit.

If the initialization here makes people confused, I will add some docs.

>>
>>   	if (msg_sys->msg_controllen > INT_MAX)
>>   		goto out;
>> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>>   			       used_address->name_len);
>>   	}
>>
>> +	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
>> +		ssize_t len = err;
>> +
>> +		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
>> +		if (!err)
>> +			err = len;
> 
> I'm a bit surprised there isn't any cleanup here if copying the cmsg to user
> fails. It seems that that __sock_sendmsg() is executed, then if we fail here,
> we just return an error without unrolling what __sock_sendmsg() did. Why is
> this ok?
> 
> Should sendmsg_copy_cmsg_to_user() be done before __sock_sendms() with a goto
> out if it fails?
> 

I did this refering to ____sys_recvmsg, in this function, if __put_user
fails, we do not unroll what sock_recvmsg did, and return the error code
of __put_user.

Before __sock_sendmsg, the content of msg_control is not updated by the
function __sock_cmsg_send, so sendmsg_copy_cmsg_to_user at this time
might be not expected.

>> +	}
>> +
>>   out_freectl:
>>   	if (ctl_buf != ctl)
>>   		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
>> @@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>>   	if (err < 0)
>>   		return err;
>>
>> -	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
>> -				allowed_msghdr_flags);
>> +	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
>> +			      allowed_msghdr_flags);
>>   	kfree(iov);
>>   	return err;
>>   }
>> @@ -2648,7 +2692,7 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
>>   long __sys_sendmsg_sock(struct socket *sock, struct msghdr *msg,
>>   			unsigned int flags)
>>   {
>> -	return ____sys_sendmsg(sock, msg, flags, NULL, 0);
>> +	return ____sys_sendmsg(sock, NULL, msg, flags, NULL, 0);
>>   }
>>
>>   long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags,
>> --
>> 2.20.1
>>

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

On 7/25/24 4:50 PM, Zijian Zhang wrote:
>>> diff --git a/net/core/sock.c b/net/core/sock.c
>>> index 9abc4fe25953..efb30668dac3 100644
>>> --- a/net/core/sock.c
>>> +++ b/net/core/sock.c
>>> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct 
>>> sock *sk, unsigned long header_len,
>>>   }
>>>   EXPORT_SYMBOL(sock_alloc_send_pskb);
>>>
>>> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>>> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct 
>>> cmsghdr *cmsg,
>>>                struct sockcm_cookie *sockc)
>>>   {
>>>       u32 tsflags;
>>> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct 
>>> cmsghdr *cmsg,
>>>       default:
>>>           return -EINVAL;
>>>       }
>>> +    if (cmsg_copy_to_user(cmsg))
>>> +        msg->msg_control_copy_to_user = true;
>>>       return 0;
>>>   }

msg_control_copy_to_user is set to true here.

>>
>>
>> This may be a lack of knowledge on my part, but i'm very confused that
>> msg_control_copy_to_user is set to false here, and then checked below, 
>> and it's
>> not touched in between. How could it evaluate to true below? Is it 
>> because something
>> overwrites the value in msg_sys between this set and the check?
>>
>> If something is overwriting it, is the initialization to false necessary?
>> I don't see other fields of msg_sys initialized this way.
>>
> 
> ```
> msg_sys->msg_control_copy_to_user = false;
> ...
> err = __sock_sendmsg(sock, msg_sys); -> __sock_cmsg_send
> ...
> if (msg && msg_sys->msg_control_copy_to_user && err >= 0)
> ```
> 
> The msg_control_copy_to_user maybe updated by the cmsg handler in
> the function __sock_cmsg_send. In patch 2/3, we have
> msg_control_copy_to_user updated to true in SCM_ZC_NOTIFICATION
> handler.

Not in patch 2/3 In this patchset msg_control_copy_to_user is set in
this patch, in __sock_cmsg_send.

> 
> As for the initialization,
> 
> msg_sys is allocated from the kernel stack, if we don't initialize
> it to false, it might be randomly true, even though there is no
> cmsg wants to be copied back.
> 
> Why is there only one initialization here? The existing bit 
> msg_control_is_user only get initialized where the following code
> path will use it. msg_control_is_user is initialized in multiple
> locations in net/socket.c. However, In function hidp_send_frame,
> msg_control_is_user is not initialized, because the following path will
> not use this bit.
> 
> We only initialize msg_control_copy_to_user in function
> ____sys_sendmsg, because only in this function will we check this bit.
> 
> If the initialization here makes people confused, I will add some docs.
> 

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Mina Almasry]

On Thu, Jul 25, 2024 at 4:51 PM Zijian Zhang <zijianzhang@bytedance.com> wrote:
...
> >> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >> -                       unsigned int flags, struct used_address *used_address,
> >> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
> >> +                                 struct user_msghdr __user *umsg)
> >> +{
> >> +    struct compat_msghdr __user *umsg_compat =
> >> +                            (struct compat_msghdr __user *)umsg;
> >> +    unsigned int flags = msg_sys->msg_flags;
> >> +    struct msghdr msg_user = *msg_sys;
> >> +    unsigned long cmsg_ptr;
> >> +    struct cmsghdr *cmsg;
> >> +    int err;
> >> +
> >> +    msg_user.msg_control_is_user = true;
> >> +    msg_user.msg_control_user = umsg->msg_control;
> >> +    cmsg_ptr = (unsigned long)msg_user.msg_control;
> >> +    for_each_cmsghdr(cmsg, msg_sys) {
> >> +            if (!CMSG_OK(msg_sys, cmsg))
> >> +                    break;
> >> +            if (cmsg_copy_to_user(cmsg))
> >> +                    put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
> >> +                             cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
> >
> > put_cmsg() can fail as far as I can tell. Any reason we don't have to check for
> > failure here?
> >
> > What happens when these failures happen. Do we end up putting the ZC
> > notification later, or is the zc notification lost forever because we did not
> > detect the failure to put_cmsg() it?
> >
>
> That's a good question,
>
> The reason why I don't have check here is that I refered to net/socket.c
> and sock.c. It turns out there is no failure check for put_cmsgs in
> these files.
>
> For example, in sock_recv_errqueue, it invokes put_cmsg without check,
> and kfree_skb anyway. In this case, if put_cmsg fails, we will lose the
> information forever. I find cases where sock_recv_errqueue is used for
> TX_TIMESTAMP. Maybe loss for timestamp is okay?
>
> However, I find that sock_recv_errqueue is also used in rds_recvmsg to
> receive the zc notifications for rds socket. The zc notification could
> also be lost forever in this case?
>
> Not sure if anyone knows the reason why there is no failure check for
> put_cmsg in net/socket.c and sock.c?
>

I don't know to be honest. I think it's fine for the put_cmsg() to
fail and the notification to be delivered later. However I'm not sure
it's OK for the notification to be lost permanently because of an
error?

For timestamp I can see it not being a big deal if the notification is
lost. For ZC notifications, I think the normal flow is that the
application holds onto the TX buffer until it receives the
notification. If the notification is lost because of an error,
wouldn't that cause a permanent memory leak in the application?

My humble opinion is try as much as possible to either fully deliver
the notification or to save the notification for a future syscall, but
not to lose it. But, I see that no other reviewers are calling this
out, so maybe it's not a big deal and you shouldn't change anything.

> > This may be a lack of knowledge on my part, but i'm very confused that
> > msg_control_copy_to_user is set to false here, and then checked below, and it's
> > not touched in between. How could it evaluate to true below? Is it because something
> > overwrites the value in msg_sys between this set and the check?
> >
> > If something is overwriting it, is the initialization to false necessary?
> > I don't see other fields of msg_sys initialized this way.
> >
>
> ```
> msg_sys->msg_control_copy_to_user = false;
> ...
> err = __sock_sendmsg(sock, msg_sys); -> __sock_cmsg_send
> ...
> if (msg && msg_sys->msg_control_copy_to_user && err >= 0)
> ```
>
> The msg_control_copy_to_user maybe updated by the cmsg handler in
> the function __sock_cmsg_send. In patch 2/3, we have
> msg_control_copy_to_user updated to true in SCM_ZC_NOTIFICATION
> handler.
>
> As for the initialization,
>
> msg_sys is allocated from the kernel stack, if we don't initialize
> it to false, it might be randomly true, even though there is no
> cmsg wants to be copied back.
>
> Why is there only one initialization here? The existing bit
> msg_control_is_user only get initialized where the following code
> path will use it. msg_control_is_user is initialized in multiple
> locations in net/socket.c. However, In function hidp_send_frame,
> msg_control_is_user is not initialized, because the following path will
> not use this bit.
>
> We only initialize msg_control_copy_to_user in function
> ____sys_sendmsg, because only in this function will we check this bit.
>
> If the initialization here makes people confused, I will add some docs.
>

Thanks for the explanation. This looks correct to me now, no need to
add docs. I just missed the intention.

> >>
> >>      if (msg_sys->msg_controllen > INT_MAX)
> >>              goto out;
> >> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >>                             used_address->name_len);
> >>      }
> >>
> >> +    if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
> >> +            ssize_t len = err;
> >> +
> >> +            err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
> >> +            if (!err)
> >> +                    err = len;
> >
> > I'm a bit surprised there isn't any cleanup here if copying the cmsg to user
> > fails. It seems that that __sock_sendmsg() is executed, then if we fail here,
> > we just return an error without unrolling what __sock_sendmsg() did. Why is
> > this ok?
> >
> > Should sendmsg_copy_cmsg_to_user() be done before __sock_sendms() with a goto
> > out if it fails?
> >
>
> I did this refering to ____sys_recvmsg, in this function, if __put_user
> fails, we do not unroll what sock_recvmsg did, and return the error code
> of __put_user.
>
> Before __sock_sendmsg, the content of msg_control is not updated by the
> function __sock_cmsg_send, so sendmsg_copy_cmsg_to_user at this time
> might be not expected.
>

I see. I don't think sendmsg_copy_cmsg_to_user() should unroll
__sock_sendmsg(), but if possible for the notification not to be lost,
I think that would be an improvement.

-- 
Thanks,
Mina

Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user space in sendmsg

[Zijian Zhang]

On 7/26/24 10:00 AM, Mina Almasry wrote:
> On Thu, Jul 25, 2024 at 4:51 PM Zijian Zhang <zijianzhang@bytedance.com> wrote:
> ...
>>>> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>>>> -                       unsigned int flags, struct used_address *used_address,
>>>> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
>>>> +                                 struct user_msghdr __user *umsg)
>>>> +{
>>>> +    struct compat_msghdr __user *umsg_compat =
>>>> +                            (struct compat_msghdr __user *)umsg;
>>>> +    unsigned int flags = msg_sys->msg_flags;
>>>> +    struct msghdr msg_user = *msg_sys;
>>>> +    unsigned long cmsg_ptr;
>>>> +    struct cmsghdr *cmsg;
>>>> +    int err;
>>>> +
>>>> +    msg_user.msg_control_is_user = true;
>>>> +    msg_user.msg_control_user = umsg->msg_control;
>>>> +    cmsg_ptr = (unsigned long)msg_user.msg_control;
>>>> +    for_each_cmsghdr(cmsg, msg_sys) {
>>>> +            if (!CMSG_OK(msg_sys, cmsg))
>>>> +                    break;
>>>> +            if (cmsg_copy_to_user(cmsg))
>>>> +                    put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
>>>> +                             cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
>>>
>>> put_cmsg() can fail as far as I can tell. Any reason we don't have to check for
>>> failure here?
>>>
>>> What happens when these failures happen. Do we end up putting the ZC
>>> notification later, or is the zc notification lost forever because we did not
>>> detect the failure to put_cmsg() it?
>>>
>>
>> That's a good question,
>>
>> The reason why I don't have check here is that I refered to net/socket.c
>> and sock.c. It turns out there is no failure check for put_cmsgs in
>> these files.
>>
>> For example, in sock_recv_errqueue, it invokes put_cmsg without check,
>> and kfree_skb anyway. In this case, if put_cmsg fails, we will lose the
>> information forever. I find cases where sock_recv_errqueue is used for
>> TX_TIMESTAMP. Maybe loss for timestamp is okay?
>>
>> However, I find that sock_recv_errqueue is also used in rds_recvmsg to
>> receive the zc notifications for rds socket. The zc notification could
>> also be lost forever in this case?
>>
>> Not sure if anyone knows the reason why there is no failure check for
>> put_cmsg in net/socket.c and sock.c?
>>
> 
> I don't know to be honest. I think it's fine for the put_cmsg() to
> fail and the notification to be delivered later. However I'm not sure
> it's OK for the notification to be lost permanently because of an
> error?
> 
> For timestamp I can see it not being a big deal if the notification is
> lost. For ZC notifications, I think the normal flow is that the
> application holds onto the TX buffer until it receives the
> notification. If the notification is lost because of an error,
> wouldn't that cause a permanent memory leak in the application?
> 
> My humble opinion is try as much as possible to either fully deliver
> the notification or to save the notification for a future syscall, but
> not to lose it. But, I see that no other reviewers are calling this
> out, so maybe it's not a big deal and you shouldn't change anything.
> 

Agree, in ZC notification case, saving the notification for a future 
syscall is better than losing it forever. The difficulties I am aware of
are as follows,

If we find put_cmsg fails in
sendmsg_copy_cmsg_to_user, we have reached the end of ____sys_sendmsg,

1. Since the skb which carries the zc information has been freed from 
the errqueue. To roll back the effect, we need to sock_omalloc an skb
and insert it back. Or, we can store the information somewhere, but I
assume we need to allocate some memory, if the allocation fails, we may
still lose the info? Shall we make sure the allocation succeeds?

2. Currently, we free the skb in the hanlder of SCM_ZC_NOTIF, can we
free it after we are sure that put_cmsg succeed? One blocking I can
think of is that because of notification coalescing, the
information in the skb might be updated, since we snapshot the 
information earlier, so the information we copy to user might be
outdated. And, we also need a post handler for this cmsg_type, in
sendmsg_copy_cmsg_to_user.

3. The above is the specific unrolling/post handling logic for 
SCM_ZC_NOTIF. Each cmsg_type, which needs to be copied back could have
their own logic, we may add a function to handle it according to
cmsg_type. And call this function in sendmsg_copy_cmsg_to_user, to make
the code generic.

So besides ABI change, this is another problem we have for this
mechanism. These two concerns make our alternative usr_addr method(v4)
shine. In v4, users pass in a user address which points to a
zc_info_elem array. In the handler of SCM_ZC_NOTIF, we do copy to user
to that usr_addr. If it fails, in the context of the handler, the
unrolling is very clean and easy. The problem with this method is that
it makes the API of msg_control hacky.

Thanks for pointing this out, more comments are welcome!

>>> This may be a lack of knowledge on my part, but i'm very confused that
>>> msg_control_copy_to_user is set to false here, and then checked below, and it's
>>> not touched in between. How could it evaluate to true below? Is it because something
>>> overwrites the value in msg_sys between this set and the check?
>>>
>>> If something is overwriting it, is the initialization to false necessary?
>>> I don't see other fields of msg_sys initialized this way.
>>>
>>
>> ```
>> msg_sys->msg_control_copy_to_user = false;
>> ...
>> err = __sock_sendmsg(sock, msg_sys); -> __sock_cmsg_send
>> ...
>> if (msg && msg_sys->msg_control_copy_to_user && err >= 0)
>> ```
>>
>> The msg_control_copy_to_user maybe updated by the cmsg handler in
>> the function __sock_cmsg_send. In patch 2/3, we have
>> msg_control_copy_to_user updated to true in SCM_ZC_NOTIFICATION
>> handler.
>>
>> As for the initialization,
>>
>> msg_sys is allocated from the kernel stack, if we don't initialize
>> it to false, it might be randomly true, even though there is no
>> cmsg wants to be copied back.
>>
>> Why is there only one initialization here? The existing bit
>> msg_control_is_user only get initialized where the following code
>> path will use it. msg_control_is_user is initialized in multiple
>> locations in net/socket.c. However, In function hidp_send_frame,
>> msg_control_is_user is not initialized, because the following path will
>> not use this bit.
>>
>> We only initialize msg_control_copy_to_user in function
>> ____sys_sendmsg, because only in this function will we check this bit.
>>
>> If the initialization here makes people confused, I will add some docs.
>>
> 
> Thanks for the explanation. This looks correct to me now, no need to
> add docs. I just missed the intention.
> 

No problem, my pleasure.

>>>>
>>>>       if (msg_sys->msg_controllen > INT_MAX)
>>>>               goto out;
>>>> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
>>>>                              used_address->name_len);
>>>>       }
>>>>
>>>> +    if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
>>>> +            ssize_t len = err;
>>>> +
>>>> +            err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
>>>> +            if (!err)
>>>> +                    err = len;
>>>
>>> I'm a bit surprised there isn't any cleanup here if copying the cmsg to user
>>> fails. It seems that that __sock_sendmsg() is executed, then if we fail here,
>>> we just return an error without unrolling what __sock_sendmsg() did. Why is
>>> this ok?
>>>
>>> Should sendmsg_copy_cmsg_to_user() be done before __sock_sendms() with a goto
>>> out if it fails?
>>>
>>
>> I did this refering to ____sys_recvmsg, in this function, if __put_user
>> fails, we do not unroll what sock_recvmsg did, and return the error code
>> of __put_user.
>>
>> Before __sock_sendmsg, the content of msg_control is not updated by the
>> function __sock_cmsg_send, so sendmsg_copy_cmsg_to_user at this time
>> might be not expected.
>>
> 
> I see. I don't think sendmsg_copy_cmsg_to_user() should unroll
> __sock_sendmsg(), but if possible for the notification not to be lost,
> I think that would be an improvement.
> 

[PATCH net-next v7 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
However, zerocopy is not a free lunch. Apart from the management of user
pages, the combination of poll + recvmsg to receive notifications incurs
unignorable overhead in the applications. We try to mitigate this overhead
with a new notification mechanism based on msg_control. Leveraging the
general framework to copy cmsgs to the user space, we copy zerocopy
notifications to the user upon returning of sendmsgs.

Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
---
 arch/alpha/include/uapi/asm/socket.h  |  2 ++
 arch/mips/include/uapi/asm/socket.h   |  2 ++
 arch/parisc/include/uapi/asm/socket.h |  2 ++
 arch/sparc/include/uapi/asm/socket.h  |  2 ++
 include/linux/socket.h                |  2 +-
 include/uapi/asm-generic/socket.h     |  2 ++
 include/uapi/linux/socket.h           | 13 ++++++++
 net/core/sock.c                       | 46 +++++++++++++++++++++++++++
 8 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h
index e94f621903fe..7c32d9dbe47f 100644
--- a/arch/alpha/include/uapi/asm/socket.h
+++ b/arch/alpha/include/uapi/asm/socket.h
@@ -140,6 +140,8 @@
 #define SO_PASSPIDFD		76
 #define SO_PEERPIDFD		77
 
+#define SCM_ZC_NOTIFICATION	78
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h
index 60ebaed28a4c..3f7fade998cb 100644
--- a/arch/mips/include/uapi/asm/socket.h
+++ b/arch/mips/include/uapi/asm/socket.h
@@ -151,6 +151,8 @@
 #define SO_PASSPIDFD		76
 #define SO_PEERPIDFD		77
 
+#define SCM_ZC_NOTIFICATION	78
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h
index be264c2b1a11..77f5bee0fdc9 100644
--- a/arch/parisc/include/uapi/asm/socket.h
+++ b/arch/parisc/include/uapi/asm/socket.h
@@ -132,6 +132,8 @@
 #define SO_PASSPIDFD		0x404A
 #define SO_PEERPIDFD		0x404B
 
+#define SCM_ZC_NOTIFICATION	0x404C
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h
index 682da3714686..eb44fc515b45 100644
--- a/arch/sparc/include/uapi/asm/socket.h
+++ b/arch/sparc/include/uapi/asm/socket.h
@@ -133,6 +133,8 @@
 #define SO_PASSPIDFD             0x0055
 #define SO_PEERPIDFD             0x0056
 
+#define SCM_ZC_NOTIFICATION      0x0057
+
 #if !defined(__KERNEL__)
 
 
diff --git a/include/linux/socket.h b/include/linux/socket.h
index 75461812a7a3..6f1b791e2de8 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -171,7 +171,7 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
 
 static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
 {
-	return 0;
+	return __cmsg->cmsg_type == SCM_ZC_NOTIFICATION;
 }
 
 static inline size_t msg_data_left(struct msghdr *msg)
diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h
index 8ce8a39a1e5f..02e9159c7944 100644
--- a/include/uapi/asm-generic/socket.h
+++ b/include/uapi/asm-generic/socket.h
@@ -135,6 +135,8 @@
 #define SO_PASSPIDFD		76
 #define SO_PEERPIDFD		77
 
+#define SCM_ZC_NOTIFICATION	78
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h
index d3fcd3b5ec53..ab361f30f3a6 100644
--- a/include/uapi/linux/socket.h
+++ b/include/uapi/linux/socket.h
@@ -2,6 +2,8 @@
 #ifndef _UAPI_LINUX_SOCKET_H
 #define _UAPI_LINUX_SOCKET_H
 
+#include <linux/types.h>
+
 /*
  * Desired design of maximum size and alignment (see RFC2553)
  */
@@ -35,4 +37,15 @@ struct __kernel_sockaddr_storage {
 #define SOCK_TXREHASH_DISABLED	0
 #define SOCK_TXREHASH_ENABLED	1
 
+struct zc_info_elem {
+	__u32 lo;
+	__u32 hi;
+	__u8 zerocopy;
+};
+
+struct zc_info {
+	__u32 size;
+	struct zc_info_elem arr[];
+};
+
 #endif /* _UAPI_LINUX_SOCKET_H */
diff --git a/net/core/sock.c b/net/core/sock.c
index efb30668dac3..e0b5162233d3 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2863,6 +2863,52 @@ int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
 	case SCM_RIGHTS:
 	case SCM_CREDENTIALS:
 		break;
+	case SCM_ZC_NOTIFICATION: {
+		struct zc_info *zc_info = CMSG_DATA(cmsg);
+		struct zc_info_elem *zc_info_arr;
+		struct sock_exterr_skb *serr;
+		int cmsg_data_len, i = 0;
+		struct sk_buff_head *q;
+		unsigned long flags;
+		struct sk_buff *skb;
+		u32 zc_info_size;
+
+		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
+			return -EINVAL;
+
+		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
+		if (cmsg_data_len < sizeof(struct zc_info))
+			return -EINVAL;
+
+		zc_info_size = zc_info->size;
+		zc_info_arr = zc_info->arr;
+		if (cmsg_data_len != sizeof(struct zc_info) +
+				     zc_info_size * sizeof(struct zc_info_elem))
+			return -EINVAL;
+
+		q = &sk->sk_error_queue;
+		spin_lock_irqsave(&q->lock, flags);
+		skb = skb_peek(q);
+		while (skb && i < zc_info_size) {
+			struct sk_buff *skb_next = skb_peek_next(skb, q);
+
+			serr = SKB_EXT_ERR(skb);
+			if (serr->ee.ee_errno == 0 &&
+			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
+				zc_info_arr[i].hi = serr->ee.ee_data;
+				zc_info_arr[i].lo = serr->ee.ee_info;
+				zc_info_arr[i].zerocopy = !(serr->ee.ee_code
+							  & SO_EE_CODE_ZEROCOPY_COPIED);
+				__skb_unlink(skb, q);
+				consume_skb(skb);
+				i++;
+			}
+			skb = skb_next;
+		}
+		spin_unlock_irqrestore(&q->lock, flags);
+		zc_info->size = i;
+		break;
+	}
 	default:
 		return -EINVAL;
 	}
-- 
2.20.1

Re: [PATCH net-next v7 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[Mina Almasry]

On Mon, Jul 08, 2024 at 09:04:04PM +0000, zijianzhang@bytedance.com wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
>
> The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
> However, zerocopy is not a free lunch. Apart from the management of user
> pages, the combination of poll + recvmsg to receive notifications incurs
> unignorable overhead in the applications. We try to mitigate this overhead
> with a new notification mechanism based on msg_control. Leveraging the
> general framework to copy cmsgs to the user space, we copy zerocopy
> notifications to the user upon returning of sendmsgs.
>
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> ---
>  arch/alpha/include/uapi/asm/socket.h  |  2 ++
>  arch/mips/include/uapi/asm/socket.h   |  2 ++
>  arch/parisc/include/uapi/asm/socket.h |  2 ++
>  arch/sparc/include/uapi/asm/socket.h  |  2 ++
>  include/linux/socket.h                |  2 +-
>  include/uapi/asm-generic/socket.h     |  2 ++
>  include/uapi/linux/socket.h           | 13 ++++++++
>  net/core/sock.c                       | 46 +++++++++++++++++++++++++++
>  8 files changed, 70 insertions(+), 1 deletion(-)
>
> diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h
> index e94f621903fe..7c32d9dbe47f 100644
> --- a/arch/alpha/include/uapi/asm/socket.h
> +++ b/arch/alpha/include/uapi/asm/socket.h
> @@ -140,6 +140,8 @@
>  #define SO_PASSPIDFD		76
>  #define SO_PEERPIDFD		77
>
> +#define SCM_ZC_NOTIFICATION	78
> +
>  #if !defined(__KERNEL__)
>
>  #if __BITS_PER_LONG == 64
> diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h
> index 60ebaed28a4c..3f7fade998cb 100644
> --- a/arch/mips/include/uapi/asm/socket.h
> +++ b/arch/mips/include/uapi/asm/socket.h
> @@ -151,6 +151,8 @@
>  #define SO_PASSPIDFD		76
>  #define SO_PEERPIDFD		77
>
> +#define SCM_ZC_NOTIFICATION	78
> +
>  #if !defined(__KERNEL__)
>
>  #if __BITS_PER_LONG == 64
> diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h
> index be264c2b1a11..77f5bee0fdc9 100644
> --- a/arch/parisc/include/uapi/asm/socket.h
> +++ b/arch/parisc/include/uapi/asm/socket.h
> @@ -132,6 +132,8 @@
>  #define SO_PASSPIDFD		0x404A
>  #define SO_PEERPIDFD		0x404B
>
> +#define SCM_ZC_NOTIFICATION	0x404C
> +
>  #if !defined(__KERNEL__)
>
>  #if __BITS_PER_LONG == 64
> diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h
> index 682da3714686..eb44fc515b45 100644
> --- a/arch/sparc/include/uapi/asm/socket.h
> +++ b/arch/sparc/include/uapi/asm/socket.h
> @@ -133,6 +133,8 @@
>  #define SO_PASSPIDFD             0x0055
>  #define SO_PEERPIDFD             0x0056
>
> +#define SCM_ZC_NOTIFICATION      0x0057
> +
>  #if !defined(__KERNEL__)
>
>
> diff --git a/include/linux/socket.h b/include/linux/socket.h
> index 75461812a7a3..6f1b791e2de8 100644
> --- a/include/linux/socket.h
> +++ b/include/linux/socket.h
> @@ -171,7 +171,7 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
>
>  static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
>  {
> -	return 0;
> +	return __cmsg->cmsg_type == SCM_ZC_NOTIFICATION;
>  }
>
>  static inline size_t msg_data_left(struct msghdr *msg)
> diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h
> index 8ce8a39a1e5f..02e9159c7944 100644
> --- a/include/uapi/asm-generic/socket.h
> +++ b/include/uapi/asm-generic/socket.h
> @@ -135,6 +135,8 @@
>  #define SO_PASSPIDFD		76
>  #define SO_PEERPIDFD		77
>
> +#define SCM_ZC_NOTIFICATION	78
> +
>  #if !defined(__KERNEL__)
>
>  #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
> diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h
> index d3fcd3b5ec53..ab361f30f3a6 100644
> --- a/include/uapi/linux/socket.h
> +++ b/include/uapi/linux/socket.h
> @@ -2,6 +2,8 @@
>  #ifndef _UAPI_LINUX_SOCKET_H
>  #define _UAPI_LINUX_SOCKET_H
>
> +#include <linux/types.h>
> +
>  /*
>   * Desired design of maximum size and alignment (see RFC2553)
>   */
> @@ -35,4 +37,15 @@ struct __kernel_sockaddr_storage {
>  #define SOCK_TXREHASH_DISABLED	0
>  #define SOCK_TXREHASH_ENABLED	1
>
> +struct zc_info_elem {
> +	__u32 lo;
> +	__u32 hi;
> +	__u8 zerocopy;

Some docs please on what each of these are, if possible. Sorry if the repeated
requests are annoying.

In particular I'm a bit confused why the zerocopy field is there. Looking at
the code, is this always set to 1?

> +};
> +
> +struct zc_info {
> +	__u32 size;
> +	struct zc_info_elem arr[];
> +};
> +
>  #endif /* _UAPI_LINUX_SOCKET_H */
> diff --git a/net/core/sock.c b/net/core/sock.c
> index efb30668dac3..e0b5162233d3 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -2863,6 +2863,52 @@ int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>  	case SCM_RIGHTS:
>  	case SCM_CREDENTIALS:
>  		break;
> +	case SCM_ZC_NOTIFICATION: {
> +		struct zc_info *zc_info = CMSG_DATA(cmsg);
> +		struct zc_info_elem *zc_info_arr;
> +		struct sock_exterr_skb *serr;
> +		int cmsg_data_len, i = 0;
> +		struct sk_buff_head *q;
> +		unsigned long flags;
> +		struct sk_buff *skb;
> +		u32 zc_info_size;
> +
> +		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
> +			return -EINVAL;
> +
> +		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
> +		if (cmsg_data_len < sizeof(struct zc_info))
> +			return -EINVAL;
> +
> +		zc_info_size = zc_info->size;
> +		zc_info_arr = zc_info->arr;

Annoying nit: To be honest zc_info->size isn't much longer to type than
zc_info_size, so I would have not added local variables.

> +		if (cmsg_data_len != sizeof(struct zc_info) +
> +				     zc_info_size * sizeof(struct zc_info_elem))
> +			return -EINVAL;
> +
> +		q = &sk->sk_error_queue;
> +		spin_lock_irqsave(&q->lock, flags);
> +		skb = skb_peek(q);
> +		while (skb && i < zc_info_size) {
> +			struct sk_buff *skb_next = skb_peek_next(skb, q);
> +
> +			serr = SKB_EXT_ERR(skb);
> +			if (serr->ee.ee_errno == 0 &&
> +			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
> +				zc_info_arr[i].hi = serr->ee.ee_data;
> +				zc_info_arr[i].lo = serr->ee.ee_info;
> +				zc_info_arr[i].zerocopy = !(serr->ee.ee_code
> +							  & SO_EE_CODE_ZEROCOPY_COPIED);
> +				__skb_unlink(skb, q);
> +				consume_skb(skb);
> +				i++;
> +			}
> +			skb = skb_next;
> +		}
> +		spin_unlock_irqrestore(&q->lock, flags);

I wonder if you should drop the spin lock in the middle of this loop somehow,
otherwise you may end up spinning for a very long time while the spinlock held
and irq disabled.

IIRC zc_info_size is user input, right? Maybe you should limit zc_info_size to
16 entries or something. So the user doesn't end up passing 100000 as
   zc_info_size and making the kernel loop for a long time here.

> +		zc_info->size = i;
> +		break;
> +	}
>  	default:
>  		return -EINVAL;
>  	}
> --
> 2.20.1
>

Re: [External] Re: [PATCH net-next v7 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[Zijian Zhang]

On 7/25/24 2:59 PM, Mina Almasry wrote:
> On Mon, Jul 08, 2024 at 09:04:04PM +0000, zijianzhang@bytedance.com wrote:
>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>
>> The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
>> However, zerocopy is not a free lunch. Apart from the management of user
>> pages, the combination of poll + recvmsg to receive notifications incurs
>> unignorable overhead in the applications. We try to mitigate this overhead
>> with a new notification mechanism based on msg_control. Leveraging the
>> general framework to copy cmsgs to the user space, we copy zerocopy
>> notifications to the user upon returning of sendmsgs.
>>
>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
>> ---
>>   arch/alpha/include/uapi/asm/socket.h  |  2 ++
>>   arch/mips/include/uapi/asm/socket.h   |  2 ++
>>   arch/parisc/include/uapi/asm/socket.h |  2 ++
>>   arch/sparc/include/uapi/asm/socket.h  |  2 ++
>>   include/linux/socket.h                |  2 +-
>>   include/uapi/asm-generic/socket.h     |  2 ++
>>   include/uapi/linux/socket.h           | 13 ++++++++
>>   net/core/sock.c                       | 46 +++++++++++++++++++++++++++
>>   8 files changed, 70 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h
>> index e94f621903fe..7c32d9dbe47f 100644
>> --- a/arch/alpha/include/uapi/asm/socket.h
>> +++ b/arch/alpha/include/uapi/asm/socket.h
>> @@ -140,6 +140,8 @@
>>   #define SO_PASSPIDFD		76
>>   #define SO_PEERPIDFD		77
>>
>> +#define SCM_ZC_NOTIFICATION	78
>> +
>>   #if !defined(__KERNEL__)
>>
>>   #if __BITS_PER_LONG == 64
>> diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h
>> index 60ebaed28a4c..3f7fade998cb 100644
>> --- a/arch/mips/include/uapi/asm/socket.h
>> +++ b/arch/mips/include/uapi/asm/socket.h
>> @@ -151,6 +151,8 @@
>>   #define SO_PASSPIDFD		76
>>   #define SO_PEERPIDFD		77
>>
>> +#define SCM_ZC_NOTIFICATION	78
>> +
>>   #if !defined(__KERNEL__)
>>
>>   #if __BITS_PER_LONG == 64
>> diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h
>> index be264c2b1a11..77f5bee0fdc9 100644
>> --- a/arch/parisc/include/uapi/asm/socket.h
>> +++ b/arch/parisc/include/uapi/asm/socket.h
>> @@ -132,6 +132,8 @@
>>   #define SO_PASSPIDFD		0x404A
>>   #define SO_PEERPIDFD		0x404B
>>
>> +#define SCM_ZC_NOTIFICATION	0x404C
>> +
>>   #if !defined(__KERNEL__)
>>
>>   #if __BITS_PER_LONG == 64
>> diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h
>> index 682da3714686..eb44fc515b45 100644
>> --- a/arch/sparc/include/uapi/asm/socket.h
>> +++ b/arch/sparc/include/uapi/asm/socket.h
>> @@ -133,6 +133,8 @@
>>   #define SO_PASSPIDFD             0x0055
>>   #define SO_PEERPIDFD             0x0056
>>
>> +#define SCM_ZC_NOTIFICATION      0x0057
>> +
>>   #if !defined(__KERNEL__)
>>
>>
>> diff --git a/include/linux/socket.h b/include/linux/socket.h
>> index 75461812a7a3..6f1b791e2de8 100644
>> --- a/include/linux/socket.h
>> +++ b/include/linux/socket.h
>> @@ -171,7 +171,7 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
>>
>>   static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
>>   {
>> -	return 0;
>> +	return __cmsg->cmsg_type == SCM_ZC_NOTIFICATION;
>>   }
>>
>>   static inline size_t msg_data_left(struct msghdr *msg)
>> diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h
>> index 8ce8a39a1e5f..02e9159c7944 100644
>> --- a/include/uapi/asm-generic/socket.h
>> +++ b/include/uapi/asm-generic/socket.h
>> @@ -135,6 +135,8 @@
>>   #define SO_PASSPIDFD		76
>>   #define SO_PEERPIDFD		77
>>
>> +#define SCM_ZC_NOTIFICATION	78
>> +
>>   #if !defined(__KERNEL__)
>>
>>   #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
>> diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h
>> index d3fcd3b5ec53..ab361f30f3a6 100644
>> --- a/include/uapi/linux/socket.h
>> +++ b/include/uapi/linux/socket.h
>> @@ -2,6 +2,8 @@
>>   #ifndef _UAPI_LINUX_SOCKET_H
>>   #define _UAPI_LINUX_SOCKET_H
>>
>> +#include <linux/types.h>
>> +
>>   /*
>>    * Desired design of maximum size and alignment (see RFC2553)
>>    */
>> @@ -35,4 +37,15 @@ struct __kernel_sockaddr_storage {
>>   #define SOCK_TXREHASH_DISABLED	0
>>   #define SOCK_TXREHASH_ENABLED	1
>>
>> +struct zc_info_elem {
>> +	__u32 lo;
>> +	__u32 hi;
>> +	__u8 zerocopy;
> 
> Some docs please on what each of these are, if possible. Sorry if the repeated
> requests are annoying.
> 
> In particular I'm a bit confused why the zerocopy field is there. Looking at
> the code, is this always set to 1?
> 
```
hi = serr->ee_data;
lo = serr->ee_info;
zerocopy = !(serr->ee_code & SO_EE_CODE_ZEROCOPY_COPIED);
```
In the original method, the above code means one notification for
sendmsg id [lo, hi], with zerocopy=n/y to denote if the zerocopy is
reverted back to copy.

So the zerocopy field aligns the same meaning of
!(serr->ee_code & SO_EE_CODE_ZEROCOPY_COPIED) in the original method.

Sorry for the confusion, I will add more docs to explain it.

>> +};
>> +
>> +struct zc_info {
>> +	__u32 size;
>> +	struct zc_info_elem arr[];
>> +};
>> +
>>   #endif /* _UAPI_LINUX_SOCKET_H */
>> diff --git a/net/core/sock.c b/net/core/sock.c
>> index efb30668dac3..e0b5162233d3 100644
>> --- a/net/core/sock.c
>> +++ b/net/core/sock.c
>> @@ -2863,6 +2863,52 @@ int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
>>   	case SCM_RIGHTS:
>>   	case SCM_CREDENTIALS:
>>   		break;
>> +	case SCM_ZC_NOTIFICATION: {
>> +		struct zc_info *zc_info = CMSG_DATA(cmsg);
>> +		struct zc_info_elem *zc_info_arr;
>> +		struct sock_exterr_skb *serr;
>> +		int cmsg_data_len, i = 0;
>> +		struct sk_buff_head *q;
>> +		unsigned long flags;
>> +		struct sk_buff *skb;
>> +		u32 zc_info_size;
>> +
>> +		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
>> +			return -EINVAL;
>> +
>> +		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
>> +		if (cmsg_data_len < sizeof(struct zc_info))
>> +			return -EINVAL;
>> +
>> +		zc_info_size = zc_info->size;
>> +		zc_info_arr = zc_info->arr;
> 
> Annoying nit: To be honest zc_info->size isn't much longer to type than
> zc_info_size, so I would have not added local variables.
> 

Agree, nice catch!

>> +		if (cmsg_data_len != sizeof(struct zc_info) +
>> +				     zc_info_size * sizeof(struct zc_info_elem))
>> +			return -EINVAL;
>> +
>> +		q = &sk->sk_error_queue;
>> +		spin_lock_irqsave(&q->lock, flags);
>> +		skb = skb_peek(q);
>> +		while (skb && i < zc_info_size) {
>> +			struct sk_buff *skb_next = skb_peek_next(skb, q);
>> +
>> +			serr = SKB_EXT_ERR(skb);
>> +			if (serr->ee.ee_errno == 0 &&
>> +			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
>> +				zc_info_arr[i].hi = serr->ee.ee_data;
>> +				zc_info_arr[i].lo = serr->ee.ee_info;
>> +				zc_info_arr[i].zerocopy = !(serr->ee.ee_code
>> +							  & SO_EE_CODE_ZEROCOPY_COPIED);
>> +				__skb_unlink(skb, q);
>> +				consume_skb(skb);
>> +				i++;
>> +			}
>> +			skb = skb_next;
>> +		}
>> +		spin_unlock_irqrestore(&q->lock, flags);
> 
> I wonder if you should drop the spin lock in the middle of this loop somehow,
> otherwise you may end up spinning for a very long time while the spinlock held
> and irq disabled.
> 
> IIRC zc_info_size is user input, right? Maybe you should limit zc_info_size to
> 16 entries or something. So the user doesn't end up passing 100000 as
>     zc_info_size and making the kernel loop for a long time here.
> 

Thanks for the suggestion, totally agree, I should limit the
zc_info_size.

>> +		zc_info->size = i;
>> +		break;
>> +	}
>>   	default:
>>   		return -EINVAL;
>>   	}
>> --
>> 2.20.1
>>

[PATCH net-next v7 3/3] selftests: add MSG_ZEROCOPY msg_control notification test

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

We update selftests/net/msg_zerocopy.c to accommodate the new mechanism,
cfg_notification_limit has the same semantics for both methods. Test
results are as follows, we update skb_orphan_frags_rx to the same as
skb_orphan_frags to support zerocopy in the localhost test.

cfg_notification_limit = 1, both method get notifications after 1 calling
of sendmsg. In this case, the new method has around 17% cpu savings in TCP
and 23% cpu savings in UDP.
+---------------------+---------+---------+---------+---------+
| Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+---------------------+---------+---------+---------+---------+
| ZCopy (MB)          | 7523    | 7706    | 7489    | 7304    |
+---------------------+---------+---------+---------+---------+
| New ZCopy (MB)      | 8834    | 8993    | 9053    | 9228    |
+---------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy   | 117.42% | 116.70% | 120.88% | 126.34% |
+---------------------+---------+---------+---------+---------+

cfg_notification_limit = 32, both get notifications after 32 calling of
sendmsg, which means more chances to coalesce notifications, and less
overhead of poll + recvmsg for the original method. In this case, the new
method has around 7% cpu savings in TCP and slightly better cpu usage in
UDP. In the context of selftest, notifications of TCP are more likely to
out of order than UDP, it's easier to coalesce more notifications in UDP.
The original method can get one notification with range of 32 in a recvmsg
most of the time. In TCP, most notifications' range is around 2, so the
original method needs around 16 recvmsgs to get notified in one round.
That's the reason for the "New ZCopy / ZCopy" diff in TCP and UDP here.
+---------------------+---------+---------+---------+---------+
| Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+---------------------+---------+---------+---------+---------+
| ZCopy (MB)          | 8842    | 8735    | 10072   | 9380    |
+---------------------+---------+---------+---------+---------+
| New ZCopy (MB)      | 9366    | 9477    | 10108   | 9385    |
+---------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy   | 106.00% | 108.28% | 100.31% | 100.01% |
+---------------------+---------+---------+---------+---------+

In conclusion, when notification interval is small or notifications are
hard to be coalesced, the new mechanism is highly recommended. Otherwise,
the performance gain from the new mechanism is very limited.

Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
---
 tools/testing/selftests/net/msg_zerocopy.c  | 111 ++++++++++++++++++--
 tools/testing/selftests/net/msg_zerocopy.sh |   1 +
 2 files changed, 105 insertions(+), 7 deletions(-)

diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c
index 7ea5fb28c93d..064d2aaf7a2c 100644
--- a/tools/testing/selftests/net/msg_zerocopy.c
+++ b/tools/testing/selftests/net/msg_zerocopy.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
 /* Evaluate MSG_ZEROCOPY
  *
  * Send traffic between two processes over one of the supported
@@ -66,6 +67,10 @@
 #define SO_ZEROCOPY	60
 #endif
 
+#ifndef SCM_ZC_NOTIFICATION
+#define SCM_ZC_NOTIFICATION	78
+#endif
+
 #ifndef SO_EE_CODE_ZEROCOPY_COPIED
 #define SO_EE_CODE_ZEROCOPY_COPIED	1
 #endif
@@ -74,6 +79,11 @@
 #define MSG_ZEROCOPY	0x4000000
 #endif
 
+enum notification_type {
+	MSG_ZEROCOPY_NOTIFY_ERRQUEUE = 1,
+	MSG_ZEROCOPY_NOTIFY_SENDMSG = 2,
+};
+
 static int  cfg_cork;
 static bool cfg_cork_mixed;
 static int  cfg_cpu		= -1;		/* default: pin to last cpu */
@@ -86,7 +96,7 @@ static int  cfg_runtime_ms	= 4200;
 static int  cfg_verbose;
 static int  cfg_waittime_ms	= 500;
 static int  cfg_notification_limit = 32;
-static bool cfg_zerocopy;
+static enum notification_type cfg_zerocopy;
 
 static socklen_t cfg_alen;
 static struct sockaddr_storage cfg_dst_addr;
@@ -97,6 +107,9 @@ static long packets, bytes, completions, expected_completions;
 static int  zerocopied = -1;
 static uint32_t next_completion;
 static uint32_t sends_since_notify;
+static char	*zc_ckbuf;
+static int	zc_info_size;
+static bool	added_zcopy_info;
 
 static unsigned long gettimeofday_ms(void)
 {
@@ -182,7 +195,26 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie)
 	memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie));
 }
 
-static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
+static void add_zcopy_info(struct msghdr *msg)
+{
+	struct zc_info *zc_info;
+	struct cmsghdr *cm;
+
+	if (!msg->msg_control)
+		error(1, errno, "NULL user arg");
+	cm = (struct cmsghdr *)msg->msg_control;
+	cm->cmsg_len = CMSG_LEN(zc_info_size);
+	cm->cmsg_level = SOL_SOCKET;
+	cm->cmsg_type = SCM_ZC_NOTIFICATION;
+
+	zc_info = (struct zc_info *)CMSG_DATA(cm);
+	zc_info->size = cfg_notification_limit;
+
+	added_zcopy_info = true;
+}
+
+static bool do_sendmsg(int fd, struct msghdr *msg,
+		       enum notification_type do_zerocopy, int domain)
 {
 	int ret, len, i, flags;
 	static uint32_t cookie;
@@ -200,6 +232,12 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
 			msg->msg_controllen = CMSG_SPACE(sizeof(cookie));
 			msg->msg_control = (struct cmsghdr *)ckbuf;
 			add_zcopy_cookie(msg, ++cookie);
+		} else if (do_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG &&
+			   sends_since_notify + 1 >= cfg_notification_limit) {
+			memset(&msg->msg_control, 0, sizeof(msg->msg_control));
+			msg->msg_controllen = CMSG_SPACE(zc_info_size);
+			msg->msg_control = (struct cmsghdr *)zc_ckbuf;
+			add_zcopy_info(msg);
 		}
 	}
 
@@ -218,7 +256,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
 		if (do_zerocopy && ret)
 			expected_completions++;
 	}
-	if (do_zerocopy && domain == PF_RDS) {
+	if (msg->msg_control) {
 		msg->msg_control = NULL;
 		msg->msg_controllen = 0;
 	}
@@ -466,6 +504,44 @@ static void do_recv_completions(int fd, int domain)
 	sends_since_notify = 0;
 }
 
+static void do_recv_completions2(void)
+{
+	struct cmsghdr *cm = (struct cmsghdr *)zc_ckbuf;
+	struct zc_info *zc_info;
+	__u32 hi, lo, range;
+	__u8 zerocopy;
+	int i;
+
+	zc_info = (struct zc_info *)CMSG_DATA(cm);
+	for (i = 0; i < zc_info->size; i++) {
+		hi = zc_info->arr[i].hi;
+		lo = zc_info->arr[i].lo;
+		zerocopy = zc_info->arr[i].zerocopy;
+		range = hi - lo + 1;
+
+		if (cfg_verbose && lo != next_completion)
+			fprintf(stderr, "gap: %u..%u does not append to %u\n",
+				lo, hi, next_completion);
+		next_completion = hi + 1;
+
+		if (zerocopied == -1)
+			zerocopied = zerocopy;
+		else if (zerocopied != zerocopy) {
+			fprintf(stderr, "serr: inconsistent\n");
+			zerocopied = zerocopy;
+		}
+
+		completions += range;
+
+		if (cfg_verbose >= 2)
+			fprintf(stderr, "completed: %u (h=%u l=%u)\n",
+				range, hi, lo);
+	}
+
+	sends_since_notify = 0;
+	added_zcopy_info = false;
+}
+
 /* Wait for all remaining completions on the errqueue */
 static void do_recv_remaining_completions(int fd, int domain)
 {
@@ -541,6 +617,14 @@ static void do_tx(int domain, int type, int protocol)
 				    sizeof(struct sockaddr_in6));
 	}
 
+	if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) {
+		zc_info_size = sizeof(struct zc_info) +
+			       sizeof(struct zc_info_elem) * cfg_notification_limit;
+		zc_ckbuf = (char *)malloc(CMSG_SPACE(zc_info_size));
+		if (!zc_ckbuf)
+			error(1, errno, "zc_ckbuf malloc failed");
+	}
+
 	iov[2].iov_base = payload;
 	iov[2].iov_len = cfg_payload_len;
 	msg.msg_iovlen++;
@@ -553,11 +637,16 @@ static void do_tx(int domain, int type, int protocol)
 		else
 			do_sendmsg(fd, &msg, cfg_zerocopy, domain);
 
-		if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit)
+		if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE &&
+		    sends_since_notify >= cfg_notification_limit)
 			do_recv_completions(fd, domain);
 
+		if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG &&
+		    added_zcopy_info)
+			do_recv_completions2();
+
 		while (!do_poll(fd, POLLOUT)) {
-			if (cfg_zerocopy)
+			if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE)
 				do_recv_completions(fd, domain);
 		}
 
@@ -566,6 +655,9 @@ static void do_tx(int domain, int type, int protocol)
 	if (cfg_zerocopy)
 		do_recv_remaining_completions(fd, domain);
 
+	if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG)
+		free(zc_ckbuf);
+
 	if (close(fd))
 		error(1, errno, "close");
 
@@ -715,7 +807,7 @@ static void parse_opts(int argc, char **argv)
 
 	cfg_payload_len = max_payload_len;
 
-	while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) {
+	while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) {
 		switch (c) {
 		case '4':
 			if (cfg_family != PF_UNSPEC)
@@ -749,6 +841,9 @@ static void parse_opts(int argc, char **argv)
 		case 'm':
 			cfg_cork_mixed = true;
 			break;
+		case 'n':
+			cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_SENDMSG;
+			break;
 		case 'p':
 			cfg_port = strtoul(optarg, NULL, 0);
 			break;
@@ -768,7 +863,7 @@ static void parse_opts(int argc, char **argv)
 			cfg_verbose++;
 			break;
 		case 'z':
-			cfg_zerocopy = true;
+			cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_ERRQUEUE;
 			break;
 		}
 	}
@@ -779,6 +874,8 @@ static void parse_opts(int argc, char **argv)
 			error(1, 0, "-D <server addr> required for PF_RDS\n");
 		if (!cfg_rx && !saddr)
 			error(1, 0, "-S <client addr> required for PF_RDS\n");
+		if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG)
+			error(1, 0, "PF_RDS does not support MSG_ZEROCOPY_NOTIFY_SENDMSG");
 	}
 	setup_sockaddr(cfg_family, daddr, &cfg_dst_addr);
 	setup_sockaddr(cfg_family, saddr, &cfg_src_addr);
diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh
index 89c22f5320e0..022a6936d86f 100755
--- a/tools/testing/selftests/net/msg_zerocopy.sh
+++ b/tools/testing/selftests/net/msg_zerocopy.sh
@@ -118,4 +118,5 @@ do_test() {
 
 do_test "${EXTRA_ARGS}"
 do_test "-z ${EXTRA_ARGS}"
+do_test "-n ${EXTRA_ARGS}"
 echo ok
-- 
2.20.1

Re: [PATCH net-next v7 3/3] selftests: add MSG_ZEROCOPY msg_control notification test

[Simon Horman]

+ Dave Miller, Jakub Kicinski, Paolo Abeni, Shuah Khan,
  linux-kselftest@vger.kernel.org

On Mon, Jul 08, 2024 at 09:04:05PM +0000, zijianzhang@bytedance.com wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
> 
> We update selftests/net/msg_zerocopy.c to accommodate the new mechanism,
> cfg_notification_limit has the same semantics for both methods. Test
> results are as follows, we update skb_orphan_frags_rx to the same as
> skb_orphan_frags to support zerocopy in the localhost test.
> 
> cfg_notification_limit = 1, both method get notifications after 1 calling
> of sendmsg. In this case, the new method has around 17% cpu savings in TCP
> and 23% cpu savings in UDP.
> +---------------------+---------+---------+---------+---------+
> | Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
> +---------------------+---------+---------+---------+---------+
> | ZCopy (MB)          | 7523    | 7706    | 7489    | 7304    |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy (MB)      | 8834    | 8993    | 9053    | 9228    |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy / ZCopy   | 117.42% | 116.70% | 120.88% | 126.34% |
> +---------------------+---------+---------+---------+---------+
> 
> cfg_notification_limit = 32, both get notifications after 32 calling of
> sendmsg, which means more chances to coalesce notifications, and less
> overhead of poll + recvmsg for the original method. In this case, the new
> method has around 7% cpu savings in TCP and slightly better cpu usage in
> UDP. In the context of selftest, notifications of TCP are more likely to
> out of order than UDP, it's easier to coalesce more notifications in UDP.
> The original method can get one notification with range of 32 in a recvmsg
> most of the time. In TCP, most notifications' range is around 2, so the
> original method needs around 16 recvmsgs to get notified in one round.
> That's the reason for the "New ZCopy / ZCopy" diff in TCP and UDP here.
> +---------------------+---------+---------+---------+---------+
> | Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
> +---------------------+---------+---------+---------+---------+
> | ZCopy (MB)          | 8842    | 8735    | 10072   | 9380    |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy (MB)      | 9366    | 9477    | 10108   | 9385    |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy / ZCopy   | 106.00% | 108.28% | 100.31% | 100.01% |
> +---------------------+---------+---------+---------+---------+
> 
> In conclusion, when notification interval is small or notifications are
> hard to be coalesced, the new mechanism is highly recommended. Otherwise,
> the performance gain from the new mechanism is very limited.
> 
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> ---
>  tools/testing/selftests/net/msg_zerocopy.c  | 111 ++++++++++++++++++--
>  tools/testing/selftests/net/msg_zerocopy.sh |   1 +
>  2 files changed, 105 insertions(+), 7 deletions(-)
> 
> diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c

...

> @@ -466,6 +504,44 @@ static void do_recv_completions(int fd, int domain)
>  	sends_since_notify = 0;
>  }
>  
> +static void do_recv_completions2(void)
> +{
> +	struct cmsghdr *cm = (struct cmsghdr *)zc_ckbuf;
> +	struct zc_info *zc_info;
> +	__u32 hi, lo, range;
> +	__u8 zerocopy;
> +	int i;
> +
> +	zc_info = (struct zc_info *)CMSG_DATA(cm);
> +	for (i = 0; i < zc_info->size; i++) {
> +		hi = zc_info->arr[i].hi;
> +		lo = zc_info->arr[i].lo;
> +		zerocopy = zc_info->arr[i].zerocopy;
> +		range = hi - lo + 1;
> +
> +		if (cfg_verbose && lo != next_completion)
> +			fprintf(stderr, "gap: %u..%u does not append to %u\n",
> +				lo, hi, next_completion);
> +		next_completion = hi + 1;
> +
> +		if (zerocopied == -1)
> +			zerocopied = zerocopy;
> +		else if (zerocopied != zerocopy) {
> +			fprintf(stderr, "serr: inconsistent\n");
> +			zerocopied = zerocopy;
> +		}

nit: If any arms of a conditional have {}, then all arms should have them

> +
> +		completions += range;
> +
> +		if (cfg_verbose >= 2)
> +			fprintf(stderr, "completed: %u (h=%u l=%u)\n",
> +				range, hi, lo);
> +	}
> +
> +	sends_since_notify = 0;
> +	added_zcopy_info = false;
> +}

...