[PATCH net-next v3 0/3] net: A lightweight zero-copy notification

[PATCH net-next v3 0/3] net: A lightweight zero-copy notification

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

Original title is "net: socket sendmsg MSG_ZEROCOPY_UARG". 

Original notification mechanism needs poll + recvmmsg which is not
easy for applcations to accommodate. And, it also incurs unignorable
overhead including extra system calls and usage of optmem.

While making maximum reuse of the existing MSG_ZEROCOPY related code,
this patch set introduces a new zerocopy socket notification mechanism.
Users of sendmsg pass a control message as a placeholder for the incoming
notifications. Upon returning, kernel embeds notifications directly into
user arguments passed in. By doing so, we can significantly reduce the
complexity and overhead for managing notifications. In an ideal pattern,
the user will keep calling sendmsg with SCM_ZC_NOTIFICATION msg_control,
and the notification will be delivered as soon as possible.

Changelog:
  v1 -> v2:
    - Reuse errormsg queue in the new notification mechanism,
      users can actually use these two mechanisms in hybrid way
      if they want to do so.
    - Update case SCM_ZC_NOTIFICATION in __sock_cmsg_send
      1. Regardless of 32-bit, 64-bit program, we will always handle
      u64 type user address.
      2. The size of data to copy_to_user is precisely calculated
      in case of kernel stack leak.
    - fix (kbuild-bot)
      1. Add SCM_ZC_NOTIFICATION to arch-specific header files.
      2. header file types.h in include/uapi/linux/socket.h

  v2 -> v3:
    - 1. Users can now pass in the address of the zc_info_elem directly
      with appropriate cmsg_len instead of the ugly user interface. Plus,
      the handler is now compatible with MSG_CMSG_COMPAT and 32-bit
      pointer.
    - 2. Suggested by Willem, another strategy of getting zc info is
      briefly taking the lock of sk_error_queue and move to a private
      list, like net_rx_action. I thought sk_error_queue is protected by
      sock_lock, so that it's impossible for the handling of zc info and
      users recvmsg from the sk_error_queue at the same time.
      However, sk_error_queue is protected by its own lock. I am afraid
      that during the time it is handling the private list, users may
      fail to get other error messages in the queue via recvmsg. Thus,
      I don't implement the splice logic in this version. Any comments?


* Performance

I extend the selftests/msg_zerocopy.c to accommodate the new mechanism,
test result is as follows,

cfg_notification_limit = 1, in this case the original method approximately
aligns with the semantics of new one. In this case, the new flag has
around 13% cpu savings in TCP and 18% cpu savings in UDP.

+---------------------+---------+---------+---------+---------+
| Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+---------------------+---------+---------+---------+---------+
| ZCopy (MB)          | 5147    | 4885    | 7489    | 7854    |
+---------------------+---------+---------+---------+---------+
| New ZCopy (MB)      | 5859    | 5505    | 9053    | 9236    |
+---------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy   | 113.83% | 112.69% | 120.88% | 117.59% |
+---------------------+---------+---------+---------+---------+


cfg_notification_limit = 32, the new mechanism performs 8% better in TCP.
For UDP, no obvious performance gain is observed and sometimes may lead
to degradation. Thus, if users don't need to retrieve the notification
ASAP in UDP, the original mechanism is preferred.

+---------------------+---------+---------+---------+---------+
| Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+---------------------+---------+---------+---------+---------+
| ZCopy (MB)          | 6272    | 6138    | 12138   | 10055   |
+---------------------+---------+---------+---------+---------+
| New ZCopy (MB)      | 6774    | 6620    | 11504   | 10355   |
+---------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy   | 108.00% | 107.85% | 94.78%  | 102.98% |
+---------------------+---------+---------+---------+---------+

Zijian Zhang (3):
  selftests: fix OOM problem in msg_zerocopy selftest
  sock: add MSG_ZEROCOPY notification mechanism based on msg_control
  selftests: add MSG_ZEROCOPY msg_control notification test

 arch/alpha/include/uapi/asm/socket.h        |   2 +
 arch/mips/include/uapi/asm/socket.h         |   2 +
 arch/parisc/include/uapi/asm/socket.h       |   2 +
 arch/sparc/include/uapi/asm/socket.h        |   2 +
 include/uapi/asm-generic/socket.h           |   2 +
 include/uapi/linux/socket.h                 |  10 ++
 net/core/sock.c                             |  68 ++++++++++++
 tools/testing/selftests/net/msg_zerocopy.c  | 114 ++++++++++++++++++--
 tools/testing/selftests/net/msg_zerocopy.sh |   1 +
 9 files changed, 196 insertions(+), 7 deletions(-)

-- 
2.20.1

[PATCH net-next v3 1/3] selftests: fix OOM problem in msg_zerocopy selftest

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

In selftests/net/msg_zerocopy.c, it has a while loop keeps calling sendmsg
on a socket with MSG_ZEROCOPY flag, and it will recv the notifications
until the socket is not writable. Typically, it will start the receiving
process after around 30+ sendmsgs. However, because of the
commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale")
the sender is always writable and does not get any chance to run recv
notifications. The selftest always exits with OUT_OF_MEMORY because the
memory used by opt_skb exceeds the core.sysctl_optmem_max.

According to our experiments, this problem can be mitigated by open the
DEBUG_LOCKDEP configuration for the kernel. But it will makes the
notifications disordered even in good commits before
commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale").

We introduce "cfg_notification_limit" to force sender to receive
notifications after some number of sendmsgs. And, notifications may not
come in order, because of the reason we present above. We have order
checking code managed by cfg_verbose.

Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
---
 tools/testing/selftests/net/msg_zerocopy.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c
index bdc03a2097e8..ba6c257f689c 100644
--- a/tools/testing/selftests/net/msg_zerocopy.c
+++ b/tools/testing/selftests/net/msg_zerocopy.c
@@ -85,6 +85,7 @@ static bool cfg_rx;
 static int  cfg_runtime_ms	= 4200;
 static int  cfg_verbose;
 static int  cfg_waittime_ms	= 500;
+static int  cfg_notification_limit = 32;
 static bool cfg_zerocopy;
 
 static socklen_t cfg_alen;
@@ -95,6 +96,8 @@ static char payload[IP_MAXPACKET];
 static long packets, bytes, completions, expected_completions;
 static int  zerocopied = -1;
 static uint32_t next_completion;
+/* The number of sendmsgs which have not received notified yet */
+static uint32_t sendmsg_counter;
 
 static unsigned long gettimeofday_ms(void)
 {
@@ -208,6 +211,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
 		error(1, errno, "send");
 	if (cfg_verbose && ret != len)
 		fprintf(stderr, "send: ret=%u != %u\n", ret, len);
+	sendmsg_counter++;
 
 	if (len) {
 		packets++;
@@ -435,7 +439,7 @@ static bool do_recv_completion(int fd, int domain)
 	/* Detect notification gaps. These should not happen often, if at all.
 	 * Gaps can occur due to drops, reordering and retransmissions.
 	 */
-	if (lo != next_completion)
+	if (cfg_verbose && lo != next_completion)
 		fprintf(stderr, "gap: %u..%u does not append to %u\n",
 			lo, hi, next_completion);
 	next_completion = hi + 1;
@@ -460,6 +464,7 @@ static bool do_recv_completion(int fd, int domain)
 static void do_recv_completions(int fd, int domain)
 {
 	while (do_recv_completion(fd, domain)) {}
+	sendmsg_counter = 0;
 }
 
 /* Wait for all remaining completions on the errqueue */
@@ -549,6 +554,9 @@ static void do_tx(int domain, int type, int protocol)
 		else
 			do_sendmsg(fd, &msg, cfg_zerocopy, domain);
 
+		if (cfg_zerocopy && sendmsg_counter >= cfg_notification_limit)
+			do_recv_completions(fd, domain);
+
 		while (!do_poll(fd, POLLOUT)) {
 			if (cfg_zerocopy)
 				do_recv_completions(fd, domain);
@@ -708,7 +716,7 @@ static void parse_opts(int argc, char **argv)
 
 	cfg_payload_len = max_payload_len;
 
-	while ((c = getopt(argc, argv, "46c:C:D:i:mp:rs:S:t:vz")) != -1) {
+	while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) {
 		switch (c) {
 		case '4':
 			if (cfg_family != PF_UNSPEC)
@@ -736,6 +744,9 @@ static void parse_opts(int argc, char **argv)
 			if (cfg_ifindex == 0)
 				error(1, errno, "invalid iface: %s", optarg);
 			break;
+		case 'l':
+			cfg_notification_limit = strtoul(optarg, NULL, 0);
+			break;
 		case 'm':
 			cfg_cork_mixed = true;
 			break;
-- 
2.20.1

[PATCH net-next v3 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
However, zerocopy is not a free lunch. Apart from the management of user
pages, the combination of poll + recvmsg to receive notifications incurs
unignorable overhead in the applications. The overhead of such sometimes
might be more than the CPU savings from zerocopy. We try to solve this
problem with a new notification mechanism based on msgcontrol.
This new mechanism aims to reduce the overhead associated with receiving
notifications by embedding them directly into user arguments passed with
each sendmsg control message. By doing so, we can significantly reduce
the complexity and overhead for managing notifications. In an ideal
pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION
msg_control, and the notification will be delivered as soon as possible.

Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
---
 arch/alpha/include/uapi/asm/socket.h  |  2 +
 arch/mips/include/uapi/asm/socket.h   |  2 +
 arch/parisc/include/uapi/asm/socket.h |  2 +
 arch/sparc/include/uapi/asm/socket.h  |  2 +
 include/uapi/asm-generic/socket.h     |  2 +
 include/uapi/linux/socket.h           | 10 ++++
 net/core/sock.c                       | 68 +++++++++++++++++++++++++++
 7 files changed, 88 insertions(+)

diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h
index e94f621903fe..7761a4e0ea2c 100644
--- a/arch/alpha/include/uapi/asm/socket.h
+++ b/arch/alpha/include/uapi/asm/socket.h
@@ -140,6 +140,8 @@
 #define SO_PASSPIDFD		76
 #define SO_PEERPIDFD		77
 
+#define SCM_ZC_NOTIFICATION 78
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h
index 60ebaed28a4c..89edc51380f0 100644
--- a/arch/mips/include/uapi/asm/socket.h
+++ b/arch/mips/include/uapi/asm/socket.h
@@ -151,6 +151,8 @@
 #define SO_PASSPIDFD		76
 #define SO_PEERPIDFD		77
 
+#define SCM_ZC_NOTIFICATION 78
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h
index be264c2b1a11..2911b43e6a9d 100644
--- a/arch/parisc/include/uapi/asm/socket.h
+++ b/arch/parisc/include/uapi/asm/socket.h
@@ -132,6 +132,8 @@
 #define SO_PASSPIDFD		0x404A
 #define SO_PEERPIDFD		0x404B
 
+#define SCM_ZC_NOTIFICATION 0x404C
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h
index 682da3714686..dc045e87cc8e 100644
--- a/arch/sparc/include/uapi/asm/socket.h
+++ b/arch/sparc/include/uapi/asm/socket.h
@@ -133,6 +133,8 @@
 #define SO_PASSPIDFD             0x0055
 #define SO_PEERPIDFD             0x0056
 
+#define SCM_ZC_NOTIFICATION 0x0057
+
 #if !defined(__KERNEL__)
 
 
diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h
index 8ce8a39a1e5f..7474c8a244bc 100644
--- a/include/uapi/asm-generic/socket.h
+++ b/include/uapi/asm-generic/socket.h
@@ -135,6 +135,8 @@
 #define SO_PASSPIDFD		76
 #define SO_PEERPIDFD		77
 
+#define SCM_ZC_NOTIFICATION 78
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h
index d3fcd3b5ec53..15cec8819f34 100644
--- a/include/uapi/linux/socket.h
+++ b/include/uapi/linux/socket.h
@@ -2,6 +2,8 @@
 #ifndef _UAPI_LINUX_SOCKET_H
 #define _UAPI_LINUX_SOCKET_H
 
+#include <linux/types.h>
+
 /*
  * Desired design of maximum size and alignment (see RFC2553)
  */
@@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage {
 #define SOCK_TXREHASH_DISABLED	0
 #define SOCK_TXREHASH_ENABLED	1
 
+#define SOCK_ZC_INFO_MAX 128
+
+struct zc_info_elem {
+	__u32 lo;
+	__u32 hi;
+	__u8 zerocopy;
+};
+
 #endif /* _UAPI_LINUX_SOCKET_H */
diff --git a/net/core/sock.c b/net/core/sock.c
index 8d6e638b5426..15da609be026 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2842,6 +2842,74 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
 	case SCM_RIGHTS:
 	case SCM_CREDENTIALS:
 		break;
+	case SCM_ZC_NOTIFICATION: {
+		int ret, i = 0;
+		int cmsg_data_len, zc_info_elem_num;
+		void __user	*usr_addr;
+		struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX];
+		unsigned long flags;
+		struct sk_buff_head *q, local_q;
+		struct sk_buff *skb, *tmp;
+		struct sock_exterr_skb *serr;
+
+		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
+			return -EINVAL;
+
+		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
+		if (cmsg_data_len % sizeof(struct zc_info_elem))
+			return -EINVAL;
+
+		zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem);
+		if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX)
+			return -EINVAL;
+
+		if (in_compat_syscall())
+			usr_addr = compat_ptr(*(compat_uptr_t *)CMSG_DATA(cmsg));
+		else
+			usr_addr = (void __user *)*(void **)CMSG_DATA(cmsg);
+		if (!access_ok(usr_addr, cmsg_data_len))
+			return -EFAULT;
+
+		q = &sk->sk_error_queue;
+		skb_queue_head_init(&local_q);
+		spin_lock_irqsave(&q->lock, flags);
+		skb = skb_peek(q);
+		while (skb && i < zc_info_elem_num) {
+			struct sk_buff *skb_next = skb_peek_next(skb, q);
+
+			serr = SKB_EXT_ERR(skb);
+			if (serr->ee.ee_errno == 0 &&
+			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
+				zc_info_kern[i].hi = serr->ee.ee_data;
+				zc_info_kern[i].lo = serr->ee.ee_info;
+				zc_info_kern[i].zerocopy = !(serr->ee.ee_code
+								& SO_EE_CODE_ZEROCOPY_COPIED);
+				__skb_unlink(skb, q);
+				__skb_queue_tail(&local_q, skb);
+				i++;
+			}
+			skb = skb_next;
+		}
+		spin_unlock_irqrestore(&q->lock, flags);
+
+		ret = copy_to_user(usr_addr,
+				   zc_info_kern,
+					i * sizeof(struct zc_info_elem));
+
+		if (unlikely(ret)) {
+			spin_lock_irqsave(&q->lock, flags);
+			skb_queue_reverse_walk_safe(&local_q, skb, tmp) {
+				__skb_unlink(skb, &local_q);
+				__skb_queue_head(q, skb);
+			}
+			spin_unlock_irqrestore(&q->lock, flags);
+			return -EFAULT;
+		}
+
+		while ((skb = __skb_dequeue(&local_q)))
+			consume_skb(skb);
+		break;
+	}
 	default:
 		return -EINVAL;
 	}
-- 
2.20.1

[PATCH net-next v3 3/3] selftests: add MSG_ZEROCOPY msg_control notification test

[zijianzhang]

From: Zijian Zhang <zijianzhang@bytedance.com>

We update selftests/net/msg_zerocopy.c to accommodate the new mechanism.

Test result from selftests/net/msg_zerocopy.c,
cfg_notification_limit = 1, in this case MSG_ZEROCOPY approximately
aligns with the semantics of MSG_ZEROCOPY_UARG. In this case, the new
flag has around 13% cpu savings in TCP and 18% cpu savings in UDP.
+---------------------+---------+---------+---------+---------+
| Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+---------------------+---------+---------+---------+---------+
| ZCopy (MB)          | 5147    | 4885    | 7489    | 7854    |
+---------------------+---------+---------+---------+---------+
| New ZCopy (MB)      | 5859    | 5505    | 9053    | 9236    |
+---------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy   | 113.83% | 112.69% | 120.88% | 117.59% |
+---------------------+---------+---------+---------+---------+

cfg_notification_limit = 32, it means less poll + recvmsg overhead,
the new mechanism performs 8% better in TCP. For UDP, no obvious
performance gain is observed and sometimes may lead to degradation.
Thus, if users don't need to retrieve the notification ASAP in UDP,
the original mechanism is preferred.
+---------------------+---------+---------+---------+---------+
| Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
+---------------------+---------+---------+---------+---------+
| ZCopy (MB)          | 6272    | 6138    | 12138   | 10055   |
+---------------------+---------+---------+---------+---------+
| New ZCopy (MB)      | 6774    | 6620    | 11504   | 10355   |
+---------------------+---------+---------+---------+---------+
| New ZCopy / ZCopy   | 108.00% | 107.85% | 94.78%  | 102.98% |
+---------------------+---------+---------+---------+---------+

Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
---
 tools/testing/selftests/net/msg_zerocopy.c  | 103 ++++++++++++++++++--
 tools/testing/selftests/net/msg_zerocopy.sh |   1 +
 2 files changed, 97 insertions(+), 7 deletions(-)

diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c
index ba6c257f689c..48750a7a162c 100644
--- a/tools/testing/selftests/net/msg_zerocopy.c
+++ b/tools/testing/selftests/net/msg_zerocopy.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
 /* Evaluate MSG_ZEROCOPY
  *
  * Send traffic between two processes over one of the supported
@@ -66,6 +67,10 @@
 #define SO_ZEROCOPY	60
 #endif
 
+#ifndef SCM_ZC_NOTIFICATION
+#define SCM_ZC_NOTIFICATION	78
+#endif
+
 #ifndef SO_EE_CODE_ZEROCOPY_COPIED
 #define SO_EE_CODE_ZEROCOPY_COPIED	1
 #endif
@@ -74,6 +79,13 @@
 #define MSG_ZEROCOPY	0x4000000
 #endif
 
+#define ZC_MSGERR_NOTIFICATION 1
+#define ZC_MSGCTL_NOTIFICATION 2
+
+#define SOCK_ZC_INFO_NUM 8
+
+#define INVALID_ZEROCOPY_VAL 2
+
 static int  cfg_cork;
 static bool cfg_cork_mixed;
 static int  cfg_cpu		= -1;		/* default: pin to last cpu */
@@ -86,13 +98,16 @@ static int  cfg_runtime_ms	= 4200;
 static int  cfg_verbose;
 static int  cfg_waittime_ms	= 500;
 static int  cfg_notification_limit = 32;
-static bool cfg_zerocopy;
+static int  cfg_zerocopy;           /* Either 1 or 2, mode for SO_ZEROCOPY */
 
 static socklen_t cfg_alen;
 static struct sockaddr_storage cfg_dst_addr;
 static struct sockaddr_storage cfg_src_addr;
 
 static char payload[IP_MAXPACKET];
+static struct zc_info_elem zc_info[SOCK_ZC_INFO_NUM];
+static char zc_ckbuf[CMSG_SPACE(sizeof(void *))];
+static struct zc_info_elem *zc_info_ptr = zc_info;
 static long packets, bytes, completions, expected_completions;
 static int  zerocopied = -1;
 static uint32_t next_completion;
@@ -170,6 +185,26 @@ static int do_accept(int fd)
 	return fd;
 }
 
+static void add_zcopy_info(struct msghdr *msg)
+{
+	int i;
+	struct cmsghdr *cm;
+
+	if (!msg->msg_control)
+		error(1, errno, "NULL user arg");
+	cm = (void *)msg->msg_control;
+	/* Although only the address of the array will be written into the
+	 * zc_ckbuf, we assign cmsg_len to CMSG_LEN(sizeof(zc_info)) to specify
+	 * the length of the array.
+	 */
+	cm->cmsg_len = CMSG_LEN(sizeof(zc_info));
+	cm->cmsg_level = SOL_SOCKET;
+	cm->cmsg_type = SCM_ZC_NOTIFICATION;
+	memcpy(CMSG_DATA(cm), &zc_info_ptr, sizeof(zc_info_ptr));
+	for (i = 0; i < SOCK_ZC_INFO_NUM; i++)
+		zc_info[i].zerocopy = INVALID_ZEROCOPY_VAL;
+}
+
 static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie)
 {
 	struct cmsghdr *cm;
@@ -183,7 +218,7 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie)
 	memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie));
 }
 
-static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
+static bool do_sendmsg(int fd, struct msghdr *msg, int do_zerocopy, int domain)
 {
 	int ret, len, i, flags;
 	static uint32_t cookie;
@@ -201,6 +236,15 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
 			msg->msg_controllen = CMSG_SPACE(sizeof(cookie));
 			msg->msg_control = (struct cmsghdr *)ckbuf;
 			add_zcopy_cookie(msg, ++cookie);
+		} else if (do_zerocopy == ZC_MSGCTL_NOTIFICATION) {
+			memset(&msg->msg_control, 0, sizeof(msg->msg_control));
+			/* Although only the address of the array will be written into the
+			 * zc_ckbuf, msg_controllen must be larger or equal than any cmsg_len
+			 * in it. Otherwise, we will get -EINVAL.
+			 */
+			msg->msg_controllen = CMSG_SPACE(sizeof(zc_info));
+			msg->msg_control = (struct cmsghdr *)zc_ckbuf;
+			add_zcopy_info(msg);
 		}
 	}
 
@@ -219,7 +263,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
 		if (do_zerocopy && ret)
 			expected_completions++;
 	}
-	if (do_zerocopy && domain == PF_RDS) {
+	if (msg->msg_control) {
 		msg->msg_control = NULL;
 		msg->msg_controllen = 0;
 	}
@@ -393,6 +437,42 @@ static bool do_recvmsg_completion(int fd)
 	return ret;
 }
 
+static void do_recv_completions2(void)
+{
+	int i;
+	__u32 hi, lo, range;
+	__u8 zerocopy;
+
+	for (i = 0; zc_info[i].zerocopy != INVALID_ZEROCOPY_VAL; i++) {
+		struct zc_info_elem elem = zc_info[i];
+
+		hi = elem.hi;
+		lo = elem.lo;
+		zerocopy = elem.zerocopy;
+		range = hi - lo + 1;
+
+		if (cfg_verbose && lo != next_completion)
+			fprintf(stderr, "gap: %u..%u does not append to %u\n",
+				lo, hi, next_completion);
+		next_completion = hi + 1;
+
+		if (zerocopied == -1)
+			zerocopied = zerocopy;
+		else if (zerocopied != zerocopy) {
+			fprintf(stderr, "serr: inconsistent\n");
+			zerocopied = zerocopy;
+		}
+
+		completions += range;
+
+		if (cfg_verbose >= 2)
+			fprintf(stderr, "completed: %u (h=%u l=%u)\n",
+				range, hi, lo);
+	}
+
+	sendmsg_counter -= i;
+}
+
 static bool do_recv_completion(int fd, int domain)
 {
 	struct sock_extended_err *serr;
@@ -554,11 +634,15 @@ static void do_tx(int domain, int type, int protocol)
 		else
 			do_sendmsg(fd, &msg, cfg_zerocopy, domain);
 
-		if (cfg_zerocopy && sendmsg_counter >= cfg_notification_limit)
+		if (cfg_zerocopy == ZC_MSGERR_NOTIFICATION &&
+			sendmsg_counter >= cfg_notification_limit)
 			do_recv_completions(fd, domain);
 
+		if (cfg_zerocopy == ZC_MSGCTL_NOTIFICATION)
+			do_recv_completions2();
+
 		while (!do_poll(fd, POLLOUT)) {
-			if (cfg_zerocopy)
+			if (cfg_zerocopy == ZC_MSGERR_NOTIFICATION)
 				do_recv_completions(fd, domain);
 		}
 
@@ -716,7 +800,7 @@ static void parse_opts(int argc, char **argv)
 
 	cfg_payload_len = max_payload_len;
 
-	while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) {
+	while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) {
 		switch (c) {
 		case '4':
 			if (cfg_family != PF_UNSPEC)
@@ -750,6 +834,9 @@ static void parse_opts(int argc, char **argv)
 		case 'm':
 			cfg_cork_mixed = true;
 			break;
+		case 'n':
+			cfg_zerocopy = ZC_MSGCTL_NOTIFICATION;
+			break;
 		case 'p':
 			cfg_port = strtoul(optarg, NULL, 0);
 			break;
@@ -769,7 +856,7 @@ static void parse_opts(int argc, char **argv)
 			cfg_verbose++;
 			break;
 		case 'z':
-			cfg_zerocopy = true;
+			cfg_zerocopy = ZC_MSGERR_NOTIFICATION;
 			break;
 		}
 	}
@@ -780,6 +867,8 @@ static void parse_opts(int argc, char **argv)
 			error(1, 0, "-D <server addr> required for PF_RDS\n");
 		if (!cfg_rx && !saddr)
 			error(1, 0, "-S <client addr> required for PF_RDS\n");
+		if (cfg_zerocopy == ZC_MSGCTL_NOTIFICATION)
+			error(1, 0, "PF_RDS does not support ZC_MSGCTL_NOTIFICATION");
 	}
 	setup_sockaddr(cfg_family, daddr, &cfg_dst_addr);
 	setup_sockaddr(cfg_family, saddr, &cfg_src_addr);
diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh
index 89c22f5320e0..022a6936d86f 100755
--- a/tools/testing/selftests/net/msg_zerocopy.sh
+++ b/tools/testing/selftests/net/msg_zerocopy.sh
@@ -118,4 +118,5 @@ do_test() {
 
 do_test "${EXTRA_ARGS}"
 do_test "-z ${EXTRA_ARGS}"
+do_test "-n ${EXTRA_ARGS}"
 echo ok
-- 
2.20.1

Re: [PATCH net-next v3 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[kernel test robot]

Hi,

kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/zijianzhang-bytedance-com/selftests-fix-OOM-problem-in-msg_zerocopy-selftest/20240511-000153
base:   net-next/main
patch link:    https://lore.kernel.org/r/20240510155900.1825946-3-zijianzhang%40bytedance.com
patch subject: [PATCH net-next v3 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control
config: arm-defconfig (https://download.01.org/0day-ci/archive/20240511/202405111306.MOClscNA-lkp@intel.com/config)
compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project f28c006a5895fc0e329fe15fead81e37457cb1d1)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240511/202405111306.MOClscNA-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202405111306.MOClscNA-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> net/core/sock.c:2808:5: warning: stack frame size (1608) exceeds limit (1024) in '__sock_cmsg_send' [-Wframe-larger-than]
   int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
       ^
   1 warning generated.


vim +/__sock_cmsg_send +2808 net/core/sock.c

^1da177e4c3f41 Linus Torvalds        2005-04-16  2807  
233baf9a1bc46f xu xin                2022-10-20 @2808  int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
f28ea365cdefc3 Edward Jee            2015-10-08  2809  		     struct sockcm_cookie *sockc)
f28ea365cdefc3 Edward Jee            2015-10-08  2810  {
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2811  	u32 tsflags;
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2812  
f28ea365cdefc3 Edward Jee            2015-10-08  2813  	switch (cmsg->cmsg_type) {
f28ea365cdefc3 Edward Jee            2015-10-08  2814  	case SO_MARK:
91f0d8a4813a9a Jakub Kicinski        2022-01-31  2815  		if (!ns_capable(sock_net(sk)->user_ns, CAP_NET_RAW) &&
91f0d8a4813a9a Jakub Kicinski        2022-01-31  2816  		    !ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN))
f28ea365cdefc3 Edward Jee            2015-10-08  2817  			return -EPERM;
f28ea365cdefc3 Edward Jee            2015-10-08  2818  		if (cmsg->cmsg_len != CMSG_LEN(sizeof(u32)))
f28ea365cdefc3 Edward Jee            2015-10-08  2819  			return -EINVAL;
f28ea365cdefc3 Edward Jee            2015-10-08  2820  		sockc->mark = *(u32 *)CMSG_DATA(cmsg);
f28ea365cdefc3 Edward Jee            2015-10-08  2821  		break;
7f1bc6e95d7840 Deepa Dinamani        2019-02-02  2822  	case SO_TIMESTAMPING_OLD:
382a32018b74f4 Thomas Lange          2024-01-04  2823  	case SO_TIMESTAMPING_NEW:
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2824  		if (cmsg->cmsg_len != CMSG_LEN(sizeof(u32)))
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2825  			return -EINVAL;
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2826  
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2827  		tsflags = *(u32 *)CMSG_DATA(cmsg);
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2828  		if (tsflags & ~SOF_TIMESTAMPING_TX_RECORD_MASK)
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2829  			return -EINVAL;
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2830  
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2831  		sockc->tsflags &= ~SOF_TIMESTAMPING_TX_RECORD_MASK;
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2832  		sockc->tsflags |= tsflags;
3dd17e63f5131b Soheil Hassas Yeganeh 2016-04-02  2833  		break;
80b14dee2bea12 Richard Cochran       2018-07-03  2834  	case SCM_TXTIME:
80b14dee2bea12 Richard Cochran       2018-07-03  2835  		if (!sock_flag(sk, SOCK_TXTIME))
80b14dee2bea12 Richard Cochran       2018-07-03  2836  			return -EINVAL;
80b14dee2bea12 Richard Cochran       2018-07-03  2837  		if (cmsg->cmsg_len != CMSG_LEN(sizeof(u64)))
80b14dee2bea12 Richard Cochran       2018-07-03  2838  			return -EINVAL;
80b14dee2bea12 Richard Cochran       2018-07-03  2839  		sockc->transmit_time = get_unaligned((u64 *)CMSG_DATA(cmsg));
80b14dee2bea12 Richard Cochran       2018-07-03  2840  		break;
779f1edec664a7 Soheil Hassas Yeganeh 2016-07-11  2841  	/* SCM_RIGHTS and SCM_CREDENTIALS are semantically in SOL_UNIX. */
779f1edec664a7 Soheil Hassas Yeganeh 2016-07-11  2842  	case SCM_RIGHTS:
779f1edec664a7 Soheil Hassas Yeganeh 2016-07-11  2843  	case SCM_CREDENTIALS:
779f1edec664a7 Soheil Hassas Yeganeh 2016-07-11  2844  		break;
274b6fd4a3053e Zijian Zhang          2024-05-10  2845  	case SCM_ZC_NOTIFICATION: {
274b6fd4a3053e Zijian Zhang          2024-05-10  2846  		int ret, i = 0;
274b6fd4a3053e Zijian Zhang          2024-05-10  2847  		int cmsg_data_len, zc_info_elem_num;
274b6fd4a3053e Zijian Zhang          2024-05-10  2848  		void __user	*usr_addr;
274b6fd4a3053e Zijian Zhang          2024-05-10  2849  		struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX];
274b6fd4a3053e Zijian Zhang          2024-05-10  2850  		unsigned long flags;
274b6fd4a3053e Zijian Zhang          2024-05-10  2851  		struct sk_buff_head *q, local_q;
274b6fd4a3053e Zijian Zhang          2024-05-10  2852  		struct sk_buff *skb, *tmp;
274b6fd4a3053e Zijian Zhang          2024-05-10  2853  		struct sock_exterr_skb *serr;
274b6fd4a3053e Zijian Zhang          2024-05-10  2854  
274b6fd4a3053e Zijian Zhang          2024-05-10  2855  		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
274b6fd4a3053e Zijian Zhang          2024-05-10  2856  			return -EINVAL;
274b6fd4a3053e Zijian Zhang          2024-05-10  2857  
274b6fd4a3053e Zijian Zhang          2024-05-10  2858  		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
274b6fd4a3053e Zijian Zhang          2024-05-10  2859  		if (cmsg_data_len % sizeof(struct zc_info_elem))
274b6fd4a3053e Zijian Zhang          2024-05-10  2860  			return -EINVAL;
274b6fd4a3053e Zijian Zhang          2024-05-10  2861  
274b6fd4a3053e Zijian Zhang          2024-05-10  2862  		zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem);
274b6fd4a3053e Zijian Zhang          2024-05-10  2863  		if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX)
274b6fd4a3053e Zijian Zhang          2024-05-10  2864  			return -EINVAL;
274b6fd4a3053e Zijian Zhang          2024-05-10  2865  
274b6fd4a3053e Zijian Zhang          2024-05-10  2866  		if (in_compat_syscall())
274b6fd4a3053e Zijian Zhang          2024-05-10  2867  			usr_addr = compat_ptr(*(compat_uptr_t *)CMSG_DATA(cmsg));
274b6fd4a3053e Zijian Zhang          2024-05-10  2868  		else
274b6fd4a3053e Zijian Zhang          2024-05-10  2869  			usr_addr = (void __user *)*(void **)CMSG_DATA(cmsg);
274b6fd4a3053e Zijian Zhang          2024-05-10  2870  		if (!access_ok(usr_addr, cmsg_data_len))
274b6fd4a3053e Zijian Zhang          2024-05-10  2871  			return -EFAULT;
274b6fd4a3053e Zijian Zhang          2024-05-10  2872  
274b6fd4a3053e Zijian Zhang          2024-05-10  2873  		q = &sk->sk_error_queue;
274b6fd4a3053e Zijian Zhang          2024-05-10  2874  		skb_queue_head_init(&local_q);
274b6fd4a3053e Zijian Zhang          2024-05-10  2875  		spin_lock_irqsave(&q->lock, flags);
274b6fd4a3053e Zijian Zhang          2024-05-10  2876  		skb = skb_peek(q);
274b6fd4a3053e Zijian Zhang          2024-05-10  2877  		while (skb && i < zc_info_elem_num) {
274b6fd4a3053e Zijian Zhang          2024-05-10  2878  			struct sk_buff *skb_next = skb_peek_next(skb, q);
274b6fd4a3053e Zijian Zhang          2024-05-10  2879  
274b6fd4a3053e Zijian Zhang          2024-05-10  2880  			serr = SKB_EXT_ERR(skb);
274b6fd4a3053e Zijian Zhang          2024-05-10  2881  			if (serr->ee.ee_errno == 0 &&
274b6fd4a3053e Zijian Zhang          2024-05-10  2882  			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
274b6fd4a3053e Zijian Zhang          2024-05-10  2883  				zc_info_kern[i].hi = serr->ee.ee_data;
274b6fd4a3053e Zijian Zhang          2024-05-10  2884  				zc_info_kern[i].lo = serr->ee.ee_info;
274b6fd4a3053e Zijian Zhang          2024-05-10  2885  				zc_info_kern[i].zerocopy = !(serr->ee.ee_code
274b6fd4a3053e Zijian Zhang          2024-05-10  2886  								& SO_EE_CODE_ZEROCOPY_COPIED);
274b6fd4a3053e Zijian Zhang          2024-05-10  2887  				__skb_unlink(skb, q);
274b6fd4a3053e Zijian Zhang          2024-05-10  2888  				__skb_queue_tail(&local_q, skb);
274b6fd4a3053e Zijian Zhang          2024-05-10  2889  				i++;
274b6fd4a3053e Zijian Zhang          2024-05-10  2890  			}
274b6fd4a3053e Zijian Zhang          2024-05-10  2891  			skb = skb_next;
274b6fd4a3053e Zijian Zhang          2024-05-10  2892  		}
274b6fd4a3053e Zijian Zhang          2024-05-10  2893  		spin_unlock_irqrestore(&q->lock, flags);
274b6fd4a3053e Zijian Zhang          2024-05-10  2894  
274b6fd4a3053e Zijian Zhang          2024-05-10  2895  		ret = copy_to_user(usr_addr,
274b6fd4a3053e Zijian Zhang          2024-05-10  2896  				   zc_info_kern,
274b6fd4a3053e Zijian Zhang          2024-05-10  2897  					i * sizeof(struct zc_info_elem));
274b6fd4a3053e Zijian Zhang          2024-05-10  2898  
274b6fd4a3053e Zijian Zhang          2024-05-10  2899  		if (unlikely(ret)) {
274b6fd4a3053e Zijian Zhang          2024-05-10  2900  			spin_lock_irqsave(&q->lock, flags);
274b6fd4a3053e Zijian Zhang          2024-05-10  2901  			skb_queue_reverse_walk_safe(&local_q, skb, tmp) {
274b6fd4a3053e Zijian Zhang          2024-05-10  2902  				__skb_unlink(skb, &local_q);
274b6fd4a3053e Zijian Zhang          2024-05-10  2903  				__skb_queue_head(q, skb);
274b6fd4a3053e Zijian Zhang          2024-05-10  2904  			}
274b6fd4a3053e Zijian Zhang          2024-05-10  2905  			spin_unlock_irqrestore(&q->lock, flags);
274b6fd4a3053e Zijian Zhang          2024-05-10  2906  			return -EFAULT;
274b6fd4a3053e Zijian Zhang          2024-05-10  2907  		}
274b6fd4a3053e Zijian Zhang          2024-05-10  2908  
274b6fd4a3053e Zijian Zhang          2024-05-10  2909  		while ((skb = __skb_dequeue(&local_q)))
274b6fd4a3053e Zijian Zhang          2024-05-10  2910  			consume_skb(skb);
274b6fd4a3053e Zijian Zhang          2024-05-10  2911  		break;
274b6fd4a3053e Zijian Zhang          2024-05-10  2912  	}
f28ea365cdefc3 Edward Jee            2015-10-08  2913  	default:
f28ea365cdefc3 Edward Jee            2015-10-08  2914  		return -EINVAL;
f28ea365cdefc3 Edward Jee            2015-10-08  2915  	}
39771b127b4123 Willem de Bruijn      2016-04-02  2916  	return 0;
39771b127b4123 Willem de Bruijn      2016-04-02  2917  }
39771b127b4123 Willem de Bruijn      2016-04-02  2918  EXPORT_SYMBOL(__sock_cmsg_send);
39771b127b4123 Willem de Bruijn      2016-04-02  2919  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH net-next v3 1/3] selftests: fix OOM problem in msg_zerocopy selftest

[Willem de Bruijn]

zijianzhang@ wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
> 
> In selftests/net/msg_zerocopy.c, it has a while loop keeps calling sendmsg
> on a socket with MSG_ZEROCOPY flag, and it will recv the notifications
> until the socket is not writable. Typically, it will start the receiving
> process after around 30+ sendmsgs. However, because of the
> commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale")
> the sender is always writable and does not get any chance to run recv
> notifications. The selftest always exits with OUT_OF_MEMORY because the
> memory used by opt_skb exceeds the core.sysctl_optmem_max.
> 
> According to our experiments, this problem can be mitigated by open the
> DEBUG_LOCKDEP configuration for the kernel. But it will makes the
> notifications disordered even in good commits before
> commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale").
> 
> We introduce "cfg_notification_limit" to force sender to receive
> notifications after some number of sendmsgs. And, notifications may not
> come in order, because of the reason we present above. We have order
> checking code managed by cfg_verbose.
> 
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> ---
>  tools/testing/selftests/net/msg_zerocopy.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c
> index bdc03a2097e8..ba6c257f689c 100644
> --- a/tools/testing/selftests/net/msg_zerocopy.c
> +++ b/tools/testing/selftests/net/msg_zerocopy.c
> @@ -85,6 +85,7 @@ static bool cfg_rx;
>  static int  cfg_runtime_ms	= 4200;
>  static int  cfg_verbose;
>  static int  cfg_waittime_ms	= 500;
> +static int  cfg_notification_limit = 32;
>  static bool cfg_zerocopy;
>  
>  static socklen_t cfg_alen;
> @@ -95,6 +96,8 @@ static char payload[IP_MAXPACKET];
>  static long packets, bytes, completions, expected_completions;
>  static int  zerocopied = -1;
>  static uint32_t next_completion;
> +/* The number of sendmsgs which have not received notified yet */
> +static uint32_t sendmsg_counter;

minor: comment typo. Consider a more self documenting variable and
drop the comment

static int sends_since_notify;

Re: [PATCH net-next v3 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[Willem de Bruijn]

zijianzhang@ wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
> 
> The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
> However, zerocopy is not a free lunch. Apart from the management of user
> pages, the combination of poll + recvmsg to receive notifications incurs
> unignorable overhead in the applications. The overhead of such sometimes
> might be more than the CPU savings from zerocopy. We try to solve this
> problem with a new notification mechanism based on msgcontrol.
> This new mechanism aims to reduce the overhead associated with receiving
> notifications by embedding them directly into user arguments passed with
> each sendmsg control message. By doing so, we can significantly reduce
> the complexity and overhead for managing notifications. In an ideal
> pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION
> msg_control, and the notification will be delivered as soon as possible.
> 
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>

> +#include <linux/types.h>
> +
>  /*
>   * Desired design of maximum size and alignment (see RFC2553)
>   */
> @@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage {
>  #define SOCK_TXREHASH_DISABLED	0
>  #define SOCK_TXREHASH_ENABLED	1
>  
> +#define SOCK_ZC_INFO_MAX 128
> +
> +struct zc_info_elem {
> +	__u32 lo;
> +	__u32 hi;
> +	__u8 zerocopy;
> +};
> +
>  #endif /* _UAPI_LINUX_SOCKET_H */
> diff --git a/net/core/sock.c b/net/core/sock.c
> index 8d6e638b5426..15da609be026 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -2842,6 +2842,74 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>  	case SCM_RIGHTS:
>  	case SCM_CREDENTIALS:
>  		break;
> +	case SCM_ZC_NOTIFICATION: {
> +		int ret, i = 0;
> +		int cmsg_data_len, zc_info_elem_num;
> +		void __user	*usr_addr;
> +		struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX];
> +		unsigned long flags;
> +		struct sk_buff_head *q, local_q;
> +		struct sk_buff *skb, *tmp;
> +		struct sock_exterr_skb *serr;

minor: reverse xmas tree

> +
> +		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
> +			return -EINVAL;

Is this mechanism supported for PF_RDS?
The next patch fails on PF_RDS + '-n'

> +
> +		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
> +		if (cmsg_data_len % sizeof(struct zc_info_elem))
> +			return -EINVAL;
> +
> +		zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem);
> +		if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX)
> +			return -EINVAL;
> +
> +		if (in_compat_syscall())
> +			usr_addr = compat_ptr(*(compat_uptr_t *)CMSG_DATA(cmsg));
> +		else
> +			usr_addr = (void __user *)*(void **)CMSG_DATA(cmsg);

The main design issue with this series is this indirection, rather
than passing the array of notifications as cmsg.

This trick circumvents having to deal with compat issues and having to
figure out copy_to_user in ____sys_sendmsg (as msg_control is an
in-kernel copy).

This is quite hacky, from an API design PoV.

As is passing a pointer, but expecting msg_controllen to hold the
length not of the pointer, but of the pointed to user buffer.

I had also hoped for more significant savings. Especially with the
higher syscall overhead due to meltdown and spectre mitigations vs
when MSG_ZEROCOPY was introduced and I last tried this optimization.

> +		if (!access_ok(usr_addr, cmsg_data_len))
> +			return -EFAULT;
> +
> +		q = &sk->sk_error_queue;
> +		skb_queue_head_init(&local_q);
> +		spin_lock_irqsave(&q->lock, flags);
> +		skb = skb_peek(q);
> +		while (skb && i < zc_info_elem_num) {
> +			struct sk_buff *skb_next = skb_peek_next(skb, q);
> +
> +			serr = SKB_EXT_ERR(skb);
> +			if (serr->ee.ee_errno == 0 &&
> +			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
> +				zc_info_kern[i].hi = serr->ee.ee_data;
> +				zc_info_kern[i].lo = serr->ee.ee_info;
> +				zc_info_kern[i].zerocopy = !(serr->ee.ee_code
> +								& SO_EE_CODE_ZEROCOPY_COPIED);
> +				__skb_unlink(skb, q);
> +				__skb_queue_tail(&local_q, skb);
> +				i++;
> +			}
> +			skb = skb_next;
> +		}
> +		spin_unlock_irqrestore(&q->lock, flags);
> +
> +		ret = copy_to_user(usr_addr,
> +				   zc_info_kern,
> +					i * sizeof(struct zc_info_elem));
> +
> +		if (unlikely(ret)) {
> +			spin_lock_irqsave(&q->lock, flags);
> +			skb_queue_reverse_walk_safe(&local_q, skb, tmp) {
> +				__skb_unlink(skb, &local_q);
> +				__skb_queue_head(q, skb);
> +			}

Can just list_splice_init?

> +			spin_unlock_irqrestore(&q->lock, flags);
> +			return -EFAULT;
> +		}
> +
> +		while ((skb = __skb_dequeue(&local_q)))
> +			consume_skb(skb);
> +		break;
> +	}
>  	default:
>  		return -EINVAL;
>  	}
> -- 
> 2.20.1
> 

Re: [PATCH net-next v3 3/3] selftests: add MSG_ZEROCOPY msg_control notification test

[Willem de Bruijn]

zijianzhang@ wrote:
> From: Zijian Zhang <zijianzhang@bytedance.com>
> 
> We update selftests/net/msg_zerocopy.c to accommodate the new mechanism.
> 
> Test result from selftests/net/msg_zerocopy.c,
> cfg_notification_limit = 1, in this case MSG_ZEROCOPY approximately
> aligns with the semantics of MSG_ZEROCOPY_UARG. In this case, the new
> flag has around 13% cpu savings in TCP and 18% cpu savings in UDP.
> +---------------------+---------+---------+---------+---------+
> | Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
> +---------------------+---------+---------+---------+---------+
> | ZCopy (MB)          | 5147    | 4885    | 7489    | 7854    |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy (MB)      | 5859    | 5505    | 9053    | 9236    |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy / ZCopy   | 113.83% | 112.69% | 120.88% | 117.59% |
> +---------------------+---------+---------+---------+---------+
> 
> cfg_notification_limit = 32, it means less poll + recvmsg overhead,
> the new mechanism performs 8% better in TCP. For UDP, no obvious
> performance gain is observed and sometimes may lead to degradation.
> Thus, if users don't need to retrieve the notification ASAP in UDP,
> the original mechanism is preferred.
> +---------------------+---------+---------+---------+---------+
> | Test Type / Protocol| TCP v4  | TCP v6  | UDP v4  | UDP v6  |
> +---------------------+---------+---------+---------+---------+
> | ZCopy (MB)          | 6272    | 6138    | 12138   | 10055   |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy (MB)      | 6774    | 6620    | 11504   | 10355   |
> +---------------------+---------+---------+---------+---------+
> | New ZCopy / ZCopy   | 108.00% | 107.85% | 94.78%  | 102.98% |
> +---------------------+---------+---------+---------+---------+
> 
> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> ---
>  tools/testing/selftests/net/msg_zerocopy.c  | 103 ++++++++++++++++++--
>  tools/testing/selftests/net/msg_zerocopy.sh |   1 +
>  2 files changed, 97 insertions(+), 7 deletions(-)
> 
> diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c
> index ba6c257f689c..48750a7a162c 100644
> --- a/tools/testing/selftests/net/msg_zerocopy.c
> +++ b/tools/testing/selftests/net/msg_zerocopy.c
> @@ -1,3 +1,4 @@
> +// SPDX-License-Identifier: GPL-2.0
>  /* Evaluate MSG_ZEROCOPY
>   *
>   * Send traffic between two processes over one of the supported
> @@ -66,6 +67,10 @@
>  #define SO_ZEROCOPY	60
>  #endif
>  
> +#ifndef SCM_ZC_NOTIFICATION
> +#define SCM_ZC_NOTIFICATION	78
> +#endif
> +
>  #ifndef SO_EE_CODE_ZEROCOPY_COPIED
>  #define SO_EE_CODE_ZEROCOPY_COPIED	1
>  #endif
> @@ -74,6 +79,13 @@
>  #define MSG_ZEROCOPY	0x4000000
>  #endif
>  
> +#define ZC_MSGERR_NOTIFICATION 1
> +#define ZC_MSGCTL_NOTIFICATION 2

Please define an enum

And consider less truncated, more descriptive, names. E.g.,

   MSG_ZEROCOPY_NOTIFY_ERRQUEUE
   MSG_ZEROCOPY_NOTIFY_SENDMSG

> +
> +#define SOCK_ZC_INFO_NUM 8
> +
> +#define INVALID_ZEROCOPY_VAL 2
> +
>  static int  cfg_cork;
>  static bool cfg_cork_mixed;
>  static int  cfg_cpu		= -1;		/* default: pin to last cpu */
> @@ -86,13 +98,16 @@ static int  cfg_runtime_ms	= 4200;
>  static int  cfg_verbose;
>  static int  cfg_waittime_ms	= 500;
>  static int  cfg_notification_limit = 32;
> -static bool cfg_zerocopy;
> +static int  cfg_zerocopy;           /* Either 1 or 2, mode for SO_ZEROCOPY */
>  
>  static socklen_t cfg_alen;
>  static struct sockaddr_storage cfg_dst_addr;
>  static struct sockaddr_storage cfg_src_addr;
>  
>  static char payload[IP_MAXPACKET];
> +static struct zc_info_elem zc_info[SOCK_ZC_INFO_NUM];
> +static char zc_ckbuf[CMSG_SPACE(sizeof(void *))];
> +static struct zc_info_elem *zc_info_ptr = zc_info;
>  static long packets, bytes, completions, expected_completions;
>  static int  zerocopied = -1;
>  static uint32_t next_completion;
> @@ -170,6 +185,26 @@ static int do_accept(int fd)
>  	return fd;
>  }
>  
> +static void add_zcopy_info(struct msghdr *msg)
> +{
> +	int i;
> +	struct cmsghdr *cm;
> +
> +	if (!msg->msg_control)
> +		error(1, errno, "NULL user arg");
> +	cm = (void *)msg->msg_control;
> +	/* Although only the address of the array will be written into the
> +	 * zc_ckbuf, we assign cmsg_len to CMSG_LEN(sizeof(zc_info)) to specify
> +	 * the length of the array.
> +	 */
> +	cm->cmsg_len = CMSG_LEN(sizeof(zc_info));
> +	cm->cmsg_level = SOL_SOCKET;
> +	cm->cmsg_type = SCM_ZC_NOTIFICATION;
> +	memcpy(CMSG_DATA(cm), &zc_info_ptr, sizeof(zc_info_ptr));
> +	for (i = 0; i < SOCK_ZC_INFO_NUM; i++)
> +		zc_info[i].zerocopy = INVALID_ZEROCOPY_VAL;
> +}
> +
>  static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie)
>  {
>  	struct cmsghdr *cm;
> @@ -183,7 +218,7 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie)
>  	memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie));
>  }
>  
> -static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
> +static bool do_sendmsg(int fd, struct msghdr *msg, int do_zerocopy, int domain)
>  {
>  	int ret, len, i, flags;
>  	static uint32_t cookie;
> @@ -201,6 +236,15 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
>  			msg->msg_controllen = CMSG_SPACE(sizeof(cookie));
>  			msg->msg_control = (struct cmsghdr *)ckbuf;
>  			add_zcopy_cookie(msg, ++cookie);
> +		} else if (do_zerocopy == ZC_MSGCTL_NOTIFICATION) {
> +			memset(&msg->msg_control, 0, sizeof(msg->msg_control));
> +			/* Although only the address of the array will be written into the
> +			 * zc_ckbuf, msg_controllen must be larger or equal than any cmsg_len
> +			 * in it. Otherwise, we will get -EINVAL.
> +			 */
> +			msg->msg_controllen = CMSG_SPACE(sizeof(zc_info));
> +			msg->msg_control = (struct cmsghdr *)zc_ckbuf;
> +			add_zcopy_info(msg);
>  		}
>  	}
>  
> @@ -219,7 +263,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain)
>  		if (do_zerocopy && ret)
>  			expected_completions++;
>  	}
> -	if (do_zerocopy && domain == PF_RDS) {
> +	if (msg->msg_control) {
>  		msg->msg_control = NULL;
>  		msg->msg_controllen = 0;
>  	}
> @@ -393,6 +437,42 @@ static bool do_recvmsg_completion(int fd)
>  	return ret;
>  }
>  
> +static void do_recv_completions2(void)
> +{
> +	int i;
> +	__u32 hi, lo, range;
> +	__u8 zerocopy;
> +
> +	for (i = 0; zc_info[i].zerocopy != INVALID_ZEROCOPY_VAL; i++) {
> +		struct zc_info_elem elem = zc_info[i];
> +
> +		hi = elem.hi;
> +		lo = elem.lo;
> +		zerocopy = elem.zerocopy;
> +		range = hi - lo + 1;
> +
> +		if (cfg_verbose && lo != next_completion)
> +			fprintf(stderr, "gap: %u..%u does not append to %u\n",
> +				lo, hi, next_completion);
> +		next_completion = hi + 1;
> +
> +		if (zerocopied == -1)
> +			zerocopied = zerocopy;
> +		else if (zerocopied != zerocopy) {
> +			fprintf(stderr, "serr: inconsistent\n");
> +			zerocopied = zerocopy;
> +		}
> +
> +		completions += range;
> +
> +		if (cfg_verbose >= 2)
> +			fprintf(stderr, "completed: %u (h=%u l=%u)\n",
> +				range, hi, lo);
> +	}
> +
> +	sendmsg_counter -= i;
> +}
> +
>  static bool do_recv_completion(int fd, int domain)
>  {
>  	struct sock_extended_err *serr;
> @@ -554,11 +634,15 @@ static void do_tx(int domain, int type, int protocol)
>  		else
>  			do_sendmsg(fd, &msg, cfg_zerocopy, domain);
>  
> -		if (cfg_zerocopy && sendmsg_counter >= cfg_notification_limit)
> +		if (cfg_zerocopy == ZC_MSGERR_NOTIFICATION &&
> +			sendmsg_counter >= cfg_notification_limit)
>  			do_recv_completions(fd, domain);
>  
> +		if (cfg_zerocopy == ZC_MSGCTL_NOTIFICATION)
> +			do_recv_completions2();
> +
>  		while (!do_poll(fd, POLLOUT)) {
> -			if (cfg_zerocopy)
> +			if (cfg_zerocopy == ZC_MSGERR_NOTIFICATION)
>  				do_recv_completions(fd, domain);
>  		}
>  
> @@ -716,7 +800,7 @@ static void parse_opts(int argc, char **argv)
>  
>  	cfg_payload_len = max_payload_len;
>  
> -	while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) {
> +	while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) {
>  		switch (c) {
>  		case '4':
>  			if (cfg_family != PF_UNSPEC)
> @@ -750,6 +834,9 @@ static void parse_opts(int argc, char **argv)
>  		case 'm':
>  			cfg_cork_mixed = true;
>  			break;
> +		case 'n':
> +			cfg_zerocopy = ZC_MSGCTL_NOTIFICATION;
> +			break;
>  		case 'p':
>  			cfg_port = strtoul(optarg, NULL, 0);
>  			break;
> @@ -769,7 +856,7 @@ static void parse_opts(int argc, char **argv)
>  			cfg_verbose++;
>  			break;
>  		case 'z':
> -			cfg_zerocopy = true;
> +			cfg_zerocopy = ZC_MSGERR_NOTIFICATION;
>  			break;
>  		}
>  	}
> @@ -780,6 +867,8 @@ static void parse_opts(int argc, char **argv)
>  			error(1, 0, "-D <server addr> required for PF_RDS\n");
>  		if (!cfg_rx && !saddr)
>  			error(1, 0, "-S <client addr> required for PF_RDS\n");
> +		if (cfg_zerocopy == ZC_MSGCTL_NOTIFICATION)
> +			error(1, 0, "PF_RDS does not support ZC_MSGCTL_NOTIFICATION");
>  	}
>  	setup_sockaddr(cfg_family, daddr, &cfg_dst_addr);
>  	setup_sockaddr(cfg_family, saddr, &cfg_src_addr);
> diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh
> index 89c22f5320e0..022a6936d86f 100755
> --- a/tools/testing/selftests/net/msg_zerocopy.sh
> +++ b/tools/testing/selftests/net/msg_zerocopy.sh
> @@ -118,4 +118,5 @@ do_test() {
>  
>  do_test "${EXTRA_ARGS}"
>  do_test "-z ${EXTRA_ARGS}"
> +do_test "-n ${EXTRA_ARGS}"
>  echo ok
> -- 
> 2.20.1
> 

Re: [External] Re: [PATCH net-next v3 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[Zijian Zhang]

On 5/12/24 5:58 PM, Willem de Bruijn wrote:
> zijianzhang@ wrote:
>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>
>> The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
>> However, zerocopy is not a free lunch. Apart from the management of user
>> pages, the combination of poll + recvmsg to receive notifications incurs
>> unignorable overhead in the applications. The overhead of such sometimes
>> might be more than the CPU savings from zerocopy. We try to solve this
>> problem with a new notification mechanism based on msgcontrol.
>> This new mechanism aims to reduce the overhead associated with receiving
>> notifications by embedding them directly into user arguments passed with
>> each sendmsg control message. By doing so, we can significantly reduce
>> the complexity and overhead for managing notifications. In an ideal
>> pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION
>> msg_control, and the notification will be delivered as soon as possible.
>>
>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
> 
>> +#include <linux/types.h>
>> +
>>   /*
>>    * Desired design of maximum size and alignment (see RFC2553)
>>    */
>> @@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage {
>>   #define SOCK_TXREHASH_DISABLED	0
>>   #define SOCK_TXREHASH_ENABLED	1
>>   
>> +#define SOCK_ZC_INFO_MAX 128
>> +
>> +struct zc_info_elem {
>> +	__u32 lo;
>> +	__u32 hi;
>> +	__u8 zerocopy;
>> +};
>> +
>>   #endif /* _UAPI_LINUX_SOCKET_H */
>> diff --git a/net/core/sock.c b/net/core/sock.c
>> index 8d6e638b5426..15da609be026 100644
>> --- a/net/core/sock.c
>> +++ b/net/core/sock.c
>> @@ -2842,6 +2842,74 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
>>   	case SCM_RIGHTS:
>>   	case SCM_CREDENTIALS:
>>   		break;
>> +	case SCM_ZC_NOTIFICATION: {
>> +		int ret, i = 0;
>> +		int cmsg_data_len, zc_info_elem_num;
>> +		void __user	*usr_addr;
>> +		struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX];
>> +		unsigned long flags;
>> +		struct sk_buff_head *q, local_q;
>> +		struct sk_buff *skb, *tmp;
>> +		struct sock_exterr_skb *serr;
> 
> minor: reverse xmas tree
> 

Ack.

>> +
>> +		if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
>> +			return -EINVAL;
> 
> Is this mechanism supported for PF_RDS?
> The next patch fails on PF_RDS + '-n'
> 

Nice catch! This mechanism does not support PF_RDS, I will update the
selftest code.

>> +
>> +		cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
>> +		if (cmsg_data_len % sizeof(struct zc_info_elem))
>> +			return -EINVAL;
>> +
>> +		zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem);
>> +		if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX)
>> +			return -EINVAL;
>> +
>> +		if (in_compat_syscall())
>> +			usr_addr = compat_ptr(*(compat_uptr_t *)CMSG_DATA(cmsg));
>> +		else
>> +			usr_addr = (void __user *)*(void **)CMSG_DATA(cmsg);
> 
> The main design issue with this series is this indirection, rather
> than passing the array of notifications as cmsg.
> 
> This trick circumvents having to deal with compat issues and having to
> figure out copy_to_user in ____sys_sendmsg (as msg_control is an
> in-kernel copy).
> 
> This is quite hacky, from an API design PoV.
> 
> As is passing a pointer, but expecting msg_controllen to hold the
> length not of the pointer, but of the pointed to user buffer.
> 
> I had also hoped for more significant savings. Especially with the
> higher syscall overhead due to meltdown and spectre mitigations vs
> when MSG_ZEROCOPY was introduced and I last tried this optimization.
>
Thanks for the summary, totally agree! It's a hard choice to design the
API like this.

>> +		if (!access_ok(usr_addr, cmsg_data_len))
>> +			return -EFAULT;
>> +
>> +		q = &sk->sk_error_queue;
>> +		skb_queue_head_init(&local_q);
>> +		spin_lock_irqsave(&q->lock, flags);
>> +		skb = skb_peek(q);
>> +		while (skb && i < zc_info_elem_num) {
>> +			struct sk_buff *skb_next = skb_peek_next(skb, q);
>> +
>> +			serr = SKB_EXT_ERR(skb);
>> +			if (serr->ee.ee_errno == 0 &&
>> +			    serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
>> +				zc_info_kern[i].hi = serr->ee.ee_data;
>> +				zc_info_kern[i].lo = serr->ee.ee_info;
>> +				zc_info_kern[i].zerocopy = !(serr->ee.ee_code
>> +								& SO_EE_CODE_ZEROCOPY_COPIED);
>> +				__skb_unlink(skb, q);
>> +				__skb_queue_tail(&local_q, skb);
>> +				i++;
>> +			}
>> +			skb = skb_next;
>> +		}
>> +		spin_unlock_irqrestore(&q->lock, flags);
>> +
>> +		ret = copy_to_user(usr_addr,
>> +				   zc_info_kern,
>> +					i * sizeof(struct zc_info_elem));
>> +
>> +		if (unlikely(ret)) {
>> +			spin_lock_irqsave(&q->lock, flags);
>> +			skb_queue_reverse_walk_safe(&local_q, skb, tmp) {
>> +				__skb_unlink(skb, &local_q);
>> +				__skb_queue_head(q, skb);
>> +			}
> 
> Can just list_splice_init?
> 

Ack.

>> +			spin_unlock_irqrestore(&q->lock, flags);
>> +			return -EFAULT;
>> +		}
>> +
>> +		while ((skb = __skb_dequeue(&local_q)))
>> +			consume_skb(skb);
>> +		break;
>> +	}
>>   	default:
>>   		return -EINVAL;
>>   	}
>> -- 
>> 2.20.1
>>
> 
> 

Re: [External] Re: [PATCH net-next v3 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control

[Zijian Zhang]

On 5/13/24 12:47 PM, Zijian Zhang wrote:
> On 5/12/24 5:58 PM, Willem de Bruijn wrote:
>> zijianzhang@ wrote:
>>> From: Zijian Zhang <zijianzhang@bytedance.com>
>>>
>>> The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
>>> However, zerocopy is not a free lunch. Apart from the management of user
>>> pages, the combination of poll + recvmsg to receive notifications incurs
>>> unignorable overhead in the applications. The overhead of such sometimes
>>> might be more than the CPU savings from zerocopy. We try to solve this
>>> problem with a new notification mechanism based on msgcontrol.
>>> This new mechanism aims to reduce the overhead associated with receiving
>>> notifications by embedding them directly into user arguments passed with
>>> each sendmsg control message. By doing so, we can significantly reduce
>>> the complexity and overhead for managing notifications. In an ideal
>>> pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION
>>> msg_control, and the notification will be delivered as soon as possible.
>>>
>>> Signed-off-by: Zijian Zhang <zijianzhang@bytedance.com>
>>> Signed-off-by: Xiaochun Lu <xiaochun.lu@bytedance.com>
>>
>>> +#include <linux/types.h>
>>> +
>>>   /*
>>>    * Desired design of maximum size and alignment (see RFC2553)
>>>    */
>>> @@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage {
>>>   #define SOCK_TXREHASH_DISABLED    0
>>>   #define SOCK_TXREHASH_ENABLED    1
>>> +#define SOCK_ZC_INFO_MAX 128
>>> +
>>> +struct zc_info_elem {
>>> +    __u32 lo;
>>> +    __u32 hi;
>>> +    __u8 zerocopy;
>>> +};
>>> +
>>>   #endif /* _UAPI_LINUX_SOCKET_H */
>>> diff --git a/net/core/sock.c b/net/core/sock.c
>>> index 8d6e638b5426..15da609be026 100644
>>> --- a/net/core/sock.c
>>> +++ b/net/core/sock.c
>>> @@ -2842,6 +2842,74 @@ int __sock_cmsg_send(struct sock *sk, struct 
>>> cmsghdr *cmsg,
>>>       case SCM_RIGHTS:
>>>       case SCM_CREDENTIALS:
>>>           break;
>>> +    case SCM_ZC_NOTIFICATION: {
>>> +        int ret, i = 0;
>>> +        int cmsg_data_len, zc_info_elem_num;
>>> +        void __user    *usr_addr;
>>> +        struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX];
>>> +        unsigned long flags;
>>> +        struct sk_buff_head *q, local_q;
>>> +        struct sk_buff *skb, *tmp;
>>> +        struct sock_exterr_skb *serr;
>>
>> minor: reverse xmas tree
>>
> 
> Ack.
> 
>>> +
>>> +        if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS)
>>> +            return -EINVAL;
>>
>> Is this mechanism supported for PF_RDS?
>> The next patch fails on PF_RDS + '-n'
>>
> 
> Nice catch! This mechanism does not support PF_RDS, I will update the
> selftest code.
> 

PF_RDS does not use MSGERR queue to store the info, thus it is not
supported by this patch. I will leave it as "unsupported" in the 
"selftest -n" now.

If possible, I may leave the support for PF_RDS in another patch set in
the future.

>>> +
>>> +        cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr);
>>> +        if (cmsg_data_len % sizeof(struct zc_info_elem))
>>> +            return -EINVAL;
>>> +
>>> +        zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem);
>>> +        if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX)
>>> +            return -EINVAL;
>>> +
>>> +        if (in_compat_syscall())
>>> +            usr_addr = compat_ptr(*(compat_uptr_t *)CMSG_DATA(cmsg));
>>> +        else
>>> +            usr_addr = (void __user *)*(void **)CMSG_DATA(cmsg);
>>
>> The main design issue with this series is this indirection, rather
>> than passing the array of notifications as cmsg.
>>
>> This trick circumvents having to deal with compat issues and having to
>> figure out copy_to_user in ____sys_sendmsg (as msg_control is an
>> in-kernel copy).
>>
>> This is quite hacky, from an API design PoV.
>>
>> As is passing a pointer, but expecting msg_controllen to hold the
>> length not of the pointer, but of the pointed to user buffer.
>>
>> I had also hoped for more significant savings. Especially with the
>> higher syscall overhead due to meltdown and spectre mitigations vs
>> when MSG_ZEROCOPY was introduced and I last tried this optimization.
>>
> Thanks for the summary, totally agree! It's a hard choice to design the
> API like this.
> 
>>> +        if (!access_ok(usr_addr, cmsg_data_len))
>>> +            return -EFAULT;
>>> +
>>> +        q = &sk->sk_error_queue;
>>> +        skb_queue_head_init(&local_q);
>>> +        spin_lock_irqsave(&q->lock, flags);
>>> +        skb = skb_peek(q);
>>> +        while (skb && i < zc_info_elem_num) {
>>> +            struct sk_buff *skb_next = skb_peek_next(skb, q);
>>> +
>>> +            serr = SKB_EXT_ERR(skb);
>>> +            if (serr->ee.ee_errno == 0 &&
>>> +                serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) {
>>> +                zc_info_kern[i].hi = serr->ee.ee_data;
>>> +                zc_info_kern[i].lo = serr->ee.ee_info;
>>> +                zc_info_kern[i].zerocopy = !(serr->ee.ee_code
>>> +                                & SO_EE_CODE_ZEROCOPY_COPIED);
>>> +                __skb_unlink(skb, q);
>>> +                __skb_queue_tail(&local_q, skb);
>>> +                i++;
>>> +            }
>>> +            skb = skb_next;
>>> +        }
>>> +        spin_unlock_irqrestore(&q->lock, flags);
>>> +
>>> +        ret = copy_to_user(usr_addr,
>>> +                   zc_info_kern,
>>> +                    i * sizeof(struct zc_info_elem));
>>> +
>>> +        if (unlikely(ret)) {
>>> +            spin_lock_irqsave(&q->lock, flags);
>>> +            skb_queue_reverse_walk_safe(&local_q, skb, tmp) {
>>> +                __skb_unlink(skb, &local_q);
>>> +                __skb_queue_head(q, skb);
>>> +            }
>>
>> Can just list_splice_init?
>>
> 
> Ack.
> 
>>> +            spin_unlock_irqrestore(&q->lock, flags);
>>> +            return -EFAULT;
>>> +        }
>>> +
>>> +        while ((skb = __skb_dequeue(&local_q)))
>>> +            consume_skb(skb);
>>> +        break;
>>> +    }
>>>       default:
>>>           return -EINVAL;
>>>       }
>>> -- 
>>> 2.20.1
>>>
>>
>>