[PATCH net-next] udp: add local "peek offset enabled" flag

[PATCH net-next] udp: add local "peek offset enabled" flag

[Paolo Abeni]

We want to re-organize the struct sock layout. The sk_peek_off
field location is problematic, as most protocols want it in the
RX read area, while UDP wants it on a cacheline different from
sk_receive_queue.

Create a local (inside udp_sock) copy of the 'peek offset is enabled'
flag and place it inside the same cacheline of reader_queue.

Check such flag before reading sk_peek_off. This will save potential
false sharing and cache misses in the fast-path.

Tested under UDP flood with small packets. The struct sock layout
update causes a 4% performance drop, and this patch restores completely
the original tput.

Signed-off-by: Paolo Abeni <pabeni@redhat.com>
---
 include/linux/udp.h | 10 ++++++++++
 net/ipv4/af_inet.c  |  2 +-
 net/ipv4/udp.c      |  2 +-
 net/ipv6/af_inet6.c |  2 +-
 4 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/include/linux/udp.h b/include/linux/udp.h
index d04188714dca..3748e82b627b 100644
--- a/include/linux/udp.h
+++ b/include/linux/udp.h
@@ -92,6 +92,9 @@ struct udp_sock {
 
 	/* This fields follows rcvbuf value, and is touched by udp_recvmsg */
 	int		forward_threshold;
+
+	/* Cache friendly copy of sk->sk_peek_off >= 0 */
+	bool		peeking_with_offset;
 };
 
 #define udp_test_bit(nr, sk)			\
@@ -109,6 +112,13 @@ struct udp_sock {
 
 #define udp_sk(ptr) container_of_const(ptr, struct udp_sock, inet.sk)
 
+static inline int udp_set_peek_off(struct sock *sk, int val)
+{
+	sk_set_peek_off(sk, val);
+	WRITE_ONCE(udp_sk(sk)->peeking_with_offset, val >= 0);
+	return 0;
+}
+
 static inline void udp_set_no_check6_tx(struct sock *sk, bool val)
 {
 	udp_assign_bit(NO_CHECK6_TX, sk, val);
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index ad278009e469..5daebdcbca32 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1103,7 +1103,7 @@ const struct proto_ops inet_dgram_ops = {
 	.recvmsg	   = inet_recvmsg,
 	.mmap		   = sock_no_mmap,
 	.splice_eof	   = inet_splice_eof,
-	.set_peek_off	   = sk_set_peek_off,
+	.set_peek_off	   = udp_set_peek_off,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl	   = inet_compat_ioctl,
 #endif
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index f631b0a21af4..38cce7cc51f6 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1589,7 +1589,7 @@ int udp_init_sock(struct sock *sk)
 
 void skb_consume_udp(struct sock *sk, struct sk_buff *skb, int len)
 {
-	if (unlikely(READ_ONCE(sk->sk_peek_off) >= 0)) {
+	if (unlikely(READ_ONCE(udp_sk(sk)->peeking_with_offset))) {
 		bool slow = lock_sock_fast(sk);
 
 		sk_peek_offset_bwd(sk, len);
diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c
index 959bfd9f6344..b90d46533cdc 100644
--- a/net/ipv6/af_inet6.c
+++ b/net/ipv6/af_inet6.c
@@ -736,7 +736,7 @@ const struct proto_ops inet6_dgram_ops = {
 	.recvmsg	   = inet6_recvmsg,		/* retpoline's sake */
 	.read_skb	   = udp_read_skb,
 	.mmap		   = sock_no_mmap,
-	.set_peek_off	   = sk_set_peek_off,
+	.set_peek_off	   = udp_set_peek_off,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl	   = inet6_compat_ioctl,
 #endif
-- 
2.43.0

Re: [PATCH net-next] udp: add local "peek offset enabled" flag

[Eric Dumazet]

On Tue, Feb 20, 2024 at 12:00 PM Paolo Abeni <pabeni@redhat.com> wrote:
>
> We want to re-organize the struct sock layout. The sk_peek_off
> field location is problematic, as most protocols want it in the
> RX read area, while UDP wants it on a cacheline different from
> sk_receive_queue.
>
> Create a local (inside udp_sock) copy of the 'peek offset is enabled'
> flag and place it inside the same cacheline of reader_queue.
>
> Check such flag before reading sk_peek_off. This will save potential
> false sharing and cache misses in the fast-path.
>
> Tested under UDP flood with small packets. The struct sock layout
> update causes a 4% performance drop, and this patch restores completely
> the original tput.
>
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> ---

SGTM, thanks !
Reviewed-by: Eric Dumazet <edumazet@google.com>

Re: [PATCH net-next] udp: add local "peek offset enabled" flag

[Willem de Bruijn]

Eric Dumazet wrote:
> On Tue, Feb 20, 2024 at 12:00 PM Paolo Abeni <pabeni@redhat.com> wrote:
> >
> > We want to re-organize the struct sock layout. The sk_peek_off
> > field location is problematic, as most protocols want it in the
> > RX read area, while UDP wants it on a cacheline different from
> > sk_receive_queue.
> >
> > Create a local (inside udp_sock) copy of the 'peek offset is enabled'
> > flag and place it inside the same cacheline of reader_queue.
> >
> > Check such flag before reading sk_peek_off. This will save potential
> > false sharing and cache misses in the fast-path.
> >
> > Tested under UDP flood with small packets. The struct sock layout
> > update causes a 4% performance drop, and this patch restores completely
> > the original tput.
> >
> > Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> > ---
> 
> SGTM, thanks !
> Reviewed-by: Eric Dumazet <edumazet@google.com>

Reviewed-by: Willem de Bruijn <willemb@google.com>

Re: [PATCH net-next] udp: add local "peek offset enabled" flag

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Tue, 20 Feb 2024 12:00:01 +0100 you wrote:
> We want to re-organize the struct sock layout. The sk_peek_off
> field location is problematic, as most protocols want it in the
> RX read area, while UDP wants it on a cacheline different from
> sk_receive_queue.
> 
> Create a local (inside udp_sock) copy of the 'peek offset is enabled'
> flag and place it inside the same cacheline of reader_queue.
> 
> [...]

Here is the summary with links:
  - [net-next] udp: add local "peek offset enabled" flag
    https://git.kernel.org/netdev/net-next/c/f796feabb9f5

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html