[PATCH net-next] udp: avoid one cache line miss in recvmsg()

[PATCH net-next] udp: avoid one cache line miss in recvmsg()

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

UDP_SKB_CB(skb)->partial_cov is located at offset 66 in skb,
requesting a cold cache line being read in cpu cache.

We can avoid this cache line miss for UDP sockets,
as partial_cov has a meaning only for UDPLite.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 net/ipv4/udp.c |    3 ++-
 net/ipv6/udp.c |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index e1fc0116e8d59d8185670c6e55d1219bde55610d..b949770fdc08398a10f3974505a50b2b4f4b2cf3 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1389,7 +1389,8 @@ int udp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int noblock,
 	 * coverage checksum (UDP-Lite), do it before the copy.
 	 */
 
-	if (copied < ulen || UDP_SKB_CB(skb)->partial_cov || peeking) {
+	if (copied < ulen || peeking ||
+	    (is_udplite && UDP_SKB_CB(skb)->partial_cov)) {
 		checksum_valid = !udp_lib_checksum_complete(skb);
 		if (!checksum_valid)
 			goto csum_copy_err;
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 4f99417d9b401f2a65c7828e7d6b86d1d6161794..8fd4d89380b86c8630f7fd27ce4e9958497a2b89 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -363,7 +363,8 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	 * coverage checksum (UDP-Lite), do it before the copy.
 	 */
 
-	if (copied < ulen || UDP_SKB_CB(skb)->partial_cov || peeking) {
+	if (copied < ulen || peeking ||
+	    (is_udplite && UDP_SKB_CB(skb)->partial_cov)) {
 		checksum_valid = !udp_lib_checksum_complete(skb);
 		if (!checksum_valid)
 			goto csum_copy_err;

Re: [PATCH net-next] udp: avoid one cache line miss in recvmsg()

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 18 Nov 2016 17:18:03 -0800

> From: Eric Dumazet <edumazet@google.com>
> 
> UDP_SKB_CB(skb)->partial_cov is located at offset 66 in skb,
> requesting a cold cache line being read in cpu cache.
> 
> We can avoid this cache line miss for UDP sockets,
> as partial_cov has a meaning only for UDPLite.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Applied.

Re: [PATCH net-next] udp: avoid one cache line miss in recvmsg()

[Paolo Abeni]

On Fri, 2016-11-18 at 17:18 -0800, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> UDP_SKB_CB(skb)->partial_cov is located at offset 66 in skb,
> requesting a cold cache line being read in cpu cache.
> 
> We can avoid this cache line miss for UDP sockets,
> as partial_cov has a meaning only for UDPLite.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> ---
>  net/ipv4/udp.c |    3 ++-
>  net/ipv6/udp.c |    3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
> index e1fc0116e8d59d8185670c6e55d1219bde55610d..b949770fdc08398a10f3974505a50b2b4f4b2cf3 100644
> --- a/net/ipv4/udp.c
> +++ b/net/ipv4/udp.c
> @@ -1389,7 +1389,8 @@ int udp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int noblock,
>  	 * coverage checksum (UDP-Lite), do it before the copy.
>  	 */
>  
> -	if (copied < ulen || UDP_SKB_CB(skb)->partial_cov || peeking) {
> +	if (copied < ulen || peeking ||
> +	    (is_udplite && UDP_SKB_CB(skb)->partial_cov)) {
>  		checksum_valid = !udp_lib_checksum_complete(skb);
>  		if (!checksum_valid)
>  			goto csum_copy_err;
> diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
> index 4f99417d9b401f2a65c7828e7d6b86d1d6161794..8fd4d89380b86c8630f7fd27ce4e9958497a2b89 100644
> --- a/net/ipv6/udp.c
> +++ b/net/ipv6/udp.c
> @@ -363,7 +363,8 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
>  	 * coverage checksum (UDP-Lite), do it before the copy.
>  	 */
>  
> -	if (copied < ulen || UDP_SKB_CB(skb)->partial_cov || peeking) {
> +	if (copied < ulen || peeking ||
> +	    (is_udplite && UDP_SKB_CB(skb)->partial_cov)) {
>  		checksum_valid = !udp_lib_checksum_complete(skb);
>  		if (!checksum_valid)
>  			goto csum_copy_err;
> 
> 

Nice catch, thank you Eric!

This gives up to 8% speed-up in my performance test (wire speed udp flood
with small packets)

Tested-by: Paolo Abeni <pabeni@redhat.com>

Re: [PATCH net-next] udp: avoid one cache line miss in recvmsg()

[Eric Dumazet]

On Mon, 2016-11-21 at 22:36 +0100, Paolo Abeni wrote:

> Nice catch, thank you Eric!
> 
> This gives up to 8% speed-up in my performance test (wire speed udp flood
> with small packets)
> 
> Tested-by: Paolo Abeni <pabeni@redhat.com>

Thanks Paolo

Note that udp6_recvmsg() hits the 3rd cache line of skb to access
skb->protocol :

is_udp4 = (skb->protocol == htons(ETH_P_IP));

We might some trick to avoid this cache line miss.

Re: [PATCH net-next] udp: avoid one cache line miss in recvmsg()

[Eric Dumazet]

On Mon, 2016-11-21 at 14:01 -0800, Eric Dumazet wrote:
> Tested-by: Paolo Abeni <pabeni@redhat.com>
> 
> Thanks Paolo
> 
> Note that udp6_recvmsg() hits the 3rd cache line of skb to access
> skb->protocol :
> 
> is_udp4 = (skb->protocol == htons(ETH_P_IP));
> 
> We might some trick to avoid this cache line miss.

Argh, ignore this.

net-next kernel has this field at offset 0xc0, so all is good (since
this 4th cache line contains other needed fields)