[PATCH net-next] net: adopt SLUB sheaves for skbuff_small_head

[Eric Dumazet <edumazet@google.com>]

skbuff_small_head is used both on receive and send paths,
serving potentially 80 million allocations and frees per second.

Tuning it on large servers has been problematic, especially
on AMD Turins platforms, where "lock cmpxch16b" latency can
be over 30,000 cycles.

Switching to SLUB sheaves fixes the issue nicely.

tcp_rr benchmark with 10,000 flows goes from 25 Mpps to 40 Mpps
on AMD Turin.

Other platforms show benefits with tcp_rr with more than 30,000
flows.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: David Rientjes <rientjes@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
---
 net/core/skbuff.c | 31 ++++++++++++++++++++-----------
 1 file changed, 20 insertions(+), 11 deletions(-)

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 513cbfed19bc34bbb6767cdd7a50dad68be430fb..79eb7eb6eea9aa4a76c555e6ddd33bf0bc84c921 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -5174,6 +5174,19 @@ static void skb_extensions_init(void) {}
 
 void __init skb_init(void)
 {
+	struct kmem_cache_args kmem_args_small_head = {
+		.align		= 0,
+		.ctor		= NULL,
+		/* usercopy should only access first SKB_SMALL_HEAD_HEADROOM
+		 * bytes.
+		 * struct skb_shared_info is located at the end of skb->head,
+		 * and should not be copied to/from user.
+		 */
+		.useroffset	= 0,
+		.usersize	= SKB_SMALL_HEAD_HEADROOM,
+		.sheaf_capacity = 32,
+	};
+
 	net_hotdata.skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache",
 					      sizeof(struct sk_buff),
 					      0,
@@ -5189,17 +5202,13 @@ void __init skb_init(void)
 						0,
 						SLAB_HWCACHE_ALIGN|SLAB_PANIC,
 						NULL);
-	/* usercopy should only access first SKB_SMALL_HEAD_HEADROOM bytes.
-	 * struct skb_shared_info is located at the end of skb->head,
-	 * and should not be copied to/from user.
-	 */
-	net_hotdata.skb_small_head_cache = kmem_cache_create_usercopy("skbuff_small_head",
-						SKB_SMALL_HEAD_CACHE_SIZE,
-						0,
-						SLAB_HWCACHE_ALIGN | SLAB_PANIC,
-						0,
-						SKB_SMALL_HEAD_HEADROOM,
-						NULL);
+
+	net_hotdata.skb_small_head_cache = kmem_cache_create(
+			"skbuff_small_head",
+			SKB_SMALL_HEAD_CACHE_SIZE,
+			&kmem_args_small_head,
+			SLAB_HWCACHE_ALIGN | SLAB_PANIC);
+
 	skb_extensions_init();
 }
 
-- 
2.53.0.473.g4a7958ca14-goog