From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 140E53019B0 for ; Sat, 28 Feb 2026 14:12:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772287976; cv=none; b=fPxEysEh83eE9FBA1TjcT730iMELMcJO/BFCXkZ/C0EdgUDVmNOx1vMbprT7BYPVY85DpO6Fk9cyeAYhTl0B5fTlB7JCOt/ESYDV3yv1QY0crw4w4XTlNq5VuE78QD+93/c7Lv7sgrKc/dhADgHPNYE6Hwrr2T/+sR2Ce2uYrr0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772287976; c=relaxed/simple; bh=QBLdIEEnHpSiWyj65PwX0yZyN3lZmmDAc5D7n7PH7ms=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=bQdsVW1ea65UPMOPmidIPi2vNod2W6vmyL4kcf0lDAYsGLk6NSygX1OfKh+z8hOuufUfnn8JgR+tsdYxb3I4dw1BvzqMOEGvm9y++j9aeCkiaTxLedxPTs3X4jV3ZVbghWLKnrr1QyTi7vnAYDxFSRawsZKhQqVozUMvU4ToabA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3fCbFevV; arc=none smtp.client-ip=209.85.222.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3fCbFevV" Received: by mail-qk1-f202.google.com with SMTP id af79cd13be357-8cb3a129cd2so3061029585a.0 for ; Sat, 28 Feb 2026 06:12:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1772287974; x=1772892774; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=5xfpeDrS28+GJWagtJ5tbfozq/tKlFdhequt2eFMJJQ=; b=3fCbFevVpnhuyhF2D1f4sBCHdrexa2/43HSQKv1DSSuiU/n/TKHUe7TJ0h4/BzXvoW fb7NcpOO7Szw0UjnVxaLnA+70jkt9e7c1vTp5qMGvo2d0ZBqx3gtVqqU5ePEDOBX0NOx qzYzBfNG8V0BqbWzixaPeAh2L1JeCdPQKh2UhyMUij6VPpj8HsXhtCs0bGE/wDnpJKmA vKot/je5lJs89ALGuyorh85sfOW1PRZdowzRyZUw+rmb+LNmM+nCIR2RPvPnLQrx3TSf rvcyU3qIXzXApVRnYf4kGuwTOyFTBBBDohW+iCyKOylmT9Y2aeC/4emj7HfkORGyAk+v HYog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772287974; x=1772892774; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=5xfpeDrS28+GJWagtJ5tbfozq/tKlFdhequt2eFMJJQ=; b=NMSKyL7kcbyCye8i17wa72vcZ2gdkh5xelmti51rfWqe+MThAZkpkgPlMX1TSzhNbG 3/S/oyjFDgxRz6jDVoWBR6C4rQWh1SrPSVIR5Bc3vjoHTGxVwn/DGGLUa/Lth194MU5a bBLV2J9fx4gT/7sUHJWAUgNelt4rbdKG0egORJiK2GZqd2PNKy0UExilph50U/zLD2Zi TMSgBzrKts6U6qf3MxRNIUCztc/HzDc5Us4mtCyTo3/HdqpN2aGEFxo+mm1f1myeNi/l R7DeLqI8fiER/eZ8RY0rIZCAPBn1aE5rvgle+IRKJxvjEJvodA1k/zwsMbX/ws5P6OcF UYIA== X-Forwarded-Encrypted: i=1; AJvYcCUjLgSDlPn5oo3A6T+ZaglduXfAXLjF2fZSbBarmCQ66WFEvDd48oxDn9ZIO+yw0nZ71Wp0sGU=@vger.kernel.org X-Gm-Message-State: AOJu0Yzj5EnHEE0c7XPKNa41ok8xZ6D03AUWMDdncZ1z1ImXTGBrJmBk 4b9hrxCi1w4pBaREvRbqSIffC4JiHHBxWeffZBKNeQXG4Dso+HVoElDOF+AUVWLwOXPixNJst3c FGqrrY/aLfOwjyA== X-Received: from qknsk6.prod.google.com ([2002:a05:620a:3a46:b0:8cb:4038:5d20]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:a05:620a:318b:b0:8a6:92d1:2dae with SMTP id af79cd13be357-8cbc8c47908mr755691885a.5.1772287973734; Sat, 28 Feb 2026 06:12:53 -0800 (PST) Date: Sat, 28 Feb 2026 14:12:52 +0000 Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.53.0.473.g4a7958ca14-goog Message-ID: <20260228141252.435509-1-edumazet@google.com> Subject: [PATCH net-next] net: adopt SLUB sheaves for skbuff_small_head From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: Simon Horman , Kuniyuki Iwashima , netdev@vger.kernel.org, eric.dumazet@gmail.com, Eric Dumazet , Vlastimil Babka , David Rientjes , Roman Gushchin Content-Type: text/plain; charset="UTF-8" skbuff_small_head is used both on receive and send paths, serving potentially 80 million allocations and frees per second. Tuning it on large servers has been problematic, especially on AMD Turins platforms, where "lock cmpxch16b" latency can be over 30,000 cycles. Switching to SLUB sheaves fixes the issue nicely. tcp_rr benchmark with 10,000 flows goes from 25 Mpps to 40 Mpps on AMD Turin. Other platforms show benefits with tcp_rr with more than 30,000 flows. Signed-off-by: Eric Dumazet Cc: Vlastimil Babka Cc: David Rientjes Cc: Roman Gushchin --- net/core/skbuff.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 513cbfed19bc34bbb6767cdd7a50dad68be430fb..79eb7eb6eea9aa4a76c555e6ddd33bf0bc84c921 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5174,6 +5174,19 @@ static void skb_extensions_init(void) {} void __init skb_init(void) { + struct kmem_cache_args kmem_args_small_head = { + .align = 0, + .ctor = NULL, + /* usercopy should only access first SKB_SMALL_HEAD_HEADROOM + * bytes. + * struct skb_shared_info is located at the end of skb->head, + * and should not be copied to/from user. + */ + .useroffset = 0, + .usersize = SKB_SMALL_HEAD_HEADROOM, + .sheaf_capacity = 32, + }; + net_hotdata.skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache", sizeof(struct sk_buff), 0, @@ -5189,17 +5202,13 @@ void __init skb_init(void) 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); - /* usercopy should only access first SKB_SMALL_HEAD_HEADROOM bytes. - * struct skb_shared_info is located at the end of skb->head, - * and should not be copied to/from user. - */ - net_hotdata.skb_small_head_cache = kmem_cache_create_usercopy("skbuff_small_head", - SKB_SMALL_HEAD_CACHE_SIZE, - 0, - SLAB_HWCACHE_ALIGN | SLAB_PANIC, - 0, - SKB_SMALL_HEAD_HEADROOM, - NULL); + + net_hotdata.skb_small_head_cache = kmem_cache_create( + "skbuff_small_head", + SKB_SMALL_HEAD_CACHE_SIZE, + &kmem_args_small_head, + SLAB_HWCACHE_ALIGN | SLAB_PANIC); + skb_extensions_init(); } -- 2.53.0.473.g4a7958ca14-goog