From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.218.49 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795052321; b=VuA5fgINZNnIKcSV7xJnvAsLxy1iXKBru5+5MxAjJH2jZg4ENJqC3yVF2UyiaX8aeqAC Fipwv9pjHprzDleZoxshOnn7jW1mS7MR2mEN6cEX262J+nq5w5e8MfI1Vlak6nrh/Grvq kf5qwIE0FhgLjH0At+6LeK5UDVFCdpT30duhf8UxDB3bm29PwKZez1xsZ6517xA2poUls Kaxm2O1P0g5oaFUsheuPJ0upThW3OpmqdVBmsKCDL5xUuWQat+7kpUQ1ol076fa4cyL31 SBJ3kW1Lka3auCQD+3B+SiV5iJNejHOhjuUKLGnYG4MD3f8hzT/Lm/Q51qqaLfjEgOg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795052321; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; bh=perFU4/B5Y0feRHDCtIfp11kwM76D74bmT1vbTvBgE4=; b=SclGHKt66W0A9POdl7dNSXAd1zSDdYMW5eJ8sZeo67jT++rMHxvRTS7+qrpvfKCW09hO 2w4z5w7lYwwMns70jl44hInCl6Xc9XH85WuUdg4vE+60aQBVdo6+UWC0xwQcz0vk9v6Hr XO6N2D0Vt4P3frhGXrnw4ynlwfMvZFOC4sSeB+F70/f4sOCyK9raRMtJUwnrFG47+DPEW yB95Jc0vwZcZfKzIzI59DGxGAr7CN7+IADZVWudBB56puq3YGwxLW+MbCgbvTacI35lv8 07BK4/P+2x4THwlfzSCr3sWXiITWwUwDzsnS/774BlwN47KMKDaEDyW6QLHL1cmshxw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=cloudflare.com; spf=pass smtp.mailfrom=cloudflare.com; dkim=pass header.d=cloudflare.com header.i=@cloudflare.com header.a=rsa-sha256 header.s=google09082023 header.b=bFPUn2O0; arc=none smtp.remote-ip=209.85.218.49 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C746606A33B for ; Thu, 19 Nov 2026 01:38:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 7C746606A33B Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.a=rsa-sha256 header.s=google09082023 header.b=bFPUn2O0 Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-c50a2efbe1dso291657766b.3 for ; Wed, 18 Nov 2026 17:38:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google09082023; t=1795052319; x=1795657119; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=perFU4/B5Y0feRHDCtIfp11kwM76D74bmT1vbTvBgE4=; b=bFPUn2O0mGRe0RmXQEMhMDC1ff3vVq6olWJgu4Ceu4rmBmUvjGd21IP9JmBHCLAmsj coyeptIAMGkBXKwTyUjuQ9sZ7hK2b1orQaPOEsPMcXcfgczcVepllJ7PFV49BqdeL/SI qVUEdBTooUMCYP27wGRA0wso7cVKMOD4coDXFHN8juME0UgK0QExT9gzE9b9UBptaHjl DAuj+gN5U3L6DcUwfsIyayr+ROFbPyd1x20JSmazw6tN7TKIB0RPvYDfKpNsC8ptfl5X di1T9Mo2IT/mn183q87V2Q2o/+1AhlDjvviR8hUQprUSrH1VTTTCccBYvgVPWHdt6P8X SItg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795052319; x=1795657119; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=perFU4/B5Y0feRHDCtIfp11kwM76D74bmT1vbTvBgE4=; b=O8bZKPGd67psZN6j85EYdWJWK0tbahG272os9Oat5yQ3ha8gbah/skde8onGTvqIeC 3ZLlf1Xncq50X8NBtnty6/Go+Ohy8bqPOtC2u+tpVzz3gDNfcW+Zj1RaryirEgYgX3Ok BH1wuNMsS3UWzGiz+Oe3AOJKCznIuGE/i1YazZMDOmRt1ZA88BW9j6R5/qKK3SqdiY1Z axq+FKneVQEHuY8VilTsspNhc4HSLZw4/0Zus2fXaTamZv9a+F9AWMd+SlD+oII3tsYe dua8Em8lE4HMdoNAwUVKgkAQgTV6UXnhxdAaD5PmzbQGEbLZTkbqxzw0nDKNnLHwPS1A 6BBA== X-Forwarded-Encrypted: i=1; AL+hErEL6Jbiyjvy8AOoEvo45+ul357tEGD784L1vDlztJhVA28IUXwoPMsF0qlLAVmlXbQJyHNRqrveF4M=@vger.kernel.org X-Gm-Message-State: APuU4Z/S5IdMjU32lcnGh7SEnHbA48UpjTDHfaEd9FA8Eni+952bmeZ/ oJHOquuTwZGmDYSQEVh8q2wMXNunV87mkPqD+gkFZLEi0EuxS8mtqukwWn2eyFdROW0= X-Gm-Gg: AYBFou3+0FLjw75rQCzVyhWlCsr0ExrjhZpZ6tEHhjG6Qhw+b/zZjCLO/+4S6OWOQqE dmu3Co7obzg9uBdYu7mqMLkcL0q7/zVKKchf+7TfMkA8F6LJaX3u1uBllVykjdNT6OOABdgC0p4 4LJueJn3h86acvF27HIC+WDx3Hso4fYKDt+IL2sjSMo3ZXoMc3tOXx2ywvSCXpjt6ocxeWyiOhz n3dIRgZvyCL10Rf8kISPvI//8txpAwylBSmI9iRdxo9e0VMnjl8LEtY+HqMFLvvfgsOVAFm/Zbq fwOOo8PTHFlL5vNLnqlLPFKrXzf0kUGYh0PuVg1QvS66z901Uo0iXudihjC9cOBp49b8rmo5xSS jYKzPF5d+IpzqVXePxRklpRrowD0gUw5XIVVG5H5MPiX3dHIvFKAUGgpyHI1elvsfiPpidzbVB5 +Lf5dgbgGwNrvrG0VFSA0nymPcYmivBOmIoP6uXWll1gc/b3E4Qx7vcF+/vxM1550YdPbMQ3DwG Lqd+2ijY0J72qI1dwTtarCQdh8r+lm5SxKvHMzSIne/VynIrqPTgo9Z0z0ADzEQjVv8WToaMvsL hAylFlEyb78aRZV/kBWaZB0W3g8sk4B+FMhK6ZC+9cAxlAZki3IqAif3Jh9XoSzRcB33KP0HsVo ijik= X-Received: by 2002:a17:907:2d24:b0:c33:923b:ad51 with SMTP id a640c23a62f3a-c510e55ffb7mr404977966b.45.1795052318970; Wed, 18 Nov 2026 17:38:38 -0800 (PST) Received: from L9JK9NPFH9 ([104.28.20.56]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c51233c4080sm59785966b.16.2026.11.18.17.38.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Nov 2026 17:38:38 -0800 (PST) From: Tianyi Gao To: netdev@vger.kernel.org, Jakub Kicinski , Paolo Abeni , Eric Dumazet Cc: "David S. Miller" , Simon Horman , David Keogh , Jesper Dangaard Brouer , Daniel Borkmann , David Woodhouse , Dragos Tatulea , Nimrod Oren , Alasdair McWilliam , Willem de Bruijn , Maciej Fijalkowski , Stanislav Fomichev , Magnus Karlsson , David Ahern , Ido Schimmel , Alexei Starovoitov , John Fastabend , Donald Hunter , Andrew Lunn , Jonathan Corbet , Shuah Khan , Shuah Khan , Randy Dunlap , Breno Leitao , =?UTF-8?q?Asbj=C3=B8rn=20Sloth=20T=C3=B8nnesen?= , Danielle Ratson , Kuniyuki Iwashima , bpf@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@cloudflare.com, nh-open-source@amazon.com, Stephen Hemminger , "Gustavo A. R. Silva" , "Remy D. Farley" , Amery Hung , Andrii Nakryiko , Andy Roulin , Anil Kaushik , Demos Ra , Eduard Zingerman , Emil Tsalapatis , Hangbin Liu , Ihor Solodrai , Jiayuan Chen , Jiri Olsa , Joe Damato , Kumar Kartikeya Dwivedi , Li RongQing , Louis Scalbert , Marc Kleine-Budde , Martin KaFai Lau , Martin Karsten , Matt Bobrowski , Michael Bommarito , Nikolay Aleksandrov , Runyu Xiao , Song Liu , Thorsten Leemhuis , Wang Zhan , Yonghong Song , Zhou Qiuyue , Tianyi Gao , Tianyi Gao Subject: [PATCH net-next v2 01/11] net: add tx_extra_headroom and netdev_get_alloc_headroom() Date: Thu, 19 Nov 2026 01:38:20 +0000 Message-ID: <20261119013830.36935-2-tianyi@cloudflare.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261119013830.36935-1-tianyi@cloudflare.com> References: <20261119013830.36935-1-tianyi@cloudflare.com> X-Mailing-List: linux-doc@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: David Keogh Programs attached at TC egress commonly push additional headers onto a packet, for example to encapsulate it. When the packet does not carry enough headroom the BPF helpers reallocate the head, which costs an allocation and a copy per packet. Locally generated UDP, raw and AF_PACKET traffic is allocated with exactly LL_RESERVED_SPACE() of headroom, which only covers the link-layer header and the driver's needed_headroom. Add a tx_extra_headroom field to struct net_device for that purpose, together with netdev_get_alloc_headroom() and netdev_get_alloc_headroom_hlen(). These add the extra reservation to the header length before the HH_DATA_MOD alignment, so the alignment of the link-layer header does not change. netdev_get_alloc_needed_headroom() returns needed_headroom plus the extra reservation, for callers that do their own alignment. The new functions are intended only for sites that allocate new packets. LL_RESERVED_SPACE() keeps its current meaning and remains the threshold used when checking whether an existing packet must be expanded, so the extra reservation never forces a reallocation on its own: packets that already have enough room for the headers being pushed, such as TCP segments built with MAX_TCP_HEADER, are left alone. Keep the extra reservation separate from needed_headroom so that driver updates of their own requirement do not clobber it. Place the field in the padding that follows needed_headroom, which keeps it in the TX read-mostly cacheline group. The field is not set anywhere yet. Later patches set it from XDP on a veth peer, from TC egress programs and from a veth attribute. Also remove LL_RESERVED_SPACE_EXTRA(). It has had no user since commit 6d5431555de0 ("caif: remove CAIF NETWORK LAYER"). It could add the extra headroom, but it reads dev->hard_header_len itself. AF_PACKET must use the value that it read once, see commit 03390aa32e66 ("packet: use consistent hard_header_len in non-ring send paths"). netdev_get_alloc_headroom_hlen() takes the header length as an argument instead. Signed-off-by: David Keogh Co-developed-by: Tianyi Gao Signed-off-by: Tianyi Gao --- v2: - Turn LL_ALLOC_SPACE() and LL_ALLOC_SPACE_EX() into the static inline functions netdev_get_alloc_headroom() and netdev_get_alloc_headroom_hlen(), and rename netdev_tx_alloc_headroom() to netdev_get_alloc_needed_headroom() (Willem). - Move the netdevices.rst section here from v1 3/8, without the IFLA part. --- .../networking/net_cachelines/net_device.rst | 1 + Documentation/networking/netdevices.rst | 13 +++++++ include/linux/netdevice.h | 34 +++++++++++++++++-- net/core/dev.c | 1 + 4 files changed, 46 insertions(+), 3 deletions(-) diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst index 7620d3fcb8dc..4341e9e443ba 100644 --- a/Documentation/networking/net_cachelines/net_device.rst +++ b/Documentation/networking/net_cachelines/net_device.rst @@ -35,6 +35,7 @@ unsigned_short gflags unsigned_short hard_header_len read_mostly read_mostly ip6_xmit(tx);gro_list_prepare(rx) unsigned_int mtu read_mostly ip_finish_output2 unsigned_short needed_headroom read_mostly LL_RESERVED_SPACE/ip_finish_output2 +unsigned_short tx_extra_headroom read_mostly netdev_get_alloc_headroom/__ip_append_data unsigned_short needed_tailroom netdev_features_t features read_mostly read_mostly HARD_TX_LOCK,netif_skb_features,sk_setup_caps(tx);netif_elide_gro(rx) netdev_features_t hw_features diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst index 6a87f09e7015..f593b242da9c 100644 --- a/Documentation/networking/netdevices.rst +++ b/Documentation/networking/netdevices.rst @@ -441,6 +441,19 @@ from a notifier. Drivers **must not** generate nested notifications of the ops-locked types. +TX headroom reservation +======================= + +``tx_extra_headroom`` is extra headroom for locally generated packets. The +stack adds it when it allocates such a packet for the device, so that +headers pushed later, for example at TC egress, fit without a +reallocation. It is at most 512 bytes. + +Unlike ``needed_headroom``, it is not a driver requirement and stacked +devices do not inherit it. It is not part of ``LL_RESERVED_SPACE()``, so a +packet that reaches the device with less headroom, such as a forwarded +packet, is not reallocated because of it. + NETDEV_INTERNAL symbol namespace ================================ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 0325c6f2a7d1..392e6fc81e34 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -311,9 +311,6 @@ struct hh_cache { & ~(HH_DATA_MOD - 1)) + HH_DATA_MOD) #define LL_RESERVED_SPACE(dev) \ LL_RESERVED_SPACE_EX(dev, (dev)->hard_header_len) -#define LL_RESERVED_SPACE_EXTRA(dev,extra) \ - ((((dev)->hard_header_len + READ_ONCE((dev)->needed_headroom) + (extra)) \ - & ~(HH_DATA_MOD - 1)) + HH_DATA_MOD) struct header_ops { int (*create) (struct sk_buff *skb, struct net_device *dev, @@ -1934,6 +1931,8 @@ enum netdev_reg_state { * * @needed_headroom: Extra headroom the hardware may need, but not in all * cases can this be guaranteed + * @tx_extra_headroom: Extra headroom for locally generated packets, for + * headers pushed after the allocation * @needed_tailroom: Extra tailroom the hardware may need, but not in all * cases can this be guaranteed. Some cases also use * LL_MAX_HEADER instead to allocate the skb @@ -2196,6 +2195,8 @@ struct net_device { */ unsigned int mtu; unsigned short needed_headroom; +#define NETDEV_TX_EXTRA_HEADROOM_MAX 512 + unsigned short tx_extra_headroom; struct netdev_tc_txq tc_to_txq[TC_MAX_QUEUE]; #ifdef CONFIG_XPS struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX]; @@ -2778,6 +2779,33 @@ struct netdev_queue *netdev_core_pick_tx(struct net_device *dev, struct sk_buff *skb, struct net_device *sb_dev); +/* Headroom to reserve when allocating a locally generated packet for dev. + * Code that checks the headroom of an existing packet keeps using + * LL_RESERVED_SPACE(), so tx_extra_headroom never forces a reallocation. + */ +static inline unsigned int +netdev_get_alloc_headroom_hlen(const struct net_device *dev, unsigned int hlen) +{ + return LL_RESERVED_SPACE_EX(dev, + hlen + READ_ONCE(dev->tx_extra_headroom)); +} + +static inline unsigned int +netdev_get_alloc_headroom(const struct net_device *dev) +{ + return netdev_get_alloc_headroom_hlen(dev, dev->hard_header_len); +} + +/* Same as netdev_get_alloc_headroom() without hard_header_len and alignment, + * for callers that size the headroom themselves. + */ +static inline unsigned int +netdev_get_alloc_needed_headroom(const struct net_device *dev) +{ + return READ_ONCE(dev->needed_headroom) + + READ_ONCE(dev->tx_extra_headroom); +} + /* returns the headroom that the master device needs to take in account * when forwarding to this dev */ diff --git a/net/core/dev.c b/net/core/dev.c index d88af07aea29..85b7c61015c7 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -13692,6 +13692,7 @@ static void __init net_dev_struct_check(void) CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, num_tc); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, mtu); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, needed_headroom); + CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tx_extra_headroom); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tc_to_txq); #ifdef CONFIG_XPS CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, xps_maps); -- 2.56.0