From: Tianyi Gao <tianyi@cloudflare.com>
To: netdev@vger.kernel.org, Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@kernel.org>
Cc: "David S. Miller" <davem@davemloft.net>,
"Simon Horman" <horms@kernel.org>,
"David Keogh" <dkeogh@amazon.com>,
"Jesper Dangaard Brouer" <hawk@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"David Woodhouse" <dwmw2@infradead.org>,
"Dragos Tatulea" <dtatulea@nvidia.com>,
"Nimrod Oren" <noren@nvidia.com>,
"Alasdair McWilliam" <alasdair@mcwilliam.dev>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Maciej Fijalkowski" <maciej.fijalkowski@intel.com>,
"Stanislav Fomichev" <sdf@fomichev.me>,
"Magnus Karlsson" <magnus.karlsson@intel.com>,
"David Ahern" <dsahern@kernel.org>,
"Ido Schimmel" <idosch@nvidia.com>,
"Alexei Starovoitov" <ast@kernel.org>,
"John Fastabend" <john.fastabend@gmail.com>,
"Donald Hunter" <donald.hunter@gmail.com>,
"Andrew Lunn" <andrew+netdev@lunn.ch>,
"Jonathan Corbet" <corbet@lwn.net>,
"Shuah Khan" <skhan@linuxfoundation.org>,
"Shuah Khan" <shuah@kernel.org>,
"Randy Dunlap" <rdunlap@infradead.org>,
"Breno Leitao" <leitao@debian.org>,
"Asbjørn Sloth Tønnesen" <ast@fiberby.net>,
"Danielle Ratson" <danieller@nvidia.com>,
"Kuniyuki Iwashima" <kuniyu@google.com>,
bpf@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kselftest@vger.kernel.org, kernel-team@cloudflare.com,
nh-open-source@amazon.com,
"Stephen Hemminger" <stephen@networkplumber.org>,
"Gustavo A. R. Silva" <gustavoars@kernel.org>,
"Remy D. Farley" <one-d-wide@protonmail.com>,
"Amery Hung" <ameryhung@gmail.com>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Andy Roulin" <aroulin@nvidia.com>,
"Anil Kaushik" <anilkaushikwireless@gmail.com>,
"Demos Ra" <demosthenes.ra@gmail.com>,
"Eduard Zingerman" <eddyz87@gmail.com>,
"Emil Tsalapatis" <emil@etsalapatis.com>,
"Hangbin Liu" <hangbin.liu@linux.dev>,
"Ihor Solodrai" <ihor.solodrai@linux.dev>,
"Jiayuan Chen" <jiayuan.chen@linux.dev>,
"Jiri Olsa" <jolsa@kernel.org>, "Joe Damato" <joe@dama.to>,
"Kumar Kartikeya Dwivedi" <memxor@gmail.com>,
"Li RongQing" <lirongqing@baidu.com>,
"Louis Scalbert" <louis.scalbert@6wind.com>,
"Marc Kleine-Budde" <mkl@pengutronix.de>,
"Martin KaFai Lau" <martin.lau@linux.dev>,
"Martin Karsten" <mkarsten@uwaterloo.ca>,
"Matt Bobrowski" <mattbobrowski@google.com>,
"Michael Bommarito" <michael.bommarito@gmail.com>,
"Nikolay Aleksandrov" <razor@blackwall.org>,
"Runyu Xiao" <runyu.xiao@seu.edu.cn>,
"Song Liu" <song@kernel.org>,
"Thorsten Leemhuis" <linux@leemhuis.info>,
"Wang Zhan" <wang.zhan@smartx.com>,
"Yonghong Song" <yonghong.song@linux.dev>,
"Zhou Qiuyue" <n0ps1edzz@gmail.com>,
"Tianyi Gao" <tianyi.gao@linux.dev>,
"Tianyi Gao" <tianyi@cloudflare.com>
Subject: [PATCH net-next v2 01/11] net: add tx_extra_headroom and netdev_get_alloc_headroom()
Date: Thu, 19 Nov 2026 01:38:20 +0000 [thread overview]
Message-ID: <20261119013830.36935-2-tianyi@cloudflare.com> (raw)
In-Reply-To: <20261119013830.36935-1-tianyi@cloudflare.com>
From: David Keogh <dkeogh@amazon.com>
Programs attached at TC egress commonly push additional headers onto a
packet, for example to encapsulate it. When the packet does not carry
enough headroom the BPF helpers reallocate the head, which costs an
allocation and a copy per packet.
Locally generated UDP, raw and AF_PACKET traffic is allocated with
exactly LL_RESERVED_SPACE() of headroom, which only covers the
link-layer header and the driver's needed_headroom.
Add a tx_extra_headroom field to struct net_device for that purpose,
together with netdev_get_alloc_headroom() and
netdev_get_alloc_headroom_hlen(). These add the extra reservation to the
header length before the HH_DATA_MOD alignment, so the alignment of the
link-layer header does not change. netdev_get_alloc_needed_headroom()
returns needed_headroom plus the extra reservation, for callers that do
their own alignment.
The new functions are intended only for sites that allocate new packets.
LL_RESERVED_SPACE() keeps its current meaning and remains the threshold
used when checking whether an existing packet must be expanded, so the
extra reservation never forces a reallocation on its own: packets that
already have enough room for the headers being pushed, such as TCP
segments built with MAX_TCP_HEADER, are left alone.
Keep the extra reservation separate from needed_headroom so that driver
updates of their own requirement do not clobber it. Place the field in
the padding that follows needed_headroom, which keeps it in the TX
read-mostly cacheline group.
The field is not set anywhere yet. Later patches set it from XDP on a
veth peer, from TC egress programs and from a veth attribute.
Also remove LL_RESERVED_SPACE_EXTRA(). It has had no user since
commit 6d5431555de0 ("caif: remove CAIF NETWORK LAYER"). It could add
the extra headroom, but it reads dev->hard_header_len itself. AF_PACKET
must use the value that it read once, see commit 03390aa32e66 ("packet:
use consistent hard_header_len in non-ring send paths").
netdev_get_alloc_headroom_hlen() takes the header length as an argument
instead.
Signed-off-by: David Keogh <dkeogh@amazon.com>
Co-developed-by: Tianyi Gao <tianyi@cloudflare.com>
Signed-off-by: Tianyi Gao <tianyi@cloudflare.com>
---
v2:
- Turn LL_ALLOC_SPACE() and LL_ALLOC_SPACE_EX() into the static inline
functions netdev_get_alloc_headroom() and
netdev_get_alloc_headroom_hlen(), and rename
netdev_tx_alloc_headroom() to netdev_get_alloc_needed_headroom()
(Willem).
- Move the netdevices.rst section here from v1 3/8, without the IFLA
part.
---
.../networking/net_cachelines/net_device.rst | 1 +
Documentation/networking/netdevices.rst | 13 +++++++
include/linux/netdevice.h | 34 +++++++++++++++++--
net/core/dev.c | 1 +
4 files changed, 46 insertions(+), 3 deletions(-)
diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
index 7620d3fcb8dc..4341e9e443ba 100644
--- a/Documentation/networking/net_cachelines/net_device.rst
+++ b/Documentation/networking/net_cachelines/net_device.rst
@@ -35,6 +35,7 @@ unsigned_short gflags
unsigned_short hard_header_len read_mostly read_mostly ip6_xmit(tx);gro_list_prepare(rx)
unsigned_int mtu read_mostly ip_finish_output2
unsigned_short needed_headroom read_mostly LL_RESERVED_SPACE/ip_finish_output2
+unsigned_short tx_extra_headroom read_mostly netdev_get_alloc_headroom/__ip_append_data
unsigned_short needed_tailroom
netdev_features_t features read_mostly read_mostly HARD_TX_LOCK,netif_skb_features,sk_setup_caps(tx);netif_elide_gro(rx)
netdev_features_t hw_features
diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst
index 6a87f09e7015..f593b242da9c 100644
--- a/Documentation/networking/netdevices.rst
+++ b/Documentation/networking/netdevices.rst
@@ -441,6 +441,19 @@ from a notifier.
Drivers **must not** generate nested notifications of the ops-locked types.
+TX headroom reservation
+=======================
+
+``tx_extra_headroom`` is extra headroom for locally generated packets. The
+stack adds it when it allocates such a packet for the device, so that
+headers pushed later, for example at TC egress, fit without a
+reallocation. It is at most 512 bytes.
+
+Unlike ``needed_headroom``, it is not a driver requirement and stacked
+devices do not inherit it. It is not part of ``LL_RESERVED_SPACE()``, so a
+packet that reaches the device with less headroom, such as a forwarded
+packet, is not reallocated because of it.
+
NETDEV_INTERNAL symbol namespace
================================
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0325c6f2a7d1..392e6fc81e34 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -311,9 +311,6 @@ struct hh_cache {
& ~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
#define LL_RESERVED_SPACE(dev) \
LL_RESERVED_SPACE_EX(dev, (dev)->hard_header_len)
-#define LL_RESERVED_SPACE_EXTRA(dev,extra) \
- ((((dev)->hard_header_len + READ_ONCE((dev)->needed_headroom) + (extra)) \
- & ~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
struct header_ops {
int (*create) (struct sk_buff *skb, struct net_device *dev,
@@ -1934,6 +1931,8 @@ enum netdev_reg_state {
*
* @needed_headroom: Extra headroom the hardware may need, but not in all
* cases can this be guaranteed
+ * @tx_extra_headroom: Extra headroom for locally generated packets, for
+ * headers pushed after the allocation
* @needed_tailroom: Extra tailroom the hardware may need, but not in all
* cases can this be guaranteed. Some cases also use
* LL_MAX_HEADER instead to allocate the skb
@@ -2196,6 +2195,8 @@ struct net_device {
*/
unsigned int mtu;
unsigned short needed_headroom;
+#define NETDEV_TX_EXTRA_HEADROOM_MAX 512
+ unsigned short tx_extra_headroom;
struct netdev_tc_txq tc_to_txq[TC_MAX_QUEUE];
#ifdef CONFIG_XPS
struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX];
@@ -2778,6 +2779,33 @@ struct netdev_queue *netdev_core_pick_tx(struct net_device *dev,
struct sk_buff *skb,
struct net_device *sb_dev);
+/* Headroom to reserve when allocating a locally generated packet for dev.
+ * Code that checks the headroom of an existing packet keeps using
+ * LL_RESERVED_SPACE(), so tx_extra_headroom never forces a reallocation.
+ */
+static inline unsigned int
+netdev_get_alloc_headroom_hlen(const struct net_device *dev, unsigned int hlen)
+{
+ return LL_RESERVED_SPACE_EX(dev,
+ hlen + READ_ONCE(dev->tx_extra_headroom));
+}
+
+static inline unsigned int
+netdev_get_alloc_headroom(const struct net_device *dev)
+{
+ return netdev_get_alloc_headroom_hlen(dev, dev->hard_header_len);
+}
+
+/* Same as netdev_get_alloc_headroom() without hard_header_len and alignment,
+ * for callers that size the headroom themselves.
+ */
+static inline unsigned int
+netdev_get_alloc_needed_headroom(const struct net_device *dev)
+{
+ return READ_ONCE(dev->needed_headroom) +
+ READ_ONCE(dev->tx_extra_headroom);
+}
+
/* returns the headroom that the master device needs to take in account
* when forwarding to this dev
*/
diff --git a/net/core/dev.c b/net/core/dev.c
index d88af07aea29..85b7c61015c7 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -13692,6 +13692,7 @@ static void __init net_dev_struct_check(void)
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, num_tc);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, mtu);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, needed_headroom);
+ CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tx_extra_headroom);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tc_to_txq);
#ifdef CONFIG_XPS
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, xps_maps);
--
2.56.0
next prev parent reply other threads:[~2026-11-19 1:38 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 1:38 [PATCH net-next v2 00/11] net: extra TX headroom for local egress Tianyi Gao
2026-11-19 1:38 ` Tianyi Gao [this message]
2026-11-19 1:38 ` [PATCH net-next v2 02/11] net: reserve " Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 03/11] net: set tx_extra_headroom of the peer on XDP attach Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 04/11] selftests: net: add flag to send a frame in xdp_helper Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 05/11] selftests: net: add tx_extra_headroom test Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 06/11] bpf, tcx: Set tx_extra_headroom from a decl tag on egress programs Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 07/11] selftests: net: tx_extra_headroom: test a push at TC egress Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 08/11] selftests/bpf: Test the tx_extra_headroom decl tag Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 09/11] veth: add VETH_INFO_TX_EXTRA_HEADROOM Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 10/11] netlink: specs: rt-link: add veth attributes Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 11/11] selftests: net: tx_extra_headroom: test the veth attribute Tianyi Gao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119013830.36935-2-tianyi@cloudflare.com \
--to=tianyi@cloudflare.com \
--cc=alasdair@mcwilliam.dev \
--cc=ameryhung@gmail.com \
--cc=andrew+netdev@lunn.ch \
--cc=andrii@kernel.org \
--cc=anilkaushikwireless@gmail.com \
--cc=aroulin@nvidia.com \
--cc=ast@fiberby.net \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=daniel@iogearbox.net \
--cc=danieller@nvidia.com \
--cc=davem@davemloft.net \
--cc=demosthenes.ra@gmail.com \
--cc=dkeogh@amazon.com \
--cc=donald.hunter@gmail.com \
--cc=dsahern@kernel.org \
--cc=dtatulea@nvidia.com \
--cc=dwmw2@infradead.org \
--cc=eddyz87@gmail.com \
--cc=edumazet@kernel.org \
--cc=emil@etsalapatis.com \
--cc=gustavoars@kernel.org \
--cc=hangbin.liu@linux.dev \
--cc=hawk@kernel.org \
--cc=horms@kernel.org \
--cc=idosch@nvidia.com \
--cc=ihor.solodrai@linux.dev \
--cc=jiayuan.chen@linux.dev \
--cc=joe@dama.to \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kernel-team@cloudflare.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leitao@debian.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux@leemhuis.info \
--cc=lirongqing@baidu.com \
--cc=louis.scalbert@6wind.com \
--cc=maciej.fijalkowski@intel.com \
--cc=magnus.karlsson@intel.com \
--cc=martin.lau@linux.dev \
--cc=mattbobrowski@google.com \
--cc=memxor@gmail.com \
--cc=michael.bommarito@gmail.com \
--cc=mkarsten@uwaterloo.ca \
--cc=mkl@pengutronix.de \
--cc=n0ps1edzz@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=nh-open-source@amazon.com \
--cc=noren@nvidia.com \
--cc=one-d-wide@protonmail.com \
--cc=pabeni@redhat.com \
--cc=razor@blackwall.org \
--cc=rdunlap@infradead.org \
--cc=runyu.xiao@seu.edu.cn \
--cc=sdf@fomichev.me \
--cc=shuah@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=song@kernel.org \
--cc=stephen@networkplumber.org \
--cc=tianyi.gao@linux.dev \
--cc=wang.zhan@smartx.com \
--cc=willemdebruijn.kernel@gmail.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox