From: Tianyi Gao <tianyi@cloudflare.com>
To: netdev@vger.kernel.org, Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@kernel.org>
Cc: "David S. Miller" <davem@davemloft.net>,
"Simon Horman" <horms@kernel.org>,
"David Keogh" <dkeogh@amazon.com>,
"Jesper Dangaard Brouer" <hawk@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"David Woodhouse" <dwmw2@infradead.org>,
"Dragos Tatulea" <dtatulea@nvidia.com>,
"Nimrod Oren" <noren@nvidia.com>,
"Alasdair McWilliam" <alasdair@mcwilliam.dev>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Maciej Fijalkowski" <maciej.fijalkowski@intel.com>,
"Stanislav Fomichev" <sdf@fomichev.me>,
"Magnus Karlsson" <magnus.karlsson@intel.com>,
"David Ahern" <dsahern@kernel.org>,
"Ido Schimmel" <idosch@nvidia.com>,
"Alexei Starovoitov" <ast@kernel.org>,
"John Fastabend" <john.fastabend@gmail.com>,
"Donald Hunter" <donald.hunter@gmail.com>,
"Andrew Lunn" <andrew+netdev@lunn.ch>,
"Jonathan Corbet" <corbet@lwn.net>,
"Shuah Khan" <skhan@linuxfoundation.org>,
"Shuah Khan" <shuah@kernel.org>,
"Randy Dunlap" <rdunlap@infradead.org>,
"Breno Leitao" <leitao@debian.org>,
"Asbjørn Sloth Tønnesen" <ast@fiberby.net>,
"Danielle Ratson" <danieller@nvidia.com>,
"Kuniyuki Iwashima" <kuniyu@google.com>,
bpf@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kselftest@vger.kernel.org, kernel-team@cloudflare.com,
nh-open-source@amazon.com,
"Stephen Hemminger" <stephen@networkplumber.org>,
"Gustavo A. R. Silva" <gustavoars@kernel.org>,
"Remy D. Farley" <one-d-wide@protonmail.com>,
"Amery Hung" <ameryhung@gmail.com>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Andy Roulin" <aroulin@nvidia.com>,
"Anil Kaushik" <anilkaushikwireless@gmail.com>,
"Demos Ra" <demosthenes.ra@gmail.com>,
"Eduard Zingerman" <eddyz87@gmail.com>,
"Emil Tsalapatis" <emil@etsalapatis.com>,
"Hangbin Liu" <hangbin.liu@linux.dev>,
"Ihor Solodrai" <ihor.solodrai@linux.dev>,
"Jiayuan Chen" <jiayuan.chen@linux.dev>,
"Jiri Olsa" <jolsa@kernel.org>, "Joe Damato" <joe@dama.to>,
"Kumar Kartikeya Dwivedi" <memxor@gmail.com>,
"Li RongQing" <lirongqing@baidu.com>,
"Louis Scalbert" <louis.scalbert@6wind.com>,
"Marc Kleine-Budde" <mkl@pengutronix.de>,
"Martin KaFai Lau" <martin.lau@linux.dev>,
"Martin Karsten" <mkarsten@uwaterloo.ca>,
"Matt Bobrowski" <mattbobrowski@google.com>,
"Michael Bommarito" <michael.bommarito@gmail.com>,
"Nikolay Aleksandrov" <razor@blackwall.org>,
"Runyu Xiao" <runyu.xiao@seu.edu.cn>,
"Song Liu" <song@kernel.org>,
"Thorsten Leemhuis" <linux@leemhuis.info>,
"Wang Zhan" <wang.zhan@smartx.com>,
"Yonghong Song" <yonghong.song@linux.dev>,
"Zhou Qiuyue" <n0ps1edzz@gmail.com>,
"Tianyi Gao" <tianyi.gao@linux.dev>,
"Tianyi Gao" <tianyi@cloudflare.com>
Subject: [PATCH net-next v2 06/11] bpf, tcx: Set tx_extra_headroom from a decl tag on egress programs
Date: Thu, 19 Nov 2026 01:38:25 +0000 [thread overview]
Message-ID: <20261119013830.36935-7-tianyi@cloudflare.com> (raw)
In-Reply-To: <20261119013830.36935-1-tianyi@cloudflare.com>
A program at TC egress can push headers, for example to encapsulate a
packet. A locally generated packet gets only LL_RESERVED_SPACE() of
headroom, so bpf_skb_adjust_room() reallocates every such packet.
The kernel cannot know how many bytes a program pushes. Let the program
declare it with a BTF decl tag on its main function:
SEC("tcx/egress")
__attribute__((btf_decl_tag("tx_extra_headroom:64")))
int prog(struct __sk_buff *skb)
The verifier parses the tag when the program is loaded, as it does for
"exception_callback:", and stores the value in struct bpf_prog_aux. The
new field fills a hole after freplace_link_cnt, so struct bpf_prog_aux
does not grow. The verifier rejects the program if the tag does not
parse or exceeds NETDEV_TX_EXTRA_HEADROOM_MAX (512 bytes).
tx_extra_headroom is then recalculated with
netdev_update_tx_extra_headroom() under RTNL whenever an egress program
is attached, detached or replaced.
The egress programs of a device run one after another, so the headers
that they push can add up. Their tags are summed and added to the
headroom for XDP on the veth peer. The total is capped at
NETDEV_TX_EXTRA_HEADROOM_MAX.
Tags on programs at tcx ingress, on tail call targets and on freplace
programs are ignored, like "exception_callback:" on subprogs. An ingress
program does not see transmitted packets. A tail call target or a
freplace program is never attached at tcx, so its tag is never added.
The tag on the attached program covers the whole chain.
Only tcx is supported. A cls_bpf filter is attached to a tc block,
which can be shared by several devices.
Suggested-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Tianyi Gao <tianyi@cloudflare.com>
---
v2:
- New patch, replaces IFLA_TX_EXTRA_HEADROOM for the TC egress case
(Dragos, Willem, Stephen).
---
Documentation/networking/netdevices.rst | 15 ++++++++++++
include/linux/bpf.h | 1 +
include/net/tcx.h | 6 +++++
kernel/bpf/tcx.c | 24 +++++++++++++++++++
kernel/bpf/verifier.c | 31 +++++++++++++++++++++++++
net/core/dev.c | 4 +++-
6 files changed, 80 insertions(+), 1 deletion(-)
diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst
index 53ca8439f73b..128a93303a6d 100644
--- a/Documentation/networking/netdevices.rst
+++ b/Documentation/networking/netdevices.rst
@@ -459,12 +459,27 @@ values:
* ``XDP_PACKET_HEADROOM`` (256 bytes) minus ``needed_headroom``, when the
peer of a veth or netkit device has an XDP program
+* the headroom that the programs at tcx egress of the device declare
When the peer has an XDP program, locally generated UDP, raw and
AF_PACKET packets and AF_XDP packets in copy mode have enough headroom
for it. ``tx_extra_headroom`` is not recalculated when a bridge or
openvswitch changes ``needed_headroom`` later.
+A program at tcx egress declares its headroom with a BTF decl tag on its
+main function. The value is a decimal number of bytes:
+
+.. code-block:: c
+
+ SEC("tcx/egress")
+ __attribute__((btf_decl_tag("tx_extra_headroom:64")))
+ int prog(struct __sk_buff *skb)
+
+The tag of the attached program covers the whole tail call chain. Tags on
+tail call targets, freplace programs and programs at tcx ingress are
+ignored. Loading a program whose tag is above 512 or does not parse fails
+with ``-EINVAL``.
+
NETDEV_INTERNAL symbol namespace
================================
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index fdbf638300fc..bca6d0b960cb 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1952,6 +1952,7 @@ struct bpf_prog_aux {
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
u16 stack_arg_sp_adjust;
u16 freplace_link_cnt; /* counts freplace links extending this prog */
+ u16 tx_extra_headroom; /* from the "tx_extra_headroom:" decl tag */
#ifdef CONFIG_SECURITY
void *security;
#endif
diff --git a/include/net/tcx.h b/include/net/tcx.h
index 23a61af13547..9ab299ecd902 100644
--- a/include/net/tcx.h
+++ b/include/net/tcx.h
@@ -164,6 +164,7 @@ int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int tcx_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int tcx_prog_detach(const union bpf_attr *attr, struct bpf_prog *prog);
void tcx_uninstall(struct net_device *dev, bool ingress);
+unsigned int tcx_tx_extra_headroom(struct net_device *dev);
int tcx_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
@@ -193,6 +194,11 @@ static inline int tcx_prog_detach(const union bpf_attr *attr,
return -EINVAL;
}
+static inline unsigned int tcx_tx_extra_headroom(struct net_device *dev)
+{
+ return 0;
+}
+
static inline int tcx_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
diff --git a/kernel/bpf/tcx.c b/kernel/bpf/tcx.c
index 02db0113b8e7..99da16e13981 100644
--- a/kernel/bpf/tcx.c
+++ b/kernel/bpf/tcx.c
@@ -46,6 +46,8 @@ int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
tcx_skeys_inc(ingress);
}
bpf_mprog_commit(entry);
+ if (!ingress)
+ netdev_update_tx_extra_headroom(dev);
} else if (created) {
tcx_entry_free(entry);
}
@@ -86,6 +88,8 @@ int tcx_prog_detach(const union bpf_attr *attr, struct bpf_prog *prog)
bpf_mprog_commit(entry);
if (!entry_new)
tcx_entry_free(entry);
+ if (!ingress)
+ netdev_update_tx_extra_headroom(dev);
}
out:
rtnl_unlock();
@@ -119,6 +123,20 @@ void tcx_uninstall(struct net_device *dev, bool ingress)
tcx_entry_free(entry);
}
+unsigned int tcx_tx_extra_headroom(struct net_device *dev)
+{
+ struct bpf_mprog_entry *entry = tcx_entry_fetch(dev, false);
+ unsigned int headroom = 0;
+ struct bpf_mprog_fp *fp;
+ struct bpf_prog *prog;
+
+ if (!entry)
+ return 0;
+ bpf_mprog_foreach_prog(entry, fp, prog)
+ headroom += prog->aux->tx_extra_headroom;
+ return headroom;
+}
+
int tcx_prog_query(const union bpf_attr *attr, union bpf_attr __user *uattr)
{
bool ingress = attr->query.attach_type == BPF_TCX_INGRESS;
@@ -160,6 +178,8 @@ static int tcx_link_prog_attach(struct bpf_link *link, u32 flags, u32 id_or_fd,
tcx_skeys_inc(ingress);
}
bpf_mprog_commit(entry);
+ if (!ingress)
+ netdev_update_tx_extra_headroom(dev);
} else if (created) {
tcx_entry_free(entry);
}
@@ -193,6 +213,8 @@ static void tcx_link_release(struct bpf_link *link)
bpf_mprog_commit(entry);
if (!entry_new)
tcx_entry_free(entry);
+ if (!ingress)
+ netdev_update_tx_extra_headroom(dev);
tcx->dev = NULL;
}
out:
@@ -237,6 +259,8 @@ static int tcx_link_update(struct bpf_link *link, struct bpf_prog *nprog,
oprog = xchg(&link->prog, nprog);
bpf_prog_put(oprog);
bpf_mprog_commit(entry);
+ if (!ingress)
+ netdev_update_tx_extra_headroom(dev);
}
out:
rtnl_unlock();
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5e4700fda66b..e241ee20927a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2683,6 +2683,33 @@ static int add_subprog(struct bpf_verifier_env *env, int off)
return i;
}
+static int bpf_find_tx_extra_headroom(struct bpf_verifier_env *env)
+{
+ struct bpf_prog_aux *aux = env->prog->aux;
+ const struct btf_type *t;
+ const char *value;
+ u16 headroom;
+
+ if (!aux->func_info_cnt)
+ return 0;
+ t = btf_type_by_id(aux->btf, aux->func_info[0].type_id);
+ value = btf_find_decl_tag_value(aux->btf, t, -1, "tx_extra_headroom:");
+ if (IS_ERR(value)) {
+ if (PTR_ERR(value) == -ENOENT)
+ return 0;
+ verbose(env,
+ "multiple tx_extra_headroom tags for main subprog\n");
+ return PTR_ERR(value);
+ }
+ if (!isdigit(*value) || kstrtou16(value, 10, &headroom) ||
+ headroom > NETDEV_TX_EXTRA_HEADROOM_MAX) {
+ verbose(env, "invalid tx_extra_headroom tag '%s'\n", value);
+ return -EINVAL;
+ }
+ aux->tx_extra_headroom = headroom;
+ return 0;
+}
+
static int bpf_find_exception_callback_insn_off(struct bpf_verifier_env *env)
{
struct bpf_prog_aux *aux = env->prog->aux;
@@ -3264,6 +3291,10 @@ static int add_subprogs(struct bpf_verifier_env *env)
}
}
+ ret = bpf_find_tx_extra_headroom(env);
+ if (ret < 0)
+ return ret;
+
ret = bpf_find_exception_callback_insn_off(env);
if (ret < 0)
return ret;
diff --git a/net/core/dev.c b/net/core/dev.c
index 6e053f612d72..6d11d5ad13e1 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -10687,8 +10687,10 @@ static void __netdev_update_tx_extra_headroom(struct net_device *dev,
if (peer && dev_xdp_prog_count(peer) &&
needed_headroom < XDP_PACKET_HEADROOM)
tx_extra_headroom = XDP_PACKET_HEADROOM - needed_headroom;
+ tx_extra_headroom += tcx_tx_extra_headroom(dev);
- WRITE_ONCE(dev->tx_extra_headroom, tx_extra_headroom);
+ WRITE_ONCE(dev->tx_extra_headroom,
+ min(tx_extra_headroom, NETDEV_TX_EXTRA_HEADROOM_MAX));
}
/**
--
2.56.0
next prev parent reply other threads:[~2026-11-19 1:38 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 1:38 [PATCH net-next v2 00/11] net: extra TX headroom for local egress Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 01/11] net: add tx_extra_headroom and netdev_get_alloc_headroom() Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 02/11] net: reserve extra TX headroom for local egress Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 03/11] net: set tx_extra_headroom of the peer on XDP attach Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 04/11] selftests: net: add flag to send a frame in xdp_helper Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 05/11] selftests: net: add tx_extra_headroom test Tianyi Gao
2026-11-19 1:38 ` Tianyi Gao [this message]
2026-11-19 1:38 ` [PATCH net-next v2 07/11] selftests: net: tx_extra_headroom: test a push at TC egress Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 08/11] selftests/bpf: Test the tx_extra_headroom decl tag Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 09/11] veth: add VETH_INFO_TX_EXTRA_HEADROOM Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 10/11] netlink: specs: rt-link: add veth attributes Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 11/11] selftests: net: tx_extra_headroom: test the veth attribute Tianyi Gao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119013830.36935-7-tianyi@cloudflare.com \
--to=tianyi@cloudflare.com \
--cc=alasdair@mcwilliam.dev \
--cc=ameryhung@gmail.com \
--cc=andrew+netdev@lunn.ch \
--cc=andrii@kernel.org \
--cc=anilkaushikwireless@gmail.com \
--cc=aroulin@nvidia.com \
--cc=ast@fiberby.net \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=daniel@iogearbox.net \
--cc=danieller@nvidia.com \
--cc=davem@davemloft.net \
--cc=demosthenes.ra@gmail.com \
--cc=dkeogh@amazon.com \
--cc=donald.hunter@gmail.com \
--cc=dsahern@kernel.org \
--cc=dtatulea@nvidia.com \
--cc=dwmw2@infradead.org \
--cc=eddyz87@gmail.com \
--cc=edumazet@kernel.org \
--cc=emil@etsalapatis.com \
--cc=gustavoars@kernel.org \
--cc=hangbin.liu@linux.dev \
--cc=hawk@kernel.org \
--cc=horms@kernel.org \
--cc=idosch@nvidia.com \
--cc=ihor.solodrai@linux.dev \
--cc=jiayuan.chen@linux.dev \
--cc=joe@dama.to \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kernel-team@cloudflare.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leitao@debian.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux@leemhuis.info \
--cc=lirongqing@baidu.com \
--cc=louis.scalbert@6wind.com \
--cc=maciej.fijalkowski@intel.com \
--cc=magnus.karlsson@intel.com \
--cc=martin.lau@linux.dev \
--cc=mattbobrowski@google.com \
--cc=memxor@gmail.com \
--cc=michael.bommarito@gmail.com \
--cc=mkarsten@uwaterloo.ca \
--cc=mkl@pengutronix.de \
--cc=n0ps1edzz@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=nh-open-source@amazon.com \
--cc=noren@nvidia.com \
--cc=one-d-wide@protonmail.com \
--cc=pabeni@redhat.com \
--cc=razor@blackwall.org \
--cc=rdunlap@infradead.org \
--cc=runyu.xiao@seu.edu.cn \
--cc=sdf@fomichev.me \
--cc=shuah@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=song@kernel.org \
--cc=stephen@networkplumber.org \
--cc=tianyi.gao@linux.dev \
--cc=wang.zhan@smartx.com \
--cc=willemdebruijn.kernel@gmail.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox