From: Tianyi Gao <tianyi@cloudflare.com>
To: netdev@vger.kernel.org, Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@kernel.org>
Cc: "David S. Miller" <davem@davemloft.net>,
"Simon Horman" <horms@kernel.org>,
"David Keogh" <dkeogh@amazon.com>,
"Jesper Dangaard Brouer" <hawk@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"David Woodhouse" <dwmw2@infradead.org>,
"Dragos Tatulea" <dtatulea@nvidia.com>,
"Nimrod Oren" <noren@nvidia.com>,
"Alasdair McWilliam" <alasdair@mcwilliam.dev>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Maciej Fijalkowski" <maciej.fijalkowski@intel.com>,
"Stanislav Fomichev" <sdf@fomichev.me>,
"Magnus Karlsson" <magnus.karlsson@intel.com>,
"David Ahern" <dsahern@kernel.org>,
"Ido Schimmel" <idosch@nvidia.com>,
"Alexei Starovoitov" <ast@kernel.org>,
"John Fastabend" <john.fastabend@gmail.com>,
"Donald Hunter" <donald.hunter@gmail.com>,
"Andrew Lunn" <andrew+netdev@lunn.ch>,
"Jonathan Corbet" <corbet@lwn.net>,
"Shuah Khan" <skhan@linuxfoundation.org>,
"Shuah Khan" <shuah@kernel.org>,
"Randy Dunlap" <rdunlap@infradead.org>,
"Breno Leitao" <leitao@debian.org>,
"Asbjørn Sloth Tønnesen" <ast@fiberby.net>,
"Danielle Ratson" <danieller@nvidia.com>,
"Kuniyuki Iwashima" <kuniyu@google.com>,
bpf@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kselftest@vger.kernel.org, kernel-team@cloudflare.com,
nh-open-source@amazon.com,
"Stephen Hemminger" <stephen@networkplumber.org>,
"Gustavo A. R. Silva" <gustavoars@kernel.org>,
"Remy D. Farley" <one-d-wide@protonmail.com>,
"Amery Hung" <ameryhung@gmail.com>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Andy Roulin" <aroulin@nvidia.com>,
"Anil Kaushik" <anilkaushikwireless@gmail.com>,
"Demos Ra" <demosthenes.ra@gmail.com>,
"Eduard Zingerman" <eddyz87@gmail.com>,
"Emil Tsalapatis" <emil@etsalapatis.com>,
"Hangbin Liu" <hangbin.liu@linux.dev>,
"Ihor Solodrai" <ihor.solodrai@linux.dev>,
"Jiayuan Chen" <jiayuan.chen@linux.dev>,
"Jiri Olsa" <jolsa@kernel.org>, "Joe Damato" <joe@dama.to>,
"Kumar Kartikeya Dwivedi" <memxor@gmail.com>,
"Li RongQing" <lirongqing@baidu.com>,
"Louis Scalbert" <louis.scalbert@6wind.com>,
"Marc Kleine-Budde" <mkl@pengutronix.de>,
"Martin KaFai Lau" <martin.lau@linux.dev>,
"Martin Karsten" <mkarsten@uwaterloo.ca>,
"Matt Bobrowski" <mattbobrowski@google.com>,
"Michael Bommarito" <michael.bommarito@gmail.com>,
"Nikolay Aleksandrov" <razor@blackwall.org>,
"Runyu Xiao" <runyu.xiao@seu.edu.cn>,
"Song Liu" <song@kernel.org>,
"Thorsten Leemhuis" <linux@leemhuis.info>,
"Wang Zhan" <wang.zhan@smartx.com>,
"Yonghong Song" <yonghong.song@linux.dev>,
"Zhou Qiuyue" <n0ps1edzz@gmail.com>,
"Tianyi Gao" <tianyi.gao@linux.dev>,
"Tianyi Gao" <tianyi@cloudflare.com>
Subject: [PATCH net-next v2 03/11] net: set tx_extra_headroom of the peer on XDP attach
Date: Thu, 19 Nov 2026 01:38:22 +0000 [thread overview]
Message-ID: <20261119013830.36935-4-tianyi@cloudflare.com> (raw)
In-Reply-To: <20261119013830.36935-1-tianyi@cloudflare.com>
XDP needs XDP_PACKET_HEADROOM (256 bytes), but locally generated packets
only get needed_headroom. XDP on the peer of a veth or netkit device
then copies every packet. Generic XDP copies it in
netif_skb_check_for_xdp(), and native XDP on veth in
veth_convert_skb_to_xdp_buff().
While the peer has an XDP program, set tx_extra_headroom of the device
to XDP_PACKET_HEADROOM minus needed_headroom.
netdev_update_tx_extra_headroom() recalculates it when an XDP program
is attached or detached, also when an upper device such as a bond
propagates its program to the peer with netif_xdp_propagate(). It does
not recalculate it:
- for IFLA_NETKIT_HEADROOM, which cannot change after the netkit
device is created,
- in dev_xdp_uninstall(), because veth and netkit unregister both ends
together,
- when bridge or openvswitch change needed_headroom later.
openvswitch changes needed_headroom under ovs_mutex, without RTNL.
There are three ways to handle a later change of needed_headroom:
- Recalculate tx_extra_headroom in netdev_set_rx_headroom(). This
races with an XDP attach or detach under RTNL. The last write can
leave tx_extra_headroom at 0 while XDP is attached.
- Do not subtract needed_headroom. This always reserves needed_headroom
bytes more than XDP needs.
- Subtract needed_headroom only when tx_extra_headroom is recalculated.
This patch uses the last one, because it does not race and reserves
only what XDP needs. If needed_headroom shrinks later, XDP on the peer
copies packets again until the next recalculation.
tx_extra_headroom is used rather than needed_headroom, because raising
needed_headroom can regress the performance of other paths.
Signed-off-by: Tianyi Gao <tianyi@cloudflare.com>
---
For native XDP on veth, this patch removes one of two copy
conditions, the missing headroom. The other condition is a head that is
not a page fragment. A follow-up veth series will fix that.
v2:
- New patch, replaces IFLA_TX_EXTRA_HEADROOM for XDP on the peer
(Stephen).
---
Documentation/networking/netdevices.rst | 11 +++++++
include/linux/netdevice.h | 1 +
net/core/dev.c | 38 +++++++++++++++++++++++++
3 files changed, 50 insertions(+)
diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst
index f593b242da9c..53ca8439f73b 100644
--- a/Documentation/networking/netdevices.rst
+++ b/Documentation/networking/netdevices.rst
@@ -454,6 +454,17 @@ devices do not inherit it. It is not part of ``LL_RESERVED_SPACE()``, so a
packet that reaches the device with less headroom, such as a forwarded
packet, is not reallocated because of it.
+The kernel sets ``tx_extra_headroom`` of a device to the sum of these
+values:
+
+* ``XDP_PACKET_HEADROOM`` (256 bytes) minus ``needed_headroom``, when the
+ peer of a veth or netkit device has an XDP program
+
+When the peer has an XDP program, locally generated UDP, raw and
+AF_PACKET packets and AF_XDP packets in copy mode have enough headroom
+for it. ``tx_extra_headroom`` is not recalculated when a bridge or
+openvswitch changes ``needed_headroom`` later.
+
NETDEV_INTERNAL symbol namespace
================================
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 392e6fc81e34..786b63f21d4f 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -4484,6 +4484,7 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf);
int dev_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf);
u8 dev_xdp_sb_prog_count(struct net_device *dev);
u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode);
+void netdev_update_tx_extra_headroom(struct net_device *dev);
u32 dev_get_min_mp_channel_count(const struct net_device *dev);
diff --git a/net/core/dev.c b/net/core/dev.c
index 85b7c61015c7..6e053f612d72 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -10666,6 +10666,7 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
if (old_prog)
bpf_prog_put(old_prog);
+ netdev_update_tx_extra_headroom(dev);
return 0;
}
EXPORT_SYMBOL_GPL(netif_xdp_propagate);
@@ -10677,6 +10678,41 @@ u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode)
return prog ? prog->aux->id : 0;
}
+static void __netdev_update_tx_extra_headroom(struct net_device *dev,
+ struct net_device *peer)
+{
+ unsigned int needed_headroom = READ_ONCE(dev->needed_headroom);
+ unsigned int tx_extra_headroom = 0;
+
+ if (peer && dev_xdp_prog_count(peer) &&
+ needed_headroom < XDP_PACKET_HEADROOM)
+ tx_extra_headroom = XDP_PACKET_HEADROOM - needed_headroom;
+
+ WRITE_ONCE(dev->tx_extra_headroom, tx_extra_headroom);
+}
+
+/**
+ * netdev_update_tx_extra_headroom - recalculate tx_extra_headroom
+ * @dev: the device to check
+ *
+ * Recalculate tx_extra_headroom of @dev and its peer. Should be called
+ * after conditions that influence tx_extra_headroom might have changed.
+ * The caller must hold the RTNL lock.
+ */
+void netdev_update_tx_extra_headroom(struct net_device *dev)
+{
+ struct net_device *peer = NULL;
+
+ rcu_read_lock();
+ if (dev->netdev_ops->ndo_get_peer_dev)
+ peer = dev->netdev_ops->ndo_get_peer_dev(dev);
+ __netdev_update_tx_extra_headroom(dev, peer);
+ if (peer)
+ __netdev_update_tx_extra_headroom(peer, dev);
+ rcu_read_unlock();
+}
+EXPORT_SYMBOL_GPL(netdev_update_tx_extra_headroom);
+
static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
bpf_op_t bpf_op, struct netlink_ext_ack *extack,
u32 flags, struct bpf_prog *prog)
@@ -10887,6 +10923,7 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack
if (cur_prog)
bpf_prog_put(cur_prog);
+ netdev_update_tx_extra_headroom(dev);
return 0;
}
@@ -10913,6 +10950,7 @@ static int dev_xdp_detach_link(struct net_device *dev,
bpf_op = dev_xdp_bpf_op(dev, mode);
WARN_ON(dev_xdp_install(dev, mode, bpf_op, NULL, 0, NULL));
dev_xdp_set_link(dev, mode, NULL);
+ netdev_update_tx_extra_headroom(dev);
return 0;
}
--
2.56.0
next prev parent reply other threads:[~2026-11-19 1:38 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 1:38 [PATCH net-next v2 00/11] net: extra TX headroom for local egress Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 01/11] net: add tx_extra_headroom and netdev_get_alloc_headroom() Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 02/11] net: reserve extra TX headroom for local egress Tianyi Gao
2026-11-19 1:38 ` Tianyi Gao [this message]
2026-11-19 1:38 ` [PATCH net-next v2 04/11] selftests: net: add flag to send a frame in xdp_helper Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 05/11] selftests: net: add tx_extra_headroom test Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 06/11] bpf, tcx: Set tx_extra_headroom from a decl tag on egress programs Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 07/11] selftests: net: tx_extra_headroom: test a push at TC egress Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 08/11] selftests/bpf: Test the tx_extra_headroom decl tag Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 09/11] veth: add VETH_INFO_TX_EXTRA_HEADROOM Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 10/11] netlink: specs: rt-link: add veth attributes Tianyi Gao
2026-11-19 1:38 ` [PATCH net-next v2 11/11] selftests: net: tx_extra_headroom: test the veth attribute Tianyi Gao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119013830.36935-4-tianyi@cloudflare.com \
--to=tianyi@cloudflare.com \
--cc=alasdair@mcwilliam.dev \
--cc=ameryhung@gmail.com \
--cc=andrew+netdev@lunn.ch \
--cc=andrii@kernel.org \
--cc=anilkaushikwireless@gmail.com \
--cc=aroulin@nvidia.com \
--cc=ast@fiberby.net \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=daniel@iogearbox.net \
--cc=danieller@nvidia.com \
--cc=davem@davemloft.net \
--cc=demosthenes.ra@gmail.com \
--cc=dkeogh@amazon.com \
--cc=donald.hunter@gmail.com \
--cc=dsahern@kernel.org \
--cc=dtatulea@nvidia.com \
--cc=dwmw2@infradead.org \
--cc=eddyz87@gmail.com \
--cc=edumazet@kernel.org \
--cc=emil@etsalapatis.com \
--cc=gustavoars@kernel.org \
--cc=hangbin.liu@linux.dev \
--cc=hawk@kernel.org \
--cc=horms@kernel.org \
--cc=idosch@nvidia.com \
--cc=ihor.solodrai@linux.dev \
--cc=jiayuan.chen@linux.dev \
--cc=joe@dama.to \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kernel-team@cloudflare.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leitao@debian.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux@leemhuis.info \
--cc=lirongqing@baidu.com \
--cc=louis.scalbert@6wind.com \
--cc=maciej.fijalkowski@intel.com \
--cc=magnus.karlsson@intel.com \
--cc=martin.lau@linux.dev \
--cc=mattbobrowski@google.com \
--cc=memxor@gmail.com \
--cc=michael.bommarito@gmail.com \
--cc=mkarsten@uwaterloo.ca \
--cc=mkl@pengutronix.de \
--cc=n0ps1edzz@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=nh-open-source@amazon.com \
--cc=noren@nvidia.com \
--cc=one-d-wide@protonmail.com \
--cc=pabeni@redhat.com \
--cc=razor@blackwall.org \
--cc=rdunlap@infradead.org \
--cc=runyu.xiao@seu.edu.cn \
--cc=sdf@fomichev.me \
--cc=shuah@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=song@kernel.org \
--cc=stephen@networkplumber.org \
--cc=tianyi.gao@linux.dev \
--cc=wang.zhan@smartx.com \
--cc=willemdebruijn.kernel@gmail.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox