[RFC net-next 0/6] net: configurable extra TX headroom for local egress

[RFC net-next 0/6] net: configurable extra TX headroom for local egress

[David Keogh]

A program attached at TC egress commonly pushes additional headers onto a
packet, for example to encapsulate it. Locally generated UDP, raw and
AF_PACKET traffic is allocated with exactly LL_RESERVED_SPACE() of headroom,
which covers the link-layer header plus the driver's needed_headroom and
alignment, leaving no slack for anything an egress program pushes, so that
push reallocates the skb head on every packet. There is currently no in-tree
way for an administrator to ask for more headroom at allocation time.

Add a per-device tx_extra_headroom, configured through a new
IFLA_TX_EXTRA_HEADROOM rtnetlink attribute. It is a separate value from
needed_headroom: it is added on top of LL_RESERVED_SPACE() at the allocation
sites only (LL_ALLOC_SPACE()), it can be set and cleared freely without
recreating the device, it is reported through its own attribute so it is never
confused with the driver's requirement, and it is not inherited by stacked
devices. needed_headroom keeps its current meaning and remains the threshold
for deciding whether an existing packet must be expanded, so the extra
reservation never forces a reallocation on its own.

The value is bounded to NETDEV_TX_EXTRA_HEADROOM_MAX (512 bytes) by the
netlink policy, enough for several stacked encapsulations (for example
VXLAN, GRE or IPIP) while keeping the growth of per-packet allocations
modest; a request above it is rejected. The attribute is purely additive, so
an older iproute2 that does not know it simply never sends it; configuring the
value requires both kernel and iproute2 support, and old userspace that
ignores unknown dump attributes is unaffected.

Why a new value rather than letting userspace raise needed_headroom: that was
the first approach and it was dropped. needed_headroom is a floor the device
and stacked devices depend on and is consulted on the expand-on-demand path
(ip_finish_output2()), so raising it reallocates every packet that lacks the
full reservation, including TCP segments, and it cannot be lowered without
recreating the device. tx_extra_headroom sidesteps both problems by only
affecting fresh allocations and by being freely clearable.

The reservation applies to the device the packet is allocated against (the one
the route selects) at skb-allocation time. A GSO super-skb is therefore
allocated once with the headroom in place; software segmentation copies that
headroom into each segment, so the push lands in reserved space for every
segment without a per-segment realloc. net/xdp/xsk.c is included among the
converted paths: AF_XDP copy mode open-codes the same needed_headroom
calculation, so it uses netdev_tx_alloc_headroom() to pick up the extra
reservation consistently with the socket paths.

Because the reservation is applied only at the allocation sites, TCP is left
alone: TCP segments are built with MAX_TCP_HEADER already reserved and are
checked against LL_RESERVED_SPACE(), not LL_ALLOC_SPACE(), so raising the knob
does not reallocate them. Covering the TCP allocation path is deliberately not
part of this series.

Measurements (net-next at commit 05b5019b0, veth, 500000 UDP datagrams, a
20-byte ip-in-ip push attached at TC egress, only tx_extra_headroom varied):

  tx_extra_headroom   pskb_expand_head reallocs / 500k
  0 (off)             500003   (~1 per packet)
  64                  4        (background)
  128                 2        (background)

Idle case, same rig with the knob set but no TC program attached: 500000 UDP
datagrams produce 0 reallocs on the device. With nothing pushing headers there
is nothing to expand, so a set-but-unused value costs only the reserved bytes
per allocation, with no added reallocation or copy.

The small background counts at 64 and 128 are control traffic on the device
(ARP/ND and pre-neighbour-resolution skbs), not the UDP push path; the idle
case reads 0 because that run primes a permanent neighbour first.

The data-plane selftest also sweeps the knob across the exact headroom the
push needs and shows the realloc count flip at the predicted byte boundary
(veth LL_RESERVED_SPACE is 16; a 14-byte L2 header leaves 2 free, so a 20-byte
push needs 18 more: reallocs persist at 17 and vanish at 18). A UDP-only encap
leaves a concurrent TCP stream delivering cleanly with the realloc count flat
as the knob is raised, confirming the TCP path is untouched.

Patch 1 adds the field and the LL_ALLOC_SPACE() macros. Patch 2 switches the
local-egress allocation sites to them (UDP, raw, AF_PACKET and AF_XDP). Patch 3
exposes the attribute through rtnetlink with documentation and the rt-link spec
entry. Patch 4 tests the attribute itself through YNL. Patches 5 and 6 add the
data-plane test: two BPF programs and the YNL test that drives them. The BPF
programs live under selftests/bpf/progs because that tree supplies the
clang -target bpf toolchain and generated vmlinux.h the CO-RE fentry counter
needs, while the test driver lives in selftests/net; this is noted here
because it has a bearing on how the series is applied.

As an RFC question: the data-plane test spans two selftest trees because the
fentry realloc counter is a CO-RE program that needs the vmlinux.h generated
under tools/testing/selftests/bpf, while the driver lives in
tools/testing/selftests/net. I would appreciate guidance on the preferred
structure: consolidate the whole data-plane test under selftests/bpf, or keep
it in selftests/net without CO-RE so it builds self-contained. The core change
(patches 1-3) is independent of this and targets net-next regardless.

David Keogh (6):
  net: add tx_extra_headroom and LL_ALLOC_SPACE()
  net: reserve extra TX headroom for local egress
  rtnetlink: add IFLA_TX_EXTRA_HEADROOM
  selftests: net: add tx_extra_headroom test
  selftests/bpf: add tx_extra_headroom data-plane programs
  selftests/net: add tx_extra_headroom data-plane test

 Documentation/netlink/specs/rt-link.yaml      |  12 ++
 .../networking/net_cachelines/net_device.rst  |   1 +
 Documentation/networking/netdevices.rst       |  34 +++
 include/linux/netdevice.h                     |  33 +++
 include/uapi/linux/if_link.h                  |   1 +
 net/core/dev.c                                |   2 +
 net/core/rtnetlink.c                          |  19 +-
 net/ipv4/ip_output.c                          |   2 +-
 net/ipv4/raw.c                                |   2 +-
 net/ipv6/ip6_output.c                         |   2 +-
 net/ipv6/raw.c                                |   2 +-
 net/packet/af_packet.c                        |   6 +-
 net/xdp/xsk.c                                 |   6 +-
 .../bpf/progs/tx_extra_headroom_probe.c       |  68 ++++++
 .../bpf/progs/tx_extra_headroom_tc.bpf.c      |  69 ++++++
 tools/testing/selftests/net/Makefile          |   2 +
 .../selftests/net/tx_extra_headroom.py        | 137 ++++++++++++
 .../net/tx_extra_headroom_dataplane.py        | 197 ++++++++++++++++++
 18 files changed, 585 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/tx_extra_headroom_probe.c
 create mode 100644 tools/testing/selftests/bpf/progs/tx_extra_headroom_tc.bpf.c
 create mode 100755 tools/testing/selftests/net/tx_extra_headroom.py
 create mode 100755 tools/testing/selftests/net/tx_extra_headroom_dataplane.py

-- 
2.50.1

[RFC net-next 1/6] net: add tx_extra_headroom and LL_ALLOC_SPACE()

[David Keogh]

Programs attached at TC egress commonly push additional headers onto a
packet, for example to encapsulate it. When the packet does not carry
enough headroom the BPF helpers reallocate the head, which costs an
allocation and a copy per packet.

Locally generated UDP, raw and AF_PACKET traffic is allocated with
exactly LL_RESERVED_SPACE() of headroom, which only covers the driver's
own requirement in needed_headroom. There is currently no way for an
administrator to ask for more headroom at allocation time without
changing the driver.

Add a tx_extra_headroom field to struct net_device for that purpose,
together with LL_ALLOC_SPACE() and LL_ALLOC_SPACE_EX(). These add the
extra reservation on top of LL_RESERVED_SPACE() while keeping the
link-layer header alignment unchanged.

The new macros are intended only for sites that allocate new packets.
LL_RESERVED_SPACE() keeps its current meaning and remains the threshold
used when checking whether an existing packet must be expanded, so the
extra reservation never forces a reallocation on its own: packets that
already have enough room for the headers being pushed, such as TCP
segments built with MAX_TCP_HEADER, are left alone.

Keep the extra reservation separate from needed_headroom so that driver
updates of their own requirement do not clobber it and so that it can
be cleared independently. Place the field in the padding that follows
needed_headroom, which keeps it in the TX read-mostly cacheline group.

The field is not set anywhere yet. A later patch exposes it through
rtnetlink.

Signed-off-by: David Keogh <dkeogh@amazon.com>
---
 .../networking/net_cachelines/net_device.rst  |  1 +
 include/linux/netdevice.h                     | 33 +++++++++++++++++++
 net/core/dev.c                                |  2 ++
 3 files changed, 36 insertions(+)

diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
index 8eceaa80b..a7e311fb7 100644
--- a/Documentation/networking/net_cachelines/net_device.rst
+++ b/Documentation/networking/net_cachelines/net_device.rst
@@ -35,6 +35,7 @@ unsigned_short                      gflags
 unsigned_short                      hard_header_len             read_mostly         read_mostly         ip6_xmit(tx);gro_list_prepare(rx)
 unsigned_int                        mtu                         read_mostly                             ip_finish_output2
 unsigned_short                      needed_headroom             read_mostly                             LL_RESERVED_SPACE/ip_finish_output2
+unsigned_short                      tx_extra_headroom           read_mostly                             LL_ALLOC_SPACE/__ip_append_data
 unsigned_short                      needed_tailroom
 netdev_features_t                   features                    read_mostly         read_mostly         HARD_TX_LOCK,netif_skb_features,sk_setup_caps(tx);netif_elide_gro(rx)
 netdev_features_t                   hw_features
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 8fbe3a5d8..9a2a12db1 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -309,6 +309,24 @@ struct hh_cache {
 	((((dev)->hard_header_len + READ_ONCE((dev)->needed_headroom) + (extra)) \
 	  & ~(HH_DATA_MOD - 1)) + HH_DATA_MOD)
 
+/*
+ * Headroom to reserve when allocating a locally generated packet for
+ * transmission through a device. On top of LL_RESERVED_SPACE() this adds
+ * the administrator-configured tx_extra_headroom, folded in before the
+ * alignment so that the link-layer header placement does not change.
+ *
+ * Only use this for new allocations. Code that checks whether an existing
+ * packet has enough headroom must keep using LL_RESERVED_SPACE(), so that
+ * the extra reservation can never force a reallocation by itself.
+ */
+#define LL_ALLOC_SPACE_EX(dev, hlen) \
+	LL_RESERVED_SPACE_EX(dev, (hlen) + READ_ONCE((dev)->tx_extra_headroom))
+#define LL_ALLOC_SPACE(dev) \
+	LL_ALLOC_SPACE_EX(dev, (dev)->hard_header_len)
+
+/* Upper bound for tx_extra_headroom, in bytes. */
+#define NETDEV_TX_EXTRA_HEADROOM_MAX	512
+
 struct header_ops {
 	int	(*create) (struct sk_buff *skb, struct net_device *dev,
 			   unsigned short type, const void *daddr,
@@ -1925,6 +1943,10 @@ enum netdev_reg_state {
  *
  *	@needed_headroom: Extra headroom the hardware may need, but not in all
  *			  cases can this be guaranteed
+ *	@tx_extra_headroom: Additional headroom reserved when allocating
+ *			    locally generated packets for transmission,
+ *			    configured by the administrator. Not a driver
+ *			    requirement; see LL_ALLOC_SPACE()
  *	@needed_tailroom: Extra tailroom the hardware may need, but not in all
  *			  cases can this be guaranteed. Some cases also use
  *			  LL_MAX_HEADER instead to allocate the skb
@@ -2187,6 +2209,7 @@ struct net_device {
 	 */
 	unsigned int		mtu;
 	unsigned short		needed_headroom;
+	unsigned short		tx_extra_headroom;
 	struct netdev_tc_txq	tc_to_txq[TC_MAX_QUEUE];
 #ifdef CONFIG_XPS
 	struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX];
@@ -2758,6 +2781,16 @@ struct netdev_queue *netdev_core_pick_tx(struct net_device *dev,
 					 struct sk_buff *skb,
 					 struct net_device *sb_dev);
 
+/* Same as LL_ALLOC_SPACE() without link-layer header and alignment, for
+ * callers that size the headroom themselves.
+ */
+static inline unsigned int
+netdev_tx_alloc_headroom(const struct net_device *dev)
+{
+	return READ_ONCE(dev->needed_headroom) +
+	       READ_ONCE(dev->tx_extra_headroom);
+}
+
 /* returns the headroom that the master device needs to take in account
  * when forwarding to this dev
  */
diff --git a/net/core/dev.c b/net/core/dev.c
index 8eb9bf1ff..a79f55b85 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -13501,6 +13501,8 @@ static void __init net_dev_struct_check(void)
 	CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, num_tc);
 	CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, mtu);
 	CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, needed_headroom);
+	CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx,
+				      tx_extra_headroom);
 	CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tc_to_txq);
 #ifdef CONFIG_XPS
 	CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, xps_maps);
-- 
2.50.1

[RFC net-next 2/6] net: reserve extra TX headroom for local egress

[David Keogh]

Use LL_ALLOC_SPACE() instead of LL_RESERVED_SPACE() where the stack
allocates packets on behalf of a local sender and sizes them for the
output device:

 - __ip_append_data() and __ip6_append_data(), covering UDP, ICMP,
   ping and raw sockets without IP_HDRINCL;
 - raw_send_hdrinc() and rawv6_send_hdrinc();
 - the three AF_PACKET transmit paths;
 - AF_XDP copy mode, which open-codes the same calculation from
   needed_headroom.

These are the main paths where a packet's headroom is decided by the
device's advertised requirement and nothing else. Adding the
administrator's extra reservation here means such packets reach the TC
egress hook with room for additional headers already in place.

Paths that check an existing packet against LL_RESERVED_SPACE() and
expand it on demand, such as ip_finish_output2() and the tunnel
transmit functions, are deliberately left alone. Using the larger value
there would force a reallocation of every packet that lacks the full
extra reservation, including TCP segments that already have enough
room for the headers an egress program actually pushes.

Signed-off-by: David Keogh <dkeogh@amazon.com>
---
 net/ipv4/ip_output.c   | 2 +-
 net/ipv4/raw.c         | 2 +-
 net/ipv6/ip6_output.c  | 2 +-
 net/ipv6/raw.c         | 2 +-
 net/packet/af_packet.c | 6 +++---
 net/xdp/xsk.c          | 6 ++++--
 6 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 3396f177a..bf718b8f6 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -988,7 +988,7 @@ static int __ip_append_data(struct sock *sk,
 	mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize;
 	paged = !!cork->gso_size;
 
-	hh_len = LL_RESERVED_SPACE(rt->dst.dev);
+	hh_len = LL_ALLOC_SPACE(rt->dst.dev);
 
 	fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0);
 	maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen;
diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c
index 9d01006d8..a1e72e634 100644
--- a/net/ipv4/raw.c
+++ b/net/ipv4/raw.c
@@ -355,7 +355,7 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4,
 	if (flags&MSG_PROBE)
 		goto out;
 
-	hlen = LL_RESERVED_SPACE(rt->dst.dev);
+	hlen = LL_ALLOC_SPACE(rt->dst.dev);
 	tlen = rt->dst.dev->needed_tailroom;
 	skb = sock_alloc_send_skb(sk,
 				  length + hlen + tlen + 15,
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 25fe0ba98..a3626d5f0 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1490,7 +1490,7 @@ static int __ip6_append_data(struct sock *sk,
 	mtu = cork->gso_size ? IP6_MAX_MTU : cork->fragsize;
 	orig_mtu = mtu;
 
-	hh_len = LL_RESERVED_SPACE(rt->dst.dev);
+	hh_len = LL_ALLOC_SPACE(rt->dst.dev);
 
 	fragheaderlen = sizeof(struct ipv6hdr) + rt->rt6i_nfheader_len +
 			(opt ? opt->opt_nflen : 0);
diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c
index b965258cf..7f2da5581 100644
--- a/net/ipv6/raw.c
+++ b/net/ipv6/raw.c
@@ -601,7 +601,7 @@ static int rawv6_send_hdrinc(struct sock *sk, struct msghdr *msg, int length,
 	struct sk_buff *skb;
 	int err;
 	struct rt6_info *rt = dst_rt6_info(*dstp);
-	int hlen = LL_RESERVED_SPACE(rt->dst.dev);
+	int hlen = LL_ALLOC_SPACE(rt->dst.dev);
 	int tlen = rt->dst.dev->needed_tailroom;
 
 	if (length > rt->dst.dev->mtu) {
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 823e0911c..8790fd038 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -2031,7 +2031,7 @@ static int packet_sendmsg_spkt(struct socket *sock, struct msghdr *msg,
 		goto out_unlock;
 
 	if (!skb) {
-		size_t reserved = LL_RESERVED_SPACE_EX(dev, hard_header_len);
+		size_t reserved = LL_ALLOC_SPACE_EX(dev, hard_header_len);
 		int tlen = dev->needed_tailroom;
 		unsigned int hhlen = dev->header_ops ? hard_header_len : 0;
 
@@ -2863,7 +2863,7 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)
 			goto tpacket_error;
 
 		status = TP_STATUS_SEND_REQUEST;
-		hlen = LL_RESERVED_SPACE_EX(dev, hard_header_len);
+		hlen = LL_ALLOC_SPACE_EX(dev, hard_header_len);
 		tlen = dev->needed_tailroom;
 		if (vnet_hdr_sz) {
 			data += vnet_hdr_sz;
@@ -3090,7 +3090,7 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len)
 		goto out_unlock;
 
 	err = -ENOBUFS;
-	hlen = LL_RESERVED_SPACE_EX(dev, hard_header_len);
+	hlen = LL_ALLOC_SPACE_EX(dev, hard_header_len);
 	tlen = dev->needed_tailroom;
 	linear = __virtio16_to_cpu(vio_le(), vnet_hdr.hdr_len);
 	linear = max(linear, min_t(int, len, hard_header_len));
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index 33475b180..58d64025d 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -1045,7 +1045,8 @@ static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs,
 	buffer = xsk_buff_raw_get_data(pool, addr);
 
 	if (!skb) {
-		hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(xs->dev->needed_headroom));
+		hr = max(NET_SKB_PAD,
+			 L1_CACHE_ALIGN(netdev_tx_alloc_headroom(xs->dev)));
 
 		skb = sock_alloc_send_skb(&xs->sk, hr, 1, &err);
 		if (unlikely(!skb))
@@ -1128,7 +1129,8 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 		len = desc->len;
 
 		if (!skb) {
-			hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(dev->needed_headroom));
+			hr = max(NET_SKB_PAD,
+				 L1_CACHE_ALIGN(netdev_tx_alloc_headroom(dev)));
 			tr = dev->needed_tailroom;
 			skb = sock_alloc_send_skb(&xs->sk, hr + len + tr, 1, &err);
 			if (unlikely(!skb))
-- 
2.50.1

[RFC net-next 3/6] rtnetlink: add IFLA_TX_EXTRA_HEADROOM

[David Keogh]

Expose tx_extra_headroom through a new IFLA_TX_EXTRA_HEADROOM u16
attribute. The value is bounded to NETDEV_TX_EXTRA_HEADROOM_MAX
(512 bytes) by the netlink policy, which is plenty for several layers
of encapsulation while keeping the growth of per-packet allocations
modest.

The attribute can be given when creating a link and when changing one.
It is included in link dumps and a change triggers the usual link
notification. Zero is the default and resets the reservation.

IFLA_HEADROOM keeps reporting needed_headroom, the driver's own
requirement, so userspace can tell the two apart.

The setting applies to the device the packet is allocated for, that is
the device selected by the route, and is not inherited by stacked
devices. Document that together with the allocation-only semantics in
netdevices.rst and add the attribute to the rt-link spec.

Signed-off-by: David Keogh <dkeogh@amazon.com>
---
 Documentation/netlink/specs/rt-link.yaml | 12 +++++++++
 Documentation/networking/netdevices.rst  | 34 ++++++++++++++++++++++++
 include/uapi/linux/if_link.h             |  1 +
 net/core/rtnetlink.c                     | 19 ++++++++++++-
 4 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/Documentation/netlink/specs/rt-link.yaml b/Documentation/netlink/specs/rt-link.yaml
index 81b586339..9da363df5 100644
--- a/Documentation/netlink/specs/rt-link.yaml
+++ b/Documentation/netlink/specs/rt-link.yaml
@@ -1098,6 +1098,16 @@ attribute-sets:
         name: pacing-offload
         type: u32
         doc: Enable EDT pacing offload (0 - disabled, 1 - enabled).
+      -
+        name: tx-extra-headroom
+        type: u16
+        doc: |
+          Additional headroom, in bytes, reserved on top of the driver's
+          requirement when allocating locally generated packets for
+          transmission through this device. Zero (the default) disables
+          the extra reservation.
+        checks:
+          max: 512
   -
     name: prop-list-link-attrs
     subset-of: link-attrs
@@ -2471,6 +2481,7 @@ operations:
             - gso-ipv4-max-size
             - gro-ipv4-max-size
             - af-spec
+            - tx-extra-headroom
     -
       name: newlink-ntf
       doc: Notify that a link has been created
@@ -2567,6 +2578,7 @@ operations:
             - gso-ipv4-max-size
             - gro-ipv4-max-size
             - pacing-offload
+            - tx-extra-headroom
       dump:
         request:
           value: 18
diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst
index 6a87f09e7..02d6b4d4b 100644
--- a/Documentation/networking/netdevices.rst
+++ b/Documentation/networking/netdevices.rst
@@ -441,6 +441,40 @@ from a notifier.
 
 Drivers **must not** generate nested notifications of the ops-locked types.
 
+TX headroom reservation
+=======================
+
+``needed_headroom`` is the driver's own additional header-space requirement.
+It is written by the driver, or by a stacked device inheriting the value of
+the device below it, and is reported read-only through ``IFLA_HEADROOM``.
+
+``tx_extra_headroom`` is a separate reservation configured by the
+administrator through the ``IFLA_TX_EXTRA_HEADROOM`` rtnetlink attribute, a
+byte count from 0 to 512 that defaults to 0. It can be supplied when a link
+is created or changed, is reported in link dumps and change notifications,
+and setting it back to 0 removes it.
+
+The extra reservation is only applied when the stack allocates a locally
+generated packet for transmission through the device. ``LL_ALLOC_SPACE()``
+is ``LL_RESERVED_SPACE()`` plus ``tx_extra_headroom`` and is used by the
+UDP, raw, AF_PACKET and AF_XDP transmit paths. Its purpose is to give
+programs attached at TC egress room to push additional headers without
+having to reallocate the packet first.
+
+The setting is deliberately not part of ``LL_RESERVED_SPACE()``, which
+remains the threshold for deciding whether an already existing packet must
+be expanded. Packets that reach the device with less than the extra
+reservation, such as forwarded packets or TCP segments, are therefore not
+reallocated because of it. Consumers must keep their usual headroom and
+clone checks.
+
+The setting is not inherited through stacked devices. Packets are allocated
+against the device selected by the route, so configure it on the VLAN,
+bond or bridge device the route points at rather than on the device
+beneath it. It survives link down/up and network namespace moves and is
+independent of any TC or BPF attachment. It does not change the MTU or
+the bytes on the wire.
+
 NETDEV_INTERNAL symbol namespace
 ================================
 
diff --git a/include/uapi/linux/if_link.h b/include/uapi/linux/if_link.h
index 245b36204..fb3a6b382 100644
--- a/include/uapi/linux/if_link.h
+++ b/include/uapi/linux/if_link.h
@@ -382,6 +382,7 @@ enum {
 	IFLA_HEADROOM,
 	IFLA_TAILROOM,
 	IFLA_PACING_OFFLOAD,
+	IFLA_TX_EXTRA_HEADROOM,
 	__IFLA_MAX
 };
 
diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index 15bffafbf..fb691db8e 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -1403,6 +1403,7 @@ static noinline size_t if_nlmsg_size(const struct net_device *dev,
 	       + nla_total_size(4)  /* IFLA_PACING_OFFLOAD */
 	       + nla_total_size(2)  /* IFLA_HEADROOM */
 	       + nla_total_size(2)  /* IFLA_TAILROOM */
+	       + nla_total_size(2)  /* IFLA_TX_EXTRA_HEADROOM */
 	       + rtnl_dev_parent_size(dev)
 	       + 0;
 
@@ -2200,7 +2201,9 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb,
 	    nla_put_u16(skb, IFLA_HEADROOM,
 			READ_ONCE(dev->needed_headroom)) ||
 	    nla_put_u16(skb, IFLA_TAILROOM,
-			READ_ONCE(dev->needed_tailroom)))
+			READ_ONCE(dev->needed_tailroom)) ||
+	    nla_put_u16(skb, IFLA_TX_EXTRA_HEADROOM,
+			READ_ONCE(dev->tx_extra_headroom)))
 		goto nla_put_failure;
 
 	if (rtnl_fill_proto_down(skb, dev))
@@ -2361,6 +2364,8 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
 	[IFLA_HEADROOM]		= { .type = NLA_REJECT },
 	[IFLA_TAILROOM]		= { .type = NLA_REJECT },
 	[IFLA_PACING_OFFLOAD]	= NLA_POLICY_MAX(NLA_U32, 1),
+	[IFLA_TX_EXTRA_HEADROOM] =
+		NLA_POLICY_MAX(NLA_U16, NETDEV_TX_EXTRA_HEADROOM_MAX),
 };
 
 static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
@@ -3368,6 +3373,15 @@ static int do_setlink(const struct sk_buff *skb, struct net_device *dev,
 		}
 	}
 
+	if (tb[IFLA_TX_EXTRA_HEADROOM]) {
+		u16 val = nla_get_u16(tb[IFLA_TX_EXTRA_HEADROOM]);
+
+		if (dev->tx_extra_headroom != val) {
+			WRITE_ONCE(dev->tx_extra_headroom, val);
+			status |= DO_SETLINK_NOTIFY;
+		}
+	}
+
 	if (tb[IFLA_OPERSTATE])
 		set_operstate(dev, nla_get_u8(tb[IFLA_OPERSTATE]));
 
@@ -3859,6 +3873,9 @@ struct net_device *rtnl_create_link(struct net *net, const char *ifname,
 		netif_set_gso_ipv4_max_size(dev, nla_get_u32(tb[IFLA_GSO_IPV4_MAX_SIZE]));
 	if (tb[IFLA_GRO_IPV4_MAX_SIZE])
 		netif_set_gro_ipv4_max_size(dev, nla_get_u32(tb[IFLA_GRO_IPV4_MAX_SIZE]));
+	if (tb[IFLA_TX_EXTRA_HEADROOM])
+		dev->tx_extra_headroom =
+			nla_get_u16(tb[IFLA_TX_EXTRA_HEADROOM]);
 
 	return dev;
 }
-- 
2.50.1

[RFC net-next 4/6] selftests: net: add tx_extra_headroom test

[David Keogh]

Exercise the IFLA_TX_EXTRA_HEADROOM attribute through YNL: the default
of zero, setting values up to the maximum and back to zero, rejection
of a value above the maximum, supplying the attribute at link creation,
the change notification, persistence across link down/up and across a
network namespace move, and that a VLAN created on top of a configured
device does not inherit the setting.

Signed-off-by: David Keogh <dkeogh@amazon.com>
---
 tools/testing/selftests/net/Makefile          |   1 +
 .../selftests/net/tx_extra_headroom.py        | 137 ++++++++++++++++++
 2 files changed, 138 insertions(+)
 create mode 100755 tools/testing/selftests/net/tx_extra_headroom.py

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 54beea2e3..0eb9a69e9 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -120,6 +120,7 @@ TEST_PROGS := \
 	test_vxlan_vnifiltering.sh \
 	tfo_passive.sh \
 	traceroute.sh \
+	tx_extra_headroom.py \
 	txtimestamp.sh \
 	udpgro.sh \
 	udpgro_bench.sh \
diff --git a/tools/testing/selftests/net/tx_extra_headroom.py b/tools/testing/selftests/net/tx_extra_headroom.py
new file mode 100755
index 000000000..f6f087e98
--- /dev/null
+++ b/tools/testing/selftests/net/tx_extra_headroom.py
@@ -0,0 +1,137 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Tests for the IFLA_TX_EXTRA_HEADROOM link attribute."""
+
+from lib.py import ksft_run, ksft_exit, ksft_eq, ksft_raises, ksft_true
+from lib.py import ip
+from lib.py import NetNS, NetNSEnter
+from lib.py import Netlink, NlError, RtnlFamily
+
+ATTR = "tx-extra-headroom"
+MAX = 512
+
+
+def ifindex(name):
+    return ip(f"link show dev {name}", json=True)[0]["ifindex"]
+
+
+def get(rtnl, idx):
+    return rtnl.getlink({"ifi-index": idx})[ATTR]
+
+
+def set_(rtnl, idx, val):
+    rtnl.setlink({"ifi-index": idx, ATTR: val})
+
+
+def test_default_and_set() -> None:
+    """Default is 0, values up to the maximum are accepted, 0 resets."""
+    with NetNS() as ns, NetNSEnter(str(ns)):
+        ip("link add d0 type dummy")
+        rtnl = RtnlFamily()
+        idx = ifindex("d0")
+
+        ksft_eq(get(rtnl, idx), 0)
+        for val in (1, 256, MAX):
+            set_(rtnl, idx, val)
+            ksft_eq(get(rtnl, idx), val)
+        set_(rtnl, idx, 0)
+        ksft_eq(get(rtnl, idx), 0)
+
+
+def test_reject_above_max() -> None:
+    """Values above the maximum are rejected by the policy."""
+    with NetNS() as ns, NetNSEnter(str(ns)):
+        ip("link add d0 type dummy")
+        rtnl = RtnlFamily()
+        idx = ifindex("d0")
+
+        set_(rtnl, idx, 128)
+        with ksft_raises(NlError):
+            set_(rtnl, idx, MAX + 1)
+        ksft_eq(get(rtnl, idx), 128)
+
+
+def test_newlink() -> None:
+    """The attribute can be supplied when the link is created."""
+    with NetNS() as ns, NetNSEnter(str(ns)):
+        rtnl = RtnlFamily()
+        rtnl.newlink({"ifname": "d0", "linkinfo": {"kind": "dummy"},
+                      ATTR: 64},
+                     flags=[Netlink.NLM_F_CREATE, Netlink.NLM_F_EXCL])
+        ksft_eq(get(rtnl, ifindex("d0")), 64)
+
+
+def test_notification() -> None:
+    """A change produces a link notification carrying the new value."""
+    with NetNS() as ns, NetNSEnter(str(ns)):
+        ip("link add d0 type dummy")
+        ip("link set dev d0 up")
+        rtnl = RtnlFamily()
+        idx = ifindex("d0")
+
+        rtnl.ntf_subscribe("rtnlgrp-link")
+        set_(rtnl, idx, 192)
+
+        seen = None
+        for msg in rtnl.poll_ntf(duration=1):
+            if msg["name"] == "getlink" and msg["msg"]["ifi-index"] == idx:
+                seen = msg["msg"][ATTR]
+                break
+        ksft_eq(seen, 192)
+
+
+def test_survives_down_up() -> None:
+    """Toggling the link does not reset the value."""
+    with NetNS() as ns, NetNSEnter(str(ns)):
+        ip("link add d0 type dummy")
+        rtnl = RtnlFamily()
+        idx = ifindex("d0")
+
+        set_(rtnl, idx, 96)
+        ip("link set dev d0 up")
+        ksft_eq(get(rtnl, idx), 96)
+        ip("link set dev d0 down")
+        ksft_eq(get(rtnl, idx), 96)
+
+
+def test_survives_netns_move() -> None:
+    """Moving the link to another namespace keeps the value."""
+    with NetNS() as ns1, NetNS() as ns2:
+        with NetNSEnter(str(ns1)):
+            ip("link add d0 type dummy")
+            rtnl = RtnlFamily()
+            set_(rtnl, ifindex("d0"), 160)
+            ip(f"link set dev d0 netns {ns2}")
+
+        with NetNSEnter(str(ns2)):
+            rtnl = RtnlFamily()
+            ksft_eq(get(rtnl, ifindex("d0")), 160)
+
+
+def test_not_inherited() -> None:
+    """Stacked devices do not inherit the value of the device below."""
+    with NetNS() as ns, NetNSEnter(str(ns)):
+        ip("link add d0 type dummy")
+        rtnl = RtnlFamily()
+        set_(rtnl, ifindex("d0"), 224)
+
+        ip("link add link d0 name d0.10 type vlan id 10")
+        ksft_eq(get(rtnl, ifindex("d0.10")), 0)
+        ksft_true(rtnl.getlink({"ifi-index": ifindex("d0.10")})["headroom"]
+                  >= 4, "vlan needed_headroom")
+
+
+def main() -> None:
+    ksft_run([test_default_and_set,
+              test_reject_above_max,
+              test_newlink,
+              test_notification,
+              test_survives_down_up,
+              test_survives_netns_move,
+              test_not_inherited])
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
-- 
2.50.1

[RFC net-next 5/6] selftests/bpf: add tx_extra_headroom data-plane programs

[David Keogh]

Add two BPF programs for a tx_extra_headroom data-plane test:

  * push_outer (SEC "tc") pushes a 20-byte outer header on UDP egress,
    forcing a head reallocation when the skb lacks the headroom.

  * count_realloc (SEC "fentry/pskb_expand_head") counts those reallocs
    for one ifindex, selected via the cfg_ifindex map.

The test that loads them is added to selftests/net in the next patch.

Signed-off-by: David Keogh <dkeogh@amazon.com>
---
 .../bpf/progs/tx_extra_headroom_probe.c       | 68 ++++++++++++++++++
 .../bpf/progs/tx_extra_headroom_tc.bpf.c      | 69 +++++++++++++++++++
 2 files changed, 137 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/tx_extra_headroom_probe.c
 create mode 100644 tools/testing/selftests/bpf/progs/tx_extra_headroom_tc.bpf.c

diff --git a/tools/testing/selftests/bpf/progs/tx_extra_headroom_probe.c b/tools/testing/selftests/bpf/progs/tx_extra_headroom_probe.c
new file mode 100644
index 000000000..a6ca66ef0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/tx_extra_headroom_probe.c
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * fentry realloc counter for the tx_extra_headroom data-plane selftest.
+ *
+ * count_realloc (fentry/pskb_expand_head) increments realloc_cnt[0] each time
+ * pskb_expand_head() runs for an skb whose egress device ifindex matches
+ * cfg_ifindex[0]. The ifindex filter scopes the count to the test interface so
+ * unrelated background reallocs are not counted. fentry is BTF/CO-RE based, so
+ * this attaches by its SEC name via "bpftool prog loadall ... autoattach" with
+ * no raw kprobe name, no bpftrace, and no user-space loader.
+ *
+ * The companion tc program (tx_extra_headroom_tc.bpf.c) is what forces the
+ * reallocation this program observes. Userspace writes cfg_ifindex[0] (the
+ * device under test) and resets realloc_cnt[0] before each run via bpftool,
+ * then reads realloc_cnt[0] after.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include <bpf/bpf_tracing.h>
+
+/* realloc_cnt[0] = number of pskb_expand_head calls on the test interface. */
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u64);
+} realloc_cnt SEC(".maps");
+
+/* cfg_ifindex[0] = ifindex of the device under test (set by userspace). */
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u32);
+} cfg_ifindex SEC(".maps");
+
+/*
+ * BPF_PROG unpacks the trampoline args, so skb is the first argument of
+ * pskb_expand_head(struct sk_buff *, int, int, gfp_t). Count only reallocs
+ * whose egress device matches the configured ifindex.
+ */
+SEC("fentry/pskb_expand_head")
+int BPF_PROG(count_realloc, struct sk_buff *skb)
+{
+	__u32 key = 0;
+	__u32 *want;
+	__u64 *cnt;
+	__u32 ifi;
+
+	want = bpf_map_lookup_elem(&cfg_ifindex, &key);
+	if (!want || *want == 0)
+		return 0;
+
+	/* CO-RE read: skb->dev may be NULL on some paths. */
+	if (!BPF_CORE_READ(skb, dev))
+		return 0;
+	ifi = BPF_CORE_READ(skb, dev, ifindex);
+	if (ifi != *want)
+		return 0;
+
+	cnt = bpf_map_lookup_elem(&realloc_cnt, &key);
+	if (cnt)
+		__sync_fetch_and_add(cnt, 1);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/tx_extra_headroom_tc.bpf.c b/tools/testing/selftests/bpf/progs/tx_extra_headroom_tc.bpf.c
new file mode 100644
index 000000000..43e3bea60
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/tx_extra_headroom_tc.bpf.c
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * tc-egress pusher for the tx_extra_headroom data-plane selftest.
+ *
+ * On egress, pushes OUTER_LEN bytes of outer header via
+ * bpf_skb_adjust_room(..., BPF_ADJ_ROOM_MAC, ...). On an skb allocated with only
+ * LL_RESERVED_SPACE() of headroom this forces skb_cow_head() ->
+ * pskb_expand_head() (a realloc-and-copy) on every packet; when the device has
+ * reserved extra headroom via tx_extra_headroom the push finds room already
+ * present and no realloc happens. The reallocation is counted by the companion
+ * fentry program in tx_extra_headroom_probe.bpf.c.
+ *
+ * UDP only, so TCP control traffic and IPv6 NDP are not corrupted. OUTER_LEN
+ * defaults to 20 (one IPv4 header) and can be overridden at build time.
+ */
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/in.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/pkt_cls.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_endian.h>
+
+#ifndef OUTER_LEN
+#define OUTER_LEN 20
+#endif
+
+SEC("tc")
+int push_outer(struct __sk_buff *skb)
+{
+	void *data = (void *)(long)skb->data;
+	void *data_end = (void *)(long)skb->data_end;
+	struct ethhdr *eth = data;
+	__u8 l4 = 0;
+
+	if ((void *)(eth + 1) > data_end)
+		return TC_ACT_OK;
+
+	if (eth->h_proto == bpf_htons(ETH_P_IP)) {
+		struct iphdr *ip = (void *)(eth + 1);
+
+		if ((void *)(ip + 1) > data_end)
+			return TC_ACT_OK;
+		l4 = ip->protocol;
+	} else if (eth->h_proto == bpf_htons(ETH_P_IPV6)) {
+		struct ipv6hdr *ip6 = (void *)(eth + 1);
+
+		if ((void *)(ip6 + 1) > data_end)
+			return TC_ACT_OK;
+		l4 = ip6->nexthdr;
+	} else {
+		return TC_ACT_OK;
+	}
+
+	/* UDP only: do not corrupt TCP or ICMPv6/NDP control traffic. */
+	if (l4 != IPPROTO_UDP)
+		return TC_ACT_OK;
+
+	/*
+	 * Grow at the MAC layer by OUTER_LEN. The helper's internal
+	 * skb_cow_head() reallocates when headroom is insufficient; that
+	 * realloc is exactly what tx_extra_headroom is meant to avoid.
+	 */
+	bpf_skb_adjust_room(skb, OUTER_LEN, BPF_ADJ_ROOM_MAC, 0);
+	return TC_ACT_OK;
+}
+
+char _license[] SEC("license") = "GPL";
-- 
2.50.1

[RFC net-next 6/6] selftests/net: add tx_extra_headroom data-plane test

[David Keogh]

The existing tx_extra_headroom test checks the IFLA_TX_EXTRA_HEADROOM
attribute itself. Add a test for its effect: with the knob at zero an
egress header-push reallocates the skb head on every packet, and with
the knob raised the realloc is eliminated. A second case checks that the
cliff sits at exactly the headroom the push needs.

The test loads the two BPF programs from the previous patch, attaching
count_realloc by SEC name with "bpftool prog loadall ... autoattach" and
reading realloc_cnt via bpf_map_dump(). The veth peer is kept in its own
netns so the datagram egresses the device under test, and a permanent
neighbor is primed so UDP is not dropped before egress.

Signed-off-by: David Keogh <dkeogh@amazon.com>
---
 tools/testing/selftests/net/Makefile          |   1 +
 .../net/tx_extra_headroom_dataplane.py        | 197 ++++++++++++++++++
 2 files changed, 198 insertions(+)
 create mode 100755 tools/testing/selftests/net/tx_extra_headroom_dataplane.py

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 0eb9a69e9..28b0444ba 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -121,6 +121,7 @@ TEST_PROGS := \
 	tfo_passive.sh \
 	traceroute.sh \
 	tx_extra_headroom.py \
+	tx_extra_headroom_dataplane.py \
 	txtimestamp.sh \
 	udpgro.sh \
 	udpgro_bench.sh \
diff --git a/tools/testing/selftests/net/tx_extra_headroom_dataplane.py b/tools/testing/selftests/net/tx_extra_headroom_dataplane.py
new file mode 100755
index 000000000..d32bb1ed6
--- /dev/null
+++ b/tools/testing/selftests/net/tx_extra_headroom_dataplane.py
@@ -0,0 +1,197 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Data-plane tests for IFLA_TX_EXTRA_HEADROOM: setting the knob eliminates the
+per-packet head reallocation that an egress header-push otherwise forces.
+
+Mechanism (fully in-tree, no bpftrace, no raw kprobe, no user-space loader):
+
+  * tx_extra_headroom_tc.bpf.o    - a tc-egress program that pushes a 20-byte
+    outer header on UDP via bpf_skb_adjust_room(..., BPF_ADJ_ROOM_MAC, ...). On
+    an skb allocated with only LL_RESERVED_SPACE() of headroom that push
+    reallocates (pskb_expand_head) on every packet; with tx_extra_headroom
+    reserving enough room at allocation time it does not.
+
+  * tx_extra_headroom_probe.bpf.o - an fentry/pskb_expand_head program that
+    counts the reallocs for the device under test (filtered by cfg_ifindex) into
+    realloc_cnt. It is attached by its SEC name via
+    "bpftool prog loadall ... autoattach" and read via bpf_map_dump().
+
+Topology is two netns (peer in a separate netns) so the datagram genuinely
+egresses the device and hits the tc-egress hook; co-resident veth ends would be
+locally delivered via loopback and never reach egress. A static ARP neighbor is
+primed so UDP egresses instead of dropping at an unresolved neighbor.
+"""
+
+import socket
+import time
+
+from lib.py import ksft_run, ksft_exit, ksft_ge, ksft_lt
+from lib.py import ip, tc, bpftool, cmd
+from lib.py import bpf_map_dump, bpf_prog_map_ids
+from lib.py import NetNS, NetNSEnter
+from lib.py import RtnlFamily
+
+ATTR = "tx-extra-headroom"
+OUTER_LEN = 20
+# veth LL_RESERVED_SPACE = ((14 + 0) & ~15) + 16 = 16; the 14-byte L2 header
+# leaves 2 free, so a 20-byte push needs 20 - 2 = 18 more from the knob.
+THRESHOLD = 18
+NPKTS = 20000
+
+TC_OBJ = "tx_extra_headroom_tc.bpf.o"
+PROBE_OBJ = "tx_extra_headroom_probe.bpf.o"
+PINDIR = "/sys/fs/bpf/tx_extra_headroom"
+
+
+def ifindex(name):
+    return ip(f"link show dev {name}", json=True)[0]["ifindex"]
+
+
+def _hex_le(value, nbytes):
+    """bpftool 'key/value hex' wants space-separated little-endian bytes."""
+    return " ".join(f"{b:02x}" for b in int(value).to_bytes(nbytes, "little"))
+
+
+def _map_set_u32(map_id, key, value):
+    bpftool(f"map update id {map_id} key hex {_hex_le(key, 4)} "
+            f"value hex {_hex_le(value, 4)}")
+
+
+def _map_set_u64(map_id, key, value):
+    bpftool(f"map update id {map_id} key hex {_hex_le(key, 4)} "
+            f"value hex {_hex_le(value, 8)}")
+
+
+def set_hr(rtnl, idx, val):
+    rtnl.setlink({"ifi-index": idx, ATTR: val})
+
+
+class Observer:
+    """Loads the fentry realloc counter (autoattach) and the tc pusher, exposes
+    a reallocs(send_fn) call, and tears both down cleanly on close()."""
+
+    def __init__(self, dev, dev_ifindex, send_ns):
+        self.dev = dev
+        self.send_ns = send_ns
+        # Fresh pin dir; a leftover one would make loadall fail to pin.
+        cmd(f"rm -f {PINDIR}/*", shell=True, fail=False)
+        cmd(f"rmdir {PINDIR}", shell=True, fail=False)
+        # Single fentry program -> autoattach attaches it by SEC name and pins
+        # its maps under PINDIR. No bpftrace, no kprobe name.
+        bpftool(f"prog loadall {PROBE_OBJ} {PINDIR} autoattach")
+        # tc pusher: self-loading via object-file, attached inside the send ns.
+        tc(f"qdisc add dev {self.dev} clsact", ns=send_ns)
+        tc(f"filter add dev {self.dev} egress bpf da object-file {TC_OBJ} sec tc",
+           ns=send_ns)
+        # Resolve the two maps by name from the fentry program.
+        prog_id = self._probe_prog_id()
+        maps = bpf_prog_map_ids(prog_id)
+        self.cfg_id = maps["cfg_ifindex"]
+        self.cnt_id = maps["realloc_cnt"]
+        # Scope the counter to the device under test (by id: a stale pinned copy
+        # of the same-named map would make a by-name update ambiguous).
+        _map_set_u32(self.cfg_id, 0, dev_ifindex)
+
+    @staticmethod
+    def _probe_prog_id():
+        progs = bpftool("prog show", json=True)
+        for p in progs:
+            if p.get("name") == "count_realloc":
+                return p["id"]
+        raise RuntimeError("count_realloc fentry program not loaded")
+
+    def reallocs(self, send_fn):
+        """Reset the counter, run send_fn (traffic), return the realloc count."""
+        _map_set_u64(self.cnt_id, 0, 0)
+        send_fn()
+        time.sleep(0.2)  # let the last in-flight reallocs land
+        return bpf_map_dump(self.cnt_id)[0]
+
+    def close(self):
+        # Detach the fentry link so it does not leak to later tests.
+        links = bpftool("link show", json=True)
+        for l in links:
+            if l.get("attach_type") == "trace_fentry":
+                cmd(f"bpftool link detach id {l['id']}", fail=False)
+        cmd(f"rm -f {PINDIR}/*", shell=True, fail=False)
+        cmd(f"rmdir {PINDIR}", shell=True, fail=False)
+
+
+def _blast_udp(dst, port, n):
+    """Send n small UDP datagrams; the receiver drops them, but the TX-side
+    realloc already happened, which is what we measure."""
+    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    s.connect((dst, port))
+    buf = b"x" * 1024
+    for _ in range(n):
+        try:
+            s.send(buf)
+        except OSError:
+            pass
+    s.close()
+
+
+def _setup_veth(peer_ns):
+    """hrt0 in the current netns (send side, tc push attached); hrt1 in peer_ns.
+    Prime a static ARP neighbor so UDP egresses instead of dropping at an
+    unresolved neighbor (which would read 0 reallocs regardless of the knob)."""
+    ip("link add hrt0 type veth peer name hrt1")
+    ip(f"link set hrt1 netns {peer_ns}")
+    ip("addr add 10.0.0.1/24 dev hrt0")
+    ip("link set hrt0 up")
+    ip("link set lo up")
+    with NetNSEnter(str(peer_ns)):
+        ip("addr add 10.0.0.2/24 dev hrt1")
+        ip("link set hrt1 up")
+        ip("link set lo up")
+        peer_mac = ip("link show dev hrt1", json=True)[0]["address"]
+    ip(f"neigh replace 10.0.0.2 lladdr {peer_mac} dev hrt0 nud permanent")
+
+
+def test_realloc_eliminated() -> None:
+    """Knob off -> push reallocates per packet; knob set -> realloc eliminated."""
+    with NetNS() as send_ns, NetNS() as peer_ns, NetNSEnter(str(send_ns)):
+        _setup_veth(peer_ns)
+        rtnl = RtnlFamily()
+        idx = ifindex("hrt0")
+        obs = Observer("hrt0", idx, str(send_ns))
+        try:
+            off = obs.reallocs(lambda: _blast_udp("10.0.0.2", 9999, NPKTS))
+            set_hr(rtnl, idx, 128)
+            on = obs.reallocs(lambda: _blast_udp("10.0.0.2", 9999, NPKTS))
+        finally:
+            obs.close()
+
+        ksft_ge(off, NPKTS // 2, "knob off: push should realloc most packets")
+        ksft_lt(on, NPKTS // 100, "knob on: realloc should be eliminated")
+
+
+def test_realloc_threshold() -> None:
+    """The realloc cliff sits at exactly the headroom the push needs."""
+    with NetNS() as send_ns, NetNS() as peer_ns, NetNSEnter(str(send_ns)):
+        _setup_veth(peer_ns)
+        rtnl = RtnlFamily()
+        idx = ifindex("hrt0")
+        obs = Observer("hrt0", idx, str(send_ns))
+        try:
+            set_hr(rtnl, idx, THRESHOLD - 1)
+            below = obs.reallocs(lambda: _blast_udp("10.0.0.2", 9999, NPKTS))
+            set_hr(rtnl, idx, THRESHOLD)
+            at = obs.reallocs(lambda: _blast_udp("10.0.0.2", 9999, NPKTS))
+        finally:
+            obs.close()
+
+        ksft_ge(below, NPKTS // 2, f"knob={THRESHOLD-1}: still reallocates")
+        ksft_lt(at, NPKTS // 100, f"knob={THRESHOLD}: realloc eliminated at cliff")
+
+
+def main() -> None:
+    ksft_run([test_realloc_eliminated,
+              test_realloc_threshold])
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
-- 
2.50.1