From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.218.48 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794022477; b=RSsd0z1A8JkSc2osM1CSIdMBaQmzaEq8TT3o9Edu4OOJ+YM5aPigTEpiiF9NUFv+16O/ aaE3xdlTX52Jyvy9M5lgec/4spS3z4XzAngT1LBsmBxR/IDFrovlcTyB+fLdULZXcV7+s xN27hpzj28ONRCNROyJN+KvWItnHXSruWVQdweLmn46gh2B/rGOPWDnR/GrJUlrxRaxVq 46al47uG6or3K5FvyuQuUaG4koPOaHUJLUoFvLWYmoAhjdwdr6Ez8N3wCs+L4ym6hmfHY r37pe48XsyTwCgeh5sZIauG6xLfuTmVq7D+pPyq8XI08livIzRfbSduNiCR3t9UZmuw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794022477; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:MIME-Version: Content-Transfer-Encoding; bh=yAQvBw1PVJuhybp5YVi1fX4+UzEv8rZ9bKF4PeZ2eGs=; b=avkU9EnCOT4B4HqAC4j80ElFFU0uF9BBYEcNZiTRMR9VnL8YmtGJGEM28ZvmNZ/zBXs2 MsmBfHYdy09F0xq1c/lQGdTcH5N7MwlvM73ABSaDE6sqFYdKo8VtzX45fB6/0AWcmoUdz fDgVBsBYI8xkZsI+w0cAiEZz/FqFSvEfeKLVwSFVKvCbjtxyabwpSb8GzDshGl256JzPq 2dOlwP3WuU4U2q/IuP3k3qpYFihAVEDFE+BCwoLiEVoI6aBXTuhwC/faB+tmFQk5WHdMz iP+Ti73JBOt8cQXXGirwUQO3EY8DhNRLOp7enPqHAXCxRy9gWfl5x3ig2834gDuOMew== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=cloudflare.com; spf=pass smtp.mailfrom=cloudflare.com; dkim=pass header.d=cloudflare.com header.i=@cloudflare.com header.a=rsa-sha256 header.s=google09082023 header.b=RqFmdIMN; arc=none smtp.remote-ip=209.85.218.48 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 294F8606A32D for ; Sat, 7 Nov 2026 03:34:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 294F8606A32D Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.a=rsa-sha256 header.s=google09082023 header.b=RqFmdIMN Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-c4d2b741ed9so192715466b.2 for ; Fri, 06 Nov 2026 19:34:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google09082023; t=1794022474; x=1794627274; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=yAQvBw1PVJuhybp5YVi1fX4+UzEv8rZ9bKF4PeZ2eGs=; b=RqFmdIMN+3kYVy1ihur1Qf8pqlW83bSprSSH1rby4KaQANpHcYhhFa0EJzYz8HdhMy PD0KljsfcP0F1HSNlX6RIjdY5+F9ty/NvBQ9WOHxw3R2lrFE4uQdv6ktZ2CPgXVdrr0m 9BldEHPTJCS1/9XQaOJwy9G56uMn145n9SPm2aD1DWoKdtskhSiV3jT6R/4EXM0k69em 7pnInrzeS06i6nFAaxy7ZMDC5tNoCjHnhGXiyMMKJcbsEc96GMrtrVK/pa3f6B6XRPZb XLM1IM1PQPyKxyvs+HR9ynUH4R84q6mwtcki4SeSEtVbmeQzTxTcmSR2GrY7LBsuGY3/ 2b6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794022474; x=1794627274; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=yAQvBw1PVJuhybp5YVi1fX4+UzEv8rZ9bKF4PeZ2eGs=; b=pW1AlpYFXUajOxod79GWbd87Vb+peyiu37ZWonJIqGSAniGQyy7F3dIAcV/voYS0y1 NIS/3XxLUvWQD4N4Qv3T1Jg7iClBb4aFu+HWXDqj9SFxxS4/bvOLjZ1f5OjCP+GvYSLx eToOHzqVSEn1Gpz/oUGKCLEqSHdIsJ08hg/t8otcOk6gFGtAmyeDjF8zpKYw55oEYyyf 9JMNWSKC+0h83u40ejV0zOj0XXd7rR8xBuZKTazO8kbXNcWZpWcsJkZUVSmGrYlldR8B U1s0CM3k+G0JOYZTkdfmgokKYqwsix60LNdxVOJRS9htdkBm7B97GcCWw/K2PjGePwHF eElQ== X-Forwarded-Encrypted: i=1; AKwUvBz4flUrIcq68Vms2hoZxJHVQxRNefSwpbnHrHRTJtW6VtKaxVKNl676FWLEUFWK25Ty73cW1u0BUEOvsRrN77E=@vger.kernel.org X-Gm-Message-State: APuU4Z+egRNwGhYeKa85vxWBapMrmTiQEN/7NZ6RZwQoXozH/159fhV4 OJEJVb+lJ0eI77Q01p4OMn3Tb1weWMGMxbB8xOHn1DXGPM6gPxAoiOSOySTIZpm0o0s= X-Gm-Gg: AYBFou3BhDgm3qUSa0CoLXYUU7XeTsa5EPQ0lS7aTElqrDJ5BjvEGhjlDnLlXzQsqWa yO+zsTXIzCaYWk1/j1Fsrah1II0hTV0tVJrWGQYAcm2SZcRv+KSiOKV9UQtzspUAPkDURDObuHR Pl2TT47sLyTRzidywdgCgs4dIbwuKKWxH0NV70bES38Ffjq3LquIZ87jTgOQZO6NDetU+SBQkqV fljbrbo7ygaXrDM8G7PYcKLeAqYX0788p1/o/uf4rhJhjTsuaIOF2rNk7jtZ6tZISsgz9bXd60W Vo/syTrq5xeFwpc45p7fEZE5O79Jm+6mqYer5yYUbV0DBkOxFZCzg13Y1/lwZBNgDN4SApUJvNc x+G6dBIMEPuha56mBwRV2jQDtZKwq6LqyNq+qAWHUXnxj+BZR6XMEqsZmIgo2Z7f6cTnHrhSKgo puAdSz0CPG5HNxPxlmhJNV7bPaamtaiNzXgzOn38Vf587ZjhZO2l0soIiSN3jnC33BH8Y3ozVvn eS748rPWywAUsBz/kmjvgMNcaHsR/SiEdCImtLBrgTGFH2nZh+TJRhNSiiCT6KqsWUFvlKzPkZr P0afEKdyta1/RkdvxgweqQykXaQw4QTVbmXShgfrvhMAQuLSBqltLB9KeeLCL70= X-Received: by 2002:a17:907:3e8f:b0:c41:4d5:e97c with SMTP id a640c23a62f3a-c4d3537fda4mr352829966b.37.1794022474460; Fri, 06 Nov 2026 19:34:34 -0800 (PST) Received: from L9JK9NPFH9.lan ([104.28.20.56]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c4d3562e00dsm121104166b.21.2026.11.06.19.34.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 06 Nov 2026 19:34:31 -0800 (PST) From: Tianyi Gao To: netdev@vger.kernel.org, Jakub Kicinski , Paolo Abeni , Eric Dumazet Cc: "David S. Miller" , Simon Horman , David Keogh , Jesper Dangaard Brouer , Daniel Borkmann , David Woodhouse , Dragos Tatulea , Nimrod Oren , Alasdair McWilliam , Willem de Bruijn , Maciej Fijalkowski , Stanislav Fomichev , Magnus Karlsson , David Ahern , Ido Schimmel , Alexei Starovoitov , John Fastabend , Donald Hunter , Andrew Lunn , Jonathan Corbet , Shuah Khan , Shuah Khan , Randy Dunlap , Breno Leitao , =?UTF-8?q?Asbj=C3=B8rn=20Sloth=20T=C3=B8nnesen?= , Danielle Ratson , Kuniyuki Iwashima , bpf@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@cloudflare.com, nh-open-source@amazon.com, Tianyi Gao Subject: [PATCH net-next 0/8] net: configurable extra TX headroom for local egress Date: Sat, 7 Nov 2026 03:34:19 +0000 Message-ID: X-Mailer: git-send-email 2.56.0 X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit A program attached at TC egress commonly pushes additional headers onto a packet, for example to encapsulate it. Locally generated UDP, raw and AF_PACKET traffic is allocated with exactly LL_RESERVED_SPACE() of headroom, which covers the link-layer header plus the driver's needed_headroom and alignment, leaving no room for headers an egress program pushes, so the push reallocates the skb head on every packet. An XDP program on a veth peer has the same problem. A local UDP, raw or AF_PACKET socket, or AF_XDP in copy mode, sends a packet on a veth device. veth_xmit() passes the skb to the peer, where the XDP program runs. The skb reaches the peer with headroom smaller than XDP_PACKET_HEADROOM (256 bytes), which XDP guarantees to its programs. With native XDP, the peer therefore has to copy the packet in veth_xdp_rcv_skb() to get the headroom required. Generic XDP on veth and other devices like macvlan and ipvlan also copies the packet in netif_receive_generic_xdp(). Add a per-device tx_extra_headroom, configured through a new IFLA_TX_EXTRA_HEADROOM rtnetlink attribute. It is a separate value from needed_headroom: LL_ALLOC_SPACE() adds it at the allocation sites only, it can be set and cleared freely without recreating the device, it is reported through its own attribute so it is never confused with the driver's requirement, and it is not inherited by stacked devices. needed_headroom keeps its current meaning and remains the threshold for deciding whether an existing packet must be expanded, so the extra reservation never forces a reallocation on its own. The value is limited to NETDEV_TX_EXTRA_HEADROOM_MAX (512 bytes). This is enough for several stacked encapsulations, for example VXLAN, GRE or IPIP, and keeps the growth of per-packet allocations modest. The netlink policy rejects a larger value. The reservation applies to the device the packet is allocated against (the one the route selects) at skb-allocation time. A GSO super-skb is therefore allocated once with the headroom in place; software segmentation copies that headroom into each segment, so the push lands in reserved space for every segment without a per-segment realloc. net/xdp/xsk.c is included among the converted paths: AF_XDP copy mode open-codes its own headroom calculation from needed_headroom, so it uses netdev_tx_alloc_headroom() to pick up the extra reservation consistently with the socket paths. Because the reservation is applied only at the allocation sites, TCP is left alone: TCP segments are built with MAX_TCP_HEADER already reserved and are checked against LL_RESERVED_SPACE(), not LL_ALLOC_SPACE(), so raising the knob does not reallocate them. Covering the TCP allocation path is deliberately not part of this series. With this series, generic XDP no longer copies packets that a local socket sends on veth. Native XDP on veth still copies them. Besides enough headroom, it also needs a head that is a page fragment. A local socket allocates the head with kmalloc, so it does not qualify. A follow-up veth patch will move this check and the copy after the program, so that XDP_PASS and XDP_DROP do not copy. One option is to raise needed_headroom instead. Setting it from user space needs driver support [1], which veth does not have. Where a driver does support it, the value is fixed at creation time. Moreover, raising it regresses current data paths, as measured below. tx_extra_headroom avoids these problems. It needs no driver support. It only affects new allocations. It can be cleared at any time. Another option, suggested in [2], is to declare the needed headroom in the BPF program itself, for example with a BTF decl tag. This works for a program at TC egress. An XDP program, however, runs on the receive side. The skb is already allocated elsewhere before it reaches the program. Auto-tuning, as suggested in [3], is out of scope for this series, but it is possible future work. ip_tunnel_adj_headroom() already does this for needed_headroom. A tuned value would also need a field that does not cause the regressions described below, and tx_extra_headroom can be that field. - Patch 1 adds the field and the LL_ALLOC_SPACE() macros. - Patch 2 switches the local-egress allocation sites to them (UDP, raw, AF_PACKET and AF_XDP). - Patch 3 exposes the attribute through rtnetlink, with documentation and the rt-link spec entry. - Patch 4 tests the attribute itself through YNL. - Patch 5 tests whether a push at TC egress reallocates the skb. - Patch 6 adds a flag to xdp_helper that sends one frame. - Patch 7 tests whether generic XDP on a veth peer copies packets from UDP and raw sockets (IPv4 and IPv6), AF_PACKET and AF_XDP. - Patch 8 tests that ip can set tx_extra_headroom on both ends of a new veth pair. - The iproute2 series "iplink: add tx_extra_headroom" is sent separately, at the same time as this series. The following case shows how raising needed_headroom regresses current data paths. Take a veth pair with veth0 in the host and veth1 in a network namespace, and an XDP program on veth1. The ideal needed_headroom for veth0 would be XDP_PACKET_HEADROOM (256 bytes). A packet allocated on veth0 then reaches the XDP program on veth1 without a copy. However, this regresses data paths that reach ip_finish_output2() or ip_forward(): a. TCP sent between the host and the namespace. TCP reserves MAX_TCP_HEADER of headroom for all its headers. After the TCP and IP headers, less headroom is left in front of the IP header than LL_RESERVED_SPACE() (272 bytes). ip_finish_output2() therefore expands the head of every TCP skb, ACKs included. b. A packet received by the host and forwarded into the namespace. mlx5 and ice running XDP leave 256 bytes in front of the Ethernet header, so today the packet reaches the XDP program on veth1 without a copy. In front of the IP header it has 270 bytes, 2 short of LL_RESERVED_SPACE() (272 bytes). ip_forward() now expands the head of every such packet. The new head is not a page fragment, so veth1 also copies the packet before XDP runs. bnxt copies the headers into a new skb with 64 bytes of headroom, so veth1 already copies its packets, and the expansion comes on top. We used patch 4/4 of Jesper Dangaard Brouer's veth reallocation RFC [4] to raise the needed_headroom of both veth0 and veth1 to 256 while XDP is attached. We then measured the throughput of cases a and b on a Xeon E5-2640 v4 with a ConnectX-4 Lx NIC, on net-next at 734bef548a96. Each value is the median of 9 rounds, and the scripts are in [5]: test tool net-next 256 (4/4 in [4]) local TCP, host to netns (a) iperf3 5.75 Gbit/s 5.59 (-2.8%) local TCP, netns to host (a) iperf3 39.14 Gbit/s 37.70 (-3.7%) TCP_RR, 1 byte (a) netperf 91582 trans/s 86296 (-5.8%) forwarded TCP (b) iperf3 7.04 Gbit/s 6.76 (-4.0%) forwarded 64B UDP (b) pktgen 873822 pps 692155 (-20.8%) bpftrace confirms that the head of every packet in these tests is expanded in the 256 column. The next table shows that tx_extra_headroom removes the reallocation caused by a push at TC egress. netperf sends 500000 UDP datagrams of 64 bytes on a veth device, and a program at TC egress pushes 20 bytes after the Ethernet header with bpf_skb_adjust_room(). Only tx_extra_headroom varies. The machine and base are the same as above, with this series applied: tx_extra_headroom tc push pskb_expand_head() calls 0 yes 500003 64 yes 0 128 yes 0 256 yes 0 256 no 0 All 5 rounds give the same counts. 500003 includes 3 datagrams that netperf sends at the end of the test. With no program pushing headers, a set value only costs the reserved bytes per allocation. It causes no reallocation or copy. [1] https://lore.kernel.org/all/a8f2f84e-7dc0-473d-afb1-af710d621f69@iogearbox.net/ [2] https://lore.kernel.org/all/bba453eb-769e-458b-88f7-8cc9bcafc1db@nvidia.com/ [3] https://lore.kernel.org/all/b8914a523b4a6b80424c5001d3d8012c6a98fd04.camel@infradead.org/ [4] https://lore.kernel.org/all/169272716651.1975370.10514711233878278884.stgit@firesoul/ [5] https://github.com/tgao-cf/tx-extra-headroom-bench/tree/main --- Changes since RFC: - Patch 1 also removes LL_RESERVED_SPACE_EXTRA(), which has no user since commit 6d5431555de0 ("caif: remove CAIF NETWORK LAYER"). - Patch 2 adds a paragraph on why ip_do_fragment() and ip6_fragment() are left alone. It also no longer says that AF_XDP copy mode uses the same calculation: xsk.c has its own, from needed_headroom. - Patch 3 changes IFLA_TX_EXTRA_HEADROOM from u16 to u32. - Patch 3 fixes the description of LL_ALLOC_SPACE() in netdevices.rst. It adds tx_extra_headroom before the HH_DATA_MOD alignment, not after. - Patch 3 names the paths that use the setting in rt-link.yaml. TCP and forwarded packets are not covered. - Patch 4 renames set_() to set_hr() and d0 to dummy0, and adds docstrings. - Patch 4 checks that the VLAN headroom is exactly 4 and that tx_extra_headroom does not change needed_headroom. - Patch 4 skips every case on a kernel without the attribute. - RFC patches 5 and 6 are mostly dropped. Patch 5 is a new test. One tc program in selftests/net compares skb->head before and after the push. It follows test_dst_clear.c in selftests/bpf. The test setup of tx_extra_headroom_dataplane.py is moved into tx_extra_headroom.py. - Patches 6-8 are new: a flag to send a frame in xdp_helper, a test of generic XDP on the veth peer, and a test that sets both ends of a veth pair at creation. - The cover letter adds the veth XDP case, the regressions from raising needed_headroom and answers to the BTF annotation and auto-tuning ideas. The tx_extra_headroom measurements are repeated on net-next 734bef548a96. The selftests run on net-next fa5196392691. The cover letter drops the RFC question, the sentence that there is no in-tree way to ask for more headroom, and the sentence about a concurrent TCP stream, which no test covers. RFC: https://lore.kernel.org/all/20261015235335.4162993-1-dkeogh@amazon.com/ David Keogh suggested that I post v1, with patches 1-4 from him: https://lore.kernel.org/all/au4j2nQdOwr90ffh@dev-dsk-dkeogh-ai2-36d3b9a7.us-west-2.amazon.com/ David Keogh has a test that tx_extra_headroom does not change the headroom of TCP packets. He will post it as a reply to this series. David Keogh (4): net: add tx_extra_headroom and LL_ALLOC_SPACE() net: reserve extra TX headroom for local egress rtnetlink: add IFLA_TX_EXTRA_HEADROOM selftests: net: add tx_extra_headroom test Tianyi Gao (4): selftests: net: tx_extra_headroom: test a push at TC egress selftests: net: add flag to send a frame in xdp_helper selftests: net: tx_extra_headroom: test generic XDP on the veth peer selftests: net: tx_extra_headroom: test veth peer at creation Documentation/netlink/specs/rt-link.yaml | 12 + .../networking/net_cachelines/net_device.rst | 1 + Documentation/networking/netdevices.rst | 35 ++ include/linux/netdevice.h | 36 +- include/uapi/linux/if_link.h | 1 + net/core/dev.c | 2 + net/core/rtnetlink.c | 19 +- net/ipv4/ip_output.c | 2 +- net/ipv4/raw.c | 2 +- net/ipv6/ip6_output.c | 2 +- net/ipv6/raw.c | 2 +- net/packet/af_packet.c | 6 +- net/xdp/xsk.c | 6 +- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/config | 1 + tools/testing/selftests/net/lib/xdp_helper.c | 113 +++++- .../selftests/net/tx_extra_headroom.bpf.c | 94 +++++ .../selftests/net/tx_extra_headroom.py | 330 ++++++++++++++++++ 18 files changed, 645 insertions(+), 20 deletions(-) create mode 100644 tools/testing/selftests/net/tx_extra_headroom.bpf.c create mode 100755 tools/testing/selftests/net/tx_extra_headroom.py base-commit: fa5196392691051963726787944db872420dfa75 -- 2.56.0