[PATCH net-next 0/8] net: configurable extra TX headroom for local egress

[Tianyi Gao <tianyi@cloudflare.com>]

A program attached at TC egress commonly pushes additional headers onto
a packet, for example to encapsulate it. Locally generated UDP, raw and
AF_PACKET traffic is allocated with exactly LL_RESERVED_SPACE() of
headroom, which covers the link-layer header plus the driver's
needed_headroom and alignment, leaving no room for headers an egress
program pushes, so the push reallocates the skb head on every packet.

An XDP program on a veth peer has the same problem. A local UDP, raw or
AF_PACKET socket, or AF_XDP in copy mode, sends a packet on a veth
device. veth_xmit() passes the skb to the peer, where the XDP program
runs. The skb reaches the peer with headroom smaller than
XDP_PACKET_HEADROOM (256 bytes), which XDP guarantees to its programs.
With native XDP, the peer therefore has to copy the packet in
veth_xdp_rcv_skb() to get the headroom required. Generic XDP on veth and
other devices like macvlan and ipvlan also copies the packet in
netif_receive_generic_xdp().

Add a per-device tx_extra_headroom, configured through a new
IFLA_TX_EXTRA_HEADROOM rtnetlink attribute. It is a separate value from
needed_headroom: LL_ALLOC_SPACE() adds it at the allocation sites only,
it can be set and cleared freely without recreating the device, it is
reported through its own attribute so it is never confused with the
driver's requirement, and it is not inherited by stacked devices.
needed_headroom keeps its current meaning and remains the threshold for
deciding whether an existing packet must be expanded, so the extra
reservation never forces a reallocation on its own.

The value is limited to NETDEV_TX_EXTRA_HEADROOM_MAX (512 bytes). This is
enough for several stacked encapsulations, for example VXLAN, GRE or IPIP,
and keeps the growth of per-packet allocations modest. The netlink policy
rejects a larger value.

The reservation applies to the device the packet is allocated against
(the one the route selects) at skb-allocation time. A GSO super-skb is
therefore allocated once with the headroom in place; software
segmentation copies that headroom into each segment, so the push lands
in reserved space for every segment without a per-segment realloc.
net/xdp/xsk.c is included among the converted paths: AF_XDP copy mode
open-codes its own headroom calculation from needed_headroom, so it uses
netdev_tx_alloc_headroom() to pick up the extra reservation consistently
with the socket paths.

Because the reservation is applied only at the allocation sites, TCP is
left alone: TCP segments are built with MAX_TCP_HEADER already reserved
and are checked against LL_RESERVED_SPACE(), not LL_ALLOC_SPACE(), so
raising the knob does not reallocate them. Covering the TCP allocation
path is deliberately not part of this series.

With this series, generic XDP no longer copies packets that a local
socket sends on veth. Native XDP on veth still copies them. Besides
enough headroom, it also needs a head that is a page fragment. A local
socket allocates the head with kmalloc, so it does not qualify. A
follow-up veth patch will move this check and the copy after the
program, so that XDP_PASS and XDP_DROP do not copy.

One option is to raise needed_headroom instead. Setting it from user space
needs driver support [1], which veth does not have. Where a driver does
support it, the value is fixed at creation time. Moreover, raising it
regresses current data paths, as measured below. tx_extra_headroom avoids
these problems. It needs no driver support. It only affects new
allocations. It can be cleared at any time.

Another option, suggested in [2], is to declare the needed headroom in
the BPF program itself, for example with a BTF decl tag. This works for a
program at TC egress. An XDP program, however, runs on the receive side.
The skb is already allocated elsewhere before it reaches the program.

Auto-tuning, as suggested in [3], is out of scope for this series, but
it is possible future work. ip_tunnel_adj_headroom() already does this
for needed_headroom. A tuned value would also need a field that does not
cause the regressions described below, and tx_extra_headroom can be that
field.

- Patch 1 adds the field and the LL_ALLOC_SPACE() macros.
- Patch 2 switches the local-egress allocation sites to them (UDP, raw,
  AF_PACKET and AF_XDP).
- Patch 3 exposes the attribute through rtnetlink, with documentation and
  the rt-link spec entry.
- Patch 4 tests the attribute itself through YNL.
- Patch 5 tests whether a push at TC egress reallocates the skb.
- Patch 6 adds a flag to xdp_helper that sends one frame.
- Patch 7 tests whether generic XDP on a veth peer copies packets from
  UDP and raw sockets (IPv4 and IPv6), AF_PACKET and AF_XDP.
- Patch 8 tests that ip can set tx_extra_headroom on both ends of a new
  veth pair.
- The iproute2 series "iplink: add tx_extra_headroom" is sent separately,
  at the same time as this series.

The following case shows how raising needed_headroom regresses current
data paths. Take a veth pair with veth0 in the host and veth1 in a network
namespace, and an XDP program on veth1. The ideal needed_headroom for
veth0 would be XDP_PACKET_HEADROOM (256 bytes). A packet allocated on
veth0 then reaches the XDP program on veth1 without a copy. However, this
regresses data paths that reach ip_finish_output2() or ip_forward():

a. TCP sent between the host and the namespace. TCP reserves
   MAX_TCP_HEADER of headroom for all its headers. After the TCP and IP
   headers, less headroom is left in front of the IP header than
   LL_RESERVED_SPACE() (272 bytes). ip_finish_output2() therefore expands
   the head of every TCP skb, ACKs included.

b. A packet received by the host and forwarded into the namespace. mlx5
   and ice running XDP leave 256 bytes in front of the Ethernet header, so
   today the packet reaches the XDP program on veth1 without a copy. In
   front of the IP header it has 270 bytes, 2 short of LL_RESERVED_SPACE()
   (272 bytes). ip_forward() now expands the head of every such packet.
   The new head is not a page fragment, so veth1 also copies the packet
   before XDP runs. bnxt copies the headers into a new skb with 64 bytes
   of headroom, so veth1 already copies its packets, and the expansion
   comes on top.

We used patch 4/4 of Jesper Dangaard Brouer's veth reallocation RFC [4] to
raise the needed_headroom of both veth0 and veth1 to 256 while XDP is
attached. We then measured the throughput of cases a and b on a Xeon
E5-2640 v4 with a ConnectX-4 Lx NIC, on net-next at 734bef548a96. Each
value is the median of 9 rounds, and the scripts are in [5]:

test                          tool     net-next       256 (4/4 in [4])
local TCP, host to netns (a)  iperf3   5.75 Gbit/s    5.59 (-2.8%)
local TCP, netns to host (a)  iperf3   39.14 Gbit/s   37.70 (-3.7%)
TCP_RR, 1 byte (a)            netperf  91582 trans/s  86296 (-5.8%)
forwarded TCP (b)             iperf3   7.04 Gbit/s    6.76 (-4.0%)
forwarded 64B UDP (b)         pktgen   873822 pps     692155 (-20.8%)

bpftrace confirms that the head of every packet in these tests is expanded
in the 256 column.

The next table shows that tx_extra_headroom removes the reallocation
caused by a push at TC egress. netperf sends 500000 UDP datagrams of 64
bytes on a veth device, and a program at TC egress pushes 20 bytes after
the Ethernet header with bpf_skb_adjust_room(). Only tx_extra_headroom
varies. The machine and base are the same as above, with this series
applied:

tx_extra_headroom  tc push  pskb_expand_head() calls
0                  yes      500003
64                 yes      0
128                yes      0
256                yes      0
256                no       0

All 5 rounds give the same counts. 500003 includes 3 datagrams that
netperf sends at the end of the test. With no program pushing headers, a
set value only costs the reserved bytes per allocation. It causes no
reallocation or copy.

[1] https://lore.kernel.org/all/a8f2f84e-7dc0-473d-afb1-af710d621f69@iogearbox.net/
[2] https://lore.kernel.org/all/bba453eb-769e-458b-88f7-8cc9bcafc1db@nvidia.com/
[3] https://lore.kernel.org/all/b8914a523b4a6b80424c5001d3d8012c6a98fd04.camel@infradead.org/
[4] https://lore.kernel.org/all/169272716651.1975370.10514711233878278884.stgit@firesoul/
[5] https://github.com/tgao-cf/tx-extra-headroom-bench/tree/main

---
Changes since RFC:
- Patch 1 also removes LL_RESERVED_SPACE_EXTRA(), which has no user since
  commit 6d5431555de0 ("caif: remove CAIF NETWORK LAYER").
- Patch 2 adds a paragraph on why ip_do_fragment() and ip6_fragment()
  are left alone. It also no longer says that AF_XDP copy mode uses
  the same calculation: xsk.c has its own, from needed_headroom.
- Patch 3 changes IFLA_TX_EXTRA_HEADROOM from u16 to u32.
- Patch 3 fixes the description of LL_ALLOC_SPACE() in netdevices.rst. It
  adds tx_extra_headroom before the HH_DATA_MOD alignment, not after.
- Patch 3 names the paths that use the setting in rt-link.yaml. TCP and
  forwarded packets are not covered.
- Patch 4 renames set_() to set_hr() and d0 to dummy0, and adds
  docstrings.
- Patch 4 checks that the VLAN headroom is exactly 4 and that
  tx_extra_headroom does not change needed_headroom.
- Patch 4 skips every case on a kernel without the attribute.
- RFC patches 5 and 6 are mostly dropped. Patch 5 is a new test. One tc
  program in selftests/net compares skb->head before and after the push.
  It follows test_dst_clear.c in selftests/bpf. The test setup of
  tx_extra_headroom_dataplane.py is moved into tx_extra_headroom.py.
- Patches 6-8 are new: a flag to send a frame in xdp_helper, a test of
  generic XDP on the veth peer, and a test that sets both ends of a veth
  pair at creation.
- The cover letter adds the veth XDP case, the regressions from raising
  needed_headroom and answers to the BTF annotation and auto-tuning
  ideas. The tx_extra_headroom measurements are repeated on net-next
  734bef548a96. The selftests run on net-next fa5196392691. The cover
  letter drops the RFC question, the sentence that there is no in-tree
  way to ask for more headroom, and the sentence about a concurrent TCP
  stream, which no test covers.

RFC: https://lore.kernel.org/all/20261015235335.4162993-1-dkeogh@amazon.com/

David Keogh suggested that I post v1, with patches 1-4 from him:
https://lore.kernel.org/all/au4j2nQdOwr90ffh@dev-dsk-dkeogh-ai2-36d3b9a7.us-west-2.amazon.com/

David Keogh has a test that tx_extra_headroom does not change the
headroom of TCP packets. He will post it as a reply to this series.

David Keogh (4):
  net: add tx_extra_headroom and LL_ALLOC_SPACE()
  net: reserve extra TX headroom for local egress
  rtnetlink: add IFLA_TX_EXTRA_HEADROOM
  selftests: net: add tx_extra_headroom test

Tianyi Gao (4):
  selftests: net: tx_extra_headroom: test a push at TC egress
  selftests: net: add flag to send a frame in xdp_helper
  selftests: net: tx_extra_headroom: test generic XDP on the veth peer
  selftests: net: tx_extra_headroom: test veth peer at creation

 Documentation/netlink/specs/rt-link.yaml      |  12 +
 .../networking/net_cachelines/net_device.rst  |   1 +
 Documentation/networking/netdevices.rst       |  35 ++
 include/linux/netdevice.h                     |  36 +-
 include/uapi/linux/if_link.h                  |   1 +
 net/core/dev.c                                |   2 +
 net/core/rtnetlink.c                          |  19 +-
 net/ipv4/ip_output.c                          |   2 +-
 net/ipv4/raw.c                                |   2 +-
 net/ipv6/ip6_output.c                         |   2 +-
 net/ipv6/raw.c                                |   2 +-
 net/packet/af_packet.c                        |   6 +-
 net/xdp/xsk.c                                 |   6 +-
 tools/testing/selftests/net/Makefile          |   1 +
 tools/testing/selftests/net/config            |   1 +
 tools/testing/selftests/net/lib/xdp_helper.c  | 113 +++++-
 .../selftests/net/tx_extra_headroom.bpf.c     |  94 +++++
 .../selftests/net/tx_extra_headroom.py        | 330 ++++++++++++++++++
 18 files changed, 645 insertions(+), 20 deletions(-)
 create mode 100644 tools/testing/selftests/net/tx_extra_headroom.bpf.c
 create mode 100755 tools/testing/selftests/net/tx_extra_headroom.py


base-commit: fa5196392691051963726787944db872420dfa75
-- 
2.56.0