Re: [PATCH net-next 1/8] net: add tx_extra_headroom and LL_ALLOC_SPACE()

[Tianyi Gao <tianyi@cloudflare.com>]

On Sun, Nov 08, 2026 at 10:13:03AM -0500, Willem de Bruijn wrote:
> Pushing back on another design choice: the UAPI change. That both cannot be
> undone, and is fragile and laborious, requiring manual admin arithmetic.

Thanks for the valuable feedback.

On fragile: a wrong value never changes the packet. If it is too
small, the push reallocates, as it does today. If it is too large,
locally generated skbs on that device use more memory. So if the value
drifts from what the programs push, only performance changes. The
software that loads the encapsulating program knows the size and can
set the value at the same time.

On manual arithmetic: user space already does this for netkit. Commit
b9ed315d3c4c ("netkit: Allow for configuring needed_{head,tail}room")
lets the Cilium CNI plugin compute the headroom and set it on the
device.

On "cannot be undone": the IFLA_TX_EXTRA_HEADROOM UAPI does not block
in-kernel mechanisms. They can write the same field later, and the UAPI
stays a bounded, explicit value.

> The RFC patch mentioned two other approaches: explicit headroom requirement
> labeling for tc and BPF, and dynamic adjustment of this field whenever a
> realloc in the stack is encountered.
>
> They are left for future work. But should those instead be implemented
> first? As said, this manual hackery cannot be undone. While in-kernel
> mechanisms can be refined and built upon.
>
> It seems to be that this series is trying to address a few related
> situations and argues that no single solution fixes all except for manual
> adjustment. But likely a combination of the above (and more) will.
>
> This does leave the veth peer to solve, but that does not seem
> fundamentally intractable.

Regarding labeling for tc and BPF: it is also a new UAPI, and a label
increases the value of the device that the program is attached to.
That is often not the device that the skb was allocated on, as in the
cases below. If we have to add a new UAPI anyway, I would prefer one
that covers all of them.

I looked at what dynamic adjustment would take. The kernel can find
the device to increase in these cases:

- XDP on a veth peer: veth_xdp_set() and generic_xdp_install() can
  increase the headroom of the peer to XDP_PACKET_HEADROOM on attach
  and reset it on detach.
- A TC egress program on the device that the skb was allocated on: a
  check around the tc egress hook catches a head reallocation, and the
  kernel increases the headroom of skb->dev.
- Stacked devices: a vlan leaves its tag in skb->vlan_tci, so the lower
  device can find the vlan with __vlan_find_dev_deep_rcu(). macvlan,
  ipvlan and team keep the dst.

It cannot in this case, which we have at Cloudflare:

- A service in a network namespace sends through a veth pair. The host
  forwards the traffic to the uplink, and a TC egress program there
  encapsulates it. ip_rcv_core() has already orphaned the skb, so the
  kernel can only guess that the veth peer of skb_iif allocated it.

Even where the kernel finds the right device, auto-tuning has problems:

- It grows too easily. Background traffic counts the same as the main
  traffic, so a few encapsulated health-check probes increase the
  headroom of every skb on the device. A threshold to filter them out
  would be one more policy value to choose.
- That growth can move skbs into a larger kmalloc size without anyone
  deciding it. On x86_64, for 1500-byte MTU UDP over IPv4 on an
  Ethernet device, 194 bytes or more moves the skb head from
  kmalloc-2048 to kmalloc-4096. That adds 2048 bytes to the memory and
  the truesize of each such skb. The IFLA_TX_EXTRA_HEADROOM UAPI can
  do the same, but then the operator chose to pay that cost.
- It only grows, unless it is reset on detach. A reset on detach must
  find every device that the auto-tuning increased, which can be a veth
  peer in another namespace.

So I prefer to keep the IFLA_TX_EXTRA_HEADROOM UAPI. If we need
auto-tuning on top of it, we can discuss it when someone actually needs
it.

Thanks again for pushing on this. If you see a simpler way to do
this, I am happy to change direction after discussing it with David
Keogh, who posted the initial RFC.