[LSF/MM/BPF TOPIC] BPF local storage for every packet

[Jakub Sitnicki <jakub@cloudflare.com>]

In the upcoming days we are going to post an RFC which proposes to
extend the concept of BPF local storage to socket buffers (sk_buff, skb)
as means to attach arbitrary metadata to packets from BPF programs [1]
(slides 41-55).

Design wise, BPF local storage is a great fit for a packet metadata
container, as it that avoids some of the shortcoming of the the XDP
metadata interface:

1. Users interact with storage through BPF maps and can take advantage
   of existing built-in BPF map types, while still being able to
   implement a custom data format,

2. Maps within local storage can have different properties controlled by
   map flags. For example, maps with BPF_F_CLONE set can survive packet
   cloning. Other flags could allow map contents to survive sk_buff
   scrubbing during encapsulation/decapsulation or pass across network
   namespace boundaries.

3. Local storage supports multiple users out of the box - each user
   creates their own map, eliminating the need to coordinate data
   layout,

4. Local storage has its own backing memory, so persisting it across
   network stack layers requires no changes to the network stack.

However, this flexibility comes at a cost. While XDP metadata requires
no allocations [2], an initial write to BPF local storage requires two:
one for bpf_local_storage_elem, and one for bpf_local_storage itself.

We would like to align this work with the needs of other BPF local
storage users (socks, cgroups, tasks, inodes), where allocation overhead
has been a concern as well [2].

Optimization ideas we would like to put up for discussion:
- slimming down bpf_local_storage so it can be embedded as an skb
  extension chunk,
- making the bpf_local_storage cache size configurable,
- allowing bpf_local_storage to be pre-allocated,
- co-allocating bpf_local_storage and bpf_local_storage_elem for the
  single-map case.

Thanks,
-jkbs

[1] https://fosdem.org/2026/schedule/event/DSC9L3-rich-packet-metadata/
[2] Assuming sufficient free headroom in the skb linear buffer.
[3] http://msgid.link/ad835a9b-e544-48d3-b6e2-ffe172fcfa6d@linux.dev