Re: [LSF/MM/BPF TOPIC] BPF local storage for every packet

[Martin KaFai Lau <martin.lau@linux.dev>]

On 2/20/26 6:56 AM, Jakub Sitnicki wrote:
> In the upcoming days we are going to post an RFC which proposes to
> extend the concept of BPF local storage to socket buffers (sk_buff, skb)
> as means to attach arbitrary metadata to packets from BPF programs [1]
> (slides 41-55).
> 
> Design wise, BPF local storage is a great fit for a packet metadata
> container, as it that avoids some of the shortcoming of the the XDP
> metadata interface:
> 
> 1. Users interact with storage through BPF maps and can take advantage
>     of existing built-in BPF map types, while still being able to
>     implement a custom data format,
> 
> 2. Maps within local storage can have different properties controlled by
>     map flags. For example, maps with BPF_F_CLONE set can survive packet
>     cloning. Other flags could allow map contents to survive sk_buff
>     scrubbing during encapsulation/decapsulation or pass across network
>     namespace boundaries.
> 
> 3. Local storage supports multiple users out of the box - each user
>     creates their own map, eliminating the need to coordinate data
>     layout,
> 
> 4. Local storage has its own backing memory, so persisting it across
>     network stack layers requires no changes to the network stack.
> 
> However, this flexibility comes at a cost. While XDP metadata requires
> no allocations [2], an initial write to BPF local storage requires two:
> one for bpf_local_storage_elem, and one for bpf_local_storage itself.
> 
> We would like to align this work with the needs of other BPF local
> storage users (socks, cgroups, tasks, inodes), where allocation overhead
> has been a concern as well [2].
> 
> Optimization ideas we would like to put up for discussion:
> - slimming down bpf_local_storage so it can be embedded as an skb
>    extension chunk,
> - making the bpf_local_storage cache size configurable,
> - allowing bpf_local_storage to be pre-allocated,
> - co-allocating bpf_local_storage and bpf_local_storage_elem for the
>    single-map case.

The sk/cgroup/task storage has a much longer lifetime. Meaning once 
allocation is done, the storage stays in the sk until the sk is closed. 
The length of lifetime is quite different from the skb. I am afraid we 
are re-purposing bpf_local_storage for a very different use case where 
skb lifecycle is much shorter.

We are planning to increase the 'sizeof(struct sock)' for perf reason. 
Saving an allocation is an upside but not the major one we are looking 
(or care) for sk. We are more looking for cacheline efficiency and 
probably remove the need for bpf_local_storage[_elem] if the user 
chooses to use the in-place spaces of a sk.

If 'sizeof(struct sk_buff)' can be increased, this should align on where 
sk local storage is going. If skb will solely depend on the existing 
bpf_local_storage and has no plan to raise sizeof(struct sk_buff) for 
perf purpose, the existing bpf_local_storage may be the wrong place to 
repurpose/optimize because the lifecycle of skb is very different.

> [1] https://fosdem.org/2026/schedule/event/DSC9L3-rich-packet-metadata/
> [2] Assuming sufficient free headroom in the skb linear buffer.
> [3] http://msgid.link/ad835a9b-e544-48d3-b6e2-ffe172fcfa6d@linux.dev
>