From: Jakub Sitnicki <jakub@cloudflare.com>
To: "Alexei Starovoitov" <alexei.starovoitov@gmail.com>
Cc: "Jakub Kicinski" <kuba@kernel.org>,
"Daniel Zahka" <daniel.zahka@gmail.com>,
<netdev@vger.kernel.org>,
"Kuniyuki Iwashima" <kuniyu@google.com>,
"Paolo Abeni" <pabeni@redhat.com>,
"Stanislav Fomichev" <sdf@fomichev.me>, <bpf@vger.kernel.org>,
<kernel-team@cloudflare.com>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"John Fastabend" <john.fastabend@gmail.com>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Eduard Zingerman" <eddyz87@gmail.com>,
"Kumar Kartikeya Dwivedi" <memxor@gmail.com>,
"Martin KaFai Lau" <martin.lau@linux.dev>,
"Song Liu" <song@kernel.org>,
"Yonghong Song" <yonghong.song@linux.dev>,
"Jiri Olsa" <jolsa@kernel.org>,
"Emil Tsalapatis" <emil@etsalapatis.com>,
"David S. Miller" <davem@davemloft.net>,
"Eric Dumazet" <edumazet@google.com>,
"Simon Horman" <horms@kernel.org>,
"Jesper Dangaard Brouer" <hawk@kernel.org>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Florian Westphal" <fw@strlen.de>,
"Jack Wang" <163wangjack@gmail.com>
Subject: Re: [PATCH net-next v2 03/14] bpf: Make BPF skb extension survive packet scrubbing
Date: Tue, 29 Sep 2026 22:32:34 +0200 [thread overview]
Message-ID: <87ecebbxy5.fsf@cloudflare.com> (raw)
In-Reply-To: <DLR2ZT65RKF3.3KZ8O8P4VKTI2@gmail.com> (Alexei Starovoitov's message of "Mon, 28 Sep 2026 16:14:04 +0000")
On Mon, Sep 28, 2026 at 04:14 PM GMT, Alexei Starovoitov wrote:
> On Mon, Sep 28, 2026 at 05:52 PM Jakub Sitnicki <jakub@cloudflare.com> wrote:
>> The correct results - as in reproducible and with lower variance - are
>> the other way around - BPF skb ext is cheaper CPU-wise than gated skb
>> tracepoints. Please see:
>>
>> https://patch.msgid.link/20260928-bpf-meta-gated-tracepoints-v1-0-844dbf3e1edf@cloudflare.com
>
> I don't think these numbers measure the mechanism either.
> cpustats.py divides busy by ncpus, so +16.4 and +6.6 are the mean
> over 6 cpus, while 146k pps is the total of 3 flows.
> So it's 6.7 usec vs 2.7 usec per packet and not 1125 ns vs 450 ns.
> That's a lot for one kmem_cache_alloc/free and a 16 byte copy.
>
> Pls share perf report for both and rerun.
My mistake. Per-packet cost shouldn't have been normalized to CPU count.
I grabbed the perf reports and flamegraphs, and added another variant to the mix
with per-CPU LRU hash instead of rhash to get an idea how much overhead comes
from locking. Gated tracepoints + per-CPU LRU hash sits in the middle with ~1.5x
the cost of skb extension:
| variant | Δ busy (mean/6) | ns/pkt @146k |
|--------------------------------|--------------------|--------------|
| gated tracepoints (rhash) | **+16.4 ± 2.8 pp** | **≈ 6750** |
| gated tracepoints (percpu LRU) | **+10.0 ± 3.3 pp** | **≈ 4100** |
| bpf skb ext | **+6.6 ± 1.1 pp** | **≈ 2720** |
% of perf samples breakdown looks like so:
| component | cnt | ext | gtplru | gtp |
|----------------------------------------|------|------|--------|------|
| tc dispatch (`tcf_classify`+`cls_bpf`) | 0.08 | 0.25 | 0.34 | 0.88 |
| skb free path (`napi_consume_skb`) | 0.05 | 0.09 | 0.16 | 0.31 |
| TX chain total (`udp_sendmsg`) | 2.51 | 2.53 | 2.98 | 3.27 |
Everything pushed to:
https://github.com/jsitnicki/skb-metadata-bench/tree/main/profile
If the numbers look fishy, after Plumbers I can set up a proper
experiment in production to see what the CPU overhead is like on bare
metal.
next prev parent reply other threads:[~2026-09-29 20:32 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-10 14:02 [PATCH net-next v2 00/14] skb extension for BPF metadata Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 01/14] bpf: Introduce per-packet metadata storage for BPF programs Jakub Sitnicki
2026-09-11 10:19 ` Jiayuan Chen
2026-09-10 14:02 ` [PATCH net-next v2 02/14] bpf: Allow access to bpf_sock_ops_kern->skb Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 03/14] bpf: Make BPF skb extension survive packet scrubbing Jakub Sitnicki
2026-09-23 17:46 ` Daniel Zahka
2026-09-24 16:52 ` Jakub Sitnicki
2026-09-25 1:07 ` Alexei Starovoitov
2026-09-25 19:18 ` Jakub Kicinski
2026-09-25 19:51 ` Alexei Starovoitov
2026-09-25 20:20 ` Jakub Kicinski
2026-09-25 20:27 ` Alexei Starovoitov
2026-09-28 15:52 ` Jakub Sitnicki
2026-09-28 16:14 ` Alexei Starovoitov
2026-09-29 20:32 ` Jakub Sitnicki [this message]
2026-09-30 9:00 ` Alexei Starovoitov
2026-10-02 12:37 ` Jakub Sitnicki
2026-09-25 12:03 ` Daniel Zahka
2026-09-25 15:24 ` Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 04/14] selftests/bpf: Add tests for bpf_dynptr_from_skb_ext Jakub Sitnicki
2026-09-11 15:09 ` sashiko-bot
2026-09-10 14:02 ` [PATCH net-next v2 05/14] selftests/bpf: Test skb_ext on cloned skbs Jakub Sitnicki
2026-09-11 15:09 ` sashiko-bot
2026-09-10 14:02 ` [PATCH net-next v2 06/14] selftests/bpf: Test skb_ext survival across veth and GRE Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 07/14] selftests/bpf: Test skb_ext read from cgroup_skb and sk_filter hooks Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 08/14] selftests/bpf: Test skb_ext read from sock_ops and LSM hooks Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 09/14] selftests/bpf: Test skb_ext read from kfree_skb tracepoint Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 10/14] selftests/bpf: Test skb_ext read from netfilter hook Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 11/14] selftests/bpf: Test skb_ext from LWT in, out, and xmit hooks Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 12/14] selftests/bpf: Test skb_ext read from seg6local End.BPF hook Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 13/14] selftests/bpf: Test skb_ext read from sk_skb stream verdict hook Jakub Sitnicki
2026-09-10 14:02 ` [PATCH net-next v2 14/14] selftests/bpf: Use non-trivial test payload in xdp_context tests Jakub Sitnicki
2026-09-10 15:56 ` [PATCH net-next v2 00/14] skb extension for BPF metadata Alexei Starovoitov
2026-09-11 11:33 ` Jakub Sitnicki
2026-09-12 3:32 ` Alexei Starovoitov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87ecebbxy5.fsf@cloudflare.com \
--to=jakub@cloudflare.com \
--cc=163wangjack@gmail.com \
--cc=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel.zahka@gmail.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=eddyz87@gmail.com \
--cc=edumazet@google.com \
--cc=emil@etsalapatis.com \
--cc=fw@strlen.de \
--cc=hawk@kernel.org \
--cc=horms@kernel.org \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kernel-team@cloudflare.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=martin.lau@linux.dev \
--cc=memxor@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sdf@fomichev.me \
--cc=song@kernel.org \
--cc=willemdebruijn.kernel@gmail.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox