* [PATCH bpf-next v3] bpf, mprog: Defer RCU synchronization until peer reuse
@ 2026-11-16 14:53 Qi Wu
2026-11-16 15:43 ` bot+bpf-ci
2026-11-17 5:20 ` patchwork-bot+netdevbpf
0 siblings, 2 replies; 3+ messages in thread
From: Qi Wu @ 2026-11-16 14:53 UTC (permalink / raw)
To: Daniel Borkmann, Alexei Starovoitov, Andrii Nakryiko,
Nikolay Aleksandrov, bpf, netdev
Cc: Martin KaFai Lau, John Fastabend, Stanislav Fomichev,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Jakub Kicinski, Paolo Abeni, Eric Dumazet, David S . Miller,
Andrew Lunn
bpf_mprog_attach() swaps the active entry and the caller then waits
for a grace period (tcx_entry_sync() / netkit_entry_sync()) before
returning. But attach releases nothing: the old entry only has to
stay intact until the next attach or detach overwrites it as the
peer. The unconditional wait buys nothing on the attach path, yet it
runs inside rtnl_lock() and serializes the whole machine's network
control plane. Creating containers in bulk pays 2N grace periods for
N devices, one first attach per direction.
Borrow the scheme used by mini_qdisc_pair_swap() in commit 267463823adb
("net: sch: eliminate unnecessary RCU waits in mini_qdisc_pair_swap()"):
record the grace-period state after the swap and only wait before the
old buffer is written again. bpf_mprog_defer_sync() records
start_poll_synchronize_rcu() on the attach swap, bpf_mprog_peer() calls
cond_synchronize_rcu() before returning the peer to be overwritten, and
a new bundle starts with get_completed_synchronize_rcu() so the first
attach never waits. The state is recorded only on the attach swap, not
in bpf_mprog_commit(): a detach or replace leaves it alone, so it does
not charge an extra grace period to the next change. This removes the
attach-path sync from both tcx and netkit with no 'created' special
case, and covers a clsact qdisc having published the entry first.
A/B on the same bpf-next base (8835113107b2), identical configs apart
from LOCALVERSION, lock/RCU debugging off, 64 vCPU KVM guest. Average
operation latency. The attach rows use 30 samples each. The detach
row measures close(link_fd), excluding the first detach in each run:
35 samples across five runs of eight links.
scenario base v3 speedup
first link attach 14.377 ms 268.4 us 54x
first prog attach 12.411 ms 290.0 us 43x
clsact qdisc first 11.562 ms 116.2 us 99x
detach after detach 6.2 ms 5.2 ms ~1x
An attach waits only if the grace period recorded by an earlier
attach swap has not completed. Detach after detach stays at the base
cost: recording the cookie on every commit makes a detach following
another detach wait twice (N links cost 2N-1 grace periods rather
than N, measured 15.0 ms), because bpf_mprog_peer() charges the
previous commit's grace period to the next detach. Recording it only
on the attach swap avoids that.
Functional checks pass and a 300-cycle attach/detach churn under live
traffic shows no splats.
BPF CI: https://github.com/kernel-patches/bpf/pull/14821 (39 passed,
0 failed; x86_64 gcc-15/llvm-21, aarch64, s390x).
The implementation and measurement were produced with the help of an
AI coding assistant; the change was reviewed by the submitter.
Suggested-by: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Signed-off-by: Qi Wu <shijing.wuqi@linux.alibaba.com>
---
drivers/net/netkit.c | 4 ++--
include/linux/bpf_mprog.h | 27 ++++++++++++++++++++++-----
kernel/bpf/tcx.c | 4 ++--
3 files changed, 26 insertions(+), 9 deletions(-)
diff --git a/drivers/net/netkit.c b/drivers/net/netkit.c
index 82e608e733d9..75a1a561e5aa 100644
--- a/drivers/net/netkit.c
+++ b/drivers/net/netkit.c
@@ -772,7 +772,7 @@ int netkit_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
if (!ret) {
if (entry != entry_new) {
netkit_entry_update(dev, entry_new);
- netkit_entry_sync();
+ bpf_mprog_defer_sync(entry);
}
bpf_mprog_commit(entry);
}
@@ -854,7 +854,7 @@ static int netkit_link_prog_attach(struct bpf_link *link, u32 flags,
if (!ret) {
if (entry != entry_new) {
netkit_entry_update(dev, entry_new);
- netkit_entry_sync();
+ bpf_mprog_defer_sync(entry);
}
bpf_mprog_commit(entry);
}
diff --git a/include/linux/bpf_mprog.h b/include/linux/bpf_mprog.h
index 0b9f4caeeb0a..2e180a43c8a0 100644
--- a/include/linux/bpf_mprog.h
+++ b/include/linux/bpf_mprog.h
@@ -30,8 +30,8 @@
* if (!ret) {
* if (entry != entry_new) {
* // swap @entry to @entry_new at attach location
- * // ensure there are no inflight users of @entry:
- * synchronize_rcu();
+ * // nothing is released, no need to wait for users of @entry:
+ * bpf_mprog_defer_sync(entry);
* }
* bpf_mprog_commit(entry);
* } else {
@@ -104,9 +104,13 @@
*
* Fast path accesses the active bpf_mprog_entry within RCU critical section
* (in case of tcx it runs in NAPI which provides RCU protection there,
- * other users might need explicit rcu_read_lock()). The bpf_mprog_commit()
- * assumes that for the old bpf_mprog_entry there are no inflight users
- * anymore.
+ * other users might need explicit rcu_read_lock()). In the detach case the
+ * bpf_mprog_commit() assumes that for the old bpf_mprog_entry there are no
+ * inflight users anymore. In the attach case nothing is released, and the
+ * old bpf_mprog_entry only has to stay intact until its users are gone:
+ * bpf_mprog_defer_sync() records the RCU state after the swap, and
+ * bpf_mprog_peer() waits for it before the old bpf_mprog_entry is written
+ * again.
*
* The READ_ONCE()/WRITE_ONCE() pairing for bpf_mprog_fp's prog access is for
* the replacement case where we don't swap the bpf_mprog_entry.
@@ -147,6 +151,7 @@ struct bpf_mprog_bundle {
struct bpf_mprog_cp cp_items[BPF_MPROG_MAX];
struct bpf_prog *ref;
atomic64_t revision;
+ unsigned long rcu_state;
u32 count;
};
@@ -158,12 +163,23 @@ struct bpf_tuple {
static inline struct bpf_mprog_entry *
bpf_mprog_peer(const struct bpf_mprog_entry *entry)
{
+ /*
+ * The peer was active until the last swap and the caller is going
+ * to overwrite it, therefore ensure that there are no inflight
+ * users on it anymore. Paired with bpf_mprog_defer_sync().
+ */
+ cond_synchronize_rcu(entry->parent->rcu_state);
if (entry == &entry->parent->a)
return &entry->parent->b;
else
return &entry->parent->a;
}
+static inline void bpf_mprog_defer_sync(struct bpf_mprog_entry *entry)
+{
+ entry->parent->rcu_state = start_poll_synchronize_rcu();
+}
+
static inline void bpf_mprog_bundle_init(struct bpf_mprog_bundle *bundle)
{
BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
@@ -172,6 +188,7 @@ static inline void bpf_mprog_bundle_init(struct bpf_mprog_bundle *bundle)
memset(bundle, 0, sizeof(*bundle));
atomic64_set(&bundle->revision, 1);
+ bundle->rcu_state = get_completed_synchronize_rcu();
bundle->a.parent = bundle;
bundle->b.parent = bundle;
}
diff --git a/kernel/bpf/tcx.c b/kernel/bpf/tcx.c
index 02db0113b8e7..12a16200cb44 100644
--- a/kernel/bpf/tcx.c
+++ b/kernel/bpf/tcx.c
@@ -42,7 +42,7 @@ int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
if (!ret) {
if (entry != entry_new) {
tcx_entry_update(dev, entry_new, ingress);
- tcx_entry_sync();
+ bpf_mprog_defer_sync(entry);
tcx_skeys_inc(ingress);
}
bpf_mprog_commit(entry);
@@ -156,7 +156,7 @@ static int tcx_link_prog_attach(struct bpf_link *link, u32 flags, u32 id_or_fd,
if (!ret) {
if (entry != entry_new) {
tcx_entry_update(dev, entry_new, ingress);
- tcx_entry_sync();
+ bpf_mprog_defer_sync(entry);
tcx_skeys_inc(ingress);
}
bpf_mprog_commit(entry);
--
2.47.3
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH bpf-next v3] bpf, mprog: Defer RCU synchronization until peer reuse
2026-11-16 14:53 [PATCH bpf-next v3] bpf, mprog: Defer RCU synchronization until peer reuse Qi Wu
@ 2026-11-16 15:43 ` bot+bpf-ci
2026-11-17 5:20 ` patchwork-bot+netdevbpf
1 sibling, 0 replies; 3+ messages in thread
From: bot+bpf-ci @ 2026-11-16 15:43 UTC (permalink / raw)
To: shijing.wuqi, daniel, ast, andrii, razor, bpf, netdev
Cc: martin.lau, john.fastabend, sdf, eddyz87, memxor, song,
yonghong.song, jolsa, emil, ihor.solodrai, kuba, pabeni, edumazet,
davem, andrew+netdev, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]
> bpf, mprog: Defer RCU synchronization until peer reuse
TL;DR: Commit message nits only: rename the "v3" column in the benchmark
table, and move the BPF CI link and pass counts below the '---' line. No code
issues raised.
> bpf, mprog: Defer RCU synchronization until peer reuse
>
> [ ... ]
>
> A/B on the same bpf-next base (8835113107b2), identical configs apart
> from LOCALVERSION, lock/RCU debugging off, 64 vCPU KVM guest. Average
> operation latency. The attach rows use 30 samples each. The detach
> row measures close(link_fd), excluding the first detach in each run:
> 35 samples across five runs of eight links.
> scenario base v3 speedup
> first link attach 14.377 ms 268.4 us 54x
> first prog attach 12.411 ms 290.0 us 43x
> clsact qdisc first 11.562 ms 116.2 us 99x
> detach after detach 6.2 ms 5.2 ms ~1x
>
> [ ... ]
>
> Functional checks pass and a 300-cycle attach/detach churn under live
> traffic shows no splats.
> BPF CI: https://github.com/kernel-patches/bpf/pull/14821 (39 passed,
> 0 failed; x86_64 gcc-15/llvm-21, aarch64, s390x).
This isn't a bug, but could the "v3" column in the benchmark table be
renamed (for example "patched"), since a patch revision label means nothing
once the commit is in git history?
Also, could the "BPF CI: https://github.com/kernel-patches/bpf/pull/14821
(39 passed, 0 failed; ...)" line and its pass counts be moved below the
'---' line so they do not end up in git history?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43894793326
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH bpf-next v3] bpf, mprog: Defer RCU synchronization until peer reuse
2026-11-16 14:53 [PATCH bpf-next v3] bpf, mprog: Defer RCU synchronization until peer reuse Qi Wu
2026-11-16 15:43 ` bot+bpf-ci
@ 2026-11-17 5:20 ` patchwork-bot+netdevbpf
1 sibling, 0 replies; 3+ messages in thread
From: patchwork-bot+netdevbpf @ 2026-11-17 5:20 UTC (permalink / raw)
To: Qi Wu
Cc: daniel, ast, andrii, razor, bpf, netdev, martin.lau,
john.fastabend, sdf, eddyz87, memxor, song, yonghong.song, jolsa,
emil, ihor.solodrai, kuba, pabeni, edumazet, davem, andrew+netdev
Hello:
This patch was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:
On Mon, 16 Nov 2026 22:53:18 +0800 you wrote:
> bpf_mprog_attach() swaps the active entry and the caller then waits
> for a grace period (tcx_entry_sync() / netkit_entry_sync()) before
> returning. But attach releases nothing: the old entry only has to
> stay intact until the next attach or detach overwrites it as the
> peer. The unconditional wait buys nothing on the attach path, yet it
> runs inside rtnl_lock() and serializes the whole machine's network
> control plane. Creating containers in bulk pays 2N grace periods for
> N devices, one first attach per direction.
>
> [...]
Here is the summary with links:
- [bpf-next,v3] bpf, mprog: Defer RCU synchronization until peer reuse
https://git.kernel.org/bpf/bpf-next/c/32b66b94c86f
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-17 5:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 14:53 [PATCH bpf-next v3] bpf, mprog: Defer RCU synchronization until peer reuse Qi Wu
2026-11-16 15:43 ` bot+bpf-ci
2026-11-17 5:20 ` patchwork-bot+netdevbpf
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox