From: "Alexei Starovoitov" <alexei.starovoitov@gmail.com>
To: "Kumar Kartikeya Dwivedi" <memxor@gmail.com>
Cc: "Yonghong Song" <yonghong.song@linux.dev>, <bpf@vger.kernel.org>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"Eduard Zingerman" <eddyz87@gmail.com>, <kernel-team@fb.com>
Subject: Re: [PATCH bpf-next v12 00/25] bpf: Run exception cleanup landing pads when bpf_unwind() unwinds
Date: Sat, 17 Oct 2026 21:41:09 +0000 [thread overview]
Message-ID: <DM7FULA7L0GQ.R21A648X547X@gmail.com> (raw)
In-Reply-To: <CAP01T75Ldbsew4rarQynmbpOvBxaRM7Lxk_1GsZL_rJFOzv-CQ@mail.gmail.com>
On Sat, Oct 17, 2026 at 10:03 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> still favor this approach over adding a runtime calling convention
> that requires checks everywhere in the normal path.
I suspect you're concerned by run-time overhead.
I was as well until I actually benchmarked it.
JIT can drop the branch and remember it in extable instead.
bpf x86 as-is x86 with extable
call check call check call check
if r3 != 0 goto pad test rdx, rdx call bpf_rcu_read_unlock
call bpf_rcu_read_unlock jne pad
call bpf_rcu_read_unlock
The idea:
at 'call foo; if r3 != 0 goto pad':
emit(call foo)
extable[n++] = { ret_addr -> pad }
at 'r3 = 1; exit':
my_ret_addr = bpf_unwind_target(my_ret_addr)
epilogue
bpf_unwind_target(ra)
{
e = search_extable(ra);
return e ? e->pad : ra;
}
Every frame fixes only its own return address.
No stack walk, no ORC. catch still works.
When JIT cannot drop the branch (private stack, constant blinding)
there is no entry. The callee returns as usual and the branch does the job.
The verifier doesn't know about any of that.
That said, I couldn't measure the branch.
N call sites in a row, user space, Zen 5, cycles per call:
N nothing test+jne
8192 4.04 4.06
12288 5.57 4.49
16384 6.60 10.57
32768 24.89 24.39
Up to 8k call sites it's 4 cycles with or without the branch.
Above 16k the calls don't fit into BTB and it's 24 cycles either way.
In between it goes both ways.
I also tried to keep the distance to the pad in a nop:
call foo
nopl dist(%rax)
'r3 = 1; exit' reads it via its return address. No lookup.
It works, but the numbers are the same as for test+jne.
No measurable difference in a microbenchmark.
I don't think we need the below diff from day one. I think we should
start with plain branches
and then measure benefits of above diff on real rust prog.
The kernel and llvm diffs are so simple.
It's a huge difference vs everything we discussed before.
On top of the kernel part from the previous email
(x86 only, in qemu, the same tests pass with and without bpf_jit_harden=2):
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 11a4e4f87aac..abe942631486 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -15,6 +15,7 @@
#include <linux/memory.h>
#include <linux/sort.h>
#include <linux/execmem.h>
+#include <linux/extable.h>
#include <asm/extable.h>
#include <asm/ftrace.h>
#include <asm/set_memory.h>
@@ -2511,6 +2512,89 @@ static void adjust_cmp_imm(u32 *op, s32 *imm)
}
}
+/*
+ * A subprog that can unwind says so in r3 and its caller does
+ * 'if r3 != 0 goto pad' after the call. Keep that off the normal path:
+ * the branch is not emitted, it becomes an extable entry
+ * return address of the call -> pad
+ * An exit with r3 != 0 looks its own return address up and returns to the pad.
+ * When there is no entry it returns as usual and the branch is there.
+ */
+static bool unwind_site(struct bpf_verifier_env *env, struct bpf_prog *prog, int i)
+{
+ const struct bpf_insn *insn = &prog->insnsi[i];
+
+ /* 'pop r9' after the call has to run on the way to the pad too */
+ if (!env || i + 1 >= prog->len || prog->aux->priv_stack_ptr)
+ return false;
+ if (!bpf_pseudo_call(insn) && !bpf_is_callx(insn))
+ return false;
+ /*
+ * No need to know the callee: the verifier lets r3 be read after
+ * a call only when every callee seen there says something in it.
+ */
+ return insn[1].code == (BPF_JMP | BPF_JNE | BPF_K) && !insn[1].imm &&
+ insn[1].dst_reg == BPF_REG_UNWIND &&
+ !bpf_is_jump_target(env, prog->aux->subprog_start + i + 1);
+}
+
+static u64 bpf_unwind_target(const struct bpf_prog_aux *aux, u64 ra)
+{
+ const struct exception_table_entry *e;
+ u32 i;
+
+ for (i = 0; i < aux->real_func_cnt; i++) {
+ struct bpf_prog *p = aux->func[i];
+ u64 start = (u64)p->bpf_func;
+
+ if (ra <= start || ra > start + p->jited_len)
+ continue;
+ /* - 1: what follows the call can have an entry of its own */
+ e = search_extable(p->aux->extable, p->aux->num_exentries, ra - 1);
+ return e ? ra + e->fixup : ra;
+ }
+ return ra;
+}
+
+enum { EXIT_RETURNS, EXIT_UNWINDS, EXIT_LOOK_AT_R3 };
+
+static int unwind_exit(struct bpf_verifier_env *env, struct bpf_prog *prog, int i)
+{
+ const struct bpf_insn *insn = &prog->insnsi[i];
+
+ if (!env || !prog->aux->ret_unwind)
+ return EXIT_RETURNS;
+ if (!i || bpf_is_jump_target(env, prog->aux->subprog_start + i) ||
+ insn[-1].code != (BPF_ALU64 | BPF_MOV | BPF_K) ||
+ insn[-1].dst_reg != BPF_REG_UNWIND)
+ return EXIT_LOOK_AT_R3;
+ return insn[-1].imm ? EXIT_UNWINDS : EXIT_RETURNS;
+}
+
+static int emit_unwind_ret_addr(u8 **pprog, struct bpf_prog *bpf_prog, u8 *ip, bool look_at_r3)
+{
+ u8 *prog = *pprog, *start = prog, *jmp = NULL;
+
+ if (look_at_r3) {
+ EMIT3(0x48, 0x85, 0xD2); /* test rdx, rdx */
+ EMIT2(X86_JE, 0);
+ jmp = prog;
+ }
+ EMIT3(0x50, 0x56, 0x52); /* push rax; push rsi; push rdx */
+ emit_mov_imm64(&prog, BPF_REG_1, (long)bpf_prog->aux >> 32, (u32)(long)bpf_prog->aux);
+ EMIT4(0x48, 0x8B, 0x75, 0x08); /* mov rsi, [rbp + 8] */
+ ip += prog - start;
+ ip += x86_call_depth_emit_accounting(&prog, bpf_unwind_target, ip);
+ if (emit_call(&prog, bpf_unwind_target, ip))
+ return -EINVAL;
+ EMIT4(0x48, 0x89, 0x45, 0x08); /* mov [rbp + 8], rax */
+ EMIT3(0x5A, 0x5E, 0x58); /* pop rdx; pop rsi; pop rax */
+ if (jmp)
+ jmp[-1] = prog - jmp;
+ *pprog = prog;
+ return 0;
+}
+
static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
{
@@ -2518,7 +2602,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
struct bpf_insn *insn = bpf_prog->insnsi;
bool callee_regs_used[4] = {};
int insn_cnt = bpf_prog->len;
- bool seen_exit = false;
+ bool seen_exit = false, skip_branch = false;
u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
void __percpu *priv_frame_ptr = NULL;
u16 out_stack_arg_cnt, outgoing_rsp;
@@ -2655,6 +2739,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
int nops;
u32 jmp_op = BPF_OP(insn->code);
s32 cmp_imm;
+ int unw;
+
+ if (skip_branch) {
+ skip_branch = false;
+ goto insn_done;
+ }
if (priv_frame_ptr) {
if (src_reg == BPF_REG_FP)
@@ -3460,6 +3550,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
ip += x86_call_depth_emit_accounting(&prog, func, ip);
if (emit_call(&prog, func, ip))
return -EINVAL;
+ if (unwind_site(env, bpf_prog, i - 1))
+ goto unwind_exentry;
if (priv_frame_ptr)
pop_r9(&prog);
/*
@@ -3487,6 +3579,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
if (emit_indirect_call(&prog, insn->dst_reg, ip))
return -EINVAL;
+ if (unwind_site(env, bpf_prog, i - 1)) {
+ struct exception_table_entry *ex;
+ int ra;
+unwind_exentry:
+ skip_branch = true;
+ if (!bpf_prog->aux->extable)
+ break;
+ if (excnt >= bpf_prog->aux->num_exentries) {
+ pr_err("unwind extable bug\n");
+ return -EFAULT;
+ }
+ ra = proglen + (prog - temp);
+ ex = &bpf_prog->aux->extable[excnt++];
+ jmp_offset = image + ra - 1 - (u8 *)&ex->insn;
+ /* switch ex to rw buffer for writes */
+ ex = (void *)rw_image + ((void *)ex - (void *)image);
+ ex->insn = jmp_offset;
+ ex->fixup = addrs[i + 1 + insn[1].off] - ra;
+ ex->data = EX_TYPE_BPF;
+ break;
+ }
if (priv_frame_ptr)
pop_r9(&prog);
break;
@@ -3772,13 +3885,17 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
break;
case BPF_JMP | BPF_EXIT:
+ unw = unwind_exit(env, bpf_prog, i - 1);
+ if (unw != EXIT_RETURNS &&
+ emit_unwind_ret_addr(&prog, bpf_prog, ip, unw == EXIT_LOOK_AT_R3))
+ return -EINVAL;
if (seen_exit) {
jmp_offset = ctx->cleanup_addr - addrs[i];
goto emit_jmp;
}
seen_exit = true;
/* Update cleanup_addr */
- ctx->cleanup_addr = proglen;
+ ctx->cleanup_addr = proglen + (prog - temp);
if (bpf_prog_was_classic(bpf_prog) &&
!ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
@@ -3811,6 +3928,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
return -EINVAL;
}
+insn_done:
ilen = prog - temp;
if (ilen > BPF_MAX_INSN_SIZE) {
pr_err("bpf_jit: fatal insn size error\n");
@@ -4864,6 +4982,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
is_atomic_fetch_op(insn))
prog->aux->num_exentries++;
}
+ for (i = 0; i < prog->len; i++)
+ if (unwind_site(env, prog, i))
+ prog->aux->num_exentries++;
addrs = kvmalloc_objs(*addrs, prog->len + 1);
if (!addrs)
prev parent reply other threads:[~2026-10-17 21:41 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-17 2:51 [PATCH bpf-next v12 00/25] bpf: Run exception cleanup landing pads when bpf_unwind() unwinds Yonghong Song
2026-10-17 2:51 ` [PATCH bpf-next v12 01/25] bpf: Accept the compiler's exception cleanup table at program load Yonghong Song
2026-10-17 2:51 ` [PATCH bpf-next v12 02/25] bpf: Add the bpf_unwind() and bpf_unwind_resume() kfuncs Yonghong Song
2026-10-17 2:51 ` [PATCH bpf-next v12 03/25] bpf: Keep a call site's landing pad in insn_aux_data, add lookups Yonghong Song
2026-10-17 2:51 ` [PATCH bpf-next v12 04/25] bpf: Mark covered call sites and check a program can take a table Yonghong Song
2026-10-17 2:51 ` [PATCH bpf-next v12 05/25] bpf: Make exception landing pads reachable in the CFG Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 06/25] bpf: Verify an unwind through landing pads and epilogues Yonghong Song
2026-10-17 3:27 ` sashiko-bot
2026-10-17 5:20 ` Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 07/25] bpf: Refuse a callback that can unwind Yonghong Song
2026-10-17 3:19 ` sashiko-bot
2026-10-17 5:23 ` Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 08/25] bpf: Refuse a landing pad that does not resume Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 09/25] bpf: Do not use a private stack for a program that can unwind Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 10/25] bpf: Prepare JITed programs for dispatching cleanup pads Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 11/25] bpf: Keep an epilogue for each subprog an unwind can pass through Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 12/25] bpf: Allow a subprog to end with a call that does not return Yonghong Song
2026-10-17 3:35 ` bot+bpf-ci
2026-10-17 5:38 ` Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 13/25] bpf: Dispatch cleanup pads by rewriting return addresses Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 14/25] bpf: Refuse a trampoline that calls a subprog that can unwind Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 15/25] bpf, x86: Dispatch exception cleanup pads at run time Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 16/25] bpf, arm64: " Yonghong Song
2026-10-17 2:52 ` [PATCH bpf-next v12 17/25] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 18/25] libbpf: Add cleanup_info to bpf_prog_load_opts Yonghong Song
2026-10-17 2:57 ` sashiko-bot
2026-10-17 2:53 ` [PATCH bpf-next v12 19/25] libbpf: Collect .bpf_cleanup records and pass them to the kernel Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 20/25] libbpf: Carry the exception cleanup table through the light skeleton Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 21/25] libbpf: Let the static linker carry .bpf_cleanup relocations Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 22/25] selftests/bpf: Add end-to-end and negative .bpf_cleanup exception tests Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 23/25] selftests/bpf: Add __set_global() and __ret_global() test tags Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 24/25] selftests/bpf: Cover more accepted .bpf_cleanup exception shapes Yonghong Song
2026-10-17 2:53 ` [PATCH bpf-next v12 25/25] selftests/bpf: Load an exception cleanup program from a light skeleton Yonghong Song
2026-10-17 17:18 ` [PATCH bpf-next v12 00/25] bpf: Run exception cleanup landing pads when bpf_unwind() unwinds Alexei Starovoitov
2026-10-17 18:26 ` Alexei Starovoitov
2026-10-17 20:03 ` Kumar Kartikeya Dwivedi
2026-10-17 21:41 ` Alexei Starovoitov [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DM7FULA7L0GQ.R21A648X547X@gmail.com \
--to=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=eddyz87@gmail.com \
--cc=kernel-team@fb.com \
--cc=memxor@gmail.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.