* [PATCH v4 bpf-next 1/6] bpf: Add unwind_goto and unwind_exit insns
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
@ 2026-10-20 18:13 ` Alexei Starovoitov
2026-10-20 18:57 ` sashiko-bot
2026-10-20 18:13 ` [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit Alexei Starovoitov
` (5 subsequent siblings)
6 siblings, 1 reply; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 18:13 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor, yonghong.song
From: Alexei Starovoitov <ast@kernel.org>
Rust Drop, C++ destructors and __attribute__((cleanup)) in C have to run
when a panic or an exception passes through a frame.
v1 [1] did it with a flag in r3 and 'if r3 != 0 goto pad' after the call.
Three bugs that the bots found there had the same cause: r3 is an ordinary
register, and JITs guessed the meaning from the shape of the code.
Kumar suggested new insns instead [2]:
unwind_goto pad BPF_JMP | BPF_JCOND, src_reg = BPF_UNWIND_GOTO
unwind_exit BPF_JMP | BPF_EXIT, src_reg = BPF_UNWIND_EXIT
'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit
and falls through when it left with exit. No register says which one it
was. In everything else unwind_exit is exit.
llvm lowers unwinding to them [3]:
ret exit
invoke foo unwind pad call foo; unwind_goto pad
resume r0 = 0; unwind_exit
call _Unwind_RaiseException r0 = 0; unwind_exit
invoke _Unwind_RaiseException goto pad
In a function that returns 16 bytes it's 'r0 = 0; r2 = 0; unwind_exit'.
Landing pads are ordinary blocks, so the verifier checks them like any
other code. A pad that forgets to drop a lock or a reference is a leak on
that path. There are no tables, no stack walk, no kfuncs, no BTF and
nothing to do for libbpf and other loaders.
In the verifier:
- unwind_goto has to follow a call or callx of a subprog and cannot be
a jump target. For cfg, liveness and the rest it's one more conditional
jump, like may_goto.
- How the callee left is one bit of the verifier state. It lives from
the exit of the callee to the next insn. states_equal() looks at it,
since the registers can be the same either way.
- unwind_exit is rejected when unwind_goto doesn't follow the call.
- A global subprog that has unwind_exit can leave either way. Both are
explored after the call, like it's done for might_throw.
- A sync callback cannot unwind. The helper that calls it doesn't expect
that: bpf_loop() would go on.
- unwind_exit of the main prog is exit, unless it's an extension. So it is
in async and exception callbacks. The kernel calls them, and there is no
frame to unwind to.
- An extension leaves either way, like the subprog that it replaced.
freplace is a jump, so the extension returns to the caller of
the subprog. An extension that has unwind_exit can replace only a subprog
that has it too. Otherwise there is no unwind_goto after the calls.
- Of tracing progs only fentry can attach to a subprog or an extension
that has unwind_exit. It would return to the trampoline that called it.
- unwind_exit changes its return address, so the cpu mispredicts
the return and runs what follows unwind_goto speculatively. Without
CAP_PERFMON the verifier looks at that path too.
- unwind_goto goes away together with a pad that turned out to be dead
code. It stays when what follows it is dead code. The cpu predicts that
the callee returns there, so 'goto pc-1' takes the place of the dead
code. Otherwise its removal would move other code there.
JITs don't emit a branch:
unwind_goto:
extable[n++] = { ret_addr - 1 -> pad }
unwind_exit:
my_ret_addr = bpf_unwind_target(my_ret_addr)
epilogue
So exit and the normal path after the call cost nothing. Every frame fixes
only its own return address. There is no stack walk. The lookup is what
the page fault handler does for bpf progs.
The interpreter cannot do that, so the prog has to be JITed, and JITs opt
in with bpf_jit_supports_unwind(). x86-64 and arm64 are in the next
patches.
BPF_JCOND is the same opcode as BPF_CALL_ARGS of the interpreter, and
BPF_UNWIND_GOTO is the same number as BPF_PSEUDO_CALL. may_goto is gone by
the time a prog is dumped, unwind_goto is not. bpf_insn_prepare_dump()
tells them apart: a prog with unwind_goto is always JITed.
Compared to v1 there is no BTF decl_tag and no 'r3 = 0' before every exit.
Tail call and ld_abs are fine in a subprog that can unwind, since they
leave it like exit does.
unwind_exit doesn't set r0. An implicit definition of r0 would need
special cases in backtracking, in liveness and in the check of the return
value of the main prog. A plain 'r0 = 0' in front of it needs none.
Rust
rust-bpf [4] builds with that llvm as it is. The programs are built with
-C panic=unwind. The panic handler prints the message and calls
_Unwind_RaiseException().
exc_cleanup.rs panics three frames deep, with preemption disabled in one
frame and rcu_read_lock held in another. All Drop impls run and the prog
returns 0.
scx_simple.rs is 2195 insns with 48 unwind_goto, 20 of them after callx,
and 16 unwind_exit. With r3 it was 2216 insns. scx_cosmos.rs is 3694 insns
with 68 and 18. It was 3716.
C++
Exceptions work too. This compiles, loads and runs:
struct rcu_guard {
rcu_guard() { bpf_rcu_read_lock(); }
~rcu_guard() { bpf_rcu_read_unlock(); }
};
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
SEC("tc") int prog(struct __sk_buff *skb)
{
try {
return inner(skb->len);
} catch (...) {
return 42;
}
}
It returns 64 for a packet of 64 bytes and 42 for a packet of 200 bytes.
The destructor runs in both cases, otherwise the verifier would reject
the prog. 'throw' is a jump to the pad with the destructor. 'catch' is
unwind_goto:
inner:
w6 = w1
call bpf_rcu_read_lock
if w6 s< 101 goto ret
r1 = exc_obj ll
*(u32 *)(r1 + 0) = 1
goto pad
ret:
call bpf_rcu_read_unlock
w0 = w6
exit
pad:
call bpf_rcu_read_unlock
r0 = 0
unwind_exit
prog:
w6 = *(u32 *)(r1 + 0)
w1 = w6
call inner
unwind_goto catch
out:
w0 = w6
exit
catch:
w6 = 42
goto out
It needs -fexceptions -funwind-tables and four one line __cxa_*()
functions, where __cxa_throw() calls _Unwind_RaiseException().
Every pad is entered whatever was thrown, so llvm takes catch (...) only.
It's a selftest in patch 5.
[1] https://lore.kernel.org/bpf/20261018010246.1643877-1-alexei.starovoitov@gmail.com/
[2] https://lore.kernel.org/bpf/DM7KR7J8VY2T.1CZIRUF6QRCWG@gmail.com/
[3] https://github.com/llvm/llvm-project/pull/233649
[4] https://github.com/4ast/rust-bpf
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 1 +
include/linux/bpf_verifier.h | 3 +
include/linux/filter.h | 20 ++++++
include/uapi/linux/bpf.h | 13 ++++
kernel/bpf/const_fold.c | 2 +-
kernel/bpf/core.c | 6 ++
kernel/bpf/disasm.c | 7 ++-
kernel/bpf/fixups.c | 18 ++++++
kernel/bpf/states.c | 3 +
kernel/bpf/syscall.c | 6 ++
kernel/bpf/verifier.c | 109 ++++++++++++++++++++++++++++++++-
tools/include/uapi/linux/bpf.h | 13 ++++
12 files changed, 197 insertions(+), 4 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 7750df973494..580f89c450b5 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1897,6 +1897,7 @@ struct bpf_prog_aux {
bool changes_pkt_data;
bool has_prologue;
bool might_sleep;
+ bool can_unwind;
bool kprobe_write_ctx;
/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
bool uses_arena_insns;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 54481da4ed2d..6606b341351e 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -532,6 +532,8 @@ struct bpf_verifier_state {
bool speculative;
bool in_sleepable;
+ /* The callee left with unwind_exit. Next insn is unwind_goto. */
+ bool unwound;
/*
* First and last insn idx of this verifier state.
@@ -957,6 +959,7 @@ struct bpf_subprog_info {
s16 fastcall_stack_off;
bool has_tail_call: 1;
bool might_throw: 1;
+ bool can_unwind: 1; /* has unwind_exit */
bool tail_call_reachable: 1;
bool has_ld_abs: 1;
bool is_cb: 1;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 6f4c9bd024a0..fccd8db33647 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -254,6 +254,25 @@ static inline bool insn_is_mov_percpu_addr(const struct bpf_insn *insn)
.off = 0, \
.imm = 1 })
+static inline bool bpf_is_unwind_goto(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_UNWIND_GOTO;
+}
+
+static inline bool bpf_is_unwind_exit(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_JMP | BPF_EXIT) && insn->src_reg == BPF_UNWIND_EXIT;
+}
+
+/*
+ * The caller of the main prog doesn't tell unwind_exit from exit. An extension
+ * is different: its caller is the caller of the subprog that it replaced.
+ */
+static inline bool bpf_prog_can_unwind(const struct bpf_prog *prog)
+{
+ return bpf_is_subprog(prog) || prog->type == BPF_PROG_TYPE_EXT;
+}
+
static inline bool insn_is_zext(const struct bpf_insn *insn)
{
return insn->code == (BPF_ALU | BPF_MOV | BPF_X) && insn->imm == 1;
@@ -1257,6 +1276,7 @@ bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_callx(void);
bool bpf_jit_supports_kfunc_ret_reg_pair(void);
+bool bpf_jit_supports_unwind(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 6f70db7515dd..f29b0aad440c 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -56,6 +56,19 @@
enum bpf_cond_pseudo_jmp {
BPF_MAY_GOTO = 0,
+ /*
+ * Follows a call of a bpf subprog. Taken when the subprog left with
+ * BPF_EXIT that has src_reg == BPF_UNWIND_EXIT.
+ */
+ BPF_UNWIND_GOTO = 1,
+};
+
+enum bpf_pseudo_exit {
+ /*
+ * When bpf_exit->src_reg == BPF_UNWIND_EXIT, it's the same as bpf_exit,
+ * except that BPF_UNWIND_GOTO after the call is taken.
+ */
+ BPF_UNWIND_EXIT = 1,
};
/* Register numbers */
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index 8dd8c5958eb8..f9edfde965dc 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -357,7 +357,7 @@ int bpf_prune_dead_branches(struct bpf_verifier_env *env)
if (!bpf_insn_is_cond_jump(insn->code))
continue;
- if (bpf_is_may_goto_insn(insn))
+ if (BPF_OP(insn->code) == BPF_JCOND)
continue;
if (!(aux->const_reg_mask & BIT(insn->dst_reg)))
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index fc22d66d194a..3030beb531d9 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3399,6 +3399,12 @@ bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
return false;
}
+/* unwind_goto and unwind_exit */
+bool __weak bpf_jit_supports_unwind(void)
+{
+ return false;
+}
+
/*
* How this arch places a by-value kfunc argument, or NULL for one that has
* not opted in and so only takes an argument of a single eightbyte, which
diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c
index 36d3228d7745..889fc074496f 100644
--- a/kernel/bpf/disasm.c
+++ b/kernel/bpf/disasm.c
@@ -375,11 +375,16 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
insn->src_reg == BPF_MAY_GOTO) {
verbose(cbs->private_data, "(%02x) may_goto pc%+d",
insn->code, insn->off);
+ } else if (insn->code == (BPF_JMP | BPF_JCOND) &&
+ insn->src_reg == BPF_UNWIND_GOTO) {
+ verbose(cbs->private_data, "(%02x) unwind_goto pc%+d",
+ insn->code, insn->off);
} else if (insn->code == (BPF_JMP32 | BPF_JA)) {
verbose(cbs->private_data, "(%02x) gotol pc%+d",
insn->code, insn->imm);
} else if (insn->code == (BPF_JMP | BPF_EXIT)) {
- verbose(cbs->private_data, "(%02x) exit", insn->code);
+ verbose(cbs->private_data, "(%02x) %s", insn->code,
+ insn->src_reg == BPF_UNWIND_EXIT ? "unwind_exit" : "exit");
} else if (BPF_SRC(insn->code) == BPF_X) {
verbose(cbs->private_data,
"(%02x) if %c%d %s %c%d goto pc%+d",
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 409b9104c723..f062ef7dd943 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -658,6 +658,7 @@ void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env)
{
struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
struct bpf_insn ja = BPF_JMP_IMM(BPF_JA, 0, 0, 0);
+ struct bpf_insn trap = BPF_JMP_IMM(BPF_JA, 0, 0, -1);
struct bpf_insn *insn = env->prog->insnsi;
const int insn_cnt = env->prog->len;
int i;
@@ -666,6 +667,20 @@ void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env)
if (!bpf_insn_is_cond_jump(insn->code))
continue;
+ if (bpf_is_unwind_goto(insn) && aux_data[i + 1 + insn->off].seen) {
+ /*
+ * The callee never exits. Its unwind_exit needs
+ * unwind_goto all the same, and the cpu predicts that it
+ * returns to what follows. Don't let the removal of dead
+ * code put something else there.
+ */
+ if (!aux_data[i + 1].seen) {
+ memcpy(insn + 1, &trap, sizeof(trap));
+ aux_data[i + 1].seen = env->pass_cnt;
+ }
+ continue;
+ }
+
if (!aux_data[i + 1].seen)
ja.off = insn->off;
else if (!aux_data[i + 1 + insn->off].seen)
@@ -1467,12 +1482,15 @@ static int jit_subprogs(struct bpf_verifier_env *env)
if (BPF_CLASS(insn->code) == BPF_STX &&
BPF_MODE(insn->code) == BPF_PROBE_ATOMIC)
num_exentries++;
+ if (bpf_is_unwind_goto(insn))
+ num_exentries++;
}
func[i]->aux->num_exentries = num_exentries;
func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable;
func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb;
func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
+ func[i]->aux->can_unwind = env->subprog_info[i].can_unwind;
func[i]->aux->token = prog->aux->token;
if (!i)
func[i]->aux->exception_boundary = env->seen_exception;
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index b3d250caaaa6..d7df6632a5c4 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -1046,6 +1046,9 @@ static bool states_equal(struct bpf_verifier_env *env,
if (old->in_sleepable != cur->in_sleepable)
return false;
+ if (old->unwound != cur->unwound)
+ return false;
+
if (!refsafe(old, cur, &env->idmap_scratch))
return false;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 849bc9136299..90bc7230afa8 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -5087,6 +5087,12 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
insns[i].imm = BPF_FUNC_tail_call;
/* fall-through */
}
+ /*
+ * BPF_JCOND is the same opcode as BPF_CALL_ARGS. The latter is for
+ * the interpreter, and a prog with unwind_goto is always JITed.
+ */
+ if (prog->jited && bpf_is_unwind_goto(&insns[i]))
+ continue;
if (code == (BPF_JMP | BPF_CALL) ||
code == (BPF_JMP | BPF_CALL_ARGS)) {
/* Restore the legacy xlated dump layout. */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 01434e1d23b5..e387320e196f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1769,6 +1769,7 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
return err;
dst_state->speculative = src->speculative;
dst_state->in_sleepable = src->in_sleepable;
+ dst_state->unwound = src->unwound;
dst_state->curframe = src->curframe;
dst_state->branches = src->branches;
dst_state->parent = src->parent;
@@ -3430,6 +3431,46 @@ static int next_callx_callee(struct bpf_verifier_env *env, int caller, int from)
return bit < end ? bit - caller * cnt : -1;
}
+/*
+ * 'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit.
+ * JITs don't emit a branch for it. unwind_exit finds the pad by its return
+ * address. So unwind_goto has to follow the call and nothing may jump to it.
+ */
+static int check_unwind_insns(struct bpf_verifier_env *env)
+{
+ struct bpf_insn *insn = env->prog->insnsi;
+ bool seen = false;
+ int i;
+
+ for (i = 0; i < env->prog->len; i++, insn++) {
+ if (bpf_is_unwind_exit(insn)) {
+ bpf_find_containing_subprog(env, i)->can_unwind = true;
+ seen = true;
+ }
+ if (!bpf_is_unwind_goto(insn))
+ continue;
+ seen = true;
+ if (!i || !(bpf_pseudo_call(insn - 1) || bpf_is_callx(insn - 1))) {
+ verbose(env, "unwind_goto at insn %d doesn't follow a call of a subprog\n",
+ i);
+ return -EINVAL;
+ }
+ if (bpf_is_jump_target(env, i)) {
+ verbose(env, "jump to unwind_goto at insn %d\n", i);
+ return -EINVAL;
+ }
+ }
+ if (!seen)
+ return 0;
+ if (!bpf_jit_supports_unwind()) {
+ verbose(env, "JIT does not support unwinding\n");
+ return -ENOTSUPP;
+ }
+ env->prog->jit_required = 1;
+ env->prog->aux->can_unwind = env->subprog_info[0].can_unwind;
+ return 0;
+}
+
/*
* Sort subprogs in topological order so that leaf subprogs come first and
* their callers come later. This is a DFS post-order traversal of the call
@@ -11584,6 +11625,19 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
}
+ if (env->subprog_info[subprog].can_unwind) {
+ struct bpf_verifier_state *branch;
+
+ if (!bpf_is_unwind_goto(insn + 1)) {
+ verbose(env, "no unwind_goto after the call of %s()\n", sub_name);
+ return -EINVAL;
+ }
+ branch = push_stack(env, *insn_idx + 1, *insn_idx, false);
+ if (IS_ERR(branch))
+ return PTR_ERR(branch);
+ branch->unwound = true;
+ }
+
if (env->subprog_info[subprog].might_throw) {
struct bpf_verifier_state *branch;
@@ -12057,6 +12111,19 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
}
caller = state->frame[state->curframe - 1];
+ if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx])) {
+ /* the helper or the kfunc that called it doesn't expect that */
+ if (callee->in_callback_fn) {
+ verbose(env, "callback cannot unwind\n");
+ return -EINVAL;
+ }
+ if (!bpf_is_unwind_goto(&env->prog->insnsi[callee->callsite + 1])) {
+ verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
+ callee->callsite);
+ return -EINVAL;
+ }
+ state->unwound = true;
+ }
if (callee->in_callback_fn) {
if (r0->type != SCALAR_VALUE) {
verbose(env, "R0 not a scalar value\n");
@@ -18759,6 +18826,23 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
return -EINVAL;
}
+ if (bpf_is_unwind_goto(insn)) {
+ if (!env->cur_state->unwound)
+ return 0;
+ env->cur_state->unwound = false;
+ /*
+ * unwind_exit changed its return address, so the return is
+ * mispredicted and what follows unwind_goto runs speculatively.
+ */
+ if (!env->bypass_spec_v1) {
+ err = sanitize_speculative_path(env, NULL, *insn_idx + 1, *insn_idx);
+ if (err)
+ return err;
+ }
+ *insn_idx += insn->off;
+ return 0;
+ }
+
if (opcode == BPF_JCOND) {
struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
int idx = *insn_idx;
@@ -21223,14 +21307,14 @@ static int check_jmp_fields(struct bpf_verifier_env *env, struct bpf_insn *insn)
return 0;
case BPF_EXIT:
if (BPF_SRC(insn->code) != BPF_K || insn->imm != 0 ||
- insn->src_reg != BPF_REG_0 || insn->dst_reg != BPF_REG_0 ||
+ insn->src_reg > BPF_UNWIND_EXIT || insn->dst_reg != BPF_REG_0 ||
class == BPF_JMP32) {
verbose(env, "BPF_EXIT uses reserved fields\n");
return -EINVAL;
}
return 0;
case BPF_JCOND:
- if (insn->code != (BPF_JMP | BPF_JCOND) || insn->src_reg != BPF_MAY_GOTO ||
+ if (insn->code != (BPF_JMP | BPF_JCOND) || insn->src_reg > BPF_UNWIND_GOTO ||
insn->dst_reg || insn->imm) {
verbose(env, "invalid may_goto imm %d\n", insn->imm);
return -EINVAL;
@@ -22436,6 +22520,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
struct bpf_prog_aux *aux = tgt_prog->aux;
bool tgt_changes_pkt_data;
bool tgt_might_sleep;
+ bool tgt_can_unwind;
if (bpf_prog_is_dev_bound(prog->aux) &&
!bpf_prog_dev_bound_match(prog, tgt_prog)) {
@@ -22464,6 +22549,16 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
prog_extension ? "Extension" : "Tracing");
return -EINVAL;
}
+ /* unwind_exit of the main prog is exit, unless it's an extension */
+ tgt_can_unwind = (subprog || tgt_prog->type == BPF_PROG_TYPE_EXT) &&
+ (aux->func ? aux->func[subprog]->aux->can_unwind
+ : aux->can_unwind);
+ /* unwind_exit would return to the trampoline that called the target */
+ if (tgt_can_unwind && !prog_extension &&
+ prog->expected_attach_type != BPF_TRACE_FENTRY) {
+ bpf_log(log, "Only fentry can attach to %s, which can unwind\n", tname);
+ return -EINVAL;
+ }
conservative = aux->func_info_aux[subprog].unreliable;
if (prog_extension) {
if (aux->func_info_aux[subprog].linkage != BTF_FUNC_GLOBAL ||
@@ -22511,6 +22606,12 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
"Extension program may sleep, while original does not\n");
return -EINVAL;
}
+ /* There is no unwind_goto after the calls of the original */
+ if (prog->aux->can_unwind && !tgt_can_unwind) {
+ bpf_log(log,
+ "Extension program can unwind, while original does not\n");
+ return -EINVAL;
+ }
}
if (!tgt_prog->jited) {
bpf_log(log, "Can attach to only JITed progs\n");
@@ -23845,6 +23946,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = check_unwind_insns(env);
+ if (ret < 0)
+ goto skip_full_check;
+
ret = bpf_compute_postorder(env);
if (ret < 0)
goto skip_full_check;
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 6f70db7515dd..f29b0aad440c 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -56,6 +56,19 @@
enum bpf_cond_pseudo_jmp {
BPF_MAY_GOTO = 0,
+ /*
+ * Follows a call of a bpf subprog. Taken when the subprog left with
+ * BPF_EXIT that has src_reg == BPF_UNWIND_EXIT.
+ */
+ BPF_UNWIND_GOTO = 1,
+};
+
+enum bpf_pseudo_exit {
+ /*
+ * When bpf_exit->src_reg == BPF_UNWIND_EXIT, it's the same as bpf_exit,
+ * except that BPF_UNWIND_GOTO after the call is taken.
+ */
+ BPF_UNWIND_EXIT = 1,
};
/* Register numbers */
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v4 bpf-next 1/6] bpf: Add unwind_goto and unwind_exit insns
2026-10-20 18:13 ` [PATCH v4 bpf-next 1/6] bpf: Add unwind_goto and unwind_exit insns Alexei Starovoitov
@ 2026-10-20 18:57 ` sashiko-bot
2026-10-20 20:04 ` Alexei Starovoitov
0 siblings, 1 reply; 13+ messages in thread
From: sashiko-bot @ 2026-10-20 18:57 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] bpf: stale insn_aux_data and unvisited unwind_goto handling in bpf_opt_hard_wire_dead_code_branches()
- [Medium] bpf: missing jit_requested and offload checks in check_unwind_insns()
--
commit d6e6c3108f32eea3c296a97e0da89132803476f1
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Add unwind_goto and unwind_exit insns
This patch introduces the BPF_UNWIND_GOTO and BPF_UNWIND_EXIT instructions
to support stack frame unwinding in BPF programs without extra register
overhead or stack walking. It adds verifier tracking, dead-code branch
fixups, and disassembler and dump support for the new instructions.
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index 409b9104c723e..f062ef7dd943d 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -658,6 +658,7 @@ void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env)
> {
> struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
> struct bpf_insn ja = BPF_JMP_IMM(BPF_JA, 0, 0, 0);
> + struct bpf_insn trap = BPF_JMP_IMM(BPF_JA, 0, 0, -1);
> struct bpf_insn *insn = env->prog->insnsi;
> const int insn_cnt = env->prog->len;
> int i;
>
> @@ -666,6 +667,20 @@ void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env)
> if (!bpf_insn_is_cond_jump(insn->code))
> continue;
>
> + if (bpf_is_unwind_goto(insn) && aux_data[i + 1 + insn->off].seen) {
> + /*
> + * The callee never exits. Its unwind_exit needs
> + * unwind_goto all the same, and the cpu predicts that it
> + * returns to what follows. Don't let the removal of dead
> + * code put something else there.
> + */
> + if (!aux_data[i + 1].seen) {
> + memcpy(insn + 1, &trap, sizeof(trap));
> + aux_data[i + 1].seen = env->pass_cnt;
> + }
> + continue;
> + }
[Severity: High]
When bpf_opt_hard_wire_dead_code_branches() overwrites insn + 1 with trap and
marks aux_data[i + 1].seen = env->pass_cnt, could stale flags in
aux_data[i + 1] corrupt the trap instruction or trigger a verifier bug?
Before do_check_main() and do_check_subprogs() run,
bpf_compute_live_registers() sets insn_aux[i + 1].zext_dst = true on the
static CFG if the original instruction at i + 1 defines a 32-bit subregister
whose upper half is live out:
kernel/bpf/liveness.c:bpf_compute_live_registers() {
...
for (i = 0; i < insn_cnt; ++i) {
int def32 = bpf_insn_def32(env->prog, &insns[i]);
u32 out = state[i].out;
u32 in = state[i].in;
insn_aux[i].live_regs_before = mask_lo(in) | mask_hi(in);
insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
}
}
In addition, when !env->bypass_spec_v1 is true, check_cond_jmp_op() calls
sanitize_speculative_path(env, NULL, *insn_idx + 1, *insn_idx) for
unwind_goto. That speculative walk leaves aux_data[i + 1].seen as 0 via
sanitize_mark_insn_seen(), while do_check_insn() can set zext_dst,
needs_zext, or arena_scalar on aux_data[i + 1].
Unlike keep_funcs_with_addr_taken() and sanitize_dead_code(), which clear
zext_dst = false when overwriting an unvisited instruction with trap,
bpf_opt_hard_wire_dead_code_branches() leaves aux_data[i + 1] flags intact.
Later, bpf_do_misc_fixups() will rewrite the BPF_JA trap instruction if
needs_zext or arena_scalar is set on aux_data[i + 1]:
kernel/bpf/fixups.c:bpf_do_misc_fixups() {
...
if (env->insn_aux_data[i + delta].needs_zext)
/* Convert BPF_CLASS(insn->code) == BPF_ALU64 to 32-bit ALU */
insn->code = BPF_ALU | BPF_OP(insn->code) | BPF_SRC(insn->code);
if (env->insn_aux_data[i + delta].arena_scalar) {
cnt = arena_scalar_access(insn, insn_buf);
...
}
And on architectures where bpf_jit_needs_zext() is true,
bpf_opt_subreg_zext_lo32_rnd_hi32() sees load_reg == -1 on the BPF_JA trap
instruction while aux[adj_idx].zext_dst is set, hitting verifier_bug_if():
kernel/bpf/fixups.c:bpf_opt_subreg_zext_lo32_rnd_hi32() {
...
if (verifier_bug_if(load_reg == -1, env,
"zext_dst is set, but no reg is defined"))
return -EFAULT;
}
Also, what happens if the call and unwind_goto at insn i are themselves in
dead code (!aux_data[i].seen), but jump to a shared landing pad that was
reached from another call (aux_data[i + 1 + insn->off].seen)? Because
aux_data[i].seen is not checked here, insn + 1 is overwritten with trap and
marked seen, so bpf_opt_remove_dead_code() removes the dead call and
unwind_goto while leaving an orphaned goto pc-1 in the program (and any
later live conditional jump with untaken target i + 1 can no longer see
!aux_data[i + 1].seen).
Should this check aux_data[i].seen before handling unwind_goto, and clear
zext_dst, needs_zext, and arena_scalar on aux_data[i + 1] when writing trap?
[ ... ]
> diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> index 849bc9136299d..90bc7230afa8a 100644
> --- a/kernel/bpf/syscall.c
> +++ b/kernel/bpf/syscall.c
> @@ -5087,6 +5087,12 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
> insns[i].imm = BPF_FUNC_tail_call;
> /* fall-through */
> }
> + /*
> + * BPF_JCOND is the same opcode as BPF_CALL_ARGS. The latter is for
> + * the interpreter, and a prog with unwind_goto is always JITed.
> + */
> + if (prog->jited && bpf_is_unwind_goto(&insns[i]))
> + continue;
> if (code == (BPF_JMP | BPF_CALL) ||
> code == (BPF_JMP | BPF_CALL_ARGS)) {
> /* Restore the legacy xlated dump layout. */
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 01434e1d23b59..e387320e196f9 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -3430,6 +3431,46 @@ static int next_callx_callee(struct bpf_verifier_env *env, int caller, int from)
> return bit < end ? bit - caller * cnt : -1;
> }
>
> +/*
> + * 'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit.
> + * JITs don't emit a branch for it. unwind_exit finds the pad by its return
> + * address. So unwind_goto has to follow the call and nothing may jump to it.
> + */
> +static int check_unwind_insns(struct bpf_verifier_env *env)
> +{
> + struct bpf_insn *insn = env->prog->insnsi;
> + bool seen = false;
> + int i;
> +
> + for (i = 0; i < env->prog->len; i++, insn++) {
> + if (bpf_is_unwind_exit(insn)) {
> + bpf_find_containing_subprog(env, i)->can_unwind = true;
> + seen = true;
> + }
> + if (!bpf_is_unwind_goto(insn))
> + continue;
> + seen = true;
> + if (!i || !(bpf_pseudo_call(insn - 1) || bpf_is_callx(insn - 1))) {
> + verbose(env, "unwind_goto at insn %d doesn't follow a call of a subprog\n",
> + i);
> + return -EINVAL;
> + }
> + if (bpf_is_jump_target(env, i)) {
> + verbose(env, "jump to unwind_goto at insn %d\n", i);
> + return -EINVAL;
> + }
> + }
> + if (!seen)
> + return 0;
> + if (!bpf_jit_supports_unwind()) {
> + verbose(env, "JIT does not support unwinding\n");
> + return -ENOTSUPP;
> + }
> + env->prog->jit_required = 1;
> + env->prog->aux->can_unwind = env->subprog_info[0].can_unwind;
> + return 0;
> +}
[Severity: Medium]
Should check_unwind_insns() also reject programs when
!env->prog->jit_requested or bpf_prog_is_offloaded(env->prog->aux)?
When CONFIG_BPF_JIT_ALWAYS_ON is enabled and a program with unwind_goto is
loaded with prog_ifindex set to an offload device (such as netdevsim),
bpf_fixup_call_args() skips bpf_jit_subprogs() because
!bpf_prog_is_offloaded(env->prog->aux) is false, and the
#ifndef CONFIG_BPF_JIT_ALWAYS_ON check on prog->jit_required is compiled out:
kernel/bpf/fixups.c:bpf_fixup_call_args() {
...
if (env->prog->jit_requested &&
!bpf_prog_is_offloaded(env->prog->aux)) {
err = bpf_jit_subprogs(env);
if (err == 0)
return 0;
if (err == -EFAULT || err == -EINTR)
return err;
}
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
if (prog->jit_required) {
verbose(env, "program requires BPF JIT compiler but it is not available\n");
return -EINVAL;
}
...
}
__bpf_prog_select_runtime() then takes the bpf_prog_offload_compile() branch
without checking jit_needed, leaving prog->jited as 0:
kernel/bpf/core.c:__bpf_prog_select_runtime() {
...
if (!bpf_prog_is_offloaded(fp->aux)) {
...
if (!fp->jited && jit_needed) {
*err = -ENOTSUPP;
return fp;
}
} else {
*err = bpf_prog_offload_compile(fp);
if (*err)
return fp;
}
}
Because prog->jited is 0 on that offloaded program, bpf_insn_prepare_dump()
skips the prog->jited && bpf_is_unwind_goto(&insns[i]) check and falls
through to the BPF_JMP | BPF_CALL_ARGS branch, where insns[i].src_reg ==
BPF_PSEUDO_CALL (both equal to 1) causes prepare_dump_pseudo_call() to
corrupt unwind_goto into BPF_JMP | BPF_CALL in xlated program dumps.
Similarly, when CONFIG_BPF_JIT_ALWAYS_ON is disabled and bpf_jit_enable is 0,
checking !env->prog->jit_requested in check_unwind_insns() (like
require_callx_jit() does) would reject unwinding before passes such as
bpf_compute_subprog_ret_regs() that gate on env->prog->jit_requested.
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020181320.1284798-1-alexei.starovoitov@gmail.com?part=1
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH v4 bpf-next 1/6] bpf: Add unwind_goto and unwind_exit insns
2026-10-20 18:57 ` sashiko-bot
@ 2026-10-20 20:04 ` Alexei Starovoitov
0 siblings, 0 replies; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 20:04 UTC (permalink / raw)
To: sashiko-reviews, bpf
On Tue, Oct 20, 2026 at 11:57 AM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
>> + if (!aux_data[i + 1].seen) {
>> + memcpy(insn + 1, &trap, sizeof(trap));
>> + aux_data[i + 1].seen = env->pass_cnt;
>> + }
>
> [Severity: High]
> When bpf_opt_hard_wire_dead_code_branches() overwrites insn + 1 with trap and
> marks aux_data[i + 1].seen = env->pass_cnt, could stale flags in
> aux_data[i + 1] corrupt the trap instruction or trigger a verifier bug?
No. That verifier_bug_if() needs bpf_jit_needs_zext(), and it's false
on x86-64 and arm64, the only JITs with bpf_jit_supports_unwind().
needs_zext and arena_scalar need an arena, and arena needs CAP_PERFMON.
Then bypass_spec_v1 is true, there is no speculative walk of i + 1,
and they cannot be set on an insn that wasn't seen.
> Also, what happens if the call and unwind_goto at insn i are themselves in
> dead code (!aux_data[i].seen), but jump to a shared landing pad that was
> reached from another call (aux_data[i + 1 + insn->off].seen)? Because
> aux_data[i].seen is not checked here, insn + 1 is overwritten with trap and
> marked seen, so bpf_opt_remove_dead_code() removes the dead call and
> unwind_goto while leaving an orphaned goto pc-1 in the program (and any
> later live conditional jump with untaken target i + 1 can no longer see
> !aux_data[i + 1].seen).
>
> Should this check aux_data[i].seen before handling unwind_goto, and clear
> zext_dst, needs_zext, and arena_scalar on aux_data[i + 1] when writing trap?
Not a bug. It's one extra 'goto pc-1' that nothing can reach. A live
cond jump to it is never taken, otherwise i + 1 would have been seen.
>> + if (!bpf_jit_supports_unwind()) {
>> + verbose(env, "JIT does not support unwinding\n");
>> + return -ENOTSUPP;
>> + }
>
> [Severity: Medium]
> Should check_unwind_insns() also reject programs when
> !env->prog->jit_requested or bpf_prog_is_offloaded(env->prog->aux)?
yeah. Only netdevsim gets there, nfp rejects the opcode in
nfp_verify_insn(), and nothing runs such a prog, so it's just a wrong
xlated dump. Will reject offloaded progs in check_unwind_insns() in v5.
> Similarly, when CONFIG_BPF_JIT_ALWAYS_ON is disabled and bpf_jit_enable is 0,
> checking !env->prog->jit_requested in check_unwind_insns() (like
> require_callx_jit() does) would reject unwinding before passes such as
> bpf_compute_subprog_ret_regs() that gate on env->prog->jit_requested.
Not a bug. jit_required is set, so bpf_fixup_call_args() rejects the
prog at the end of verification. It's only a matter of when.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
2026-10-20 18:13 ` [PATCH v4 bpf-next 1/6] bpf: Add unwind_goto and unwind_exit insns Alexei Starovoitov
@ 2026-10-20 18:13 ` Alexei Starovoitov
2026-10-20 18:55 ` bot+bpf-ci
2026-10-20 18:56 ` sashiko-bot
2026-10-20 18:13 ` [PATCH v4 bpf-next 3/6] bpf, arm64: " Alexei Starovoitov
` (4 subsequent siblings)
6 siblings, 2 replies; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 18:13 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor, yonghong.song
From: Alexei Starovoitov <ast@kernel.org>
bpf x86
call inner call inner
unwind_goto pad
w0 = w6 mov eax, ebx
r0 = 0 xor eax, eax
unwind_exit push rax
push rsi
mov rdi, [rbp + 8]
call bpf_unwind_target
mov [rbp + 8], rax
pop rsi
pop rax
leave
ret
unwind_goto is an extable entry and no code. unwind_exit looks its return
address up there and returns to the pad. r0 and r2 are what a subprog
returns, so rax and rsi are saved around the lookup.
When there is no entry, the kernel called the subprog as an async or an
exception callback, and unwind_exit is exit.
The caller of the main prog doesn't tell exit from unwind_exit, so there
it's exit. An extension is different. freplace is a jump, so its return
address is the one of the subprog that it replaced.
With private stack r9 is saved around a call, and 'pop r9' has to be done
on the way to the pad too. In such progs unwind_goto is:
push r9
call foo
pop r9 <- exit returns here
jmp 1f
pop r9 <- unwind_exit returns here
jmp pad
1:
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 101 +++++++++++++++++++++++++++++++++++-
1 file changed, 100 insertions(+), 1 deletion(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 72a2a0a89e03..d3ab36281144 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -15,6 +15,7 @@
#include <linux/memory.h>
#include <linux/sort.h>
#include <linux/execmem.h>
+#include <linux/extable.h>
#include <asm/extable.h>
#include <asm/ftrace.h>
#include <asm/set_memory.h>
@@ -2511,6 +2512,68 @@ static void adjust_cmp_imm(u32 *op, s32 *imm)
}
}
+/*
+ * There is no branch for unwind_goto, only an extable entry
+ * { .insn = ret_addr - 1, .fixup = target - ret_addr }
+ * where ret_addr is the return address of the call in front of it. unwind_exit
+ * replaces the return address of its frame with bpf_unwind_target() of it.
+ * ret_addr - 1, because what follows the call can have an entry of its own.
+ * No insn starts there, so a fault cannot hit the entry.
+ */
+static int add_unwind_exentry(struct bpf_prog *prog, u8 *image, u8 *rw_image, int *excnt,
+ int ret_addr, int target)
+{
+ struct exception_table_entry *ex;
+ s64 delta;
+
+ if (!prog->aux->extable)
+ return 0;
+ if (*excnt >= prog->aux->num_exentries) {
+ pr_err("unwind extable bug\n");
+ return -EFAULT;
+ }
+ ex = &prog->aux->extable[(*excnt)++];
+ delta = image + ret_addr - 1 - (u8 *)&ex->insn;
+ if (!is_simm32(delta)) {
+ pr_err("extable->insn doesn't fit into 32-bit\n");
+ return -EFAULT;
+ }
+ /* switch ex to rw buffer for writes */
+ ex = (void *)rw_image + ((void *)ex - (void *)image);
+ ex->insn = delta;
+ ex->fixup = target - ret_addr;
+ ex->data = EX_TYPE_BPF;
+ return 0;
+}
+
+static u64 bpf_unwind_target(u64 ret_addr)
+{
+ const struct exception_table_entry *e = search_bpf_extables(ret_addr - 1);
+
+ /*
+ * The verifier made sure that unwind_goto follows a call in a prog. When
+ * the kernel called an async or an exception callback, it's exit.
+ */
+ return e ? ret_addr + e->fixup : ret_addr;
+}
+
+/* unwind_exit: fix the return address of this frame. r0 and r2 are returned. */
+static int emit_unwind_ret_addr(u8 **pprog, u8 *ip)
+{
+ u8 *prog = *pprog, *start = prog;
+
+ EMIT2(0x50, 0x56); /* push rax; push rsi */
+ EMIT4(0x48, 0x8B, 0x7D, 0x08); /* mov rdi, [rbp + 8] */
+ ip += prog - start;
+ ip += x86_call_depth_emit_accounting(&prog, bpf_unwind_target, ip);
+ if (emit_call(&prog, bpf_unwind_target, ip))
+ return -EINVAL;
+ EMIT4(0x48, 0x89, 0x45, 0x08); /* mov [rbp + 8], rax */
+ EMIT2(0x5E, 0x58); /* pop rsi; pop rax */
+ *pprog = prog;
+ return 0;
+}
+
static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
{
@@ -3493,6 +3556,34 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
pop_r9(&prog);
break;
+ /* unwind_goto. The verifier expanded may_goto. */
+ case BPF_JMP | BPF_JCOND:
+ if (!priv_frame_ptr) {
+ err = add_unwind_exentry(bpf_prog, image, rw_image, &excnt,
+ addrs[i - 1], addrs[i + insn->off]);
+ if (err)
+ return err;
+ break;
+ }
+ /*
+ * 'pop r9' after the call has to be done on the way to
+ * the pad too:
+ * call foo
+ * pop r9 <- exit returns here
+ * jmp 1f
+ * pop r9 <- unwind_exit returns here
+ * jmp pad
+ * 1:
+ */
+ err = add_unwind_exentry(bpf_prog, image, rw_image, &excnt,
+ addrs[i - 1] - 2, addrs[i - 1] + 2);
+ if (err)
+ return err;
+ EMIT2(0xEB, 7);
+ pop_r9(&prog);
+ EMIT1_off32(0xE9, addrs[i + insn->off] - (addrs[i - 1] + 9));
+ break;
+
case BPF_JMP | BPF_TAIL_CALL:
if (imm32)
emit_bpf_tail_call_direct(bpf_prog,
@@ -3774,13 +3865,16 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
break;
case BPF_JMP | BPF_EXIT:
+ if (src_reg == BPF_UNWIND_EXIT && bpf_prog_can_unwind(bpf_prog) &&
+ emit_unwind_ret_addr(&prog, ip))
+ return -EINVAL;
if (seen_exit) {
jmp_offset = ctx->cleanup_addr - addrs[i];
goto emit_jmp;
}
seen_exit = true;
/* Update cleanup_addr */
- ctx->cleanup_addr = proglen;
+ ctx->cleanup_addr = proglen + (prog - temp);
if (bpf_prog_was_classic(bpf_prog) &&
!ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
@@ -5027,6 +5121,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
return true;
}
+bool bpf_jit_supports_unwind(void)
+{
+ return true;
+}
+
const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
{
return &x86_arg_abi;
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit
2026-10-20 18:13 ` [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit Alexei Starovoitov
@ 2026-10-20 18:55 ` bot+bpf-ci
2026-10-20 18:56 ` sashiko-bot
1 sibling, 0 replies; 13+ messages in thread
From: bot+bpf-ci @ 2026-10-20 18:55 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, yonghong.song, ast, andrii,
daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4832 bytes --]
> bpf, x86: Support unwind_goto and unwind_exit
TL;DR: An exception callback's unwind_exit may return into an unwind pad the
verifier never checked for that path (via freplace or tail call), possibly
yielding unsafe register state; JIT should skip it there.
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
[ ... ]
> +static u64 bpf_unwind_target(u64 ret_addr)
> +{
> + const struct exception_table_entry *e = search_bpf_extables(ret_addr - 1);
> +
> + /*
> + * The verifier made sure that unwind_goto follows a call in a prog. When
> + * the kernel called an async or an exception callback, it's exit.
> + */
> + return e ? ret_addr + e->fixup : ret_addr;
> +}
> +
> +/* unwind_exit: fix the return address of this frame. r0 and r2 are returned. */
> +static int emit_unwind_ret_addr(u8 **pprog, u8 *ip)
> +{
[ ... ]
> + EMIT4(0x48, 0x8B, 0x7D, 0x08); /* mov rdi, [rbp + 8] */
[ ... ]
> @@ -3774,13 +3865,16 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
> break;
>
> case BPF_JMP | BPF_EXIT:
> + if (src_reg == BPF_UNWIND_EXIT && bpf_prog_can_unwind(bpf_prog) &&
> + emit_unwind_ret_addr(&prog, ip))
> + return -EINVAL;
> if (seen_exit) {
Can an exception callback's unwind_exit land in a pad that the verifier
never explored for that path?
An exception callback is a subprog (func_idx != 0), so
bpf_prog_can_unwind() is true for it and do_jit() emits
emit_unwind_ret_addr() for its unwind_exit. The verifier treats
unwind_exit in an exception callback as a plain exit, and
prepare_func_exit() only handles curframe > 0. The comment in
bpf_unwind_target() relies on the same assumption:
/*
* The verifier made sure that unwind_goto follows a call in a prog. When
* the kernel called an async or an exception callback, it's exit.
*/
The exception callback does not run in a frame of its own, though.
bpf_throw() calls ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0)
with the frame of the first non-subprog prog on the stack, and
emit_prologue() does "mov rbp, rdx" for is_exception_cb. So
"mov rdi, [rbp + 8]" reads the return address of the main prog's frame,
and that address is not always in the kernel:
- Extension: freplace is a jmp from the replaced subprog's nop5
(bpf_arch_text_poke(..., BPF_MOD_JUMP, ..., prog->bpf_func)), so
[rbp + 8] of the extension's frame is the return address of "call b" in
the target prog.
- Tail call from a subprog: emit_bpf_tail_call_indirect() does not pop rbp
and jumps past push rbp / mov rbp, rsp, so the tail-called main prog runs
in the subprog's frame and [rbp + 8] is the return address of "call S" in
the caller.
If an unwind_goto follows that call, search_bpf_extables() finds its entry
and the exception callback returns into the pad.
One example with extensions: target prog P has
call a; unwind_goto pad; ...; call b; unwind_goto pad
where global a() can unwind and global b() cannot. The pad is seen via
a(), so bpf_opt_hard_wire_dead_code_branches() keeps the unwind_goto after
"call b" and the JIT emits an extable entry for it.
Extension E replaces b(). E's main has no unwind_exit, so
prog->aux->can_unwind (which is only subprog_info[0].can_unwind) is false
and the "Extension program can unwind, while original does not" check
passes. E calls bpf_throw(), and E's exception callback ends with
"r0 = 0; unwind_exit".
At runtime the callback returns into P's pad with the registers and
references that follow "call b". The verifier only checked the pad with
the state that follows "call a" unwinding, e.g. r6 as a map value pointer
there, while P could set r6 to an arbitrary scalar before "call b".
The tail call route does not need an extension. A subprog S with
unwind_exit (for example one that returns a pointer in r0 on its unwind
path) tail-calls T, T throws, and T's exception callback unwind_exit sends
it into P's pad after "call S" with r0 taken from the callback.
Should do_jit() skip emit_unwind_ret_addr() when bpf_prog->aux->exception_cb
is set, so the JIT matches the verifier's "exception callback unwind_exit
is exit" model? The alternative is for the verifier to account for the
exception callback's can_unwind.
The arm64 patch in this series, c1df16f0ed1d ("bpf, arm64: Support
unwind_goto and unwind_exit"), has the same structure, and none of the
later commits in the series change this.
> jmp_offset = ctx->cleanup_addr - addrs[i];
> goto emit_jmp;
> }
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39474515830
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit
2026-10-20 18:13 ` [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit Alexei Starovoitov
2026-10-20 18:55 ` bot+bpf-ci
@ 2026-10-20 18:56 ` sashiko-bot
1 sibling, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-10-20 18:56 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf, x86: emit_unwind_ret_addr() in exception_cb unwinds caller frame
--
commit c3cf97c5b42c930ee82a85d0896d91eed8f4f5f6
Author: Alexei Starovoitov <ast@kernel.org>
bpf, x86: Support unwind_goto and unwind_exit
This commit adds x86 JIT support for the BPF unwind_goto and unwind_exit
instructions using exception table entries and return address fixups.
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 72a2a0a89e033..d3ab36281144d 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
[ ... ]
> +static u64 bpf_unwind_target(u64 ret_addr)
> +{
> + const struct exception_table_entry *e = search_bpf_extables(ret_addr - 1);
> +
> + /*
> + * The verifier made sure that unwind_goto follows a call in a prog. When
> + * the kernel called an async or an exception callback, it's exit.
> + */
> + return e ? ret_addr + e->fixup : ret_addr;
> +}
> +
> +/* unwind_exit: fix the return address of this frame. r0 and r2 are returned. */
> +static int emit_unwind_ret_addr(u8 **pprog, u8 *ip)
> +{
> + u8 *prog = *pprog, *start = prog;
> +
> + EMIT2(0x50, 0x56); /* push rax; push rsi */
> + EMIT4(0x48, 0x8B, 0x7D, 0x08); /* mov rdi, [rbp + 8] */
> + ip += prog - start;
> + ip += x86_call_depth_emit_accounting(&prog, bpf_unwind_target, ip);
> + if (emit_call(&prog, bpf_unwind_target, ip))
> + return -EINVAL;
> + EMIT4(0x48, 0x89, 0x45, 0x08); /* mov [rbp + 8], rax */
> + EMIT2(0x5E, 0x58); /* pop rsi; pop rax */
> + *pprog = prog;
> + return 0;
> +}
[ ... ]
> @@ -3774,13 +3865,16 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
> break;
>
> case BPF_JMP | BPF_EXIT:
> + if (src_reg == BPF_UNWIND_EXIT && bpf_prog_can_unwind(bpf_prog) &&
> + emit_unwind_ret_addr(&prog, ip))
> + return -EINVAL;
[Severity: High]
Can emitting emit_unwind_ret_addr() inside an exception callback subprogram
cause unwind_exit to unwind the exception boundary's caller frame into an
unverified landing pad?
In include/linux/filter.h, bpf_prog_can_unwind() returns true for any
subprogram:
include/linux/filter.h:bpf_prog_can_unwind() {
return bpf_is_subprog(prog) || prog->type == BPF_PROG_TYPE_EXT;
}
Because an exception callback is a subprogram (bpf_prog->aux->func_idx != 0
with bpf_prog->aux->exception_cb set), bpf_prog_can_unwind(bpf_prog) is true
in do_jit(), while the verifier treats BPF_UNWIND_EXIT in an exception
callback as a normal frame-0 exit in process_bpf_exit_full() and ignores the
exception callback when setting prog->aux->can_unwind in check_unwind_insns().
The comment in bpf_unwind_target() assumes that when the kernel invokes an
exception callback, the return address at [rbp + 8] belongs to the kernel
caller and will not match any BPF extable entry. However, in
arch/x86/net/bpf_jit_comp.c, emit_prologue() resets rbp to the third argument
(ctx.bp of the exception boundary frame found by bpf_stack_walker()):
arch/x86/net/bpf_jit_comp.c:emit_prologue() {
if (is_exception_cb) {
EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
...
EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */
}
As a result, [rbp + 8] inside the exception callback is the return address of
the exception boundary frame (func[0]), not bpf_throw().
If the exception boundary frame was entered via a tail call from a subprogram
that was called before an unwind_goto instruction, or if an extension program
(BPF_PROG_TYPE_EXT) with can_unwind == false on func[0] replaces a global
subprogram followed by an unwind_goto instruction and then calls bpf_throw(),
[rbp + 8] points to the instruction after the subprogram call in the caller.
When the exception callback executes BPF_UNWIND_EXIT, bpf_unwind_target()
finds the caller's unwind_goto extable entry and redirects [rbp + 8] to the
landing pad with register and stack state that the verifier never checked for
that path.
Should bpf_prog_can_unwind() (or the check in do_jit()) exclude
prog->aux->exception_cb?
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020181320.1284798-1-alexei.starovoitov@gmail.com?part=2
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v4 bpf-next 3/6] bpf, arm64: Support unwind_goto and unwind_exit
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
2026-10-20 18:13 ` [PATCH v4 bpf-next 1/6] bpf: Add unwind_goto and unwind_exit insns Alexei Starovoitov
2026-10-20 18:13 ` [PATCH v4 bpf-next 2/6] bpf, x86: Support unwind_goto and unwind_exit Alexei Starovoitov
@ 2026-10-20 18:13 ` Alexei Starovoitov
2026-10-20 18:55 ` bot+bpf-ci
2026-10-20 18:13 ` [PATCH v4 bpf-next 4/6] selftests/bpf: Add tests for " Alexei Starovoitov
` (3 subsequent siblings)
6 siblings, 1 reply; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 18:13 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor, yonghong.song
From: Alexei Starovoitov <ast@kernel.org>
bpf arm64
call inner bl inner
mov x8, x0
unwind_goto pad
w0 = w6 mov w8, w19
Like on x86 unwind_goto is an extable entry and no code.
unwind_exit returns to the pad past 'mov x8, x0'. That's fine: r0 is x8,
and the epilogue of the callee copies it to x0 and leaves x8 as it is.
With pointer authentication LR in the frame record is signed. unwind_exit
doesn't sign the address of the pad. It looks up the offset of the pad from
its return address. Then it's the epilogue as it is, and the offset is
added to LR after autiasp:
stp x8, x1, [sp, #-16]!
ldr x0, [x29, #8]
bl bpf_unwind_fixup
mov x10, x0
ldp x8, x1, [sp], #16
...
ldp x29, x30, [sp], #16
mov x0, x8
autiasp
add x30, x30, x10
ret
So LR is authenticated like in exit, and the frame record stays as it is.
bpf_unwind_fixup() strips the signature for the lookup.
RET is not subject to BTI.
r0 and r2 are what a subprog returns, so x8 and x1 are saved around the
lookup.
Like on x86 the offset is 0 when the kernel called the subprog as
a callback, and unwind_exit of the main prog is exit, unless it's
an extension.
The private stack pointer is in a callee saved register. Unlike x86 there
is nothing to undo after the call.
Tested in qemu with -cpu max and -cpu cortex-a72, i.e. with and without
pointer authentication.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/arm64/net/bpf_jit_comp.c | 97 +++++++++++++++++++++++++++++++++--
1 file changed, 94 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 1d5f4cf88d33..b3efe0668e86 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -11,6 +11,7 @@
#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/cfi.h>
+#include <linux/extable.h>
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
@@ -1044,8 +1045,9 @@ static void __maybe_unused build_bhb_mitigation(struct jit_ctx *ctx)
}
}
-static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
+static void build_epilogue(struct jit_ctx *ctx, bool was_classic, bool unwind)
{
+ const u8 tmp = bpf2a64[TMP_REG_1];
const u8 r0 = bpf2a64[BPF_REG_0];
const u8 ptr = bpf2a64[TCCNT_PTR];
@@ -1073,6 +1075,10 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
emit(A64_AUTIASP, ctx);
+ /* tmp is the offset of the pad, see emit_unwind_exit() */
+ if (unwind)
+ emit(A64_ADD(1, A64_LR, A64_LR, tmp), ctx);
+
emit(A64_RET(A64_LR), ctx);
}
@@ -1221,6 +1227,76 @@ static int add_exception_handler(const struct bpf_insn *insn,
return 0;
}
+/*
+ * There is no branch for unwind_goto, only an extable entry
+ * { .insn = ret_addr - 1, .fixup = pad - ret_addr }
+ * where ret_addr is the return address of the call in front of it. unwind_exit
+ * adds .fixup to its LR.
+ * ret_addr - 1, because what follows the call can have an entry of its own.
+ * No insn starts there, so a fault cannot hit the entry.
+ * 'bl foo; mov x8, x0' is in front of unwind_goto. The pad doesn't miss the mov:
+ * r0 is x8 and the epilogue of the callee copies it to x0 and leaves x8 as it is.
+ */
+static int add_unwind_exentry(struct jit_ctx *ctx, int pad)
+{
+ int ret_addr = ctx->idx - 1;
+ struct exception_table_entry *ex;
+ off_t ins_offset;
+
+ if (!ctx->image)
+ /* First pass */
+ return 0;
+
+ if (!ctx->prog->aux->extable ||
+ WARN_ON_ONCE(ctx->exentry_idx >= ctx->prog->aux->num_exentries))
+ return -EINVAL;
+
+ ex = &ctx->prog->aux->extable[ctx->exentry_idx];
+ ins_offset = (long)&ctx->ro_image[ret_addr] - 1 - (long)&ex->insn;
+ if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN))
+ return -ERANGE;
+
+ /* switch ex to rw buffer for writing */
+ ex = (void *)ctx->image + ((void *)ex - (void *)ctx->ro_image);
+ ex->insn = ins_offset;
+ ex->fixup = (ctx->offset[pad] - ret_addr) * AARCH64_INSN_SIZE;
+ ex->type = EX_TYPE_BPF;
+
+ ctx->exentry_idx++;
+ return 0;
+}
+
+static long bpf_unwind_fixup(u64 lr)
+{
+ const struct exception_table_entry *e;
+
+ e = search_bpf_extables(ptrauth_strip_kernel_insn_pac(lr) - 1);
+ /*
+ * The verifier made sure that unwind_goto follows a call in a prog. When
+ * the kernel called an async or an exception callback, it's exit.
+ */
+ return e ? e->fixup : 0;
+}
+
+/*
+ * unwind_exit: return to the pad. LR in the frame record is signed. Instead of
+ * signing the address of the pad, let the epilogue authenticate LR as it is and
+ * add the offset of the pad to it. r0 and r2 are returned.
+ */
+static void emit_unwind_exit(struct jit_ctx *ctx)
+{
+ const u8 tmp = bpf2a64[TMP_REG_1];
+ const u8 r0 = bpf2a64[BPF_REG_0];
+ const u8 r2 = bpf2a64[BPF_REG_2];
+
+ emit(A64_PUSH(r0, r2, A64_SP), ctx);
+ emit(A64_LDR64I(A64_R(0), A64_FP, 8), ctx);
+ emit_call((u64)bpf_unwind_fixup, ctx);
+ emit(A64_MOV(1, tmp, A64_R(0)), ctx);
+ emit(A64_POP(r0, r2, A64_SP), ctx);
+ build_epilogue(ctx, false, true);
+}
+
static const struct bpf_jit_arg_abi arm64_arg_abi = {
.nr_arg_regs = 8,
.even_reg_align = true,
@@ -1806,8 +1882,18 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
if (emit_bpf_tail_call(ctx))
return -EFAULT;
break;
+ /* unwind_goto. The verifier expanded may_goto. */
+ case BPF_JMP | BPF_JCOND:
+ ret = add_unwind_exentry(ctx, i + 1 + off);
+ if (ret)
+ return ret;
+ break;
/* function return */
case BPF_JMP | BPF_EXIT:
+ if (bpf_is_unwind_exit(insn) && bpf_prog_can_unwind(ctx->prog)) {
+ emit_unwind_exit(ctx);
+ break;
+ }
/* Optimization: when last instruction is EXIT,
simply fallthrough to epilogue. */
if (i == ctx->prog->len - 1)
@@ -2312,7 +2398,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
goto out_off;
ctx.epilogue_offset = ctx.idx;
- build_epilogue(&ctx, was_classic);
+ build_epilogue(&ctx, was_classic, false);
build_plt(&ctx);
extable_align = __alignof__(struct exception_table_entry);
@@ -2369,7 +2455,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset))
goto out_free_hdr;
- build_epilogue(&ctx, was_classic);
+ build_epilogue(&ctx, was_classic, false);
build_plt(&ctx);
/* Extra pass to validate JITed code. */
@@ -2466,6 +2552,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
return true;
}
+bool bpf_jit_supports_unwind(void)
+{
+ return true;
+}
+
const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
{
return &arm64_arg_abi;
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v4 bpf-next 3/6] bpf, arm64: Support unwind_goto and unwind_exit
2026-10-20 18:13 ` [PATCH v4 bpf-next 3/6] bpf, arm64: " Alexei Starovoitov
@ 2026-10-20 18:55 ` bot+bpf-ci
0 siblings, 0 replies; 13+ messages in thread
From: bot+bpf-ci @ 2026-10-20 18:55 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, yonghong.song, ast, andrii,
daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5457 bytes --]
> bpf, arm64: Support unwind_goto and unwind_exit
TL;DR: In arm64 JIT, unwind_exit in an exception callback may return into an
unverified unwind_goto pad (via extension or tail call), possibly allowing
type confusion and out-of-bounds kernel memory access.
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 1d5f4cf88d33..b3efe0668e86 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
[ ... ]
> @@ -1221,6 +1227,76 @@ static int add_exception_handler(const struct bpf_insn *insn,
[ ... ]
> +static long bpf_unwind_fixup(u64 lr)
> +{
> + const struct exception_table_entry *e;
> +
> + e = search_bpf_extables(ptrauth_strip_kernel_insn_pac(lr) - 1);
> + /*
> + * The verifier made sure that unwind_goto follows a call in a prog. When
> + * the kernel called an async or an exception callback, it's exit.
> + */
> + return e ? e->fixup : 0;
> +}
[ ... ]
> @@ -1806,8 +1882,18 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
> /* function return */
> case BPF_JMP | BPF_EXIT:
> + if (bpf_is_unwind_exit(insn) && bpf_prog_can_unwind(ctx->prog)) {
> + emit_unwind_exit(ctx);
> + break;
> + }
> /* Optimization: when last instruction is EXIT,
> simply fallthrough to epilogue. */
> if (i == ctx->prog->len - 1)
Can this unwind_exit land in a pad that the verifier never checked, when it
is emitted for an exception callback?
The verifier treats unwind_exit in an exception callback as a plain exit
(process_bpf_exit_full() -> check_return_code()), so prepare_func_exit()
never checks for an unwind_goto at that call site. The JIT doesn't follow
that rule. An exception callback is func[i] with func_idx != 0, so
bpf_prog_can_unwind() returns true and emit_unwind_exit() is emitted. The
comment in bpf_unwind_fixup() ("When the kernel called an async or an
exception callback, it's exit") then relies on search_bpf_extables()
failing.
An exception callback has no frame record of its own. build_prologue()
does "mov x29, x2", and x2 is ctx.bp from bpf_stack_walker(), which is the
FP of the first func_idx == 0 frame. So the "ldr x0, [x29, #8]" in
emit_unwind_exit() reads the return address of the main prog that threw,
not an address in bpf_throw(). That address is in BPF code in at least two
supported setups:
1. Extension. Prog P calls global foo() as "call foo; unwind_goto pad".
Extension E replaces foo(). freplace is a B from foo's patch site
(bpf_arch_text_poke(), BPF_MOD_JUMP), so E's frame record LR is the
return address in P after "bl foo". E calls bpf_throw() and has
__exception_cb(cb), and cb ends with "r0 = 0; unwind_exit". Extensions
with exception callbacks are used in progs/exceptions_ext.c.
2. Tail call. P's subprog s tail-calls Q. On arm64 Q runs on s's frame
record (emit_bpf_tail_call() branches past the FP/LR push), so Q's
exception callback reads the return address in P after "bl s".
In both cases search_bpf_extables() finds the entry that
add_unwind_exentry() created for P's unwind_goto, and the callback's
epilogue adds .fixup to LR and returns into P's pad.
Is that path checked anywhere? In case 1, check_unwind_insns() sets
prog->aux->can_unwind from subprog_info[0] only, so an unwind_exit in E's
exception callback doesn't count. The bpf_check_attach_target() check
"Extension program can unwind, while original does not" then lets E replace
a foo() that cannot unwind, and check_func_call() never pushes the unwound
branch for that call. In case 2, check_helper_call() models a taken tail
call as a normal return only.
The unwind_goto and its extable entry survive whenever the pad is reached
from another path, for example another jump or another call whose callee
does unwind, because bpf_opt_hard_wire_dead_code_branches() keeps it:
if (bpf_is_unwind_goto(insn) && aux_data[i + 1 + insn->off].seen)
continue;
At runtime the pad then runs with r6-r9 and the stack as they were at the
"call foo" or "call s" site, plus r0 from the callback, while the verifier
only checked the pad with the states from its other predecessors. Could
that mismatch allow type confusion, for example a register that is a scalar
at the call site but a map value pointer on the verified path into the pad,
and from there an out-of-bounds kernel memory access?
The commit message statement "the offset is 0 when the kernel called the
subprog as a callback" and the comment in bpf_unwind_fixup() don't appear to
hold for exception callbacks.
Would it work to treat unwind_exit as exit when
ctx->prog->aux->exception_cb is set, for example:
if (bpf_is_unwind_exit(insn) && bpf_prog_can_unwind(ctx->prog) &&
!ctx->prog->aux->exception_cb)
or else to make the verifier and the attach check account for unwind_exit
in exception callbacks?
The x86 JIT in 9fa54d36ee99 ("bpf, x86: Support unwind_goto and
unwind_exit") uses the same condition,
"src_reg == BPF_UNWIND_EXIT && bpf_prog_can_unwind(bpf_prog)". No fix for
this was found in the rest of the series (c1df16f0ed1d..72e419bb0b07), which
only adds selftests, and those cover the exception callback case only for a
non-extension main prog.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39474515830
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v4 bpf-next 4/6] selftests/bpf: Add tests for unwind_goto and unwind_exit
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
` (2 preceding siblings ...)
2026-10-20 18:13 ` [PATCH v4 bpf-next 3/6] bpf, arm64: " Alexei Starovoitov
@ 2026-10-20 18:13 ` Alexei Starovoitov
2026-10-20 18:13 ` [PATCH v4 bpf-next 5/6] selftests/bpf: Add a C++ prog with exceptions Alexei Starovoitov
` (2 subsequent siblings)
6 siblings, 0 replies; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 18:13 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor, yonghong.song
From: Alexei Starovoitov <ast@kernel.org>
The tests spell the two insns out as raw insns, so they don't depend on
llvm version.
verifier_unwind:
- A static subprog, a global one and a callee of callx leave either way at
run time, through one and two frames. The verifier follows a static
subprog, so the tests derive the argument from skb->len or read it from
a map.
- r0 of unwind_exit, r6-r9 and the stack are there in the pad. r1-r5
are not readable.
- A pad is checked like any other code: it has to drop rcu_read_lock.
- How the callee left is a part of the verifier state: two states that
differ only in that are not pruned.
- The pad of a call of a global subprog is always checked.
- A callee that never returns, where unwind_goto stays and 'goto pc-1'
follows it, and one that never unwinds, where it goes away with the pad.
- What follows unwind_goto runs speculatively: with CAP_BPF only there is
nospec in front of an insn that isn't safe with r0 of unwind_exit.
- unwind_exit without unwind_goto after the call.
- unwind_goto that doesn't follow a call of a subprog or is a jump target.
- Reserved fields.
- A sync callback cannot unwind. What it calls can.
- unwind_exit of an async and of an exception callback is exit.
- Tail call and ld_abs in a subprog that can unwind.
- unwind_exit of the main prog is exit.
- What x86-64 and arm64 JITs emit. With KASAN x86-64 JIT wraps a load in
pushes, a call and pops, so the patterns start after the loads.
- Private stack. On x86-64 r9 has to be restored on the way to the pad.
The last one has to look at the return value of a prog that uses private
stack. raw_tp is such a prog. Its test_run doesn't take a packet, so let
test_loader run it like it runs syscall progs.
unwind_attach, with the caller run after every attach:
- fentry on a subprog that unwinds. fexit is refused.
- freplace of a subprog that unwinds by an extension that returns and by
one that unwinds. The caller goes on after the call or in the pad.
- An extension that unwinds cannot replace a subprog that doesn't.
Every check that the series adds was removed in turn to see that a test
fails.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/unwind_attach.c | 112 ++
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/unwind_attach.c | 43 +
.../selftests/bpf/progs/verifier_unwind.c | 1211 +++++++++++++++++
tools/testing/selftests/bpf/test_loader.c | 3 +-
5 files changed, 1370 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_attach.c
create mode 100644 tools/testing/selftests/bpf/progs/unwind_attach.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_unwind.c
diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_attach.c b/tools/testing/selftests/bpf/prog_tests/unwind_attach.c
new file mode 100644
index 000000000000..ada8b3273ccb
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/unwind_attach.c
@@ -0,0 +1,112 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "verifier_unwind.skel.h"
+#include "unwind_attach.skel.h"
+
+static char log_buf[16 * 1024];
+
+static int run(int fd)
+{
+ char data[64] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .data_in = data,
+ .data_size_in = sizeof(data),
+ .repeat = 1,
+ );
+
+ ASSERT_OK(bpf_prog_test_run_opts(fd, &opts), "test_run");
+ return opts.retval;
+}
+
+/*
+ * msg: why it cannot be loaded. Otherwise retval is what the target returns
+ * with the prog attached.
+ */
+static void attach(int tgt_fd, const char *func, const char *name, const char *msg, int retval)
+{
+ struct bpf_link *link = NULL;
+ struct unwind_attach *skel;
+ struct bpf_program *prog;
+ int err;
+
+ skel = unwind_attach__open();
+ if (!ASSERT_OK_PTR(skel, "unwind_attach__open"))
+ return;
+
+ prog = bpf_object__find_program_by_name(skel->obj, name);
+ if (!ASSERT_OK_PTR(prog, name))
+ goto out;
+ bpf_program__set_autoload(prog, true);
+ log_buf[0] = 0;
+ bpf_program__set_log_buf(prog, log_buf, sizeof(log_buf));
+ err = bpf_program__set_attach_target(prog, tgt_fd, func);
+ if (!ASSERT_OK(err, "set_attach_target"))
+ goto out;
+
+ err = unwind_attach__load(skel);
+ if (msg) {
+ ASSERT_ERR(err, name);
+ ASSERT_HAS_SUBSTR(log_buf, msg, name);
+ goto out;
+ }
+ if (!ASSERT_OK(err, name))
+ goto out;
+
+ link = bpf_program__attach(prog);
+ if (!ASSERT_OK_PTR(link, "attach"))
+ goto out;
+ ASSERT_EQ(run(tgt_fd), retval, "retval");
+ if (prog == skel->progs.fentry_unwind)
+ ASSERT_EQ(skel->bss->fentry_cnt, 1, "fentry_cnt");
+out:
+ bpf_link__destroy(link);
+ unwind_attach__destroy(skel);
+}
+
+void test_unwind_attach(void)
+{
+ struct verifier_unwind *tgt;
+ struct bpf_program *prog;
+ int tgt_fd, plain_fd, err;
+
+ if (!is_jit_enabled()) {
+ printf("%s:SKIP:JIT is disabled\n", __func__);
+ test__skip();
+ return;
+ }
+ tgt = verifier_unwind__open();
+ if (!ASSERT_OK_PTR(tgt, "verifier_unwind__open"))
+ return;
+ bpf_object__for_each_program(prog, tgt->obj)
+ bpf_program__set_autoload(prog, prog == tgt->progs.calls_global_unwinds ||
+ prog == tgt->progs.calls_global_never_unwinds);
+ err = verifier_unwind__load(tgt);
+ if (err == -ENOTSUPP) {
+ printf("%s:SKIP:JIT does not support unwinding\n", __func__);
+ test__skip();
+ goto out;
+ }
+ if (!ASSERT_OK(err, "verifier_unwind__load"))
+ goto out;
+ /*
+ * It calls global_unwinds(), which unwinds with r0 = 5, and returns r0 + 10
+ * from the pad.
+ */
+ tgt_fd = bpf_program__fd(tgt->progs.calls_global_unwinds);
+ plain_fd = bpf_program__fd(tgt->progs.calls_global_never_unwinds);
+ ASSERT_EQ(run(tgt_fd), 15, "retval");
+
+ attach(tgt_fd, "global_unwinds", "fentry_unwind", NULL, 15);
+ /* unwind_exit would return to the trampoline */
+ attach(tgt_fd, "global_unwinds", "fexit_unwind",
+ "Only fentry can attach to global_unwinds", 0);
+ /* An extension leaves either way, like what it replaced */
+ attach(tgt_fd, "global_unwinds", "freplace_returns", NULL, 9);
+ attach(tgt_fd, "global_unwinds", "freplace_unwinds", NULL, 13);
+ /* There is no unwind_goto after the call */
+ attach(plain_fd, "global_never_unwinds", "freplace_unwinds",
+ "Extension program can unwind, while original does not", 0);
+ ASSERT_EQ(run(tgt_fd), 15, "retval");
+out:
+ verifier_unwind__destroy(tgt);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 6df440d82675..04740ba37e91 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -130,6 +130,7 @@
#include "verifier_typedef.skel.h"
#include "verifier_uninit.skel.h"
#include "verifier_unpriv.skel.h"
+#include "verifier_unwind.skel.h"
#include "verifier_unpriv_perf.skel.h"
#include "verifier_value_adj_spill.skel.h"
#include "verifier_value.skel.h"
@@ -320,6 +321,7 @@ void test_verifier_tailcall_jit(void) { RUN(verifier_tailcall_jit); }
void test_verifier_typedef(void) { RUN(verifier_typedef); }
void test_verifier_uninit(void) { RUN(verifier_uninit); }
void test_verifier_unpriv(void) { RUN(verifier_unpriv); }
+void test_verifier_unwind(void) { RUN(verifier_unwind); }
void test_verifier_unpriv_perf(void) { RUN(verifier_unpriv_perf); }
void test_verifier_value_adj_spill(void) { RUN(verifier_value_adj_spill); }
void test_verifier_value(void) { RUN(verifier_value); }
diff --git a/tools/testing/selftests/bpf/progs/unwind_attach.c b/tools/testing/selftests/bpf/progs/unwind_attach.c
new file mode 100644
index 000000000000..df73c26c1ff4
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/unwind_attach.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/* The targets are global_unwinds() and global_never_unwinds() in verifier_unwind.c */
+
+int fentry_cnt;
+
+SEC("?fentry/global_unwinds")
+int BPF_PROG(fentry_unwind)
+{
+ fentry_cnt++;
+ return 0;
+}
+
+SEC("?fexit/global_unwinds")
+int BPF_PROG(fexit_unwind)
+{
+ return 0;
+}
+
+SEC("?freplace/global_unwinds")
+unsigned long freplace_returns(void)
+{
+ return 9;
+}
+
+SEC("?freplace/global_unwinds")
+__naked unsigned long freplace_unwinds(void)
+{
+ asm volatile (
+ "r0 = 3;"
+ ".8byte %[unwind_exit];"
+ :
+ : __imm_insn(unwind_exit,
+ BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+ );
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_unwind.c b/tools/testing/selftests/bpf/progs/verifier_unwind.c
new file mode 100644
index 000000000000..0a75cb8af67f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_unwind.c
@@ -0,0 +1,1211 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/*
+ * llvm lowers unwinding to two insns:
+ * invoke foo unwind pad call foo; unwind_goto pad
+ * resume unwind_exit
+ * The tests spell them out as raw insns, so they don't depend on llvm version.
+ * The argument of unwind_goto() is the number of insns to skip.
+ */
+#define unwind_goto(off) ".8byte %[insn_unwind_goto_" #off "];"
+#define unwind_exit ".8byte %[insn_unwind_exit];"
+
+#define __imm_unwind_goto(off) \
+ __imm_insn(insn_unwind_goto_##off, \
+ BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, off, 0))
+#define __imm_unwind_exit \
+ __imm_insn(insn_unwind_exit, \
+ BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+extern void bpf_throw(__u64 cookie) __ksym;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+} prog_array SEC(".maps");
+
+struct timer_elem {
+ struct bpf_timer t;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct timer_elem);
+} timer_map SEC(".maps");
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+unsigned long one = 1;
+
+/*
+ * The verifier follows a static subprog, so it knows which way it leaves when
+ * it knows the argument. To have both ways at run time the tests pass skb->len,
+ * which is 64, or skb->len - 64.
+ *
+ * r1 == 0: returns 7, otherwise unwinds with r0 = 5
+ */
+__naked __noinline __used
+static unsigned long may_unwind(void)
+{
+ asm volatile (
+ "if r1 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 5;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+/* owns nothing: its pad only carries on */
+__naked __noinline __used
+static unsigned long passes_on(void)
+{
+ asm volatile (
+ "call may_unwind;"
+ unwind_goto(2)
+ "r0 += 1;"
+ "exit;"
+ unwind_exit
+ :
+ : __imm_unwind_goto(2),
+ __imm_unwind_exit
+ );
+}
+
+__naked __noinline __used
+static unsigned long always_unwinds(void)
+{
+ asm volatile (
+ "r0 = 5;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+__naked __noinline __used
+static unsigned long never_unwinds(void)
+{
+ asm volatile (
+ "r0 = 7;"
+ "exit;"
+ );
+}
+
+/* Both can unwind. This one returns 7. */
+__naked __noinline __used
+unsigned long global_returns(void)
+{
+ asm volatile (
+ "r1 = %[zero] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 5;"
+ unwind_exit
+ :
+ : __imm_addr(zero),
+ __imm_unwind_exit
+ );
+}
+
+/* This one unwinds with r0 = 5 */
+__naked __noinline __used
+unsigned long global_unwinds(void)
+{
+ asm volatile (
+ "r1 = %[one] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 5;"
+ unwind_exit
+ :
+ : __imm_addr(one),
+ __imm_unwind_exit
+ );
+}
+
+SEC("tc")
+__description("unwind: the callee returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int callee_returns(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r1 -= 64;"
+ "call may_unwind;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: the callee unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int callee_unwinds(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call may_unwind;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: return through two frames")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(8)
+__naked int two_frames_return(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r1 -= 64;"
+ "call passes_on;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+/* unwind_exit returns r0 like exit does */
+SEC("tc")
+__description("unwind: unwind through two frames, r0 reaches the pad")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(15)
+__naked int two_frames_unwind(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call passes_on;"
+ unwind_goto(1)
+ "exit;"
+ "r0 += 10;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: r6-r9 and the stack are intact in the pad")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(1111)
+__naked int pad_sees_callee_saved(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r6 = 1;"
+ "r7 = 10;"
+ "r8 = 100;"
+ "r9 = 400;"
+ "*(u64 *)(r10 - 8) = 600;"
+ "call passes_on;"
+ unwind_goto(2)
+ "r0 = 0;"
+ "exit;"
+ "r0 = r6;"
+ "r0 += r7;"
+ "r0 += r8;"
+ "r0 += r9;"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(2)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: r1-r5 are not readable in the pad")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R3 !read_ok")
+__naked int pad_reads_r3(void)
+{
+ asm volatile (
+ "r3 = 0;"
+ "call always_unwinds;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = r3;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind_exit of the main prog is exit")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(3)
+__naked int main_prog_unwinds(void)
+{
+ asm volatile (
+ "r0 = 3;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind_exit of the main prog needs r0")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R0 !read_ok")
+__naked int main_prog_unwinds_no_r0(void)
+{
+ asm volatile (
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ : __clobber_all
+ );
+}
+
+/*
+ * What follows unwind_goto is dead code. unwind_exit still has to find the pad,
+ * and the cpu predicts that the callee returns: there is a trap for that.
+ */
+SEC("tc")
+__description("unwind: the callee never returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__xlated("0: call pc+4")
+__xlated("1: unwind_goto pc+1")
+__xlated("2: goto pc-1")
+__xlated("3: r0 = 42")
+__xlated("4: exit")
+__naked int callee_never_returns(void)
+{
+ asm volatile (
+ "call always_unwinds;"
+ unwind_goto(2)
+ "r0 = 1;"
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(2)
+ : __clobber_all
+ );
+}
+
+/* The pad is dead code and goes away together with unwind_goto */
+SEC("tc")
+__description("unwind: the callee never unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__xlated("0: call pc+1")
+__xlated("1: exit")
+__xlated("2: r0 = 7")
+__xlated("3: exit")
+__naked int callee_never_unwinds(void)
+{
+ asm volatile (
+ "call never_unwinds;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = *(u64 *)(r0 + 0);"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: the pad drops what the frame owns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int pad_unlocks(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "call %[bpf_rcu_read_lock];"
+ "r1 = r6;"
+ "call may_unwind;"
+ unwind_goto(3)
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 0;"
+ "exit;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock),
+ __imm_unwind_goto(3)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: the pad is checked like any other code")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("in main prog cannot be used inside bpf_rcu_read_lock-ed region")
+__naked int pad_does_not_unlock(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "call %[bpf_rcu_read_lock];"
+ "r1 = r6;"
+ "call may_unwind;"
+ unwind_goto(3)
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 0;"
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock),
+ __imm_unwind_goto(3)
+ : __clobber_all
+ );
+}
+
+/* Leaves either way with the same registers */
+__naked __noinline __used
+static unsigned long same_regs(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "if r1 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+/*
+ * After the call the two states differ only in the way the callee left.
+ * The second one must not be pruned, or nobody looks at the pad.
+ */
+SEC("tc")
+__description("unwind: the way the callee left is a part of the state")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R0 invalid mem access 'scalar'")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked int no_pruning_after_the_call(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call same_regs;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = *(u64 *)(r0 + 0);"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+/* Returns ctx, or unwinds with a number in r0 */
+__naked __noinline __used
+static unsigned long ctx_or_unwind(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[len]);"
+ "if r2 > 100 goto 1f;"
+ "r0 = r1;"
+ "exit;"
+ "1:"
+ "r0 = 0;"
+ unwind_exit
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_exit
+ );
+}
+
+/*
+ * unwind_exit changes its return address, so the cpu mispredicts the return and
+ * runs what follows unwind_goto with r0 of unwind_exit.
+ */
+SEC("socket")
+__description("unwind: what follows unwind_goto runs speculatively")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(50) /* 64 without the ethernet header */
+__success_unpriv __retval_unpriv(50)
+__caps_unpriv(CAP_BPF)
+#ifdef SPEC_V1
+__xlated_unpriv("unwind_goto pc+{{.*}}")
+__xlated_unpriv("nospec") /* inserted to prevent `R0 invalid mem access 'scalar'` */
+__xlated_unpriv("r0 = *(u32 *)(r0 +{{.*}})")
+#endif
+__naked int speculative_fall_through(void)
+{
+ asm volatile (
+ "call ctx_or_unwind;"
+ unwind_goto(2)
+ "r0 = *(u32 *)(r0 + %[len]);"
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(2)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: a global subprog returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int calls_global_returns(void)
+{
+ asm volatile (
+ "call global_returns;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: a global subprog unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(15)
+__naked int calls_global_unwinds(void)
+{
+ asm volatile (
+ "call global_unwinds;"
+ unwind_goto(1)
+ "exit;"
+ "r0 += 10;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+/* The verifier doesn't look inside */
+SEC("tc")
+__description("unwind: the pad of a call of a global subprog is always checked")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R0 invalid mem access 'scalar'")
+__naked int global_pad_is_checked(void)
+{
+ asm volatile (
+ "call global_returns;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = *(u64 *)(r0 + 0);"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+unsigned long global_never_unwinds(void)
+{
+ asm volatile (
+ "r0 = 7;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: a global subprog that cannot unwind")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int calls_global_never_unwinds(void)
+{
+ asm volatile (
+ "call global_never_unwinds;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: no unwind_goto after a call of a global subprog")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("no unwind_goto after the call of global_returns()")
+__naked int global_no_unwind_goto(void)
+{
+ asm volatile (
+ "call global_returns;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: no unwind_goto after a call of a static subprog")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("unwind_exit, but no unwind_goto after the call at insn 1")
+__naked int static_no_unwind_goto(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call may_unwind;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+/* The verifier sees that it doesn't get to unwind_exit */
+SEC("tc")
+__description("unwind: no unwind_goto is fine when the callee doesn't unwind")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int static_does_not_get_there(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "call may_unwind;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: callx, the callee returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int callx_returns(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r1 -= 64;"
+ "r6 = %[may_unwind] ll;"
+ "callx r6;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_addr(may_unwind),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: callx, the callee unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int callx_unwinds(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r6 = %[may_unwind] ll;"
+ "callx r6;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_addr(may_unwind),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind_goto after a plain insn")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("unwind_goto at insn 1 doesn't follow a call of a subprog")
+__naked int unwind_goto_after_mov(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ unwind_goto(0)
+ "exit;"
+ :
+ : __imm_unwind_goto(0)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind_goto is the first insn")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("unwind_goto at insn 0 doesn't follow a call of a subprog")
+__naked int unwind_goto_first(void)
+{
+ asm volatile (
+ unwind_goto(0)
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_unwind_goto(0)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind_goto after a call of a helper")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("unwind_goto at insn 1 doesn't follow a call of a subprog")
+__naked int unwind_goto_after_helper(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ unwind_goto(0)
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_unwind_goto(0)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind_goto after a call of a kfunc")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("unwind_goto at insn 1 doesn't follow a call of a subprog")
+__naked int unwind_goto_after_kfunc(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ unwind_goto(0)
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock),
+ __imm_unwind_goto(0)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: jump to unwind_goto")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("jump to unwind_goto at insn 3")
+__naked int jump_to_unwind_goto(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "if r1 != 0 goto 1f;"
+ "call never_unwinds;"
+ "1:"
+ unwind_goto(0)
+ "exit;"
+ :
+ : __imm_unwind_goto(0)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: reserved fields of unwind_goto")
+__failure __msg("invalid may_goto imm 1")
+__naked int unwind_goto_reserved(void)
+{
+ asm volatile (
+ "call never_unwinds;"
+ ".8byte %[bad];"
+ "exit;"
+ :
+ : __imm_insn(bad, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, 0, 1))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unknown kind of jcond")
+__failure __msg("invalid may_goto imm 0")
+__naked int jcond_unknown(void)
+{
+ asm volatile (
+ "call never_unwinds;"
+ ".8byte %[bad];"
+ "exit;"
+ :
+ : __imm_insn(bad, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO + 1, 0, 0))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unknown kind of exit")
+__failure __msg("BPF_EXIT uses reserved fields")
+__naked int exit_unknown(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ ".8byte %[bad];"
+ :
+ : __imm_insn(bad, BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT + 1, 0, 0))
+ : __clobber_all
+ );
+}
+
+/* r1 is the index: unwinds in the second iteration */
+__naked __noinline __used
+static unsigned long cb_unwinds(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "if r1 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+SEC("tc")
+__description("unwind: callback unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("callback cannot unwind")
+__naked int callback_unwinds(void)
+{
+ asm volatile (
+ "r1 = 3;"
+ "r2 = %[cb_unwinds];"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "exit;"
+ :
+ : __imm_ptr(cb_unwinds),
+ __imm(bpf_loop)
+ : __clobber_common
+ );
+}
+
+__naked __noinline __used
+static unsigned long async_cb_unwinds(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+/* The kernel calls it. It's the top frame like the main prog: unwind_exit is exit. */
+SEC("tc")
+__description("unwind: async callback unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int async_callback_unwinds(void *ctx)
+{
+ struct timer_elem *val;
+ int key = 0;
+
+ val = bpf_map_lookup_elem(&timer_map, &key);
+ if (!val)
+ return 0;
+
+ bpf_timer_init(&val->t, &timer_map, 0);
+ bpf_timer_set_callback(&val->t, (void *)async_cb_unwinds);
+ bpf_timer_start(&val->t, 0, 0);
+ return 0;
+}
+
+/* Not naked: the argument has to have a name in BTF */
+__noinline int exception_cb_unwinds(__u64 cookie)
+{
+ if (cookie)
+ asm volatile ("r0 = 7;" unwind_exit :: __imm_unwind_exit : "r0");
+ return 0;
+}
+
+SEC("tc")
+__description("unwind: exception callback unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__attribute__((btf_decl_tag("exception_callback:exception_cb_unwinds")))
+__success __retval(7)
+int exception_callback_unwinds(struct __sk_buff *skb)
+{
+ bpf_throw(skb->len);
+ return 0;
+}
+
+/* What it calls can unwind as long as it doesn't let it out */
+__naked __noinline __used
+static unsigned long cb_catches(void)
+{
+ asm volatile (
+ "call may_unwind;"
+ unwind_goto(0)
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_unwind_goto(0)
+ );
+}
+
+SEC("tc")
+__description("unwind: callback catches")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(3)
+__naked int callback_catches(void)
+{
+ asm volatile (
+ "r1 = 3;"
+ "r2 = %[cb_catches];"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "exit;"
+ :
+ : __imm_ptr(cb_catches),
+ __imm(bpf_loop)
+ : __clobber_common
+ );
+}
+
+/* Both leave the subprog like exit does */
+__naked __noinline __used
+static unsigned long does_tail_call(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r2 = %[prog_array] ll;"
+ "r3 = 0;"
+ "call %[bpf_tail_call];"
+ "r0 = 7;"
+ "if r6 == 64 goto 1f;"
+ unwind_exit
+ "1:"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm(bpf_tail_call),
+ __imm_addr(prog_array),
+ __imm_unwind_exit
+ );
+}
+
+SEC("tc")
+__description("unwind: tail call in a subprog that can unwind")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int tail_call_in_subprog(void)
+{
+ asm volatile (
+ "call does_tail_call;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+/* The packet is shorter than that: ld_abs leaves the subprog with r0 = 0 */
+__naked __noinline __used
+static unsigned long does_ld_abs(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "r0 = *(u8 *)skb[1000];"
+ "r0 = 5;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+SEC("tc")
+__description("unwind: ld_abs in a subprog that can unwind")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int ld_abs_in_subprog(void)
+{
+ asm volatile (
+ "call does_ld_abs;"
+ unwind_goto(2)
+ "r0 += 7;"
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(2)
+ : __clobber_all
+ );
+}
+
+/*
+ * JITs don't emit anything for unwind_goto. It's an extable entry, where
+ * unwind_exit looks its return address up.
+ * With KASAN x86-64 JIT wraps a load in pushes, a call and pops. 'r2 = 0x1234'
+ * is where the patterns start, so that they don't take that call for the one
+ * of the subprog.
+ */
+SEC("tc")
+__description("unwind: jit")
+__success __retval(42)
+__load_if_JITed()
+__arch_x86_64
+__jited(" movl $0x1234, %esi")
+__jited("...") /* call depth accounting */
+__jited(" callq 0x{{.*}}")
+__jited(" movl $0x7, %eax")
+__jited("...")
+__jited(" movl $0x5, %eax")
+__jited(" pushq %rax")
+__jited(" pushq %rsi")
+__jited(" movq 0x8(%rbp), %rdi")
+__jited("...") /* call depth accounting */
+__jited(" callq 0x{{.*}}")
+__jited(" movq %rax, 0x8(%rbp)")
+__jited(" popq %rsi")
+__jited(" popq %rax")
+__arch_arm64
+__jited(" mov x1, #0x1234")
+__jited(" bl {{.*}}")
+__jited(" mov x8, x0")
+__jited(" mov x8, #0x7")
+__jited("...")
+__jited(" mov x8, #0x5")
+__jited(" stp x8, x1, [sp, #-0x10]!")
+__jited(" ldr x0, [x29, #0x8]")
+__jited("...")
+__jited(" mov x10, x0")
+__jited(" ldp x8, x1, [sp], #0x10")
+__jited("...")
+__jited(" add x30, x30, x10")
+__jited(" ret")
+__naked int jit(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r2 = 0x1234;"
+ "call may_unwind;"
+ unwind_goto(2)
+ "r0 = 7;"
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_unwind_goto(2)
+ : __clobber_all
+ );
+}
+
+/* r1 == 0: returns 7, otherwise unwinds with r0 = 5 */
+__naked __noinline __used
+static unsigned long may_unwind_big_stack(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 200) = r1;"
+ "if r1 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 5;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+/*
+ * x86-64 keeps the pointer to the private stack in r9 and saves it around a call.
+ * The value that is read back in the pad is there only when r9 is restored.
+ */
+SEC("raw_tp")
+__description("unwind: private stack, the callee unwinds")
+__success __retval(47)
+__load_if_JITed()
+__arch_x86_64
+__jited(" movl $0x1234, %esi") /* past the load, see above */
+__jited(" pushq %r9")
+__jited("...") /* call depth accounting */
+__jited(" callq 0x{{.*}}")
+__jited(" popq %r9")
+__jited(" jmp L{{.*}}")
+__jited(" popq %r9")
+__jited(" jmp L{{.*}}")
+__arch_arm64
+__naked int private_stack_unwinds(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 200) = 42;"
+ "r1 = %[one] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "r2 = 0x1234;"
+ "call may_unwind_big_stack;"
+ unwind_goto(1)
+ "exit;"
+ "r1 = *(u64 *)(r10 - 200);"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_addr(one),
+ __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("raw_tp")
+__description("unwind: private stack, the callee returns")
+__success __retval(49)
+__load_if_JITed()
+__arch_x86_64
+__arch_arm64
+__naked int private_stack_returns(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 200) = 42;"
+ "r1 = %[zero] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "call may_unwind_big_stack;"
+ unwind_goto(3)
+ "r1 = *(u64 *)(r10 - 200);"
+ "r0 += r1;"
+ "exit;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(zero),
+ __imm_unwind_goto(3)
+ : __clobber_all
+ );
+}
+
+/*
+ * BTF FUNC records are not generated for kfuncs referenced
+ * from inline assembly. These records are necessary for
+ * libbpf to link the program. The function below is a hack
+ * to ensure that BTF FUNC records are generated.
+ */
+void __kfunc_btf_root(void)
+{
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c
index 18d47e925400..824e5912ebcb 100644
--- a/tools/testing/selftests/bpf/test_loader.c
+++ b/tools/testing/selftests/bpf/test_loader.c
@@ -1546,7 +1546,8 @@ void run_subtest(struct test_loader *tester,
}
err = do_prog_test_run(bpf_program__fd(tprog), &retval,
- bpf_program__type(tprog) == BPF_PROG_TYPE_SYSCALL ? true : false,
+ bpf_program__type(tprog) == BPF_PROG_TYPE_SYSCALL ||
+ bpf_program__type(tprog) == BPF_PROG_TYPE_RAW_TRACEPOINT,
spec->linear_sz);
if (!err && retval != subspec->retval && subspec->retval != POINTER_VALUE) {
PRINT_FAIL("Unexpected retval: %d != %d\n", retval, subspec->retval);
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v4 bpf-next 5/6] selftests/bpf: Add a C++ prog with exceptions
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
` (3 preceding siblings ...)
2026-10-20 18:13 ` [PATCH v4 bpf-next 4/6] selftests/bpf: Add tests for " Alexei Starovoitov
@ 2026-10-20 18:13 ` Alexei Starovoitov
2026-10-20 18:13 ` [PATCH v4 bpf-next 6/6] selftests/bpf: Test unwinding from R0:R2 subprog Alexei Starovoitov
2026-10-20 21:00 ` [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit patchwork-bot+netdevbpf
6 siblings, 0 replies; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 18:13 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor, yonghong.song
From: Alexei Starovoitov <ast@kernel.org>
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
static __noinline int middle(int len)
{
preempt_guard guard;
return inner(len) + 1;
}
SEC("tc") int catches(struct __sk_buff *skb)
{
try {
return middle(skb->len);
} catch (...) {
return 42;
}
}
The exception passes through two frames. The destructors call
bpf_rcu_read_unlock() and bpf_preempt_enable(). The verifier insists on
both, and the test checks that they ran in that order.
In the second prog nobody catches. The destructors run and the prog
returns 0.
All that is left of the C++ runtime is four one line functions.
__cxa_throw() calls _Unwind_RaiseException(), which llvm lowers.
progs/*.cpp are built like progs/*.rs: by clang only and without skeletons.
vmlinux.h and bpf_helpers.h are not C++, so the prog declares what it
needs.
clang that lowers unwinding [1] defines __BPF_FEATURE_UNWIND. Without it
the object has no progs and the test is skipped.
[1] https://github.com/llvm/llvm-project/pull/233649
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile.skel | 11 ++
.../selftests/bpf/prog_tests/unwind_cxx.c | 77 ++++++++++++
.../selftests/bpf/progs/unwind_cxx.cpp | 115 ++++++++++++++++++
3 files changed, 203 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
create mode 100644 tools/testing/selftests/bpf/progs/unwind_cxx.cpp
diff --git a/tools/testing/selftests/bpf/Makefile.skel b/tools/testing/selftests/bpf/Makefile.skel
index 06e297a17156..6570c48f92b0 100644
--- a/tools/testing/selftests/bpf/Makefile.skel
+++ b/tools/testing/selftests/bpf/Makefile.skel
@@ -153,4 +153,15 @@ $(RUST_OBJS): $(RDIR)/%.bpf.o: progs/%.rs $(RUST_CORE) | $(RDIR)
endif
endif
+# Programs in C++. No skeletons: gcc doesn't build them.
+ifeq ($(BPF_CC),$(CLANG))
+CXX_OBJS := $(patsubst progs/%.cpp,$(RDIR)/%.bpf.o,$(wildcard progs/*.cpp))
+BPF_OBJS += $(CXX_OBJS)
+
+$(CXX_OBJS): $(RDIR)/%.bpf.o: progs/%.cpp | $(RDIR)
+ $(call msg,$(BPF_CC_MSG),$(BINARY),$@)
+ $(Q)$(BPF_CC) -x c++ -g -Wall -Werror -fexceptions -funwind-tables -fno-rtti \
+ $(BPF_CC_FLAGS) -c $< -o $@ $(call skip_on_fail,BPF)
+endif
+
endif # BPF_CC
diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
new file mode 100644
index 000000000000..a049a807cd04
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <bpf/btf.h>
+
+/* The progs are under #ifdef __BPF_FEATURE_UNWIND */
+static bool llvm_lowers_unwinding(const char *file)
+{
+ struct btf *btf = btf__parse(file, NULL);
+ bool ret;
+
+ ret = btf && btf__find_by_name_kind(btf, "catches", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ret;
+}
+
+static void run(struct bpf_object *obj, const char *name, int len, int retval, int dtors)
+{
+ struct bpf_program *prog = bpf_object__find_program_by_name(obj, name);
+ struct bpf_map *map = bpf_object__find_map_by_name(obj, ".data.dtors");
+ char data[256] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .data_in = data,
+ .data_size_in = len,
+ .repeat = 1,
+ );
+ int key = 0, val = 0;
+
+ if (!ASSERT_OK_PTR(prog, name) || !ASSERT_OK_PTR(map, "dtors"))
+ return;
+ ASSERT_OK(bpf_map__update_elem(map, &key, sizeof(key), &val, sizeof(val), 0), "clear");
+ ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(prog), &opts), "test_run");
+ ASSERT_EQ(opts.retval, retval, "retval");
+ ASSERT_OK(bpf_map__lookup_elem(map, &key, sizeof(key), &val, sizeof(val), 0), "lookup");
+ ASSERT_EQ(val, dtors, "dtors");
+}
+
+void test_unwind_cxx(void)
+{
+ const char *file = "./unwind_cxx.bpf.o";
+ struct bpf_object *obj;
+ int err;
+
+ if (access(file, R_OK)) {
+ printf("%s:SKIP:gcc doesn't build %s\n", __func__, file);
+ test__skip();
+ return;
+ }
+ if (!is_jit_enabled()) {
+ printf("%s:SKIP:JIT is disabled\n", __func__);
+ test__skip();
+ return;
+ }
+ if (!llvm_lowers_unwinding(file)) {
+ printf("%s:SKIP:llvm doesn't lower unwinding\n", __func__);
+ test__skip();
+ return;
+ }
+ obj = bpf_object__open_file(file, NULL);
+ if (!ASSERT_OK_PTR(obj, "open"))
+ return;
+ err = bpf_object__load(obj);
+ if (err == -ENOTSUPP) {
+ printf("%s:SKIP:JIT does not support unwinding\n", __func__);
+ test__skip();
+ goto out;
+ }
+ if (!ASSERT_OK(err, "load"))
+ goto out;
+
+ /* inner's destructor, then middle's */
+ run(obj, "catches", 64, 65, 12);
+ run(obj, "catches", 200, 42, 12);
+ run(obj, "does_not_catch", 64, 65, 12);
+ run(obj, "does_not_catch", 200, 0, 12);
+out:
+ bpf_object__close(obj);
+}
diff --git a/tools/testing/selftests/bpf/progs/unwind_cxx.cpp b/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
new file mode 100644
index 000000000000..d46ee41fef85
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
@@ -0,0 +1,115 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ exceptions. llvm lowers unwinding to two insns:
+ * invoke foo unwind pad call foo; unwind_goto pad
+ * resume r0 = 0; unwind_exit
+ * so 'throw' is a jump or an exit, 'catch (...)' and the destructors that run on
+ * the way are ordinary blocks.
+ *
+ * vmlinux.h and bpf_helpers.h are not C++.
+ */
+#define SEC(name) __attribute__((section(name), used))
+#define __ksym __attribute__((section(".ksyms")))
+#define __noinline __attribute__((noinline))
+#define __always_inline inline __attribute__((always_inline))
+
+#ifdef __BPF_FEATURE_UNWIND
+
+struct __sk_buff {
+ unsigned int len;
+};
+
+extern "C" {
+void bpf_rcu_read_lock(void) noexcept __ksym;
+void bpf_rcu_read_unlock(void) noexcept __ksym;
+void bpf_preempt_disable(void) noexcept __ksym;
+void bpf_preempt_enable(void) noexcept __ksym;
+
+/* Starts an unwind. Not a kfunc: llvm lowers the call and it doesn't come back. */
+int _Unwind_RaiseException(void *exc);
+
+/* All that is left of the C++ runtime. Nobody looks at the exception object. */
+static char exc_obj[16];
+
+__always_inline void *__cxa_allocate_exception(unsigned long size) noexcept
+{
+ return exc_obj;
+}
+
+__always_inline void __cxa_throw(void *exc, void *tinfo, void (*dtor)(void *))
+{
+ _Unwind_RaiseException(exc);
+}
+
+__always_inline void *__cxa_begin_catch(void *exc) noexcept
+{
+ return exc;
+}
+
+__always_inline void __cxa_end_catch(void) noexcept
+{
+}
+}
+
+/* One decimal digit per destructor, in the order they ran */
+volatile int dtors SEC(".data.dtors");
+
+/* The verifier insists that both are undone on every path out of the prog */
+struct rcu_guard {
+ __always_inline rcu_guard()
+ {
+ bpf_rcu_read_lock();
+ }
+ __always_inline ~rcu_guard()
+ {
+ bpf_rcu_read_unlock();
+ dtors = dtors * 10 + 1;
+ }
+};
+
+struct preempt_guard {
+ __always_inline preempt_guard()
+ {
+ bpf_preempt_disable();
+ }
+ __always_inline ~preempt_guard()
+ {
+ bpf_preempt_enable();
+ dtors = dtors * 10 + 2;
+ }
+};
+
+static __noinline int inner(int len)
+{
+ rcu_guard guard;
+
+ if (len > 100)
+ throw 1;
+ return len;
+}
+
+static __noinline int middle(int len)
+{
+ preempt_guard guard;
+
+ return inner(len) + 1;
+}
+
+extern "C" SEC("tc") int catches(struct __sk_buff *skb)
+{
+ try {
+ return middle(skb->len);
+ } catch (...) {
+ return 42;
+ }
+}
+
+/* The exception leaves the prog: it returns 0 */
+extern "C" SEC("tc") int does_not_catch(struct __sk_buff *skb)
+{
+ return middle(skb->len);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v4 bpf-next 6/6] selftests/bpf: Test unwinding from R0:R2 subprog
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
` (4 preceding siblings ...)
2026-10-20 18:13 ` [PATCH v4 bpf-next 5/6] selftests/bpf: Add a C++ prog with exceptions Alexei Starovoitov
@ 2026-10-20 18:13 ` Alexei Starovoitov
2026-10-20 21:00 ` [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit patchwork-bot+netdevbpf
6 siblings, 0 replies; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-20 18:13 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor, yonghong.song
From: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Global subprograms returning a 16-byte scalar use R0:R2. The
verifier therefore checks both registers at every ordinary and unwind
exit.
LLVM lowers a resume to "r0 = 0; unwind_exit". If cleanup code does not
happen to initialize R2, the verifier rejects the unwind path with:
R2 !read_ok
Extend unwind_cxx with a global unsigned __int128-returning subprogram
whose cleanup only releases its RCU read lock. Load it separately so
the existing cases continue to run and the failure is attributed to
this case. Check both the normal aggregate return and caught exception.
The test passes once LLVM defines R2 before emitting unwind_exit.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/unwind_cxx.c | 24 ++++++++++++-
.../selftests/bpf/progs/unwind_cxx.cpp | 35 +++++++++++++++++++
2 files changed, 58 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
index a049a807cd04..9faba99133ad 100644
--- a/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
+++ b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
@@ -38,7 +38,8 @@ void test_unwind_cxx(void)
{
const char *file = "./unwind_cxx.bpf.o";
struct bpf_object *obj;
- int err;
+ struct bpf_program *prog;
+ int err = -EINVAL;
if (access(file, R_OK)) {
printf("%s:SKIP:gcc doesn't build %s\n", __func__, file);
@@ -58,6 +59,10 @@ void test_unwind_cxx(void)
obj = bpf_object__open_file(file, NULL);
if (!ASSERT_OK_PTR(obj, "open"))
return;
+ prog = bpf_object__find_program_by_name(obj, "catches_aggregate");
+ if (!ASSERT_OK_PTR(prog, "catches_aggregate"))
+ goto out;
+ bpf_program__set_autoload(prog, false);
err = bpf_object__load(obj);
if (err == -ENOTSUPP) {
printf("%s:SKIP:JIT does not support unwinding\n", __func__);
@@ -74,4 +79,21 @@ void test_unwind_cxx(void)
run(obj, "does_not_catch", 200, 0, 12);
out:
bpf_object__close(obj);
+ if (err)
+ return;
+
+ if (!test__start_subtest("aggregate_return"))
+ return;
+ obj = bpf_object__open_file(file, NULL);
+ if (!ASSERT_OK_PTR(obj, "open_aggregate"))
+ return;
+ bpf_object__for_each_program(prog, obj)
+ bpf_program__set_autoload(prog,
+ strcmp(bpf_program__name(prog), "catches_aggregate") == 0);
+ if (!ASSERT_OK(bpf_object__load(obj), "load_aggregate"))
+ goto out_aggregate;
+ run(obj, "catches_aggregate", 64, 71, 0);
+ run(obj, "catches_aggregate", 200, 42, 0);
+out_aggregate:
+ bpf_object__close(obj);
}
diff --git a/tools/testing/selftests/bpf/progs/unwind_cxx.cpp b/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
index d46ee41fef85..3a113ffea29c 100644
--- a/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
+++ b/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
@@ -67,6 +67,17 @@ struct rcu_guard {
}
};
+struct plain_rcu_guard {
+ __always_inline plain_rcu_guard()
+ {
+ bpf_rcu_read_lock();
+ }
+ __always_inline ~plain_rcu_guard()
+ {
+ bpf_rcu_read_unlock();
+ }
+};
+
struct preempt_guard {
__always_inline preempt_guard()
{
@@ -95,6 +106,18 @@ static __noinline int middle(int len)
return inner(len) + 1;
}
+using u128 = unsigned __int128;
+
+/* Global subprogs returning 16 bytes use R0:R2, including unwind exits. */
+extern "C" __noinline u128 aggregate_return(int len)
+{
+ plain_rcu_guard guard;
+
+ if (len > 100)
+ throw 1;
+ return (static_cast<u128>(len) << 64) | 7;
+}
+
extern "C" SEC("tc") int catches(struct __sk_buff *skb)
{
try {
@@ -110,6 +133,18 @@ extern "C" SEC("tc") int does_not_catch(struct __sk_buff *skb)
return middle(skb->len);
}
+extern "C" SEC("tc") int catches_aggregate(struct __sk_buff *skb)
+{
+ try {
+ u128 value = aggregate_return(skb->len);
+
+ return static_cast<unsigned long>(value) +
+ static_cast<unsigned long>(value >> 64);
+ } catch (...) {
+ return 42;
+ }
+}
+
#endif
char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit
2026-10-20 18:13 [PATCH v4 bpf-next 0/6] bpf: Unwinding with unwind_goto and unwind_exit Alexei Starovoitov
` (5 preceding siblings ...)
2026-10-20 18:13 ` [PATCH v4 bpf-next 6/6] selftests/bpf: Test unwinding from R0:R2 subprog Alexei Starovoitov
@ 2026-10-20 21:00 ` patchwork-bot+netdevbpf
6 siblings, 0 replies; 13+ messages in thread
From: patchwork-bot+netdevbpf @ 2026-10-20 21:00 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87, memxor, yonghong.song
Hello:
This series was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:
On Tue, 20 Oct 2026 18:13:14 +0000 you wrote:
> From: Alexei Starovoitov <ast@kernel.org>
>
> 1. What is the problem?
>
> A rust panic or a C++ exception has to run Drop impls or destructors of
> every frame it passes through. bpf_throw() discards the frames, so the
> verifier doesn't let a frame that owns a lock or a reference throw. In rust
> every index and every unwrap() can panic.
>
> [...]
Here is the summary with links:
- [v4,bpf-next,1/6] bpf: Add unwind_goto and unwind_exit insns
https://git.kernel.org/bpf/bpf-next/c/a2d274771321
- [v4,bpf-next,2/6] bpf, x86: Support unwind_goto and unwind_exit
https://git.kernel.org/bpf/bpf-next/c/cff94a3d78d8
- [v4,bpf-next,3/6] bpf, arm64: Support unwind_goto and unwind_exit
https://git.kernel.org/bpf/bpf-next/c/fc85015a5e86
- [v4,bpf-next,4/6] selftests/bpf: Add tests for unwind_goto and unwind_exit
https://git.kernel.org/bpf/bpf-next/c/86cec0897108
- [v4,bpf-next,5/6] selftests/bpf: Add a C++ prog with exceptions
https://git.kernel.org/bpf/bpf-next/c/0935acd1a8a7
- [v4,bpf-next,6/6] selftests/bpf: Test unwinding from R0:R2 subprog
https://git.kernel.org/bpf/bpf-next/c/55c34496ee99
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 13+ messages in thread