* [PATCH bpf-next 1/5] bpf: Support unwinding via r3
2026-10-18 1:02 [PATCH bpf-next 0/5] bpf: Unwinding via r3 Alexei Starovoitov
@ 2026-10-18 1:02 ` Alexei Starovoitov
2026-10-18 1:32 ` sashiko-bot
2026-10-18 1:48 ` bot+bpf-ci
2026-10-18 1:02 ` [PATCH bpf-next 2/5] bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable entry Alexei Starovoitov
` (4 subsequent siblings)
5 siblings, 2 replies; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-18 1:02 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Rust Drop, C++ destructors and __attribute__((cleanup)) in C have to run
when a panic or an exception passes through a frame.
llvm lowers unwinding to plain insns [1]. A function that can unwind says
in r3 how it returns, and its caller looks at r3 after the call:
ret r3 = 0; exit
invoke foo unwind pad call foo; if r3 != 0 goto pad
resume r0 = 0; r3 = 1; exit
call _Unwind_RaiseException r0 = 0; r3 = 1; exit
invoke _Unwind_RaiseException r0 = 0; goto pad
Landing pads are ordinary blocks reached by ordinary branches, so the
verifier checks them like any other code. A pad that forgets to drop
a lock or a reference is a leak on that path. There are no tables, no stack
walk, no uapi, no kfuncs and nothing to do for libbpf and other loaders.
The kernel only has to know which subprogs say something in r3.
llvm marks them with BTF decl_tag "unwind". For such a subprog:
- It has to exit with a scalar in r3, which reaches the caller like r0
does. After a call to a global subprog it's an unknown scalar.
- As a callback it has to exit with r3 == 0. The helper that calls it
doesn't look at r3.
- Tail call and ld_abs are rejected. Both return to the caller of the
subprog with whatever is in r3.
- Only fentry can attach to it. A trampoline that calls the subprog and
an extension that replaces it don't keep r3.
- The prog has to be JITed. The interpreter returns r0 only.
Without the tag r3 is not readable after a call, as before.
r3 steers control flow. A JIT that loses it between 'exit' and the insn
after the call would skip a pad that the verifier saw running. So JITs opt
in with bpf_jit_supports_unwind(). x86-64 does: r3 is rdx and the epilogue
doesn't touch it.
Rust
rust-bpf is switched to it [2]. The programs are built with
-C panic=unwind. The panic handler prints the message and calls
_Unwind_RaiseException(). The IR post-processing used to turn invoke into
call and drop the pads. Now it leaves invoke, landingpad and resume to
llvm.
A panic that cannot unwind, e.g. one that reaches an extern "C" function,
still ends in bpf_throw().
exc_cleanup.rs panics three frames deep, with preemption disabled in one
frame and rcu_read_lock held in another. All Drop impls run and the prog
returns 0.
scx_simple.rs is 2216 insns. 13 of 37 direct calls of subprogs and all 20
indirect calls are followed by the branch. In scx_cosmos.rs it's 21 of 61
and 20 of 20 in 3716 insns.
C++
Exceptions work too. This compiles, loads and runs:
struct rcu_guard {
rcu_guard() { bpf_rcu_read_lock(); }
~rcu_guard() { bpf_rcu_read_unlock(); }
};
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
SEC("tc") int prog(struct __sk_buff *skb)
{
try {
return inner(skb->len);
} catch (...) {
return 42;
}
}
It returns 64 for a packet of 64 bytes and 42 for a packet of 200 bytes.
The destructor runs in both cases, otherwise the verifier would reject
the prog. 'throw' is a jump to the pad with the destructor. 'catch' is
the branch after the call:
inner:
w6 = w1
call bpf_rcu_read_lock
if w6 s< 101 goto ret
goto pad
ret:
call bpf_rcu_read_unlock
w0 = w6
r3 = 0
exit
pad:
call bpf_rcu_read_unlock
r0 = 0
r3 = 1
exit
prog:
w6 = *(u32 *)(r1 + 0)
w1 = w6
call inner
if r3 != 0 goto catch
out:
w0 = w6
exit
catch:
w6 = 42
goto out
It needs -fexceptions -funwind-tables and four one line __cxa_*()
functions, where __cxa_throw() calls _Unwind_RaiseException().
Every pad is entered whatever was thrown, so llvm takes catch (...) only.
It's a selftest in the last patch.
[1] https://github.com/llvm/llvm-project/pull/233140
[2] https://github.com/4ast/rust-bpf/commit/b3225d1
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 5 +++
include/linux/bpf.h | 1 +
include/linux/bpf_verifier.h | 3 ++
include/linux/filter.h | 1 +
kernel/bpf/backtrack.c | 10 ++++--
kernel/bpf/core.c | 9 +++++
kernel/bpf/fixups.c | 1 +
kernel/bpf/liveness.c | 7 ++--
kernel/bpf/verifier.c | 66 ++++++++++++++++++++++++++++++++++--
9 files changed, 96 insertions(+), 7 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 11a4e4f87aac..16e847c615bc 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -5025,6 +5025,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
return true;
}
+bool bpf_jit_supports_unwind(void)
+{
+ return true;
+}
+
const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
{
return &x86_arg_abi;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 22947fa192e4..5f824c4242be 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1894,6 +1894,7 @@ struct bpf_prog_aux {
bool priv_stack_requested;
bool changes_pkt_data;
bool might_sleep;
+ bool ret_unwind;
bool kprobe_write_ctx;
struct {
s32 keyring_serial;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index e420d211f423..0f25454094a7 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -949,6 +949,7 @@ struct bpf_subprog_info {
s16 fastcall_stack_off;
bool has_tail_call: 1;
bool might_throw: 1;
+ bool ret_unwind: 1; /* says in r3 whether it returns or unwinds */
bool tail_call_reachable: 1;
bool has_ld_abs: 1;
bool is_cb: 1;
@@ -1219,6 +1220,8 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
return &env->subprog_info[subprog];
}
+#define BPF_REG_UNWIND BPF_REG_3
+
static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
{
return subprog_info(env, subprog)->ret_reg_pair;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 9339c6131f8f..15600d0e1b57 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1244,6 +1244,7 @@ bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_callx(void);
bool bpf_jit_supports_kfunc_ret_reg_pair(void);
+bool bpf_jit_supports_unwind(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 43d5740034c0..0f8dbacdf9ca 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -448,6 +448,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_clear_reg(bt, BPF_REG_0);
/* and if it does not set R2, main pass would catch it */
bt_clear_reg(bt, BPF_REG_2);
+ bt_clear_reg(bt, BPF_REG_UNWIND);
/* r1-r5 are invalidated after subprog call,
* so for global func call it shouldn't be set
* anymore
@@ -548,7 +549,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
- bool from_subprog_call, r0_precise, r2_precise;
+ bool from_subprog_call, r0_precise, r2_precise, r3_precise;
/* BPF_EXIT in subprog or callback always returns
* right after the call instruction, so by checking
@@ -565,6 +566,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
+ r3_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_UNWIND);
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
@@ -578,8 +580,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* might be a precise helper argument as well, so leave it set.
*/
bt_clear_reg(bt, BPF_REG_0);
- if (from_subprog_call)
+ if (from_subprog_call) {
bt_clear_reg(bt, BPF_REG_2);
+ bt_clear_reg(bt, BPF_REG_UNWIND);
+ }
if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
verifier_bug(env, "backtracking exit unexpected regs %x",
bt_reg_mask(bt));
@@ -593,6 +597,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_set_reg(bt, BPF_REG_0);
if (r2_precise)
bt_set_reg(bt, BPF_REG_2);
+ if (r3_precise)
+ bt_set_reg(bt, BPF_REG_UNWIND);
/* r6-r9 and stack slots will stay set in caller frame
* bitmasks until we return back from callee(s)
*/
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index e1216927373a..b16d74d2681a 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3322,6 +3322,15 @@ bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
return false;
}
+/*
+ * What a subprog leaves in r3 has to reach its caller: nothing from 'exit' to
+ * the insn after the call may clobber it.
+ */
+bool __weak bpf_jit_supports_unwind(void)
+{
+ return false;
+}
+
/*
* How this arch places a by-value kfunc argument, or NULL for one that has
* not opted in and so only takes an argument of a single eightbyte, which
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index fe59e3114d82..5e3e3cd378ee 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1383,6 +1383,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb;
func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
+ func[i]->aux->ret_unwind = env->subprog_info[i].ret_unwind;
func[i]->aux->token = prog->aux->token;
if (!i)
func[i]->aux->exception_boundary = env->seen_exception;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index e657e6d95408..51ccb8b080a9 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2482,7 +2482,7 @@ static int callee_subprog(struct bpf_verifier_env *env, const struct bpf_insn *i
static void compute_insn_live_regs(struct bpf_verifier_env *env,
struct bpf_insn *insn,
struct insn_live_regs *info,
- bool ret_reg_pair)
+ bool ret_reg_pair, bool ret_unwind)
{
struct bpf_call_summary cs;
const u8 class = BPF_CLASS(insn->code);
@@ -2616,6 +2616,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
case BPF_EXIT:
def = 0;
use = ret_reg_pair ? (r0 | r2) : r0;
+ if (ret_unwind)
+ use |= reg64_mask(BPF_REG_UNWIND);
break;
case BPF_CALL:
def = ALL_CALLER_SAVED_REGS;
@@ -2698,7 +2700,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
ret_reg_pair = bpf_ret_reg_pair(env, subprog);
for (i = start; i < end; ++i)
- compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+ compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair,
+ subprog && env->subprog_info[subprog].ret_unwind);
}
/* Forward pass: resolve stack access through FP-derived pointers */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 2c812a5f1da8..7f1aefafa0b9 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -440,7 +440,7 @@ static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
{
const struct btf *btf = env->prog->aux->btf;
- const struct btf_type *type;
+ const struct btf_type *type, *func;
int subprog;
u32 size;
@@ -454,8 +454,26 @@ static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
* refuses to replace a function returning more than 8 bytes.
*/
for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
+ struct bpf_subprog_info *sub = subprog_info(env, subprog);
+
type = subprog_ret_type(env, subprog);
- if (!type || btf_type_is_void(type))
+ if (!type)
+ continue;
+ func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
+ if (btf_find_next_decl_tag(btf, func, -1, "unwind", 0) > 0) {
+ if (!bpf_jit_supports_unwind()) {
+ verbose(env, "JIT does not support unwinding\n");
+ return -ENOTSUPP;
+ }
+ /* They return to the caller with whatever is in r3. */
+ if (sub->has_tail_call || sub->has_ld_abs) {
+ verbose(env, "tail call or ld_abs in a subprog that can unwind\n");
+ return -EINVAL;
+ }
+ sub->ret_unwind = true;
+ env->prog->jit_required = 1;
+ }
+ if (btf_type_is_void(type))
continue;
if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
"cannot size return type of subprog %d", subprog))
@@ -11471,6 +11489,9 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
}
+ if (env->subprog_info[subprog].ret_unwind)
+ mark_reg_unknown(env, caller->regs, BPF_REG_UNWIND);
+
if (env->subprog_info[subprog].might_throw) {
struct bpf_verifier_state *branch;
@@ -11923,13 +11944,14 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
struct bpf_verifier_state *state = env->cur_state, *prev_st;
struct bpf_func_state *caller, *callee;
struct bpf_reg_state *r0;
- bool in_callback_fn;
+ bool in_callback_fn, ret_unwind;
u32 i, nregs;
int err;
callee = state->frame[state->curframe];
r0 = &callee->regs[BPF_REG_0];
nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+ ret_unwind = subprog_info(env, callee->subprogno)->ret_unwind;
for (i = 0; i < nregs; i++) {
if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
continue;
@@ -11961,6 +11983,16 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
"At callback return", "R0");
return -EINVAL;
}
+ /* the helper that called it does not look at r3 */
+ if (ret_unwind) {
+ if (!bpf_register_is_null(&callee->regs[BPF_REG_UNWIND])) {
+ verbose(env, "callback cannot unwind\n");
+ return -EINVAL;
+ }
+ err = mark_chain_precision(env, BPF_REG_UNWIND);
+ if (err)
+ return err;
+ }
if (!bpf_calls_callback(env, callee->callsite)) {
verifier_bug(env, "in callback at %d, callsite %d !calls_callback",
*insn_idx, callee->callsite);
@@ -11979,6 +12011,22 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
caller->regs[regno] = callee->regs[regno];
bpf_diag_mod_end(env);
}
+ if (ret_unwind) {
+ struct bpf_reg_state *r3 = &callee->regs[BPF_REG_UNWIND];
+
+ err = check_reg_arg(env, BPF_REG_UNWIND, SRC_OP);
+ if (err)
+ return err;
+ if (r3->type != SCALAR_VALUE) {
+ verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+ BPF_REG_UNWIND, reg_type_str(env, r3->type));
+ return -EINVAL;
+ }
+ bpf_diag_mod_begin(env, &caller->regs[BPF_REG_UNWIND], r3,
+ BPF_DIAG_MOD_WRITE);
+ caller->regs[BPF_REG_UNWIND] = *r3;
+ bpf_diag_mod_end(env);
+ }
}
/* Invalidate callee-local dynptrs and their slices before the frame goes away. */
@@ -19335,6 +19383,12 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
u32 i, nregs;
int err;
+ if (subprog_info(env, subprog)->ret_unwind) {
+ err = check_global_ret_scalar_reg(env, BPF_REG_UNWIND);
+ if (err)
+ return err;
+ }
+
if (subprog_returns_void(env, subprog))
return 0;
@@ -22226,6 +22280,12 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
prog_extension ? "Extension" : "Tracing");
return -EINVAL;
}
+ /* Neither a trampoline that calls the subprog nor an extension keeps r3. */
+ if (aux->func && subprog && aux->func[subprog]->aux->ret_unwind &&
+ prog->expected_attach_type != BPF_TRACE_FENTRY) {
+ bpf_log(log, "Only fentry can attach to %s, which can unwind\n", tname);
+ return -EINVAL;
+ }
conservative = aux->func_info_aux[subprog].unreliable;
if (prog_extension) {
if (conservative) {
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next 1/5] bpf: Support unwinding via r3
2026-10-18 1:02 ` [PATCH bpf-next 1/5] bpf: Support unwinding " Alexei Starovoitov
@ 2026-10-18 1:32 ` sashiko-bot
2026-10-18 1:48 ` bot+bpf-ci
1 sibling, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-10-18 1:32 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: missing r3 unwind check for async and exception callbacks in process_bpf_exit_full()
--
commit 2eb5a83ebef2f014ae77160f4cea6d3a54b2f738
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Support unwinding via r3
This commit adds verifier and JIT support for subprograms annotated with the
BTF decl_tag "unwind", which use r3 (BPF_REG_UNWIND) at exit to indicate
whether they return normally or unwind to a caller's landing pad.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 2c812a5f1da87..7f1aefafa0b9b 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -11961,6 +11983,16 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
> "At callback return", "R0");
> return -EINVAL;
> }
> + /* the helper that called it does not look at r3 */
> + if (ret_unwind) {
> + if (!bpf_register_is_null(&callee->regs[BPF_REG_UNWIND])) {
> + verbose(env, "callback cannot unwind\n");
> + return -EINVAL;
> + }
> + err = mark_chain_precision(env, BPF_REG_UNWIND);
> + if (err)
> + return err;
> + }
> if (!bpf_calls_callback(env, callee->callsite)) {
> verifier_bug(env, "in callback at %d, callsite %d !calls_callback",
> *insn_idx, callee->callsite);
[Severity: Medium]
Does this check in prepare_func_exit() miss asynchronous callbacks and
exception callbacks that are annotated with the "unwind" BTF decl_tag?
When a subprogram is verified as an asynchronous callback (via push_async_cb()
with in_async_callback_fn = true) or as an exception callback (via
do_check_common() with in_exception_callback_fn = true), it runs at
curframe == 0.
When BPF_EXIT is reached in process_bpf_exit_full():
kernel/bpf/verifier.c:process_bpf_exit_full() {
...
if (env->cur_state->curframe) {
/* exit from nested function */
err = prepare_func_exit(env, &env->insn_idx);
...
}
...
if (cur_frame->subprogno &&
!cur_frame->in_async_callback_fn &&
!cur_frame->in_exception_callback_fn)
err = check_global_subprog_return_code(env);
else
err = check_return_code(env, BPF_REG_0, "R0");
}
Both prepare_func_exit() and check_global_subprog_return_code() are skipped,
and check_return_code() only validates R0:
kernel/bpf/verifier.c:check_return_code() {
...
if (frame->in_async_callback_fn) {
exit_ctx = "At async callback return";
range = frame->callback_ret_range;
goto enforce_retval;
}
}
Because llvm lowers unwind paths to r0 = 0; r3 = 1; exit, R0 satisfies the
callback return range while BPF_REG_UNWIND (R3) is never checked in
check_return_code(). Meanwhile, bpf_compute_live_registers() marks
BPF_REG_UNWIND as live at BPF_EXIT for all subprog > 0 with ret_unwind set.
Should check_return_code() (or process_bpf_exit_full()) also enforce that
BPF_REG_UNWIND is 0 when subprog_info(env, frame->subprogno)->ret_unwind is
set for async and exception callbacks?
[ ... ]
> @@ -19335,6 +19383,12 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
> u32 i, nregs;
> int err;
>
> + if (subprog_info(env, subprog)->ret_unwind) {
> + err = check_global_ret_scalar_reg(env, BPF_REG_UNWIND);
> + if (err)
> + return err;
> + }
> +
> if (subprog_returns_void(env, subprog))
> return 0;
>
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018010246.1643877-1-alexei.starovoitov@gmail.com?part=1
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next 1/5] bpf: Support unwinding via r3
2026-10-18 1:02 ` [PATCH bpf-next 1/5] bpf: Support unwinding " Alexei Starovoitov
2026-10-18 1:32 ` sashiko-bot
@ 2026-10-18 1:48 ` bot+bpf-ci
1 sibling, 0 replies; 13+ messages in thread
From: bot+bpf-ci @ 2026-10-18 1:48 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 6600 bytes --]
> bpf: Support unwinding via r3
TL;DR:
- Possible wrong r3 after fentry on an unwind subprog (rdx not restored), so
the verifier may miss a reachable pad
- "unwind" decl_tag matched by prefix, may reject unrelated programs
- Missing diagnostic write record for R3 (nit)
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 11a4e4f87aac..16e847c615bc 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -5025,6 +5025,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
>
> +bool bpf_jit_supports_unwind(void)
> +{
> + return true;
> +}
> +
> const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
Can the r3 that reaches the caller still be clobbered when fentry is attached
to a subprog that can unwind on x86-64?
prepare_func_exit() only checks that r3 is readable and a scalar at exit:
err = check_reg_arg(env, BPF_REG_UNWIND, SRC_OP);
...
caller->regs[BPF_REG_UNWIND] = *r3;
set_callee_state() copies r1-r5 into the callee no matter how many arguments
the callee declares. So a static unwind subprog that never writes r3 passes
verification whenever its caller set r3 before the call. The selftest 'static
subprog doesn't set r3' fails only because its caller leaves r3 uninitialized.
This means r3 does not behave 'like r0', which is NOT_INIT when the callee
starts.
An fentry-only trampoline is built with BPF_TRAMP_F_RESTORE_REGS, and it
restores only the argument slots in the target's func model:
if (flags & BPF_TRAMP_F_RESTORE_REGS)
restore_regs(&prog, pos_of_slot, nr_regs, regs_off);
For a static subprog whose BTF passes btf_check_subprog_call(),
func_info_aux[].unreliable stays false, so bpf_check_attach_target() sets
conservative == false and btf_distill_func_proto() uses the real BTF argument
count instead of 5. If that count is below 3, rdx is never restored. By then
rdx has been overwritten by __bpf_prog_enter*(), by the fentry prog, and by
__bpf_prog_exit*(), which receives run_ctx in rdx.
For example:
caller: r3 = 0; r1 = ...; call S; if r3 != 0 goto pad
S (static, BTF 'int S(int)', decl_tag "unwind"):
r0 = r1; exit
The verifier knows r3 == 0 after the call, marks it precise, and never
explores the pad from this path. If the pad is also reachable from another
path, bpf_opt_hard_wire_dead_code_branches() does not hard-wire the branch, so
it stays in the image. Once fentry is attached to S, S returns whatever rdx
the trampoline left behind, and the branch can be taken at run time with a
register state the verifier never checked for the pad.
The later extable conversion in this series (f2c9eb1fde9a "bpf, x86: Replace
'if r3 != 0 goto pad' after a call with an extable entry") does not change
this. An exit not preceded by 'r3 = imm' gets 'test rdx, rdx' plus the
extable lookup, and returns to the pad. No later commit in the range fixes
it.
The same trampoline behaviour already affects a static subprog that reads an
argument register it did not declare, so this widens an existing gap. Here,
though, r3 directly decides which landing pads the verifier treats as
reachable.
Would it work to mark r3 NOT_INIT on entry to an unwind subprog when r3 is not
a declared argument (as is done for r0), or to treat unwind subprogs as
conservative for fentry so that all five argument registers are restored?
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 63fff641d985..70b2087c7185 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -454,8 +454,26 @@ static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
> for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
> [ ... ]
> type = subprog_ret_type(env, subprog);
> - if (!type || btf_type_is_void(type))
> + if (!type)
> + continue;
> + func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
> + if (btf_find_next_decl_tag(btf, func, -1, "unwind", 0) > 0) {
Does this match only the tag "unwind"?
btf_find_next_decl_tag() does strncmp(__btf_name_by_offset(btf, t->name_off),
tag_key, strlen(tag_key)), so it matches by prefix. The other callers rely on
that because they pass key prefixes that end in a delimiter ("arg:",
"exception_callback:") and then parse the value. Here the key is the full tag
name "unwind" with no delimiter.
Any decl_tag on a subprog whose name starts with "unwind" (for example
"unwind_safe", "unwinding", or a later "unwind:<x>" form) sets
sub->ret_unwind, even though the commit message says only the tag "unwind"
means this. Such a subprog loaded fine before this patch. Now:
1) on every JIT other than x86-64, bpf_check() fails with -ENOTSUPP
("JIT does not support unwinding")
2) on x86-64, prepare_func_exit() and check_global_subprog_return_code()
require an initialized scalar in R3 at every exit, so a subprog that
never sets r3 (it is normally scratched by helper calls) is rejected
with "R3 !read_ok"
3) tail calls and ld_abs in it are rejected
4) fexit and freplace can no longer attach to it
5) jit_required is forced
Would checking the name of the found tag with strcmp() against "unwind" limit
this to the tag llvm emits?
[ ... ]
> @@ -11489,6 +11507,9 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
> [ ... ]
>
> + if (env->subprog_info[subprog].ret_unwind)
> + mark_reg_unknown(env, caller->regs, BPF_REG_UNWIND);
Should this R3 write be wrapped in bpf_diag_mod_begin()/bpf_diag_mod_end()
with BPF_DIAG_MOD_WRITE?
In the global subprog path, every return register write is wrapped that way:
R0, and R2 for a register pair. The new R3 write in prepare_func_exit() in
this patch is wrapped the same way.
Just before this, clear_caller_saved_regs() -> bpf_diag_record_caller_saved()
logs a BPF_DIAG_MOD_CALLER_SAVED scrub event for R3, and nothing is logged
afterwards. When a later error involves R3 after a call to a global subprog
that can unwind, the "Causal path" printed by diag_print_mod() ends with:
R3: call invalidated this caller-saved register; previous value was ...
It never shows that the call wrote an unknown scalar into R3, which can
mislead the user. R2 in the pair case gets the scrub event followed by a
write event, which is the intended pattern.
This only affects diagnostics, not verification results.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39085585797
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH bpf-next 2/5] bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable entry
2026-10-18 1:02 [PATCH bpf-next 0/5] bpf: Unwinding via r3 Alexei Starovoitov
2026-10-18 1:02 ` [PATCH bpf-next 1/5] bpf: Support unwinding " Alexei Starovoitov
@ 2026-10-18 1:02 ` Alexei Starovoitov
2026-10-18 1:28 ` sashiko-bot
2026-10-18 1:02 ` [PATCH bpf-next 3/5] bpf, arm64: Support unwinding via r3 Alexei Starovoitov
` (3 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-18 1:02 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A subprog that can unwind says so in r3 and its caller does
'if r3 != 0 goto pad' after the call. Keep that off the normal path.
bpf x86 before x86 after
call inner call inner call inner
if r3 != 0 goto pad test rdx, rdx mov eax, ebx
w0 = w6 jne pad
mov eax, ebx
at 'call foo; if r3 != 0 goto pad':
emit(call foo)
extable[n++] = { ret_addr -> pad }
at 'r3 = 1; exit':
my_ret_addr = bpf_unwind_target(my_ret_addr)
epilogue
bpf_unwind_target(ret_addr)
{
e = search_bpf_extables(ret_addr - 1);
return e ? ret_addr + e->fixup : ret_addr;
}
Every frame fixes only its own return address. There is no stack walk and
no ORC. The lookup is what the page fault handler does for bpf progs.
- 1, because what follows the call can have an entry of its own, and
no insn starts at that address, so a fault cannot hit the entry.
The verifier doesn't know about any of that.
When there is no entry the callee returns as usual and the branch does the
job. That's the case when:
- the branch is a jump target
- constant blinding rewrote the branch
- the prog uses private stack: 'pop r9' after the call has to run on the
way to the pad too
When 'exit' doesn't follow 'r3 = imm' the JIT emits 'test rdx, rdx; je'
around the lookup.
bpf_jit_unwind_site() and bpf_jit_unwind_exit() look at bpf insns only.
They're in core.c for other JITs to use.
I couldn't measure the branch in a microbenchmark [1]. In rust it follows
every indirect call and a third of direct calls of subprogs.
[1] https://lore.kernel.org/bpf/DM7FULA7L0GQ.R21A648X547X@gmail.com/
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 102 ++++++++++++++++++++++++++++++++++-
include/linux/bpf_verifier.h | 2 +-
include/linux/filter.h | 10 ++++
kernel/bpf/core.c | 44 +++++++++++++++
4 files changed, 155 insertions(+), 3 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 16e847c615bc..1b8c4708ba6f 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -15,6 +15,7 @@
#include <linux/memory.h>
#include <linux/sort.h>
#include <linux/execmem.h>
+#include <linux/extable.h>
#include <asm/extable.h>
#include <asm/ftrace.h>
#include <asm/set_memory.h>
@@ -2511,6 +2512,71 @@ static void adjust_cmp_imm(u32 *op, s32 *imm)
}
}
+/* See bpf_jit_unwind_site() */
+static bool unwind_site(struct bpf_verifier_env *env, struct bpf_prog *prog, int i)
+{
+ /* 'pop r9' after the call has to run on the way to the pad too */
+ return !prog->aux->priv_stack_ptr && bpf_jit_unwind_site(env, prog, i);
+}
+
+/* The branch after the call is not emitted. Remember where it goes. */
+static int add_unwind_exentry(struct bpf_prog *prog, u8 *image, u8 *rw_image, int *excnt,
+ int ret_addr, int pad)
+{
+ struct exception_table_entry *ex;
+ s64 delta;
+
+ if (!prog->aux->extable)
+ return 0;
+ if (*excnt >= prog->aux->num_exentries) {
+ pr_err("unwind extable bug\n");
+ return -EFAULT;
+ }
+ ex = &prog->aux->extable[(*excnt)++];
+ delta = image + ret_addr - 1 - (u8 *)&ex->insn;
+ if (!is_simm32(delta)) {
+ pr_err("extable->insn doesn't fit into 32-bit\n");
+ return -EFAULT;
+ }
+ /* switch ex to rw buffer for writes */
+ ex = (void *)rw_image + ((void *)ex - (void *)image);
+ ex->insn = delta;
+ ex->fixup = pad - ret_addr;
+ ex->data = EX_TYPE_BPF;
+ return 0;
+}
+
+static u64 bpf_unwind_target(u64 ret_addr)
+{
+ const struct exception_table_entry *e = search_bpf_extables(ret_addr - 1);
+
+ return e ? ret_addr + e->fixup : ret_addr;
+}
+
+/* r0, r2 and r3 are what the caller may look at */
+static int emit_unwind_ret_addr(u8 **pprog, u8 *ip, bool look_at_r3)
+{
+ u8 *prog = *pprog, *start = prog, *jmp = NULL;
+
+ if (look_at_r3) {
+ EMIT3(0x48, 0x85, 0xD2); /* test rdx, rdx */
+ EMIT2(X86_JE, 0);
+ jmp = prog;
+ }
+ EMIT3(0x50, 0x56, 0x52); /* push rax; push rsi; push rdx */
+ EMIT4(0x48, 0x8B, 0x7D, 0x08); /* mov rdi, [rbp + 8] */
+ ip += prog - start;
+ ip += x86_call_depth_emit_accounting(&prog, bpf_unwind_target, ip);
+ if (emit_call(&prog, bpf_unwind_target, ip))
+ return -EINVAL;
+ EMIT4(0x48, 0x89, 0x45, 0x08); /* mov [rbp + 8], rax */
+ EMIT3(0x5A, 0x5E, 0x58); /* pop rdx; pop rsi; pop rax */
+ if (jmp)
+ jmp[-1] = prog - jmp;
+ *pprog = prog;
+ return 0;
+}
+
static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
{
@@ -2518,7 +2584,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
struct bpf_insn *insn = bpf_prog->insnsi;
bool callee_regs_used[4] = {};
int insn_cnt = bpf_prog->len;
- bool seen_exit = false;
+ bool seen_exit = false, skip_branch = false;
u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
void __percpu *priv_frame_ptr = NULL;
u16 out_stack_arg_cnt, outgoing_rsp;
@@ -2655,6 +2721,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
int nops;
u32 jmp_op = BPF_OP(insn->code);
s32 cmp_imm;
+ enum bpf_unwind_exit unw;
+
+ if (skip_branch) {
+ skip_branch = false;
+ goto insn_done;
+ }
if (priv_frame_ptr) {
if (src_reg == BPF_REG_FP)
@@ -3460,6 +3532,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
ip += x86_call_depth_emit_accounting(&prog, func, ip);
if (emit_call(&prog, func, ip))
return -EINVAL;
+ if (unwind_site(env, bpf_prog, i - 1)) {
+ skip_branch = true;
+ err = add_unwind_exentry(bpf_prog, image, rw_image, &excnt,
+ proglen + (prog - temp),
+ addrs[i + 1 + insn[1].off]);
+ if (err)
+ return err;
+ break;
+ }
if (priv_frame_ptr)
pop_r9(&prog);
/*
@@ -3487,6 +3568,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
if (emit_indirect_call(&prog, insn->dst_reg, ip))
return -EINVAL;
+ if (unwind_site(env, bpf_prog, i - 1)) {
+ skip_branch = true;
+ err = add_unwind_exentry(bpf_prog, image, rw_image, &excnt,
+ proglen + (prog - temp),
+ addrs[i + 1 + insn[1].off]);
+ if (err)
+ return err;
+ break;
+ }
if (priv_frame_ptr)
pop_r9(&prog);
break;
@@ -3772,13 +3862,17 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
break;
case BPF_JMP | BPF_EXIT:
+ unw = bpf_jit_unwind_exit(env, bpf_prog, i - 1);
+ if (unw != BPF_EXIT_RETURNS &&
+ emit_unwind_ret_addr(&prog, ip, unw == BPF_EXIT_LOOK_AT_R3))
+ return -EINVAL;
if (seen_exit) {
jmp_offset = ctx->cleanup_addr - addrs[i];
goto emit_jmp;
}
seen_exit = true;
/* Update cleanup_addr */
- ctx->cleanup_addr = proglen;
+ ctx->cleanup_addr = proglen + (prog - temp);
if (bpf_prog_was_classic(bpf_prog) &&
!ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
@@ -3811,6 +3905,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
return -EINVAL;
}
+insn_done:
ilen = prog - temp;
if (ilen > BPF_MAX_INSN_SIZE) {
pr_err("bpf_jit: fatal insn size error\n");
@@ -4864,6 +4959,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
is_atomic_fetch_op(insn))
prog->aux->num_exentries++;
}
+ for (i = 0; i < prog->len; i++)
+ if (unwind_site(env, prog, i))
+ prog->aux->num_exentries++;
addrs = kvmalloc_objs(*addrs, prog->len + 1);
if (!addrs)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 0f25454094a7..e38e4b87d9d2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1326,7 +1326,7 @@ static inline void mark_jump_target(struct bpf_verifier_env *env, int idx)
env->insn_aux_data[idx].jump_target = true;
}
-static inline bool bpf_is_jump_target(struct bpf_verifier_env *env, int insn_idx)
+static inline bool bpf_is_jump_target(const struct bpf_verifier_env *env, int insn_idx)
{
return env->insn_aux_data[insn_idx].jump_target;
}
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 15600d0e1b57..6e4f07093c7c 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1245,6 +1245,16 @@ bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_callx(void);
bool bpf_jit_supports_kfunc_ret_reg_pair(void);
bool bpf_jit_supports_unwind(void);
+
+enum bpf_unwind_exit {
+ BPF_EXIT_RETURNS,
+ BPF_EXIT_UNWINDS,
+ BPF_EXIT_LOOK_AT_R3,
+};
+
+bool bpf_jit_unwind_site(const struct bpf_verifier_env *env, const struct bpf_prog *prog, int i);
+enum bpf_unwind_exit bpf_jit_unwind_exit(const struct bpf_verifier_env *env,
+ const struct bpf_prog *prog, int i);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index b16d74d2681a..14696555e8a5 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3331,6 +3331,50 @@ bool __weak bpf_jit_supports_unwind(void)
return false;
}
+/*
+ * A subprog that can unwind says so in r3 and its caller does
+ * 'if r3 != 0 goto pad' after the call. A JIT can keep that off the normal path.
+ * It doesn't emit the branch and adds an extable entry instead:
+ * insn = return address of the call - 1
+ * fixup = pad - return address of the call
+ * - 1, because what follows the call can have an entry of its own.
+ * An exit with r3 != 0 looks its own return address up and returns to the pad.
+ * When there is no entry it returns as usual and the branch is there.
+ *
+ * Is insn i of the prog such a call?
+ */
+bool bpf_jit_unwind_site(const struct bpf_verifier_env *env, const struct bpf_prog *prog, int i)
+{
+ const struct bpf_insn *insn = &prog->insnsi[i];
+
+ if (!env || i + 1 >= prog->len)
+ return false;
+ if (!bpf_pseudo_call(insn) && !bpf_is_callx(insn))
+ return false;
+ /*
+ * No need to know the callee: the verifier lets r3 be read after
+ * a call only when every callee seen there says something in it.
+ */
+ return insn[1].code == (BPF_JMP | BPF_JNE | BPF_K) && !insn[1].imm &&
+ insn[1].dst_reg == BPF_REG_UNWIND &&
+ !bpf_is_jump_target(env, prog->aux->subprog_start + i + 1);
+}
+
+/* What does 'exit' at insn i of the prog do? */
+enum bpf_unwind_exit bpf_jit_unwind_exit(const struct bpf_verifier_env *env,
+ const struct bpf_prog *prog, int i)
+{
+ const struct bpf_insn *insn = &prog->insnsi[i];
+
+ if (!env || !prog->aux->ret_unwind)
+ return BPF_EXIT_RETURNS;
+ if (!i || bpf_is_jump_target(env, prog->aux->subprog_start + i) ||
+ insn[-1].code != (BPF_ALU64 | BPF_MOV | BPF_K) ||
+ insn[-1].dst_reg != BPF_REG_UNWIND)
+ return BPF_EXIT_LOOK_AT_R3;
+ return insn[-1].imm ? BPF_EXIT_UNWINDS : BPF_EXIT_RETURNS;
+}
+
/*
* How this arch places a by-value kfunc argument, or NULL for one that has
* not opted in and so only takes an argument of a single eightbyte, which
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next 2/5] bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable entry
2026-10-18 1:02 ` [PATCH bpf-next 2/5] bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable entry Alexei Starovoitov
@ 2026-10-18 1:28 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-10-18 1:28 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] bpf: stale jump_target in bpf_jit_unwind_site() and bpf_jit_unwind_exit()
--
commit 5cf9f95ceda5d2950b3e2ae8af36899c1bc4137e
Author: Alexei Starovoitov <ast@kernel.org>
bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable entry
This commit optimizes unwinding subprogram calls on x86-64 by replacing the
post-call 'if r3 != 0 goto pad' branch with an exception table entry and
redirecting the caller's return address at subprogram exit when unwinding.
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index b16d74d2681a9..14696555e8a51 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -3331,6 +3331,50 @@ bool __weak bpf_jit_supports_unwind(void)
> return false;
> }
>
> +/*
> + * A subprog that can unwind says so in r3 and its caller does
> + * 'if r3 != 0 goto pad' after the call. A JIT can keep that off the normal path.
> + * It doesn't emit the branch and adds an extable entry instead:
> + * insn = return address of the call - 1
> + * fixup = pad - return address of the call
> + * - 1, because what follows the call can have an entry of its own.
> + * An exit with r3 != 0 looks its own return address up and returns to the pad.
> + * When there is no entry it returns as usual and the branch is there.
> + *
> + * Is insn i of the prog such a call?
> + */
> +bool bpf_jit_unwind_site(const struct bpf_verifier_env *env, const struct bpf_prog *prog, int i)
> +{
> + const struct bpf_insn *insn = &prog->insnsi[i];
> +
> + if (!env || i + 1 >= prog->len)
> + return false;
> + if (!bpf_pseudo_call(insn) && !bpf_is_callx(insn))
> + return false;
> + /*
> + * No need to know the callee: the verifier lets r3 be read after
> + * a call only when every callee seen there says something in it.
> + */
> + return insn[1].code == (BPF_JMP | BPF_JNE | BPF_K) && !insn[1].imm &&
> + insn[1].dst_reg == BPF_REG_UNWIND &&
> + !bpf_is_jump_target(env, prog->aux->subprog_start + i + 1);
> +}
> +
> +/* What does 'exit' at insn i of the prog do? */
> +enum bpf_unwind_exit bpf_jit_unwind_exit(const struct bpf_verifier_env *env,
> + const struct bpf_prog *prog, int i)
> +{
> + const struct bpf_insn *insn = &prog->insnsi[i];
> +
> + if (!env || !prog->aux->ret_unwind)
> + return BPF_EXIT_RETURNS;
> + if (!i || bpf_is_jump_target(env, prog->aux->subprog_start + i) ||
> + insn[-1].code != (BPF_ALU64 | BPF_MOV | BPF_K) ||
> + insn[-1].dst_reg != BPF_REG_UNWIND)
> + return BPF_EXIT_LOOK_AT_R3;
> + return insn[-1].imm ? BPF_EXIT_UNWINDS : BPF_EXIT_RETURNS;
> +}
[Severity: Critical]
Is env->insn_aux_data[].jump_target still accurate when bpf_jit_unwind_site()
and bpf_jit_unwind_exit() query bpf_is_jump_target() during JIT compilation?
env->insn_aux_data[].jump_target is populated once during bpf_check_cfg()
before main verification, and several post-verification passes in
kernel/bpf/fixups.c and kernel/bpf/const_fold.c modify instructions without
preserving jump_target on the resulting jump targets:
1. NOP removal in bpf_opt_remove_nops() -> verifier_remove_insns():
Dead conditional branches folded to 'ja +0' (by bpf_prune_dead_branches()
or bpf_opt_hard_wire_dead_code_branches()) and removed fastcall
spills/fills (by bpf_remove_fastcall_spills_fills()) are deleted unless
aux[i].indirect_target is set:
kernel/bpf/fixups.c:bpf_opt_remove_nops() {
...
if (aux[i].indirect_target)
continue;
err = verifier_remove_insns(env, i, 1);
}
verifier_remove_insns() redirects incoming jumps from off to the next
instruction, but overwrites aux_data[off] with aux_data[off + cnt]:
kernel/bpf/fixups.c:verifier_remove_insns() {
...
memmove(aux_data + off, aux_data + off + cnt,
sizeof(*aux_data) * (orig_prog_len - off - cnt));
}
If a jump targeted the removed NOP at k (aux_data[k].jump_target == true)
while k + 1 was only reached by fallthrough (aux_data[k + 1].jump_target ==
false), the instruction shifting into k inherits the incoming jump with
jump_target == false.
In an unwinding subprogram such as:
r3 = 1
if r1 == 0 goto L
r3 = 0
L: if r2 == 0 goto dead /* folded to ja +0 and removed */
exit
removing the NOP at L leaves exit at L with jump_target == false and
insn[-1] == 'r3 = 0'. bpf_jit_unwind_exit() then returns BPF_EXIT_RETURNS
instead of BPF_EXIT_LOOK_AT_R3, so do_jit() omits emit_unwind_ret_addr().
When the 'r1 == 0' branch is taken with r3 = 1, the subprogram returns to
the caller's normal post-call instruction (where do_jit() set
skip_branch = true and stripped 'if r3 != 0 goto pad') instead of the
landing pad, bypassing cleanup on the unwind path. Conversely, if
insn[-1] is 'r3 = 1', bpf_jit_unwind_exit() returns BPF_EXIT_UNWINDS and
unwinds unconditionally even when a jumping path had r3 = 0.
Similarly, if a jump-targeted NOP between a call and 'if r3 != 0 goto pad'
is removed, 'if r3 != 0 goto pad' shifts into insn[1] after the call with
jump_target == false, causing bpf_jit_unwind_site() to return true and
do_jit() to elide the branch via skip_branch = true even though another
jump targets it.
2. PROBE_MEM fixup in bpf_do_misc_fixups():
When bpf_arch_uaddress_limit() is non-zero (such as on x86-64), a
BPF_LDX | BPF_PROBE_MEM (or BPF_PROBE_MEMSX) load into r3 immediately
before exit is expanded via bpf_patch_insn_data():
kernel/bpf/fixups.c:bpf_do_misc_fixups() {
...
*patch++ = BPF_JMP_IMM(BPF_JLE, BPF_REG_AX, uaddress_limit >> 32, 2);
*patch++ = *insn;
*patch++ = BPF_JMP_IMM(BPF_JA, 0, 0, 1);
*patch++ = BPF_MOV64_IMM(insn->dst_reg, 0);
cnt = patch - insn_buf;
new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
}
The synthesized 'goto +1' jumps over 'r3 = 0' to exit without marking
exit as a jump target, and places 'r3 = 0' at insn[-1].
bpf_jit_unwind_exit() then sees jump_target == false and insn[-1] ==
'r3 = 0', returning BPF_EXIT_RETURNS instead of BPF_EXIT_LOOK_AT_R3.
3. Instruction patching in adjust_insn_aux_data():
When expanding an instruction in BPF_PATCH_KEEP_TARGET mode,
indirect_target is moved from data[off + cnt - 1] back to data[off],
whereas jump_target is left on data[off + cnt - 1] and cleared on
data[off]:
kernel/bpf/fixups.c:adjust_insn_aux_data() {
...
if (mode == BPF_PATCH_KEEP_TARGET && data[off + cnt - 1].indirect_target) {
data[off].indirect_target = 1;
data[off + cnt - 1].indirect_target = 0;
}
}
Should jump_target be maintained across verifier_remove_insns(),
adjust_insn_aux_data(), and fixup patches (or recomputed after all instruction
rewrites before JIT compilation)?
> /*
> * How this arch places a by-value kfunc argument, or NULL for one that has
> * not opted in and so only takes an argument of a single eightbyte, which
[ ... ]
>
endasi
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018010246.1643877-1-alexei.starovoitov@gmail.com?part=2
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH bpf-next 3/5] bpf, arm64: Support unwinding via r3
2026-10-18 1:02 [PATCH bpf-next 0/5] bpf: Unwinding via r3 Alexei Starovoitov
2026-10-18 1:02 ` [PATCH bpf-next 1/5] bpf: Support unwinding " Alexei Starovoitov
2026-10-18 1:02 ` [PATCH bpf-next 2/5] bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable entry Alexei Starovoitov
@ 2026-10-18 1:02 ` Alexei Starovoitov
2026-10-18 1:34 ` sashiko-bot
2026-10-18 1:02 ` [PATCH bpf-next 4/5] selftests/bpf: Add tests for " Alexei Starovoitov
` (2 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-18 1:02 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
r3 is x2. The epilogue doesn't touch it, so say so in
bpf_jit_supports_unwind().
Like x86, replace 'if r3 != 0 goto pad' after a call with an extable entry:
bpf arm64 before arm64 after
call inner bl inner bl inner
if r3 != 0 goto pad mov x8, x0 mov x8, x0
w0 = w6 cmp x2, #0 mov w8, w19
b.ne pad
mov w8, w19
The callee returns to the pad past 'mov x8, x0'. That's fine: r0 is x8 and
the epilogue of the callee copies it to x0 and leaves x8 as it is.
An exit with r3 != 0 fixes LR in its frame record. With pointer
authentication it's signed with SP at the entry of the prog, which is
FP + 16:
ldr x17, [x29, #8]
add x16, x29, #16
autia1716
mov x0, x17
bl bpf_unwind_target
mov x17, x0
add x16, x29, #16
pacia1716
str x17, [x29, #8]
So the lookup gets an authenticated address, and autiasp in the epilogue
checks the new one. autia1716 and pacia1716 are hints like paciasp and
autiasp: nops when the cpu doesn't have pointer authentication.
RET is not subject to BTI.
The private stack pointer is in a callee saved register. Unlike x86 there
is nothing to undo after the call, so it doesn't get in the way.
Tested in qemu with -cpu max and -cpu cortex-a72, i.e. with and without
pointer authentication.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/arm64/net/bpf_jit.h | 2 +
arch/arm64/net/bpf_jit_comp.c | 110 ++++++++++++++++++++++++++++++++++
2 files changed, 112 insertions(+)
diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h
index b2fe6e6dcf44..50029dd9baa6 100644
--- a/arch/arm64/net/bpf_jit.h
+++ b/arch/arm64/net/bpf_jit.h
@@ -316,6 +316,8 @@
#define A64_PACIASP A64_HINT(AARCH64_INSN_HINT_PACIASP)
#define A64_AUTIASP A64_HINT(AARCH64_INSN_HINT_AUTIASP)
+#define A64_PACIA1716 A64_HINT(AARCH64_INSN_HINT_PACIA_1716)
+#define A64_AUTIA1716 A64_HINT(AARCH64_INSN_HINT_AUTIA_1716)
/* BTI */
#define A64_BTI_C A64_HINT(AARCH64_INSN_HINT_BTIC)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 506c4201fec8..9b702930486a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -11,6 +11,7 @@
#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/cfi.h>
+#include <linux/extable.h>
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
@@ -1221,6 +1222,86 @@ static int add_exception_handler(const struct bpf_insn *insn,
return 0;
}
+/*
+ * See bpf_jit_unwind_site(). The branch after the call that was just emitted is
+ * not emitted. Remember where it goes. The pad is an index into ctx->offset.
+ */
+static int add_unwind_exentry(struct jit_ctx *ctx, int pad)
+{
+ struct exception_table_entry *ex;
+ off_t ins_offset;
+
+ if (!ctx->image)
+ /* First pass */
+ return 0;
+
+ if (!ctx->prog->aux->extable ||
+ WARN_ON_ONCE(ctx->exentry_idx >= ctx->prog->aux->num_exentries))
+ return -EINVAL;
+
+ ex = &ctx->prog->aux->extable[ctx->exentry_idx];
+ ins_offset = (long)&ctx->ro_image[ctx->idx] - 1 - (long)&ex->insn;
+ if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN))
+ return -ERANGE;
+
+ /* switch ex to rw buffer for writing */
+ ex = (void *)ctx->image + ((void *)ex - (void *)ctx->ro_image);
+ ex->insn = ins_offset;
+ ex->fixup = (ctx->offset[pad] - ctx->idx) * AARCH64_INSN_SIZE;
+ ex->type = EX_TYPE_BPF;
+
+ ctx->exentry_idx++;
+ return 0;
+}
+
+static u64 bpf_unwind_target(u64 ret_addr)
+{
+ const struct exception_table_entry *e = search_bpf_extables(ret_addr - 1);
+
+ return e ? ret_addr + e->fixup : ret_addr;
+}
+
+/*
+ * Make an exit with r3 != 0 return to the pad.
+ * LR in the frame record is signed with SP at the entry of the prog, which is
+ * FP + 16. The x16/x17 forms are hints like PACIASP: nops without pointer auth.
+ * r0, r2 and r3 are what the caller may look at.
+ */
+static void emit_unwind_ret_addr(struct jit_ctx *ctx, bool look_at_r3)
+{
+ const bool ptr_auth = IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL);
+ const u8 r0 = bpf2a64[BPF_REG_0];
+ const u8 r2 = bpf2a64[BPF_REG_2];
+ const u8 r3 = bpf2a64[BPF_REG_3];
+ __le32 *branch = ctx->image + ctx->idx;
+
+ if (look_at_r3)
+ emit(A64_NOP, ctx);
+
+ emit(A64_PUSH(r0, r2, A64_SP), ctx);
+ emit(A64_PUSH(r3, A64_ZR, A64_SP), ctx);
+
+ emit(A64_LDR64I(A64_R(17), A64_FP, 8), ctx);
+ if (ptr_auth) {
+ emit(A64_ADD_I(1, A64_R(16), A64_FP, 16), ctx);
+ emit(A64_AUTIA1716, ctx);
+ }
+ emit(A64_MOV(1, A64_R(0), A64_R(17)), ctx);
+ emit_call((u64)bpf_unwind_target, ctx);
+ emit(A64_MOV(1, A64_R(17), A64_R(0)), ctx);
+ if (ptr_auth) {
+ emit(A64_ADD_I(1, A64_R(16), A64_FP, 16), ctx);
+ emit(A64_PACIA1716, ctx);
+ }
+ emit(A64_STR64I(A64_R(17), A64_FP, 8), ctx);
+
+ emit(A64_POP(r3, A64_ZR, A64_SP), ctx);
+ emit(A64_POP(r0, r2, A64_SP), ctx);
+
+ if (look_at_r3 && ctx->image)
+ *branch = cpu_to_le32(A64_CBZ(1, r3, &ctx->image[ctx->idx] - branch));
+}
+
static const struct bpf_jit_arg_abi arm64_arg_abi = {
.nr_arg_regs = 8,
.even_reg_align = true,
@@ -1378,6 +1459,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
int off_adj;
int ret;
bool sign_extend;
+ enum bpf_unwind_exit unw;
if (bpf_insn_is_indirect_target(env, ctx->prog, i))
emit_bti(A64_BTI_J, ctx);
@@ -1781,6 +1863,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
emit_kfunc_arg_moves(ctx, fm);
}
emit_call(func_addr, ctx);
+ if (bpf_jit_unwind_site(env, ctx->prog, i)) {
+ ret = add_unwind_exentry(ctx, i + 2 + insn[1].off);
+ if (ret)
+ return ret;
+ emit(A64_MOV(1, r0, A64_R(0)), ctx);
+ return 1;
+ }
/*
* Call to arch_bpf_timed_may_goto() is emitted by the
* verifier and called with custom calling convention with
@@ -1799,6 +1888,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
* counter and the private stack are callee saved.
*/
emit(A64_BLR(dst), ctx);
+ if (bpf_jit_unwind_site(env, ctx->prog, i)) {
+ ret = add_unwind_exentry(ctx, i + 2 + insn[1].off);
+ if (ret)
+ return ret;
+ emit(A64_MOV(1, bpf2a64[BPF_REG_0], A64_R(0)), ctx);
+ return 1;
+ }
emit(A64_MOV(1, bpf2a64[BPF_REG_0], A64_R(0)), ctx);
break;
/* tail call */
@@ -1808,6 +1904,9 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
break;
/* function return */
case BPF_JMP | BPF_EXIT:
+ unw = bpf_jit_unwind_exit(env, ctx->prog, i);
+ if (unw != BPF_EXIT_RETURNS)
+ emit_unwind_ret_addr(ctx, unw == BPF_EXIT_LOOK_AT_R3);
/* Optimization: when last instruction is EXIT,
simply fallthrough to epilogue. */
if (i == ctx->prog->len - 1)
@@ -2240,6 +2339,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
int body_idx;
int exentry_idx;
int out_cnt;
+ int i;
if (!prog->jit_requested)
return prog;
@@ -2313,6 +2413,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
build_epilogue(&ctx, was_classic);
build_plt(&ctx);
+ /* The extra pass doesn't get here, so the count is adjusted once. */
+ for (i = 0; i < prog->len; i++)
+ if (bpf_jit_unwind_site(env, prog, i))
+ prog->aux->num_exentries++;
+
extable_align = __alignof__(struct exception_table_entry);
extable_size = prog->aux->num_exentries *
sizeof(struct exception_table_entry);
@@ -2464,6 +2569,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
return true;
}
+bool bpf_jit_supports_unwind(void)
+{
+ return true;
+}
+
const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
{
return &arm64_arg_abi;
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next 3/5] bpf, arm64: Support unwinding via r3
2026-10-18 1:02 ` [PATCH bpf-next 3/5] bpf, arm64: Support unwinding via r3 Alexei Starovoitov
@ 2026-10-18 1:34 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-10-18 1:34 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] bpf, arm64: stale jump_target in bpf_jit_unwind_exit() skips landing pad
--
commit e9dd702da2c478ecd8523b3f071fba585759acfc
Author: Alexei Starovoitov <ast@kernel.org>
bpf, arm64: Support unwinding via r3
This commit adds arm64 JIT support for unwinding via r3 by replacing the
post-call conditional branch with an exception table entry and rewriting the
return address in the callee frame record upon unwinding exit.
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 506c4201fec85..9b702930486a1 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
[ ... ]
> @@ -1781,6 +1863,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
> emit_kfunc_arg_moves(ctx, fm);
> }
> emit_call(func_addr, ctx);
> + if (bpf_jit_unwind_site(env, ctx->prog, i)) {
> + ret = add_unwind_exentry(ctx, i + 2 + insn[1].off);
> + if (ret)
> + return ret;
> + emit(A64_MOV(1, r0, A64_R(0)), ctx);
> + return 1;
> + }
> /*
> * Call to arch_bpf_timed_may_goto() is emitted by the
> * verifier and called with custom calling convention with
> @@ -1799,6 +1888,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
> * counter and the private stack are callee saved.
> */
> emit(A64_BLR(dst), ctx);
> + if (bpf_jit_unwind_site(env, ctx->prog, i)) {
> + ret = add_unwind_exentry(ctx, i + 2 + insn[1].off);
> + if (ret)
> + return ret;
> + emit(A64_MOV(1, bpf2a64[BPF_REG_0], A64_R(0)), ctx);
> + return 1;
> + }
> emit(A64_MOV(1, bpf2a64[BPF_REG_0], A64_R(0)), ctx);
> break;
> /* tail call */
> @@ -1808,6 +1904,9 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
> break;
> /* function return */
> case BPF_JMP | BPF_EXIT:
> + unw = bpf_jit_unwind_exit(env, ctx->prog, i);
> + if (unw != BPF_EXIT_RETURNS)
> + emit_unwind_ret_addr(ctx, unw == BPF_EXIT_LOOK_AT_R3);
[Severity: Critical]
Could stale jump_target metadata in env->insn_aux_data cause
bpf_jit_unwind_exit() or bpf_jit_unwind_site() to skip the landing pad when
unwinding?
In build_insn(), bpf_jit_unwind_site() elides the caller's post-call branch
and relies on the callee rewriting its return address via
emit_unwind_ret_addr(). Both bpf_jit_unwind_site() and bpf_jit_unwind_exit()
use bpf_is_jump_target() at JIT time to determine whether an instruction can
be reached by a jump rather than falling through from the preceding
instruction:
kernel/bpf/core.c:bpf_jit_unwind_exit() {
...
if (!i || bpf_is_jump_target(env, prog->aux->subprog_start + i) ||
insn[-1].code != (BPF_ALU64 | BPF_MOV | BPF_K) ||
insn[-1].dst_reg != BPF_REG_UNWIND)
return BPF_EXIT_LOOK_AT_R3;
return insn[-1].imm ? BPF_EXIT_UNWINDS : BPF_EXIT_RETURNS;
}
However, env->insn_aux_data[].jump_target is populated during bpf_check_cfg()
(which only marks branch targets, not fallthrough edges) and is not preserved
across post-CFG instruction removal passes in verifier_remove_insns():
kernel/bpf/fixups.c:verifier_remove_insns() {
...
memmove(aux_data + off, aux_data + off + cnt,
sizeof(*aux_data) * (orig_prog_len - off - cnt));
env->insn_aux_data_len -= cnt;
}
For example, in a subprogram annotated with btf_decl_tag("unwind"):
1. Path A sets r3 = 1; goto L, while path B sets r3 = 0 and falls through to
L, where L is a conditional branch whose target is dead (such as
if r6 != 0 goto dead; followed by exit;).
2. bpf_check_cfg() marks L with jump_target = true (from goto L), but leaves
the fallthrough exit at L + 1 with jump_target = false.
3. bpf_prune_dead_branches() or bpf_opt_hard_wire_dead_code_branches()
rewrites L into goto +0 without setting jump_target on L + 1.
4. bpf_opt_remove_nops() -> verifier_remove_insns() removes the NOP at L,
shifting exit into slot L (so goto L now targets exit directly with
r3 = 1) and overwriting aux_data[L].jump_target with
aux_data[L + 1].jump_target (false). A similar loss of jump_target can
happen when bpf_remove_fastcall_spills_fills() turns a jump-target spill
into NOPs that bpf_opt_remove_nops() removes, or when a NOP before the
post-call if r3 != 0 goto pad is removed.
When bpf_int_jit_compile() -> build_body() -> build_insn() runs,
bpf_jit_unwind_exit() sees !bpf_is_jump_target() and insn[-1] as r3 = 0,
returning BPF_EXIT_RETURNS. As a result, build_insn() omits
emit_unwind_ret_addr() while also eliding the caller's if r3 != 0 goto pad
branch via bpf_jit_unwind_site().
When path A executes at runtime and exits with r3 = 1, won't execution bypass
the landing pad and fall through into the caller's normal return path where
the verifier assumed r3 == 0?
> /* Optimization: when last instruction is EXIT,
> simply fallthrough to epilogue. */
> if (i == ctx->prog->len - 1)
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018010246.1643877-1-alexei.starovoitov@gmail.com?part=3
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH bpf-next 4/5] selftests/bpf: Add tests for unwinding via r3
2026-10-18 1:02 [PATCH bpf-next 0/5] bpf: Unwinding via r3 Alexei Starovoitov
` (2 preceding siblings ...)
2026-10-18 1:02 ` [PATCH bpf-next 3/5] bpf, arm64: Support unwinding via r3 Alexei Starovoitov
@ 2026-10-18 1:02 ` Alexei Starovoitov
2026-10-18 1:38 ` bot+bpf-ci
2026-10-18 1:02 ` [PATCH bpf-next 5/5] selftests/bpf: Add a C++ prog with exceptions Alexei Starovoitov
2026-10-18 1:31 ` [PATCH bpf-next 0/5] bpf: Unwinding via r3 Kumar Kartikeya Dwivedi
5 siblings, 1 reply; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-18 1:02 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The tests spell out the insns that llvm emits and put BTF decl_tag "unwind"
on the subprogs by hand, so they don't depend on llvm version.
verifier_unwind:
- r3 of a static and of a global subprog at run time, through one and
two frames. The verifier removes 'if r3 != 0' when it knows r3, so the
tests derive it from skb->len.
- A landing pad is checked like any other code: it has to drop
rcu_read_lock.
- r3 is not readable after a call to a subprog without the tag.
- A subprog with the tag has to exit with a scalar in r3.
- Precision of r3 is tracked into the callee.
- A callback has to exit with r3 == 0.
- Tail call and ld_abs are rejected.
- x86-64 and arm64 JITs don't emit the branch after the call, unless it's
a jump target, and an exit with r3 != 0 fixes its return address.
unwind_attach: only fentry can attach to a subprog that can unwind.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/unwind_attach.c | 67 ++
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/unwind_attach.c | 32 +
.../selftests/bpf/progs/verifier_unwind.c | 759 ++++++++++++++++++
4 files changed, 860 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_attach.c
create mode 100644 tools/testing/selftests/bpf/progs/unwind_attach.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_unwind.c
diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_attach.c b/tools/testing/selftests/bpf/prog_tests/unwind_attach.c
new file mode 100644
index 000000000000..6927757a15b6
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/unwind_attach.c
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "verifier_unwind.skel.h"
+#include "unwind_attach.skel.h"
+
+/*
+ * A trampoline that calls the subprog and an extension that replaces it don't
+ * keep r3, so only fentry can attach to a subprog that can unwind.
+ */
+static void attach(int tgt_fd, const char *name, bool ok)
+{
+ struct unwind_attach *skel;
+ struct bpf_program *prog;
+ int err;
+
+ skel = unwind_attach__open();
+ if (!ASSERT_OK_PTR(skel, "unwind_attach__open"))
+ return;
+
+ prog = bpf_object__find_program_by_name(skel->obj, name);
+ if (!ASSERT_OK_PTR(prog, name))
+ goto out;
+ bpf_program__set_autoload(prog, true);
+ err = bpf_program__set_attach_target(prog, tgt_fd, "global_unwinds");
+ if (!ASSERT_OK(err, "set_attach_target"))
+ goto out;
+
+ err = unwind_attach__load(skel);
+ if (ok)
+ ASSERT_OK(err, name);
+ else
+ ASSERT_ERR(err, name);
+out:
+ unwind_attach__destroy(skel);
+}
+
+void test_unwind_attach(void)
+{
+#if defined(__x86_64__) || defined(__aarch64__)
+ struct verifier_unwind *tgt;
+ struct bpf_program *prog;
+ int tgt_fd;
+
+ if (!is_jit_enabled()) {
+ test__skip();
+ return;
+ }
+
+ tgt = verifier_unwind__open();
+ if (!ASSERT_OK_PTR(tgt, "verifier_unwind__open"))
+ return;
+ bpf_object__for_each_program(prog, tgt->obj)
+ bpf_program__set_autoload(prog, prog == tgt->progs.calls_global_unwinds);
+ if (!ASSERT_OK(verifier_unwind__load(tgt), "verifier_unwind__load"))
+ goto out;
+ tgt_fd = bpf_program__fd(tgt->progs.calls_global_unwinds);
+
+ attach(tgt_fd, "fentry_unwind", true);
+ attach(tgt_fd, "fexit_unwind", false);
+ attach(tgt_fd, "fmod_ret_unwind", false);
+ attach(tgt_fd, "freplace_unwind", false);
+out:
+ verifier_unwind__destroy(tgt);
+#else
+ test__skip();
+#endif
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index c69c0f7e9cc8..2c15d6251b5a 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -130,6 +130,7 @@
#include "verifier_uninit.skel.h"
#include "verifier_unpriv.skel.h"
#include "verifier_unpriv_perf.skel.h"
+#include "verifier_unwind.skel.h"
#include "verifier_value_adj_spill.skel.h"
#include "verifier_value.skel.h"
#include "verifier_value_illegal_alu.skel.h"
@@ -319,6 +320,7 @@ void test_verifier_typedef(void) { RUN(verifier_typedef); }
void test_verifier_uninit(void) { RUN(verifier_uninit); }
void test_verifier_unpriv(void) { RUN(verifier_unpriv); }
void test_verifier_unpriv_perf(void) { RUN(verifier_unpriv_perf); }
+void test_verifier_unwind(void) { RUN(verifier_unwind); }
void test_verifier_value_adj_spill(void) { RUN(verifier_value_adj_spill); }
void test_verifier_value(void) { RUN(verifier_value); }
void test_verifier_value_illegal_alu(void) { RUN(verifier_value_illegal_alu); }
diff --git a/tools/testing/selftests/bpf/progs/unwind_attach.c b/tools/testing/selftests/bpf/progs/unwind_attach.c
new file mode 100644
index 000000000000..269d349139fc
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/unwind_attach.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/* The target is global_unwinds() in verifier_unwind.c */
+
+SEC("?fentry/global_unwinds")
+int BPF_PROG(fentry_unwind)
+{
+ return 0;
+}
+
+SEC("?fexit/global_unwinds")
+int BPF_PROG(fexit_unwind)
+{
+ return 0;
+}
+
+SEC("?fmod_ret/global_unwinds")
+int BPF_PROG(fmod_ret_unwind)
+{
+ return 0;
+}
+
+SEC("?freplace/global_unwinds")
+unsigned long freplace_unwind(void)
+{
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_unwind.c b/tools/testing/selftests/bpf/progs/verifier_unwind.c
new file mode 100644
index 000000000000..18b4774f4292
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_unwind.c
@@ -0,0 +1,759 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/*
+ * llvm lowers unwinding to plain insns:
+ * ret r3 = 0; exit
+ * invoke foo unwind pad call foo; if r3 != 0 goto pad
+ * resume r0 = 0; r3 = 1; exit
+ * and puts this tag on the functions that say something in r3.
+ * The tests spell the insns out, so they don't depend on llvm version.
+ */
+#define __unwind __attribute__((btf_decl_tag("unwind")))
+
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+} prog_array SEC(".maps");
+
+/*
+ * The verifier removes 'if r3 != 0' when it knows r3. To have it at run time
+ * the tests pass skb->len, which is 64, or skb->len - 64.
+ *
+ * r1 == 0: returns 7, otherwise unwinds
+ */
+__naked __noinline __used __unwind
+static unsigned long may_unwind(void)
+{
+ asm volatile (
+ "if r1 != 0 goto 1f;"
+ "r0 = 7;"
+ "r3 = 0;"
+ "exit;"
+ "1:"
+ "r0 = 0;"
+ "r3 = 1;"
+ "exit;"
+ );
+}
+
+/* owns nothing: its pad only resumes */
+__naked __noinline __used __unwind
+static unsigned long passes_on(void)
+{
+ asm volatile (
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "r0 += 1;"
+ "r3 = 0;"
+ "exit;"
+ "1:"
+ "r0 = 0;"
+ "r3 = 1;"
+ "exit;"
+ );
+}
+
+__naked __noinline __used __unwind
+unsigned long global_returns(void)
+{
+ asm volatile (
+ "r0 = 7;"
+ "r3 = 0;"
+ "exit;"
+ );
+}
+
+__naked __noinline __used __unwind
+unsigned long global_unwinds(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = 1;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: the callee returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int callee_returns(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r1 -= 64;"
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: the callee unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int callee_unwinds(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: return through two frames")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(8)
+__naked int two_frames_return(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r1 -= 64;"
+ "call passes_on;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: unwind through two frames")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int two_frames_unwind(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call passes_on;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+/* 'if r3 != 0' makes r3 precise. The chain goes into the callee through its exit. */
+SEC("tc")
+__description("unwind: precision of r3 is tracked into the callee")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __log_level(2)
+__msg("mark_precise: frame0: regs=r3 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r3 stack= before 11: (b7) r3 = 1")
+__naked int r3_precision(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: the pad drops what the frame owns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success
+__naked int pad_unlocks(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "call %[bpf_get_prandom_u32];"
+ "r1 = r0;"
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 0;"
+ "exit;"
+ "1:"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock),
+ __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: the pad is checked like any other code")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_rcu_read_lock-ed region")
+__naked int pad_forgets_to_unlock(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "call %[bpf_get_prandom_u32];"
+ "r1 = r0;"
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "call %[bpf_rcu_read_unlock];"
+ "1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock),
+ __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/* The verifier doesn't look into a global subprog at the call: it may unwind. */
+SEC("tc")
+__description("unwind: r3 is unknown after a call to a global subprog")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_rcu_read_lock-ed region")
+__naked int global_pad_forgets_to_unlock(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "call global_returns;"
+ "if r3 != 0 goto 1f;"
+ "call %[bpf_rcu_read_unlock];"
+ "1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: a global subprog returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(7)
+__naked int calls_global_returns(void)
+{
+ asm volatile (
+ "call global_returns;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: a global subprog unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int calls_global_unwinds(void)
+{
+ asm volatile (
+ "call global_unwinds;"
+ "if r3 != 0 goto 1f;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long no_tag(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: r3 is not readable after a call to a subprog without the tag")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R3 !read_ok")
+__naked int callee_has_no_tag(void)
+{
+ asm volatile (
+ "call no_tag;"
+ "if r3 != 0 goto 1f;"
+ "1:"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used __unwind
+static unsigned long r3_not_set(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: static subprog doesn't set r3")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R3 !read_ok")
+__naked int static_r3_not_set(void)
+{
+ asm volatile (
+ "call r3_not_set;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used __unwind
+static unsigned long r3_is_pointer(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = r10;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: static subprog leaves a pointer in r3")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("At subprogram exit the register R3 is not a scalar value (fp)")
+__naked int static_r3_is_pointer(void)
+{
+ asm volatile (
+ "call r3_is_pointer;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used __unwind
+unsigned long global_r3_not_set(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: global subprog doesn't set r3")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("R3 !read_ok")
+__naked int calls_global_r3_not_set(void)
+{
+ asm volatile (
+ "call global_r3_not_set;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used __unwind
+unsigned long global_r3_is_pointer(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = r10;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: global subprog leaves a pointer in r3")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("At subprogram exit the register R3 is not a scalar value (fp)")
+__naked int calls_global_r3_is_pointer(void)
+{
+ asm volatile (
+ "call global_r3_is_pointer;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/* The helper that calls a callback doesn't look at r3. */
+__naked __noinline __used __unwind
+static unsigned long cb_returns(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = 0;"
+ "exit;"
+ );
+}
+
+__naked __noinline __used __unwind
+static unsigned long cb_unwinds(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = 1;"
+ "exit;"
+ );
+}
+
+__naked __noinline __used __unwind
+static unsigned long cb_r3_not_set(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: callback returns")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(3)
+__naked int callback_returns(void)
+{
+ asm volatile (
+ "r1 = 3;"
+ "r2 = %[cb_returns];"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "exit;"
+ :
+ : __imm_ptr(cb_returns),
+ __imm(bpf_loop)
+ : __clobber_common
+ );
+}
+
+SEC("tc")
+__description("unwind: callback unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("callback cannot unwind")
+__naked int callback_unwinds(void)
+{
+ asm volatile (
+ "r1 = 3;"
+ "r2 = %[cb_unwinds];"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "exit;"
+ :
+ : __imm_ptr(cb_unwinds),
+ __imm(bpf_loop)
+ : __clobber_common
+ );
+}
+
+SEC("tc")
+__description("unwind: callback doesn't set r3")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("callback cannot unwind")
+__naked int callback_r3_not_set(void)
+{
+ asm volatile (
+ "r1 = 3;"
+ "r2 = %[cb_r3_not_set];"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "exit;"
+ :
+ : __imm_ptr(cb_r3_not_set),
+ __imm(bpf_loop)
+ : __clobber_common
+ );
+}
+
+/* Both return to the caller of the subprog with whatever is in r3. */
+__naked __noinline __used __unwind
+static unsigned long does_tail_call(void)
+{
+ asm volatile (
+ "r2 = %[prog_array] ll;"
+ "r3 = 0;"
+ "call %[bpf_tail_call];"
+ "r0 = 0;"
+ "r3 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_tail_call),
+ __imm_addr(prog_array)
+ );
+}
+
+SEC("tc")
+__description("unwind: tail call in a subprog that can unwind")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("tail call or ld_abs in a subprog that can unwind")
+__naked int tail_call_in_subprog(void)
+{
+ asm volatile (
+ "call does_tail_call;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used __unwind
+static unsigned long does_ld_abs(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "r0 = *(u8*)skb[0];"
+ "r3 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: ld_abs in a subprog that can unwind")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__failure __msg("tail call or ld_abs in a subprog that can unwind")
+__naked int ld_abs_in_subprog(void)
+{
+ asm volatile (
+ "call does_ld_abs;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/*
+ * JITs don't emit 'if r3 != 0 goto pad' after a call. It becomes an extable entry.
+ * An exit with r3 != 0 looks its return address up there.
+ */
+SEC("tc")
+__description("unwind: jit: no branch after the call")
+__success __retval(42)
+__load_if_JITed()
+__arch_x86_64
+__jited(" callq 0x{{.*}}")
+__jited(" movl $0x7, %eax")
+__jited("...")
+__jited(" movl $0x1, %edx")
+__jited(" pushq %rax")
+__jited(" pushq %rsi")
+__jited(" pushq %rdx")
+__jited(" movq 0x8(%rbp), %rdi")
+__jited(" callq 0x{{.*}}")
+__jited(" movq %rax, 0x8(%rbp)")
+__jited(" popq %rdx")
+__jited(" popq %rsi")
+__jited(" popq %rax")
+__arch_arm64
+__jited(" bl {{.*}}")
+__jited(" mov x8, x0")
+__jited(" mov x8, #0x7")
+__jited("...")
+__jited(" mov x2, #0x1")
+__jited(" stp x8, x1, [sp, #-0x10]!")
+__jited(" stp x2, xzr, [sp, #-0x10]!")
+__jited(" ldr x17, [x29, #0x8]")
+__jited("...")
+__jited(" str x17, [x29, #0x8]")
+__jited(" ldp x2, xzr, [sp], #0x10")
+__jited(" ldp x8, x1, [sp], #0x10")
+__naked int jit_no_branch(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call may_unwind;"
+ "if r3 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+/* The exit cannot tell what is in r3 */
+__naked __noinline __used __unwind
+static unsigned long r3_from_r1(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r3 = r1;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__description("unwind: jit: exit looks at r3, returns")
+__success __retval(7)
+__load_if_JITed()
+__arch_x86_64
+__jited(" movq %rdi, %rdx")
+__jited(" testq %rdx, %rdx")
+__jited(" je {{.*}}")
+__jited(" pushq %rax")
+__arch_arm64
+__jited(" mov x2, x0")
+__jited(" cbz x2, {{.*}}")
+__jited(" stp x8, x1, [sp, #-0x10]!")
+__naked int jit_exit_looks_at_r3_returns(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "r1 -= 64;"
+ "call r3_from_r1;"
+ "if r3 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("unwind: jit: exit looks at r3, unwinds")
+__arch_x86_64
+__arch_arm64
+__load_if_JITed()
+__success __retval(42)
+__naked int jit_exit_looks_at_r3_unwinds(void)
+{
+ asm volatile (
+ "r1 = *(u32 *)(r1 + %[len]);"
+ "call r3_from_r1;"
+ "if r3 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len))
+ : __clobber_all
+ );
+}
+
+/* Something else jumps to the branch, so it has to be there */
+SEC("tc")
+__description("unwind: jit: the branch is a jump target")
+__success
+__load_if_JITed()
+__arch_x86_64
+__jited(" movl $0x1, %edi")
+__jited(" callq 0x{{.*}}")
+__jited("{{.*}} testq %rdx, %rdx")
+__jited(" jne {{.*}}")
+__arch_arm64
+__jited(" mov x0, #0x1")
+__jited(" bl {{.*}}")
+__jited(" mov x8, x0")
+__jited("{{.*}} cmp x2, #0x0")
+__jited(" b.ne {{.*}}")
+__naked int jit_branch_is_jump_target(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r3 = 0;"
+ "if r0 == 0 goto 2f;"
+ "r1 = 1;"
+ "call may_unwind;"
+ "2:"
+ "if r3 != 0 goto 1f;"
+ "r0 = 7;"
+ "exit;"
+ "1:"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * BTF FUNC records are not generated for kfuncs referenced
+ * from inline assembly. These records are necessary for
+ * libbpf to link the program. The function below is a hack
+ * to ensure that BTF FUNC records are generated.
+ */
+void __kfunc_btf_root(void)
+{
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next 4/5] selftests/bpf: Add tests for unwinding via r3
2026-10-18 1:02 ` [PATCH bpf-next 4/5] selftests/bpf: Add tests for " Alexei Starovoitov
@ 2026-10-18 1:38 ` bot+bpf-ci
0 siblings, 0 replies; 13+ messages in thread
From: bot+bpf-ci @ 2026-10-18 1:38 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3832 bytes --]
> selftests/bpf: Add tests for unwinding via r3
TL;DR:
- unwind_attach.c: unused attach() on non-x86/arm64 may fail the whole
selftest build under -Werror
- fmod_ret negative test passes without the new check; error-only assertions
are weak
> diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_attach.c b/tools/testing/selftests/bpf/prog_tests/unwind_attach.c
> new file mode 100644
> index 000000000000..6927757a15b6
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/unwind_attach.c
> @@ -0,0 +1,67 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <test_progs.h>
> +#include "verifier_unwind.skel.h"
> +#include "unwind_attach.skel.h"
> +
> +/*
> + * A trampoline that calls the subprog and an extension that replaces it don't
> + * keep r3, so only fentry can attach to a subprog that can unwind.
> + */
> +static void attach(int tgt_fd, const char *name, bool ok)
> +{
Will this build on architectures other than x86_64 and aarch64?
attach() is a non-inline static function, but every caller is inside the
#if defined(__x86_64__) || defined(__aarch64__) block in
test_unwind_attach(). On s390x, riscv64, ppc64le, loongarch and the rest,
the #else branch only calls test__skip(), so attach() is defined but never
used.
The selftests build prog_tests/*.c with the flags from Makefile.buildvars:
COMMON_CFLAGS = -g $(OPT_FLAGS) -rdynamic -std=gnu11 \
-Wall -Werror -fno-omit-frame-pointer ...
-Wall enables -Wunused-function, so compiling unwind_attach.test.o should
fail there with "'attach' defined but not used
[-Werror=unused-function]", which breaks the test_progs build for the whole
suite. s390x and riscv64 both have DENYLIST files in the selftests tree, so
they are built.
The skeleton helpers from the two included .skel.h files are static inline
and don't warn, so only attach() is affected. No later patch in this series
touches this file.
Could attach() move inside the arch #if block, or be marked __maybe_unused?
[ ... ]
> +void test_unwind_attach(void)
> +{
> +#if defined(__x86_64__) || defined(__aarch64__)
[ ... ]
> + attach(tgt_fd, "fentry_unwind", true);
> + attach(tgt_fd, "fexit_unwind", false);
> + attach(tgt_fd, "fmod_ret_unwind", false);
> + attach(tgt_fd, "freplace_unwind", false);
This isn't a bug, but does the fmod_ret_unwind case test the new restriction?
Every expected failure in attach() is checked only with ASSERT_ERR(err,
name), and bpf_check_attach_target() already rejects BPF_MODIFY_RETURN
against any BPF program target, whether or not the target can unwind:
} else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
if (tgt_prog) {
module_put(mod);
bpf_log(log, "can't modify return codes of BPF programs\n");
return -EINVAL;
}
So with the new "Only fentry can attach to %s, which can unwind" check the
load returns -EINVAL, and without it the load still returns -EINVAL from the
check above. The fmod_ret case can't catch a regression in the restriction
described in the commit message.
The fexit and freplace cases do discriminate, since fexit on a BPF subprog
is otherwise allowed and freplace_unwind() matches global_unwinds() in
btf_check_func_type_match(). But they accept any error, so an unrelated
load failure would make them pass too.
fexit_bpf2bpf.c's test_obj_load_failure_common() captures the verifier log
and checks it with ASSERT_HAS_SUBSTR(log_buf, exp_msg, ...). Could the
negative cases here check for "which can unwind" in the log in the same way,
so each one exercises the new check?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39085585797
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH bpf-next 5/5] selftests/bpf: Add a C++ prog with exceptions
2026-10-18 1:02 [PATCH bpf-next 0/5] bpf: Unwinding via r3 Alexei Starovoitov
` (3 preceding siblings ...)
2026-10-18 1:02 ` [PATCH bpf-next 4/5] selftests/bpf: Add tests for " Alexei Starovoitov
@ 2026-10-18 1:02 ` Alexei Starovoitov
2026-10-18 1:38 ` bot+bpf-ci
2026-10-18 1:31 ` [PATCH bpf-next 0/5] bpf: Unwinding via r3 Kumar Kartikeya Dwivedi
5 siblings, 1 reply; 13+ messages in thread
From: Alexei Starovoitov @ 2026-10-18 1:02 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
static __noinline int middle(int len)
{
preempt_guard guard;
return inner(len) + 1;
}
SEC("tc") int catches(struct __sk_buff *skb)
{
try {
return middle(skb->len);
} catch (...) {
return 42;
}
}
The exception passes through two frames. The destructors call
bpf_rcu_read_unlock() and bpf_preempt_enable(). The verifier insists on
both, and the test checks that they ran in that order.
In the second prog nobody catches. The destructors run and the prog
returns 0.
All that is left of the C++ runtime is four one line functions.
__cxa_throw() calls _Unwind_RaiseException(), which llvm lowers.
progs/*.cpp are built like progs/*.rs: by clang only and without skeletons.
vmlinux.h and bpf_helpers.h are not C++, so the prog declares what it
needs.
The test is skipped when llvm doesn't lower unwinding [1]. It looks for BTF
decl_tag "unwind" in the object, since the version of llvm doesn't tell.
[1] https://github.com/llvm/llvm-project/pull/233140
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile.skel | 11 ++
.../selftests/bpf/prog_tests/unwind_cxx.c | 76 ++++++++++++
.../selftests/bpf/progs/unwind_cxx.cpp | 117 ++++++++++++++++++
3 files changed, 204 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
create mode 100644 tools/testing/selftests/bpf/progs/unwind_cxx.cpp
diff --git a/tools/testing/selftests/bpf/Makefile.skel b/tools/testing/selftests/bpf/Makefile.skel
index 06e297a17156..6570c48f92b0 100644
--- a/tools/testing/selftests/bpf/Makefile.skel
+++ b/tools/testing/selftests/bpf/Makefile.skel
@@ -153,4 +153,15 @@ $(RUST_OBJS): $(RDIR)/%.bpf.o: progs/%.rs $(RUST_CORE) | $(RDIR)
endif
endif
+# Programs in C++. No skeletons: gcc doesn't build them.
+ifeq ($(BPF_CC),$(CLANG))
+CXX_OBJS := $(patsubst progs/%.cpp,$(RDIR)/%.bpf.o,$(wildcard progs/*.cpp))
+BPF_OBJS += $(CXX_OBJS)
+
+$(CXX_OBJS): $(RDIR)/%.bpf.o: progs/%.cpp | $(RDIR)
+ $(call msg,$(BPF_CC_MSG),$(BINARY),$@)
+ $(Q)$(BPF_CC) -x c++ -g -Wall -Werror -fexceptions -funwind-tables -fno-rtti \
+ $(BPF_CC_FLAGS) -c $< -o $@ $(call skip_on_fail,BPF)
+endif
+
endif # BPF_CC
diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
new file mode 100644
index 000000000000..6a189045851f
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <bpf/btf.h>
+
+/*
+ * llvm tags the functions that say in r3 whether they return or unwind.
+ * Without that _Unwind_RaiseException() is an unknown extern and libbpf doesn't
+ * open the object, so look before that.
+ */
+static bool llvm_lowers_unwinding(const char *file)
+{
+ struct btf *btf = btf__parse(file, NULL);
+ int i, n = btf ? btf__type_cnt(btf) : 0;
+ bool ret = false;
+
+ for (i = 1; i < n && !ret; i++) {
+ const struct btf_type *t = btf__type_by_id(btf, i);
+
+ ret = btf_is_decl_tag(t) &&
+ !strcmp(btf__name_by_offset(btf, t->name_off), "unwind");
+ }
+ btf__free(btf);
+ return ret;
+}
+
+static void run(struct bpf_object *obj, const char *name, int len, int retval, int dtors)
+{
+ struct bpf_program *prog = bpf_object__find_program_by_name(obj, name);
+ struct bpf_map *map = bpf_object__find_map_by_name(obj, ".data.dtors");
+ char data[256] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .data_in = data,
+ .data_size_in = len,
+ .repeat = 1,
+ );
+ int key = 0, val = 0;
+
+ if (!ASSERT_OK_PTR(prog, name) || !ASSERT_OK_PTR(map, "dtors"))
+ return;
+ ASSERT_OK(bpf_map__update_elem(map, &key, sizeof(key), &val, sizeof(val), 0), "clear");
+ ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(prog), &opts), "test_run");
+ ASSERT_EQ(opts.retval, retval, "retval");
+ ASSERT_OK(bpf_map__lookup_elem(map, &key, sizeof(key), &val, sizeof(val), 0), "lookup");
+ ASSERT_EQ(val, dtors, "dtors");
+}
+
+void test_unwind_cxx(void)
+{
+#if defined(__x86_64__) || defined(__aarch64__)
+ const char *file = "./unwind_cxx.bpf.o";
+ struct bpf_object *obj;
+
+ /* gcc flavor doesn't build it */
+ if (access(file, R_OK) || !llvm_lowers_unwinding(file) || !is_jit_enabled()) {
+ test__skip();
+ return;
+ }
+ obj = bpf_object__open_file(file, NULL);
+ if (!ASSERT_OK_PTR(obj, "open"))
+ return;
+ if (!ASSERT_OK(bpf_object__load(obj), "load"))
+ goto out;
+ /* libbpf goes on without BTF when the kernel doesn't take it */
+ ASSERT_GE(bpf_object__btf_fd(obj), 0, "btf_fd");
+
+ /* inner's destructor, then middle's */
+ run(obj, "catches", 64, 65, 12);
+ run(obj, "catches", 200, 42, 12);
+ run(obj, "does_not_catch", 64, 65, 12);
+ run(obj, "does_not_catch", 200, 0, 12);
+out:
+ bpf_object__close(obj);
+#else
+ test__skip();
+#endif
+}
diff --git a/tools/testing/selftests/bpf/progs/unwind_cxx.cpp b/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
new file mode 100644
index 000000000000..fa9ddd87319d
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/unwind_cxx.cpp
@@ -0,0 +1,117 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ exceptions. llvm lowers unwinding to plain insns:
+ * ret r3 = 0; exit
+ * invoke foo unwind pad call foo; if r3 != 0 goto pad
+ * resume r0 = 0; r3 = 1; exit
+ * so 'throw' is a jump or an exit, 'catch (...)' and the destructors that run on
+ * the way are ordinary blocks reached by ordinary branches.
+ *
+ * vmlinux.h and bpf_helpers.h are not C++.
+ */
+#define SEC(name) __attribute__((section(name), used))
+#define __ksym __attribute__((section(".ksyms")))
+#define __noinline __attribute__((noinline))
+#define __always_inline inline __attribute__((always_inline))
+
+#if __clang_major__ >= 24
+
+struct __sk_buff {
+ unsigned int len;
+};
+
+extern "C" {
+void bpf_rcu_read_lock(void) noexcept __ksym;
+void bpf_rcu_read_unlock(void) noexcept __ksym;
+void bpf_preempt_disable(void) noexcept __ksym;
+void bpf_preempt_enable(void) noexcept __ksym;
+
+/* Starts an unwind. Not a kfunc: llvm lowers the call and it doesn't come back. */
+int _Unwind_RaiseException(void *exc);
+
+/* All that is left of the C++ runtime. Nobody looks at the exception object. */
+static char exc_obj[16];
+
+__always_inline void *__cxa_allocate_exception(unsigned long size) noexcept
+{
+ return exc_obj;
+}
+
+__always_inline void __cxa_throw(void *exc, void *tinfo, void (*dtor)(void *))
+{
+ _Unwind_RaiseException(exc);
+ __builtin_trap();
+}
+
+__always_inline void *__cxa_begin_catch(void *exc) noexcept
+{
+ return exc;
+}
+
+__always_inline void __cxa_end_catch(void) noexcept
+{
+}
+}
+
+/* One decimal digit per destructor, in the order they ran */
+volatile int dtors SEC(".data.dtors");
+
+/* The verifier insists that both are undone on every path out of the prog */
+struct rcu_guard {
+ __always_inline rcu_guard()
+ {
+ bpf_rcu_read_lock();
+ }
+ __always_inline ~rcu_guard()
+ {
+ bpf_rcu_read_unlock();
+ dtors = dtors * 10 + 1;
+ }
+};
+
+struct preempt_guard {
+ __always_inline preempt_guard()
+ {
+ bpf_preempt_disable();
+ }
+ __always_inline ~preempt_guard()
+ {
+ bpf_preempt_enable();
+ dtors = dtors * 10 + 2;
+ }
+};
+
+static __noinline int inner(int len)
+{
+ rcu_guard guard;
+
+ if (len > 100)
+ throw 1;
+ return len;
+}
+
+static __noinline int middle(int len)
+{
+ preempt_guard guard;
+
+ return inner(len) + 1;
+}
+
+extern "C" SEC("tc") int catches(struct __sk_buff *skb)
+{
+ try {
+ return middle(skb->len);
+ } catch (...) {
+ return 42;
+ }
+}
+
+/* The exception leaves the prog: it returns 0 */
+extern "C" SEC("tc") int does_not_catch(struct __sk_buff *skb)
+{
+ return middle(skb->len);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next 5/5] selftests/bpf: Add a C++ prog with exceptions
2026-10-18 1:02 ` [PATCH bpf-next 5/5] selftests/bpf: Add a C++ prog with exceptions Alexei Starovoitov
@ 2026-10-18 1:38 ` bot+bpf-ci
0 siblings, 0 replies; 13+ messages in thread
From: bot+bpf-ci @ 2026-10-18 1:38 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]
> selftests/bpf: Add a C++ prog with exceptions
TL;DR: Test nit only: test_unwind_cxx() skips silently in three different
cases (no object, llvm lacks unwinding lowering, JIT disabled); consider
printing a skip reason, though existing tests do the same.
> diff --git a/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
> new file mode 100644
> index 000000000000..6a189045851f
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
[ ... ]
> +void test_unwind_cxx(void)
> +{
> +#if defined(__x86_64__) || defined(__aarch64__)
> + const char *file = "./unwind_cxx.bpf.o";
> + struct bpf_object *obj;
> +
> + /* gcc flavor doesn't build it */
> + if (access(file, R_OK) || !llvm_lowers_unwinding(file) || !is_jit_enabled()) {
> + test__skip();
> + return;
> + }
This isn't a bug, but three different conditions end in the same bare
test__skip() here, and none of them prints a reason:
access(file, R_OK) the object was not built (gcc flavor)
!llvm_lowers_unwinding(file) llvm has no BTF decl_tag "unwind" lowering
!is_jit_enabled() the JIT is disabled
Would it be worth printing which check failed before calling test__skip(),
for example:
printf("%s:SKIP:llvm does not lower unwinding\n", __func__);
so that a reader of the test_progs output can tell an old toolchain from
bpf_jit_enable being 0?
The subsystem selftests guidance asks for a reason string with every skip.
The same pattern is used by test_rust() in prog_tests/data_in_arena.c, so
this may simply follow existing practice.
> + obj = bpf_object__open_file(file, NULL);
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39085585797
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH bpf-next 0/5] bpf: Unwinding via r3
2026-10-18 1:02 [PATCH bpf-next 0/5] bpf: Unwinding via r3 Alexei Starovoitov
` (4 preceding siblings ...)
2026-10-18 1:02 ` [PATCH bpf-next 5/5] selftests/bpf: Add a C++ prog with exceptions Alexei Starovoitov
@ 2026-10-18 1:31 ` Kumar Kartikeya Dwivedi
5 siblings, 0 replies; 13+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-18 1:31 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87
On Sun Oct 18, 2026 at 3:02 AM CEST, Alexei Starovoitov wrote:
> From: Alexei Starovoitov <ast@anthropic.com>
>
> This is the r3 alternative from the discussion of v12 of Yonghong's series
> [1] as patches, with x86-64 and arm64 JITs and selftests.
>
> 1. What is the problem?
>
> A rust panic or a C++ exception has to run Drop impls or destructors of
> every frame it passes through. bpf_throw() discards the frames, so the
> verifier doesn't let a frame that owns a lock or a reference throw. In rust
> every index and every unwrap() can panic.
>
> 2. Is it worth solving?
>
> Without it rust-bpf [2] drops all landing pads and a panic is bpf_throw().
> A prog that can panic with a guard held doesn't load.
>
> 3. Is there a simpler way?
>
> [1] keeps the normal path free of unwind code. llvm emits a table, libbpf
> passes it to the kernel, bpf_unwind() walks the stack and rewrites return
> addresses, and the verifier models hidden edges from calls to pads. That's
> 1400 lines in the kernel, with new uapi and two kfuncs, and 440 lines in
> libbpf.
>
> Here llvm emits plain insns [3]:
>
> ret r3 = 0; exit
> invoke foo unwind pad call foo; if r3 != 0 goto pad
> resume r0 = 0; r3 = 1; exit
>
> Pads are ordinary code, so the verifier part is 96 lines in patch 1.
> The price is a branch after a call that may unwind. Patches 2 and 3 take it
> off the normal path in the JITs without telling the verifier. 360 lines
> with them.
>
> 4. Will the api and the layers last?
>
> No uapi, no kfuncs, no changes in libbpf and other loaders.
> r3 and BTF decl_tag "unwind" become an ABI between llvm and the kernel.
> The source level entry point is _Unwind_RaiseException() of the Itanium
> ABI. It's what the runtimes of rust and C++ call on other architectures.
>
> 5. Is it good taste?
>
> Every frame returns through its own epilogue. The only question is where it
> returns to. That's one bit.
>
> It's not specific to rust. A C++ prog with throw, catch and destructors
> compiles, loads and runs. It's a selftest in patch 5.
>
> Since the sketches in [1]:
> - bpf_unwind() is gone. _Unwind_RaiseException() starts an unwind.
> - llvm applies the convention to functions with uwtable only, so C progs
> are not affected.
> - JITs opt in. A JIT that loses r3 would skip a pad.
> - A subprog with the tag has to exit with a scalar in r3.
> - A callback that doesn't set r3 was hitting verifier_bug().
> - The extable lookup is search_bpf_extables().
> - 'Functions end with exit' was not quite right. A function can still end
> with a call of a noreturn function that cannot unwind, like bpf_throw().
> rust-bpf adds 'exit' after it.
>
> Tested in qemu only, x86-64 and arm64 with and without pointer
> authentication: the selftests, C, C++ and rust progs built with [3],
> with and without bpf_jit_harden=2. test_progs was run on x86-64. On arm64
> the same objects were loaded and run by a small loader, and __jited
> patterns were matched against the JIT output by a script.
>
> [1] https://lore.kernel.org/bpf/20261017025129.1914859-1-yonghong.song@linux.dev/
> [2] https://github.com/4ast/rust-bpf
> [3] https://github.com/llvm/llvm-project/pull/233140
>
> Alexei Starovoitov (5):
> bpf: Support unwinding via r3
> bpf, x86: Replace 'if r3 != 0 goto pad' after a call with an extable
> entry
> bpf, arm64: Support unwinding via r3
> selftests/bpf: Add tests for unwinding via r3
> selftests/bpf: Add a C++ prog with exceptions
>
> arch/arm64/net/bpf_jit.h | 2 +
> arch/arm64/net/bpf_jit_comp.c | 110 +++
> arch/x86/net/bpf_jit_comp.c | 107 ++-
> include/linux/bpf.h | 1 +
> include/linux/bpf_verifier.h | 5 +-
> include/linux/filter.h | 11 +
> kernel/bpf/backtrack.c | 10 +-
> kernel/bpf/core.c | 53 ++
> kernel/bpf/fixups.c | 1 +
> kernel/bpf/liveness.c | 7 +-
> kernel/bpf/verifier.c | 66 +-
> tools/testing/selftests/bpf/Makefile.skel | 11 +
> .../selftests/bpf/prog_tests/unwind_attach.c | 67 ++
> .../selftests/bpf/prog_tests/unwind_cxx.c | 76 ++
> .../selftests/bpf/prog_tests/verifier.c | 2 +
> .../selftests/bpf/progs/unwind_attach.c | 32 +
> .../selftests/bpf/progs/unwind_cxx.cpp | 117 +++
> .../selftests/bpf/progs/verifier_unwind.c | 759 ++++++++++++++++++
> 18 files changed, 1427 insertions(+), 10 deletions(-)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_attach.c
> create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_cxx.c
> create mode 100644 tools/testing/selftests/bpf/progs/unwind_attach.c
> create mode 100644 tools/testing/selftests/bpf/progs/unwind_cxx.cpp
> create mode 100644 tools/testing/selftests/bpf/progs/verifier_unwind.c
>
>
> base-commit: c058636f788f8751e419b407206516280226e568
Overall this makes sense, I guess my one small nit is that since we've
crossed the bridge on changing the compiler, we could just use a
cleaner encoding of some new instruction, but that is a matter of
preference, it would be slightly more explicit and cleaner. Claude
suggests "BPF_JMP | BPF_JCOND with a new value in imm" with the
following benefits:
- The verifier treats it exactly as JNE r3, 0, so no new verifier
semantics.
- The JIT elides it by opcode, with no lookahead and no shape test.
- Blinding never touches it, since JCOND carries no blindable
constant, so the blinding fallback disappears.
- The kernel can require that it directly follow a call and not be a
jump target, and reject otherwise, which makes elision
unconditional rather than best-effort.
- Dumps show the intent, which also matters when someone reads an
xlated program.
If we encode unwind exit as another jcond variant to replace
resume/raise_exception then you can get rid of the decl tag as well
and just compute it from the subprog body while we walk all
instructions.
Concretely, two imm values in JCOND next to may_goto, and the whole
convention in the object becomes:
in the object meaning
-------------------------------- ----------------------------------
unwind_goto pad (after a call) if the callee unwound, go to pad
unwind_exit leave this frame unwinding, r0 = 0
exit return normally
^ permalink raw reply [flat|nested] 13+ messages in thread