* [PATCH bpf-next v1 01/29] bpf: Record where JITs save callee-saved registers
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
@ 2026-11-11 12:56 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 02/29] bpf: Add bpf_throw_unwind() and an arch entry hook Kumar Kartikeya Dwivedi
` (28 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:56 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently bpf_throw() discards the frames between the throw site and the
main program without running their epilogues. The main program restores
the callee-saved registers that these frames clobbered: its prologue
saves all of them, and the exception callback pops them.
To restore the registers from the discarded frames instead, the unwinder
must know where each JITed function saved the R6-R9 of its caller.
Walking outwards from the throw site, a register is either:
- still in the physical register, if no inner frame saved it; or
- in the prologue save area of the nearest inner frame that saved it.
Add callee_reg_off[4] to struct bpf_prog_aux. Each element is the offset
of a saved register from the native frame pointer that
arch_bpf_stack_walk() reports for the frame, or 0 if the register is not
saved. The x86-64 and arm64 JITs fill it in. Nothing uses it yet.
On x86-64 the frame holds, below rbp from high to low addresses:
- the program stack of round_up(stack_depth, 8) bytes, or nothing with a
private stack;
- the tail call count and its pointer, when tail_call_reachable;
- r12, when the program is an exception boundary or uses an arena;
- rbx, r13, r14 and r15 in push order, each only if it is pushed. An
exception boundary pushes all four.
On arm64:
- the frame record is at x29, and the tail call count pair is right
below it;
- the callee-saved registers follow as 16 byte pairs, in the order that
find_used_callee_regs() lists them, with the first register of a pair
at the lower address. The i-th pushed register is therefore at
-32 - 16 * (i / 2) + 8 * (i % 2);
- an exception boundary pushes x19-x28 in order, so R6-R9 take the first
four slots.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
arch/arm64/net/bpf_jit_comp.c | 37 +++++++++++++++++++++++++++++++++++
arch/x86/net/bpf_jit_comp.c | 33 +++++++++++++++++++++++++++++++
include/linux/bpf.h | 6 ++++++
3 files changed, 76 insertions(+)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index b8d765d6ab07..83a858c6d472 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -854,6 +854,42 @@ static void find_used_callee_regs(struct jit_ctx *ctx)
ctx->nr_used_callee_reg = i;
}
+/*
+ * Offset from the frame pointer of the @idx-th register pushed by push_callee_regs(). The
+ * frame record sits at the frame pointer, the tail call count pair right below it, and
+ * then the pushed pairs, each 16 bytes with its first register at the lower address.
+ */
+static int callee_reg_slot_off(int idx)
+{
+ return -32 - 16 * (idx / 2) + 8 * (idx % 2);
+}
+
+/*
+ * Record where push_callee_regs() saved the caller's R6..R9, relative to the frame
+ * pointer that arch_bpf_stack_walk() reports for this frame, for the exception unwinder.
+ * An exception boundary pushes x19..x28 in order, so R6..R9 take the first four slots.
+ */
+static void record_callee_reg_off(struct jit_ctx *ctx)
+{
+ struct bpf_prog_aux *aux = ctx->prog->aux;
+ int i, j;
+
+ if (aux->exception_boundary) {
+ for (i = 0; i < 4; i++)
+ aux->callee_reg_off[i] = callee_reg_slot_off(i);
+ return;
+ }
+ for (i = 0; i < 4; i++) {
+ aux->callee_reg_off[i] = 0;
+ for (j = 0; j < ctx->nr_used_callee_reg; j++) {
+ if (ctx->used_callee_reg[j] != bpf2a64[BPF_REG_6 + i])
+ continue;
+ aux->callee_reg_off[i] = callee_reg_slot_off(j);
+ break;
+ }
+ }
+}
+
/* Save callee-saved registers */
static void push_callee_regs(struct jit_ctx *ctx)
{
@@ -884,6 +920,7 @@ static void push_callee_regs(struct jit_ctx *ctx)
emit(A64_PUSH(reg1, A64_ZR, A64_SP), ctx);
}
}
+ record_callee_reg_off(ctx);
}
/* Restore callee-saved registers */
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 5bc744c42f7f..19fef39f1445 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -382,6 +382,24 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
*pprog = prog;
}
+/*
+ * Record where push_callee_regs() saved the caller's R6..R9 relative to rbp, for the
+ * exception unwinder. @off is the offset of rsp from rbp right before the first push.
+ */
+static void record_callee_reg_off(struct bpf_prog *prog, s32 off, bool *callee_regs_used)
+{
+ int i;
+
+ for (i = 0; i < 4; i++) {
+ if (!callee_regs_used[i]) {
+ prog->aux->callee_reg_off[i] = 0;
+ continue;
+ }
+ off -= 8;
+ prog->aux->callee_reg_off[i] = off;
+ }
+}
+
/* add rsp, depth */
static void emit_add_rsp(u8 **pprog, u32 depth)
{
@@ -2402,6 +2420,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* it has to be saved around calls of helpers and kfuncs */
bool arena_sp_save = arena_sp && calls_clobber_arena_sp(bpf_prog);
s32 arena_sp_off = 0;
+ s32 callee_regs_off;
int err;
stack_depth = bpf_prog->aux->stack_depth;
@@ -2472,6 +2491,20 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
push_callee_regs(&prog, callee_regs_used);
}
+ /*
+ * Record where the pushes above saved the caller's R6..R9, relative to the
+ * rbp that arch_bpf_stack_walk() reports for this frame. Below rbp lie, from
+ * high to low addresses: round_up(stack_depth, 8) bytes of program stack
+ * (none with a private stack), the tail call count and its pointer (16 bytes)
+ * when tail_call_reachable, r12 when exception_boundary or arena_vm_start,
+ * and then rbx, r13, r14 and r15 in push order, each only when pushed.
+ */
+ callee_regs_off = -(s32)round_up(stack_depth, 8) - (tail_call_reachable ? 16 : 0);
+ if (bpf_prog->aux->exception_boundary || arena_vm_start)
+ callee_regs_off -= 8;
+ record_callee_reg_off(bpf_prog, callee_regs_off,
+ bpf_prog->aux->exception_boundary ? all_callee_regs_used : callee_regs_used);
+
/* Compute callee-saved register area size. */
callee_saved_size = 0;
if (bpf_prog->aux->exception_boundary || arena_vm_start)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 520fd72e21fd..da42dbd32fd4 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1997,6 +1997,12 @@ struct bpf_prog_aux {
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
u16 stack_arg_sp_adjust;
u16 freplace_link_cnt; /* counts freplace links extending this prog */
+ /*
+ * Offsets from the native frame pointer of this JITed program at which
+ * its prologue saved the caller's R6..R9, or 0 when that register is not
+ * saved. Filled by the JIT; consumed by the exception unwinder.
+ */
+ s16 callee_reg_off[4];
#ifdef CONFIG_SECURITY
void *security;
#endif
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 02/29] bpf: Add bpf_throw_unwind() and an arch entry hook
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
2026-11-11 12:56 ` [PATCH bpf-next v1 01/29] bpf: Record where JITs save callee-saved registers Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:45 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 03/29] bpf, x86: Capture registers in a native bpf_throw() entry Kumar Kartikeya Dwivedi
` (27 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently bpf_throw() is a C function. It walks the stack to the frame of
the main program and calls its exception callback, which resets the stack
to that frame and returns from the program. To restore the callee-saved
registers of the frames it discards, the unwinder needs their values at
the throw site. By the time C code runs, the compiler may have clobbered
them.
Move the body of bpf_throw() into bpf_throw_unwind(), which takes the
callee-saved registers of the throw site, R6-R9, along with the cookie:
- An architecture can provide a native entry for the kfunc, which
captures the registers and then calls bpf_throw_unwind().
bpf_arch_throw_entry() returns that entry.
- The verifier redirects a call to bpf_throw() to the entry when it
specializes the kfunc. The JIT still finds the function model of the
call through the specialized descriptor.
- Without an entry, bpf_throw() calls bpf_throw_unwind() with NULL, as
before.
Nothing uses the registers yet.
bpf_throw_unwind() never returns. It transfers to a JITed callback, which
returns to the kernel code that called the program. On arm64 with a
shadow call stack, the function's prologue would push a return address
that is never popped. The caller of the program would then pop it instead
of its own return address, and trap when it returns. So:
- Mark the function __noscs, which keeps a static shadow call stack out
of it.
- A dynamic shadow call stack is patched in at boot, in place of the
pointer authentication instructions of each function. Under
CONFIG_DYNAMIC_SCS, build the function without pointer authentication
but with branch target identification, as drivers/misc/lkdtm/cfi.c
does for the same reason.
The function loses nothing by this, since it never returns through its
return address.
For the same reason, mark bpf_throw_unwind() notrace, and add it and
bpf_throw() to the functions that an fexit program cannot attach to: a
function that does not return never leaves the trampoline.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf.h | 1 +
include/linux/bpf_throw.h | 52 +++++++++++++++++++++++++++++++++++++++
kernel/bpf/core.c | 9 +++++++
kernel/bpf/helpers.c | 17 ++++++++++---
kernel/bpf/verifier.c | 7 ++++++
5 files changed, 83 insertions(+), 3 deletions(-)
create mode 100644 include/linux/bpf_throw.h
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index da42dbd32fd4..b01820e67795 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -7,6 +7,7 @@
#include <uapi/linux/bpf.h>
#include <uapi/linux/filter.h>
#include <linux/bpf_defs.h>
+#include <linux/bpf_throw.h>
#include <crypto/sha2.h>
#include <linux/workqueue.h>
diff --git a/include/linux/bpf_throw.h b/include/linux/bpf_throw.h
new file mode 100644
index 000000000000..57d974458750
--- /dev/null
+++ b/include/linux/bpf_throw.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * The interface between the architecture entries of bpf_throw() and the
+ * generic unwinder.
+ */
+#ifndef _LINUX_BPF_THROW_H
+#define _LINUX_BPF_THROW_H
+
+/* The callee-saved registers the unwinder tracks: R6..R9. */
+#define BPF_THROW_NR_CALLEE_SAVED 4
+/* The size of struct bpf_throw_regs, for the entries. */
+#define BPF_THROW_REGS_SIZE (8 * BPF_THROW_NR_CALLEE_SAVED)
+
+#ifndef __ASSEMBLER__
+#include <linux/build_bug.h>
+#include <linux/types.h>
+
+/*
+ * The callee-saved registers captured at a throw site, indexed like
+ * bpf_prog_aux::callee_reg_off.
+ */
+struct bpf_throw_regs {
+ u64 saved[BPF_THROW_NR_CALLEE_SAVED];
+};
+static_assert(sizeof(struct bpf_throw_regs) == BPF_THROW_REGS_SIZE);
+
+/*
+ * bpf_throw_unwind() never returns: it transfers to a JITed callback, which
+ * returns to the kernel code that called the program. With a shadow call
+ * stack the return address its prologue pushed would never be popped, and the
+ * program's caller would pop it in place of its own. __noscs keeps a static
+ * shadow call stack out of the function. A dynamic one is patched in at boot
+ * in place of the pointer authentication of every function that has it, see
+ * UNWIND_PATCH_PAC_INTO_SCS, so the function has to do without that as well,
+ * which costs it nothing, as it never returns through it; the branch target
+ * identification stays.
+ */
+#if defined(CONFIG_DYNAMIC_SCS) && defined(CONFIG_ARM64_BTI_KERNEL)
+#define __bpf_unwind_entry __noscs __attribute__((__target__("branch-protection=bti")))
+#elif defined(CONFIG_DYNAMIC_SCS)
+#define __bpf_unwind_entry __noscs __attribute__((__target__("branch-protection=none")))
+#else
+#define __bpf_unwind_entry __noscs
+#endif
+
+void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie);
+/* The native entry of bpf_throw(), on an architecture that has one. */
+void bpf_throw_entry(void);
+void *bpf_arch_throw_entry(void);
+#endif /* __ASSEMBLER__ */
+
+#endif /* _LINUX_BPF_THROW_H */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index f88a1c041a7a..7a25d1cb2c30 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3930,6 +3930,15 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
{
}
+/*
+ * The native entry of bpf_throw(), which captures the BPF register file of
+ * the throw site, or NULL when the architecture calls the kfunc directly.
+ */
+void * __weak bpf_arch_throw_entry(void)
+{
+ return NULL;
+}
+
bool __weak bpf_jit_supports_timed_may_goto(void)
{
return false;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 937a0d7d6839..de31fdd18411 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3460,7 +3460,13 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
return false;
}
-__bpf_kfunc void bpf_throw(u64 cookie)
+/*
+ * Unwind the stack to the exception callback of the main program. @regs holds
+ * the callee-saved registers of the throw site, or is NULL when the
+ * architecture has no native entry and the kfunc was called directly.
+ */
+__bpf_unwind_entry
+notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
{
struct bpf_throw_ctx ctx = {};
@@ -3472,8 +3478,8 @@ __bpf_kfunc void bpf_throw(u64 cookie)
WARN_ON_ONCE(!ctx.cnt);
/*
* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
- * deeper stack depths than ctx.sp as we do not return from bpf_throw,
- * which skips compiler generated instrumentation to do the same. Some
+ * deeper stack depths than ctx.sp as we do not return from here, which
+ * skips compiler generated instrumentation to do the same. Some
* architectures cannot recover sp while unwinding, so fall back to bp.
*/
kasan_unpoison_task_stack_below((void *)(long)(ctx.sp ?: ctx.bp));
@@ -3481,6 +3487,11 @@ __bpf_kfunc void bpf_throw(u64 cookie)
WARN(1, "A call to BPF exception callback should never return\n");
}
+__bpf_kfunc void bpf_throw(u64 cookie)
+{
+ bpf_throw_unwind(NULL, cookie);
+}
+
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 20b1b9ff574c..c5b4025ee222 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -24434,6 +24434,8 @@ BTF_ID(func, do_exit)
BTF_ID(func, do_group_exit)
BTF_ID(func, kthread_complete_and_exit)
BTF_ID(func, make_task_dead)
+BTF_ID(func, bpf_throw)
+BTF_ID(func, bpf_throw_unwind)
BTF_SET_END(noreturn_deny)
static bool can_be_sleepable(struct bpf_prog *prog)
@@ -24793,6 +24795,11 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc
} else if (func_id == special_kfunc_list[KF_bpf_arena_free_pages]) {
if (env->insn_aux_data[insn_idx].non_sleepable)
addr = (unsigned long)bpf_arena_free_pages_non_sleepable;
+ } else if (func_id == special_kfunc_list[KF_bpf_throw]) {
+ void *entry = bpf_arch_throw_entry();
+
+ if (entry)
+ addr = (unsigned long)entry;
}
desc->addr = addr;
return 0;
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 02/29] bpf: Add bpf_throw_unwind() and an arch entry hook
2026-11-11 12:57 ` [PATCH bpf-next v1 02/29] bpf: Add bpf_throw_unwind() and an arch entry hook Kumar Kartikeya Dwivedi
@ 2026-11-11 14:45 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 14:45 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] bpf: missing __bpf_unwind_entry and notrace on non-returning
bpf_throw()
- [Medium] bpf: missing noinline and __used/__retain/__noclone on
bpf_throw_unwind()
It also noticed 1 related pre-existing issue this patch does not
introduce:
- [Critical] bpf: unprotected dentry mutation via static kfunc rewrite
in specialize_kfunc() [1]
--
commit 83fef6290827fc9875d42699c4b52cdbf2a7cf72
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Add bpf_throw_unwind() and an arch entry hook
This commit moves the stack-unwinding body of bpf_throw() into
bpf_throw_unwind() and adds a weak bpf_arch_throw_entry() hook so an
architecture can capture callee-saved registers before unwinding.
> diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
> index 937a0d7d68390..de31fdd18411e 100644
> --- a/kernel/bpf/helpers.c
> +++ b/kernel/bpf/helpers.c
> @@ -3460,7 +3460,13 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
> return false;
> }
>
> -__bpf_kfunc void bpf_throw(u64 cookie)
> +/*
> + * Unwind the stack to the exception callback of the main program. @regs holds
> + * the callee-saved registers of the throw site, or is NULL when the
> + * architecture has no native entry and the kfunc was called directly.
> + */
> +__bpf_unwind_entry
> +notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
> {
> struct bpf_throw_ctx ctx = {};
[Severity: Medium]
Should bpf_throw_unwind() keep the noinline, __used, __retain, and
__noclone attributes that bpf_throw() had via __bpf_kfunc?
Without noinline, the compiler can inline bpf_throw_unwind(NULL, cookie)
into its same-file caller bpf_throw(), running the unwinding body inside a
function that is neither __noscs nor notrace. And on architectures without
an assembly caller to bpf_throw_unwind(), LTO, --gc-sections, or IPA-SRA can
drop or clone the standalone symbol so resolve_btfids fails on
BTF_ID(func, bpf_throw_unwind) in noreturn_deny.
> @@ -3481,6 +3487,11 @@ __bpf_kfunc void bpf_throw(u64 cookie)
> WARN(1, "A call to BPF exception callback should never return\n");
> }
>
> +__bpf_kfunc void bpf_throw(u64 cookie)
> +{
> + bpf_throw_unwind(NULL, cookie);
> +}
[Severity: High]
Does bpf_throw() also need __bpf_unwind_entry and notrace?
Whenever bpf_arch_throw_entry() returns NULL, specialize_kfunc() leaves
calls to bpf_throw() targeting this C wrapper, which never returns either.
Under CONFIG_FRAME_POINTER=y (-fno-optimize-sibling-calls), bpf_throw()
cannot tail-call bpf_throw_unwind(), so on arm64 with shadow call stack
enabled its prologue still pushes a return address onto the shadow call
stack that is never popped, corrupting the return address of the BPF
program's caller. Likewise, without notrace, function_graph tracing on
bpf_throw() pushes an entry onto current->ret_stack that is never popped on
unwind.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 20b1b9ff574c7..c5b4025ee2223 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -24793,6 +24795,11 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc
> } else if (func_id == special_kfunc_list[KF_bpf_arena_free_pages]) {
> if (env->insn_aux_data[insn_idx].non_sleepable)
> addr = (unsigned long)bpf_arena_free_pages_non_sleepable;
> + } else if (func_id == special_kfunc_list[KF_bpf_throw]) {
> + void *entry = bpf_arch_throw_entry();
> +
> + if (entry)
> + addr = (unsigned long)entry;
> }
> desc->addr = addr;
> return 0;
[Severity: Critical]
This problem wasn't introduced by this patch, but can specialize_kfunc()
cause an unlocked dentry xattr mutation [1] when rewriting
KF_bpf_set_dentry_xattr and KF_bpf_remove_dentry_xattr?
kernel/bpf/verifier.c:specialize_kfunc() {
...
} else if (func_id == special_kfunc_list[KF_bpf_set_dentry_xattr]) {
if (bpf_lsm_has_d_inode_locked(prog))
addr = (unsigned long)bpf_set_dentry_xattr_locked;
} else if (func_id == special_kfunc_list[KF_bpf_remove_dentry_xattr]) {
if (bpf_lsm_has_d_inode_locked(prog))
addr = (unsigned long)bpf_remove_dentry_xattr_locked;
...
}
The rewrite checks only whether the LSM hook holds i_rwsem on the hook's
own dentry, not whether the dentry argument passed to the kfunc is that
same dentry. If a program attached to such a hook passes an unrelated
dentry (for example, obtained via bpf_task_get_open_file()), the _locked
variant runs __vfs_setxattr() or __vfs_removexattr() on that dentry without
holding its inode's i_rwsem.
[1] https://sashiko.dev/#/bug/linux-1f1b04a2-315c-40c0-91e1-333f3c841441
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=2
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 03/29] bpf, x86: Capture registers in a native bpf_throw() entry
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
2026-11-11 12:56 ` [PATCH bpf-next v1 01/29] bpf: Record where JITs save callee-saved registers Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 02/29] bpf: Add bpf_throw_unwind() and an arch entry hook Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 04/29] bpf, arm64: " Kumar Kartikeya Dwivedi
` (26 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add bpf_throw_entry, which the verifier calls instead of bpf_throw(). It
saves the BPF registers of the throw site, R0-R9, on the stack in the
layout of struct bpf_throw_regs, passes the cookie from R1, and calls
bpf_throw_unwind(). That function does not return, so a ud2 follows the
call.
The unwinder needs the callee-saved R6-R9 to restore them for the frames
it discards. The entry also captures R0-R5:
- later patches describe which register holds a reference at a throw
site, and any of R0-R9 can hold one there;
- bpf_throw() does not return, so the compiler has no reason to move a
reference out of a caller-saved register before the call.
R0-R5 are stored before R6-R9, so the unwinder can index the saved
registers by BPF register number. An architecture may append callee-saved
registers that do not map to a BPF register after R9. None does yet.
The JIT calls the entry directly through __bpf_call_base + imm, so it
needs no ENDBR. objtool generates ORC data for its frame, so the stack
walk in bpf_throw_unwind() passes through the entry to the BPF frames,
where the frame pointer fallback of the ORC unwinder takes over as
before.
The entry calls bpf_throw_unwind(), which is only built with
CONFIG_BPF_SYSCALL. Build the entry with CONFIG_BPF_SYSCALL as well, and
have the JIT return it only then. The JIT itself is built with
CONFIG_BPF_JIT alone, for cBPF.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
arch/x86/net/Makefile | 1 +
arch/x86/net/bpf_jit_comp.c | 8 +++++++
arch/x86/net/bpf_throw.S | 43 +++++++++++++++++++++++++++++++++++++
include/linux/bpf_throw.h | 17 ++++++++++-----
4 files changed, 64 insertions(+), 5 deletions(-)
create mode 100644 arch/x86/net/bpf_throw.S
diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
index dddbefc0f439..daef0aa8b28c 100644
--- a/arch/x86/net/Makefile
+++ b/arch/x86/net/Makefile
@@ -7,4 +7,5 @@ ifeq ($(CONFIG_X86_32),y)
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
else
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+ obj-$(CONFIG_BPF_SYSCALL) += bpf_throw.o
endif
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 19fef39f1445..bdfc5329a00c 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -5135,6 +5135,14 @@ void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp
#endif
}
+#ifdef CONFIG_BPF_SYSCALL
+/* arch/x86/net/bpf_throw.S, which calls bpf_throw_unwind(); both need the syscall. */
+void *bpf_arch_throw_entry(void)
+{
+ return (void *)bpf_throw_entry;
+}
+#endif
+
void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,
struct bpf_prog *new, struct bpf_prog *old)
{
diff --git a/arch/x86/net/bpf_throw.S b/arch/x86/net/bpf_throw.S
new file mode 100644
index 000000000000..67983e5077a6
--- /dev/null
+++ b/arch/x86/net/bpf_throw.S
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/linkage.h>
+#include <asm/nospec-branch.h>
+
+ .code64
+ .section .text, "ax"
+
+/*
+ * Native entry of the bpf_throw() kfunc. The verifier retargets the call to
+ * bpf_throw() here, so the BPF register file of the throw site is still intact
+ * on entry. Capture it on the stack as a struct bpf_throw_regs and hand it to
+ * bpf_throw_unwind(), which never returns.
+ */
+SYM_FUNC_START(bpf_throw_entry)
+ ANNOTATE_NOENDBR
+
+ /* Setup frame. */
+ pushq %rbp
+ movq %rsp, %rbp
+
+ /* Save R9..R0 so that [rsp + 8 * i] is r[i]. */
+ pushq %r15 /* R9 */
+ pushq %r14 /* R8 */
+ pushq %r13 /* R7 */
+ pushq %rbx /* R6 */
+ pushq %r8 /* R5 */
+ pushq %rcx /* R4 */
+ pushq %rdx /* R3 */
+ pushq %rsi /* R2 */
+ pushq %rdi /* R1 */
+ pushq %rax /* R0 */
+
+ /* bpf_throw_unwind(regs, cookie); the cookie is R1 of the caller. */
+ movq %rdi, %rsi
+ movq %rsp, %rdi
+
+ /* Emit call depth accounting for call below. */
+ CALL_DEPTH_ACCOUNT
+ call bpf_throw_unwind
+ ud2
+SYM_FUNC_END(bpf_throw_entry)
diff --git a/include/linux/bpf_throw.h b/include/linux/bpf_throw.h
index 57d974458750..2bb2c18796ac 100644
--- a/include/linux/bpf_throw.h
+++ b/include/linux/bpf_throw.h
@@ -6,21 +6,28 @@
#ifndef _LINUX_BPF_THROW_H
#define _LINUX_BPF_THROW_H
+/*
+ * The callee-saved registers of the architecture that no BPF register maps
+ * to, which an entry captures after R9. None yet.
+ */
+#define BPF_THROW_NR_EXTRA 0
/* The callee-saved registers the unwinder tracks: R6..R9. */
#define BPF_THROW_NR_CALLEE_SAVED 4
-/* The size of struct bpf_throw_regs, for the entries. */
-#define BPF_THROW_REGS_SIZE (8 * BPF_THROW_NR_CALLEE_SAVED)
+/* The size of struct bpf_throw_regs, for the entries: R0..R9 and the extra ones. */
+#define BPF_THROW_REGS_SIZE (8 * (10 + BPF_THROW_NR_EXTRA))
#ifndef __ASSEMBLER__
#include <linux/build_bug.h>
#include <linux/types.h>
+#include <uapi/linux/bpf.h>
/*
- * The callee-saved registers captured at a throw site, indexed like
- * bpf_prog_aux::callee_reg_off.
+ * The register file captured at a throw site: the BPF registers R0..R9, then
+ * the extra callee-saved registers of the architecture; from R6 on, indexed
+ * like bpf_prog_aux::callee_reg_off.
*/
struct bpf_throw_regs {
- u64 saved[BPF_THROW_NR_CALLEE_SAVED];
+ u64 r[BPF_REG_10 + BPF_THROW_NR_EXTRA];
};
static_assert(sizeof(struct bpf_throw_regs) == BPF_THROW_REGS_SIZE);
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 04/29] bpf, arm64: Capture registers in a native bpf_throw() entry
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (2 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 03/29] bpf, x86: Capture registers in a native bpf_throw() entry Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 05/29] bpf: Restore callee-saved registers from unwound frames Kumar Kartikeya Dwivedi
` (25 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add a native entry for bpf_throw() on arm64, as on x86-64. The verifier
redirects a call to bpf_throw() to bpf_throw_entry(), which:
- pushes a frame record whose return address points into the program;
- saves the BPF registers of the throw site next to it as a struct
bpf_throw_regs: R0 from x8, R1-R5 from x0-x4, and R6-R9 from x19-x22;
- calls bpf_throw_unwind() with it.
Nothing uses the registers yet.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
arch/arm64/net/Makefile | 1 +
arch/arm64/net/bpf_jit_comp.c | 8 ++++++++
arch/arm64/net/bpf_throw.S | 30 ++++++++++++++++++++++++++++++
3 files changed, 39 insertions(+)
create mode 100644 arch/arm64/net/bpf_throw.S
diff --git a/arch/arm64/net/Makefile b/arch/arm64/net/Makefile
index 3ae382bfca87..70d18f067d01 100644
--- a/arch/arm64/net/Makefile
+++ b/arch/arm64/net/Makefile
@@ -3,3 +3,4 @@
# ARM64 networking code
#
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+obj-$(CONFIG_BPF_SYSCALL) += bpf_throw.o
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 83a858c6d472..33cc50268f3a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -10,6 +10,7 @@
#include <linux/arm-smccc.h>
#include <linux/bitfield.h>
#include <linux/bpf.h>
+#include <linux/bpf_throw.h>
#include <linux/cfi.h>
#include <linux/extable.h>
#include <linux/filter.h>
@@ -3965,6 +3966,13 @@ bool bpf_jit_supports_timed_may_goto(void)
return true;
}
+#ifdef CONFIG_BPF_SYSCALL
+void *bpf_arch_throw_entry(void)
+{
+ return (void *)bpf_throw_entry;
+}
+#endif
+
bool bpf_jit_inlines_helper_call(s32 imm)
{
switch (imm) {
diff --git a/arch/arm64/net/bpf_throw.S b/arch/arm64/net/bpf_throw.S
new file mode 100644
index 000000000000..93f2d208ccc8
--- /dev/null
+++ b/arch/arm64/net/bpf_throw.S
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/linkage.h>
+#include <linux/bpf_throw.h>
+
+/*
+ * Native entry of the bpf_throw() kfunc. The verifier retargets the call to
+ * bpf_throw() here, so the BPF register file of the throw site is still intact
+ * on entry and x30 is the return address into the program. Push a frame record
+ * whose link register is that address and save the file below it as a struct
+ * bpf_throw_regs: r[i] sits at [sp + 16 + 8 * i], with R0 in x8, R1-R5 in
+ * x0-x4 and R6-R9 in x19-x22. Then hand it to bpf_throw_unwind(), which never
+ * returns.
+ */
+SYM_FUNC_START(bpf_throw_entry)
+ stp x29, x30, [sp, #-(16 + BPF_THROW_REGS_SIZE)]!
+ mov x29, sp
+ stp x8, x0, [sp, #16]
+ stp x1, x2, [sp, #32]
+ stp x3, x4, [sp, #48]
+ stp x19, x20, [sp, #64]
+ stp x21, x22, [sp, #80]
+
+ /* bpf_throw_unwind(regs, cookie); the cookie is R1 of the caller. */
+ mov x1, x0
+ add x0, sp, #16
+ bl bpf_throw_unwind
+ brk #0x1
+SYM_FUNC_END(bpf_throw_entry)
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 05/29] bpf: Restore callee-saved registers from unwound frames
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (3 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 04/29] bpf, arm64: " Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:41 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 06/29] bpf: Keep last insn's aux data in add_hidden_subprog() Kumar Kartikeya Dwivedi
` (24 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently a program that may throw is an exception boundary. Its prologue
pushes every callee-saved register that the JIT could clobber, five on
x86-64 and ten on arm64, whether or not the program uses them. The
exception callback pops them all from the boundary frame before it runs.
This restores the registers that a discarded frame clobbered, since the
epilogues of the frames between the throw site and the boundary never
run.
Every invocation of such a program pays for the pushes and pops: 18 bytes
of code on x86-64 for a program that uses none of the registers, and 80
bytes of stack on arm64.
Have the unwinder restore the registers instead, on the throw path only:
- Each JITed function already records where its prologue saved the R6-R9
of its caller. Extend this to every other callee-saved register: r12
on x86-64 and x23-x28 on arm64.
- The native entries capture the extra registers after R9, before any C
code runs.
- The unwinder starts with the values at the throw site. Walking
outwards, it takes from each frame the values that the frame saved
for its caller. After the main frame, these are the values that the
caller of the program expects.
- The exception callback receives them as its fourth argument and loads
them before it reuses the main frame, instead of popping the saved
registers of the boundary. It then sets up the rest of its frame like
any function.
The extra registers are:
- r12 on x86-64, which holds the arena base;
- x23 on arm64, the stack pointer of the stack in arena;
- x25-x28 on arm64: the BPF frame pointer, the tail call count pointer,
the private stack pointer and the arena base;
- x24 on arm64. No JITed code touches it, but the C code of the unwinder
may still be using it when it enters the callback, whose epilogue
never runs.
The boundary prologue becomes an ordinary prologue. The x86-64 stack
pointer adjustment that the callback needed to find the saved registers
of the boundary is removed along with the pops.
Architectures without a native entry, s390 and powerpc, are not affected.
There bpf_throw() calls the unwinder with NULL, the boundary prologue
still saves every register, and the callback restores them from the frame
as before.
bpf_prog_frame_aux() and bpf_throw_caller_regs() are the two steps of the
walk. A later patch reuses them to release the resources that the frames
hold.
The JITed code of the two verifier_private_stack programs that throw
changes on arm64 with the prologue, so update the expected output.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
arch/arm64/net/bpf_jit_comp.c | 126 +++++++----------
arch/arm64/net/bpf_throw.S | 7 +-
arch/x86/net/bpf_jit_comp.c | 132 +++++++-----------
arch/x86/net/bpf_throw.S | 6 +-
include/linux/bpf.h | 41 +++++-
include/linux/bpf_throw.h | 20 ++-
kernel/bpf/fixups.c | 1 -
kernel/bpf/helpers.c | 30 +++-
.../bpf/progs/verifier_private_stack.c | 12 +-
9 files changed, 190 insertions(+), 185 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 33cc50268f3a..c9f851bd5003 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -866,24 +866,29 @@ static int callee_reg_slot_off(int idx)
}
/*
- * Record where push_callee_regs() saved the caller's R6..R9, relative to the frame
- * pointer that arch_bpf_stack_walk() reports for this frame, for the exception unwinder.
- * An exception boundary pushes x19..x28 in order, so R6..R9 take the first four slots.
+ * Record where the prologue saved the caller's callee-saved registers, relative to
+ * the frame pointer that arch_bpf_stack_walk() reports for this frame, for the
+ * exception unwinder. The registers are x19..x28 in the order of struct
+ * bpf_throw_regs, R6..R9 first: the tail call count pointer is saved with the frame
+ * record, the others where push_callee_regs() put them, and x24 nowhere, since
+ * JITed code leaves it alone.
*/
static void record_callee_reg_off(struct jit_ctx *ctx)
{
struct bpf_prog_aux *aux = ctx->prog->aux;
int i, j;
- if (aux->exception_boundary) {
- for (i = 0; i < 4; i++)
- aux->callee_reg_off[i] = callee_reg_slot_off(i);
- return;
- }
- for (i = 0; i < 4; i++) {
+ BUILD_BUG_ON(BPF_THROW_NR_CALLEE_SAVED != 10);
+ for (i = 0; i < BPF_THROW_NR_CALLEE_SAVED; i++) {
+ u8 reg = A64_R(19 + i);
+
aux->callee_reg_off[i] = 0;
+ if (reg == bpf2a64[TCCNT_PTR]) {
+ aux->callee_reg_off[i] = -8;
+ continue;
+ }
for (j = 0; j < ctx->nr_used_callee_reg; j++) {
- if (ctx->used_callee_reg[j] != bpf2a64[BPF_REG_6 + i])
+ if (ctx->used_callee_reg[j] != reg)
continue;
aux->callee_reg_off[i] = callee_reg_slot_off(j);
break;
@@ -891,35 +896,25 @@ static void record_callee_reg_off(struct jit_ctx *ctx)
}
}
-/* Save callee-saved registers */
+/*
+ * Save the callee-saved registers this function uses. An exception boundary
+ * saves nothing more: the unwinder recovers the registers of the frames it
+ * discards and hands them to the exception callback.
+ */
static void push_callee_regs(struct jit_ctx *ctx)
{
int reg1, reg2, i;
- /*
- * Program acting as exception boundary should save all ARM64
- * Callee-saved registers as the exception callback needs to recover
- * all ARM64 Callee-saved registers in its epilogue.
- */
- if (ctx->prog->aux->exception_boundary) {
- emit(A64_PUSH(A64_R(19), A64_R(20), A64_SP), ctx);
- emit(A64_PUSH(A64_R(21), A64_R(22), A64_SP), ctx);
- emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);
- emit(A64_PUSH(A64_R(25), A64_R(26), A64_SP), ctx);
- emit(A64_PUSH(A64_R(27), A64_R(28), A64_SP), ctx);
- ctx->fp_used = true;
- } else {
- find_used_callee_regs(ctx);
- for (i = 0; i + 1 < ctx->nr_used_callee_reg; i += 2) {
- reg1 = ctx->used_callee_reg[i];
- reg2 = ctx->used_callee_reg[i + 1];
- emit(A64_PUSH(reg1, reg2, A64_SP), ctx);
- }
- if (i < ctx->nr_used_callee_reg) {
- reg1 = ctx->used_callee_reg[i];
- /* keep SP 16-byte aligned */
- emit(A64_PUSH(reg1, A64_ZR, A64_SP), ctx);
- }
+ find_used_callee_regs(ctx);
+ for (i = 0; i + 1 < ctx->nr_used_callee_reg; i += 2) {
+ reg1 = ctx->used_callee_reg[i];
+ reg2 = ctx->used_callee_reg[i + 1];
+ emit(A64_PUSH(reg1, reg2, A64_SP), ctx);
+ }
+ if (i < ctx->nr_used_callee_reg) {
+ reg1 = ctx->used_callee_reg[i];
+ /* keep SP 16-byte aligned */
+ emit(A64_PUSH(reg1, A64_ZR, A64_SP), ctx);
}
record_callee_reg_off(ctx);
}
@@ -927,34 +922,19 @@ static void push_callee_regs(struct jit_ctx *ctx)
/* Restore callee-saved registers */
static void pop_callee_regs(struct jit_ctx *ctx)
{
- struct bpf_prog_aux *aux = ctx->prog->aux;
int reg1, reg2, i;
- /*
- * Program acting as exception boundary pushes R23 and R24 in addition
- * to BPF callee-saved registers. Exception callback uses the boundary
- * program's stack frame, so recover these extra registers in the above
- * two cases.
- */
- if (aux->exception_boundary || aux->exception_cb) {
- emit(A64_POP(A64_R(27), A64_R(28), A64_SP), ctx);
- emit(A64_POP(A64_R(25), A64_R(26), A64_SP), ctx);
- emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
- emit(A64_POP(A64_R(21), A64_R(22), A64_SP), ctx);
- emit(A64_POP(A64_R(19), A64_R(20), A64_SP), ctx);
- } else {
- i = ctx->nr_used_callee_reg - 1;
- if (ctx->nr_used_callee_reg % 2 != 0) {
- reg1 = ctx->used_callee_reg[i];
- emit(A64_POP(reg1, A64_ZR, A64_SP), ctx);
- i--;
- }
- while (i > 0) {
- reg1 = ctx->used_callee_reg[i - 1];
- reg2 = ctx->used_callee_reg[i];
- emit(A64_POP(reg1, reg2, A64_SP), ctx);
- i -= 2;
- }
+ i = ctx->nr_used_callee_reg - 1;
+ if (ctx->nr_used_callee_reg % 2 != 0) {
+ reg1 = ctx->used_callee_reg[i];
+ emit(A64_POP(reg1, A64_ZR, A64_SP), ctx);
+ i--;
+ }
+ while (i > 0) {
+ reg1 = ctx->used_callee_reg[i - 1];
+ reg2 = ctx->used_callee_reg[i];
+ emit(A64_POP(reg1, reg2, A64_SP), ctx);
+ i -= 2;
}
}
@@ -1055,24 +1035,22 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
/* BTI landing pad for the tail call, done with a BR */
emit_bti(A64_BTI_J, ctx);
}
- push_callee_regs(ctx);
} else {
/*
- * Exception callback receives FP of Main Program as third
- * parameter
+ * The exception callback receives the main program's frame
+ * pointer as its third parameter and, as its fourth, the
+ * callee-saved registers x19..x28 the unwinder recovered from
+ * the frames it discarded: the values the main program's caller
+ * expects back. Load them, then build the rest of the frame
+ * below the main program's frame record like any function.
*/
+ for (int i = 0; i < BPF_THROW_NR_CALLEE_SAVED; i++)
+ emit(A64_LDR64I(A64_R(19 + i), A64_R(3), 8 * i), ctx);
emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
- /*
- * Main Program already pushed the frame record and the
- * callee-saved registers. The exception callback will not push
- * anything and re-use the main program's stack.
- *
- * 12 registers are on the stack
- */
- emit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);
- /* The callback may use its own BPF stack, set up fp for it. */
- ctx->fp_used = true;
+ emit(A64_MOV(1, A64_SP, A64_FP), ctx);
+ prepare_bpf_tail_call_cnt(ctx);
}
+ push_callee_regs(ctx);
/* Stack must be multiples of 16B */
ctx->stack_size = round_up(prog->aux->stack_depth, 16);
diff --git a/arch/arm64/net/bpf_throw.S b/arch/arm64/net/bpf_throw.S
index 93f2d208ccc8..776a339b3234 100644
--- a/arch/arm64/net/bpf_throw.S
+++ b/arch/arm64/net/bpf_throw.S
@@ -10,8 +10,8 @@
* on entry and x30 is the return address into the program. Push a frame record
* whose link register is that address and save the file below it as a struct
* bpf_throw_regs: r[i] sits at [sp + 16 + 8 * i], with R0 in x8, R1-R5 in
- * x0-x4 and R6-R9 in x19-x22. Then hand it to bpf_throw_unwind(), which never
- * returns.
+ * x0-x4, R6-R9 in x19-x22 and the extra callee-saved registers x23-x28 after
+ * them. Then hand it to bpf_throw_unwind(), which never returns.
*/
SYM_FUNC_START(bpf_throw_entry)
stp x29, x30, [sp, #-(16 + BPF_THROW_REGS_SIZE)]!
@@ -21,6 +21,9 @@ SYM_FUNC_START(bpf_throw_entry)
stp x3, x4, [sp, #48]
stp x19, x20, [sp, #64]
stp x21, x22, [sp, #80]
+ stp x23, x24, [sp, #96]
+ stp x25, x26, [sp, #112]
+ stp x27, x28, [sp, #128]
/* bpf_throw_unwind(regs, cookie); the cookie is R1 of the caller. */
mov x1, x0
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index bdfc5329a00c..9b7466ca1a82 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -35,8 +35,6 @@ void __asan_load8(void *p);
void __asan_store8(void *p);
#endif
-static bool all_callee_regs_used[4] = {true, true, true, true};
-
static void *trampoline_size_image;
static int __init init_trampoline_size_image(void)
@@ -383,13 +381,19 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
}
/*
- * Record where push_callee_regs() saved the caller's R6..R9 relative to rbp, for the
- * exception unwinder. @off is the offset of rsp from rbp right before the first push.
+ * Record where the prologue saved the caller's callee-saved registers relative to rbp,
+ * for the exception unwinder: r12 when @r12 was pushed, then rbx, r13, r14 and r15
+ * as push_callee_regs() pushed them. @off is the offset of rsp from rbp right before
+ * the first push.
*/
-static void record_callee_reg_off(struct bpf_prog *prog, s32 off, bool *callee_regs_used)
+static void record_callee_reg_off(struct bpf_prog *prog, s32 off, bool *callee_regs_used,
+ bool r12)
{
int i;
+ prog->aux->callee_reg_off[4] = r12 ? off - 8 : 0;
+ if (r12)
+ off -= 8;
for (i = 0; i < 4; i++) {
if (!callee_regs_used[i]) {
prog->aux->callee_reg_off[i] = 0;
@@ -569,18 +573,21 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
/* Keep the same instruction layout. */
emit_nops(&prog, 3); /* nop3 */
}
- /* Exception callback receives FP as third parameter */
+ /*
+ * The exception callback receives the main program's frame pointer as
+ * its third parameter and, as its fourth, the callee-saved registers
+ * the unwinder recovered from the frames it discarded, rbx, r13, r14,
+ * r15 and r12 in that order: the values the main program's caller
+ * expects back. Load them, then reuse the main program's frame.
+ */
if (is_exception_cb) {
- EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
- EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
- /* The main frame must have exception_boundary as true, so we
- * first restore those callee-saved regs from stack, before
- * reusing the stack frame.
- */
- pop_callee_regs(&prog, all_callee_regs_used);
- pop_r12(&prog);
- /* Reset the stack frame. */
- EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */
+ EMIT3(0x48, 0x8B, 0x19); /* mov rbx, [rcx] */
+ EMIT4(0x4C, 0x8B, 0x69, 0x08); /* mov r13, [rcx + 8] */
+ EMIT4(0x4C, 0x8B, 0x71, 0x10); /* mov r14, [rcx + 16] */
+ EMIT4(0x4C, 0x8B, 0x79, 0x18); /* mov r15, [rcx + 24] */
+ EMIT4(0x4C, 0x8B, 0x61, 0x20); /* mov r12, [rcx + 32] */
+ EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
+ EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */
} else {
EMIT1(0x55); /* push rbp */
EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */
@@ -856,14 +863,9 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
/* Inc tail_call_cnt if the slot is populated. */
EMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */
- if (bpf_prog->aux->exception_boundary) {
- pop_callee_regs(&prog, all_callee_regs_used);
+ pop_callee_regs(&prog, callee_regs_used);
+ if (bpf_prog_uses_arena_insns(bpf_prog))
pop_r12(&prog);
- } else {
- pop_callee_regs(&prog, callee_regs_used);
- if (bpf_prog_uses_arena_insns(bpf_prog))
- pop_r12(&prog);
- }
/* Pop tail_call_cnt_ptr. */
EMIT1(0x58); /* pop rax */
@@ -922,14 +924,9 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
/* Inc tail_call_cnt if the slot is populated. */
EMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */
- if (bpf_prog->aux->exception_boundary) {
- pop_callee_regs(&prog, all_callee_regs_used);
+ pop_callee_regs(&prog, callee_regs_used);
+ if (bpf_prog_uses_arena_insns(bpf_prog))
pop_r12(&prog);
- } else {
- pop_callee_regs(&prog, callee_regs_used);
- if (bpf_prog_uses_arena_insns(bpf_prog))
- pop_r12(&prog);
- }
/* Pop tail_call_cnt_ptr. */
EMIT1(0x58); /* pop rax */
@@ -2475,49 +2472,31 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
bpf_prog->aux->ksym.fp_start = prog - temp;
- /* Exception callback will clobber callee regs for its own use, and
- * restore the original callee regs from main prog's stack frame.
+ /*
+ * Save the callee-saved registers this function uses. An exception
+ * boundary saves nothing more: the unwinder recovers the registers of
+ * the frames it discards and hands them to the exception callback.
*/
- if (bpf_prog->aux->exception_boundary) {
- /* We also need to save r12, which is not mapped to any BPF
- * register, as we throw after entry into the kernel, which may
- * overwrite r12.
- */
+ if (arena_vm_start)
push_r12(&prog);
- push_callee_regs(&prog, all_callee_regs_used);
- } else {
- if (arena_vm_start)
- push_r12(&prog);
- push_callee_regs(&prog, callee_regs_used);
- }
+ push_callee_regs(&prog, callee_regs_used);
/*
- * Record where the pushes above saved the caller's R6..R9, relative to the
- * rbp that arch_bpf_stack_walk() reports for this frame. Below rbp lie, from
- * high to low addresses: round_up(stack_depth, 8) bytes of program stack
- * (none with a private stack), the tail call count and its pointer (16 bytes)
- * when tail_call_reachable, r12 when exception_boundary or arena_vm_start,
- * and then rbx, r13, r14 and r15 in push order, each only when pushed.
+ * Record where the pushes above saved the caller's registers, relative to
+ * the rbp that arch_bpf_stack_walk() reports for this frame. Below rbp lie,
+ * from high to low addresses: round_up(stack_depth, 8) bytes of program
+ * stack (none with a private stack), the tail call count and its pointer
+ * (16 bytes) when tail_call_reachable, r12 when arena_vm_start, and then
+ * rbx, r13, r14 and r15 in push order, each only when pushed.
*/
callee_regs_off = -(s32)round_up(stack_depth, 8) - (tail_call_reachable ? 16 : 0);
- if (bpf_prog->aux->exception_boundary || arena_vm_start)
- callee_regs_off -= 8;
- record_callee_reg_off(bpf_prog, callee_regs_off,
- bpf_prog->aux->exception_boundary ? all_callee_regs_used : callee_regs_used);
+ record_callee_reg_off(bpf_prog, callee_regs_off, callee_regs_used, arena_vm_start);
/* Compute callee-saved register area size. */
- callee_saved_size = 0;
- if (bpf_prog->aux->exception_boundary || arena_vm_start)
- callee_saved_size += 8; /* r12 */
- if (bpf_prog->aux->exception_boundary) {
- callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
- } else {
- int j;
-
- for (j = 0; j < 4; j++)
- if (callee_regs_used[j])
- callee_saved_size += 8;
- }
+ callee_saved_size = arena_vm_start ? 8 : 0; /* r12 */
+ for (int j = 0; j < 4; j++)
+ if (callee_regs_used[j])
+ callee_saved_size += 8;
/*
* Base offset from rbp for translating BPF outgoing args 7+
* to native offsets. BPF uses negative offsets from r11
@@ -2546,15 +2525,6 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
if (arena_sp)
outgoing_rsp = out_stack_arg_cnt * 8;
- /*
- * bpf_throw() sees rsp of this frame as it is at a call. The callee-saved
- * regs are above the outgoing args, or above r9, which is pushed around
- * every call when the stack is private. With BPF_REG_ARENA_SP the slot
- * that keeps r9 over calls is between the two.
- */
- if (bpf_prog->aux->exception_boundary)
- bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp +
- (priv_frame_ptr || arena_sp_save ? 8 : 0);
emit_sub_rsp(&prog, outgoing_rsp);
if (arena_vm_start)
@@ -3381,10 +3351,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
if (err < 0)
return err;
ip += err;
- /*
- * Other kfuncs leave rsp alone. bpf_throw() is one of
- * them, see stack_arg_sp_adjust.
- */
+ /* Other kfuncs leave rsp alone. */
arena_sp_args = arena_sp && kfunc_has_stack_args(fm);
if (arena_sp_args) {
u8 *start = prog;
@@ -3762,14 +3729,9 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
/* Deallocate outgoing args 7+ area and the slot of BPF_REG_ARENA_SP. */
emit_add_rsp(&prog, outgoing_rsp + (arena_sp_save ? 8 : 0));
- if (bpf_prog->aux->exception_boundary) {
- pop_callee_regs(&prog, all_callee_regs_used);
+ pop_callee_regs(&prog, callee_regs_used);
+ if (arena_vm_start)
pop_r12(&prog);
- } else {
- pop_callee_regs(&prog, callee_regs_used);
- if (arena_vm_start)
- pop_r12(&prog);
- }
EMIT1(0xC9); /* leave */
bpf_prog->aux->ksym.fp_end = prog - temp;
diff --git a/arch/x86/net/bpf_throw.S b/arch/x86/net/bpf_throw.S
index 67983e5077a6..856b3f216a76 100644
--- a/arch/x86/net/bpf_throw.S
+++ b/arch/x86/net/bpf_throw.S
@@ -10,7 +10,8 @@
/*
* Native entry of the bpf_throw() kfunc. The verifier retargets the call to
* bpf_throw() here, so the BPF register file of the throw site is still intact
- * on entry. Capture it on the stack as a struct bpf_throw_regs and hand it to
+ * on entry. Capture it on the stack as a struct bpf_throw_regs, along with
+ * r12, the callee-saved register no BPF register maps to, and hand it to
* bpf_throw_unwind(), which never returns.
*/
SYM_FUNC_START(bpf_throw_entry)
@@ -20,7 +21,8 @@ SYM_FUNC_START(bpf_throw_entry)
pushq %rbp
movq %rsp, %rbp
- /* Save R9..R0 so that [rsp + 8 * i] is r[i]. */
+ /* Save r12, then R9..R0 so that [rsp + 8 * i] is r[i]. */
+ pushq %r12 /* r[10] */
pushq %r15 /* R9 */
pushq %r14 /* R8 */
pushq %r13 /* R7 */
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index b01820e67795..2b9adc4866bc 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1995,15 +1995,21 @@ struct bpf_prog_aux {
int cgroup_atype; /* enum cgroup_bpf_attach_type */
struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
char name[BPF_OBJ_NAME_LEN];
- u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
- u16 stack_arg_sp_adjust;
+ /*
+ * The exception callback of the main program: @cookie is the value of
+ * the throw, @sp and @bp the native stack and frame pointers of the main
+ * program's frame, @saved the callee-saved registers the unwinder
+ * restored, R6..R9 then the architecture's own, see struct bpf_throw_regs.
+ */
+ u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 saved, u64);
u16 freplace_link_cnt; /* counts freplace links extending this prog */
/*
* Offsets from the native frame pointer of this JITed program at which
- * its prologue saved the caller's R6..R9, or 0 when that register is not
- * saved. Filled by the JIT; consumed by the exception unwinder.
+ * its prologue saved the caller's callee-saved registers, R6..R9 then
+ * the architecture's own, or 0 when that register is not saved. Filled
+ * by the JIT; consumed by the exception unwinder.
*/
- s16 callee_reg_off[4];
+ s16 callee_reg_off[BPF_THROW_NR_CALLEE_SAVED];
#ifdef CONFIG_SECURITY
void *security;
#endif
@@ -4436,6 +4442,31 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog)
return prog->aux->func_idx != 0;
}
+/*
+ * The aux of the JITed function a frame of @prog runs: the symbol of a
+ * program JITed as a set of functions covers its first function.
+ */
+static inline const struct bpf_prog_aux *bpf_prog_frame_aux(const struct bpf_prog *prog)
+{
+ return !prog->is_func && prog->aux->func ? prog->aux->func[0]->aux : prog->aux;
+}
+
+/*
+ * The exception unwinder leaves a frame of the JITed function @fn, whose
+ * native frame pointer is @bp, for its caller. @saved holds the callee-saved
+ * registers the frame has, R6..R9 then the architecture's own: the caller has
+ * the same, but for the ones the prologue of the frame saved, which are where
+ * it saved them.
+ */
+static inline void bpf_throw_caller_regs(const struct bpf_prog_aux *fn, u64 bp, u64 *saved)
+{
+ int i;
+
+ for (i = 0; i < BPF_THROW_NR_CALLEE_SAVED; i++)
+ if (fn->callee_reg_off[i])
+ saved[i] = *(u64 *)(long)(bp + fn->callee_reg_off[i]);
+}
+
const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off);
struct bpf_linfo_source {
const char *file;
diff --git a/include/linux/bpf_throw.h b/include/linux/bpf_throw.h
index 2bb2c18796ac..6449757c18a9 100644
--- a/include/linux/bpf_throw.h
+++ b/include/linux/bpf_throw.h
@@ -8,11 +8,23 @@
/*
* The callee-saved registers of the architecture that no BPF register maps
- * to, which an entry captures after R9. None yet.
+ * to, which the entries capture after R9 and the unwinder restores along
+ * with R6..R9 when it discards frames: r12 on x86-64, which holds the arena
+ * base, and x23..x28 on arm64, of which x23 is the stack pointer of the
+ * stack in arena and x25..x28 are the BPF frame pointer, the tail call count
+ * pointer, the private stack pointer and the arena base, while x24 is left
+ * alone by JITed code but may be in use by the C code of the unwinder when
+ * it enters the exception callback, whose epilogue never runs.
*/
-#define BPF_THROW_NR_EXTRA 0
-/* The callee-saved registers the unwinder tracks: R6..R9. */
-#define BPF_THROW_NR_CALLEE_SAVED 4
+#if defined(CONFIG_X86_64)
+#define BPF_THROW_NR_EXTRA 1
+#elif defined(CONFIG_ARM64)
+#define BPF_THROW_NR_EXTRA 6
+#else
+#define BPF_THROW_NR_EXTRA 0
+#endif
+/* The callee-saved registers the unwinder tracks: R6..R9, then the extra ones. */
+#define BPF_THROW_NR_CALLEE_SAVED (4 + BPF_THROW_NR_EXTRA)
/* The size of struct bpf_throw_regs, for the entries: R0..R9 and the extra ones. */
#define BPF_THROW_REGS_SIZE (8 * (10 + BPF_THROW_NR_EXTRA))
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 1d33a78d9373..eb5a5bca5c8d 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -2103,7 +2103,6 @@ static int jit_subprogs(struct bpf_verifier_env *env)
prog->aux->real_func_cnt = env->subprog_cnt;
prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
prog->aux->exception_boundary = func[0]->aux->exception_boundary;
- prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
bpf_prog_jit_attempt_done(prog);
return 0;
out_free:
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index de31fdd18411..0164f183aade 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3434,6 +3434,13 @@ struct bpf_throw_ctx {
u64 sp;
u64 bp;
int cnt;
+ /*
+ * The callee-saved registers of the frame being walked, R6-R9 then the
+ * architecture's own: what the throw site had, then what each frame
+ * recovered of its caller's. After the main frame they are the ones
+ * its caller expects back.
+ */
+ u64 saved[BPF_THROW_NR_CALLEE_SAVED];
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
@@ -3451,6 +3458,12 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
+ /*
+ * Recover the caller's callee-saved registers: those this frame's
+ * prologue saved sit in its frame, the rest are still the values the
+ * frame itself has.
+ */
+ bpf_throw_caller_regs(bpf_prog_frame_aux(prog), bp, ctx->saved);
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
@@ -3461,15 +3474,24 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
}
/*
- * Unwind the stack to the exception callback of the main program. @regs holds
- * the callee-saved registers of the throw site, or is NULL when the
- * architecture has no native entry and the kfunc was called directly.
+ * Unwind the stack to the exception callback of the main program. @regs is the
+ * register file of the throw site, which the entry of the architecture
+ * captured: the callee-saved registers of the frames unwound are restored from
+ * it for the callback. It is NULL when the architecture has no native entry
+ * and the kfunc was called directly.
*/
__bpf_unwind_entry
notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
{
struct bpf_throw_ctx ctx = {};
+ /*
+ * Without the file, on an architecture that has no native entry, the
+ * prologue of the main program has saved every callee-saved register
+ * and the callback restores them from its frame, not from @saved.
+ */
+ if (regs)
+ memcpy(ctx.saved, ®s->r[BPF_REG_6], sizeof(ctx.saved));
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
@@ -3483,7 +3505,7 @@ notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
* architectures cannot recover sp while unwinding, so fall back to bp.
*/
kasan_unpoison_task_stack_below((void *)(long)(ctx.sp ?: ctx.bp));
- ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
+ ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, (u64)(long)ctx.saved, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
diff --git a/tools/testing/selftests/bpf/progs/verifier_private_stack.c b/tools/testing/selftests/bpf/progs/verifier_private_stack.c
index e7f410a74a21..14e5c6bdd4d8 100644
--- a/tools/testing/selftests/bpf/progs/verifier_private_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_private_stack.c
@@ -287,11 +287,7 @@ __jited(" mov x29, sp")
__jited(" stp xzr, x26, [sp, #-0x10]!")
__jited(" mov x26, sp")
__jited("...")
-__jited(" stp x19, x20, [sp, #-0x10]!")
-__jited(" stp x21, x22, [sp, #-0x10]!")
-__jited(" stp x23, x24, [sp, #-0x10]!")
-__jited(" stp x25, x26, [sp, #-0x10]!")
-__jited(" stp x27, x28, [sp, #-0x10]!")
+__jited(" stp x25, x27, [sp, #-0x10]!")
__jited(" mov x27, {{.*}}")
__jited(" movk x27, {{.*}}, lsl #16")
__jited(" movk x27, {{.*}}")
@@ -303,7 +299,7 @@ __jited(" str x0, [x27]")
__jited(" mov x0, #0x0")
__jited(" bl 0x{{.*}}")
__jited(" mov x8, x0")
-__jited(" ldp x27, x28, [sp], #0x10")
+__jited(" ldp x25, x27, [sp], #0x10")
int private_stack_exception_main_prog(void)
{
asm volatile (" \
@@ -331,7 +327,7 @@ __jited("...")
__jited(" callq")
__jited(" popq %r9")
__arch_arm64
-__jited(" stp x27, x28, [sp, #-0x10]!")
+__jited(" stp x25, x27, [sp, #-0x10]!")
__jited(" mov x27, {{.*}}")
__jited(" movk x27, {{.*}}, lsl #16")
__jited(" movk x27, {{.*}}")
@@ -342,7 +338,7 @@ __jited(" mov x0, #0x2a")
__jited(" str x0, [x27]")
__jited(" bl 0x{{.*}}")
__jited(" mov x8, x0")
-__jited(" ldp x27, x28, [sp], #0x10")
+__jited(" ldp x25, x27, [sp], #0x10")
int private_stack_exception_sub_prog(void)
{
asm volatile (" \
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 05/29] bpf: Restore callee-saved registers from unwound frames
2026-11-11 12:57 ` [PATCH bpf-next v1 05/29] bpf: Restore callee-saved registers from unwound frames Kumar Kartikeya Dwivedi
@ 2026-11-11 14:41 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 14:41 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] bpf: NULL pointer dereference in bpf_throw() [1]
--
commit 1abcf2f1b7a0733f9332d1db13a62945d7715633
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Restore callee-saved registers from unwound frames
This commit restores callee-saved registers from unwound frames during
bpf_throw() unwinding and passes them to the exception callback instead of
saving and popping all callee-saved registers in the exception boundary
prologue.
> diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
> index de31fdd18411e..0164f183aade2 100644
> --- a/kernel/bpf/helpers.c
> +++ b/kernel/bpf/helpers.c
[ ... ]
> @@ -3461,15 +3474,24 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
> }
>
> /*
> - * Unwind the stack to the exception callback of the main program. @regs holds
> - * the callee-saved registers of the throw site, or is NULL when the
> - * architecture has no native entry and the kfunc was called directly.
> + * Unwind the stack to the exception callback of the main program. @regs is the
> + * register file of the throw site, which the entry of the architecture
> + * captured: the callee-saved registers of the frames unwound are restored from
> + * it for the callback. It is NULL when the architecture has no native entry
> + * and the kfunc was called directly.
> */
> __bpf_unwind_entry
> notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
> {
> struct bpf_throw_ctx ctx = {};
>
> + /*
> + * Without the file, on an architecture that has no native entry, the
> + * prologue of the main program has saved every callee-saved register
> + * and the callback restores them from its frame, not from @saved.
> + */
> + if (regs)
> + memcpy(ctx.saved, ®s->r[BPF_REG_6], sizeof(ctx.saved));
> arch_bpf_stack_walk(bpf_stack_walker, &ctx);
> WARN_ON_ONCE(!ctx.aux);
> if (ctx.aux)
> @@ -3483,7 +3505,7 @@ notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
> * architectures cannot recover sp while unwinding, so fall back to bp.
> */
> kasan_unpoison_task_stack_below((void *)(long)(ctx.sp ?: ctx.bp));
> - ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
> + ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, (u64)(long)ctx.saved, 0);
[Severity: Critical]
This problem wasn't introduced by this patch, but can this cause a NULL
pointer dereference [1] on ctx.aux if arch_bpf_stack_walk() fails to
locate the main BPF program frame?
Although the code checks whether ctx.aux is non-NULL before testing
ctx.aux->exception_boundary above, it then dereferences ctx.aux
unconditionally when invoking bpf_exception_cb().
[1] https://sashiko.dev/#/bug/linux-ba8dbe22-2e39-4c7e-90bf-e30065ad3c90
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=5
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 06/29] bpf: Keep last insn's aux data in add_hidden_subprog()
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (4 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 05/29] bpf: Restore callee-saved registers from unwound frames Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:38 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots Kumar Kartikeya Dwivedi
` (23 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
add_hidden_subprog() appends its instructions by patching the last
instruction of the program with a sequence that starts with a copy of
that instruction. Like every patch, this moves the aux data of the
patched instruction to the end of the new sequence. The aux data of the
original last instruction therefore ends up on the exit of the hidden
subprog, and the copy of the instruction, which stays in place, loses it.
Today only the JITs read the aux data of the last instruction after this
point, for indirect_target, which the patch leaves with the instruction.
But a noreturn call such as bpf_throw() can be the last instruction once
the code after it is removed, and later patches attach per-instruction
data to such calls.
Swap the two entries back after the patch, except for indirect_target.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
kernel/bpf/fixups.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index eb5a5bca5c8d..8597b443d561 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -2280,8 +2280,10 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
{
struct bpf_subprog_info *info = env->subprog_info;
int err, cnt = env->subprog_cnt;
+ struct bpf_insn_aux_data *aux;
struct bpf_insn *last;
struct bpf_prog *prog;
+ bool indirect;
/* We only reserve one slot for hidden subprogs in subprog_info. */
if (env->hidden_subprog_cnt) {
@@ -2299,6 +2301,17 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
if (err)
return err;
prog = env->prog;
+ /*
+ * The patch starts with the original last insn, but the commit moved that
+ * insn's aux data to the end of the patch: give it back to the insn. All
+ * but indirect_target, which the commit left with the insn, since an
+ * indirect jump to it lands there still, see patch_list_adj_aux().
+ */
+ aux = &env->insn_aux_data[prog->len - len];
+ indirect = aux->indirect_target;
+ swap(aux[0], aux[len - 1]);
+ aux[0].indirect_target = indirect;
+ aux[len - 1].indirect_target = false;
info[cnt + 1].start = info[cnt].start;
info[cnt].start = prog->len - len + 1;
/* The patch ends with the exit. */
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 06/29] bpf: Keep last insn's aux data in add_hidden_subprog()
2026-11-11 12:57 ` [PATCH bpf-next v1 06/29] bpf: Keep last insn's aux data in add_hidden_subprog() Kumar Kartikeya Dwivedi
@ 2026-11-11 14:38 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 14:38 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] bpf: misplaced nospec flag in adjust_insn_aux_data() [1]
--
commit f29fbac112320caad2087aefcfcc0181b977759d
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Keep last insn's aux data in add_hidden_subprog()
This commit swaps the aux data of the original last instruction and the end
of the patch in add_hidden_subprog() so that the original instruction keeps
its aux data (while leaving indirect_target on the first instruction).
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index eb5a5bca5c8d5..8597b443d5619 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -2280,8 +2280,10 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
> {
> struct bpf_subprog_info *info = env->subprog_info;
> int err, cnt = env->subprog_cnt;
> + struct bpf_insn_aux_data *aux;
> struct bpf_insn *last;
> struct bpf_prog *prog;
> + bool indirect;
>
> /* We only reserve one slot for hidden subprogs in subprog_info. */
> if (env->hidden_subprog_cnt) {
> @@ -2299,6 +2301,17 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
> if (err)
> return err;
> prog = env->prog;
> + /*
> + * The patch starts with the original last insn, but the commit moved that
> + * insn's aux data to the end of the patch: give it back to the insn. All
> + * but indirect_target, which the commit left with the insn, since an
> + * indirect jump to it lands there still, see patch_list_adj_aux().
> + */
> + aux = &env->insn_aux_data[prog->len - len];
> + indirect = aux->indirect_target;
> + swap(aux[0], aux[len - 1]);
> + aux[0].indirect_target = indirect;
> + aux[len - 1].indirect_target = false;
> info[cnt + 1].start = info[cnt].start;
> info[cnt].start = prog->len - len + 1;
> /* The patch ends with the exit. */
[Severity: Critical]
This problem wasn't introduced by this patch, but can patch_list_adj_aux()
misplace the nospec flag [1] when expanding a single instruction into a
multi-instruction sequence?
kernel/bpf/fixups.c:patch_list_adj_aux() {
...
*last = *old;
/* Recompute zext_dst: the last insn of the patch need not be the patched one. */
last->zext_dst = bpf_insn_def32(prog, &patch[p->len - 1]) >= 0;
memset(slot, 0, sizeof(*slot) * (p->len - 1));
...
/* indirect jumps to the replaced insn land on p->target */
if (last->indirect_target && p->target != p->len - 1) {
slot[p->target].indirect_target = 1;
last->indirect_target = 0;
}
}
During verification with Spectre v1 mitigation active, a bpf_loop() call can
be eligible for inlining on an architectural path while a speculative path to
the same instruction fails in check_func_args() with -EACCES, causing
do_check() to set insn_aux->nospec on the call instruction and stop verifying
that speculative path.
When bpf_optimize_bpf_loop() -> inline_bpf_loop() later replaces the call
with a 19-instruction sequence, patch_list_adj_aux() moves old aux data to
last and zeroes the preceding entries while only restoring indirect_target to
slot[p->target]. As a result, bpf_convert_ctx_accesses() emits BPF_ST_NOSPEC()
before the last instruction of the inlined loop instead of at its entry,
allowing speculative execution to enter the loop body and invoke the callback
with unverified state.
[1] https://sashiko.dev/#/bug/linux-9fe39327-eb21-4b95-9330-3ea552a3b9fc
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=6
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (5 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 06/29] bpf: Keep last insn's aux data in add_hidden_subprog() Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 15:15 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry Kumar Kartikeya Dwivedi
` (22 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Rewrites after verification need stack space that the program does not
know about:
- the may_goto count, or the register that a timed may_goto saves;
- the spill area of an inlined bpf_loop();
- the context pointer that a generated epilogue reads back.
Each user computes its own offsets below the verified stack depth and
bumps stack_depth at a subprogram boundary in its own loop. may_goto also
keeps stack_extra, so that a prologue patch at the end of
bpf_do_misc_fixups() knows how many slots to initialize, with a bug check
that nothing else uses it.
Later patches need more hidden slots, for the handles of resources that
an unwinder has to find. Add one allocator:
- bpf_reserve_stack_slots() hands out 8-byte slots, top down, below the
verified depth of a subprogram rounded up to 8 bytes. It bumps
stack_depth and stack_extra.
- bpf_stack_slot_init() records the value that a slot must hold on
entry.
- bpf_emit_stack_inits() patches all initializing stores into the
subprogram prologues in one place, in chunks that fit the instruction
buffer. Jumps to the first instruction are redirected past the stores,
as the may_goto patch did.
Convert the three users:
- The may_goto slot is reserved at the first may_goto of a subprogram
and shared by the rest. The bpf_loop() spill area works the same way.
- The epilogue slot keeps its offset by asking for the next slot before
gen_epilogue() decides whether it is needed.
The "stack size %d(extra %d) is too large" check for interpreted programs
moves into the allocator, so it covers every user. Slots now start at an
8-byte aligned offset. Before, a subprogram with an unaligned depth got
unaligned may_goto slots.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 12 ++
kernel/bpf/fixups.c | 265 +++++++++++++++++++++++------------
kernel/bpf/verifier.c | 1 +
3 files changed, 190 insertions(+), 88 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 4771c55d8f41..7a2b6507f562 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1023,6 +1023,7 @@ struct bpf_subprog_info {
u32 exit_idx;
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
u16 stack_depth; /* max. stack depth used by this function */
+ /* bytes below the verified stack depth reserved for hidden slots, see bpf_reserve_stack_slots() */
u16 stack_extra;
u16 arena_stack_depth; /* size of the frame in the stack in arena */
u32 insns_total;
@@ -1208,6 +1209,13 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
/* single container for all structs
* one verifier_env per bpf_check() call
*/
+/* Initialization of a hidden stack slot, emitted in the subprogram's prologue. */
+struct bpf_stack_init {
+ u32 subprog;
+ s16 off;
+ s32 imm;
+};
+
struct bpf_verifier_env {
u32 insn_idx;
u32 prev_insn_idx;
@@ -1233,6 +1241,8 @@ struct bpf_verifier_env {
u32 insn_array_map_cnt; /* number of used maps of type BPF_MAP_TYPE_INSN_ARRAY */
u32 id_gen; /* used to generate unique reg IDs */
u32 hidden_subprog_cnt; /* number of hidden subprogs */
+ struct bpf_stack_init *stack_inits; /* hidden slot initializations, see bpf_stack_slot_init() */
+ u32 stack_init_cnt;
int exception_callback_subprog;
bool explore_alu_limits;
bool allow_ptr_leaks;
@@ -2174,6 +2184,8 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
/* Functions in fixups.c, called from bpf_check() */
void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len);
+int bpf_reserve_stack_slots(struct bpf_verifier_env *env, int subprog, int nr, s16 *off);
+int bpf_stack_slot_init(struct bpf_verifier_env *env, int subprog, s16 off, s32 imm);
int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 8597b443d561..04d89113e596 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -918,6 +918,87 @@ int bpf_insert_recursion_checks(struct bpf_verifier_env *env)
return 0;
}
+/*
+ * Hidden stack slots. Post-verification rewrites need stack space the program
+ * does not know about: may_goto counters, the spill area of an inlined
+ * bpf_loop(), the context pointer a generated epilogue reads back. Such slots
+ * are reserved below the stack depth verification established for a
+ * subprogram, after bpf_remove_fastcall_spills_fills() has shrunk it, so that
+ * they sit directly under the program's own frame. Slots are 8 bytes, 8-byte
+ * aligned and handed out top down: the first slot of a subprogram lies right
+ * below its verified depth rounded up to 8 bytes.
+ */
+static s16 next_stack_slot_off(const struct bpf_subprog_info *info)
+{
+ return -(s32)(round_up(info->stack_depth, BPF_REG_SIZE) + BPF_REG_SIZE);
+}
+
+/* Reserve @nr slots in @subprog; @off receives the offset of the topmost one. */
+int bpf_reserve_stack_slots(struct bpf_verifier_env *env, int subprog, int nr, s16 *off)
+{
+ struct bpf_subprog_info *info = &env->subprog_info[subprog];
+ u32 depth = round_up(info->stack_depth, BPF_REG_SIZE) + nr * BPF_REG_SIZE;
+ u32 extra = depth - info->stack_depth;
+
+ if ((depth > MAX_BPF_STACK && !env->prog->jit_requested) || depth > S16_MAX + 1U) {
+ verbose(env, "stack size %d(extra %d) is too large\n", depth, info->stack_extra + extra);
+ return -EINVAL;
+ }
+ *off = next_stack_slot_off(info);
+ info->stack_depth = depth;
+ info->stack_extra += extra;
+ return 0;
+}
+
+/* Initialize the hidden slot at @off of @subprog to @imm in its prologue. */
+int bpf_stack_slot_init(struct bpf_verifier_env *env, int subprog, s16 off, s32 imm)
+{
+ struct bpf_stack_init *inits;
+ u32 cnt = env->stack_init_cnt;
+
+ inits = kvrealloc(env->stack_inits, array_size(cnt + 1, sizeof(*inits)),
+ GFP_KERNEL_ACCOUNT);
+ if (!inits)
+ return -ENOMEM;
+ inits[cnt] = (struct bpf_stack_init) { .subprog = subprog, .off = off, .imm = imm };
+ env->stack_inits = inits;
+ env->stack_init_cnt = cnt + 1;
+ return 0;
+}
+
+/*
+ * Emit the stores that initialize hidden slots at the start of each subprogram.
+ * They go in front of the first instruction, which a jump to it skips, so a
+ * loop back to the entry does not rerun the initialization. The patch list is
+ * empty on entry and committed on return.
+ */
+static int bpf_emit_stack_inits(struct bpf_verifier_env *env)
+{
+ struct bpf_insn *insn_buf = env->insn_buf;
+ u32 i, j, cnt;
+ int ret;
+
+ for (i = 0; i < env->subprog_cnt; i++) {
+ j = 0;
+ do {
+ cnt = 0;
+ for (; j < env->stack_init_cnt && cnt < INSN_BUF_SIZE; j++) {
+ const struct bpf_stack_init *init = &env->stack_inits[j];
+
+ if (init->subprog == i)
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, init->off,
+ init->imm);
+ }
+ if (!cnt)
+ break;
+ ret = bpf_patch_list_insert(env, env->subprog_info[i].start, insn_buf, cnt);
+ if (ret)
+ return ret;
+ } while (j < env->stack_init_cnt);
+ }
+ return bpf_patch_list_commit(env);
+}
+
/* Clean up dynamically allocated fields of aux data for instructions [start, ...] */
void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
{
@@ -1461,16 +1542,22 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
int epilogue_idx = 0;
if (ops->gen_epilogue) {
- epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog,
- -(subprogs[0].stack_depth + 8));
+ s16 ctx_off = next_stack_slot_off(&subprogs[0]);
+
+ epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, ctx_off);
if (epilogue_cnt >= INSN_BUF_SIZE) {
verifier_bug(env, "epilogue is too long");
return -EFAULT;
} else if (epilogue_cnt) {
+ s16 off;
+
/* Save the ARG_PTR_TO_CTX for the epilogue to use */
- subprogs[0].stack_depth += 8;
- insn_buf[0] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_1,
- -subprogs[0].stack_depth);
+ ret = bpf_reserve_stack_slots(env, 0, 1, &off);
+ if (ret)
+ return ret;
+ if (verifier_bug_if(off != ctx_off, env, "epilogue context slot moved"))
+ return -EFAULT;
+ insn_buf[0] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_1, off);
ret = bpf_patch_list_insert(env, 0, insn_buf, 1);
if (ret)
return ret;
@@ -2321,6 +2408,25 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
return 0;
}
+/*
+ * Reserve the hidden slot of the may_goto instructions of @subprog once, and
+ * @init it: the plain form keeps its count there. The timed form keeps the
+ * count and the timestamp in per-cpu memory of the prog, see struct
+ * bpf_timed_may_goto, and needs the slot only to save R0 around its insns
+ * when every register is live. @off receives the offset of the slot.
+ */
+static int may_goto_stack_slot(struct bpf_verifier_env *env, int subprog, bool init, s16 *off)
+{
+ int ret;
+
+ if (*off)
+ return 0;
+ ret = bpf_reserve_stack_slots(env, subprog, 1, off);
+ if (!ret && init)
+ ret = bpf_stack_slot_init(env, subprog, *off, BPF_MAX_LOOPS);
+ return ret;
+}
+
/*
* Expand may_goto: load the count from the stack, jump to the target of
* may_goto when it is zero, then the tail that updates the count.
@@ -2616,11 +2722,10 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
struct bpf_map *map_ptr;
int i, ret, cnt, cur_subprog = 0;
struct bpf_subprog_info *subprogs = env->subprog_info;
- u16 stack_depth = subprogs[cur_subprog].stack_depth;
- u16 stack_depth_extra = 0;
/* arch_bpf_timed_may_goto() takes and returns AX in a cpu register: for JITs only */
bool timed_may_goto = prog->jit_requested && bpf_jit_supports_timed_may_goto();
struct bpf_timed_may_goto __percpu *tmg = NULL;
+ s16 may_goto_slot = 0;
if (env->seen_exception && !env->exception_callback_subprog) {
struct bpf_insn *patch = insn_buf;
@@ -2879,9 +2984,12 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
if (!(live & BIT(reg)))
break;
if (reg > BPF_REG_9) {
+ /* R0 is saved in a hidden slot, shared by the may_goto of the subprog */
+ ret = may_goto_stack_slot(env, cur_subprog, false, &may_goto_slot);
+ if (ret)
+ return ret;
reg = BPF_REG_0;
- save_off = -stack_depth - 8;
- stack_depth_extra = 8;
+ save_off = may_goto_slot;
}
if (!tmg) {
tmg = alloc_percpu_gfp(struct bpf_timed_may_goto,
@@ -2899,16 +3007,16 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
return PTR_ERR(insn);
goto next_insn;
} else if (bpf_is_may_goto_insn(insn)) {
- int stack_off = -stack_depth - 8;
- struct bpf_insn tail[] = {
- BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, 1),
- BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_AX, stack_off),
- };
+ struct bpf_insn tail[2];
- stack_depth_extra = 8;
- cnt = may_goto_expand(insn_buf, insn->off,
- may_goto_off(env, insn, i),
- stack_off, tail, ARRAY_SIZE(tail));
+ /* the count is in a hidden slot, shared by the may_goto of the subprog */
+ ret = may_goto_stack_slot(env, cur_subprog, true, &may_goto_slot);
+ if (ret)
+ return ret;
+ tail[0] = BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, 1);
+ tail[1] = BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_AX, may_goto_slot);
+ cnt = may_goto_expand(insn_buf, insn->off, may_goto_off(env, insn, i),
+ may_goto_slot, tail, ARRAY_SIZE(tail));
insn = bpf_patch_list_add(env, i, insn_buf, cnt);
if (IS_ERR(insn))
@@ -3431,18 +3539,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
insn->imm = BPF_CALL_IMM(fn->func);
next_insn:
if (subprogs[cur_subprog + 1].start == i + 1) {
- subprogs[cur_subprog].stack_depth += stack_depth_extra;
- subprogs[cur_subprog].stack_extra = stack_depth_extra;
-
- stack_depth = subprogs[cur_subprog].stack_depth;
- if (stack_depth > MAX_BPF_STACK && !prog->jit_requested) {
- verbose(env, "stack size %d(extra %d) is too large\n",
- stack_depth, stack_depth_extra);
- return -EINVAL;
- }
cur_subprog++;
- stack_depth = subprogs[cur_subprog].stack_depth;
- stack_depth_extra = 0;
+ may_goto_slot = 0;
}
i++;
insn = &prog->insnsi[i];
@@ -3453,57 +3551,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
return ret;
prog = env->prog;
- env->prog->aux->stack_depth = subprogs[0].stack_depth;
- for (i = 0; i < env->subprog_cnt; i++) {
- int stack_slots = subprogs[i].stack_extra / 8;
+ for (i = 0; tmg && i < env->subprog_cnt; i++) {
/* an invocation begins, see bpf_timed_may_goto_save() for async callbacks */
- bool main_prog = tmg && !i;
+ bool main_prog = !i;
/* only a helper calls it, in a loop that ends when it returns 1 */
- bool loop_cb = tmg && i && subprogs[i].is_loop_cb && !subprogs[i].is_called &&
+ bool loop_cb = i && subprogs[i].is_loop_cb && !subprogs[i].is_called &&
!subprogs[i].is_other_cb && !subprogs[i].is_async_cb;
-
- if (timed_may_goto ? !main_prog && !loop_cb : !stack_slots)
+ struct bpf_insn ld_tmg[] = {
+ BPF_LD_IMM64(BPF_REG_0, (__force unsigned long)tmg)
+ };
+ int count_off = offsetof(struct bpf_timed_may_goto, count);
+ int timestamp_off = offsetof(struct bpf_timed_may_goto, timestamp);
+ int owner_off = offsetof(struct bpf_timed_may_goto, owner);
+
+ if (!main_prog && !loop_cb)
continue;
- if (stack_slots > 1) {
- verifier_bug(env, "stack_slots supports may_goto only");
- return -EFAULT;
- }
cnt = 0;
- stack_depth = subprogs[i].stack_depth;
- if (timed_may_goto) {
- struct bpf_insn ld_tmg[] = {
- BPF_LD_IMM64(BPF_REG_0, (__force unsigned long)tmg)
- };
- int count_off = offsetof(struct bpf_timed_may_goto, count);
- int timestamp_off = offsetof(struct bpf_timed_may_goto, timestamp);
- int owner_off = offsetof(struct bpf_timed_may_goto, owner);
-
- /* R0 has nothing at the entry */
- insn_buf[cnt++] = ld_tmg[0];
- insn_buf[cnt++] = ld_tmg[1];
- insn_buf[cnt++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
- if (main_prog) {
- insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, count_off,
- BPF_MAX_TIMED_LOOPS);
- insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, timestamp_off, 0);
- insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, owner_off, 0);
- } else {
- /*
- * The time is up, but the helper would call the function again and
- * again. End its loop. The verifier has seen that: the loop can end
- * after each call.
- */
- insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0,
- count_off);
- insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2);
- insn_buf[cnt++] = BPF_MOV64_IMM(BPF_REG_0, 1);
- insn_buf[cnt++] = BPF_EXIT_INSN();
- }
+ /* R0 has nothing at the entry */
+ insn_buf[cnt++] = ld_tmg[0];
+ insn_buf[cnt++] = ld_tmg[1];
+ insn_buf[cnt++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
+ if (main_prog) {
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, count_off,
+ BPF_MAX_TIMED_LOOPS);
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, timestamp_off, 0);
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, owner_off, 0);
} else {
- /* Add ST insn to subprog prologue to init extra stack */
- insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, -stack_depth,
- BPF_MAX_LOOPS);
+ /*
+ * The time is up, but the helper would call the function again and
+ * again. End its loop. The verifier has seen that: the loop can end
+ * after each call.
+ */
+ insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0,
+ count_off);
+ insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2);
+ insn_buf[cnt++] = BPF_MOV64_IMM(BPF_REG_0, 1);
+ insn_buf[cnt++] = BPF_EXIT_INSN();
}
/* jumps to the first insn of the subprog skip the setup */
ret = bpf_patch_list_insert(env, subprogs[i].start, insn_buf, cnt);
@@ -3512,9 +3596,15 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
ret = bpf_patch_list_commit(env);
+ if (ret)
+ return ret;
+
+ /* The hidden slots get their values in the prologues too. */
+ ret = bpf_emit_stack_inits(env);
if (ret)
return ret;
prog = env->prog;
+ prog->aux->stack_depth = subprogs[0].stack_depth;
/* Since poke tab is now finalized, publish aux to tracker. */
for (i = 0; i < prog->aux->size_poke_tab; i++) {
@@ -3613,14 +3703,14 @@ static bool is_bpf_loop_call(struct bpf_insn *insn)
insn->imm == BPF_FUNC_loop;
}
-/* For all sub-programs in the program (including main) check
+/*
+ * For all sub-programs in the program (including main) check
* insn_aux_data to see if there are bpf_loop calls that require
* inlining. If such calls are found the calls are replaced with a
* sequence of instructions produced by `inline_bpf_loop` function and
- * subprog stack_depth is increased by the size of 3 registers.
- * This stack space is used to spill values of the R6, R7, R8. These
- * registers are used to store the loop bound, counter and context
- * variables.
+ * three hidden stack slots are reserved in the subprog to spill the
+ * values of R6, R7, R8. These registers are used to store the loop
+ * bound, counter and context variables.
*/
int bpf_optimize_bpf_loop(struct bpf_verifier_env *env)
{
@@ -3628,9 +3718,7 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env)
int i, cur_subprog = 0, err;
struct bpf_insn *insn = env->prog->insnsi;
int insn_cnt = env->prog->len;
- u16 stack_depth = subprogs[cur_subprog].stack_depth;
- u16 stack_depth_roundup = round_up(stack_depth, 8) - stack_depth;
- u16 stack_depth_extra = 0;
+ s16 loop_off = 0;
for (i = 0; i < insn_cnt; i++, insn++) {
struct bpf_loop_inline_state *inline_state =
@@ -3639,20 +3727,21 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env)
/* the callback that takes a stack in arena is not for a call from the program */
if (is_bpf_loop_call(insn) && inline_state->fit_for_inline &&
!subprogs[inline_state->callback_subprogno].arena_stack_entry) {
- stack_depth_extra = BPF_REG_SIZE * 3 + stack_depth_roundup;
- err = inline_bpf_loop(env, i,
- -(stack_depth + stack_depth_extra),
+ /* Three slots spill R6, R7 and R8, shared by the subprog's loops. */
+ if (!loop_off) {
+ err = bpf_reserve_stack_slots(env, cur_subprog, 3, &loop_off);
+ if (err)
+ return err;
+ }
+ err = inline_bpf_loop(env, i, loop_off - 2 * BPF_REG_SIZE,
inline_state->callback_subprogno);
if (err)
return err;
}
if (subprogs[cur_subprog + 1].start == i + 1) {
- subprogs[cur_subprog].stack_depth += stack_depth_extra;
cur_subprog++;
- stack_depth = subprogs[cur_subprog].stack_depth;
- stack_depth_roundup = round_up(stack_depth, 8) - stack_depth;
- stack_depth_extra = 0;
+ loop_off = 0;
}
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c5b4025ee222..0684ac1f4f22 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -25732,6 +25732,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
for (i = 0; i < env->func_set_cnt; i++)
kfree(env->func_sets[i]);
kfree(env->func_sets);
+ kvfree(env->stack_inits);
bpf_diag_free(env);
kvfree(env->idoms);
kvfree(env);
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots
2026-11-11 12:57 ` [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 15:15 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5601 bytes --]
> bpf: Add one allocator for hidden stack slots
TL;DR:
- Stack slot allocator only caps JITed frames at 32 KiB, so later patches in
the series may overgrow stack limits
- insert_arena_stack_get() bypasses the allocator, contradicting comments
- Misplaced struct comment (nit)
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
[ ... ]
> @@ -1208,6 +1209,13 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
> /* single container for all structs
> * one verifier_env per bpf_check() call
> */
> +/* Initialization of a hidden stack slot, emitted in the subprogram's prologue. */
> +struct bpf_stack_init {
> + u32 subprog;
> + s16 off;
> + s32 imm;
> +};
> +
> struct bpf_verifier_env {
This isn't a bug, but struct bpf_stack_init now sits between the comment that
describes struct bpf_verifier_env ("single container for all structs / one
verifier_env per bpf_check() call") and the struct itself. Two comment
blocks are stacked on struct bpf_stack_init, and struct bpf_verifier_env has
lost its comment.
Could struct bpf_stack_init go above the bpf_verifier_env comment instead?
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -918,6 +918,87 @@ int bpf_insert_recursion_checks(struct bpf_verifier_env *env)
> return 0;
> }
>
> +/*
> + * Hidden stack slots. Post-verification rewrites need stack space the program
> + * does not know about: may_goto counters, the spill area of an inlined
> + * bpf_loop(), the context pointer a generated epilogue reads back. Such slots
> + * are reserved below the stack depth verification established for a
> + * subprogram, after bpf_remove_fastcall_spills_fills() has shrunk it, so that
> + * they sit directly under the program's own frame. Slots are 8 bytes, 8-byte
> + * aligned and handed out top down: the first slot of a subprogram lies right
> + * below its verified depth rounded up to 8 bytes.
> + */
> +static s16 next_stack_slot_off(const struct bpf_subprog_info *info)
The comment and the commit message describe this as the one allocator, with
the first slot lying right below the verified depth rounded up to 8 bytes,
and say the stack size check "covers every user".
insert_arena_stack_get() still works the old way. It runs between
bpf_optimize_bpf_loop() and bpf_do_misc_fixups() and bumps stack_depth by
hand:
if (s) {
spill = round_up(sub->stack_depth, BPF_REG_SIZE);
sub->stack_depth = spill + MAX_BPF_FUNC_REG_ARGS * BPF_REG_SIZE;
...
*patch++ = BPF_STX_MEM(BPF_DW, BPF_REG_FP, i, -(spill + i * BPF_REG_SIZE));
For an arena_stack_entry callback that has a may_goto, may_goto_stack_slot()
-> bpf_reserve_stack_slots() puts its slot below the arena spill area, not
right below the verified depth. Those arena bytes are also not counted in
stack_extra, so the "stack size %d(extra %d)" message under-reports them.
Nothing functional breaks today, since the arena pass checks env->stack_limit
itself. Should insert_arena_stack_get() reserve its area through
bpf_reserve_stack_slots(), or should the comment and commit message drop the
"right below its verified depth" and "every user" claims?
> +{
> + return -(s32)(round_up(info->stack_depth, BPF_REG_SIZE) + BPF_REG_SIZE);
> +}
> +
> +/* Reserve @nr slots in @subprog; @off receives the offset of the topmost one. */
> +int bpf_reserve_stack_slots(struct bpf_verifier_env *env, int subprog, int nr, s16 *off)
> +{
> + struct bpf_subprog_info *info = &env->subprog_info[subprog];
> + u32 depth = round_up(info->stack_depth, BPF_REG_SIZE) + nr * BPF_REG_SIZE;
> + u32 extra = depth - info->stack_depth;
> +
> + if ((depth > MAX_BPF_STACK && !env->prog->jit_requested) || depth > S16_MAX + 1U) {
For a JITed program the only limit applied here is S16_MAX + 1, and
check_max_stack_depth() has already run by the time this is called. It is
not compared against env->stack_limit or MAX_BPF_STACK_JIT.
In this commit that is harmless, since every caller adds a small constant per
subprog (3 slots for bpf_loop(), 1 for the epilogue, 1 for may_goto, plus at
most 7 bytes of alignment).
The commit message says the allocator is meant for later patches that need
more hidden slots. In this series, cd4657cd0e50 ("bpf: Track held resources
in hidden stack slots") calls it once per acquiring instruction from
bpf_frame_desc_resolve():
ret = bpf_reserve_stack_slots(env, acq_subprog(env, a, new_of), 1, &a->off);
026679e2eda5 ("bpf: Describe held resources at throw sites") narrows this to
acquisitions that need a slot, but there is still no per-subprog cap, and no
later commit in the series adds a bound. A subprog with N slot-needing
acquisitions grows by 8*N bytes after the stack budget was checked, up to
32 KiB per frame. The x86 JIT emits that size directly with
emit_sub_rsp(&prog, round_up(stack_depth, 8)), while
bpf_jit_supports_large_stack() documents JIT frames only up to
MAX_BPF_STACK_JIT (2048).
insert_arena_stack_get() rejects a frame that exceeds env->stack_limit.
Should the allocator also bound JITed frames, for example against
env->stack_limit plus a small fixed allowance for the constant per-subprog
users, since it is now the only place that checks?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots
2026-11-11 12:57 ` [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 15:15 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 15:15 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] bpf: stack buffer overflow in ___bpf_prog_run() [1]
--
commit 12562841fbd23150e8e97550f85ed29a199cd037
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Add one allocator for hidden stack slots
This commit introduces a unified hidden stack slot allocator for
post-verification rewrites and converts may_goto, inlined bpf_loop(), and
epilogue context saving to use it.
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index 8597b443d5619..04d89113e5963 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
[ ... ]
> @@ -3512,9 +3596,15 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
> }
>
> ret = bpf_patch_list_commit(env);
> + if (ret)
> + return ret;
> +
> + /* The hidden slots get their values in the prologues too. */
> + ret = bpf_emit_stack_inits(env);
> if (ret)
> return ret;
> prog = env->prog;
> + prog->aux->stack_depth = subprogs[0].stack_depth;
[Severity: Critical]
This problem wasn't introduced by this patch, but can a tail-calling program
that fails JIT compilation overflow the kernel stack in ___bpf_prog_run()
[1]?
In bpf_do_misc_fixups(), when a BPF_FUNC_tail_call instruction is seen, the
verifier only sets prog->aux->stack_depth to MAX_BPF_STACK when
!bpf_allow_tail_call_in_subprogs(env) (and then overwrites
prog->aux->stack_depth with subprogs[0].stack_depth at the end of the
function):
kernel/bpf/fixups.c:bpf_do_misc_fixups() {
...
if (insn->imm == BPF_FUNC_tail_call) {
prog->cb_access = 1;
if (!bpf_allow_tail_call_in_subprogs(env))
prog->aux->stack_depth = MAX_BPF_STACK;
...
}
On architectures where bpf_allow_tail_call_in_subprogs() is true because
jit_requested is set, if JIT compilation later fails (for example, on memory
allocation failure) and falls back to the interpreter,
bpf_prog_select_interpreter() selects an interpreter function sized only for
the caller's unexpanded stack_depth. When JMP_TAIL_CALL in
___bpf_prog_run() switches insn to a target program that uses a deeper stack
(up to MAX_BPF_STACK) without allocating a new C stack frame, the target
program writes past the caller's fixed-size stack array and corrupts the
kernel stack.
[1] https://sashiko.dev/#/bug/linux-57b05589-4ccc-4b7c-94c0-c3316f3932e4
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=7
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (6 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 07/29] bpf: Add one allocator for hidden stack slots Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 13:33 ` bot+bpf-ci
2026-11-11 15:36 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types Kumar Kartikeya Dwivedi
` (21 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
The BTF destructor registry maps a type to the void (*)(void *) function
that destroys a kptr of that type when a map element is freed. This is
its only use.
An unwinder that releases the references a program holds when it throws
needs the same table, with two additions:
- An entry must say whether the unwinder may use it.
- An entry must name the kfunc that a program calls to release such a
reference. The verifier uses this to decide whether a destructor may
run at a given point: if the program could call the release kfunc
there, running the equivalent destructor is allowed.
Add a flags field to struct btf_id_dtor_kfunc:
- BPF_DTOR_KPTR keeps today's use. btf_find_dtor_kfunc() takes the
flags that an entry must have, so the kptr lookup in btf_parse_kptr()
sees the same entries as before.
- BPF_DTOR_CLEANUP marks an entry that the unwinder may use. Such an
entry carries release_btf_id: the KF_RELEASE kfunc of the type, or the
KF_ITER_DESTROY kfunc of an iterator type.
btf_find_dtor() returns the whole entry for the unwinder. For a type of a
module BTF it also searches the base BTF, and falls back to a vmlinux
struct of the same name.
Registration rejects an entry without flags, a cleanup entry without a
release kfunc, and a release kfunc that is not registered with KF_RELEASE
or KF_ITER_DESTROY in a kfunc id set of the same BTF. The check runs at
registration time, so a module or initcall must register its kfunc id
sets before its destructors. Every current registration already does.
Mark the existing entries BPF_DTOR_KPTR. Also mark them BPF_DTOR_CLEANUP,
with their release kfunc, where the type has one: task_struct, cgroup,
bpf_cpumask, bpf_crypto_ctx, bpf_ksock, the sk_buff of a qdisc program,
the two test types in test_run.c, and bpf_testmod_ctx.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/btf.h | 23 +++-
kernel/bpf/btf.c | 101 ++++++++++++++++--
kernel/bpf/cpumask.c | 7 +-
kernel/bpf/crypto.c | 7 +-
kernel/bpf/helpers.c | 14 ++-
net/bpf/test_run.c | 14 ++-
net/core/bpf_ksock.c | 3 +
net/sched/bpf_qdisc.c | 10 +-
.../selftests/bpf/test_kmods/bpf_testmod.c | 5 +-
9 files changed, 155 insertions(+), 29 deletions(-)
diff --git a/include/linux/btf.h b/include/linux/btf.h
index 87fad4076f0e..045c48bcf98c 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -141,9 +141,19 @@ struct btf_kfunc_id_set {
btf_kfunc_filter_t filter;
};
+/*
+ * Flags for struct btf_id_dtor_kfunc::flags: BPF_DTOR_KPTR destroys the kptr fields of a map
+ * element when it is freed, BPF_DTOR_CLEANUP may release a reference held by a program that
+ * is being unwound.
+ */
+#define BPF_DTOR_KPTR (1U << 0)
+#define BPF_DTOR_CLEANUP (1U << 1)
+
struct btf_id_dtor_kfunc {
- u32 btf_id;
- u32 kfunc_btf_id;
+ u32 btf_id; /* type the destructor applies to */
+ u32 kfunc_btf_id; /* BTF id of the void (*)(void *) destructor */
+ u32 release_btf_id; /* KF_RELEASE kfunc of the type, required for BPF_DTOR_CLEANUP */
+ u32 flags;
};
struct btf_struct_meta {
@@ -627,7 +637,8 @@ u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
const struct btf_kfunc_id_set *s);
int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset);
-s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);
+const struct btf_id_dtor_kfunc *btf_find_dtor(struct btf **btfp, u32 btf_id);
+s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id, u32 flags);
int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
struct module *owner);
struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id);
@@ -718,7 +729,11 @@ static inline int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
{
return 0;
}
-static inline s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id)
+static inline const struct btf_id_dtor_kfunc *btf_find_dtor(struct btf **btfp, u32 btf_id)
+{
+ return NULL;
+}
+static inline s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id, u32 flags)
{
return -ENOENT;
}
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 228b93d4f8c9..6359c5e343bd 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -4073,7 +4073,7 @@ static int btf_parse_kptr(const struct btf *btf, struct btf_field *field,
* can be used as a referenced pointer and be stored in a map at
* the same time.
*/
- dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id);
+ dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id, BPF_DTOR_KPTR);
if (dtor_btf_id < 0) {
ret = dtor_btf_id;
goto end_btf;
@@ -10190,23 +10190,86 @@ int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset)
}
EXPORT_SYMBOL_GPL(register_btf_fmodret_id_set);
-s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id)
+static const struct btf_id_dtor_kfunc *btf_dtor_tab_find(const struct btf *btf, u32 btf_id)
{
struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
- struct btf_id_dtor_kfunc *dtor;
if (!tab)
- return -ENOENT;
- /* Even though the size of tab->dtors[0] is > sizeof(u32), we only need
+ return NULL;
+ /*
+ * Even though the size of tab->dtors[0] is > sizeof(u32), we only need
* to compare the first u32 with btf_id, so we can reuse btf_id_cmp_func.
*/
BUILD_BUG_ON(offsetof(struct btf_id_dtor_kfunc, btf_id) != 0);
- dtor = bsearch(&btf_id, tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func);
- if (!dtor)
+ return bsearch(&btf_id, tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func);
+}
+
+/*
+ * Find the destructor entry of type @btf_id of *@btfp, and set *@btfp to the
+ * BTF the entry and its kfunc ids belong to. A module's BTF is split on top of
+ * vmlinux's, so a type below its first id is vmlinux's, with its destructor
+ * registered there; and a struct a module defines under the name of a vmlinux
+ * struct is the same type to the verifier, which passes it to vmlinux kfuncs
+ * by name, so it is destroyed as that type.
+ */
+const struct btf_id_dtor_kfunc *btf_find_dtor(struct btf **btfp, u32 btf_id)
+{
+ const struct btf_id_dtor_kfunc *dtor;
+ struct btf *btf = *btfp, *vmlinux;
+ const struct btf_type *t;
+ s32 id;
+
+ while (btf_id < btf->start_id)
+ btf = btf->base_btf;
+ dtor = btf_dtor_tab_find(btf, btf_id);
+ if (dtor || !btf_is_module(btf)) {
+ *btfp = btf;
+ return dtor;
+ }
+ t = btf_type_by_id(btf, btf_id);
+ if (!t || !btf_type_is_struct(t))
+ return NULL;
+ vmlinux = bpf_get_btf_vmlinux();
+ if (IS_ERR_OR_NULL(vmlinux))
+ return NULL;
+ id = btf_find_by_name_kind(vmlinux, btf_name_by_offset(btf, t->name_off), BTF_KIND_STRUCT);
+ if (id < 0)
+ return NULL;
+ dtor = btf_dtor_tab_find(vmlinux, id);
+ if (dtor)
+ *btfp = vmlinux;
+ return dtor;
+}
+
+/* Return the destructor of btf_id if its entry has all of the BPF_DTOR_* bits in flags */
+s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id, u32 flags)
+{
+ const struct btf_id_dtor_kfunc *dtor = btf_find_dtor(&btf, btf_id);
+
+ if (!dtor || (dtor->flags & flags) != flags)
return -ENOENT;
return dtor->kfunc_btf_id;
}
+/*
+ * The release kfunc of a BPF_DTOR_CLEANUP entry is what a program calls to release a
+ * reference of the type itself, so it must be a KF_RELEASE kfunc, or the KF_ITER_DESTROY
+ * kfunc of an iterator type. The kfunc id sets of the BTF are consulted, so they must be
+ * registered before the destructors.
+ */
+static int btf_check_release_kfunc(struct btf *btf, u32 release_btf_id)
+{
+ const struct btf_type *t;
+
+ t = btf_type_by_id(btf, release_btf_id);
+ if (!t || !btf_type_is_func(t))
+ return -EINVAL;
+ if (btf_kfunc_check_flag(btf, release_btf_id, KF_RELEASE) == 1 ||
+ btf_kfunc_check_flag(btf, release_btf_id, KF_ITER_DESTROY) == 1)
+ return 0;
+ return -EINVAL;
+}
+
static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc *dtors, u32 cnt)
{
const struct btf_type *dtor_func, *dtor_func_proto, *t;
@@ -10217,6 +10280,22 @@ static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc
for (i = 0; i < cnt; i++) {
dtor_btf_id = btf_relocate_id(btf, dtors[i].kfunc_btf_id);
+ if (!dtors[i].flags) {
+ pr_err("no flags for the destructor of btf_id %u\n", dtors[i].btf_id);
+ return -EINVAL;
+ }
+ if ((dtors[i].flags & BPF_DTOR_CLEANUP) && !dtors[i].release_btf_id) {
+ pr_err("no release kfunc for the cleanup destructor of btf_id %u\n",
+ dtors[i].btf_id);
+ return -EINVAL;
+ }
+ if (dtors[i].release_btf_id &&
+ btf_check_release_kfunc(btf, btf_relocate_id(btf, dtors[i].release_btf_id))) {
+ pr_err("btf_id %u is not a release kfunc for the destructor of btf_id %u\n",
+ dtors[i].release_btf_id, dtors[i].btf_id);
+ return -EINVAL;
+ }
+
dtor_func = btf_type_by_id(btf, dtor_btf_id);
if (!dtor_func || !btf_type_is_func(dtor_func))
return -EINVAL;
@@ -10311,8 +10390,12 @@ int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_c
/* remap BTF ids based on BTF relocation (if any) */
for (i = tab_cnt; i < tab_cnt + add_cnt; i++) {
- tab->dtors[i].btf_id = btf_relocate_id(btf, tab->dtors[i].btf_id);
- tab->dtors[i].kfunc_btf_id = btf_relocate_id(btf, tab->dtors[i].kfunc_btf_id);
+ struct btf_id_dtor_kfunc *dtor = &tab->dtors[i];
+
+ dtor->btf_id = btf_relocate_id(btf, dtor->btf_id);
+ dtor->kfunc_btf_id = btf_relocate_id(btf, dtor->kfunc_btf_id);
+ if (dtor->release_btf_id)
+ dtor->release_btf_id = btf_relocate_id(btf, dtor->release_btf_id);
}
tab->cnt += add_cnt;
diff --git a/kernel/bpf/cpumask.c b/kernel/bpf/cpumask.c
index 1336a4efa755..f552472d69b1 100644
--- a/kernel/bpf/cpumask.c
+++ b/kernel/bpf/cpumask.c
@@ -511,14 +511,17 @@ static const struct btf_kfunc_id_set cpumask_kfunc_set = {
BTF_ID_LIST(cpumask_dtor_ids)
BTF_ID(struct, bpf_cpumask)
BTF_ID(func, bpf_cpumask_release_dtor)
+BTF_ID(func, bpf_cpumask_release)
static int __init cpumask_kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc cpumask_dtors[] = {
{
- .btf_id = cpumask_dtor_ids[0],
- .kfunc_btf_id = cpumask_dtor_ids[1]
+ .btf_id = cpumask_dtor_ids[0],
+ .kfunc_btf_id = cpumask_dtor_ids[1],
+ .release_btf_id = cpumask_dtor_ids[2],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
};
diff --git a/kernel/bpf/crypto.c b/kernel/bpf/crypto.c
index 69bc29d0a4d4..cd65da3dffcf 100644
--- a/kernel/bpf/crypto.c
+++ b/kernel/bpf/crypto.c
@@ -336,14 +336,17 @@ static const struct btf_kfunc_id_set crypt_kfunc_set = {
BTF_ID_LIST(bpf_crypto_dtor_ids)
BTF_ID(struct, bpf_crypto_ctx)
BTF_ID(func, bpf_crypto_ctx_release_dtor)
+BTF_ID(func, bpf_crypto_ctx_release)
static int __init crypto_kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc bpf_crypto_dtors[] = {
{
- .btf_id = bpf_crypto_dtor_ids[0],
- .kfunc_btf_id = bpf_crypto_dtor_ids[1]
+ .btf_id = bpf_crypto_dtor_ids[0],
+ .kfunc_btf_id = bpf_crypto_dtor_ids[1],
+ .release_btf_id = bpf_crypto_dtor_ids[2],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
};
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 0164f183aade..cbed0c8952a7 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -5317,9 +5317,11 @@ static const struct btf_kfunc_id_set user_mem_kfunc_set = {
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
+BTF_ID(func, bpf_task_release)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
+BTF_ID(func, bpf_cgroup_release)
#endif
BTF_KFUNCS_START(common_btf_ids)
@@ -5429,13 +5431,17 @@ static int __init kfunc_init(void)
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
- .btf_id = generic_dtor_ids[0],
- .kfunc_btf_id = generic_dtor_ids[1]
+ .btf_id = generic_dtor_ids[0],
+ .kfunc_btf_id = generic_dtor_ids[1],
+ .release_btf_id = generic_dtor_ids[2],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
#ifdef CONFIG_CGROUPS
{
- .btf_id = generic_dtor_ids[2],
- .kfunc_btf_id = generic_dtor_ids[3]
+ .btf_id = generic_dtor_ids[3],
+ .kfunc_btf_id = generic_dtor_ids[4],
+ .release_btf_id = generic_dtor_ids[5],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
#endif
};
diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
index d6c9dc62182e..0eb11688aaa4 100644
--- a/net/bpf/test_run.c
+++ b/net/bpf/test_run.c
@@ -1888,19 +1888,25 @@ static const struct btf_kfunc_id_set bpf_prog_test_kfunc_set = {
BTF_ID_LIST(bpf_prog_test_dtor_kfunc_ids)
BTF_ID(struct, prog_test_ref_kfunc)
BTF_ID(func, bpf_kfunc_call_test_release_dtor)
+BTF_ID(func, bpf_kfunc_call_test_release)
BTF_ID(struct, prog_test_member)
BTF_ID(func, bpf_kfunc_call_memb_release_dtor)
+BTF_ID(func, bpf_kfunc_call_memb_release)
static int __init bpf_prog_test_run_init(void)
{
const struct btf_id_dtor_kfunc bpf_prog_test_dtor_kfunc[] = {
{
- .btf_id = bpf_prog_test_dtor_kfunc_ids[0],
- .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[1]
+ .btf_id = bpf_prog_test_dtor_kfunc_ids[0],
+ .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[1],
+ .release_btf_id = bpf_prog_test_dtor_kfunc_ids[2],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
{
- .btf_id = bpf_prog_test_dtor_kfunc_ids[2],
- .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[3],
+ .btf_id = bpf_prog_test_dtor_kfunc_ids[3],
+ .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[4],
+ .release_btf_id = bpf_prog_test_dtor_kfunc_ids[5],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
};
int ret;
diff --git a/net/core/bpf_ksock.c b/net/core/bpf_ksock.c
index 25111712b944..e7fba11cdb12 100644
--- a/net/core/bpf_ksock.c
+++ b/net/core/bpf_ksock.c
@@ -366,6 +366,7 @@ static const struct btf_kfunc_id_set ksock_kfunc_set = {
BTF_ID_LIST(bpf_ksock_dtor_ids)
BTF_ID(struct, bpf_ksock)
BTF_ID(func, bpf_ksock_release_dtor)
+BTF_ID(func, bpf_ksock_release)
static int __init bpf_ksock_kfunc_init(void)
{
@@ -374,6 +375,8 @@ static int __init bpf_ksock_kfunc_init(void)
{
.btf_id = bpf_ksock_dtor_ids[0],
.kfunc_btf_id = bpf_ksock_dtor_ids[1],
+ .release_btf_id = bpf_ksock_dtor_ids[2],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
};
diff --git a/net/sched/bpf_qdisc.c b/net/sched/bpf_qdisc.c
index 5691c13781a8..70e7a6dafc58 100644
--- a/net/sched/bpf_qdisc.c
+++ b/net/sched/bpf_qdisc.c
@@ -453,15 +453,19 @@ static struct bpf_struct_ops bpf_Qdisc_ops = {
.owner = THIS_MODULE,
};
-BTF_ID_LIST_SINGLE(bpf_sk_buff_dtor_ids, func, bpf_kfree_skb_dtor)
+BTF_ID_LIST(bpf_sk_buff_dtor_ids)
+BTF_ID(func, bpf_kfree_skb_dtor)
+BTF_ID(func, bpf_kfree_skb)
static int __init bpf_qdisc_kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc skb_kfunc_dtors[] = {
{
- .btf_id = bpf_sk_buff_ids[0],
- .kfunc_btf_id = bpf_sk_buff_dtor_ids[0]
+ .btf_id = bpf_sk_buff_ids[0],
+ .kfunc_btf_id = bpf_sk_buff_dtor_ids[0],
+ .release_btf_id = bpf_sk_buff_dtor_ids[1],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
};
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 41ff36f0ca77..891f777a265a 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -1084,6 +1084,7 @@ BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)
BTF_ID_LIST(bpf_testmod_dtor_ids)
BTF_ID(struct, bpf_testmod_ctx)
BTF_ID(func, bpf_testmod_ctx_release_dtor)
+BTF_ID(func, bpf_testmod_ctx_release)
static const struct btf_kfunc_id_set bpf_testmod_common_kfunc_set = {
.owner = THIS_MODULE,
@@ -2622,7 +2623,9 @@ static int bpf_testmod_init(void)
const struct btf_id_dtor_kfunc bpf_testmod_dtors[] = {
{
.btf_id = bpf_testmod_dtor_ids[0],
- .kfunc_btf_id = bpf_testmod_dtor_ids[1]
+ .kfunc_btf_id = bpf_testmod_dtor_ids[1],
+ .release_btf_id = bpf_testmod_dtor_ids[2],
+ .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
};
void **tramp;
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry
2026-11-11 12:57 ` [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry Kumar Kartikeya Dwivedi
@ 2026-11-11 13:33 ` bot+bpf-ci
2026-11-11 15:36 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 13:33 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3664 bytes --]
> bpf: Add flags and release kfuncs to the dtor registry
TL;DR: btf_find_dtor() may fall back by name to a vmlinux destructor for a
same-named module struct, so on bpf_throw() the unwinder could run it on an
object of a different type. Needs a colliding module type.
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
[ ... ]
> +/*
> + * Find the destructor entry of type @btf_id of *@btfp, and set *@btfp to the
> + * BTF the entry and its kfunc ids belong to. A module's BTF is split on top of
> + * vmlinux's, so a type below its first id is vmlinux's, with its destructor
> + * registered there; and a struct a module defines under the name of a vmlinux
> + * struct is the same type to the verifier, which passes it to vmlinux kfuncs
> + * by name, so it is destroyed as that type.
> + */
> +const struct btf_id_dtor_kfunc *btf_find_dtor(struct btf **btfp, u32 btf_id)
> +{
[ ... ]
> + t = btf_type_by_id(btf, btf_id);
> + if (!t || !btf_type_is_struct(t))
> + return NULL;
> + vmlinux = bpf_get_btf_vmlinux();
> + if (IS_ERR_OR_NULL(vmlinux))
> + return NULL;
> + id = btf_find_by_name_kind(vmlinux, btf_name_by_offset(btf, t->name_off), BTF_KIND_STRUCT);
> + if (id < 0)
> + return NULL;
> + dtor = btf_dtor_tab_find(vmlinux, id);
> + if (dtor)
> + *btfp = vmlinux;
> + return dtor;
> +}
The comment says a module struct with the name of a vmlinux struct is the
same type to the verifier and is passed to vmlinux kfuncs by name. Is that
accurate?
For kfunc pointer arguments, process_arg_ptr_to_btf_id() calls
btf_struct_ids_match(), which uses btf_types_are_same():
if (id1 != id2)
return false;
if (btf1 == btf2)
return true;
return btf_type_by_id(btf1, id1) == btf_type_by_id(btf2, id2);
For a released argument the match is also strict, so the struct is not
walked. Matching by name only happens for the __sk_buff/xdp_md projections
in btf_is_projection_of().
So a program holding a reference to a module's own struct X cannot pass it
to the vmlinux release kfunc of X, yet btf_find_dtor() returns the vmlinux
entry for X. The commit message says the rule is "if the program could
call the release kfunc there, running the equivalent destructor is
allowed", and this fallback seems to break that rule.
The later consumer, resource_registry() in kernel/bpf/frame_desc.c, only
checks btf_kfunc_flags()/btf_kfunc_is_allowed() for dtor->release_btf_id
against the program. It does not check that reg->btf and reg->btf_id would
pass the release kfunc's strict type match.
Suppose a module defines its own struct named like a vmlinux cleanup type
that is private to a .c file (bpf_cpumask, bpf_crypto_ctx, bpf_ksock or
prog_test_ref_kfunc), returns it from a KF_ACQUIRE kfunc, and registers no
destructor for it. Can a program that holds that reference and calls
bpf_throw() then have the unwinder run the vmlinux destructor, for example
bpf_cpumask_release_dtor(), on an object of a different type?
In this patch the fallback is not reachable from btf_parse_kptr(), since
bpf_find_btf_id() searches vmlinux first by name and kind. No in-tree
module defines such a colliding, non-deduplicated struct, so this needs a
module type to trigger. The fallback is still present at the end of the
series.
Could the fallback be dropped, or could it require type identity before
returning a vmlinux entry for a module type?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry
2026-11-11 12:57 ` [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry Kumar Kartikeya Dwivedi
2026-11-11 13:33 ` bot+bpf-ci
@ 2026-11-11 15:36 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 15:36 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] bpf: global state corruption in
register_btf_id_dtor_kfuncs() [1]
--
commit 296a6e3ce795b555a37d881e7098d3e3144bc374
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Add flags and release kfuncs to the dtor registry
This commit adds flags and release_btf_id fields to struct btf_id_dtor_kfunc
so the unwinder can look up and validate cleanup destructors alongside map
kptr destructors.
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 228b93d4f8c94..6359c5e343bd4 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
[ ... ]
> @@ -10217,6 +10280,22 @@ static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc
> for (i = 0; i < cnt; i++) {
> dtor_btf_id = btf_relocate_id(btf, dtors[i].kfunc_btf_id);
>
> + if (!dtors[i].flags) {
> + pr_err("no flags for the destructor of btf_id %u\n", dtors[i].btf_id);
> + return -EINVAL;
> + }
> + if ((dtors[i].flags & BPF_DTOR_CLEANUP) && !dtors[i].release_btf_id) {
> + pr_err("no release kfunc for the cleanup destructor of btf_id %u\n",
> + dtors[i].btf_id);
> + return -EINVAL;
> + }
> + if (dtors[i].release_btf_id &&
> + btf_check_release_kfunc(btf, btf_relocate_id(btf, dtors[i].release_btf_id))) {
> + pr_err("btf_id %u is not a release kfunc for the destructor of btf_id %u\n",
> + dtors[i].release_btf_id, dtors[i].btf_id);
> + return -EINVAL;
> + }
> +
> dtor_func = btf_type_by_id(btf, dtor_btf_id);
> if (!dtor_func || !btf_type_is_func(dtor_func))
> return -EINVAL;
> @@ -10311,8 +10390,12 @@ int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_c
>
> /* remap BTF ids based on BTF relocation (if any) */
> for (i = tab_cnt; i < tab_cnt + add_cnt; i++) {
> - tab->dtors[i].btf_id = btf_relocate_id(btf, tab->dtors[i].btf_id);
> - tab->dtors[i].kfunc_btf_id = btf_relocate_id(btf, tab->dtors[i].kfunc_btf_id);
> + struct btf_id_dtor_kfunc *dtor = &tab->dtors[i];
> +
> + dtor->btf_id = btf_relocate_id(btf, dtor->btf_id);
> + dtor->kfunc_btf_id = btf_relocate_id(btf, dtor->kfunc_btf_id);
> + if (dtor->release_btf_id)
> + dtor->release_btf_id = btf_relocate_id(btf, dtor->release_btf_id);
> }
>
> tab->cnt += add_cnt;
[Severity: Critical]
This problem wasn't introduced by this patch, but can a failed call to
register_btf_id_dtor_kfuncs() wipe out all previously registered destructors
in the shared vmlinux BTF [1]?
Multiple initcalls append entries to the global vmlinux BTF's dtor_kfunc_tab
via krealloc(). However, if a later registration fails in
btf_check_dtor_kfuncs(), exceeds BTF_DTOR_KFUNC_MAX_CNT, or fails krealloc(),
the error path unconditionally frees the entire table:
kernel/bpf/btf.c:register_btf_id_dtor_kfuncs() {
...
end:
if (ret)
btf_free_dtor_kfunc_tab(btf);
btf_put(btf);
return ret;
}
Because the error paths in register_btf_id_dtor_kfuncs() run before the new
entries are appended to btf->dtor_kfunc_tab, dropping the
btf_free_dtor_kfunc_tab(btf) call on error would leave existing vmlinux
entries intact (while module BTFs still free their table in btf_free() if
module init fails).
[1] https://sashiko.dev/#/bug/linux-5f5fa9b2-fc98-4120-b5d5-adb2695404ab
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=8
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (7 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 08/29] bpf: Add flags and release kfuncs to the dtor registry Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 15:00 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 10/29] bpf: Record where JITed frames keep the BPF stack Kumar Kartikeya Dwivedi
` (20 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Register a BPF_DTOR_CLEANUP entry for every referenced type that a
program can hold but that has no kptr destructor, so that an unwinder can
release the references a program holds when it throws. For an open-coded
iterator, the type is the iterator struct and the release kfunc is its
KF_ITER_DESTROY kfunc.
The release kfuncs take a typed pointer, and calling one through a
void (*)(void *) pointer is a kCFI violation. Each entry therefore uses a
void (*)(void *) wrapper next to the kfunc, as the kptr destructors do.
In kernel/bpf, these are the num, task, task_vma, css, css_task, bits,
kmem_cache and dmabuf iterators, and bpf_key. They are registered from
kfunc_init() with one list of ids per entry: the groups under
CONFIG_CGROUPS, CONFIG_KEYS and CONFIG_DMA_SHARED_BUFFER would otherwise
shift the indices of a single list. The common kfunc set contains the
iterator destroy kfuncs, so it is now registered before the destructors.
Outside kernel/bpf, these are:
- xfrm_state and hid_bpf_ctx;
- nf_conn, released by bpf_ct_release();
- nf_conn___init, an entry that was allocated but not inserted. Its
destructor frees the entry, as bpf_ct_insert_entry() does when the
insert fails. bpf_ct_insert_entry() is the only kfunc that takes such
an entry, so it is the release kfunc;
- mem_cgroup, released by bpf_put_mem_cgroup();
- file, released by bpf_put_file();
- bpf_ws_lock, the wakeup sources read lock. Its handle is the SRCU
index plus one, never a pointer;
- the DSQ iterator of sched_ext;
- cpumask, for the masks that sched_ext hands out.
scx_bpf_put_cpumask() and scx_bpf_put_idle_cpumask() both do nothing,
so one entry covers both.
bpf_testmod gets entries for its two iterators.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
drivers/base/power/wakeup.c | 25 +++++-
drivers/hid/bpf/hid_bpf_dispatch.c | 23 +++++
drivers/hid/bpf/hid_bpf_dispatch.h | 1 +
fs/bpf_fs_kfuncs.c | 27 +++++-
include/linux/bpf.h | 14 ++++
include/net/netfilter/nf_conntrack_bpf.h | 1 +
include/net/xfrm.h | 1 +
kernel/bpf/bpf_iter.c | 6 ++
kernel/bpf/cgroup_iter.c | 6 ++
kernel/bpf/dmabuf_iter.c | 6 ++
kernel/bpf/helpers.c | 83 ++++++++++++++-----
kernel/bpf/kmem_cache_iter.c | 6 ++
kernel/bpf/task_iter.c | 18 ++++
kernel/sched/ext/ext.c | 41 +++++++++
mm/bpf_memcontrol.c | 28 ++++++-
net/netfilter/nf_conntrack_bpf.c | 38 +++++++++
net/xfrm/xfrm_state_bpf.c | 26 +++++-
.../selftests/bpf/test_kmods/bpf_testmod.c | 30 +++++++
18 files changed, 350 insertions(+), 30 deletions(-)
diff --git a/drivers/base/power/wakeup.c b/drivers/base/power/wakeup.c
index 80b497de2deb..16ccafbfcd6b 100644
--- a/drivers/base/power/wakeup.c
+++ b/drivers/base/power/wakeup.c
@@ -1170,6 +1170,7 @@ static const struct file_operations wakeup_sources_stats_fops = {
#ifdef CONFIG_BPF_SYSCALL
#include <linux/btf.h>
+#include <linux/btf_ids.h>
__bpf_kfunc_start_defs();
@@ -1202,6 +1203,12 @@ __bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock)
wakeup_sources_read_unlock((int)(long)lock - 1);
}
+__bpf_kfunc void bpf_wakeup_sources_read_unlock_dtor(void *lock)
+{
+ bpf_wakeup_sources_read_unlock(lock);
+}
+CFI_NOSEAL(bpf_wakeup_sources_read_unlock_dtor);
+
/**
* bpf_wakeup_sources_get_head - Get the head of the wakeup sources list
*
@@ -1224,9 +1231,25 @@ static const struct btf_kfunc_id_set wakeup_source_kfunc_set = {
.set = &wakeup_source_kfunc_ids,
};
+BTF_ID_LIST(wakeup_source_dtor_ids)
+BTF_ID(struct, bpf_ws_lock)
+BTF_ID(func, bpf_wakeup_sources_read_unlock_dtor)
+BTF_ID(func, bpf_wakeup_sources_read_unlock)
+
static void __init wakeup_sources_bpf_init(void)
{
- if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set))
+ const struct btf_id_dtor_kfunc wakeup_source_dtors[] = {
+ {
+ .btf_id = wakeup_source_dtor_ids[0],
+ .kfunc_btf_id = wakeup_source_dtor_ids[1],
+ .release_btf_id = wakeup_source_dtor_ids[2],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
+
+ if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set) ||
+ register_btf_id_dtor_kfuncs(wakeup_source_dtors, ARRAY_SIZE(wakeup_source_dtors),
+ THIS_MODULE))
pm_pr_dbg("Wakeup: failed to register BTF kfuncs\n");
}
#else
diff --git a/drivers/hid/bpf/hid_bpf_dispatch.c b/drivers/hid/bpf/hid_bpf_dispatch.c
index 6b467c74601c..897bae3f8c1b 100644
--- a/drivers/hid/bpf/hid_bpf_dispatch.c
+++ b/drivers/hid/bpf/hid_bpf_dispatch.c
@@ -357,6 +357,12 @@ hid_bpf_release_context(struct hid_bpf_ctx *ctx)
hid_put_device(hid);
}
+__bpf_kfunc void hid_bpf_release_context_dtor(void *ctx)
+{
+ hid_bpf_release_context(ctx);
+}
+CFI_NOSEAL(hid_bpf_release_context_dtor);
+
static int
__hid_bpf_hw_check_params(struct hid_bpf_ctx *ctx, __u8 *buf, size_t *buf__sz,
enum hid_report_type rtype, bool hw_request)
@@ -659,8 +665,21 @@ int hid_bpf_device_init(struct hid_device *hdev)
}
EXPORT_SYMBOL_GPL(hid_bpf_device_init);
+BTF_ID_LIST(hid_bpf_dtor_ids)
+BTF_ID(struct, hid_bpf_ctx)
+BTF_ID(func, hid_bpf_release_context_dtor)
+BTF_ID(func, hid_bpf_release_context)
+
static int __init hid_bpf_init(void)
{
+ const struct btf_id_dtor_kfunc hid_bpf_dtors[] = {
+ {
+ .btf_id = hid_bpf_dtor_ids[0],
+ .kfunc_btf_id = hid_bpf_dtor_ids[1],
+ .release_btf_id = hid_bpf_dtor_ids[2],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
int err;
/* Note: if we exit with an error any time here, we would entirely break HID, which
@@ -681,6 +700,10 @@ static int __init hid_bpf_init(void)
return 0;
}
+ err = register_btf_id_dtor_kfuncs(hid_bpf_dtors, ARRAY_SIZE(hid_bpf_dtors), THIS_MODULE);
+ if (err)
+ pr_warn("error while registering HID BPF destructors: %d\n", err);
+
return 0;
}
diff --git a/drivers/hid/bpf/hid_bpf_dispatch.h b/drivers/hid/bpf/hid_bpf_dispatch.h
index 44c6ea22233f..a198700a4394 100644
--- a/drivers/hid/bpf/hid_bpf_dispatch.h
+++ b/drivers/hid/bpf/hid_bpf_dispatch.h
@@ -16,6 +16,7 @@ void hid_put_device(struct hid_device *hid);
int hid_bpf_allocate_event_data(struct hid_device *hdev);
void __hid_bpf_ops_destroy_device(struct hid_device *hdev);
int hid_bpf_reconnect(struct hid_device *hdev);
+void hid_bpf_release_context_dtor(void *ctx);
struct bpf_prog;
diff --git a/fs/bpf_fs_kfuncs.c b/fs/bpf_fs_kfuncs.c
index 35ccd98a4de6..0c8e1d16a1ef 100644
--- a/fs/bpf_fs_kfuncs.c
+++ b/fs/bpf_fs_kfuncs.c
@@ -63,6 +63,12 @@ __bpf_kfunc void bpf_put_file(struct file *file)
fput(file);
}
+__bpf_kfunc void bpf_put_file_dtor(void *file)
+{
+ bpf_put_file(file);
+}
+CFI_NOSEAL(bpf_put_file_dtor);
+
/**
* bpf_path_d_path - resolve the pathname for the supplied path
* @path: path to resolve the pathname for
@@ -584,15 +590,28 @@ static const struct btf_kfunc_id_set bpf_fs_kfunc_set = {
.filter = bpf_fs_kfuncs_filter,
};
+BTF_ID_LIST(bpf_fs_dtor_ids)
+BTF_ID(struct, file)
+BTF_ID(func, bpf_put_file_dtor)
+BTF_ID(func, bpf_put_file)
+
static int __init bpf_fs_kfuncs_init(void)
{
+ const struct btf_id_dtor_kfunc bpf_fs_dtors[] = {
+ {
+ .btf_id = bpf_fs_dtor_ids[0],
+ .kfunc_btf_id = bpf_fs_dtor_ids[1],
+ .release_btf_id = bpf_fs_dtor_ids[2],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
int ret;
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_LSM, &bpf_fs_kfunc_set);
- if (ret || !IS_ENABLED(CONFIG_BINFMT_MISC_BPF))
- return ret;
- return register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
- &bpf_fs_kfunc_set);
+ if (!ret && IS_ENABLED(CONFIG_BINFMT_MISC_BPF))
+ ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_fs_kfunc_set);
+ return ret ?: register_btf_id_dtor_kfuncs(bpf_fs_dtors, ARRAY_SIZE(bpf_fs_dtors),
+ THIS_MODULE);
}
late_initcall(bpf_fs_kfuncs_init);
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 2b9adc4866bc..7f2db4e9afe0 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -3301,6 +3301,19 @@ int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
int bpf_iter_map_check_value_write(const struct bpf_prog *prog,
const struct bpf_map *map);
+/*
+ * void (*)(void *) wrappers of the open-coded iterator destroy kfuncs, which the BTF
+ * destructor registry refers to.
+ */
+void bpf_iter_num_destroy_dtor(void *it);
+void bpf_iter_task_destroy_dtor(void *it);
+void bpf_iter_task_vma_destroy_dtor(void *it);
+void bpf_iter_css_destroy_dtor(void *it);
+void bpf_iter_css_task_destroy_dtor(void *it);
+void bpf_iter_bits_destroy_dtor(void *it);
+void bpf_iter_kmem_cache_destroy_dtor(void *it);
+void bpf_iter_dmabuf_destroy_dtor(void *it);
+
int map_set_for_each_callback_args(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
struct bpf_func_state *callee);
@@ -4105,6 +4118,7 @@ struct bpf_key *bpf_lookup_system_key(u64 id);
struct bpf_key *bpf_lookup_keyring(void);
bool bpf_keyring_enforced(void);
void bpf_key_put(struct bpf_key *bkey);
+void bpf_key_put_dtor(void *bkey);
int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
const struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring);
diff --git a/include/net/netfilter/nf_conntrack_bpf.h b/include/net/netfilter/nf_conntrack_bpf.h
index 2d0da478c8e0..60ab309452c5 100644
--- a/include/net/netfilter/nf_conntrack_bpf.h
+++ b/include/net/netfilter/nf_conntrack_bpf.h
@@ -15,6 +15,7 @@ struct nf_conn___init {
extern int register_nf_conntrack_bpf(void);
extern void cleanup_nf_conntrack_bpf(void);
+void bpf_ct_release_dtor(void *nfct);
#else
diff --git a/include/net/xfrm.h b/include/net/xfrm.h
index 298ca9bb0083..561418052d33 100644
--- a/include/net/xfrm.h
+++ b/include/net/xfrm.h
@@ -2378,6 +2378,7 @@ static inline int register_xfrm_interface_bpf(void)
#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF)
int register_xfrm_state_bpf(void);
+void bpf_xdp_xfrm_state_release_dtor(void *x);
#else
static inline int register_xfrm_state_bpf(void)
{
diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c
index f0b29a170935..c5cc7ed1b918 100644
--- a/kernel/bpf/bpf_iter.c
+++ b/kernel/bpf/bpf_iter.c
@@ -846,4 +846,10 @@ __bpf_kfunc void bpf_iter_num_destroy(struct bpf_iter_num *it)
/* no-op */
}
+__bpf_kfunc void bpf_iter_num_destroy_dtor(void *it)
+{
+ bpf_iter_num_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_num_destroy_dtor);
+
__bpf_kfunc_end_defs();
diff --git a/kernel/bpf/cgroup_iter.c b/kernel/bpf/cgroup_iter.c
index fd51fe3d92cc..d1298d0d2404 100644
--- a/kernel/bpf/cgroup_iter.c
+++ b/kernel/bpf/cgroup_iter.c
@@ -372,4 +372,10 @@ __bpf_kfunc void bpf_iter_css_destroy(struct bpf_iter_css *it)
{
}
+__bpf_kfunc void bpf_iter_css_destroy_dtor(void *it)
+{
+ bpf_iter_css_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_css_destroy_dtor);
+
__bpf_kfunc_end_defs();
diff --git a/kernel/bpf/dmabuf_iter.c b/kernel/bpf/dmabuf_iter.c
index cd500248abd9..ac140a6cb570 100644
--- a/kernel/bpf/dmabuf_iter.c
+++ b/kernel/bpf/dmabuf_iter.c
@@ -189,4 +189,10 @@ __bpf_kfunc void bpf_iter_dmabuf_destroy(struct bpf_iter_dmabuf *it)
dma_buf_put(kit->dmabuf);
}
+__bpf_kfunc void bpf_iter_dmabuf_destroy_dtor(void *it)
+{
+ bpf_iter_dmabuf_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_dmabuf_destroy_dtor);
+
__bpf_kfunc_end_defs();
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index cbed0c8952a7..cc6d3a6a4de0 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3725,6 +3725,12 @@ __bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
bpf_mem_free(&bpf_global_ma, kit->bits);
}
+__bpf_kfunc void bpf_iter_bits_destroy_dtor(void *it)
+{
+ bpf_iter_bits_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_bits_destroy_dtor);
+
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
@@ -4584,6 +4590,12 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
kfree(bkey);
}
+__bpf_kfunc void bpf_key_put_dtor(void *bkey)
+{
+ bpf_key_put(bkey);
+}
+CFI_NOSEAL(bpf_key_put_dtor);
+
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
@@ -5314,16 +5326,40 @@ static const struct btf_kfunc_id_set user_mem_kfunc_set = {
.filter = user_mem_kfunc_filter,
};
-BTF_ID_LIST(generic_dtor_ids)
-BTF_ID(struct, task_struct)
-BTF_ID(func, bpf_task_release_dtor)
-BTF_ID(func, bpf_task_release)
+/* A type, its void (*)(void *) destructor, and the kfunc a program calls to release it */
+#define BPF_DTOR_ID_LIST(name, type, dtor, release) \
+BTF_ID_LIST(name) \
+BTF_ID(struct, type) \
+BTF_ID(func, dtor) \
+BTF_ID(func, release)
+
+BPF_DTOR_ID_LIST(task_dtor_ids, task_struct, bpf_task_release_dtor, bpf_task_release)
+BPF_DTOR_ID_LIST(iter_num_dtor_ids, bpf_iter_num, bpf_iter_num_destroy_dtor, bpf_iter_num_destroy)
+BPF_DTOR_ID_LIST(iter_task_dtor_ids, bpf_iter_task, bpf_iter_task_destroy_dtor,
+ bpf_iter_task_destroy)
+BPF_DTOR_ID_LIST(iter_task_vma_dtor_ids, bpf_iter_task_vma, bpf_iter_task_vma_destroy_dtor,
+ bpf_iter_task_vma_destroy)
+BPF_DTOR_ID_LIST(iter_bits_dtor_ids, bpf_iter_bits, bpf_iter_bits_destroy_dtor,
+ bpf_iter_bits_destroy)
+BPF_DTOR_ID_LIST(iter_kmem_cache_dtor_ids, bpf_iter_kmem_cache, bpf_iter_kmem_cache_destroy_dtor,
+ bpf_iter_kmem_cache_destroy)
#ifdef CONFIG_CGROUPS
-BTF_ID(struct, cgroup)
-BTF_ID(func, bpf_cgroup_release_dtor)
-BTF_ID(func, bpf_cgroup_release)
+BPF_DTOR_ID_LIST(cgroup_dtor_ids, cgroup, bpf_cgroup_release_dtor, bpf_cgroup_release)
+BPF_DTOR_ID_LIST(iter_css_dtor_ids, bpf_iter_css, bpf_iter_css_destroy_dtor, bpf_iter_css_destroy)
+BPF_DTOR_ID_LIST(iter_css_task_dtor_ids, bpf_iter_css_task, bpf_iter_css_task_destroy_dtor,
+ bpf_iter_css_task_destroy)
+#endif
+#ifdef CONFIG_KEYS
+BPF_DTOR_ID_LIST(key_dtor_ids, bpf_key, bpf_key_put_dtor, bpf_key_put)
+#endif
+#ifdef CONFIG_DMA_SHARED_BUFFER
+BPF_DTOR_ID_LIST(iter_dmabuf_dtor_ids, bpf_iter_dmabuf, bpf_iter_dmabuf_destroy_dtor,
+ bpf_iter_dmabuf_destroy)
#endif
+#define BPF_DTOR_ENTRY(ids, dtor_flags) \
+ { .btf_id = ids[0], .kfunc_btf_id = ids[1], .release_btf_id = ids[2], .flags = dtor_flags }
+
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL | KF_PERFMON)
@@ -5430,19 +5466,22 @@ static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
- {
- .btf_id = generic_dtor_ids[0],
- .kfunc_btf_id = generic_dtor_ids[1],
- .release_btf_id = generic_dtor_ids[2],
- .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
- },
+ BPF_DTOR_ENTRY(task_dtor_ids, BPF_DTOR_KPTR | BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_num_dtor_ids, BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_task_dtor_ids, BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_task_vma_dtor_ids, BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_bits_dtor_ids, BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_kmem_cache_dtor_ids, BPF_DTOR_CLEANUP),
#ifdef CONFIG_CGROUPS
- {
- .btf_id = generic_dtor_ids[3],
- .kfunc_btf_id = generic_dtor_ids[4],
- .release_btf_id = generic_dtor_ids[5],
- .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
- },
+ BPF_DTOR_ENTRY(cgroup_dtor_ids, BPF_DTOR_KPTR | BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_css_dtor_ids, BPF_DTOR_CLEANUP),
+ BPF_DTOR_ENTRY(iter_css_task_dtor_ids, BPF_DTOR_CLEANUP),
+#endif
+#ifdef CONFIG_KEYS
+ BPF_DTOR_ENTRY(key_dtor_ids, BPF_DTOR_CLEANUP),
+#endif
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ BPF_DTOR_ENTRY(iter_dmabuf_dtor_ids, BPF_DTOR_CLEANUP),
#endif
};
@@ -5453,10 +5492,10 @@ static int __init kfunc_init(void)
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
- ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
- ARRAY_SIZE(generic_dtors),
+ ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
+ /* The destructors refer to release kfuncs of both sets, so they come last */
+ return ret ?: register_btf_id_dtor_kfuncs(generic_dtors, ARRAY_SIZE(generic_dtors),
THIS_MODULE);
- return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
diff --git a/kernel/bpf/kmem_cache_iter.c b/kernel/bpf/kmem_cache_iter.c
index 3ae2158d767f..c629473a35ee 100644
--- a/kernel/bpf/kmem_cache_iter.c
+++ b/kernel/bpf/kmem_cache_iter.c
@@ -100,6 +100,12 @@ __bpf_kfunc void bpf_iter_kmem_cache_destroy(struct bpf_iter_kmem_cache *it)
kmem_cache_destroy(s);
}
+__bpf_kfunc void bpf_iter_kmem_cache_destroy_dtor(void *it)
+{
+ bpf_iter_kmem_cache_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_kmem_cache_destroy_dtor);
+
__bpf_kfunc_end_defs();
struct bpf_iter__kmem_cache {
diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c
index 2ee7b3b3c10b..df7c89fb61d0 100644
--- a/kernel/bpf/task_iter.c
+++ b/kernel/bpf/task_iter.c
@@ -1044,6 +1044,12 @@ __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
}
}
+__bpf_kfunc void bpf_iter_task_vma_destroy_dtor(void *it)
+{
+ bpf_iter_task_vma_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_task_vma_destroy_dtor);
+
__bpf_kfunc_end_defs();
#ifdef CONFIG_CGROUPS
@@ -1102,6 +1108,12 @@ __bpf_kfunc void bpf_iter_css_task_destroy(struct bpf_iter_css_task *it)
bpf_mem_free(&bpf_global_ma, kit->css_it);
}
+__bpf_kfunc void bpf_iter_css_task_destroy_dtor(void *it)
+{
+ bpf_iter_css_task_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_css_task_destroy_dtor);
+
__bpf_kfunc_end_defs();
#endif /* CONFIG_CGROUPS */
@@ -1192,6 +1204,12 @@ __bpf_kfunc void bpf_iter_task_destroy(struct bpf_iter_task *it)
{
}
+__bpf_kfunc void bpf_iter_task_destroy_dtor(void *it)
+{
+ bpf_iter_task_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_task_destroy_dtor);
+
__bpf_kfunc_end_defs();
DEFINE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index f74ffb3b1aca..0b75cdbf1b23 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -11068,6 +11068,12 @@ __bpf_kfunc void bpf_iter_scx_dsq_destroy(struct bpf_iter_scx_dsq *it)
kit->dsq = NULL;
}
+__bpf_kfunc void bpf_iter_scx_dsq_destroy_dtor(void *it)
+{
+ bpf_iter_scx_dsq_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_scx_dsq_destroy_dtor);
+
/**
* scx_bpf_dsq_peek - Lockless peek at the first element.
* @dsq_id: DSQ to examine.
@@ -11753,6 +11759,13 @@ __bpf_kfunc void scx_bpf_put_cpumask(const struct cpumask *cpumask)
*/
}
+/* Releases what scx_bpf_put_idle_cpumask() does as well: neither holds a reference. */
+__bpf_kfunc void scx_bpf_put_cpumask_dtor(void *cpumask)
+{
+ scx_bpf_put_cpumask(cpumask);
+}
+CFI_NOSEAL(scx_bpf_put_cpumask_dtor);
+
/**
* scx_bpf_task_running - Is task the current scheduling context?
* @p: task of interest
@@ -12328,8 +12341,30 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id)
return -EACCES;
}
+BTF_ID_LIST(scx_dtor_ids)
+BTF_ID(struct, cpumask)
+BTF_ID(func, scx_bpf_put_cpumask_dtor)
+BTF_ID(func, scx_bpf_put_cpumask)
+BTF_ID(struct, bpf_iter_scx_dsq)
+BTF_ID(func, bpf_iter_scx_dsq_destroy_dtor)
+BTF_ID(func, bpf_iter_scx_dsq_destroy)
+
static int __init scx_init(void)
{
+ const struct btf_id_dtor_kfunc scx_dtors[] = {
+ {
+ .btf_id = scx_dtor_ids[0],
+ .kfunc_btf_id = scx_dtor_ids[1],
+ .release_btf_id = scx_dtor_ids[2],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ {
+ .btf_id = scx_dtor_ids[3],
+ .kfunc_btf_id = scx_dtor_ids[4],
+ .release_btf_id = scx_dtor_ids[5],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
int ret;
/*
@@ -12439,6 +12474,12 @@ static int __init scx_init(void)
return ret;
}
+ ret = register_btf_id_dtor_kfuncs(scx_dtors, ARRAY_SIZE(scx_dtors), THIS_MODULE);
+ if (ret) {
+ pr_err("sched_ext: Failed to register destructors (%d)\n", ret);
+ return ret;
+ }
+
ret = scx_idle_init();
if (ret) {
pr_err("sched_ext: Failed to initialize idle tracking (%d)\n", ret);
diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
index d8f579c28560..4cd37301df23 100644
--- a/mm/bpf_memcontrol.c
+++ b/mm/bpf_memcontrol.c
@@ -82,6 +82,12 @@ __bpf_kfunc void bpf_put_mem_cgroup(struct mem_cgroup *memcg)
css_put(&memcg->css);
}
+__bpf_kfunc void bpf_put_mem_cgroup_dtor(void *memcg)
+{
+ bpf_put_mem_cgroup(memcg);
+}
+CFI_NOSEAL(bpf_put_mem_cgroup_dtor);
+
/**
* bpf_mem_cgroup_vm_events - Read memory cgroup's vm event counter
* @memcg: memory cgroup
@@ -235,8 +241,21 @@ static const struct btf_kfunc_id_set bpf_memcontrol_reclaim_kfunc_set = {
.set = &bpf_memcontrol_reclaim_kfuncs,
};
+BTF_ID_LIST(bpf_memcontrol_dtor_ids)
+BTF_ID(struct, mem_cgroup)
+BTF_ID(func, bpf_put_mem_cgroup_dtor)
+BTF_ID(func, bpf_put_mem_cgroup)
+
static int __init bpf_memcontrol_init(void)
{
+ const struct btf_id_dtor_kfunc bpf_memcontrol_dtors[] = {
+ {
+ .btf_id = bpf_memcontrol_dtor_ids[0],
+ .kfunc_btf_id = bpf_memcontrol_dtor_ids[1],
+ .release_btf_id = bpf_memcontrol_dtor_ids[2],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
int err;
err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
@@ -248,8 +267,15 @@ static int __init bpf_memcontrol_init(void)
err = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
&bpf_memcontrol_reclaim_kfunc_set);
- if (err)
+ if (err) {
pr_warn("error registering bpf reclaim kfuncs: %d\n", err);
+ return err;
+ }
+
+ err = register_btf_id_dtor_kfuncs(bpf_memcontrol_dtors, ARRAY_SIZE(bpf_memcontrol_dtors),
+ THIS_MODULE);
+ if (err)
+ pr_warn("error registering bpf memcontrol destructors: %d\n", err);
return err;
}
diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c
index c2df7c948281..9d9d4e97ca4e 100644
--- a/net/netfilter/nf_conntrack_bpf.c
+++ b/net/netfilter/nf_conntrack_bpf.c
@@ -457,6 +457,22 @@ __bpf_kfunc void bpf_ct_release(struct nf_conn *nfct)
nf_ct_put(nfct);
}
+__bpf_kfunc void bpf_ct_release_dtor(void *nfct)
+{
+ bpf_ct_release(nfct);
+}
+CFI_NOSEAL(bpf_ct_release_dtor);
+
+/*
+ * An entry that was allocated and not inserted has no reference to put: it is
+ * freed, as bpf_ct_insert_entry() does with one it cannot insert.
+ */
+__bpf_kfunc void bpf_ct_free_dtor(void *nfct)
+{
+ nf_conntrack_free(nfct);
+}
+CFI_NOSEAL(bpf_ct_free_dtor);
+
/* bpf_ct_set_timeout - Set timeout of allocated nf_conn
*
* Sets the default timeout of newly allocated nf_conn before insertion.
@@ -537,12 +553,34 @@ static const struct btf_kfunc_id_set nf_conntrack_kfunc_set = {
.set = &nf_ct_kfunc_set,
};
+BTF_ID_LIST(nf_ct_dtor_ids)
+BTF_ID(func, bpf_ct_release_dtor)
+BTF_ID(func, bpf_ct_release)
+BTF_ID(func, bpf_ct_free_dtor)
+BTF_ID(func, bpf_ct_insert_entry)
+
int register_nf_conntrack_bpf(void)
{
+ /* The only kfunc that takes an allocated entry off a program is bpf_ct_insert_entry() */
+ const struct btf_id_dtor_kfunc nf_ct_dtors[] = {
+ {
+ .btf_id = btf_nf_conn_ids[0],
+ .kfunc_btf_id = nf_ct_dtor_ids[0],
+ .release_btf_id = nf_ct_dtor_ids[1],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ {
+ .btf_id = btf_nf_conn_ids[1],
+ .kfunc_btf_id = nf_ct_dtor_ids[2],
+ .release_btf_id = nf_ct_dtor_ids[3],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
int ret;
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &nf_conntrack_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &nf_conntrack_kfunc_set);
+ ret = ret ?: register_btf_id_dtor_kfuncs(nf_ct_dtors, ARRAY_SIZE(nf_ct_dtors), THIS_MODULE);
if (!ret) {
mutex_lock(&nf_conn_btf_access_lock);
nfct_btf_struct_access = _nf_conntrack_btf_struct_access;
diff --git a/net/xfrm/xfrm_state_bpf.c b/net/xfrm/xfrm_state_bpf.c
index 4180c317f9bc..560b444aee80 100644
--- a/net/xfrm/xfrm_state_bpf.c
+++ b/net/xfrm/xfrm_state_bpf.c
@@ -115,6 +115,12 @@ __bpf_kfunc void bpf_xdp_xfrm_state_release(struct xfrm_state *x)
xfrm_state_put(x);
}
+__bpf_kfunc void bpf_xdp_xfrm_state_release_dtor(void *x)
+{
+ bpf_xdp_xfrm_state_release(x);
+}
+CFI_NOSEAL(bpf_xdp_xfrm_state_release_dtor);
+
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(xfrm_state_kfunc_set)
@@ -127,8 +133,24 @@ static const struct btf_kfunc_id_set xfrm_state_xdp_kfunc_set = {
.set = &xfrm_state_kfunc_set,
};
+BTF_ID_LIST(xfrm_state_dtor_ids)
+BTF_ID(struct, xfrm_state)
+BTF_ID(func, bpf_xdp_xfrm_state_release_dtor)
+BTF_ID(func, bpf_xdp_xfrm_state_release)
+
int __init register_xfrm_state_bpf(void)
{
- return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP,
- &xfrm_state_xdp_kfunc_set);
+ const struct btf_id_dtor_kfunc xfrm_state_dtors[] = {
+ {
+ .btf_id = xfrm_state_dtor_ids[0],
+ .kfunc_btf_id = xfrm_state_dtor_ids[1],
+ .release_btf_id = xfrm_state_dtor_ids[2],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ };
+ int ret;
+
+ ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &xfrm_state_xdp_kfunc_set);
+ return ret ?: register_btf_id_dtor_kfuncs(xfrm_state_dtors, ARRAY_SIZE(xfrm_state_dtors),
+ THIS_MODULE);
}
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 891f777a265a..db1a7073aee3 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -281,6 +281,12 @@ __bpf_kfunc void bpf_iter_testmod_seq_destroy(struct bpf_iter_testmod_seq *it)
it->cnt = 0;
}
+__bpf_kfunc void bpf_iter_testmod_seq_destroy_dtor(void *it)
+{
+ bpf_iter_testmod_seq_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_testmod_seq_destroy_dtor);
+
__bpf_kfunc int bpf_iter_testmod_current_new(struct bpf_iter_testmod_current *it)
{
it->remaining = 2;
@@ -302,6 +308,12 @@ __bpf_kfunc void bpf_iter_testmod_current_destroy(struct bpf_iter_testmod_curren
it->remaining = 0;
}
+__bpf_kfunc void bpf_iter_testmod_current_destroy_dtor(void *it)
+{
+ bpf_iter_testmod_current_destroy(it);
+}
+CFI_NOSEAL(bpf_iter_testmod_current_destroy_dtor);
+
__bpf_kfunc void bpf_kfunc_common_test(void)
{
}
@@ -1085,6 +1097,12 @@ BTF_ID_LIST(bpf_testmod_dtor_ids)
BTF_ID(struct, bpf_testmod_ctx)
BTF_ID(func, bpf_testmod_ctx_release_dtor)
BTF_ID(func, bpf_testmod_ctx_release)
+BTF_ID(struct, bpf_iter_testmod_seq)
+BTF_ID(func, bpf_iter_testmod_seq_destroy_dtor)
+BTF_ID(func, bpf_iter_testmod_seq_destroy)
+BTF_ID(struct, bpf_iter_testmod_current)
+BTF_ID(func, bpf_iter_testmod_current_destroy_dtor)
+BTF_ID(func, bpf_iter_testmod_current_destroy)
static const struct btf_kfunc_id_set bpf_testmod_common_kfunc_set = {
.owner = THIS_MODULE,
@@ -2627,6 +2645,18 @@ static int bpf_testmod_init(void)
.release_btf_id = bpf_testmod_dtor_ids[2],
.flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP,
},
+ {
+ .btf_id = bpf_testmod_dtor_ids[3],
+ .kfunc_btf_id = bpf_testmod_dtor_ids[4],
+ .release_btf_id = bpf_testmod_dtor_ids[5],
+ .flags = BPF_DTOR_CLEANUP,
+ },
+ {
+ .btf_id = bpf_testmod_dtor_ids[6],
+ .kfunc_btf_id = bpf_testmod_dtor_ids[7],
+ .release_btf_id = bpf_testmod_dtor_ids[8],
+ .flags = BPF_DTOR_CLEANUP,
+ },
};
void **tramp;
int ret;
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types
2026-11-11 12:57 ` [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 15:00 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3354 bytes --]
> bpf: Register cleanup destructors for more types
TL;DR: With CONFIG_NF_CONNTRACK=m, the nf_conn cleanup destructor is
registered in module BTF but btf_find_dtor() may never find it, so XDP/TC
programs holding nf_conn that can bpf_throw() would be rejected; no selftest
covers it.
> diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c
> --- a/net/netfilter/nf_conntrack_bpf.c
> +++ b/net/netfilter/nf_conntrack_bpf.c
[ ... ]
> @@ -537,12 +553,34 @@ static const struct btf_kfunc_id_set nf_conntrack_kfunc_set = {
> int register_nf_conntrack_bpf(void)
> {
> + /* The only kfunc that takes an allocated entry off a program is bpf_ct_insert_entry() */
> + const struct btf_id_dtor_kfunc nf_ct_dtors[] = {
> + {
> + .btf_id = btf_nf_conn_ids[0],
> + .kfunc_btf_id = nf_ct_dtor_ids[0],
> + .release_btf_id = nf_ct_dtor_ids[1],
> + .flags = BPF_DTOR_CLEANUP,
> + },
[ ... ]
> ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &nf_conntrack_kfunc_set);
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &nf_conntrack_kfunc_set);
> + ret = ret ?: register_btf_id_dtor_kfuncs(nf_ct_dtors, ARRAY_SIZE(nf_ct_dtors), THIS_MODULE);
With CONFIG_NF_CONNTRACK=m, can the nf_conn entry ever be found here?
struct nf_conn is always in vmlinux BTF when conntrack is enabled, since
net/core/flow_dissector.o is built in and skb_flow_dissect_ct()
dereferences a struct nf_conn under IS_ENABLED(CONFIG_NF_CONNTRACK), which
is also true for =m. Module BTF is deduplicated against vmlinux, so
BTF_ID(struct, nf_conn) in nf_conntrack.ko (btf_nf_conn_ids[0]) resolves
to the vmlinux type id, below the module BTF's start_id.
register_btf_id_dtor_kfuncs(..., THIS_MODULE) stores that entry in the
module BTF's dtor_kfunc_tab. But btf_find_dtor() does:
while (btf_id < btf->start_id)
btf = btf->base_btf;
dtor = btf_dtor_tab_find(btf, btf_id);
if (dtor || !btf_is_module(btf)) {
*btfp = btf;
return dtor;
}
The reference returned by bpf_xdp_ct_lookup(), bpf_skb_ct_lookup() or
bpf_ct_insert_entry() has reg->btf set to the nf_conntrack module BTF and
reg->btf_id set to the vmlinux id. The loop moves to vmlinux BTF, finds no
nf_conn entry there, and returns NULL because vmlinux is not a module. The
module table is never searched, which breaks the assumption stated in the
function's own comment that a type below its first id is vmlinux's, with
its destructor registered there.
With conntrack built as a module, resource_registry() in
kernel/bpf/frame_desc.c (added later in the series) would then fail with
"frame_desc: no cleanup destructor registered for object nf_conn", so an
XDP or TC program holding a looked-up or inserted nf_conn where it can
reach bpf_throw() is still rejected. Only CONFIG_NF_CONNTRACK=y, as the
selftests use, makes the entry reachable, and no selftest in the series
covers nf_conn.
The nf_conn___init entry is not affected, since that type exists only in
the module's BTF. Should the nf_conn entry be registered against vmlinux
BTF, or btf_find_dtor() be taught to also search the module's table?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types
2026-11-11 12:57 ` [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 15:00 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 15:00 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] bpf: btf_find_dtor() misses nf_conn destructors when
CONFIG_NF_CONNTRACK=m
--
commit ddb65cf317bc73659c05f7b86c4aa44d5480e3c1
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Register cleanup destructors for more types
This commit registers BPF_DTOR_CLEANUP destructors and void (*)(void *)
wrappers for open-coded iterators and referenced types across the kernel
so that stack unwinding on bpf_throw() can release held references.
> diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c
> index c2df7c9482812..9d9d4e97ca4e5 100644
> --- a/net/netfilter/nf_conntrack_bpf.c
> +++ b/net/netfilter/nf_conntrack_bpf.c
[ ... ]
> @@ -537,12 +553,34 @@ static const struct btf_kfunc_id_set nf_conntrack_kfunc_set = {
> .set = &nf_ct_kfunc_set,
> };
>
> +BTF_ID_LIST(nf_ct_dtor_ids)
> +BTF_ID(func, bpf_ct_release_dtor)
> +BTF_ID(func, bpf_ct_release)
> +BTF_ID(func, bpf_ct_free_dtor)
> +BTF_ID(func, bpf_ct_insert_entry)
> +
> int register_nf_conntrack_bpf(void)
> {
> + /* The only kfunc that takes an allocated entry off a program is bpf_ct_insert_entry() */
> + const struct btf_id_dtor_kfunc nf_ct_dtors[] = {
> + {
> + .btf_id = btf_nf_conn_ids[0],
> + .kfunc_btf_id = nf_ct_dtor_ids[0],
> + .release_btf_id = nf_ct_dtor_ids[1],
> + .flags = BPF_DTOR_CLEANUP,
> + },
> + {
> + .btf_id = btf_nf_conn_ids[1],
> + .kfunc_btf_id = nf_ct_dtor_ids[2],
> + .release_btf_id = nf_ct_dtor_ids[3],
> + .flags = BPF_DTOR_CLEANUP,
> + },
> + };
> int ret;
>
> ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &nf_conntrack_kfunc_set);
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &nf_conntrack_kfunc_set);
> + ret = ret ?: register_btf_id_dtor_kfuncs(nf_ct_dtors, ARRAY_SIZE(nf_ct_dtors), THIS_MODULE);
[Severity: Medium]
Can btf_find_dtor() find nf_ct_dtors when CONFIG_NF_CONNTRACK=m?
tc_cls_act_func_proto() in net/core/filter.c executes
BTF_TYPE_EMIT(struct nf_conn___init), so both struct nf_conn
(btf_nf_conn_ids[0]) and struct nf_conn___init (btf_nf_conn_ids[1]) are
emitted into vmlinux base BTF and have BTF IDs below nf_conntrack's module
btf->start_id. However, register_btf_id_dtor_kfuncs() stores nf_ct_dtors in
nf_conntrack's module dtor_kfunc_tab via THIS_MODULE.
When resource_registry() looks up the destructor, btf_find_dtor() walks to
base_btf before searching the module's table:
kernel/bpf/btf.c:btf_find_dtor() {
...
while (btf_id < btf->start_id)
btf = btf->base_btf;
dtor = btf_dtor_tab_find(btf, btf_id);
if (dtor || !btf_is_module(btf)) {
*btfp = btf;
return dtor;
}
...
}
Because btf_id < btf->start_id, btf_find_dtor() skips nf_conntrack's
dtor_kfunc_tab, searches vmlinux's table where nf_ct_dtors was not
registered, and returns NULL. As a result, the verifier rejects any XDP or
sched_cls program that calls bpf_throw() while holding a conntrack reference
when CONFIG_NF_CONNTRACK=m.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=9
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 10/29] bpf: Record where JITed frames keep the BPF stack
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (8 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 09/29] bpf: Register cleanup destructors for more types Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 11/29] bpf: Give every reference an owner frame Kumar Kartikeya Dwivedi
` (19 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
The unwinder will release resources that a frame keeps on its BPF stack.
The verifier gives their locations relative to the frame's R10. The stack
walk reports the native frame pointer of a frame, and the position of R10
relative to it depends on the architecture and on the stack that the
frame uses:
- On x86-64, R10 is the native frame pointer.
- On arm64, R10 is the stack pointer after the frame record, the tail
call count pair and the callee-saved pairs are pushed. This is
-16 - 16 * ceil(pushed / 2) below x29. Record the distance as
frame_fp_off in the program's aux data. x86-64 leaves it 0.
- A frame that uses a private stack keeps its BPF stack in the per-CPU
private area, so its R10 is not derived from the frame pointer. The
JIT knows that R10 is a fixed distance into the per-CPU private stack
of the function. Record the distance as priv_frame_off.
The arm64 JIT sets up R10 only for a program that uses it, but records
the distances for every program.
Add bpf_jit_supports_exception_cleanup(), which tells the verifier
whether the JIT records these. The other JITs that support exceptions do
not record them yet, and keep the rule that bpf_throw() must be reached
with no resources held.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
arch/arm64/net/bpf_jit_comp.c | 16 ++++++++++++++++
arch/x86/net/bpf_jit_comp.c | 7 +++++++
include/linux/bpf.h | 4 ++++
include/linux/filter.h | 1 +
kernel/bpf/core.c | 10 ++++++++++
5 files changed, 38 insertions(+)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c9f851bd5003..418bda24020c 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1055,6 +1055,16 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
/* Stack must be multiples of 16B */
ctx->stack_size = round_up(prog->aux->stack_depth, 16);
+ /*
+ * Where the exception unwinder finds this frame's BPF stack: the
+ * private stack, or below the frame record, the tail call count pair
+ * and the pairs push_callee_regs() pushed.
+ */
+ if (ctx->priv_sp_used)
+ prog->aux->priv_frame_off = PRIV_STACK_GUARD_SZ + ctx->stack_size;
+ else
+ prog->aux->frame_fp_off = -16 - 16 * DIV_ROUND_UP(ctx->nr_used_callee_reg, 2);
+
if (ctx->fp_used) {
if (ctx->priv_sp_used) {
/* Set up private stack pointer */
@@ -3904,6 +3914,12 @@ bool bpf_jit_supports_exceptions(void)
return true;
}
+/* build_prologue() records where each frame keeps its BPF stack. */
+bool bpf_jit_supports_exception_cleanup(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_arena(void)
{
return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 9b7466ca1a82..9170cbe3fa16 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2425,6 +2425,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
if (priv_stack_ptr) {
priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
+ bpf_prog->aux->priv_frame_off = PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
stack_depth = 0;
}
@@ -5066,6 +5067,12 @@ bool bpf_jit_supports_exceptions(void)
return IS_ENABLED(CONFIG_UNWINDER_ORC);
}
+/* On x86-64 R10 is the frame pointer; do_jit() records where a private stack has it. */
+bool bpf_jit_supports_exception_cleanup(void)
+{
+ return bpf_jit_supports_exceptions();
+}
+
bool bpf_jit_supports_private_stack(void)
{
return true;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 7f2db4e9afe0..17b13b16899d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2010,6 +2010,10 @@ struct bpf_prog_aux {
* by the JIT; consumed by the exception unwinder.
*/
s16 callee_reg_off[BPF_THROW_NR_CALLEE_SAVED];
+ /* BPF frame pointer relative to the native frame pointer the stack walker reports */
+ s16 frame_fp_off;
+ /* With a private stack, the frame's R10 is this far into this CPU's priv_stack_ptr. */
+ u32 priv_frame_off;
#ifdef CONFIG_SECURITY
void *security;
#endif
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 61448be53f8d..08ad906f649f 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1391,6 +1391,7 @@ bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_exception_cleanup(void);
bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 7a25d1cb2c30..59478a08b247 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3896,6 +3896,16 @@ bool __weak bpf_jit_supports_exceptions(void)
return false;
}
+/*
+ * Does the JIT record where each frame keeps its BPF stack, see
+ * bpf_prog_aux::frame_fp_off, so that the unwinder can release what a frame
+ * holds when bpf_throw() discards it?
+ */
+bool __weak bpf_jit_supports_exception_cleanup(void)
+{
+ return false;
+}
+
bool __weak bpf_jit_supports_private_stack(void)
{
return false;
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 11/29] bpf: Give every reference an owner frame
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (9 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 10/29] bpf: Record where JITed frames keep the BPF stack Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 13:33 ` bot+bpf-ci
2026-11-11 12:57 ` [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw() Kumar Kartikeya Dwivedi
` (18 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
The unwinder will release the resources of each frame from that frame's
own stack slots. Each reference in the verifier state must therefore
belong to exactly one frame. The verifier has no such notion today: the
state holds the references, and a frame only knows how many of the
references held at its entry are still held.
Record the owner frame in the reference state:
- The frame that acquires a reference owns it.
- When a frame returns, the references it still holds pass to its
caller, which is the only frame left that can release them.
- A function that is verified on its own inherits the references of its
callers without their frames, see call_func_on_its_own(). No frame of
the state owns these inherited references, so their owner is -1.
References that the function returns with, beyond the inherited ones,
pass to the caller when the call is resumed in bpf_flow_return().
Callbacks release everything they acquire, so nothing changes owner when
a callback returns, and the transfer in prepare_func_exit() has nothing
to do for them.
RCU read and preemption disabled sections have no reference state. The
verifier tracks, per frame, how many of the sections held at its entry
are still held, and release_context_state() lowers that count on every
unlock. A frame that left a section of its caller therefore looks like a
frame that was entered with fewer sections. Also count how many sections
each frame released below its entry count, so that the exit of the frame
can tell that it returns with fewer sections than it was called with.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 13 +++++++++++++
kernel/bpf/verifier.c | 22 ++++++++++++++++++++--
2 files changed, 33 insertions(+), 2 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7a2b6507f562..c18efe4bd54a 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -327,6 +327,12 @@ struct bpf_reference_state {
* is used purely to inform the user of a reference leak.
*/
int insn_idx;
+ /*
+ * The frame that owns the reference: the one that acquired it, or the
+ * caller a returning frame handed it to. -1 for what the callers of a
+ * function verified on its own hold, which are not in the state.
+ */
+ s8 frameno;
union {
/* For REF_TYPE_PTR */
int parent_id;
@@ -503,6 +509,13 @@ struct bpf_func_state {
u32 entry_refs;
u32 entry_rcu_locks;
u32 entry_preempt_locks;
+ /*
+ * How many of the RCU read and preemption disabled sections held when the
+ * frame was entered it left, or a callee of it did: a frame that returns
+ * after that is in fewer sections than it was called in.
+ */
+ u32 rcu_released_below;
+ u32 preempt_released_below;
/* The following fields should be last. See copy_func_state() */
/* The state of the stack. Each element of the array describes BPF_REG_SIZE
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0684ac1f4f22..6c71ceb1dd4c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1545,6 +1545,7 @@ static struct bpf_reference_state *acquire_reference_state(struct bpf_verifier_e
if (err)
return NULL;
state->refs[new_ofs].insn_idx = insn_idx;
+ state->refs[new_ofs].frameno = state->curframe;
return &state->refs[new_ofs];
}
@@ -1625,7 +1626,11 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx)
return;
}
-/* bpf_rcu_read_unlock() or bpf_preempt_enable() is done, see release_reference_state() */
+/*
+ * bpf_rcu_read_unlock() or bpf_preempt_enable() is done, see release_reference_state().
+ * A frame that was entered with more sections than are held now has left one of its
+ * callers', and counts that for its exit to tell.
+ */
static void release_context_state(struct bpf_verifier_state *state)
{
struct bpf_func_state *frame;
@@ -1633,6 +1638,11 @@ static void release_context_state(struct bpf_verifier_state *state)
for (fr = 0; fr <= state->curframe; fr++) {
frame = state->frame[fr];
+ if (frame->entry_rcu_locks > state->active_rcu_locks)
+ frame->rcu_released_below += frame->entry_rcu_locks - state->active_rcu_locks;
+ if (frame->entry_preempt_locks > state->active_preempt_locks)
+ frame->preempt_released_below += frame->entry_preempt_locks -
+ state->active_preempt_locks;
frame->entry_rcu_locks = min(frame->entry_rcu_locks, state->active_rcu_locks);
frame->entry_preempt_locks = min(frame->entry_preempt_locks,
state->active_preempt_locks);
@@ -11709,6 +11719,8 @@ static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int i
entry.refs[i].id = inherited_id(state, entry.refs[i].id);
if (entry.refs[i].type == REF_TYPE_PTR)
entry.refs[i].parent_id = inherited_id(state, entry.refs[i].parent_id);
+ /* the frames that own what is inherited are not in the state */
+ entry.refs[i].frameno = -1;
}
entry.active_irq_id = inherited_id(state, entry.active_irq_id);
entry.active_lock_id = inherited_id(state, entry.active_lock_id);
@@ -11878,11 +11890,12 @@ int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *cal
reg->parent_id = fresh_id(env, idmap, reg->parent_id);
reg->map_uid = fresh_id(env, idmap, reg->map_uid);
}
- /* those that were held are the first */
+ /* those that were held are the first; what the function took is the caller's now */
for (i = nheld; i < caller->acquired_refs; i++) {
caller->refs[i].id = fresh_id(env, idmap, caller->refs[i].id);
if (caller->refs[i].type == REF_TYPE_PTR)
caller->refs[i].parent_id = fresh_id(env, idmap, caller->refs[i].parent_id);
+ caller->refs[i].frameno = caller->curframe;
}
caller->active_irq_id = fresh_id(env, idmap, caller->active_irq_id);
caller->active_lock_id = fresh_id(env, idmap, caller->active_lock_id);
@@ -12688,6 +12701,11 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return err;
invalidate_stack_slices(env, callee->frameno, INT_MIN, 0);
+ /* What the frame holds when it returns is the caller's to release. */
+ for (i = 0; i < state->acquired_refs; i++)
+ if (state->refs[i].frameno == callee->frameno)
+ state->refs[i].frameno = caller->frameno;
+
/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
* there function call logic would reschedule callback visit.
*/
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 11/29] bpf: Give every reference an owner frame
2026-11-11 12:57 ` [PATCH bpf-next v1 11/29] bpf: Give every reference an owner frame Kumar Kartikeya Dwivedi
@ 2026-11-11 13:33 ` bot+bpf-ci
0 siblings, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 13:33 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3339 bytes --]
> bpf: Give every reference an owner frame
TL;DR: Commit message only: the claim that callbacks release everything they
acquire may be inaccurate, since IRQ or lock references acquired in a callback
seem to pass to the caller in prepare_func_exit().
> bpf: Give every reference an owner frame
>
> The unwinder will release the resources of each frame from that frame's
> own stack slots. Each reference in the verifier state must therefore
> belong to exactly one frame. The verifier has no such notion today: the
> state holds the references, and a frame only knows how many of the
> references held at its entry are still held.
>
> [ ... ]
>
> Callbacks release everything they acquire, so nothing changes owner when
> a callback returns, and the transfer in prepare_func_exit() has nothing
> to do for them.
>
> [ ... ]
>
> Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Is the statement that callbacks release everything they acquire accurate
for every reference type?
At a callback's BPF_EXIT, process_bpf_exit_full() calls check_resource_leak()
with check_lock = exception_exit || !curframe, which is false in a callback
frame. So active_locks and active_irq_id are not checked there, and
check_reference_leak() returns 0 for frameno != 0.
The only callback-specific check in prepare_func_exit() skips everything
that is not REF_TYPE_PTR:
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -12688,6 +12701,11 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
> return err;
> invalidate_stack_slices(env, callee->frameno, INT_MIN, 0);
>
> + /* What the frame holds when it returns is the caller's to release. */
> + for (i = 0; i < state->acquired_refs; i++)
> + if (state->refs[i].frameno == callee->frameno)
> + state->refs[i].frameno = caller->frameno;
> +
A REF_TYPE_IRQ, REF_TYPE_LOCK or REF_TYPE_RES_LOCK* entry acquired by a
callback survives the callback's exit with frameno == callee->frameno, and
this loop then reassigns it to the caller.
For example, a bpf_loop() callback can call bpf_local_irq_save() on a flag
that lives in the caller's stack, since bpf_loop() passes ctx as a
PTR_TO_STACK of the caller and process_irq_flag()/mark_stack_slot_irq_flag()
accept a slot in any frame. The callback returns with IRQs still saved, and
the caller restores them after the loop.
For a bpf_spin_lock taken in a callback, the transfer runs too, although
the re-simulated helper call at the callsite later rejects the program.
Handing the IRQ reference to the frame whose stack holds the flag looks like
a sensible owner. But the sentence in the changelog does not seem to match
this, and later patches rely on the ownership model it describes (for
example, prepare_func_exit() later calls bpf_frame_desc_record_return() for
every handed reference, including those of callbacks).
Could the changelog say that only REF_TYPE_PTR references are guaranteed to
be released by a callback, and that IRQ/lock entries can still pass to the
caller here?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw()
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (10 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 11/29] bpf: Give every reference an owner frame Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 15:30 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots Kumar Kartikeya Dwivedi
` (17 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently bpf_throw() is only allowed where the program holds no
resources. The frames between the throw and the main program are
discarded without releasing anything, so the verifier rejects a throw
that is reached with a reference, a lock, or a section held.
Add the runtime half of releasing the resources of these frames. A frame
will keep the handles of the kernel resources it holds in hidden stack
slots, which the verifier maintains from the next patch on. A handle is
the value that the release takes: a pointer to an object, a ringbuf
record or a lock, or the address of an iterator or of saved IRQ flags on
the stack.
A frame descriptor lists the hidden slots in the frames of one JITed
function:
- Each entry gives the offset of a slot from the frame's R10 and the
resource class of the handle in it. An empty slot holds 0.
- A resource class says how to release a handle: the kind of resource,
and the destructor or the struct meta of the object.
- RCU read and preemption disabled sections have no handle. A frame
counts the sections it entered in one slot per kind.
The descriptors and resource classes of a program live in a pool owned by
the main program. Each JITed function refers to its descriptor by index,
and identical descriptors are shared.
bpf_throw() releases the resources in the slots of each frame before it
walks the stack for the exception callback. The release order is the same
for all frames, so the frames are walked once per phase:
1. locks and IRQ flags;
2. objects whose destructor needs RCU protection;
3. the sections, using counts that the first walk summed over the
frames;
4. all other objects.
In the first phase, IRQ flags that a frame saved while interrupts were
enabled are restored last. Every later save found interrupts disabled, so
a frame has at most one such set, and the locks and the other flags can
be released in any order before it.
The unwinder releases a socket with bpf_sk_release(), and the reader of a
file dynptr with the function that bpf_dynptr_file_discard() frees it
with. So bpf_sk_release_proto is no longer static, and that function
becomes bpf_dynptr_file_free().
Nothing generates descriptors yet, so nothing is released and the
verifier's rules do not change.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf.h | 91 +++++++++++++
include/linux/bpf_verifier.h | 4 +
kernel/bpf/Makefile | 2 +-
kernel/bpf/core.c | 1 +
kernel/bpf/fixups.c | 3 +
kernel/bpf/helpers.c | 17 ++-
kernel/bpf/unwind.c | 243 +++++++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 5 +
net/core/filter.c | 2 +-
9 files changed, 364 insertions(+), 4 deletions(-)
create mode 100644 kernel/bpf/unwind.c
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 17b13b16899d..e7cce9080531 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1829,6 +1829,72 @@ struct bpf_jit_poke_descriptor {
u32 insn_idx;
};
+/*
+ * Frame descriptors: the hidden stack slots in which the frames of a function
+ * keep the handles of what they hold, and how the unwinder releases them, see
+ * kernel/bpf/frame_desc.c.
+ */
+/* Kinds of resources the unwinder can release. */
+enum bpf_resource_kind {
+ BPF_RESOURCE_OBJ, /* bpf_obj_new() object: bpf_obj_drop_impl(p, meta) */
+ BPF_RESOURCE_PERCPU_OBJ, /* bpf_percpu_obj_new() object: bpf_percpu_obj_drop_impl() */
+ BPF_RESOURCE_KPTR, /* referenced kernel object: the BTF registry's dtor(p) */
+ BPF_RESOURCE_SOCK, /* socket acquired by a helper: bpf_sk_release() */
+ BPF_RESOURCE_RINGBUF, /* ringbuf record, also of a dynptr: bpf_ringbuf_discard() */
+ BPF_RESOURCE_DYNPTR_FILE, /* reader of a file dynptr: bpf_dynptr_file_free() */
+ BPF_RESOURCE_ITER, /* open-coded iterator on the stack: its destroy kfunc */
+ BPF_RESOURCE_SPIN_LOCK, /* bpf_spin_lock() held: bpf_spin_unlock() */
+ BPF_RESOURCE_RES_LOCK, /* bpf_res_spin_lock() held: bpf_res_spin_unlock() */
+ BPF_RESOURCE_IRQ_FLAG, /* saved IRQ flags on the stack: bpf_local_irq_restore() */
+ BPF_RESOURCE_RCU, /* bpf_rcu_read_lock() sections: unlock each */
+ BPF_RESOURCE_PREEMPT, /* bpf_preempt_disable() sections: enable each */
+};
+
+/* A resource class of a program; descriptor entries refer to it by index. */
+struct bpf_resource {
+ u8 kind; /* enum bpf_resource_kind */
+ bool sleeps; /* the destructor may sleep */
+ bool rcu; /* the destructor needs RCU protection */
+ u32 btf_id;
+ struct btf *btf;
+ union {
+ void (*dtor)(void *);
+ struct btf_struct_meta *meta;
+ };
+};
+
+/*
+ * The slot holds a count of sections instead of a handle, signed: the
+ * sections the frame entered less those of its callers it left.
+ */
+#define BPF_FRAME_ENT_COUNTER BIT(0)
+
+/*
+ * One hidden slot of a frame: it holds the handle of a resource of the class
+ * the frame holds, or 0. The handle is what the release takes: the pointer
+ * to an object, to a ringbuf record or to a lock, or the address of an
+ * iterator or of saved IRQ flags on the stack of some frame.
+ */
+struct bpf_frame_desc_entry {
+ u16 res; /* index into the program's resource classes */
+ s16 off; /* the slot, relative to the frame pointer */
+ u16 flags;
+};
+
+struct bpf_frame_desc {
+ u32 size; /* total size in bytes, including this header */
+ u32 nent; /* number of entries that follow */
+ struct bpf_frame_desc_entry ent[];
+};
+
+/* Owned by the main program and shared by its subprograms. */
+struct bpf_frame_desc_pool {
+ u32 cnt;
+ u32 resource_cnt;
+ struct bpf_resource *resources;
+ struct bpf_frame_desc *desc[];
+};
+
/* reg_type info for ctx arguments */
struct bpf_ctx_arg_aux {
u32 offset;
@@ -2053,6 +2119,14 @@ struct bpf_prog_aux {
struct module *mod;
u32 num_exentries;
struct exception_table_entry *extable;
+ /*
+ * The frame descriptor of this JITed function in the pool of the main
+ * program, plus one, or 0: the hidden slots its frames keep the handles
+ * of their resources in, see frame_desc.c. The main and sub progs share
+ * the pool, which is freed with the main prog only.
+ */
+ u32 frame_desc;
+ struct bpf_frame_desc_pool *frame_desc_pool;
union {
struct work_struct work;
struct rcu_head rcu;
@@ -2999,6 +3073,12 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
void bpf_obj_cancel_fields(struct bpf_map *map, void *obj);
void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
+void bpf_obj_drop_impl(void *p__alloc, void *meta__ign);
+void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign);
+void bpf_rcu_read_unlock(void);
+void bpf_preempt_enable(void);
+void bpf_local_irq_restore(unsigned long *flags__irq_flag);
+void bpf_res_spin_unlock(struct bpf_res_spin_lock *lock);
struct bpf_map *bpf_map_get(u32 ufd);
struct bpf_map *bpf_map_get_with_uref(u32 ufd);
@@ -4218,6 +4298,7 @@ extern const struct bpf_func_proto bpf_ringbuf_output_proto;
extern const struct bpf_func_proto bpf_ringbuf_reserve_proto;
extern const struct bpf_func_proto bpf_ringbuf_submit_proto;
extern const struct bpf_func_proto bpf_ringbuf_discard_proto;
+extern const struct bpf_func_proto bpf_sk_release_proto;
extern const struct bpf_func_proto bpf_ringbuf_query_proto;
extern const struct bpf_func_proto bpf_ringbuf_reserve_dynptr_proto;
extern const struct bpf_func_proto bpf_ringbuf_submit_dynptr_proto;
@@ -4508,11 +4589,21 @@ void bpf_insn_array_remap(struct bpf_map *map, const u32 *new_off, const u32 *ne
#ifdef CONFIG_BPF_SYSCALL
void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image);
+void bpf_unwind_release(const struct bpf_resource *res, void *handle);
+void bpf_dynptr_file_free(void *reader);
+void bpf_unwind_release_frames(void);
+void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool);
#else
static inline void
bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
{
}
+static inline void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool)
+{
+}
+static inline void bpf_unwind_release_frames(void)
+{
+}
#endif
static inline bool bpf_map_is_percpu_map(enum bpf_map_type map_type)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index c18efe4bd54a..98d26a4c4a91 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1035,6 +1035,8 @@ struct bpf_subprog_info {
*/
u32 exit_idx;
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
+ /* the frame descriptor of this subprogram in the pool of the program, plus one; see frame_desc.c */
+ u32 frame_desc;
u16 stack_depth; /* max. stack depth used by this function */
/* bytes below the verified stack depth reserved for hidden slots, see bpf_reserve_stack_slots() */
u16 stack_extra;
@@ -1293,6 +1295,8 @@ struct bpf_verifier_env {
/* What a pointer that was read from a table of functions may point to. */
struct bpf_func_set **func_sets;
u32 func_set_cnt;
+ /* frame descriptors of the subprograms, handed to prog->aux once verification succeeds */
+ struct bpf_frame_desc_pool *frame_desc_pool;
bool has_callx;
/*
* Call graph edges created by callx instructions. A bitmap of
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index d26c62326e58..2b6b4af882d1 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o unwind.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 59478a08b247..80321ab16b32 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3459,6 +3459,7 @@ static void bpf_prog_free_images(struct work_struct *work)
int i;
aux = container_of(to_rcu_work(work), struct bpf_prog_aux, rwork);
+ bpf_frame_desc_pool_free(aux->frame_desc_pool);
for (i = 0; i < aux->real_func_cnt; i++) {
/* We can just unlink the subprog poke descriptor table as
* it was originally linked to the main program and is also
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 04d89113e596..4e46f8931775 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -2047,6 +2047,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->nr_linfo = prog->aux->nr_linfo;
func[i]->aux->jited_linfo = prog->aux->jited_linfo;
func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx;
+ /* The pool is freed with the main prog. */
+ func[i]->aux->frame_desc = env->subprog_info[i].frame_desc;
+ func[i]->aux->frame_desc_pool = env->frame_desc_pool;
func[i]->aux->arena = prog->aux->arena;
func[i]->aux->arena_stack_size = prog->aux->arena_stack_size;
func[i]->aux->arena_stack_entry = env->subprog_info[i].arena_stack_entry;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index cc6d3a6a4de0..793eb457fb62 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3492,6 +3492,8 @@ notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
*/
if (regs)
memcpy(ctx.saved, ®s->r[BPF_REG_6], sizeof(ctx.saved));
+ /* What the frames about to be discarded hold is released first. */
+ bpf_unwind_release_frames();
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
@@ -5136,6 +5138,18 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynp
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
+/*
+ * Free the reader of a file dynptr, which the dynptr and its clones point
+ * at: what the unwinder does for a dynptr the program did not discard.
+ */
+void bpf_dynptr_file_free(void *reader)
+{
+ struct bpf_dynptr_file_impl *df = reader;
+
+ freader_cleanup(&df->freader);
+ kfree_nolock(df);
+}
+
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
@@ -5144,8 +5158,7 @@ __bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
if (!df)
return 0;
- freader_cleanup(&df->freader);
- kfree_nolock(df);
+ bpf_dynptr_file_free(df);
bpf_dynptr_set_null(ptr);
return 0;
}
diff --git a/kernel/bpf/unwind.c b/kernel/bpf/unwind.c
new file mode 100644
index 000000000000..3b488a4bd759
--- /dev/null
+++ b/kernel/bpf/unwind.c
@@ -0,0 +1,243 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Releasing what the frames of a BPF program hold when bpf_throw() discards
+ * them. Before the exception callback takes over, the frames from the throw
+ * up to the main program are walked once per phase of the release order, and
+ * each frame releases what its frame descriptor says it holds of that phase,
+ * see frame_desc.c.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/filter.h>
+#include <linux/irqflags.h>
+
+void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool)
+{
+ u32 i;
+
+ if (!pool)
+ return;
+ for (i = 0; i < pool->cnt; i++)
+ kvfree(pool->desc[i]);
+ kvfree(pool->resources);
+ kvfree(pool);
+}
+
+static bool resource_is_lock(u8 kind)
+{
+ return kind == BPF_RESOURCE_SPIN_LOCK || kind == BPF_RESOURCE_RES_LOCK;
+}
+
+static bool resource_is_section(u8 kind)
+{
+ return kind == BPF_RESOURCE_RCU || kind == BPF_RESOURCE_PREEMPT;
+}
+
+/*
+ * Release one resource by its handle, at run time: the pointer to an object,
+ * the address of a stack object or of a lock, or nothing for a section. Not
+ * inlined so that every release of an unwind can be observed by attaching to
+ * this function.
+ */
+noinline void bpf_unwind_release(const struct bpf_resource *res, void *handle)
+{
+ /* A destructor that needs RCU protection gets it here, whatever the frames held. */
+ if (res->rcu)
+ rcu_read_lock();
+ switch (res->kind) {
+ case BPF_RESOURCE_OBJ:
+ bpf_obj_drop_impl(handle, res->meta);
+ break;
+ case BPF_RESOURCE_PERCPU_OBJ:
+ bpf_percpu_obj_drop_impl(handle, res->meta);
+ break;
+ case BPF_RESOURCE_KPTR:
+ case BPF_RESOURCE_ITER:
+ res->dtor(handle);
+ break;
+#ifdef CONFIG_NET
+ case BPF_RESOURCE_SOCK:
+ bpf_sk_release_proto.func((u64)(long)handle, 0, 0, 0, 0);
+ break;
+#endif
+ case BPF_RESOURCE_RINGBUF:
+ bpf_ringbuf_discard_proto.func((u64)(long)handle, 0, 0, 0, 0);
+ break;
+ case BPF_RESOURCE_DYNPTR_FILE:
+ bpf_dynptr_file_free(handle);
+ break;
+ case BPF_RESOURCE_SPIN_LOCK:
+ bpf_spin_unlock_proto.func((u64)(long)handle, 0, 0, 0, 0);
+ break;
+ case BPF_RESOURCE_RES_LOCK:
+ bpf_res_spin_unlock(handle);
+ break;
+ case BPF_RESOURCE_IRQ_FLAG:
+ bpf_local_irq_restore(handle);
+ break;
+ case BPF_RESOURCE_RCU:
+ bpf_rcu_read_unlock();
+ break;
+ case BPF_RESOURCE_PREEMPT:
+ bpf_preempt_enable();
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ }
+ if (res->rcu)
+ rcu_read_unlock();
+}
+
+/* The sections of a frame are counted, per kind. */
+enum {
+ BPF_UNWIND_SECTION_RCU,
+ BPF_UNWIND_SECTION_PREEMPT,
+ BPF_UNWIND_NR_SECTIONS,
+};
+
+static int section_index(u8 kind)
+{
+ return kind == BPF_RESOURCE_RCU ? BPF_UNWIND_SECTION_RCU : BPF_UNWIND_SECTION_PREEMPT;
+}
+
+/*
+ * The release order. Objects never run under a lock or with IRQs disabled by
+ * any frame, and a sleeping destructor runs after every section is left; an
+ * RCU-protected object is destroyed before the sections end.
+ */
+enum {
+ BPF_UNWIND_PHASE_LOCKS, /* locks and IRQ flags */
+ BPF_UNWIND_PHASE_RCU_OBJS, /* objects whose destructor needs RCU protection */
+ BPF_UNWIND_PHASE_SECTIONS, /* RCU and preemption sections */
+ BPF_UNWIND_PHASE_OBJS, /* everything else */
+ BPF_UNWIND_NR_PHASES,
+};
+
+static int release_phase(const struct bpf_resource *res)
+{
+ if (resource_is_lock(res->kind) || res->kind == BPF_RESOURCE_IRQ_FLAG)
+ return BPF_UNWIND_PHASE_LOCKS;
+ if (resource_is_section(res->kind))
+ return BPF_UNWIND_PHASE_SECTIONS;
+ return res->rcu ? BPF_UNWIND_PHASE_RCU_OBJS : BPF_UNWIND_PHASE_OBJS;
+}
+
+struct bpf_unwind_ctx {
+ int phase; /* BPF_UNWIND_PHASE_*: what this walk releases */
+ int cnt; /* BPF frames passed */
+ /* The sections the frames hold, per kind, summed by the first walk. */
+ const struct bpf_resource *section[BPF_UNWIND_NR_SECTIONS];
+ s64 nr_sections[BPF_UNWIND_NR_SECTIONS];
+};
+
+/*
+ * The R10 of a frame of the function @fn whose native frame pointer is @bp: a
+ * fixed distance from it, or, with a private stack, this CPU's private frame.
+ */
+static u64 bpf_unwind_fp(const struct bpf_prog_aux *fn, u64 bp)
+{
+ if (fn->priv_stack_ptr)
+ return (u64)(long)this_cpu_ptr(fn->priv_stack_ptr) + fn->priv_frame_off;
+ return bp + fn->frame_fp_off;
+}
+
+/*
+ * Release what the slots of the frame of @prog hold of the phase of this
+ * walk; @bp is its native frame pointer. The first walk sums the frame's
+ * counts of sections for their phase, and releases the locks and IRQ flags
+ * in two passes: the flags a frame saved with interrupts enabled, of which
+ * it can have one at most, since every later save found them disabled, are
+ * restored last, after the locks and the other flags in whatever order the
+ * frame took them.
+ */
+static void bpf_unwind_release_frame(struct bpf_unwind_ctx *ctx, const struct bpf_prog *prog, u64 bp)
+{
+ const struct bpf_prog_aux *fn = bpf_prog_frame_aux(prog);
+ const struct bpf_frame_desc_pool *pool = fn->frame_desc_pool;
+ int passes = ctx->phase == BPF_UNWIND_PHASE_LOCKS ? 2 : 1, pass, s;
+ const struct bpf_frame_desc *d;
+ bool late;
+ u64 fp;
+ u32 i;
+
+ if (!fn->frame_desc)
+ return;
+ d = pool->desc[fn->frame_desc - 1];
+ fp = bpf_unwind_fp(fn, bp);
+ for (pass = 0; pass < passes; pass++) {
+ for (i = 0; i < d->nent; i++) {
+ const struct bpf_frame_desc_entry *ent = &d->ent[i];
+ const struct bpf_resource *res = &pool->resources[ent->res];
+ u64 *slot = (u64 *)(long)(fp + ent->off);
+ void *handle = (void *)(long)*slot;
+
+ if (ent->flags & BPF_FRAME_ENT_COUNTER) {
+ if (ctx->phase != BPF_UNWIND_PHASE_LOCKS || pass)
+ continue;
+ s = section_index(res->kind);
+ ctx->section[s] = res;
+ ctx->nr_sections[s] += (s64)*slot;
+ continue;
+ }
+ if (!handle || release_phase(res) != ctx->phase)
+ continue;
+ late = res->kind == BPF_RESOURCE_IRQ_FLAG &&
+ !arch_irqs_disabled_flags(*(unsigned long *)handle);
+ if (passes == 2 && late != (pass == 1))
+ continue;
+ bpf_unwind_release(res, handle);
+ }
+ }
+}
+
+static bool bpf_unwind_walker(void *cookie, u64 ip, u64 sp, u64 bp)
+{
+ struct bpf_unwind_ctx *ctx = cookie;
+ struct bpf_prog *prog;
+
+ /*
+ * The RCU read lock is held to safely traverse the latch tree, but we
+ * don't need its protection when accessing the prog, since it has an
+ * active stack frame on the current stack trace, and won't disappear.
+ */
+ rcu_read_lock();
+ prog = bpf_prog_ksym_find(ip);
+ rcu_read_unlock();
+ /* the frames of bpf_throw() and of the unwinder come first */
+ if (!prog)
+ return !ctx->cnt;
+ ctx->cnt++;
+ bpf_unwind_release_frame(ctx, prog, bp);
+ /* the main program is where the exception callback takes over */
+ return bpf_is_subprog(prog);
+}
+
+/*
+ * Release what the frames between bpf_throw() and the main program hold, the
+ * main program's included, in four phases: locks and IRQ flags,
+ * RCU-protected objects, sections, then the other objects. Each phase but
+ * the sections' is a walk over the frames, innermost first, that releases
+ * each frame's entries of the phase in the order of its descriptor; the
+ * frames are released as the walk passes them, so their number is not
+ * bounded. The sections need no walk of their own: the first one summed
+ * their counts, and sections of one kind are all the same.
+ */
+void bpf_unwind_release_frames(void)
+{
+ struct bpf_unwind_ctx ctx = {};
+ int s;
+
+ for (ctx.phase = 0; ctx.phase < BPF_UNWIND_NR_PHASES; ctx.phase++) {
+ if (ctx.phase != BPF_UNWIND_PHASE_SECTIONS) {
+ ctx.cnt = 0;
+ arch_bpf_stack_walk(bpf_unwind_walker, &ctx);
+ continue;
+ }
+ for (s = 0; s < BPF_UNWIND_NR_SECTIONS; s++) {
+ WARN_ONCE(ctx.nr_sections[s] < 0, "bpf: %lld sections to leave\n",
+ ctx.nr_sections[s]);
+ while (ctx.nr_sections[s]-- > 0)
+ bpf_unwind_release(ctx.section[s], NULL);
+ }
+ }
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6c71ceb1dd4c..cf3cfc734ed4 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -25695,6 +25695,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (env->prog->type == BPF_PROG_TYPE_EXT)
env->prog->expected_attach_type = 0;
+ /* The program owns the frame descriptors from here on. */
+ env->prog->aux->frame_desc_pool = env->frame_desc_pool;
+ env->frame_desc_pool = NULL;
+
env->prog = __bpf_prog_select_runtime(env, env->prog, &ret);
err_release_maps:
@@ -25731,6 +25735,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
release_btfs(env);
err_free_env:
bpf_free_subprog_jts(env);
+ bpf_frame_desc_pool_free(env->frame_desc_pool);
if (env->insn_aux_data)
bpf_clear_insn_aux_data(env, 0, env->insn_aux_data_len);
vfree(env->insn_aux_data);
diff --git a/net/core/filter.c b/net/core/filter.c
index 9245009d9f1e..9f59b1c002d6 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -7577,7 +7577,7 @@ BPF_CALL_1(bpf_sk_release, struct sock *, sk)
return 0;
}
-static const struct bpf_func_proto bpf_sk_release_proto = {
+const struct bpf_func_proto bpf_sk_release_proto = {
.func = bpf_sk_release,
.gpl_only = false,
.ret_type = RET_INTEGER,
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw()
2026-11-11 12:57 ` [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw() Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 15:30 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4036 bytes --]
> bpf: Release resources in hidden slots on bpf_throw()
TL;DR:
- Possible link failure with CONFIG_NET=y, INET=n: bpf_sk_release_proto is
guarded by INET, not NET
- bpf_throw() unwind may release spin locks and IRQ flags out of order,
leaving IRQs disabled or the held-lock table wrong
> diff --git a/kernel/bpf/unwind.c b/kernel/bpf/unwind.c
> new file mode 100644
> index 000000000000..3b488a4bd759
> --- /dev/null
> +++ b/kernel/bpf/unwind.c
[ ... ]
> +noinline void bpf_unwind_release(const struct bpf_resource *res, void *handle)
> +{
[ ... ]
> +#ifdef CONFIG_NET
> + case BPF_RESOURCE_SOCK:
> + bpf_sk_release_proto.func((u64)(long)handle, 0, 0, 0, 0);
> + break;
> +#endif
Can this fail to link with CONFIG_NET=y, CONFIG_INET=n and
CONFIG_BPF_SYSCALL=y?
unwind.o is built whenever CONFIG_BPF_SYSCALL is set, and this case is
compiled under CONFIG_NET. But bpf_sk_release_proto is defined in
net/core/filter.c inside the CONFIG_INET block:
net/core/filter.c:
#ifdef CONFIG_INET
...
const struct bpf_func_proto bpf_sk_release_proto = {
...
#endif /* CONFIG_INET */
INET is a separate bool under NET and BPF_SYSCALL does not select it, so
vmlinux would end up with an undefined reference to bpf_sk_release_proto.
Should the guard here be CONFIG_INET, matching the definition (as
kernel/bpf/cgroup.c does for bpf_tcp_sock_proto)?
[ ... ]
> +static void bpf_unwind_release_frame(struct bpf_unwind_ctx *ctx, const struct bpf_prog *prog, u64 bp)
> +{
[ ... ]
> + if (!handle || release_phase(res) != ctx->phase)
> + continue;
> + late = res->kind == BPF_RESOURCE_IRQ_FLAG &&
> + !arch_irqs_disabled_flags(*(unsigned long *)handle);
> + if (passes == 2 && late != (pass == 1))
> + continue;
> + bpf_unwind_release(res, handle);
> + }
> + }
> +}
Only BPF_RESOURCE_IRQ_FLAG entries can be "late" here, but doesn't
bpf_spin_lock() also save and restore IRQ state? The commit message says
"Every later save found interrupts disabled ... the locks and the other
flags can be released in any order before it."
__bpf_spin_lock_irqsave() does local_irq_save() into the per-CPU
irqsave_flags, and bpf_spin_unlock() ends up in
__bpf_spin_unlock_irqrestore(), which does local_irq_restore() of that
value.
Consider a frame that does:
bpf_spin_lock(&v->lock); /* IRQs on -> off */
if (bpf_res_spin_lock_irqsave(&v->rl, &flags)) /* flags saved: off */
...
bpf_throw(0);
The IRQ_FLAG for flags is not "late", so pass 0 releases the SPIN_LOCK,
RES_LOCK and that flag in descriptor order. If the descriptor lists them in
acquisition order, bpf_spin_unlock() turns interrupts back on and then
bpf_local_irq_restore(&flags) turns them off again. Pass 1 has no late flag
to restore, so the exception callback runs and the program returns to its
caller with IRQs disabled, although it was entered with them enabled.
Can the same happen when the flags are in this frame's stack but a callee
saved them under this frame's spin lock?
Separately, releasing entries of a frame in acquisition order breaks the
LIFO assumption of bpf_res_spin_unlock():
rqh->locks[cnt - 1]
is cleared whichever lock is passed in. Does the held-lock table then name
the wrong lock while the unwind runs, so that AA detection is missed or
misdiagnosed?
Would it work to treat a SPIN_LOCK whose saved irqsave_flags has IRQs
enabled as "late" too, or to release the lock and IRQ entries of a frame in
reverse acquisition order?
This commit generates no descriptors, so the path is not reached yet. It
gets enabled later in the series by cd4657cd0e50 ("bpf: Track held
resources in hidden stack slots") and faacef7a1935 ("bpf: Describe locks at
throw sites"), and no later commit in the series changes this ordering.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw()
2026-11-11 12:57 ` [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw() Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 15:30 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 15:30 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] bpf: broken IRQ state restoration for spinlocks in
bpf_unwind_release_frame()
--
commit bc130ef20556bd747c8015574438bb24ed4f95b8
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Release resources in hidden slots on bpf_throw()
This commit adds runtime support in bpf_throw() to walk discarded BPF
stack frames and release held locks, IRQ flags, sections, and referenced
objects recorded in frame descriptors.
[ ... ]
> diff --git a/kernel/bpf/unwind.c b/kernel/bpf/unwind.c
> new file mode 100644
> index 0000000000000..3b488a4bd7592
> --- /dev/null
> +++ b/kernel/bpf/unwind.c
[ ... ]
> +static void bpf_unwind_release_frame(struct bpf_unwind_ctx *ctx, const struct bpf_prog *prog, u64 bp)
> +{
> + const struct bpf_prog_aux *fn = bpf_prog_frame_aux(prog);
> + const struct bpf_frame_desc_pool *pool = fn->frame_desc_pool;
> + int passes = ctx->phase == BPF_UNWIND_PHASE_LOCKS ? 2 : 1, pass, s;
> + const struct bpf_frame_desc *d;
> + bool late;
> + u64 fp;
> + u32 i;
> +
> + if (!fn->frame_desc)
> + return;
> + d = pool->desc[fn->frame_desc - 1];
> + fp = bpf_unwind_fp(fn, bp);
> + for (pass = 0; pass < passes; pass++) {
> + for (i = 0; i < d->nent; i++) {
> + const struct bpf_frame_desc_entry *ent = &d->ent[i];
> + const struct bpf_resource *res = &pool->resources[ent->res];
> + u64 *slot = (u64 *)(long)(fp + ent->off);
> + void *handle = (void *)(long)*slot;
> +
> + if (ent->flags & BPF_FRAME_ENT_COUNTER) {
> + if (ctx->phase != BPF_UNWIND_PHASE_LOCKS || pass)
> + continue;
> + s = section_index(res->kind);
> + ctx->section[s] = res;
> + ctx->nr_sections[s] += (s64)*slot;
> + continue;
> + }
> + if (!handle || release_phase(res) != ctx->phase)
> + continue;
> + late = res->kind == BPF_RESOURCE_IRQ_FLAG &&
> + !arch_irqs_disabled_flags(*(unsigned long *)handle);
> + if (passes == 2 && late != (pass == 1))
> + continue;
> + bpf_unwind_release(res, handle);
> + }
> + }
> +}
[Severity: High]
Does releasing BPF_RESOURCE_SPIN_LOCK in pass 0 corrupt the CPU interrupt
state when a frame holds both a bpf_spin_lock and a nested
bpf_res_spin_lock_irqsave()?
bpf_spin_lock() saves interrupt flags in per-CPU irqsave_flags and
disables local interrupts, and bpf_spin_unlock() restores irqsave_flags:
kernel/bpf/helpers.c:__bpf_spin_unlock_irqrestore() {
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
bpf_res_spin_lock() and bpf_res_spin_lock_irqsave() are marked
KF_SPINLOCK_SAFE and can be acquired while a bpf_spin_lock is held. When
a frame acquires bpf_spin_lock() with interrupts enabled and then calls
bpf_res_spin_lock_irqsave(), irqsave_flags holds the enabled state while
the stack slot for BPF_RESOURCE_IRQ_FLAG records interrupts as disabled.
Because only BPF_RESOURCE_IRQ_FLAG is checked for late, pass 0 runs
bpf_spin_unlock() first, re-enabling interrupts while the resilient
spinlock is still held, and then runs bpf_local_irq_restore() on the
BPF_RESOURCE_IRQ_FLAG slot, leaving local interrupts permanently disabled
through sleeping destructors in BPF_UNWIND_PHASE_OBJS and after the BPF
program returns. In addition, releasing BPF_RESOURCE_RES_LOCK in pass 0
calls preempt_enable() in bpf_res_spin_unlock() before pass 1 restores
the interrupt state.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=12
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (11 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 12/29] bpf: Release resources in hidden slots on bpf_throw() Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:12 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 14/29] bpf: Describe held resources at throw sites Kumar Kartikeya Dwivedi
` (16 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently nothing generates frame descriptors, so bpf_throw() releases
nothing and the verifier rejects a throw that is reached with a
reference, a lock, or a section held.
Add the verifier half. In a program that may throw, every call that
acquires a resource gets its own hidden stack slot in the frame that
makes the call. The verifier inserts instructions that keep the handle of
the resource in the slot while it is held. It makes no assumption about
where the program itself keeps the handle.
The main pass records, per instruction:
- each call that acquires a resource, with the resource class and the
location of the handle: R0 or an argument register;
- each call that releases a resource, with the register that holds the
handle and the acquisitions that it may release;
- each call that enters or leaves an RCU read or preemption disabled
section.
After the pass, the verifier reserves the slots and patches the program:
- A handle returned in R0 is stored after the call. A pointer into the
object, as bpf_list_pop_front() returns, is first adjusted to the
start of the object.
- A handle passed as an argument, the address of a lock, an iterator or
saved IRQ flags, is stored before the call, so that a throw from
inside an iterator loop or a critical section finds it. If the call
can fail to acquire, the slot is cleared after a failed call.
- A releasing call compares its handle with the slot of each
acquisition it may release, and clears the matching slot before the
call.
- The handle of a dynptr, its ringbuf record or file reader, is loaded
from the dynptr on the stack after the call that initializes it.
- A function that enters or leaves a section counts it in one slot per
kind.
- A subprog that returns a reference in R0 returns holding it. The call
is then the caller's acquisition, like a call to a kfunc that returns
a reference.
- The referenced arguments of a struct_ops program are loaded from the
context when the main program is entered.
- The slots of a frame are zeroed at its entry.
The frame descriptor of each function lists these slots, and bpf_throw()
walks them.
A slot belongs to one frame and holds one handle. A program that throws
is therefore rejected when:
- a callee releases a resource held by its caller, or leaves a section
that its caller entered;
- a subprog returns holding anything other than one reference in R0, or
in different sections than it was called in;
- a callee creates an iterator or saves IRQ flags on its caller's stack;
- a frame can hold two resources acquired by the same instruction;
- an instruction acquires a resource only on some paths, such as a
lookup that returns a socket from a map of sockets and a plain value
from another map, or resources of different classes on different
paths;
- an object is released while the lock inside it is held;
- a reference has no destructor that the unwinder may run: its type has
no cleanup destructor registered, or the program may not call its
release kfunc.
Later patches lift most of the first five.
The unwinder destroys iterators and restores IRQ flags on the program's
stack, so they must be intact at a throw:
- A program that throws must not overwrite an iterator or saved IRQ
flags on its stack.
- A verifier state can stand for another only if the stack objects of
both are in the same slots. This already holds: a stack object stays
in a state whether or not it is read again, and is always compared.
- An iterator that needs RCU protection must not outlive the RCU section
it was created in.
States are now also compared by the owner frame, the resource class and
the acquiring instruction of each reference, since the instructions
inserted for a release depend on them.
Frames are described only where the JIT reports
bpf_jit_supports_exception_cleanup(), and only in a program where some
subprog may throw. Other architectures and programs keep the existing
rules. A throw from a callback stays rejected.
exceptions_fail has programs that expect a throw with resources held to
be rejected. That still holds where the JIT does not support the cleanup,
so restrict them to s390x, the other architecture with exceptions that
the test loader can select. Two others throw from a callback and are
still rejected here, but by that rule: update their expected message and
restrict them to x86-64 and arm64.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 84 ++
kernel/bpf/Makefile | 2 +-
kernel/bpf/fixups.c | 2 +-
kernel/bpf/frame_desc.c | 1186 +++++++++++++++++
kernel/bpf/join.c | 18 +-
kernel/bpf/states.c | 23 +-
kernel/bpf/verifier.c | 296 +++-
.../selftests/bpf/progs/exceptions_fail.c | 16 +-
8 files changed, 1584 insertions(+), 43 deletions(-)
create mode 100644 kernel/bpf/frame_desc.c
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 98d26a4c4a91..5cbac24102f4 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -333,6 +333,14 @@ struct bpf_reference_state {
* function verified on its own hold, which are not in the state.
*/
s8 frameno;
+ /*
+ * The resource class the unwinder releases it as, plus one, see
+ * frame_desc.c; 0 for a reference the frames keep no handle of, which
+ * a throw may not be reached with.
+ */
+ u16 res;
+ /* The acquisition whose slot holds its handle, plus one, see struct bpf_res_acq. */
+ u16 acq;
union {
/* For REF_TYPE_PTR */
int parent_id;
@@ -908,6 +916,13 @@ struct bpf_insn_aux_data {
u64 jump_target:1;
unsigned int orig_idx; /* original instruction index, initialized once */
+ /*
+ * Some visit of the call acquired nothing a hidden slot can hold where
+ * another may have acquired a resource, see frame_desc.c.
+ */
+ bool res_none;
+ /* Some visit of the call left an R0 that is not a resource. */
+ bool res_none_r0;
/* registers alive before this instruction. */
u16 live_regs_before;
/*
@@ -1037,6 +1052,8 @@ struct bpf_subprog_info {
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
/* the frame descriptor of this subprogram in the pool of the program, plus one; see frame_desc.c */
u32 frame_desc;
+ /* hidden slots: the counts of the sections the frame entered, by kind; 0 when it has none */
+ s16 section_off[2];
u16 stack_depth; /* max. stack depth used by this function */
/* bytes below the verified stack depth reserved for hidden slots, see bpf_reserve_stack_slots() */
u16 stack_extra;
@@ -1224,6 +1241,42 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
/* single container for all structs
* one verifier_env per bpf_check() call
*/
+#define BPF_RES_F_ARG BIT(0) /* the handle is the pointer the call takes in the register */
+#define BPF_RES_F_MAY_FAIL BIT(1) /* acquired only when the call returns 0 */
+#define BPF_RES_F_DEREF BIT(2) /* the handle is the first word of a dynptr on the stack */
+#define BPF_RES_F_CTX BIT(3) /* a __ref argument the kernel passes the main program */
+
+/*
+ * A call that acquires a resource, see frame_desc.c: the handle of what it
+ * acquired is kept in a hidden slot of the frame that runs the call, one slot
+ * per acquiring call.
+ */
+struct bpf_res_acq {
+ u32 insn; /* the insn as the main pass named it; the argument with BPF_RES_F_CTX */
+ u16 res; /* the class of what it acquires */
+ u8 reg; /* R0 for what the call returns, else the argument that is the handle */
+ u8 flags; /* BPF_RES_F_* */
+ s16 adj; /* R0 points that many bytes into the object: the node of a list */
+ s16 obj_off; /* BPF_RES_F_DEREF: the dynptr, relative to R10; BPF_RES_F_CTX: in the context */
+ s16 off; /* the slot, relative to R10 */
+};
+
+/* A call that releases what an acquisition acquired: the slot of the acquisition is cleared. */
+struct bpf_res_rel {
+ u32 insn;
+ u16 acq; /* index of the acquisition */
+ u8 reg; /* the argument the call releases */
+ u8 flags; /* BPF_RES_F_DEREF */
+ s16 adj; /* the argument points that many bytes into the object */
+};
+
+/* A call that enters or leaves a section, for the count of its frame, see frame_desc.c. */
+struct bpf_section_insn {
+ u32 insn;
+ u8 kind; /* 0 RCU, 1 preemption */
+ s8 delta;
+};
+
/* Initialization of a hidden stack slot, emitted in the subprogram's prologue. */
struct bpf_stack_init {
u32 subprog;
@@ -1258,6 +1311,19 @@ struct bpf_verifier_env {
u32 hidden_subprog_cnt; /* number of hidden subprogs */
struct bpf_stack_init *stack_inits; /* hidden slot initializations, see bpf_stack_slot_init() */
u32 stack_init_cnt;
+ struct bpf_resource *resources; /* resource classes of frame descriptors */
+ u32 resource_cnt;
+ struct bpf_res_acq *res_acqs; /* see frame_desc.c */
+ u32 res_acq_cnt;
+ struct bpf_res_rel *res_rels;
+ u32 res_rel_cnt;
+ struct bpf_section_insn *section_insns;
+ u32 section_insn_cnt;
+ int release_regno; /* the argument the current call releases, or -1 */
+ u32 frame_desc_slots; /* hidden slots that hold handles and counts */
+ u32 frame_desc_insns; /* instructions keeping them up to date */
+ u32 frame_desc_bytes; /* descriptors and resource classes */
+ bool describe_frames; /* frame descriptors are generated: the program may throw */
int exception_callback_subprog;
bool explore_alu_limits;
bool allow_ptr_leaks;
@@ -1929,6 +1995,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
bool bpf_is_throw_kfunc(struct bpf_insn *insn);
+bool bpf_nmi_unsafe_prog(struct bpf_verifier_env *env);
int bpf_compute_const_regs(struct bpf_verifier_env *env);
int bpf_prune_dead_branches(struct bpf_verifier_env *env);
int bpf_check_cfg(struct bpf_verifier_env *env);
@@ -2202,7 +2269,24 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
/* Functions in fixups.c, called from bpf_check() */
void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len);
int bpf_reserve_stack_slots(struct bpf_verifier_env *env, int subprog, int nr, s16 *off);
+int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta);
int bpf_stack_slot_init(struct bpf_verifier_env *env, int subprog, s16 off, s32 imm);
+
+/* Functions in frame_desc.c */
+int bpf_frame_desc_record_acquire(struct bpf_verifier_env *env, int idx, int regno, bool after,
+ bool may_fail);
+void bpf_frame_desc_record_none(struct bpf_verifier_env *env, const struct bpf_reg_state *r0);
+int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx);
+int bpf_frame_desc_record_ctx_arg(struct bpf_verifier_env *env, int idx, int argno);
+int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delta);
+int bpf_frame_desc_record_exit(struct bpf_verifier_env *env);
+int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int idx, int callsite, const struct bpf_reg_state *r0);
+int bpf_frame_desc_resolve(struct bpf_verifier_env *env);
+int bpf_frame_desc_emit(struct bpf_verifier_env *env);
+int bpf_frame_desc_build(struct bpf_verifier_env *env);
+void bpf_frame_desc_reset(struct bpf_verifier_env *env);
+void bpf_frame_desc_env_free(struct bpf_verifier_env *env);
int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 2b6b4af882d1..da4d62466645 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o unwind.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o frame_desc.o unwind.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 4e46f8931775..984d6287241a 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -553,7 +553,7 @@ int bpf_patch_list_commit(struct bpf_verifier_env *env)
* insn was moved down by delta insns inside its own patch. Operands relative
* to the pc that point in front of the old position did not move with it.
*/
-static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
+int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
{
u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
s64 off = insn->imm, off_min = S32_MIN;
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
new file mode 100644
index 000000000000..b0db3feeb9b0
--- /dev/null
+++ b/kernel/bpf/frame_desc.c
@@ -0,0 +1,1186 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Frame descriptors: the hidden stack slots in which the frames of a program
+ * that may throw keep the handles of the kernel resources they hold, for
+ * bpf_throw() to release, see unwind.c.
+ *
+ * Nothing is inferred about where a program keeps a reference. The handle of
+ * a resource is written down when the resource is acquired: the pointer a
+ * call returns, the address of the lock, the iterator or the saved IRQ flags
+ * a call takes, the record or the reader of a dynptr. Every call that
+ * acquires has a slot of its own in the frame that runs it. The handle goes
+ * into the slot around the call, and the call that releases the resource
+ * clears the slot when it holds the handle the call takes. The sections a
+ * frame enters, RCU read and preemption disabled, have no handle and are
+ * counted in a slot per kind.
+ *
+ * A slot belongs to one frame and holds one handle, so a program that throws
+ * keeps to what that can express:
+ *
+ * - A frame releases what it acquired itself. A callee cannot reach the
+ * slot of its caller, so it may not release what its caller holds.
+ * - A frame that returns holds nothing but a reference it returns in R0,
+ * which the call acquires for the caller like the call of a kfunc that
+ * returns one, and is in the sections it was called in.
+ * - A frame holds one resource of an acquiring call at a time.
+ *
+ * The main pass records every call that acquires or releases and which
+ * acquisition a release may be of. The slots are reserved and the
+ * instructions that keep them emitted after the pass.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/kallsyms.h>
+#include <linux/sort.h>
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+/* Intern a resource class; descriptors refer to it by index. */
+static int resource_intern(struct bpf_verifier_env *env, const struct bpf_resource *res)
+{
+ struct bpf_resource *tab;
+ u32 i, cnt = env->resource_cnt;
+
+ for (i = 0; i < cnt; i++)
+ if (!memcmp(&env->resources[i], res, sizeof(*res)))
+ return i;
+ if (cnt >= U16_MAX)
+ return -E2BIG;
+ tab = kvrealloc(env->resources, array_size(cnt + 1, sizeof(*tab)), GFP_KERNEL_ACCOUNT);
+ if (!tab)
+ return -ENOMEM;
+ tab[cnt] = *res;
+ env->resources = tab;
+ env->resource_cnt = cnt + 1;
+ return cnt;
+}
+
+/*
+ * Resolve the destructor of a referenced kernel object through the BTF
+ * destructor registry, and check that the program could have called the
+ * matching release kfunc itself: that is the context in which running the
+ * destructor is legal. A class interned for the type already has it resolved.
+ */
+static int resource_registry(struct bpf_verifier_env *env, struct btf *btf, u32 btf_id,
+ struct bpf_resource *res, const char *what, bool report)
+{
+ const char *name = btf_name_by_offset(btf, btf_type_by_id(btf, btf_id)->name_off);
+ struct btf *dtor_btf = btf;
+ const struct btf_id_dtor_kfunc *dtor = btf_find_dtor(&dtor_btf, btf_id);
+ const struct btf_type *t;
+ u32 i, *flags;
+
+ for (i = 0; i < env->resource_cnt; i++) {
+ const struct bpf_resource *r = &env->resources[i];
+
+ if (r->kind == res->kind && r->btf == btf && r->btf_id == btf_id) {
+ res->dtor = r->dtor;
+ res->sleeps = r->sleeps;
+ return 0;
+ }
+ }
+ if (!dtor || !(dtor->flags & BPF_DTOR_CLEANUP)) {
+ if (report)
+ verbose(env, "frame_desc: no cleanup destructor registered for %s %s\n", what,
+ name);
+ return -EOPNOTSUPP;
+ }
+ flags = btf_kfunc_flags(dtor_btf, dtor->release_btf_id, env->prog);
+ if (!flags || !btf_kfunc_is_allowed(dtor_btf, dtor->release_btf_id, env->prog)) {
+ if (report)
+ verbose(env, "frame_desc: release kfunc of %s %s is not callable by this program\n",
+ what, name);
+ return -EOPNOTSUPP;
+ }
+ res->sleeps = *flags & KF_SLEEPABLE;
+ t = btf_type_by_id(dtor_btf, dtor->kfunc_btf_id);
+ res->dtor = t ? (void *)kallsyms_lookup_name(btf_name_by_offset(dtor_btf, t->name_off)) :
+ NULL;
+ if (!res->dtor) {
+ if (report)
+ verbose(env, "frame_desc: cannot resolve the destructor of %s %s\n", what, name);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static bool helper_ok(struct bpf_verifier_env *env, enum bpf_func_id func_id)
+{
+ return env->ops->get_func_proto && env->ops->get_func_proto(func_id, env->prog);
+}
+
+/*
+ * Does @reg point to one of the socket structs of vmlinux BTF? A referenced
+ * one is the cast of an acquired socket by a helper such as
+ * bpf_skc_to_tcp_sock(), which carries the socket's reference under the
+ * struct's type.
+ */
+static bool reg_is_sock_btf(const struct bpf_reg_state *reg)
+{
+#ifdef CONFIG_NET
+ int i;
+
+ if (!btf_is_vmlinux(reg->btf))
+ return false;
+ for (i = 0; i < MAX_BTF_SOCK_TYPE; i++)
+ if (reg->btf_id == btf_sock_ids[i])
+ return true;
+#endif
+ return false;
+}
+
+/*
+ * Classify the reference a register holds, @report why it cannot be. A
+ * reference of a type that no destructor is registered for, or whose release
+ * kfunc the program may not call, has no class: the frames keep no handle of
+ * it, and a throw may not be reached while it is held.
+ */
+static int resource_describe(struct bpf_verifier_env *env, const struct bpf_reg_state *reg,
+ struct bpf_resource *resp, bool report)
+{
+ u32 flags = type_flag(reg->type) & ~PTR_MAYBE_NULL;
+ struct bpf_resource res;
+ int ret;
+
+ memset(&res, 0, sizeof(res));
+ switch (base_type(reg->type)) {
+ case PTR_TO_BTF_ID:
+ if (flags & MEM_ALLOC) {
+ if (flags & ~(MEM_ALLOC | MEM_PERCPU))
+ goto unsupported;
+ res.kind = flags & MEM_PERCPU ? BPF_RESOURCE_PERCPU_OBJ : BPF_RESOURCE_OBJ;
+ res.meta = btf_find_struct_meta(reg->btf, reg->btf_id);
+ /* where the program may not drop the object, the unwinder may not either */
+ if (res.meta && bpf_nmi_unsafe_prog(env) &&
+ btf_record_has_nmi_unsafe_fields(res.meta->record))
+ goto unsupported;
+ /* the types without fields to destroy are freed alike: one class */
+ break;
+ }
+ if (flags & ~PTR_TRUSTED)
+ goto unsupported;
+ res.btf = reg->btf;
+ res.btf_id = reg->btf_id;
+ /* A socket cast to one of its structs is still what bpf_sk_release() takes. */
+ if (reg_is_sock_btf(reg) && helper_ok(env, BPF_FUNC_sk_release)) {
+ res.kind = BPF_RESOURCE_SOCK;
+ break;
+ }
+ res.kind = BPF_RESOURCE_KPTR;
+ ret = resource_registry(env, reg->btf, reg->btf_id, &res, "object", report);
+ if (ret)
+ return ret;
+ break;
+ case PTR_TO_SOCKET:
+ case PTR_TO_SOCK_COMMON:
+ case PTR_TO_TCP_SOCK:
+ if (flags || !helper_ok(env, BPF_FUNC_sk_release))
+ goto unsupported;
+ res.kind = BPF_RESOURCE_SOCK;
+ break;
+ case PTR_TO_MEM:
+ if (flags != MEM_RINGBUF || !helper_ok(env, BPF_FUNC_ringbuf_discard))
+ goto unsupported;
+ res.kind = BPF_RESOURCE_RINGBUF;
+ break;
+ default:
+ goto unsupported;
+ }
+ *resp = res;
+ return 0;
+unsupported:
+ if (report)
+ verbose(env, "frame_desc: cannot release a reference of type %s\n",
+ reg_type_str(env, reg->type));
+ return -EOPNOTSUPP;
+}
+
+/*
+ * The class of the iterator in @reg, the first slot of an iterator. An
+ * RCU-protected iterator is destroyed before the sections of its frame end,
+ * unless its destructor sleeps: that runs after every section, which is
+ * the only place the program could have called it.
+ */
+static int resource_of_iter(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
+{
+ struct bpf_resource res;
+ int ret;
+
+ /* nothing may release it once its RCU protection ended */
+ if (reg->type & PTR_UNTRUSTED)
+ return -EOPNOTSUPP;
+ memset(&res, 0, sizeof(res));
+ res.kind = BPF_RESOURCE_ITER;
+ res.btf = reg->iter.btf;
+ res.btf_id = reg->iter.btf_id;
+ ret = resource_registry(env, reg->iter.btf, reg->iter.btf_id, &res, "iterator", false);
+ if (ret)
+ return ret;
+ res.rcu = (reg->type & MEM_RCU) && !res.sleeps;
+ return resource_intern(env, &res);
+}
+
+/*
+ * Does @reg point into the object of reference @id, @off bytes in: the node
+ * of a list or a tree is popped and pushed that way, and the object it is
+ * in is released.
+ */
+static bool reg_ref_off(const struct bpf_reg_state *reg, int id, s16 *off)
+{
+#ifdef CONFIG_NET
+ /*
+ * bpf_skc_to_mptcp_sock() gives its result the id of the subflow it
+ * was passed, and the result is the socket of the connection.
+ */
+ if (base_type(reg->type) == PTR_TO_BTF_ID && btf_is_vmlinux(reg->btf) &&
+ reg->btf_id == btf_sock_ids[BTF_SOCK_TYPE_MPTCP])
+ return false;
+#endif
+ if (reg->type == SCALAR_VALUE || reg->type == NOT_INIT || reg->id != id ||
+ !tnum_is_const(reg->var_off) || reg->var_off.value != (u64)(s16)reg->var_off.value)
+ return false;
+ *off = reg->var_off.value;
+ return true;
+}
+
+/*
+ * Is @slot the stack object of reference @id: the first slot of an iterator,
+ * of a ringbuf dynptr, which tracks the record's reference as its parent,
+ * or saved IRQ flags?
+ */
+static bool stack_slot_is_object(const struct bpf_stack_state *slot, int id)
+{
+ const struct bpf_reg_state *reg = &slot->spilled_ptr;
+
+ switch (slot->slot_type[BPF_REG_SIZE - 1]) {
+ case STACK_DYNPTR:
+ return reg->dynptr.first_slot && reg->parent_id == id &&
+ (reg->dynptr.type == BPF_DYNPTR_TYPE_RINGBUF ||
+ reg->dynptr.type == BPF_DYNPTR_TYPE_FILE);
+ case STACK_ITER:
+ case STACK_IRQ_FLAG:
+ return reg->id == id;
+ default:
+ return false;
+ }
+}
+
+/*
+ * The frame and slot of the stack object of reference @ref, if it is one:
+ * the object may be in the stack of another frame, created through a
+ * pointer to it.
+ */
+static bool stack_object_find(const struct bpf_verifier_state *st,
+ const struct bpf_reference_state *ref, int *frameno, int *spi)
+{
+ int f, k;
+
+ for (f = st->curframe; f >= 0; f--) {
+ const struct bpf_func_state *frame = st->frame[f];
+
+ for (k = 0; k < bpf_stack_nr_slots(frame); k++) {
+ if (!stack_slot_is_object(bpf_stack_slot(frame, k), ref->id))
+ continue;
+ *frameno = f;
+ *spi = k;
+ return true;
+ }
+ }
+ return false;
+}
+
+/* The class of the sections of @kind, 0 RCU read and 1 preemption disabled. */
+static int resource_of_section(struct bpf_verifier_env *env, u8 kind)
+{
+ struct bpf_resource res;
+
+ memset(&res, 0, sizeof(res));
+ res.kind = kind ? BPF_RESOURCE_PREEMPT : BPF_RESOURCE_RCU;
+ return resource_intern(env, &res);
+}
+
+static int resource_of_kind(struct bpf_verifier_env *env, u8 kind)
+{
+ struct bpf_resource res;
+
+ memset(&res, 0, sizeof(res));
+ res.kind = kind;
+ return resource_intern(env, &res);
+}
+
+/* The offset from R10 of the stack slot @spi of a frame. */
+static s16 spi_off(int spi)
+{
+ return -(spi + 1) * BPF_REG_SIZE;
+}
+
+/*
+ * The class of the resource refs[@idx] the current insn acquires, and how the
+ * insn has its handle: in R@regno @after the call, the pointer the call
+ * returns, which may point @adj bytes into the object, or in the register in
+ * front of it, the pointer the call takes to a lock or to a stack object. The
+ * handle of a dynptr is the record or the reader, the first word of the
+ * dynptr at @obj_off, so that a clone releases it as well as the dynptr it was
+ * created in. -EOPNOTSUPP for a resource the unwinder cannot release, -EXDEV
+ * for a stack object in the stack of another frame.
+ */
+static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ const struct bpf_reference_state *ref, int regno, bool after, u8 *flags,
+ s16 *adj, s16 *obj_off)
+{
+ const struct bpf_func_state *frame = st->frame[st->curframe];
+ const struct bpf_stack_state *slot;
+ struct bpf_resource res;
+ struct bpf_reg_state reg;
+ int objframe, spi, ret;
+
+ *adj = 0;
+ *obj_off = 0;
+ if (after) {
+ reg = frame->regs[regno];
+ if (!reg_ref_off(®, ref->id, adj))
+ *adj = 0;
+ reg.var_off = tnum_const(0);
+ ret = resource_describe(env, ®, &res, false);
+ return ret ? -EOPNOTSUPP : resource_intern(env, &res);
+ }
+ *flags |= BPF_RES_F_ARG;
+ if (ref->type & REF_TYPE_LOCK_MASK)
+ return resource_of_kind(env, ref->type == REF_TYPE_LOCK ? BPF_RESOURCE_SPIN_LOCK :
+ BPF_RESOURCE_RES_LOCK);
+ if (!stack_object_find(st, ref, &objframe, &spi))
+ return -EOPNOTSUPP;
+ if (objframe != st->curframe)
+ return -EXDEV;
+ slot = bpf_stack_slot(st->frame[objframe], spi);
+ switch (slot->slot_type[BPF_REG_SIZE - 1]) {
+ case STACK_DYNPTR:
+ *flags |= BPF_RES_F_DEREF;
+ *obj_off = spi_off(spi);
+ return resource_of_kind(env, slot->spilled_ptr.dynptr.type == BPF_DYNPTR_TYPE_FILE ?
+ BPF_RESOURCE_DYNPTR_FILE : BPF_RESOURCE_RINGBUF);
+ case STACK_IRQ_FLAG:
+ return resource_of_kind(env, BPF_RESOURCE_IRQ_FLAG);
+ default:
+ ret = resource_of_iter(env, &slot->spilled_ptr);
+ return ret < 0 ? -EOPNOTSUPP : ret;
+ }
+}
+
+/*
+ * The acquisition of insn @insn through R@reg: the same on every visit, since
+ * the insns that keep its slot are emitted once. Returns its index.
+ */
+static int acq_add(struct bpf_verifier_env *env, u32 insn, u16 res, u8 reg, u8 flags, s16 adj,
+ s16 obj_off)
+{
+ u32 i, cnt = env->res_acq_cnt;
+ struct bpf_res_acq *a;
+
+ for (i = 0; i < cnt; i++) {
+ a = &env->res_acqs[i];
+ if (a->insn != insn || a->reg != reg || (a->flags ^ flags) & BPF_RES_F_CTX)
+ continue;
+ if (a->res != res || a->flags != flags || a->adj != adj || a->obj_off != obj_off) {
+ verbose(env, "frame_desc: insn %d acquires resources of different types on different paths\n",
+ insn);
+ return -EINVAL;
+ }
+ return i;
+ }
+ if (cnt >= U16_MAX - 1)
+ return -E2BIG;
+ a = kvrealloc(env->res_acqs, array_size(cnt + 1, sizeof(*a)), GFP_KERNEL_ACCOUNT);
+ if (!a)
+ return -ENOMEM;
+ a[cnt] = (struct bpf_res_acq) { .insn = insn, .res = res, .reg = reg, .flags = flags,
+ .adj = adj, .obj_off = obj_off };
+ env->res_acqs = a;
+ env->res_acq_cnt = cnt + 1;
+ return cnt;
+}
+
+/*
+ * refs[@idx] is what acquisition @acq acquired, held by the frame that owns
+ * it: its handle is in the slot of the acquisition, and the slot holds one.
+ */
+static int ref_set_acq(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int idx,
+ int acq)
+{
+ struct bpf_reference_state *ref = &st->refs[idx];
+ int i;
+
+ for (i = 0; i < st->acquired_refs; i++) {
+ if (i == idx || st->refs[i].acq != acq + 1 || st->refs[i].frameno != ref->frameno)
+ continue;
+ verbose(env, "frame_desc: a frame holds two resources of insn %d at once, which is not supported in a program that throws\n",
+ env->res_acqs[acq].insn);
+ return -EINVAL;
+ }
+ ref->res = env->res_acqs[acq].res + 1;
+ ref->acq = acq + 1;
+ return 0;
+}
+
+/*
+ * Record that the current insn acquires the resource refs[@idx], whose handle
+ * is in R@regno @after the insn, the pointer a call returns, or in front of
+ * it, the pointer to a stack object or to a lock the call takes. A call that
+ * @may_fail, a resilient lock, has acquired it only when it returns 0. A
+ * resource the unwinder cannot release gets no slot, and a throw may not be
+ * reached while it is held, see bpf_reference_state::res.
+ */
+int bpf_frame_desc_record_acquire(struct bpf_verifier_env *env, int idx, int regno, bool after,
+ bool may_fail)
+{
+ struct bpf_verifier_state *st = env->cur_state;
+ struct bpf_reference_state *ref = &st->refs[idx];
+ u8 flags = may_fail ? BPF_RES_F_MAY_FAIL : 0;
+ s16 adj, obj_off;
+ int res, acq;
+
+ ref->res = 0;
+ ref->acq = 0;
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ res = acquired_class(env, st, ref, regno, after, &flags, &adj, &obj_off);
+ if (res == -EOPNOTSUPP) {
+ /* the insns of another visit must not take this for a resource of theirs */
+ env->insn_aux_data[env->insn_idx].res_none = true;
+ return 0;
+ }
+ if (res == -EXDEV) {
+ verbose(env, "frame_desc: insn %d creates an object in the stack of another frame, which is not supported in a program that throws\n",
+ env->insn_idx);
+ return -EINVAL;
+ }
+ if (res < 0)
+ return res;
+ if (regno < BPF_REG_0 || regno > BPF_REG_5) {
+ verbose(env, "frame_desc: insn %d takes the handle of a resource in no register\n",
+ env->insn_idx);
+ return -EINVAL;
+ }
+ acq = acq_add(env, env->insn_idx, res, regno, flags, adj, obj_off);
+ return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
+}
+
+/*
+ * The current insn is a call that acquires nothing on this visit and leaves
+ * @r0. It may acquire what R0 carries on another visit, as a lookup does in a
+ * map of sockets and not in another map: the insns that keep the slot of that
+ * would take this R0 for the handle, see bpf_frame_desc_resolve().
+ */
+void bpf_frame_desc_record_none(struct bpf_verifier_env *env, const struct bpf_reg_state *r0)
+{
+ if (!env->describe_frames || env->cur_state->speculative)
+ return;
+ if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
+ env->insn_aux_data[env->insn_idx].res_none_r0 = true;
+}
+
+/*
+ * Did some visit of insn @idx acquire nothing a slot can hold where
+ * acquisition @a has a resource? An R0 that is no resource matters to an
+ * acquisition whose handle is R0: one that takes its handle in an argument,
+ * a lock for one, stores the argument.
+ */
+static bool acquires_sometimes(const struct bpf_verifier_env *env, u32 idx,
+ const struct bpf_res_acq *a)
+{
+ const struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+
+ return aux->res_none || (aux->res_none_r0 && !(a->flags & BPF_RES_F_ARG));
+}
+
+/*
+ * The kernel holds a reference for the __ref argument @argno of a struct_ops
+ * program, refs[@idx], and the program releases it or hands it back: the main
+ * program holds it from its first insn, in a slot its entry fills from the
+ * context. An argument of a type the unwinder cannot release gets no slot,
+ * like any such reference.
+ */
+int bpf_frame_desc_record_ctx_arg(struct bpf_verifier_env *env, int idx, int argno)
+{
+ const struct bpf_ctx_arg_aux *info = &env->prog->aux->ctx_arg_info[argno];
+ struct bpf_verifier_state *st = env->cur_state;
+ struct bpf_reference_state *ref = &st->refs[idx];
+ struct bpf_reg_state reg = {};
+ struct bpf_resource res;
+ int ret;
+
+ ref->res = 0;
+ ref->acq = 0;
+ if (!env->describe_frames)
+ return 0;
+ reg.type = info->reg_type & ~PTR_MAYBE_NULL;
+ reg.btf = info->btf;
+ reg.btf_id = info->btf_id;
+ if (resource_describe(env, ®, &res, false))
+ return 0;
+ ret = resource_intern(env, &res);
+ if (ret < 0)
+ return ret;
+ ret = acq_add(env, argno, ret, BPF_REG_1, BPF_RES_F_CTX, 0, info->offset);
+ return ret < 0 ? ret : ref_set_acq(env, st, idx, ret);
+}
+
+/*
+ * Record that the current insn releases the resource refs[@idx], before it
+ * is released: its handle, in the argument the call releases, see
+ * bpf_verifier_env::release_regno, leaves the slot of its acquisition in
+ * front of the call. A frame releases what it acquired itself: nothing tells
+ * a callee where the slot of its caller is.
+ */
+int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
+{
+ struct bpf_verifier_state *st = env->cur_state;
+ const struct bpf_reference_state *ref = &st->refs[idx];
+ const struct bpf_func_state *frame = st->frame[st->curframe];
+ u32 i, cnt = env->res_rel_cnt, insn = env->insn_idx;
+ int regno = env->release_regno;
+ const struct bpf_res_acq *acq;
+ struct bpf_res_rel *r;
+ u8 flags = 0;
+ s16 adj = 0;
+
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ /* the lock would be unlocked in memory that may be gone by then */
+ for (i = 0; ref->type == REF_TYPE_PTR && i < st->acquired_refs; i++) {
+ if (!(st->refs[i].type & REF_TYPE_LOCK_MASK) || st->refs[i].id != ref->id)
+ continue;
+ verbose(env, "frame_desc: insn %d releases an object whose lock is held, which is not supported in a program that throws\n",
+ insn);
+ return -EINVAL;
+ }
+ if (!ref->res)
+ return 0;
+ if (ref->frameno != st->curframe) {
+ verbose(env, "frame_desc: insn %d releases what another frame holds, which is not supported in a program that throws\n",
+ insn);
+ return -EINVAL;
+ }
+ acq = &env->res_acqs[ref->acq - 1];
+ if (regno < BPF_REG_1 || regno > BPF_REG_5) {
+ verbose(env, "frame_desc: insn %d releases a resource whose handle is in no argument register\n",
+ insn);
+ return -EINVAL;
+ }
+ if (acq->flags & BPF_RES_F_DEREF) {
+ flags = BPF_RES_F_DEREF;
+ } else if (!(acq->flags & BPF_RES_F_ARG) &&
+ !reg_ref_off(&frame->regs[regno], ref->id, &adj)) {
+ verbose(env, "frame_desc: insn %d releases a resource R%d does not point to\n", insn,
+ regno);
+ return -EINVAL;
+ }
+ for (i = 0; i < cnt; i++) {
+ r = &env->res_rels[i];
+ if (r->insn != insn || r->reg != regno)
+ continue;
+ /* the argument is taken the same way whatever it releases */
+ if (r->flags != flags || r->adj != adj) {
+ verbose(env, "frame_desc: insn %d takes the handle of what it releases differently on different paths\n",
+ insn);
+ return -EINVAL;
+ }
+ if (r->acq == ref->acq - 1)
+ return 0;
+ }
+ r = kvrealloc(env->res_rels, array_size(cnt + 1, sizeof(*r)), GFP_KERNEL_ACCOUNT);
+ if (!r)
+ return -ENOMEM;
+ r[cnt] = (struct bpf_res_rel) { .insn = insn, .acq = ref->acq - 1, .reg = regno,
+ .flags = flags, .adj = adj };
+ env->res_rels = r;
+ env->res_rel_cnt = cnt + 1;
+ return 0;
+}
+
+/* The current insn enters (@delta 1) or leaves (-1) a section of @kind: counted by its frame. */
+int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delta)
+{
+ struct bpf_verifier_state *st = env->cur_state;
+ struct bpf_section_insn *si;
+ u32 i, cnt = env->section_insn_cnt;
+
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ for (i = 0; i < cnt; i++)
+ if (env->section_insns[i].insn == env->insn_idx)
+ return 0;
+ si = kvrealloc(env->section_insns, array_size(cnt + 1, sizeof(*si)), GFP_KERNEL_ACCOUNT);
+ if (!si)
+ return -ENOMEM;
+ si[cnt] = (struct bpf_section_insn) { .insn = env->insn_idx, .kind = kind, .delta = delta };
+ env->section_insns = si;
+ env->section_insn_cnt = cnt + 1;
+ return 0;
+}
+
+/*
+ * The current frame returns to its caller. Its slots and its counts of
+ * sections go with the frame, and nothing moves them to the frame it returns
+ * to, so it returns holding nothing but a reference in R0, which the call
+ * acquires for the caller, see bpf_frame_desc_record_return(), and in as many
+ * sections as it was called in. The entry counts of the sections are lowered
+ * as the frame leaves its callers' sections, see release_context_state(),
+ * which counts those apart.
+ */
+int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *st = env->cur_state;
+ const struct bpf_func_state *frame = st->frame[st->curframe];
+ int i, held = 0;
+ s16 adj;
+
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ if (st->active_rcu_locks != frame->entry_rcu_locks || frame->rcu_released_below ||
+ st->active_preempt_locks != frame->entry_preempt_locks || frame->preempt_released_below) {
+ verbose(env, "frame_desc: a subprog returns in other sections than it was called in at insn %d, which is not supported in a program that throws\n",
+ env->insn_idx);
+ return -EINVAL;
+ }
+ for (i = 0; i < st->acquired_refs; i++) {
+ const struct bpf_reference_state *ref = &st->refs[i];
+
+ if (!ref->res || ref->frameno != st->curframe)
+ continue;
+ if (ref->type == REF_TYPE_PTR && !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
+ reg_ref_off(&frame->regs[BPF_REG_0], ref->id, &adj) && !held++)
+ continue;
+ verbose(env, "frame_desc: a subprog returns holding what R0 does not carry at insn %d, which is not supported in a program that throws\n",
+ env->insn_idx);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+/*
+ * A subprog called at insn @callsite returned to the current frame of @st
+ * with @r0. The reference refs[@idx] it returned holding is the caller's
+ * from here on, acquired by the call like the reference a kfunc returns; with
+ * @idx < 0 it returned holding nothing, and what it left in R0 must not be
+ * taken for a reference by the insns another visit has follow the call.
+ */
+int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int idx, int callsite, const struct bpf_reg_state *r0)
+{
+ struct bpf_reference_state *ref;
+ s16 adj = 0;
+ int acq;
+
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ ref = idx < 0 ? NULL : &st->refs[idx];
+ if (!ref || !ref->res) {
+ if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
+ env->insn_aux_data[callsite].res_none = true;
+ return 0;
+ }
+ if (!reg_ref_off(r0, ref->id, &adj)) {
+ verbose(env, "frame_desc: the subprog called at insn %d returns holding what R0 does not carry, which is not supported in a program that throws\n",
+ callsite);
+ return -EINVAL;
+ }
+ acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
+ return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
+}
+
+/*
+ * A round of the main pass is dropped, see do_check_all(): what its paths
+ * recorded goes with it.
+ */
+void bpf_frame_desc_reset(struct bpf_verifier_env *env)
+{
+ u32 i;
+
+ env->res_acq_cnt = 0;
+ env->res_rel_cnt = 0;
+ env->section_insn_cnt = 0;
+ for (i = 0; i < env->prog->len; i++) {
+ env->insn_aux_data[i].res_none = false;
+ env->insn_aux_data[i].res_none_r0 = false;
+ }
+}
+
+void bpf_frame_desc_env_free(struct bpf_verifier_env *env)
+{
+ kvfree(env->res_acqs);
+ kvfree(env->res_rels);
+ kvfree(env->section_insns);
+ kvfree(env->resources);
+ env->res_acqs = NULL;
+ env->res_rels = NULL;
+ env->section_insns = NULL;
+ env->resources = NULL;
+}
+
+/*
+ * The pass named the insns of the program it verified. The insns of a patch
+ * all name the insn they replaced, and its aux data went to the last of them,
+ * see patch_list_adj_aux(): map every insn of them to that one, the call
+ * itself when insns were put in front of it. Returns NULL without memory.
+ */
+static u32 *insns_of_pass(struct bpf_verifier_env *env, u32 *orig_len)
+{
+ u32 i, len = env->prog->len, n = 0, *new_of;
+
+ /* a hidden subprog was appended to the last insn, and its insns name that one */
+ if (env->hidden_subprog_cnt)
+ len = env->subprog_info[env->subprog_cnt - 1].start;
+ for (i = 0; i < len; i++)
+ n = max(n, env->insn_aux_data[i].orig_idx + 1);
+ new_of = kvcalloc(n, sizeof(*new_of), GFP_KERNEL_ACCOUNT);
+ if (!new_of)
+ return NULL;
+ for (i = 0; i < len; i++)
+ new_of[env->insn_aux_data[i].orig_idx] = i;
+ *orig_len = n;
+ return new_of;
+}
+
+/* The subprog an acquisition is in: the main program for an argument of the context. */
+static int acq_subprog(struct bpf_verifier_env *env, const struct bpf_res_acq *a, const u32 *new_of)
+{
+ if (a->flags & BPF_RES_F_CTX)
+ return 0;
+ return bpf_find_containing_subprog(env, new_of[a->insn]) - env->subprog_info;
+}
+
+static bool subprog_has_section(struct bpf_verifier_env *env, int subprog, const u32 *new_of, u8 kind)
+{
+ u32 i;
+
+ for (i = 0; i < env->section_insn_cnt; i++) {
+ const struct bpf_section_insn *si = &env->section_insns[i];
+
+ if (si->kind == kind &&
+ bpf_find_containing_subprog(env, new_of[si->insn]) - env->subprog_info == subprog)
+ return true;
+ }
+ return false;
+}
+
+/* An upper bound of the insns put around one call for one acquisition or release. */
+#define RES_INSNS_MAX 6
+
+/* In front of a call that takes the handle in an argument: the handle goes into the slot. */
+static int emit_acquire_before(struct bpf_insn *buf, const struct bpf_res_acq *a)
+{
+ int cnt = 0;
+
+ if ((a->flags & (BPF_RES_F_ARG | BPF_RES_F_DEREF)) == BPF_RES_F_ARG)
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, a->reg, a->off);
+ return cnt;
+}
+
+/*
+ * After a call that acquires: the pointer it returned goes into the slot, as
+ * a pointer to the object, and so does the record or the reader of the
+ * dynptr it created; NULL when it acquired nothing. A call that may fail has
+ * acquired what it took only when it returned 0.
+ */
+static int emit_acquire_after(struct bpf_insn *buf, const struct bpf_res_acq *a)
+{
+ int cnt = 0;
+
+ if (a->flags & BPF_RES_F_DEREF) {
+ buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_FP, a->obj_off);
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_AX, a->off);
+ } else if (a->flags & BPF_RES_F_ARG) {
+ if (a->flags & BPF_RES_F_MAY_FAIL) {
+ buf[cnt++] = BPF_JMP32_IMM(BPF_JEQ, BPF_REG_0, 0, 1);
+ buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, a->off, 0);
+ }
+ } else if (a->adj) {
+ buf[cnt++] = BPF_MOV64_REG(BPF_REG_AX, BPF_REG_0);
+ buf[cnt++] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, 1);
+ buf[cnt++] = BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, a->adj);
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_AX, a->off);
+ } else {
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_0, a->off);
+ }
+ return cnt;
+}
+
+/*
+ * Reserve the hidden slots, once the pass has seen every frame: one for each
+ * acquisition, in the frame of the subprog it is in, and a count of the
+ * sections of a kind for a subprog that enters or leaves them. The subprogs
+ * are numbered as they are now and looked up again whenever they are needed,
+ * since removing a dead one renumbers them.
+ */
+int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
+{
+ u32 i, k, orig_len, *new_of;
+ int s, ret = 0;
+
+ if (!env->res_acq_cnt && !env->section_insn_cnt)
+ return 0;
+ new_of = insns_of_pass(env, &orig_len);
+ if (!new_of)
+ return -ENOMEM;
+ for (i = 0; i < env->res_acq_cnt && !ret; i++) {
+ struct bpf_res_acq *a = &env->res_acqs[i];
+
+ if (!(a->flags & BPF_RES_F_CTX) &&
+ verifier_bug_if(a->insn >= orig_len, env, "resource at insn %d", a->insn)) {
+ ret = -EFAULT;
+ break;
+ }
+ /* some visit of the insn has nothing a slot can hold where this one has */
+ if (!(a->flags & BPF_RES_F_CTX) && acquires_sometimes(env, new_of[a->insn], a)) {
+ verbose(env, "frame_desc: insn %d acquires a resource on some paths only, or one of a type that cannot be released\n",
+ a->insn);
+ ret = -EINVAL;
+ break;
+ }
+ ret = bpf_reserve_stack_slots(env, acq_subprog(env, a, new_of), 1, &a->off);
+ env->frame_desc_slots++;
+ }
+ for (s = 0; s < env->subprog_cnt && !ret; s++) {
+ for (k = 0; k < ARRAY_SIZE(env->subprog_info[s].section_off) && !ret; k++) {
+ if (!subprog_has_section(env, s, new_of, k))
+ continue;
+ ret = bpf_reserve_stack_slots(env, s, 1, &env->subprog_info[s].section_off[k]);
+ env->frame_desc_slots++;
+ }
+ }
+ kvfree(new_of);
+ return ret;
+}
+
+/* What is kept at an insn of the current image, in the order it is emitted in. */
+enum {
+ ITEM_SECTION,
+ ITEM_RELEASE,
+ ITEM_ACQUIRE,
+};
+
+struct res_item {
+ u32 idx; /* insn of the current image */
+ u32 rec; /* which section insn, release or acquisition */
+ u8 kind;
+ u8 reg; /* ITEM_RELEASE: the argument the call releases */
+};
+
+static int item_cmp(const void *a, const void *b)
+{
+ const struct res_item *x = a, *y = b;
+
+ if (x->idx != y->idx)
+ return x->idx < y->idx ? -1 : 1;
+ if (x->kind != y->kind)
+ return x->kind - y->kind;
+ if (x->reg != y->reg)
+ return x->reg - y->reg;
+ return x->rec < y->rec ? -1 : x->rec > y->rec;
+}
+
+/*
+ * In front of a call that releases what one of the acquisitions of the @n
+ * items from @it acquired, in one argument: the slot that holds the handle
+ * the call takes is cleared, one slot, since two references to one object
+ * have equal handles in a slot each. No slot is touched when the call takes
+ * NULL or what another acquisition acquired. R0 is free in front of a call.
+ */
+static int emit_release(struct bpf_verifier_env *env, struct bpf_insn *buf,
+ const struct res_item *it, int n)
+{
+ const struct bpf_res_rel *r = &env->res_rels[it->rec];
+ int i, cnt = 0, reg = r->reg, per = r->adj ? 5 : 4;
+
+ if (r->flags & BPF_RES_F_DEREF) {
+ buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, r->reg, 0);
+ reg = BPF_REG_0;
+ }
+ for (i = 0; i < n; i++) {
+ s16 off = env->res_acqs[env->res_rels[it[i].rec].acq].off;
+ bool last = i == n - 1;
+
+ buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_FP, off);
+ if (r->adj)
+ buf[cnt++] = BPF_ALU64_IMM(BPF_ADD, BPF_REG_AX, r->adj);
+ buf[cnt++] = BPF_JMP_REG(BPF_JNE, BPF_REG_AX, reg, last ? 1 : 2);
+ buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, off, 0);
+ /* past the slots that follow: the last one has no jump */
+ if (!last)
+ buf[cnt++] = BPF_JMP_A((n - 1 - i) * per - 1);
+ }
+ return cnt;
+}
+
+/*
+ * The insns around the call @n items from @it are kept at: in @buf, those in
+ * front of the call, the call, and those that follow it. Returns how many
+ * follow it, and in @before how many come in front.
+ */
+static int emit_items(struct bpf_verifier_env *env, const struct res_item *it, int n,
+ struct bpf_insn *buf, int *before)
+{
+ const struct bpf_subprog_info *sub = bpf_find_containing_subprog(env, it->idx);
+ int i, k, cnt = 0, after;
+
+ for (i = 0; i < n; i++) {
+ const struct bpf_section_insn *si;
+ s16 off;
+
+ switch (it[i].kind) {
+ case ITEM_SECTION:
+ si = &env->section_insns[it[i].rec];
+ off = sub->section_off[si->kind];
+ buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_FP, off);
+ buf[cnt++] = BPF_ALU64_IMM(BPF_ADD, BPF_REG_AX, si->delta);
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_AX, off);
+ break;
+ case ITEM_RELEASE:
+ /* the acquisitions one argument may release what was acquired by */
+ for (k = i + 1; k < n && it[k].kind == ITEM_RELEASE && it[k].reg == it[i].reg; k++)
+ ;
+ cnt += emit_release(env, buf + cnt, &it[i], k - i);
+ i = k - 1;
+ break;
+ case ITEM_ACQUIRE:
+ cnt += emit_acquire_before(buf + cnt, &env->res_acqs[it[i].rec]);
+ break;
+ }
+ }
+ *before = cnt;
+ buf[cnt++] = env->prog->insnsi[it->idx];
+ after = cnt;
+ for (i = 0; i < n; i++)
+ if (it[i].kind == ITEM_ACQUIRE)
+ cnt += emit_acquire_after(buf + cnt, &env->res_acqs[it[i].rec]);
+ return cnt - after;
+}
+
+/*
+ * At the entry of each subprog: its slots and counts start out empty, and
+ * the main program takes the references the kernel passed it from the
+ * context, which R1 points at. The insns go in front of the first one, which
+ * a jump to it skips.
+ */
+static int emit_entries(struct bpf_verifier_env *env)
+{
+ u32 i, k, orig_len, *new_of;
+ struct bpf_insn *buf;
+ int s, cnt, ret = 0;
+
+ new_of = insns_of_pass(env, &orig_len);
+ buf = kvcalloc(2 * (env->res_acq_cnt + 2), sizeof(*buf), GFP_KERNEL_ACCOUNT);
+ if (!new_of || !buf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ for (s = 0; s < env->subprog_cnt && !ret; s++) {
+ const struct bpf_subprog_info *sub = &env->subprog_info[s];
+
+ cnt = 0;
+ for (i = 0; i < env->res_acq_cnt; i++) {
+ const struct bpf_res_acq *a = &env->res_acqs[i];
+
+ if (acq_subprog(env, a, new_of) != s)
+ continue;
+ if (a->flags & BPF_RES_F_CTX) {
+ buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_1, a->obj_off);
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_AX, a->off);
+ } else {
+ buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, a->off, 0);
+ }
+ }
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++)
+ if (sub->section_off[k])
+ buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, sub->section_off[k], 0);
+ if (!cnt)
+ continue;
+ ret = bpf_patch_list_insert(env, sub->start, buf, cnt);
+ env->frame_desc_insns += cnt;
+ }
+ ret = ret ?: bpf_patch_list_commit(env);
+out:
+ kvfree(buf);
+ kvfree(new_of);
+ return ret;
+}
+
+/*
+ * Keep the slots and the counts up to date, through the patch list. What goes
+ * in front of a call is patched in with the call, so that the jumps to it
+ * land on it; what follows it is inserted in front of the next insn, which a
+ * jump to that insn skips, as it skips the call.
+ */
+int bpf_frame_desc_emit(struct bpf_verifier_env *env)
+{
+ u32 i, k, n = 0, len = env->prog->len, orig_len, *new_of, max = 0;
+ struct bpf_insn *buf = NULL, *insn;
+ struct res_item *items = NULL;
+ int ret = 0, before, after;
+
+ if (!env->res_acq_cnt && !env->section_insn_cnt)
+ return 0;
+ new_of = insns_of_pass(env, &orig_len);
+ items = kvcalloc(env->res_acq_cnt + env->res_rel_cnt + env->section_insn_cnt, sizeof(*items),
+ GFP_KERNEL_ACCOUNT);
+ if (!new_of || !items) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ for (i = 0; i < env->section_insn_cnt; i++)
+ items[n++] = (struct res_item) { new_of[env->section_insns[i].insn], i, ITEM_SECTION };
+ for (i = 0; i < env->res_rel_cnt; i++)
+ items[n++] = (struct res_item) { new_of[env->res_rels[i].insn], i, ITEM_RELEASE,
+ env->res_rels[i].reg };
+ for (i = 0; i < env->res_acq_cnt; i++)
+ if (!(env->res_acqs[i].flags & BPF_RES_F_CTX))
+ items[n++] = (struct res_item) { new_of[env->res_acqs[i].insn], i, ITEM_ACQUIRE };
+ sort(items, n, sizeof(*items), item_cmp, NULL);
+ for (i = 0; i < n; i = k) {
+ for (k = i; k < n && items[k].idx == items[i].idx; k++)
+ ;
+ max = max(max, k - i);
+ }
+ buf = kvcalloc(max * RES_INSNS_MAX + 1, sizeof(*buf), GFP_KERNEL_ACCOUNT);
+ if (!buf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ for (i = 0; i < n && !ret; i = k) {
+ u32 idx = items[i].idx;
+
+ for (k = i; k < n && items[k].idx == idx; k++)
+ ;
+ if (verifier_bug_if(idx >= len || env->prog->insnsi[idx].code != (BPF_JMP | BPF_CALL),
+ env, "resource kept at insn %d, not a call", idx)) {
+ ret = -EFAULT;
+ break;
+ }
+ after = emit_items(env, &items[i], k - i, buf, &before);
+ /* the call of a subprog in front of it does not move with the call */
+ ret = bpf_adj_moved_insn(&buf[before], before);
+ if (ret)
+ break;
+ /*
+ * A callee that unwound returns right after the call and goes where
+ * the unwind_goto says: nothing may come between the two.
+ */
+ if (after && idx + 1 < len && bpf_is_unwind_goto(&env->prog->insnsi[idx + 1])) {
+ verbose(env, "frame_desc: the call at insn %d acquires a resource and may unwind, which is not supported in a program that throws\n",
+ idx);
+ ret = -EINVAL;
+ break;
+ }
+ if (before) {
+ insn = bpf_patch_list_add(env, idx, buf, before + 1);
+ ret = PTR_ERR_OR_ZERO(insn);
+ }
+ if (!ret && after)
+ ret = bpf_patch_list_insert(env, idx + 1, buf + before + 1, after);
+ env->frame_desc_insns += before + after;
+ }
+ ret = ret ?: bpf_patch_list_commit(env);
+ ret = ret ?: emit_entries(env);
+out:
+ kvfree(buf);
+ kvfree(items);
+ kvfree(new_of);
+ return ret;
+}
+
+/* A stack object may use a referenced object the frame holds as well: it goes first. */
+static bool resource_goes_first(const struct bpf_resource *res)
+{
+ return res->kind == BPF_RESOURCE_ITER || res->kind == BPF_RESOURCE_DYNPTR_FILE;
+}
+
+/*
+ * The frame descriptor of every subprog that has slots: the slot of each of
+ * its acquisitions, which holds the handle of what the acquisition acquired
+ * or NULL, and the counts of its sections. The unwinder releases what the
+ * slots of a frame hold in the order of the entries, within a phase: the
+ * iterators and the readers of file dynptrs come before the references,
+ * which they may be iterating over or reading. Identical descriptors share a
+ * body in the pool of the program, which takes the resource classes over.
+ */
+int bpf_frame_desc_build(struct bpf_verifier_env *env)
+{
+ u32 i, k, n, s, orig_len, *new_of;
+ struct bpf_frame_desc_pool *pool;
+ int res, ret = -ENOMEM;
+
+ if (!env->res_acq_cnt && !env->section_insn_cnt)
+ return 0;
+ new_of = insns_of_pass(env, &orig_len);
+ pool = kvzalloc(struct_size(pool, desc, 0), GFP_KERNEL_ACCOUNT);
+ if (!new_of || !pool) {
+ kvfree(pool);
+ goto out;
+ }
+ env->frame_desc_pool = pool;
+ for (s = 0; s < env->subprog_cnt; s++) {
+ struct bpf_subprog_info *sub = &env->subprog_info[s];
+ struct bpf_frame_desc *d;
+
+ for (i = 0, n = 0; i < env->res_acq_cnt; i++)
+ n += acq_subprog(env, &env->res_acqs[i], new_of) == s;
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++)
+ n += !!sub->section_off[k];
+ if (!n)
+ continue;
+ d = kvzalloc(struct_size(d, ent, n), GFP_KERNEL_ACCOUNT);
+ if (!d)
+ goto out;
+ d->size = struct_size(d, ent, n);
+ for (i = 0; i < 2 * env->res_acq_cnt; i++) {
+ const struct bpf_res_acq *a = &env->res_acqs[i % env->res_acq_cnt];
+ bool first = i < env->res_acq_cnt;
+
+ if (acq_subprog(env, a, new_of) != s ||
+ resource_goes_first(&env->resources[a->res]) != first)
+ continue;
+ d->ent[d->nent++] = (struct bpf_frame_desc_entry) { .res = a->res, .off = a->off };
+ }
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++) {
+ if (!sub->section_off[k])
+ continue;
+ res = resource_of_section(env, k);
+ if (res < 0) {
+ kvfree(d);
+ ret = res;
+ goto out;
+ }
+ d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
+ .res = res, .off = sub->section_off[k], .flags = BPF_FRAME_ENT_COUNTER,
+ };
+ }
+ for (i = 0; i < pool->cnt; i++)
+ if (pool->desc[i]->size == d->size && !memcmp(pool->desc[i], d, d->size))
+ break;
+ if (i < pool->cnt) {
+ kvfree(d);
+ sub->frame_desc = i + 1;
+ continue;
+ }
+ pool = kvrealloc(pool, struct_size(pool, desc, pool->cnt + 1), GFP_KERNEL_ACCOUNT);
+ if (!pool) {
+ kvfree(d);
+ goto out;
+ }
+ env->frame_desc_pool = pool;
+ pool->desc[pool->cnt] = d;
+ env->frame_desc_bytes += d->size;
+ sub->frame_desc = ++pool->cnt;
+ }
+ pool->resources = env->resources;
+ pool->resource_cnt = env->resource_cnt;
+ env->frame_desc_bytes += env->resource_cnt * sizeof(*env->resources);
+ env->resources = NULL;
+ ret = 0;
+out:
+ kvfree(new_of);
+ return ret;
+}
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index cc4494aef187..ba1176dc868e 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -462,6 +462,10 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
fa->entry_rcu_locks != fb->entry_rcu_locks ||
fa->entry_preempt_locks != fb->entry_preempt_locks)
return false;
+ /* a frame that left a section of its callers hands its count over, see frame_desc.c */
+ if (env->describe_frames && (fa->rcu_released_below != fb->rcu_released_below ||
+ fa->preempt_released_below != fb->preempt_released_below))
+ return false;
if (!join_loop_stack(fa, fb))
return false;
}
@@ -523,7 +527,7 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
}
/* Things that can't be different in two states that are joined */
-static bool join_check_state(const struct bpf_verifier_state *a,
+static bool join_check_state(struct bpf_verifier_env *env, const struct bpf_verifier_state *a,
const struct bpf_verifier_state *b)
{
int i;
@@ -546,6 +550,16 @@ static bool join_check_state(const struct bpf_verifier_state *a,
for (i = 0; i < a->acquired_refs; i++) {
if (a->refs[i].type != b->refs[i].type)
return false;
+ /*
+ * The frames of a program that may throw keep the handle of a
+ * reference in the slot of the call that acquired it, in the frame
+ * that holds it, see frame_desc.c, and the result keeps those of
+ * 'a'.
+ */
+ if (env->describe_frames &&
+ (a->refs[i].frameno != b->refs[i].frameno || a->refs[i].res != b->refs[i].res ||
+ (a->refs[i].frameno >= 0 && a->refs[i].acq != b->refs[i].acq)))
+ return false;
switch (a->refs[i].type) {
case REF_TYPE_PTR:
case REF_TYPE_IRQ:
@@ -632,7 +646,7 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
js->npairs = 0;
js->pairs_overflow = false;
- if (!join_check_state(a, b))
+ if (!join_check_state(env, a, b))
return 1;
err = bpf_copy_verifier_state(dst, a);
if (err)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index efc0870b85c2..8f357fee6104 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -562,8 +562,8 @@ static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *
return true;
}
-static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
- struct bpf_idmap *idmap)
+static bool refsafe(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+ struct bpf_verifier_state *cur, struct bpf_idmap *idmap)
{
int i;
@@ -590,6 +590,19 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
if (!check_ids(old->refs[i].id, cur->refs[i].id, idmap) ||
old->refs[i].type != cur->refs[i].type)
return false;
+ /*
+ * The frames of a program that may throw keep the handle of a
+ * reference in the slot of the call that acquired it, in the frame
+ * that holds it, see frame_desc.c: a state doesn't stand for one
+ * that holds the reference as another class, in another frame, or
+ * from another call. What the callers of a function verified on its
+ * own hold has no slot in its frames.
+ */
+ if (env->describe_frames &&
+ (old->refs[i].frameno != cur->refs[i].frameno ||
+ old->refs[i].res != cur->refs[i].res ||
+ (old->refs[i].frameno >= 0 && old->refs[i].acq != cur->refs[i].acq)))
+ return false;
switch (old->refs[i].type) {
case REF_TYPE_PTR:
if (!check_ids(old->refs[i].parent_id, cur->refs[i].parent_id, idmap))
@@ -695,6 +708,10 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
old->entry_rcu_locks != cur->entry_rcu_locks ||
old->entry_preempt_locks != cur->entry_preempt_locks)
return false;
+ /* a frame that left a section of its callers hands its count over, see frame_desc.c */
+ if (env->describe_frames && (old->rcu_released_below != cur->rcu_released_below ||
+ old->preempt_released_below != cur->preempt_released_below))
+ return false;
for (i = 0; i < MAX_BPF_REG; i++)
if (((1 << i) & live_regs) &&
@@ -748,7 +765,7 @@ bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state
if (old->unwound != cur->unwound || old->changed_pkt_data != cur->changed_pkt_data)
return false;
- if (!refsafe(old, cur, &env->idmap_scratch))
+ if (!refsafe(env, old, cur, &env->idmap_scratch))
return false;
/* for states to be equal callsites have to be the same
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index cf3cfc734ed4..4d85a42a18d5 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -205,13 +205,14 @@ struct bpf_verifier_stack_elem {
static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id);
static int __release_reference_nomark(struct bpf_verifier_state *state, int id);
+static void set_release_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg);
static int release_reference_nomark(struct bpf_verifier_env *env, int id);
static int release_reference(struct bpf_verifier_env *env, int id);
static int invalidate_reference(struct bpf_verifier_env *env, int id);
static u32 reg_lifetime_id(struct bpf_verifier_env *env,
const struct bpf_reg_state *reg);
static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
-static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env);
+static int invalidate_rcu_protected_refs(struct bpf_verifier_env *env);
static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
static bool is_tracing_prog_type(enum bpf_prog_type type);
static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -812,6 +813,10 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
mark_dynptr_stack_regs(env, &bpf_stack_slot(state, spi)->spilled_ptr,
&bpf_stack_slot(state, spi - 1)->spilled_ptr, type, parent_id);
+ /* a referenced dynptr is a stack object the unwinder discards at its address */
+ if (dynptr->type == BPF_DYNPTR_TYPE_INVALID && dynptr_type_referenced(type))
+ return bpf_frame_desc_record_acquire(env, env->cur_state->acquired_refs - 1,
+ reg - cur_regs(env), false, false);
return 0;
}
@@ -1076,7 +1081,9 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
mark_stack_slot_scratched(env, spi - i);
}
- return 0;
+ /* a stack object the unwinder destroys at its address, which the call takes */
+ return bpf_frame_desc_record_acquire(env, env->cur_state->acquired_refs - 1,
+ reg - cur_regs(env), false, false);
}
static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
@@ -1094,6 +1101,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
struct bpf_reg_state *st = &slot->spilled_ptr;
if (i == 0) {
+ set_release_reg(env, reg);
err = release_reference(env, st->id);
if (err)
return err;
@@ -1202,7 +1210,9 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env,
slot->slot_type[i] = STACK_IRQ_FLAG;
mark_stack_slot_scratched(env, spi);
- return 0;
+ /* a stack object the unwinder restores at its address, which the call takes; irqsave may fail */
+ return bpf_frame_desc_record_acquire(env, env->cur_state->acquired_refs - 1,
+ reg - cur_regs(env), false, kfunc_class == IRQ_LOCK_KFUNC);
}
static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
@@ -1239,6 +1249,17 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r
return -EINVAL;
}
+ /* the frame descriptors take the flags out of their slot, see frame_desc.c */
+ set_release_reg(env, reg);
+ for (i = 0; i < env->cur_state->acquired_refs; i++) {
+ if (env->cur_state->refs[i].type != REF_TYPE_IRQ ||
+ env->cur_state->refs[i].id != st->id)
+ continue;
+ err = bpf_frame_desc_record_release(env, i);
+ if (err)
+ return err;
+ break;
+ }
err = release_irq_state(env, st->id);
WARN_ON_ONCE(err && err != -EACCES);
if (err) {
@@ -3715,6 +3736,25 @@ static void check_fastcall_stack_contract(struct bpf_verifier_env *env,
}
}
+/*
+ * A frame of a program that may throw keeps the address of an iterator and of
+ * saved IRQ flags it holds in a hidden slot, and the unwinder destroys or
+ * restores what is at the address, see frame_desc.c: the program may not write
+ * over one. Without a throw, the leak such a write leaves refuses the program
+ * at its exit.
+ */
+static int check_stack_object_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
+ int spi)
+{
+ u8 type = bpf_stack_slot(state, spi)->slot_type[BPF_REG_SIZE - 1];
+
+ if (!env->describe_frames || (type != STACK_ITER && type != STACK_IRQ_FLAG))
+ return 0;
+ verbose(env, "cannot write over %s in a program that throws\n",
+ type == STACK_ITER ? "an iterator" : "saved irq flags");
+ return -EINVAL;
+}
+
static void scrub_special_slot(struct bpf_func_state *state, int spi)
{
struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
@@ -3784,7 +3824,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
env->insn_aux_data[insn_idx].nospec_result = true;
}
- err = destroy_if_dynptr_stack_slot(env, state, spi);
+ err = check_stack_object_write(env, state, spi);
+ err = err ?: destroy_if_dynptr_stack_slot(env, state, spi);
if (err)
return err;
@@ -3886,6 +3927,13 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
(!value_reg && is_bpf_st_mem(insn) && insn->imm == 0))
writing_zero = true;
+ for (i = bpf_get_spi(max_off - 1); i <= bpf_get_spi(min_off); i++) {
+ if (i >= bpf_stack_nr_slots(state))
+ break;
+ err = check_stack_object_write(env, state, i);
+ if (err)
+ return err;
+ }
err = destroy_dynptrs_in_stack_slots(env, state, bpf_get_spi(max_off - 1),
bpf_get_spi(min_off));
if (err)
@@ -8472,6 +8520,11 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state
verbose(env, "Failed to acquire lock state\n");
return err;
}
+ /* the unwinder releases the lock at the address the call takes; a res lock may fail */
+ err = bpf_frame_desc_record_acquire(env, cur->acquired_refs - 1,
+ reg - cur_regs(env), false, is_res_lock);
+ if (err)
+ return err;
} else {
void *ptr;
int type;
@@ -8518,6 +8571,14 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state
"Release nested locks in the reverse order they were acquired.", lock);
return -EINVAL;
}
+ /* the frame descriptors take the lock out of its slot, see frame_desc.c */
+ set_release_reg(env, reg);
+ lock = find_lock_state(cur, type, reg->id, ptr);
+ if (lock) {
+ err = bpf_frame_desc_record_release(env, lock - cur->refs);
+ if (err)
+ return err;
+ }
if (release_lock_state(env, type, reg->id, ptr)) {
verbose(env, "%s_unlock of different lock\n", lock_str);
bpf_diag_lock(
@@ -8533,8 +8594,11 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state
*/
invalidate_non_owning_refs(env);
- if (!in_rcu_cs(env))
- invalidate_rcu_protected_refs(env);
+ if (!in_rcu_cs(env)) {
+ err = invalidate_rcu_protected_refs(env);
+ if (err)
+ return err;
+ }
}
return 0;
}
@@ -11108,11 +11172,32 @@ static int __release_reference_nomark(struct bpf_verifier_state *state, int id)
return -EINVAL;
}
+/*
+ * The current call releases what its argument @reg holds or points to: the
+ * frame descriptors take the handle from that argument, see frame_desc.c. An
+ * argument passed on the stack has no register.
+ */
+static void set_release_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg)
+{
+ const struct bpf_reg_state *regs = cur_regs(env);
+
+ env->release_regno = reg >= regs && reg < regs + MAX_BPF_REG ? reg - regs : -1;
+}
+
static int release_reference_nomark(struct bpf_verifier_env *env, int id)
{
- int err;
+ struct bpf_verifier_state *state = env->cur_state;
+ int i, err;
- err = __release_reference_nomark(env->cur_state, id);
+ for (i = 0; i < state->acquired_refs; i++) {
+ if (state->refs[i].type != REF_TYPE_PTR || state->refs[i].id != id)
+ continue;
+ err = bpf_frame_desc_record_release(env, i);
+ if (err)
+ return err;
+ break;
+ }
+ err = __release_reference_nomark(state, id);
if (!err)
bpf_diag_record_ref_release(env, env->insn_idx, id);
return err;
@@ -11213,9 +11298,13 @@ static int invalidate_reference(struct bpf_verifier_env *env, int id)
static int release_reference(struct bpf_verifier_env *env, int id)
{
- if (find_reference_state(env->cur_state, id))
- WARN_ON_ONCE(release_reference_nomark(env, id));
+ int err;
+ if (find_reference_state(env->cur_state, id)) {
+ err = release_reference_nomark(env, id);
+ if (err)
+ return err;
+ }
return invalidate_reference(env, id);
}
@@ -11232,21 +11321,34 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env)
}));
}
-static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
+static int invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
{
struct bpf_stack_state *stack;
struct bpf_func_state *state;
struct bpf_reg_state *reg;
u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER);
+ bool iter = false;
bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({
if (reg->type & MEM_RCU) {
+ iter |= stack && stack->slot_type[BPF_REG_SIZE - 1] == STACK_ITER;
bpf_diag_mod_begin(env, reg, NULL, BPF_DIAG_MOD_WRITE);
reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL | NON_OWN_REF);
reg->type |= PTR_UNTRUSTED;
bpf_diag_mod_end(env);
}
}));
+ /*
+ * Nothing may destroy an RCU protected iterator once its section ended:
+ * not the program, which is refused when it leaks one, and not the
+ * unwinder, which destroys what a frame holds at a throw, see
+ * frame_desc.c.
+ */
+ if (iter && env->describe_frames) {
+ verbose(env, "an RCU protected iterator may not outlive its section in a program that throws\n");
+ return -EINVAL;
+ }
+ return 0;
}
static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id)
@@ -11776,6 +11878,10 @@ static int exit_func_on_its_own(struct bpf_verifier_env *env)
if (err)
return err;
+ /* what the function returns with is its caller's from here on */
+ err = bpf_frame_desc_record_exit(env);
+ if (err)
+ return err;
account_current_path(env);
err = bpf_flow_exit(env, state, env->insn_idx);
return err ?: PROCESS_BPF_EXIT;
@@ -11896,6 +12002,16 @@ int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *cal
if (caller->refs[i].type == REF_TYPE_PTR)
caller->refs[i].parent_id = fresh_id(env, idmap, caller->refs[i].parent_id);
caller->refs[i].frameno = caller->curframe;
+ /* the call acquired it for the caller, see frame_desc.c */
+ err = bpf_frame_desc_record_return(env, caller, i, call_idx, &frame->regs[BPF_REG_0]);
+ if (err)
+ return err;
+ }
+ if (nheld == caller->acquired_refs) {
+ err = bpf_frame_desc_record_return(env, caller, -1, call_idx,
+ &callee->regs[BPF_REG_0]);
+ if (err)
+ return err;
}
caller->active_irq_id = fresh_id(env, idmap, caller->active_irq_id);
caller->active_lock_id = fresh_id(env, idmap, caller->active_lock_id);
@@ -12620,7 +12736,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
struct bpf_func_state *caller, *callee;
struct bpf_reg_state *r0;
u32 i, nregs;
- int err;
+ int err, handed;
callee = state->frame[state->curframe];
r0 = &callee->regs[BPF_REG_0];
@@ -12702,9 +12818,26 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
invalidate_stack_slices(env, callee->frameno, INT_MIN, 0);
/* What the frame holds when it returns is the caller's to release. */
- for (i = 0; i < state->acquired_refs; i++)
- if (state->refs[i].frameno == callee->frameno)
- state->refs[i].frameno = caller->frameno;
+ err = bpf_frame_desc_record_exit(env);
+ if (err)
+ return err;
+ for (i = 0, handed = -1; i < state->acquired_refs; i++) {
+ if (state->refs[i].frameno != callee->frameno)
+ continue;
+ state->refs[i].frameno = caller->frameno;
+ handed = i;
+ /* the call acquired it for the caller, see frame_desc.c */
+ err = bpf_frame_desc_record_return(env, state, i, callee->callsite,
+ &callee->regs[BPF_REG_0]);
+ if (err)
+ return err;
+ }
+ if (handed < 0 && !callee->in_callback_fn) {
+ err = bpf_frame_desc_record_return(env, state, -1, callee->callsite,
+ &callee->regs[BPF_REG_0]);
+ if (err)
+ return err;
+ }
/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
* there function call logic would reschedule callback visit.
@@ -12723,9 +12856,9 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
account_processed_insns(env, callee, caller);
/* clear everything in the callee. In case of exceptional exits using
* bpf_throw, this will be done by do_check_block() for extra frames. */
- free_func_state(callee);
state->frame[state->curframe--] = NULL;
invalidate_outgoing_stack_args(env, caller);
+ free_func_state(callee);
return 0;
}
@@ -12877,6 +13010,14 @@ static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exi
for (i = 0; i < state->acquired_refs; i++) {
if (state->refs[i].type != REF_TYPE_PTR)
continue;
+ /*
+ * The unwinder releases what the slots of the frames hold, see
+ * frame_desc.c: every reference but one of a class it cannot
+ * release, which no slot holds.
+ */
+ if (exception_exit && bpf_jit_supports_exception_cleanup() &&
+ (state->refs[i].res || state->speculative))
+ continue;
/* Allow struct_ops programs to return a referenced kptr back to
* kernel. Type checks are performed later in check_return_code.
*/
@@ -13093,6 +13234,8 @@ static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
if (bpf_register_is_null(reg))
return 0;
+ set_release_reg(env, reg);
+
if (release_dynptr)
err = unmark_stack_slots_dynptr(env, reg);
else if (convert_rcu)
@@ -13534,6 +13677,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
return id;
regs[BPF_REG_0].id = id;
+ /* the unwinder releases the pointer the call returns */
+ err = bpf_frame_desc_record_acquire(env, env->cur_state->acquired_refs - 1,
+ BPF_REG_0, true, false);
+ if (err)
+ return err;
+ } else {
+ /* the same call may acquire on another visit, see frame_desc.c */
+ bpf_frame_desc_record_none(env, ®s[BPF_REG_0]);
}
if (func_id == BPF_FUNC_dynptr_data)
@@ -14931,8 +15082,11 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
if (err)
return err;
- if (!in_rcu_cs(env))
- invalidate_rcu_protected_refs(env);
+ if (!in_rcu_cs(env)) {
+ err = invalidate_rcu_protected_refs(env);
+ if (err)
+ return err;
+ }
}
return 0;
}
@@ -14958,14 +15112,15 @@ static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state
return 0;
}
-static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id)
+static int ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id)
{
struct bpf_func_state *unused;
struct bpf_reg_state *reg;
int err;
err = release_reference_nomark(env, id);
- WARN_ON_ONCE(err);
+ if (err)
+ return err;
bpf_for_each_reg_in_vstate(env->cur_state, unused, reg, ({
if (reg->id == id) {
@@ -14974,7 +15129,7 @@ static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id)
}
}));
- return;
+ return 0;
}
/* Implementation details:
@@ -16001,11 +16156,24 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
static int check_return_code(struct bpf_verifier_env *env, int regno, const char *reg_name);
+/*
+ * A tracing program may run in NMI context, where the destructors of some
+ * fields cannot run: bpf_obj_drop() is refused there on a type that has them.
+ */
+bool bpf_nmi_unsafe_prog(struct bpf_verifier_env *env)
+{
+ enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
+
+ return is_tracing_prog_type(prog_type) ||
+ /* is_tracing_prog_type() for now doesn't cover non-iterator tracing progs. */
+ (prog_type == BPF_PROG_TYPE_TRACING &&
+ env->prog->expected_attach_type != BPF_TRACE_ITER && !env->prog->sleepable);
+}
+
static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
int *insn_idx_p)
{
bool sleepable, rcu_lock, rcu_unlock, preempt_disable, preempt_enable;
- enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
struct bpf_reg_state *regs = cur_regs(env);
const char *func_name, *ptr_type_name;
const struct btf_type *t, *ptr_type;
@@ -16123,10 +16291,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
return err;
if ((is_bpf_obj_drop_kfunc(meta.func_id) ||
- is_bpf_percpu_obj_drop_kfunc(meta.func_id)) && (is_tracing_prog_type(prog_type) ||
- /* is_tracing_prog_type() for now doesn't cover non-iterator tracing progs. */
- (prog_type == BPF_PROG_TYPE_TRACING && env->prog->expected_attach_type != BPF_TRACE_ITER
- && !env->prog->sleepable))) {
+ is_bpf_percpu_obj_drop_kfunc(meta.func_id)) && bpf_nmi_unsafe_prog(env)) {
struct btf_struct_meta *struct_meta;
struct_meta = btf_find_struct_meta(meta.arg_btf, meta.arg_btf_id);
@@ -16183,6 +16348,13 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
preempt_disable = is_kfunc_bpf_preempt_disable(&meta);
preempt_enable = is_kfunc_bpf_preempt_enable(&meta);
+ /* the descriptors may count the sections of the frame in a hidden slot */
+ if (rcu_lock || rcu_unlock || preempt_disable || preempt_enable) {
+ err = bpf_frame_desc_record_section(env, rcu_lock || rcu_unlock ? 0 : 1,
+ rcu_lock || preempt_disable ? 1 : -1);
+ if (err)
+ return err;
+ }
if (rcu_lock) {
env->cur_state->active_rcu_locks++;
bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_RCU, true,
@@ -16199,8 +16371,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
release_context_state(env->cur_state);
bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_RCU, false,
env->cur_state->active_rcu_locks);
- if (!in_rcu_cs(env))
- invalidate_rcu_protected_refs(env);
+ if (!in_rcu_cs(env)) {
+ err = invalidate_rcu_protected_refs(env);
+ if (err)
+ return err;
+ }
} else if (preempt_disable) {
env->cur_state->active_preempt_locks++;
bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_PREEMPT, true,
@@ -16217,8 +16392,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
release_context_state(env->cur_state);
bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_PREEMPT, false,
env->cur_state->active_preempt_locks);
- if (!in_rcu_cs(env))
- invalidate_rcu_protected_refs(env);
+ if (!in_rcu_cs(env)) {
+ err = invalidate_rcu_protected_refs(env);
+ if (err)
+ return err;
+ }
}
if (in_rbtree_lock_required_cb(env) && (rcu_lock || rcu_unlock)) {
@@ -16247,7 +16425,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
id = regs[BPF_REG_2].id;
insn_aux->insert_off = regs[BPF_REG_2].var_off.value;
insn_aux->kptr_struct_meta = btf_find_struct_meta(meta.arg_btf, meta.arg_btf_id);
- ref_convert_owning_non_owning(env, id);
+ /* the node goes to the list or the tree: the second argument is released */
+ set_release_reg(env, ®s[BPF_REG_2]);
+ err = ref_convert_owning_non_owning(env, id);
+ if (err)
+ return err;
}
if ((is_kfunc_call(&meta, special_kfunc_list[KF_bpf_dynptr_slice]) ||
@@ -16461,6 +16643,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (id < 0)
return id;
regs[BPF_REG_0].id = id;
+ /* the unwinder releases the pointer the call returns */
+ err = bpf_frame_desc_record_acquire(env, env->cur_state->acquired_refs - 1,
+ BPF_REG_0, true, false);
+ if (err)
+ return err;
} else if (is_rbtree_node_type(ptr_type) || is_list_node_type(ptr_type)) {
ref_set_non_owning(env, ®s[BPF_REG_0]);
}
@@ -20864,10 +21051,12 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
* prepare_func_exit to handle the case when
* state->curframe > 0, it may be a callback function,
* for which reference_state must match caller reference
- * state when it exits.
+ * state when it exits. An unwinder that releases resources
+ * leaves locks and sections too.
*/
int err = check_resource_leak(env, exception_exit,
- exception_exit || !env->cur_state->curframe,
+ exception_exit ? !bpf_jit_supports_exception_cleanup() :
+ !env->cur_state->curframe,
exception_exit ? "bpf_throw" :
"BPF_EXIT instruction in main prog");
if (err)
@@ -21155,7 +21344,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
insn->imm != BPF_FUNC_spin_unlock &&
insn->imm != BPF_FUNC_kptr_xchg) ||
(insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
- !kfunc_spin_allowed(env, insn->imm, insn->off)))) {
+ !kfunc_spin_allowed(env, insn->imm, insn->off) &&
+ !(bpf_is_throw_kfunc(insn) && bpf_jit_supports_exception_cleanup())))) {
verbose(env,
"function calls are not allowed while holding a lock\n");
bpf_diag_ctx_active(
@@ -23335,6 +23525,12 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
goto out;
aux->ctx_arg_info[i].ref_id = ret;
+ /* the kernel passed it: the main program holds it from its entry */
+ if (!ret)
+ continue;
+ ret = bpf_frame_desc_record_ctx_arg(env, state->acquired_refs - 1, i);
+ if (ret)
+ goto out;
}
}
@@ -23475,6 +23671,7 @@ static int do_check_all(struct bpf_verifier_env *env)
bpf_vlog_reset(&env->log, min(log_pos, env->log.end_pos));
for (i = 0; i < env->prog->len; i++)
env->insn_aux_data[i].seen = 0;
+ bpf_frame_desc_reset(env);
for (i = 1; env->prog->aux->func_info && i < env->subprog_cnt; i++) {
sub_aux = subprog_aux(env, i);
memset(sub_aux->called, 0, sizeof(sub_aux->called));
@@ -23544,6 +23741,11 @@ static void print_verification_stats(struct bpf_verifier_env *env)
env->insn_processed,
env->max_states_per_insn, env->total_states,
env->peak_states);
+ if (env->log.level & BPF_LOG_STATS)
+ verbose(env, "frame descriptors: slots %u insns %u classes %u bytes %u\n",
+ env->frame_desc_slots, env->frame_desc_insns,
+ env->frame_desc_pool ? env->frame_desc_pool->resource_cnt : env->resource_cnt,
+ env->frame_desc_bytes);
}
int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
@@ -25485,6 +25687,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
ret = bpf_check_cfg(env);
if (ret < 0)
goto skip_full_check;
+ /*
+ * A program that may throw keeps what its frames hold in slots the
+ * unwinder releases, see frame_desc.c, where the architecture has the
+ * entry the unwinder needs; elsewhere it throws holding nothing.
+ */
+ for (i = 0; bpf_jit_supports_exception_cleanup() && i < env->subprog_cnt; i++)
+ env->describe_frames |= env->subprog_info[i].might_throw;
ret = check_unwind_insns(env);
if (ret < 0)
@@ -25600,6 +25809,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = check_max_stack_depth(env);
+ /*
+ * The pass has seen what every frame holds and the stack is final:
+ * reserve the slots the frames keep their handles in and emit the insns
+ * that keep them up to date, with the insns still those the main pass
+ * recorded.
+ */
+ if (ret == 0)
+ ret = bpf_frame_desc_resolve(env);
+ if (ret == 0)
+ ret = bpf_frame_desc_emit(env);
+
+ /* Account for the slots reserved by frame descriptor resolution. */
+ if (ret == 0)
+ ret = check_max_stack_depth(env);
+
/* instruction rewrites happen after this point */
if (ret == 0)
ret = bpf_optimize_bpf_loop(env);
@@ -25641,6 +25865,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
: false;
}
+ if (ret == 0)
+ ret = bpf_frame_desc_build(env);
+
if (ret == 0)
ret = bpf_fixup_call_args(env);
@@ -25756,6 +25983,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kfree(env->func_sets[i]);
kfree(env->func_sets);
kvfree(env->stack_inits);
+ bpf_frame_desc_env_free(env);
bpf_diag_free(env);
kvfree(env->idoms);
kvfree(env);
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index 22503cf62e9f..f02882224a03 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -133,6 +133,7 @@ __noinline static int subprog_lock(struct __sk_buff *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("function calls are not allowed while holding a lock")
int reject_with_lock(void *ctx)
{
@@ -142,6 +143,7 @@ int reject_with_lock(void *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("function calls are not allowed while holding a lock")
int reject_subprog_with_lock(void *ctx)
{
@@ -149,6 +151,7 @@ int reject_subprog_with_lock(void *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("bpf_throw cannot be used inside bpf_rcu_read_lock-ed region")
int reject_with_rcu_read_lock(void *ctx)
{
@@ -165,6 +168,7 @@ __noinline static int throwing_subprog(struct __sk_buff *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("bpf_throw cannot be used inside bpf_rcu_read_lock-ed region")
int reject_subprog_with_rcu_read_lock(void *ctx)
{
@@ -181,7 +185,8 @@ static bool rbless(struct bpf_rb_node *n1, const struct bpf_rb_node *n2)
}
SEC("?tc")
-__failure __msg("function calls are not allowed while holding a lock")
+__arch_x86_64 __arch_arm64
+__failure __msg("cannot be called from callback subprog")
int reject_with_rbtree_add_throw(void *ctx)
{
struct foo *f;
@@ -196,6 +201,7 @@ int reject_with_rbtree_add_throw(void *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("Unreleased reference")
int reject_with_reference(void *ctx)
{
@@ -216,6 +222,7 @@ __noinline int global_subprog_may_throw(struct __sk_buff *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("Unreleased reference")
int reject_global_subprog_throw_with_reference(struct __sk_buff *ctx)
{
@@ -248,7 +255,8 @@ __noinline static int subprog_cb_ref(u32 i, void *ctx)
}
SEC("?tc")
-__failure __msg("Unreleased reference")
+__arch_x86_64 __arch_arm64
+__failure __msg("cannot be called from callback subprog")
int reject_with_cb_reference(void *ctx)
{
struct foo *f;
@@ -270,6 +278,7 @@ int reject_with_cb(void *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("Unreleased reference")
int reject_with_subprog_reference(void *ctx)
{
@@ -417,6 +426,7 @@ __noinline static int rcu_lock_then_throw(void)
}
SEC("?tc")
+__arch_s390x
__failure __msg("bpf_throw cannot be used inside bpf_rcu_read_lock-ed region")
int reject_subprog_rcu_lock_throw(void *ctx)
{
@@ -425,6 +435,7 @@ int reject_subprog_rcu_lock_throw(void *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("bpf_throw cannot be used inside bpf_preempt_disable-ed region")
int reject_subprog_throw_preempt_lock(void *ctx)
{
@@ -435,6 +446,7 @@ int reject_subprog_throw_preempt_lock(void *ctx)
}
SEC("?tc")
+__arch_s390x
__failure __msg("bpf_throw cannot be used inside bpf_local_irq_save-ed region")
int reject_subprog_throw_irq_lock(void *ctx)
{
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots
2026-11-11 12:57 ` [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:12 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 12623 bytes --]
> bpf: Track held resources in hidden stack slots
TL;DR:
- Two aliasing references may be double-released by bpf_throw() unwinding
(possible refcount underflow/use-after-free)
- Programs using callx or cross-frame resources may fail to load, breaking
bisection
- Comment and message nits
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> new file mode 100644
> index 000000000000..b0db3feeb9b0
> --- /dev/null
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> +int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
> +{
[ ... ]
> + for (i = 0; i < cnt; i++) {
> + r = &env->res_rels[i];
> + if (r->insn != insn || r->reg != regno)
> + continue;
> + /* the argument is taken the same way whatever it releases */
> + if (r->flags != flags || r->adj != adj) {
> + verbose(env, "frame_desc: insn %d takes the handle of what it releases differently on different paths\n",
> + insn);
> + return -EINVAL;
> + }
> + if (r->acq == ref->acq - 1)
> + return 0;
> + }
Can this lead to a double release of an object that has two references?
bpf_frame_desc_record_release() links a releasing insn only to the
acquisitions it was seen releasing on some verified path, and
emit_release() compares the handle against those slots in recording order
and clears only the first slot that matches. Its comment says this is
enough because "two references to one object have equal handles in a slot
each".
That does not hold when one frame holds two references to the same object
from two different acquiring insns, A and B. Two acquire kfunc calls on
the same kernel object do this, and so does bpf_obj_new() followed by
bpf_refcount_acquire(). Both slots then hold the same pointer.
Suppose one release insn Y releases A on one path and B on another, so Y
has rel entries for both. At runtime Y can clear A's slot when the
verifier's path released B, because the handles are equal. If the
remaining reference is then released by an insn that was verified only on
paths releasing A, that insn's rel list contains only A. It compares A's
already-cleared slot, finds no match and leaves B's slot holding the
pointer.
After that nothing the verifier tracks is still held, but B's hidden slot
is non-NULL. bpf_unwind_release_frame() releases every non-NULL slot:
if (!handle || ...) continue;
...
bpf_unwind_release(res, handle);
so a later bpf_throw() runs the destructor (put_task_struct(),
bpf_obj_drop_impl(), ...) again for an object whose references were
already dropped. That looks like a refcount underflow and a use-after-free.
The verifier appears to accept such a program: the two paths keep
different reference ids at Y, so they are not pruned against each other,
and a branch whose condition is already known lets each path release the
remaining reference at a different insn.
Would it work for every release insn to compare all acquisitions of the
same class in the frame, to reject a release insn that may release two
acquisitions whose handles can alias, or to clear every matching slot
(which turns the double release into a leak)?
The later commit 026679e2eda5 ("Describe held resources at throw sites")
keeps hidden slots for resources whose location differs between visits of
a site, and the unwinder still releases every non-NULL hidden slot, so this
does not look fixed in the rest of the series.
[ ... ]
> +int bpf_frame_desc_emit(struct bpf_verifier_env *env)
> +{
[ ... ]
> + for (i = 0; i < n && !ret; i = k) {
> + u32 idx = items[i].idx;
> +
> + for (k = i; k < n && items[k].idx == idx; k++)
> + ;
> + if (verifier_bug_if(idx >= len || env->prog->insnsi[idx].code != (BPF_JMP | BPF_CALL),
> + env, "resource kept at insn %d, not a call", idx)) {
> + ret = -EFAULT;
> + break;
> + }
Can a callx reach this check?
bpf_frame_desc_record_return() makes the call site the acquisition of a
reference that a subprog returns in R0:
acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
The call site can be a callx. check_func_callx() ->
check_func_callx_one() -> check_static_func_call() -> setup_func_entry()
records the callx insn as callee->callsite, and prepare_func_exit() passes
that callsite to bpf_frame_desc_record_return(). For a function verified
on its own, bpf_flow_return() passes call_idx, which is also the callx.
Nothing in this patch rejects callx when env->describe_frames is set, and a
static subprog returning an acquired reference is legal because
check_reference_leak() returns 0 in any frame other than frame 0.
A callx has code BPF_JMP | BPF_CALL | BPF_X (see bpf_is_callx()), which is
not equal to (BPF_JMP | BPF_CALL), so verifier_bug_if() fires and the load
fails with -EFAULT. With CONFIG_DEBUG_KERNEL the BPF_WARN_ONCE() also
produces a WARN, which panics with panic_on_warn.
The trigger is a program on x86-64 or arm64, where
bpf_jit_supports_exception_cleanup() and bpf_jit_supports_callx() both
hold. Some subprog in it calls bpf_throw(), so describe_frames is set, and
another place makes a callx through a function pointer to a static subprog
that returns, say, the result of bpf_task_acquire() or bpf_obj_new(). The
throw does not need to be reached with anything held. Such a program
loaded before this patch and now fails with "verifier bug: resource kept
at insn N, not a call".
emit_items() itself handles the callx correctly, since it only emits
after-insns for an R0 acquisition, so only this check is too narrow.
This is fixed later in the series by acd08268bd12 ("bpf: Support callx
when a callee releases resources"), which also accepts
bpf_is_callx(&env->prog->insnsi[idx]) here. This commit still breaks
bisection for such programs.
> + after = emit_items(env, &items[i], k - i, buf, &before);
> + /* the call of a subprog in front of it does not move with the call */
> + ret = bpf_adj_moved_insn(&buf[before], before);
> + if (ret)
> + break;
> + /*
> + * A callee that unwound returns right after the call and goes where
> + * the unwind_goto says: nothing may come between the two.
> + */
> + if (after && idx + 1 < len && bpf_is_unwind_goto(&env->prog->insnsi[idx + 1])) {
> + verbose(env, "frame_desc: the call at insn %d acquires a resource and may unwind, which is not supported in a program that throws\n",
> + idx);
Does this message name the insn the user wrote?
idx is an index in the current image. By this point
bpf_insert_loop_guards() (and append_budget_exit()) may already have
inserted insns, such as may_goto guards of 10-14 insns each plus budget
exits, so idx can be past the insn of the loaded program. The other
frame_desc messages in this file print main-pass indices (a->insn,
env->insn_idx), which equal insn_aux_data[].orig_idx, and
bpf_patch_list_insert() also translates with
env->insn_aux_data[off].orig_idx for the same reason.
Should this use env->insn_aux_data[idx].orig_idx so that the -EINVAL
rejection points at the right call? The end of the series still prints idx.
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index cf3cfc734ed4..4d85a42a18d5 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -812,6 +813,10 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
> mark_dynptr_stack_regs(env, &bpf_stack_slot(state, spi)->spilled_ptr,
> &bpf_stack_slot(state, spi - 1)->spilled_ptr, type, parent_id);
>
> + /* a referenced dynptr is a stack object the unwinder discards at its address */
> + if (dynptr->type == BPF_DYNPTR_TYPE_INVALID && dynptr_type_referenced(type))
> + return bpf_frame_desc_record_acquire(env, env->cur_state->acquired_refs - 1,
> + reg - cur_regs(env), false, false);
> return 0;
> }
This isn't a bug, but is this comment accurate? It says the unwinder
discards the referenced dynptr "at its address", while the code does
something else.
For a STACK_DYNPTR object, acquired_class() sets BPF_RES_F_DEREF and
obj_off = spi_off(spi). emit_acquire_after() then emits
BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_FP, a->obj_off) and stores that
value in the hidden slot. That value is the first word of the dynptr, the
ringbuf record or the file reader, not the dynptr's stack address.
unwind.c then calls bpf_ringbuf_discard_proto.func((u64)(long)handle, ...)
on that handle, and the dynptr may even be overwritten later without
affecting the slot.
The comment looks copied from the iterator and IRQ-flag sites, where "at
its address, which the call takes" is accurate because the slot holds the
object's address. acquired_class() describes the dynptr case correctly
("The handle of a dynptr is the record or the reader, the first word of the
dynptr at @obj_off"). Could this comment say that the slot holds the
record or reader loaded from the dynptr after the call?
[ ... ]
> @@ -12702,9 +12818,26 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
> invalidate_stack_slices(env, callee->frameno, INT_MIN, 0);
>
> /* What the frame holds when it returns is the caller's to release. */
> - for (i = 0; i < state->acquired_refs; i++)
> - if (state->refs[i].frameno == callee->frameno)
> - state->refs[i].frameno = caller->frameno;
> + err = bpf_frame_desc_record_exit(env);
> + if (err)
> + return err;
Do programs that load at the parent commit get rejected here even when no
bpf_throw() can be reached while the resource is held?
On x86-64 with CONFIG_UNWINDER_ORC and on arm64,
bpf_jit_supports_exception_cleanup() is true, so bpf_check() sets
env->describe_frames for any program where some subprog has might_throw,
which includes any program using bpf_assert() anywhere. The new calls then
reject cross-frame resources wherever they occur, not only at a throw:
- bpf_frame_desc_record_exit(), called from prepare_func_exit() and
exit_func_on_its_own(), fails with "frame_desc: a subprog returns
holding what R0 does not carry ..." for any ref owned by the returning
frame that is not a single REF_TYPE_PTR carried in R0 (a bpf_spin_lock or
res lock, saved IRQ flags, an iterator). It also fails with "returns in
other sections" for a subprog that enters RCU or preempt-disabled
sections for its caller.
- bpf_frame_desc_record_release(), reached from process_spin_lock(),
unmark_stack_slot_irq_flag() and release_reference_nomark(), fails with
"releases what another frame holds" when ref->frameno != st->curframe.
- bpf_frame_desc_record_acquire(), reached from mark_stack_slots_iter(),
mark_stack_slot_irq_flag() and mark_stack_slots_dynptr(), returns -EXDEV
("creates an object in the stack of another frame") for an iterator, IRQ
flags or a ringbuf/file dynptr created through a pointer to the caller's
stack.
For example, this loads before the patch and is rejected after it:
static __noinline void lock_it(struct val *v) { bpf_spin_lock(&v->lock); }
static __noinline void unlock_it(struct val *v) { bpf_spin_unlock(&v->lock); }
SEC("tc") int prog(struct __sk_buff *ctx) {
struct val *v = bpf_map_lookup_elem(&m, &(int){0});
if (!v) return 0;
lock_it(v); v->cnt++; unlock_it(v);
bpf_assert(ctx->len > 0);
return 0;
}
In lock_it(), process_spin_lock() -> bpf_frame_desc_record_acquire() gives
the REF_TYPE_LOCK entry a non-zero ->res with frameno 1. At its BPF_EXIT,
bpf_frame_desc_record_exit() returns -EINVAL because the type is not
REF_TYPE_PTR. The throw is never reached with the lock held, and the
verifier guide says a BPF_EXIT from a static subprog with a lock held is
allowed.
The commit message lists these restrictions and says "Later patches lift
most of the first five". The ringbuf/file dynptr case is not in that list.
The fixes come later in the series: 99e07a877ad7 ("bpf: Allow a callee to
leave resources with its caller") and 0bbb9cb47a2c ("bpf: Allow a callee to
release its caller's resources"). 99e07a877ad7 itself says "These programs
are rejected even if no throw can find the resource held, although they
load today." This leaves a bisection window in which existing programs
that use bpf_assert() or bpf_throw() fail to load on x86-64 and arm64.
Could the restriction apply only when a throw is reached with the resource
held, or could the series be reordered so that no intermediate commit
rejects programs that load today?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots
2026-11-11 12:57 ` [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 16:12 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 16:12 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 6 potential
issues to consider:
- [High] bpf: emit_entries() hides arena stack adjustment from recursion
check
- [High] bpf: RCU iterator outlives section when subprog unlocks and
throws
- [High] bpf: double-free on unwind due to wrong slot cleared in
emit_release()
- [High] bpf: non-LIFO lock unwind in bpf_frame_desc_build() corrupts
IRQ and lock state
- [High] bpf: hidden slot overwritten when subprog call returns NULL in
frame_desc
- [Medium] bpf: duplicate check_max_stack_depth() inflates recursive
frame depth
--
commit 197c825766b69a79fd4f8ec64eb1bed1f952135c
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Track held resources in hidden stack slots
This commit tracks held kernel resources, locks, and RCU or preemption
sections in hidden stack slots and builds per-subprogram frame descriptors so
bpf_throw() can release them during unwinding.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> new file mode 100644
> index 0000000000000..b0db3feeb9b0f
> --- /dev/null
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> +int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
> + int idx, int callsite, const struct bpf_reg_state *r0)
> +{
> + struct bpf_reference_state *ref;
> + s16 adj = 0;
> + int acq;
> +
> + if (!env->describe_frames || st->speculative)
> + return 0;
> + ref = idx < 0 ? NULL : &st->refs[idx];
> + if (!ref || !ref->res) {
> + if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
> + env->insn_aux_data[callsite].res_none = true;
> + return 0;
> + }
[Severity: High]
Can a second call at the same callsite that returns NULL overwrite a live
handle in the callsite's hidden stack slot?
emit_acquire_after() unconditionally stores R0 into the callsite's slot
after every return. If a subprog can return either an acquired reference or
NULL (R0 == 0, idx < 0), and the caller invokes that callsite a second time
while still holding the reference from the first call,
bpf_frame_desc_record_return() returns 0 without checking whether st->refs[]
already holds a reference from that callsite. At runtime, the second return
writes 0 into the hidden slot, so a later bpf_throw() sees NULL and leaks the
object from the first call.
[ ... ]
> +static int emit_release(struct bpf_verifier_env *env, struct bpf_insn *buf,
> + const struct res_item *it, int n)
> +{
> + const struct bpf_res_rel *r = &env->res_rels[it->rec];
> + int i, cnt = 0, reg = r->reg, per = r->adj ? 5 : 4;
> +
> + if (r->flags & BPF_RES_F_DEREF) {
> + buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, r->reg, 0);
> + reg = BPF_REG_0;
> + }
> + for (i = 0; i < n; i++) {
> + s16 off = env->res_acqs[env->res_rels[it[i].rec].acq].off;
> + bool last = i == n - 1;
> +
> + buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_FP, off);
> + if (r->adj)
> + buf[cnt++] = BPF_ALU64_IMM(BPF_ADD, BPF_REG_AX, r->adj);
> + buf[cnt++] = BPF_JMP_REG(BPF_JNE, BPF_REG_AX, reg, last ? 1 : 2);
> + buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, off, 0);
> + /* past the slots that follow: the last one has no jump */
> + if (!last)
> + buf[cnt++] = BPF_JMP_A((n - 1 - i) * per - 1);
> + }
> + return cnt;
> +}
[Severity: High]
Can emit_release() clear the wrong hidden slot when a frame holds two
references to the same underlying pointer?
Suppose a frame acquires a = bpf_obj_new(...) (acq 0, slot0) and
b = bpf_refcount_acquire(a) (acq 1, slot1), so both slots hold the same
pointer value p. If a first release call R_A takes a on one path and b on
another, R_A is recorded with both acq 0 and acq 1. On the path that passes
b to R_A and then passes a to a second release call R_B, the verifier removes
b's ref at R_A and records only acq 0 at R_B.
At runtime on that path, R_A checks slot0 first, sees slot0 == p, clears
slot0, and jumps past slot1. R_B then only checks slot0 (already 0), leaving
p in slot1 after both references have been dropped, so a subsequent
bpf_throw() releases p a third time and causes a double free or refcount
underflow.
[ ... ]
> +static int emit_entries(struct bpf_verifier_env *env)
> +{
[ ... ]
> + for (k = 0; k < ARRAY_SIZE(sub->section_off); k++)
> + if (sub->section_off[k])
> + buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, sub->section_off[k], 0);
> + if (!cnt)
> + continue;
> + ret = bpf_patch_list_insert(env, sub->start, buf, cnt);
> + env->frame_desc_insns += cnt;
> + }
> + ret = ret ?: bpf_patch_list_commit(env);
[Severity: High]
Does inserting slot-zeroing instructions at sub->start hide a recursive
subprog's arena stack adjustment from bpf_insert_recursion_checks()?
bpf_frame_desc_emit() runs before bpf_insert_recursion_checks() and commits
the prologue stores at sub->start, which moves sub->start in front of the
subprog's leading is_arena_sp_adj() instruction:
kernel/bpf/fixups.c:bpf_insert_recursion_checks() {
...
/* The exit drops the frame in the stack in arena. Take it before the check. */
if (is_arena_sp_adj(&env->prog->insnsi[si[s].start])) {
buf[n++] = env->prog->insnsi[si[s].start];
...
}
Because si[s].start is now a BPF_ST_MEM instruction instead of
is_arena_sp_adj(), bpf_insert_recursion_checks() does not hoist the arena
frame allocation ahead of the recursion check's bailout exit (and also looks
up live_regs_before at the wrong offset). bpf_insert_arena_stack() still
prepends r12 -= arena_stack_depth to every BPF_EXIT in the subprog, including
the bailout exit, so hitting the recursion or may_goto budget check decrements
BPF_REG_ARENA_SP without the matching entry increment and corrupts the
caller's arena stack pointer.
[ ... ]
> +int bpf_frame_desc_build(struct bpf_verifier_env *env)
> +{
[ ... ]
> + for (i = 0; i < 2 * env->res_acq_cnt; i++) {
> + const struct bpf_res_acq *a = &env->res_acqs[i % env->res_acq_cnt];
> + bool first = i < env->res_acq_cnt;
> +
> + if (acq_subprog(env, a, new_of) != s ||
> + resource_goes_first(&env->resources[a->res]) != first)
> + continue;
> + d->ent[d->nent++] = (struct bpf_frame_desc_entry) { .res = a->res, .off = a->off };
> + }
[Severity: High]
Should bpf_frame_desc_build() emit lock and IRQ flag entries in reverse
acquisition order?
d->ent[] is populated in forward env->res_acqs[] order, and
bpf_unwind_release_frame() walks d->ent[] forward during
BPF_UNWIND_PHASE_LOCKS. If a frame entered with interrupts enabled calls
bpf_spin_lock() (saving IF=1 in per-CPU irqsave_flags and disabling IRQs),
then calls bpf_res_spin_lock_irqsave() (saving IF=0 into the stack flag slot),
and throws, pass 0 of BPF_UNWIND_PHASE_LOCKS first runs bpf_spin_unlock(),
which restores IF=1 while the resilient spinlock is still held, and then runs
bpf_local_irq_restore() on the stack slot (since arch_irqs_disabled_flags()
is true, late is false), leaving local CPU interrupts disabled during sleeping
destructors in BPF_UNWIND_PHASE_OBJS and after bpf_throw() returns.
When a frame holds two nested resilient spinlocks, unlocking them in forward
order also pops the wrong entry in res_spin_unlock() (which clears
rqh->locks[rqh->cnt - 1]), corrupting the per-CPU held-lock stack.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index cf3cfc734ed4d..4d85a42a18d5f 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -11232,21 +11321,34 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env)
> }));
> }
>
> -static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
> +static int invalidate_rcu_protected_refs(struct bpf_verifier_env *env)
> {
> struct bpf_stack_state *stack;
> struct bpf_func_state *state;
> struct bpf_reg_state *reg;
> u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER);
> + bool iter = false;
>
> bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({
> if (reg->type & MEM_RCU) {
> + iter |= stack && stack->slot_type[BPF_REG_SIZE - 1] == STACK_ITER;
> bpf_diag_mod_begin(env, reg, NULL, BPF_DIAG_MOD_WRITE);
> reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL | NON_OWN_REF);
> reg->type |= PTR_UNTRUSTED;
> bpf_diag_mod_end(env);
> }
> }));
> + /*
> + * Nothing may destroy an RCU protected iterator once its section ended:
> + * not the program, which is refused when it leaks one, and not the
> + * unwinder, which destroys what a frame holds at a throw, see
> + * frame_desc.c.
> + */
> + if (iter && env->describe_frames) {
> + verbose(env, "an RCU protected iterator may not outlive its section in a program that throws\n");
> + return -EINVAL;
> + }
> + return 0;
> }
[Severity: High]
Does this miss RCU-protected iterators on caller frames when a subprog is
verified on its own and calls bpf_rcu_read_unlock() followed by bpf_throw()?
When call_func_on_its_own() verifies a subprog, the caller frames are not in
env->cur_state->frame[], so bpf_for_each_reg_in_vstate_mask() does not see a
caller's STACK_ITER slot (while env->cur_state->refs[] carries the inherited
iterator reference with frameno == -1). If the subprog then calls
bpf_throw() instead of returning normally, process_bpf_exit_full() skips
exit_func_on_its_own(), so bpf_frame_desc_record_exit() never checks
rcu_released_below and bpf_flow_return() never sets releases_inherited. The
subprog can then drop the caller's RCU lock (and sleep in a sleepable prog)
before throwing, causing the unwinder to destroy the caller's RCU-protected
iterator after its RCU section has ended.
[ ... ]
> @@ -25600,6 +25809,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
> if (ret == 0)
> ret = check_max_stack_depth(env);
>
> + /*
> + * The pass has seen what every frame holds and the stack is final:
> + * reserve the slots the frames keep their handles in and emit the insns
> + * that keep them up to date, with the insns still those the main pass
> + * recorded.
> + */
> + if (ret == 0)
> + ret = bpf_frame_desc_resolve(env);
> + if (ret == 0)
> + ret = bpf_frame_desc_emit(env);
> +
> + /* Account for the slots reserved by frame descriptor resolution. */
> + if (ret == 0)
> + ret = check_max_stack_depth(env);
[Severity: Medium]
Does calling check_max_stack_depth() both before and after
bpf_frame_desc_resolve() inflate stack_depth for recursive subprogs that use
the arena stack?
During the first check_max_stack_depth() call, pad_recursive_frames()
overwrites si[s].stack_depth in place with max(si[s].stack_depth, frame).
bpf_frame_desc_resolve() -> bpf_reserve_stack_slots() then reserves hidden
slots on top of that already-padded stack_depth instead of the verified stack
depth, which can push stack_depth past env->stack_limit on the second
check_max_stack_depth() call and cause false verifier rejections.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=13
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 14/29] bpf: Describe held resources at throw sites
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (12 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 13/29] bpf: Track held resources in hidden stack slots Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 12:57 ` [PATCH bpf-next v1 15/29] bpf: Describe handles in registers " Kumar Kartikeya Dwivedi
` (15 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently every call that acquires a resource in a program that throws
keeps the handle in a hidden slot of its frame. This costs a stack slot,
a store on acquire, and a compare and a store on release, whether or not
a throw can find the resource held.
A frame can only be unwound at a call it has not returned from:
bpf_throw(), or a call to a subprog that may throw. Such a call is a
site. Often the frame already keeps its resources on its own stack at a
site, in the same location on every path:
- an iterator and saved IRQ flags are objects on the stack;
- a reference is spilled, or is the first word of a dynptr;
- the frame is in the same number of RCU read and preemption disabled
sections.
Add a descriptor per site for these. On every visit to a site, the main
pass records where each frame of the state keeps its resources:
- at bpf_throw() and at a call to a global subprog that may throw: for
the frame that makes the call and for each of its callers, at the
call it has not returned from;
- at a call to a subprog that is verified on its own and may throw: for
the frames of the caller. Such a subprog has no pointer into their
stack, so its throw finds them as they are at the call.
After the pass, each acquisition is one of three cases:
- No site finds the resource held: nothing is kept for it.
- Every site that finds it held does so on every visit, with the handle
in the same location: the site descriptors give that location.
- Otherwise the handle is kept in a hidden slot, as before.
Sections work the same way. A subprog counts its sections in a hidden
slot only if the visits to one of its sites disagree on the number of
sections. Otherwise the site descriptor gives the count, which is
negative for a callee that left a section of its caller.
A referenced argument of a struct_ops program is described like any
other reference. Its hidden slot, and the load and the store at program
entry that fill it, remain only for an argument that some site cannot
describe. Locks always take a hidden slot when a site finds them held.
Two things limit which stack slots a site can describe:
- The verifier forgets the contents of a dead slot where paths join, see
bpf_clean_verifier_state(). A reference spilled in a slot that is not
read again would then be unknown at the site, and nothing is read
after bpf_throw(). So a slot that holds a pointer to a resource held
by its frame is kept even when dead: it stays in the state, and a
state can stand for another only if both have the same contents in
it. Stack liveness itself does not change.
- The slots that the spills of a bpf_fastcall pattern go to hold
nothing at runtime once bpf_remove_fastcall_spills_fills() removes
the spills. They are never described.
At runtime, each JITed function has a table of its sites that have a
descriptor, sorted by native offset. The verifier builds the table from
the instruction index of each call when the function is split off for
the JIT, after constant blinding has moved the instructions. The x86-64
and arm64 JITs fill in the native range of each call. The unwinder looks
up the return address of a frame, minus one, and releases the resources
listed in the site descriptor together with those in the hidden slots of
the function.
A descriptor entry now says what kind of location it describes: a slot
or a spill that holds a pointer, possibly into the object, a stack
object, a slot that counts sections, or a constant count. Iterators and
file readers are flagged to be released before the references of their
frame, in both descriptors of the frame.
The rejections for a program that throws do not change, with one
exception: an instruction that acquires a resource only on some paths is
rejected only if the resource needs a hidden slot.
A subprog that calls through a function pointer it was passed can throw
without containing a throw or taking the address of a function. Mark
every subprog that contains a callx as one that may throw if any function
whose address is taken may throw, so that the frames of its callers are
visited at the call.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
arch/arm64/net/bpf_jit_comp.c | 1 +
arch/x86/net/bpf_jit_comp.c | 4 +-
include/linux/bpf.h | 64 +++-
include/linux/bpf_verifier.h | 61 +++-
kernel/bpf/cfg.c | 23 +-
kernel/bpf/core.c | 1 +
kernel/bpf/fixups.c | 8 +-
kernel/bpf/frame_desc.c | 603 +++++++++++++++++++++++++++-------
kernel/bpf/join.c | 3 +-
kernel/bpf/states.c | 8 +-
kernel/bpf/unwind.c | 156 ++++++---
kernel/bpf/verifier.c | 22 +-
12 files changed, 780 insertions(+), 174 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 418bda24020c..801b507a4644 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2902,6 +2902,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
* reasons, expects to point to the next instruction)
*/
bpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);
+ bpf_prog_fill_frame_desc_sites(prog, ctx.offset, ctx.ro_image);
out_off:
if (!ro_header && priv_stack_ptr) {
free_percpu(priv_stack_ptr);
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 9170cbe3fa16..2c2d571b21c4 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -4932,8 +4932,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
}
if (!image || !prog->is_func || extra_pass) {
- if (image)
+ if (image) {
bpf_prog_fill_jited_linfo(prog, addrs + 1);
+ bpf_prog_fill_frame_desc_sites(prog, addrs, image);
+ }
out_addrs:
kvfree(addrs);
if (!image && priv_stack_ptr) {
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index e7cce9080531..e8eda32bc05d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1864,29 +1864,66 @@ struct bpf_resource {
};
/*
- * The slot holds a count of sections instead of a handle, signed: the
- * sections the frame entered less those of its callers it left.
+ * What an entry of a frame descriptor names, relative to the frame pointer:
+ *
+ * - by default a location that holds the handle of a resource of the class,
+ * or 0 when the frame holds none. The handle is what the release takes: the
+ * pointer to an object, to a ringbuf record or to a lock, or the address of
+ * an iterator or of saved IRQ flags on the stack. The location is a hidden
+ * slot of the frame, or a slot of the program's own stack that holds the
+ * pointer whenever the frame is at the site of the descriptor; the pointer
+ * may then point adj bytes into the object;
+ * - with BPF_FRAME_ENT_OBJ the resource itself, an iterator or saved IRQ
+ * flags the frame has on its stack whenever it is at the site;
+ * - with BPF_FRAME_ENT_COUNTER a hidden slot that holds a count of sections,
+ * signed: the sections the frame entered less those of its callers it left;
+ * - with BPF_FRAME_ENT_COUNT no location: the count of the sections the frame
+ * is in whenever it is at the site.
*/
#define BPF_FRAME_ENT_COUNTER BIT(0)
+#define BPF_FRAME_ENT_OBJ BIT(1)
+#define BPF_FRAME_ENT_COUNT BIT(2)
+/* Released before the other entries of the frame, which it may be using. */
+#define BPF_FRAME_ENT_FIRST BIT(3)
-/*
- * One hidden slot of a frame: it holds the handle of a resource of the class
- * the frame holds, or 0. The handle is what the release takes: the pointer
- * to an object, to a ringbuf record or to a lock, or the address of an
- * iterator or of saved IRQ flags on the stack of some frame.
- */
struct bpf_frame_desc_entry {
u16 res; /* index into the program's resource classes */
- s16 off; /* the slot, relative to the frame pointer */
- u16 flags;
+ u16 flags; /* BPF_FRAME_ENT_* */
+ union {
+ s16 off; /* the location, relative to the frame pointer */
+ s16 count; /* BPF_FRAME_ENT_COUNT */
+ };
+ s16 adj;
};
+/*
+ * A frame has two descriptors: the one of its function lists its hidden
+ * slots, and the one of the site it is at, the call a throw found it in, what
+ * the verifier found in the same place on every path to the site.
+ */
struct bpf_frame_desc {
u32 size; /* total size in bytes, including this header */
u32 nent; /* number of entries that follow */
struct bpf_frame_desc_entry ent[];
};
+/* A call of a JITed function that a frame may be unwound at. */
+struct bpf_frame_desc_site {
+ u32 off; /* native offset of the first byte of the call from prog->bpf_func */
+ u32 end; /* native offset one past its last byte */
+ u32 desc; /* index into the descriptor pool of the main program */
+};
+
+/*
+ * The sites of one JITed function, sorted by off. Until the JIT fills it, off
+ * holds the index of the instruction in the function and end is unused.
+ */
+struct bpf_frame_desc_sites {
+ u32 cnt;
+ bool filled;
+ struct bpf_frame_desc_site site[];
+};
+
/* Owned by the main program and shared by its subprograms. */
struct bpf_frame_desc_pool {
u32 cnt;
@@ -2127,6 +2164,8 @@ struct bpf_prog_aux {
*/
u32 frame_desc;
struct bpf_frame_desc_pool *frame_desc_pool;
+ /* The sites of this JITed function that have a descriptor, freed with it. */
+ struct bpf_frame_desc_sites *frame_desc_sites;
union {
struct work_struct work;
struct rcu_head rcu;
@@ -4589,6 +4628,7 @@ void bpf_insn_array_remap(struct bpf_map *map, const u32 *new_off, const u32 *ne
#ifdef CONFIG_BPF_SYSCALL
void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image);
+void bpf_prog_fill_frame_desc_sites(struct bpf_prog *prog, const u32 *offsets, void *image);
void bpf_unwind_release(const struct bpf_resource *res, void *handle);
void bpf_dynptr_file_free(void *reader);
void bpf_unwind_release_frames(void);
@@ -4598,6 +4638,10 @@ static inline void
bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
{
}
+static inline void
+bpf_prog_fill_frame_desc_sites(struct bpf_prog *prog, const u32 *offsets, void *image)
+{
+}
static inline void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool)
{
}
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5cbac24102f4..ca50c85006d7 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1052,7 +1052,10 @@ struct bpf_subprog_info {
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
/* the frame descriptor of this subprogram in the pool of the program, plus one; see frame_desc.c */
u32 frame_desc;
- /* hidden slots: the counts of the sections the frame entered, by kind; 0 when it has none */
+ /*
+ * hidden slots: the counts of the sections the frame entered, by kind; 0
+ * when it has none: it enters none, or its sites say how many it is in
+ */
s16 section_off[2];
u16 stack_depth; /* max. stack depth used by this function */
/* bytes below the verified stack depth reserved for hidden slots, see bpf_reserve_stack_slots() */
@@ -1064,6 +1067,11 @@ struct bpf_subprog_info {
* are used for bpf_fastcall spills and fills.
*/
s16 fastcall_stack_off;
+ /*
+ * Where the longest pattern ends: its spills are removed as well, so
+ * the slots below hold nothing at run time, see frame_desc.c.
+ */
+ s16 fastcall_stack_end;
bool has_tail_call: 1;
bool might_throw: 1;
bool can_unwind: 1; /* has unwind_exit */
@@ -1247,8 +1255,10 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
#define BPF_RES_F_CTX BIT(3) /* a __ref argument the kernel passes the main program */
/*
- * A call that acquires a resource, see frame_desc.c: the handle of what it
- * acquired is kept in a hidden slot of the frame that runs the call, one slot
+ * A call that acquires a resource, see frame_desc.c. When a frame can be
+ * unwound while it holds what the call acquired, and the sites it can be
+ * unwound at do not all find the handle in the same place of its stack, the
+ * handle is kept in a hidden slot of the frame that runs the call, one slot
* per acquiring call.
*/
struct bpf_res_acq {
@@ -1259,6 +1269,41 @@ struct bpf_res_acq {
s16 adj; /* R0 points that many bytes into the object: the node of a list */
s16 obj_off; /* BPF_RES_F_DEREF: the dynptr, relative to R10; BPF_RES_F_CTX: in the context */
s16 off; /* the slot, relative to R10 */
+ bool held; /* some frame holds what it acquired at a site */
+ bool slot; /* and some site cannot say where: the handle is kept in a slot */
+};
+
+/* How many places of its stack a site remembers a frame keeping one handle in. */
+#define BPF_RES_LOC_MAX 3
+
+/*
+ * A site: a call a frame can be unwound at, bpf_throw() or the call of a
+ * subprog that may throw, and what every visit of it found of the frame that
+ * runs it. A site has a descriptor of its own for what the visits agree on,
+ * see frame_desc.c.
+ */
+struct bpf_res_site {
+ u32 insn; /* the call, as the main pass named it */
+ u32 visits;
+ s16 sections[2]; /* the sections the frame is in, RCU and preemption */
+ bool sections_vary[2]; /* the visits differ: the frame counts them in a slot */
+ u32 desc; /* its descriptor in the pool of the program, plus one */
+};
+
+/*
+ * Where the frame at a site keeps the handle of what an acquisition acquired:
+ * the slots of its stack that every visit holding it found the handle in.
+ */
+struct bpf_res_loc {
+ u32 site; /* index of the site */
+ u32 visits; /* how many visits of the site found the frame holding it */
+ u16 acq; /* index of the acquisition */
+ u8 cnt; /* the places left */
+ bool obj; /* they are the resource itself, an object on the stack */
+ struct {
+ s16 off; /* relative to R10 */
+ s16 adj; /* the pointer there points that many bytes into the object */
+ } at[BPF_RES_LOC_MAX];
};
/* A call that releases what an acquisition acquired: the slot of the acquisition is cleared. */
@@ -1319,8 +1364,13 @@ struct bpf_verifier_env {
u32 res_rel_cnt;
struct bpf_section_insn *section_insns;
u32 section_insn_cnt;
+ struct bpf_res_site *res_sites;
+ u32 res_site_cnt;
+ struct bpf_res_loc *res_locs;
+ u32 res_loc_cnt;
int release_regno; /* the argument the current call releases, or -1 */
u32 frame_desc_slots; /* hidden slots that hold handles and counts */
+ u32 frame_desc_sites; /* sites that have a descriptor */
u32 frame_desc_insns; /* instructions keeping them up to date */
u32 frame_desc_bytes; /* descriptors and resource classes */
bool describe_frames; /* frame descriptors are generated: the program may throw */
@@ -2280,11 +2330,16 @@ int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx);
int bpf_frame_desc_record_ctx_arg(struct bpf_verifier_env *env, int idx, int argno);
int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delta);
int bpf_frame_desc_record_exit(struct bpf_verifier_env *env);
+bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
+ int f, const struct bpf_reg_state *reg);
+int bpf_frame_desc_record_site(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx);
int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
int idx, int callsite, const struct bpf_reg_state *r0);
int bpf_frame_desc_resolve(struct bpf_verifier_env *env);
int bpf_frame_desc_emit(struct bpf_verifier_env *env);
int bpf_frame_desc_build(struct bpf_verifier_env *env);
+struct bpf_frame_desc_sites *bpf_frame_desc_sites(struct bpf_verifier_env *env, int subprog);
void bpf_frame_desc_reset(struct bpf_verifier_env *env);
void bpf_frame_desc_env_free(struct bpf_verifier_env *env);
int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index bbcf42eae9d9..fad8affe7fbf 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -535,22 +535,41 @@ static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
static void close_callee_effects(struct bpf_verifier_env *env)
{
struct bpf_insn *insn = env->prog->insnsi;
+ bool changed, taken_throws = false, throws;
+ struct bpf_subprog_info *sub;
struct bpf_func_ptr *ptrs;
- bool changed;
u32 cnt, j;
int t;
do {
changed = false;
+ throws = taken_throws;
for (t = 0; t < env->prog->len; t++) {
if (bpf_pseudo_call(&insn[t]) || bpf_pseudo_func(&insn[t]))
changed |= merge_callee_effects(env, t, t + insn[t].imm + 1);
+ if (bpf_pseudo_func(&insn[t]))
+ taken_throws |= bpf_find_containing_subprog(env, t + insn[t].imm + 1)->might_throw;
ptrs = insn_func_ptrs(env, t, &cnt);
- for (j = 0; j < cnt; j++)
+ for (j = 0; j < cnt; j++) {
changed |= merge_callee_effects(env, t, ptrs[j].xlated_off);
+ taken_throws |= bpf_find_containing_subprog(env, ptrs[j].xlated_off)->might_throw;
+ }
+ /*
+ * It is not known which function a callx calls: the subprog may
+ * throw when a function whose address is taken may. The marks
+ * above are for the subprog that takes the address, which need
+ * not be the one that calls through it, and the frames of the
+ * callers of the one that does are unwound, see frame_desc.c.
+ */
+ if (taken_throws && bpf_is_callx(&insn[t])) {
+ sub = bpf_find_containing_subprog(env, t);
+ changed |= !sub->might_throw;
+ sub->might_throw = true;
+ }
if (bpf_is_ldimm64(&insn[t]))
t++;
}
+ changed |= throws != taken_throws;
} while (changed);
}
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 80321ab16b32..66815fd8f856 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -295,6 +295,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
mutex_destroy(&fp->aux->dst_mutex);
kfree(fp->aux->poke_tab);
free_percpu(fp->aux->timed_may_goto);
+ kvfree(fp->aux->frame_desc_sites);
kfree(fp->aux);
}
free_percpu(fp->stats);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 984d6287241a..0dfb33bb8bb8 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -2047,9 +2047,15 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->nr_linfo = prog->aux->nr_linfo;
func[i]->aux->jited_linfo = prog->aux->jited_linfo;
func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx;
- /* The pool is freed with the main prog. */
+ /* The pool is freed with the main prog, the sites with func[i]. */
func[i]->aux->frame_desc = env->subprog_info[i].frame_desc;
func[i]->aux->frame_desc_pool = env->frame_desc_pool;
+ func[i]->aux->frame_desc_sites = bpf_frame_desc_sites(env, i);
+ if (IS_ERR(func[i]->aux->frame_desc_sites)) {
+ err = PTR_ERR(func[i]->aux->frame_desc_sites);
+ func[i]->aux->frame_desc_sites = NULL;
+ goto out_free;
+ }
func[i]->aux->arena = prog->aux->arena;
func[i]->aux->arena_stack_size = prog->aux->arena_stack_size;
func[i]->aux->arena_stack_entry = env->subprog_info[i].arena_stack_entry;
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index b0db3feeb9b0..b09cd868d8d9 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -1,18 +1,30 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
- * Frame descriptors: the hidden stack slots in which the frames of a program
- * that may throw keep the handles of the kernel resources they hold, for
- * bpf_throw() to release, see unwind.c.
+ * Frame descriptors: where the frames of a program that may throw keep the
+ * handles of the kernel resources they hold, for bpf_throw() to release, see
+ * unwind.c.
*
- * Nothing is inferred about where a program keeps a reference. The handle of
- * a resource is written down when the resource is acquired: the pointer a
- * call returns, the address of the lock, the iterator or the saved IRQ flags
- * a call takes, the record or the reader of a dynptr. Every call that
- * acquires has a slot of its own in the frame that runs it. The handle goes
- * into the slot around the call, and the call that releases the resource
- * clears the slot when it holds the handle the call takes. The sections a
- * frame enters, RCU read and preemption disabled, have no handle and are
- * counted in a slot per kind.
+ * A frame is unwound at a site: the call it has not returned from,
+ * bpf_throw() or the call of a subprog that may throw. The main pass visits a
+ * site with every state a frame can be in there, and records what the frame
+ * keeps in its own stack on each visit: the iterators and saved IRQ flags it
+ * holds, which are on its stack, the slots a reference it holds is spilled
+ * in, and how many RCU read and preemption disabled sections it is in. What
+ * every visit finds in the same place goes into the descriptor of the site,
+ * and costs the program nothing.
+ *
+ * What the visits of some site do not agree on is written down when it is
+ * acquired, in a hidden stack slot. Nothing is inferred about where the
+ * program keeps it. The handle is the pointer a call returns, the address of
+ * the lock, the iterator or the saved IRQ flags a call takes, the record or
+ * the reader of a dynptr. Every call that acquires such a resource has a slot
+ * of its own in the frame that runs it. The handle goes into the slot around
+ * the call, and the call that releases the resource clears the slot when it
+ * holds the handle the call takes. The sections a frame enters have no handle
+ * and are counted in a slot per kind. The slots of a subprog are listed in
+ * its own descriptor.
+ *
+ * What no frame holds at any site needs neither.
*
* A slot belongs to one frame and holds one handle, so a program that throws
* keeps to what that can express:
@@ -24,9 +36,10 @@
* returns one, and is in the sections it was called in.
* - A frame holds one resource of an acquiring call at a time.
*
- * The main pass records every call that acquires or releases and which
- * acquisition a release may be of. The slots are reserved and the
- * instructions that keep them emitted after the pass.
+ * The main pass records every call that acquires or releases, which
+ * acquisition a release may be of, and the visits of the sites. What needs a
+ * slot is decided after the pass: the slots are reserved, the instructions
+ * that keep them emitted and the descriptors built then.
*/
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
@@ -245,6 +258,35 @@ static bool reg_ref_off(const struct bpf_reg_state *reg, int id, s16 *off)
return true;
}
+/*
+ * Does @reg, a register of frame @f of @st or one spilled in its stack, point
+ * to what the frame holds a reference to? A site names such a place, see
+ * ref_locations().
+ *
+ * The program may have read the pointer for the last time while the frame
+ * holds the resource still, at the latest in front of a bpf_throw(), which
+ * nothing follows. A state forgets what is dead, see
+ * bpf_clean_verifier_state(), and no site could name the place then. So such
+ * a place is kept: it stays in the state, and a state stands for another only
+ * when both have the same in it. What is dead otherwise is forgotten as it
+ * was.
+ */
+bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
+ int f, const struct bpf_reg_state *reg)
+{
+ s16 off;
+ int i;
+
+ if (!env->describe_frames || reg->type == NOT_INIT || reg->type == SCALAR_VALUE || !reg->id)
+ return false;
+ /* a frame above may hold it for now and give it back, see bpf_frame_desc_record_exit() */
+ for (i = 0; i < st->acquired_refs; i++)
+ if (st->refs[i].frameno >= f && st->refs[i].acq && st->refs[i].id &&
+ reg_ref_off(reg, st->refs[i].id, &off))
+ return true;
+ return false;
+}
+
/*
* Is @slot the stack object of reference @id: the first slot of an iterator,
* of a ringbuf dynptr, which tracks the record's reference as its parent,
@@ -498,9 +540,9 @@ static bool acquires_sometimes(const struct bpf_verifier_env *env, u32 idx,
/*
* The kernel holds a reference for the __ref argument @argno of a struct_ops
* program, refs[@idx], and the program releases it or hands it back: the main
- * program holds it from its first insn, in a slot its entry fills from the
- * context. An argument of a type the unwinder cannot release gets no slot,
- * like any such reference.
+ * program holds it from its first insn. When it needs a slot, the entry of the
+ * program fills the slot from the context. An argument of a type the unwinder
+ * cannot release is not recorded, like any such reference.
*/
int bpf_frame_desc_record_ctx_arg(struct bpf_verifier_env *env, int idx, int argno)
{
@@ -691,6 +733,162 @@ int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifi
return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
}
+/*
+ * How many sections of @kind frame @f of @st is in: those entered since it
+ * was called, less those of its callers it left. A frame is entered with the
+ * sections its callers are in, which entry_*_locks counts, less the ones the
+ * frame or its callees have left since, which *_released_below counts.
+ */
+static int frame_sections(const struct bpf_verifier_state *st, int f, int kind)
+{
+ const struct bpf_func_state *frame = st->frame[f], *callee;
+ u32 entry, now;
+
+ entry = kind ? frame->entry_preempt_locks + frame->preempt_released_below :
+ frame->entry_rcu_locks + frame->rcu_released_below;
+ if (f == st->curframe) {
+ now = kind ? st->active_preempt_locks : st->active_rcu_locks;
+ } else {
+ callee = st->frame[f + 1];
+ now = kind ? callee->entry_preempt_locks + callee->preempt_released_below :
+ callee->entry_rcu_locks + callee->rcu_released_below;
+ }
+ return (int)now - (int)entry;
+}
+
+/*
+ * The places in the stack of frame @f of @st that hold the handle of @ref,
+ * which the frame owns, in @loc: the stack object the reference is of, or
+ * the slots a pointer to the object is spilled in. A lock has none. The slots
+ * a spill is removed from again, see bpf_remove_fastcall_spills_fills(), hold
+ * nothing at run time.
+ */
+static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifier_state *st, int f,
+ const struct bpf_reference_state *ref, struct bpf_res_loc *loc)
+{
+ const struct bpf_res_acq *acq = &env->res_acqs[ref->acq - 1];
+ const struct bpf_func_state *frame = st->frame[f];
+ s16 adj, lim = env->subprog_info[frame->subprogno].fastcall_stack_end;
+ const struct bpf_stack_state *slot;
+ int k;
+
+ loc->cnt = 0;
+ loc->obj = false;
+ if (ref->type & REF_TYPE_LOCK_MASK)
+ return;
+ for (k = 0; k < bpf_stack_nr_slots(frame) && loc->cnt < BPF_RES_LOC_MAX; k++) {
+ slot = bpf_stack_slot(frame, k);
+ adj = 0;
+ if (spi_off(k) < lim)
+ break;
+ if (stack_slot_is_object(slot, ref->id)) {
+ /* the handle of a dynptr is its first word, as when it was created */
+ if (slot->slot_type[BPF_REG_SIZE - 1] != STACK_DYNPTR) {
+ loc->obj = true;
+ loc->cnt = 0;
+ }
+ } else if (ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG) ||
+ !bpf_is_spilled_reg(slot) || !reg_ref_off(&slot->spilled_ptr, ref->id, &adj)) {
+ continue;
+ }
+ loc->at[loc->cnt].off = spi_off(k);
+ loc->at[loc->cnt++].adj = adj;
+ if (loc->obj)
+ break;
+ }
+}
+
+/*
+ * A throw may find frame @f of @st at the call @insn. Record what the frame
+ * keeps in its stack there, for the descriptor of the site: the visits of a
+ * site have to agree on it, since the site has one descriptor.
+ */
+static int site_visit(struct bpf_verifier_env *env, const struct bpf_verifier_state *st, int f,
+ u32 insn)
+{
+ struct bpf_res_loc cur, *loc;
+ struct bpf_res_site *site;
+ u32 i, j, k, n, s;
+
+ for (s = 0; s < env->res_site_cnt; s++)
+ if (env->res_sites[s].insn == insn)
+ break;
+ if (s == env->res_site_cnt) {
+ site = kvrealloc(env->res_sites, array_size(s + 1, sizeof(*site)), GFP_KERNEL_ACCOUNT);
+ if (!site)
+ return -ENOMEM;
+ env->res_sites = site;
+ site[s] = (struct bpf_res_site) { .insn = insn };
+ env->res_site_cnt = s + 1;
+ }
+ site = &env->res_sites[s];
+ for (k = 0; k < ARRAY_SIZE(site->sections); k++) {
+ int cnt = frame_sections(st, f, k);
+
+ if (!site->visits)
+ site->sections[k] = cnt;
+ else if (site->sections[k] != cnt)
+ site->sections_vary[k] = true;
+ }
+ site->visits++;
+ for (i = 0; i < st->acquired_refs; i++) {
+ const struct bpf_reference_state *ref = &st->refs[i];
+
+ if (ref->frameno != f || !ref->acq)
+ continue;
+ env->res_acqs[ref->acq - 1].held = true;
+ ref_locations(env, st, f, ref, &cur);
+ for (j = 0; j < env->res_loc_cnt; j++)
+ if (env->res_locs[j].site == s && env->res_locs[j].acq == ref->acq - 1)
+ break;
+ if (j == env->res_loc_cnt) {
+ loc = kvrealloc(env->res_locs, array_size(j + 1, sizeof(*loc)), GFP_KERNEL_ACCOUNT);
+ if (!loc)
+ return -ENOMEM;
+ env->res_locs = loc;
+ cur.site = s;
+ cur.acq = ref->acq - 1;
+ cur.visits = 1;
+ loc[j] = cur;
+ env->res_loc_cnt = j + 1;
+ continue;
+ }
+ /* the places this visit finds the handle in as well */
+ loc = &env->res_locs[j];
+ loc->visits++;
+ for (k = 0, n = 0; k < loc->cnt; k++) {
+ for (j = 0; j < cur.cnt; j++)
+ if (cur.at[j].off == loc->at[k].off && cur.at[j].adj == loc->at[k].adj)
+ break;
+ if (j < cur.cnt && cur.obj == loc->obj)
+ loc->at[n++] = loc->at[k];
+ }
+ loc->cnt = n;
+ }
+ return 0;
+}
+
+/*
+ * A throw may find the frames of @st where they are: the current one at
+ * @insn_idx, bpf_throw() or the call of a subprog that may throw, and each of
+ * its callers at the call it has not returned from. The frames that called a
+ * function verified on its own were recorded when they called it.
+ */
+int bpf_frame_desc_record_site(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx)
+{
+ int f, err;
+
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ for (f = st->curframe; f >= 0; f--) {
+ err = site_visit(env, st, f, f == st->curframe ? insn_idx : st->frame[f + 1]->callsite);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
/*
* A round of the main pass is dropped, see do_check_all(): what its paths
* recorded goes with it.
@@ -702,6 +900,8 @@ void bpf_frame_desc_reset(struct bpf_verifier_env *env)
env->res_acq_cnt = 0;
env->res_rel_cnt = 0;
env->section_insn_cnt = 0;
+ env->res_site_cnt = 0;
+ env->res_loc_cnt = 0;
for (i = 0; i < env->prog->len; i++) {
env->insn_aux_data[i].res_none = false;
env->insn_aux_data[i].res_none_r0 = false;
@@ -713,10 +913,14 @@ void bpf_frame_desc_env_free(struct bpf_verifier_env *env)
kvfree(env->res_acqs);
kvfree(env->res_rels);
kvfree(env->section_insns);
+ kvfree(env->res_sites);
+ kvfree(env->res_locs);
kvfree(env->resources);
env->res_acqs = NULL;
env->res_rels = NULL;
env->section_insns = NULL;
+ env->res_sites = NULL;
+ env->res_locs = NULL;
env->resources = NULL;
}
@@ -752,20 +956,6 @@ static int acq_subprog(struct bpf_verifier_env *env, const struct bpf_res_acq *a
return bpf_find_containing_subprog(env, new_of[a->insn]) - env->subprog_info;
}
-static bool subprog_has_section(struct bpf_verifier_env *env, int subprog, const u32 *new_of, u8 kind)
-{
- u32 i;
-
- for (i = 0; i < env->section_insn_cnt; i++) {
- const struct bpf_section_insn *si = &env->section_insns[i];
-
- if (si->kind == kind &&
- bpf_find_containing_subprog(env, new_of[si->insn]) - env->subprog_info == subprog)
- return true;
- }
- return false;
-}
-
/* An upper bound of the insns put around one call for one acquisition or release. */
#define RES_INSNS_MAX 6
@@ -809,22 +999,37 @@ static int emit_acquire_after(struct bpf_insn *buf, const struct bpf_res_acq *a)
}
/*
- * Reserve the hidden slots, once the pass has seen every frame: one for each
- * acquisition, in the frame of the subprog it is in, and a count of the
- * sections of a kind for a subprog that enters or leaves them. The subprogs
- * are numbered as they are now and looked up again whenever they are needed,
- * since removing a dead one renumbers them.
+ * Decide what needs a hidden slot, once the pass has seen every frame, and
+ * reserve it.
+ *
+ * A frame is unwound at a site, and the site has a descriptor for what every
+ * visit of it found in the same place of the frame's stack. So an
+ * acquisition needs no slot when no site finds a frame holding what it
+ * acquired, and none when every site that does finds the handle in a place
+ * all its visits agree on, on every visit. The others get a slot in the
+ * frame of the subprog they are in. Likewise a subprog counts its sections
+ * of a kind in a slot only when the visits of one of its sites differ in how
+ * many the frame is in.
+ *
+ * The subprogs are numbered as they are now and looked up again whenever
+ * they are needed, since removing a dead one renumbers them.
*/
int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
{
u32 i, k, orig_len, *new_of;
- int s, ret = 0;
+ int ret = 0;
if (!env->res_acq_cnt && !env->section_insn_cnt)
return 0;
new_of = insns_of_pass(env, &orig_len);
if (!new_of)
return -ENOMEM;
+ for (i = 0; i < env->res_loc_cnt; i++) {
+ const struct bpf_res_loc *loc = &env->res_locs[i];
+
+ if (!loc->cnt || loc->visits != env->res_sites[loc->site].visits)
+ env->res_acqs[loc->acq].slot = true;
+ }
for (i = 0; i < env->res_acq_cnt && !ret; i++) {
struct bpf_res_acq *a = &env->res_acqs[i];
@@ -833,6 +1038,8 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
ret = -EFAULT;
break;
}
+ if (!a->slot)
+ continue;
/* some visit of the insn has nothing a slot can hold where this one has */
if (!(a->flags & BPF_RES_F_CTX) && acquires_sometimes(env, new_of[a->insn], a)) {
verbose(env, "frame_desc: insn %d acquires a resource on some paths only, or one of a type that cannot be released\n",
@@ -843,11 +1050,19 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
ret = bpf_reserve_stack_slots(env, acq_subprog(env, a, new_of), 1, &a->off);
env->frame_desc_slots++;
}
- for (s = 0; s < env->subprog_cnt && !ret; s++) {
- for (k = 0; k < ARRAY_SIZE(env->subprog_info[s].section_off) && !ret; k++) {
- if (!subprog_has_section(env, s, new_of, k))
+ for (i = 0; i < env->res_site_cnt && !ret; i++) {
+ const struct bpf_res_site *site = &env->res_sites[i];
+ struct bpf_subprog_info *sub;
+
+ if (verifier_bug_if(site->insn >= orig_len, env, "site at insn %d", site->insn)) {
+ ret = -EFAULT;
+ break;
+ }
+ sub = bpf_find_containing_subprog(env, new_of[site->insn]);
+ for (k = 0; k < ARRAY_SIZE(sub->section_off) && !ret; k++) {
+ if (!site->sections_vary[k] || sub->section_off[k])
continue;
- ret = bpf_reserve_stack_slots(env, s, 1, &env->subprog_info[s].section_off[k]);
+ ret = bpf_reserve_stack_slots(env, sub - env->subprog_info, 1, &sub->section_off[k]);
env->frame_desc_slots++;
}
}
@@ -961,9 +1176,9 @@ static int emit_items(struct bpf_verifier_env *env, const struct res_item *it, i
/*
* At the entry of each subprog: its slots and counts start out empty, and
- * the main program takes the references the kernel passed it from the
- * context, which R1 points at. The insns go in front of the first one, which
- * a jump to it skips.
+ * the main program takes the references the kernel passed it that it keeps
+ * in a slot from the context, which R1 points at. The insns go in front of
+ * the first one, which a jump to it skips.
*/
static int emit_entries(struct bpf_verifier_env *env)
{
@@ -984,7 +1199,7 @@ static int emit_entries(struct bpf_verifier_env *env)
for (i = 0; i < env->res_acq_cnt; i++) {
const struct bpf_res_acq *a = &env->res_acqs[i];
- if (acq_subprog(env, a, new_of) != s)
+ if (!a->slot || acq_subprog(env, a, new_of) != s)
continue;
if (a->flags & BPF_RES_F_CTX) {
buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_1, a->obj_off);
@@ -1009,7 +1224,7 @@ static int emit_entries(struct bpf_verifier_env *env)
}
/*
- * Keep the slots and the counts up to date, through the patch list. What goes
+ * Keep the slots and the counts there are up to date, through the patch list. What goes
* in front of a call is patched in with the call, so that the jumps to it
* land on it; what follows it is inserted in front of the next insn, which a
* jump to that insn skips, as it skips the call.
@@ -1030,13 +1245,19 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
ret = -ENOMEM;
goto out;
}
- for (i = 0; i < env->section_insn_cnt; i++)
- items[n++] = (struct res_item) { new_of[env->section_insns[i].insn], i, ITEM_SECTION };
+ /* what has no slot is kept by nothing */
+ for (i = 0; i < env->section_insn_cnt; i++) {
+ const struct bpf_section_insn *si = &env->section_insns[i];
+
+ if (bpf_find_containing_subprog(env, new_of[si->insn])->section_off[si->kind])
+ items[n++] = (struct res_item) { new_of[si->insn], i, ITEM_SECTION };
+ }
for (i = 0; i < env->res_rel_cnt; i++)
- items[n++] = (struct res_item) { new_of[env->res_rels[i].insn], i, ITEM_RELEASE,
- env->res_rels[i].reg };
+ if (env->res_acqs[env->res_rels[i].acq].slot)
+ items[n++] = (struct res_item) { new_of[env->res_rels[i].insn], i, ITEM_RELEASE,
+ env->res_rels[i].reg };
for (i = 0; i < env->res_acq_cnt; i++)
- if (!(env->res_acqs[i].flags & BPF_RES_F_CTX))
+ if (env->res_acqs[i].slot && !(env->res_acqs[i].flags & BPF_RES_F_CTX))
items[n++] = (struct res_item) { new_of[env->res_acqs[i].insn], i, ITEM_ACQUIRE };
sort(items, n, sizeof(*items), item_cmp, NULL);
for (i = 0; i < n; i = k) {
@@ -1097,20 +1318,150 @@ static bool resource_goes_first(const struct bpf_resource *res)
return res->kind == BPF_RESOURCE_ITER || res->kind == BPF_RESOURCE_DYNPTR_FILE;
}
+/* Add @d to the pool of the program, or free it when the pool has it: its index plus one. */
+static int pool_add(struct bpf_verifier_env *env, struct bpf_frame_desc *d)
+{
+ struct bpf_frame_desc_pool *pool = env->frame_desc_pool;
+ u32 i;
+
+ for (i = 0; i < pool->cnt; i++) {
+ if (pool->desc[i]->size != d->size || memcmp(pool->desc[i], d, d->size))
+ continue;
+ kvfree(d);
+ return i + 1;
+ }
+ pool = kvrealloc(pool, struct_size(pool, desc, pool->cnt + 1), GFP_KERNEL_ACCOUNT);
+ if (!pool) {
+ kvfree(d);
+ return -ENOMEM;
+ }
+ env->frame_desc_pool = pool;
+ pool->desc[pool->cnt] = d;
+ env->frame_desc_bytes += d->size;
+ return ++pool->cnt;
+}
+
+/*
+ * The descriptor of a subprog that has slots: the slot of each acquisition in
+ * it that has one, which holds the handle of what the acquisition acquired or
+ * NULL, and the counts of its sections.
+ */
+static int build_slots(struct bpf_verifier_env *env, int s, const u32 *new_of)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[s];
+ struct bpf_frame_desc *d;
+ u32 i, k, n = 0;
+ int res;
+
+ for (i = 0; i < env->res_acq_cnt; i++)
+ n += env->res_acqs[i].slot && acq_subprog(env, &env->res_acqs[i], new_of) == s;
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++)
+ n += !!sub->section_off[k];
+ if (!n)
+ return 0;
+ d = kvzalloc(struct_size(d, ent, n), GFP_KERNEL_ACCOUNT);
+ if (!d)
+ return -ENOMEM;
+ d->size = struct_size(d, ent, n);
+ for (i = 0; i < env->res_acq_cnt; i++) {
+ const struct bpf_res_acq *a = &env->res_acqs[i];
+
+ if (!a->slot || acq_subprog(env, a, new_of) != s)
+ continue;
+ d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
+ .res = a->res, .off = a->off,
+ .flags = resource_goes_first(&env->resources[a->res]) ? BPF_FRAME_ENT_FIRST : 0,
+ };
+ }
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++) {
+ if (!sub->section_off[k])
+ continue;
+ res = resource_of_section(env, k);
+ if (res < 0) {
+ kvfree(d);
+ return res;
+ }
+ d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
+ .res = res, .off = sub->section_off[k], .flags = BPF_FRAME_ENT_COUNTER,
+ };
+ }
+ res = pool_add(env, d);
+ if (res < 0)
+ return res;
+ sub->frame_desc = res;
+ return 0;
+}
+
+/*
+ * The descriptor of site @idx: where the frame at the site keeps, in its own
+ * stack, what it holds that has no slot, and how many sections it is in when
+ * its subprog does not count them. A site that has nothing to say has none.
+ */
+static int build_site(struct bpf_verifier_env *env, u32 idx, const u32 *new_of)
+{
+ struct bpf_res_site *site = &env->res_sites[idx];
+ const struct bpf_subprog_info *sub = bpf_find_containing_subprog(env, new_of[site->insn]);
+ struct bpf_frame_desc *d;
+ u32 i, k, n = 0;
+ int res;
+
+ for (i = 0; i < env->res_loc_cnt; i++)
+ n += env->res_locs[i].site == idx && !env->res_acqs[env->res_locs[i].acq].slot;
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++)
+ n += !sub->section_off[k] && site->sections[k];
+ if (!n)
+ return 0;
+ d = kvzalloc(struct_size(d, ent, n), GFP_KERNEL_ACCOUNT);
+ if (!d)
+ return -ENOMEM;
+ d->size = struct_size(d, ent, n);
+ for (i = 0; i < env->res_loc_cnt; i++) {
+ const struct bpf_res_loc *loc = &env->res_locs[i];
+ const struct bpf_res_acq *a = &env->res_acqs[loc->acq];
+ u16 flags = loc->obj ? BPF_FRAME_ENT_OBJ : 0;
+
+ if (loc->site != idx || a->slot)
+ continue;
+ if (resource_goes_first(&env->resources[a->res]))
+ flags |= BPF_FRAME_ENT_FIRST;
+ d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
+ .res = a->res, .off = loc->at[0].off, .adj = loc->at[0].adj, .flags = flags,
+ };
+ }
+ for (k = 0; k < ARRAY_SIZE(sub->section_off); k++) {
+ if (sub->section_off[k] || !site->sections[k])
+ continue;
+ res = resource_of_section(env, k);
+ if (res < 0) {
+ kvfree(d);
+ return res;
+ }
+ d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
+ .res = res, .count = site->sections[k], .flags = BPF_FRAME_ENT_COUNT,
+ };
+ }
+ res = pool_add(env, d);
+ if (res < 0)
+ return res;
+ site->desc = res;
+ env->frame_desc_sites++;
+ return 0;
+}
+
/*
- * The frame descriptor of every subprog that has slots: the slot of each of
- * its acquisitions, which holds the handle of what the acquisition acquired
- * or NULL, and the counts of its sections. The unwinder releases what the
- * slots of a frame hold in the order of the entries, within a phase: the
- * iterators and the readers of file dynptrs come before the references,
- * which they may be iterating over or reading. Identical descriptors share a
- * body in the pool of the program, which takes the resource classes over.
+ * The frame descriptors of the program: one for every subprog that has
+ * slots, and one for every site that says what the frame at it keeps in its
+ * own stack. The unwinder releases what a frame holds in the order of the
+ * entries, within a phase, but for the iterators and the readers of file
+ * dynptrs: they come before the references, which they may be iterating over
+ * or reading. Identical descriptors share a body in the pool of the program,
+ * which takes the resource classes over.
*/
int bpf_frame_desc_build(struct bpf_verifier_env *env)
{
- u32 i, k, n, s, orig_len, *new_of;
struct bpf_frame_desc_pool *pool;
- int res, ret = -ENOMEM;
+ u32 i, orig_len, *new_of;
+ int ret = -ENOMEM;
if (!env->res_acq_cnt && !env->section_insn_cnt)
return 0;
@@ -1121,66 +1472,88 @@ int bpf_frame_desc_build(struct bpf_verifier_env *env)
goto out;
}
env->frame_desc_pool = pool;
- for (s = 0; s < env->subprog_cnt; s++) {
- struct bpf_subprog_info *sub = &env->subprog_info[s];
- struct bpf_frame_desc *d;
+ for (i = 0, ret = 0; i < env->subprog_cnt && !ret; i++)
+ ret = build_slots(env, i, new_of);
+ for (i = 0; i < env->res_site_cnt && !ret; i++)
+ ret = build_site(env, i, new_of);
+ if (ret)
+ goto out;
+ pool = env->frame_desc_pool;
+ pool->resources = env->resources;
+ pool->resource_cnt = env->resource_cnt;
+ env->frame_desc_bytes += env->resource_cnt * sizeof(*env->resources);
+ env->resources = NULL;
+out:
+ kvfree(new_of);
+ return ret;
+}
- for (i = 0, n = 0; i < env->res_acq_cnt; i++)
- n += acq_subprog(env, &env->res_acqs[i], new_of) == s;
- for (k = 0; k < ARRAY_SIZE(sub->section_off); k++)
- n += !!sub->section_off[k];
- if (!n)
+static int site_cmp(const void *a, const void *b)
+{
+ const struct bpf_frame_desc_site *x = a, *y = b;
+
+ return x->off < y->off ? -1 : x->off > y->off;
+}
+
+/*
+ * The sites of @subprog that have a descriptor, for its JITed function, or
+ * NULL when it has none: each by the index of its call in the function, until
+ * the JIT has put the call somewhere, see bpf_prog_fill_frame_desc_sites().
+ * The insns of now are not those the pass named the sites by, and the insns
+ * put around a call name the call as well.
+ */
+struct bpf_frame_desc_sites *bpf_frame_desc_sites(struct bpf_verifier_env *env, int subprog)
+{
+ const struct bpf_subprog_info *sub = &env->subprog_info[subprog];
+ struct bpf_frame_desc_sites *tab;
+ u32 i, idx, n = 0, orig_len, *new_of;
+
+ for (i = 0; i < env->res_site_cnt; i++)
+ n += !!env->res_sites[i].desc;
+ if (!n)
+ return NULL;
+ new_of = insns_of_pass(env, &orig_len);
+ tab = kvzalloc(struct_size(tab, site, n), GFP_KERNEL_ACCOUNT);
+ if (!new_of || !tab) {
+ kvfree(new_of);
+ kvfree(tab);
+ return ERR_PTR(-ENOMEM);
+ }
+ for (i = 0; i < env->res_site_cnt; i++) {
+ const struct bpf_res_site *site = &env->res_sites[i];
+ const struct bpf_insn *insn;
+
+ if (!site->desc)
continue;
- d = kvzalloc(struct_size(d, ent, n), GFP_KERNEL_ACCOUNT);
- if (!d)
- goto out;
- d->size = struct_size(d, ent, n);
- for (i = 0; i < 2 * env->res_acq_cnt; i++) {
- const struct bpf_res_acq *a = &env->res_acqs[i % env->res_acq_cnt];
- bool first = i < env->res_acq_cnt;
-
- if (acq_subprog(env, a, new_of) != s ||
- resource_goes_first(&env->resources[a->res]) != first)
- continue;
- d->ent[d->nent++] = (struct bpf_frame_desc_entry) { .res = a->res, .off = a->off };
- }
- for (k = 0; k < ARRAY_SIZE(sub->section_off); k++) {
- if (!sub->section_off[k])
- continue;
- res = resource_of_section(env, k);
- if (res < 0) {
- kvfree(d);
- ret = res;
- goto out;
+ idx = site->insn < orig_len ? new_of[site->insn] : 0;
+ for (;; idx--) {
+ insn = &env->prog->insnsi[idx];
+ if (env->insn_aux_data[idx].orig_idx != site->insn) {
+ idx = env->prog->len;
+ break;
}
- d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
- .res = res, .off = sub->section_off[k], .flags = BPF_FRAME_ENT_COUNTER,
- };
- }
- for (i = 0; i < pool->cnt; i++)
- if (pool->desc[i]->size == d->size && !memcmp(pool->desc[i], d, d->size))
+ if ((BPF_CLASS(insn->code) == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) || !idx)
break;
- if (i < pool->cnt) {
- kvfree(d);
- sub->frame_desc = i + 1;
- continue;
}
- pool = kvrealloc(pool, struct_size(pool, desc, pool->cnt + 1), GFP_KERNEL_ACCOUNT);
- if (!pool) {
- kvfree(d);
- goto out;
+ insn = idx < env->prog->len ? &env->prog->insnsi[idx] : NULL;
+ if (verifier_bug_if(!insn || BPF_CLASS(insn->code) != BPF_JMP ||
+ BPF_OP(insn->code) != BPF_CALL, env,
+ "site at insn %d is not a call", site->insn)) {
+ kvfree(new_of);
+ kvfree(tab);
+ return ERR_PTR(-EFAULT);
}
- env->frame_desc_pool = pool;
- pool->desc[pool->cnt] = d;
- env->frame_desc_bytes += d->size;
- sub->frame_desc = ++pool->cnt;
+ if (idx < sub->start || idx >= (sub + 1)->start)
+ continue;
+ tab->site[tab->cnt++] = (struct bpf_frame_desc_site) {
+ .off = idx - sub->start, .desc = site->desc - 1,
+ };
}
- pool->resources = env->resources;
- pool->resource_cnt = env->resource_cnt;
- env->frame_desc_bytes += env->resource_cnt * sizeof(*env->resources);
- env->resources = NULL;
- ret = 0;
-out:
kvfree(new_of);
- return ret;
+ if (!tab->cnt) {
+ kvfree(tab);
+ return NULL;
+ }
+ sort(tab->site, tab->cnt, sizeof(tab->site[0]), site_cmp, NULL);
+ return tab;
}
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index ba1176dc868e..4ec834e31dff 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -16,7 +16,8 @@
*
* Inputs have to be at the same insn and cleaned with
* bpf_clean_verifier_state(): dead registers and stack slots are the same on
- * both sides then, but for dynptrs, iterators and irq flags: they stay.
+ * both sides then, but for dynptrs, iterators and irq flags: they stay. So
+ * does what a program that may throw keeps for it, see bpf_frame_desc_keeps().
*/
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 8f357fee6104..33cf3376f276 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -129,6 +129,7 @@ static bool check_scalar_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
}
static void __clean_func_state(struct bpf_verifier_env *env,
+ const struct bpf_verifier_state *vst,
struct bpf_func_state *st,
u16 live_regs, int frame)
{
@@ -168,6 +169,11 @@ static void __clean_func_state(struct bpf_verifier_env *env,
stype == STACK_IRQ_FLAG)
continue;
+ /* a throw finds what the frame holds through it, see frame_desc.c */
+ if (stype == STACK_SPILL &&
+ bpf_frame_desc_keeps(env, vst, frame, &bpf_stack_slot(st, i)->spilled_ptr))
+ continue;
+
/*
* Only scalar spills can be degraded to raw stack bytes
* when their high half is dead. Pointer spills need the
@@ -209,7 +215,7 @@ int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_s
u32 ip = bpf_frame_insn_idx(st, i);
u16 live_regs = env->insn_aux_data[ip].live_regs_before;
- __clean_func_state(env, st->frame[i], live_regs, i);
+ __clean_func_state(env, st, st->frame[i], live_regs, i);
}
return 0;
}
diff --git a/kernel/bpf/unwind.c b/kernel/bpf/unwind.c
index 3b488a4bd759..b8845d7a0220 100644
--- a/kernel/bpf/unwind.c
+++ b/kernel/bpf/unwind.c
@@ -3,7 +3,7 @@
* Releasing what the frames of a BPF program hold when bpf_throw() discards
* them. Before the exception callback takes over, the frames from the throw
* up to the main program are walked once per phase of the release order, and
- * each frame releases what its frame descriptor says it holds of that phase,
+ * each frame releases what its frame descriptors say it holds of that phase,
* see frame_desc.c.
*/
#include <linux/bpf.h>
@@ -11,6 +11,57 @@
#include <linux/filter.h>
#include <linux/irqflags.h>
+/*
+ * Convert the instruction indices of the sites of a JITed function to native
+ * offsets. offsets[k] is the first byte of instruction k relative to image
+ * and offsets[prog->len] the end of the function, as for
+ * bpf_prog_update_insn_ptrs().
+ */
+void bpf_prog_fill_frame_desc_sites(struct bpf_prog *prog, const u32 *offsets, void *image)
+{
+ struct bpf_frame_desc_sites *tab = prog->aux->frame_desc_sites;
+ u32 i;
+
+ if (!tab || tab->filled)
+ return;
+ for (i = 0; i < tab->cnt; i++) {
+ struct bpf_frame_desc_site *site = &tab->site[i];
+
+ site->end = image + offsets[site->off + 1] - (void *)prog->bpf_func;
+ site->off = image + offsets[site->off] - (void *)prog->bpf_func;
+ }
+ tab->filled = true;
+}
+
+/*
+ * The descriptor of the site of @fn that @ip is in, or NULL when the site has
+ * none. A frame is at the call it has not returned from: @ip is the return
+ * address less one, since the return address may be the first byte of the
+ * next instruction.
+ */
+static const struct bpf_frame_desc *bpf_unwind_site(const struct bpf_prog_aux *fn,
+ const struct bpf_prog *prog, u64 ip)
+{
+ const struct bpf_frame_desc_sites *tab = fn->frame_desc_sites;
+ u64 off = ip - (u64)(long)prog->bpf_func;
+ u32 l = 0, r, m;
+
+ if (!tab || !tab->filled)
+ return NULL;
+ /* The sites are disjoint and sorted: find the first one that ends after off. */
+ r = tab->cnt;
+ while (l < r) {
+ m = l + (r - l) / 2;
+ if (tab->site[m].end <= off)
+ l = m + 1;
+ else
+ r = m;
+ }
+ if (l == tab->cnt || tab->site[l].off > off)
+ return NULL;
+ return fn->frame_desc_pool->desc[tab->site[l].desc];
+}
+
void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool)
{
u32 i;
@@ -142,50 +193,81 @@ static u64 bpf_unwind_fp(const struct bpf_prog_aux *fn, u64 bp)
}
/*
- * Release what the slots of the frame of @prog hold of the phase of this
- * walk; @bp is its native frame pointer. The first walk sums the frame's
- * counts of sections for their phase, and releases the locks and IRQ flags
- * in two passes: the flags a frame saved with interrupts enabled, of which
- * it can have one at most, since every later save found them disabled, are
- * restored last, after the locks and the other flags in whatever order the
- * frame took them.
+ * Release what the entries of @d say a frame holds of the phase of this walk,
+ * the entries that go @first or the others; @fp is the R10 of the frame. The
+ * first walk sums the frame's counts of sections for their phase, and
+ * releases the locks and IRQ flags in two passes, see
+ * bpf_unwind_release_frame(): @late selects the flags saved with interrupts
+ * enabled.
+ */
+static void bpf_unwind_release_desc(struct bpf_unwind_ctx *ctx, const struct bpf_frame_desc_pool *pool,
+ const struct bpf_frame_desc *d, u64 fp, bool first, bool late)
+{
+ bool locks = ctx->phase == BPF_UNWIND_PHASE_LOCKS;
+ void *handle;
+ u32 i;
+ int s;
+
+ for (i = 0; d && i < d->nent; i++) {
+ const struct bpf_frame_desc_entry *ent = &d->ent[i];
+ const struct bpf_resource *res = &pool->resources[ent->res];
+ u64 *loc = (u64 *)(long)(fp + ent->off);
+
+ if (ent->flags & (BPF_FRAME_ENT_COUNTER | BPF_FRAME_ENT_COUNT)) {
+ if (!locks || first || late)
+ continue;
+ s = section_index(res->kind);
+ ctx->section[s] = res;
+ ctx->nr_sections[s] += ent->flags & BPF_FRAME_ENT_COUNT ? ent->count : (s64)*loc;
+ continue;
+ }
+ if (release_phase(res) != ctx->phase || !(ent->flags & BPF_FRAME_ENT_FIRST) != !first)
+ continue;
+ if (ent->flags & BPF_FRAME_ENT_OBJ) {
+ handle = loc;
+ } else {
+ if (!*loc)
+ continue;
+ handle = (void *)(long)(*loc - ent->adj);
+ }
+ if (locks && late != (res->kind == BPF_RESOURCE_IRQ_FLAG &&
+ !arch_irqs_disabled_flags(*(unsigned long *)handle)))
+ continue;
+ bpf_unwind_release(res, handle);
+ }
+}
+
+/*
+ * Release what the frame of @prog holds of the phase of this walk; @bp is its
+ * native frame pointer and @ip the address it returns to. The frame is
+ * described by the hidden slots of its function and by what the site it is at
+ * says of its stack; the entries that go first, an iterator or the reader of
+ * a file, of both come before the references they may be using. The locks and
+ * IRQ flags are released in two passes: the flags a frame saved with
+ * interrupts enabled, of which it can have one at most, since every later
+ * save found them disabled, are restored last, after the locks and the other
+ * flags in whatever order the frame took them.
*/
-static void bpf_unwind_release_frame(struct bpf_unwind_ctx *ctx, const struct bpf_prog *prog, u64 bp)
+static void bpf_unwind_release_frame(struct bpf_unwind_ctx *ctx, const struct bpf_prog *prog,
+ u64 ip, u64 bp)
{
const struct bpf_prog_aux *fn = bpf_prog_frame_aux(prog);
const struct bpf_frame_desc_pool *pool = fn->frame_desc_pool;
- int passes = ctx->phase == BPF_UNWIND_PHASE_LOCKS ? 2 : 1, pass, s;
- const struct bpf_frame_desc *d;
- bool late;
+ int passes = ctx->phase == BPF_UNWIND_PHASE_LOCKS ? 2 : 1, pass, first;
+ const struct bpf_frame_desc *d, *site;
u64 fp;
- u32 i;
- if (!fn->frame_desc)
+ if (!pool)
+ return;
+ d = fn->frame_desc ? pool->desc[fn->frame_desc - 1] : NULL;
+ site = bpf_unwind_site(fn, prog, ip - 1);
+ if (!d && !site)
return;
- d = pool->desc[fn->frame_desc - 1];
fp = bpf_unwind_fp(fn, bp);
for (pass = 0; pass < passes; pass++) {
- for (i = 0; i < d->nent; i++) {
- const struct bpf_frame_desc_entry *ent = &d->ent[i];
- const struct bpf_resource *res = &pool->resources[ent->res];
- u64 *slot = (u64 *)(long)(fp + ent->off);
- void *handle = (void *)(long)*slot;
-
- if (ent->flags & BPF_FRAME_ENT_COUNTER) {
- if (ctx->phase != BPF_UNWIND_PHASE_LOCKS || pass)
- continue;
- s = section_index(res->kind);
- ctx->section[s] = res;
- ctx->nr_sections[s] += (s64)*slot;
- continue;
- }
- if (!handle || release_phase(res) != ctx->phase)
- continue;
- late = res->kind == BPF_RESOURCE_IRQ_FLAG &&
- !arch_irqs_disabled_flags(*(unsigned long *)handle);
- if (passes == 2 && late != (pass == 1))
- continue;
- bpf_unwind_release(res, handle);
+ for (first = 1; first >= 0; first--) {
+ bpf_unwind_release_desc(ctx, pool, d, fp, first, pass);
+ bpf_unwind_release_desc(ctx, pool, site, fp, first, pass);
}
}
}
@@ -207,7 +289,7 @@ static bool bpf_unwind_walker(void *cookie, u64 ip, u64 sp, u64 bp)
if (!prog)
return !ctx->cnt;
ctx->cnt++;
- bpf_unwind_release_frame(ctx, prog, bp);
+ bpf_unwind_release_frame(ctx, prog, ip, bp);
/* the main program is where the exception callback takes over */
return bpf_is_subprog(prog);
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4d85a42a18d5..cbe99c346731 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3727,6 +3727,7 @@ static void check_fastcall_stack_contract(struct bpf_verifier_env *env,
* fastcall_stack_off to a value smaller than any possible offset.
*/
subprog->fastcall_stack_off = S16_MIN;
+ subprog->fastcall_stack_end = S16_MIN;
/* reset fastcall aux flags within subprogram,
* happens at most once per subprogram
*/
@@ -11847,6 +11848,18 @@ static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int i
inherited_id(state, callee->stack_arg_regs[i].map_uid);
}
+ /*
+ * A throw of the function finds the frames of the caller as they are
+ * now: it has nothing that points to their stack.
+ */
+ if (env->subprog_info[subprog].might_throw) {
+ err = bpf_frame_desc_record_site(env, state, insn_idx);
+ if (err) {
+ bpf_free_verifier_state(&entry, false);
+ return err;
+ }
+ }
+
account_current_path(env);
env->subprog_info[subprog].on_its_own = true;
err = bpf_flow_call(env, state, &entry, subprog, insn_idx);
@@ -20836,6 +20849,7 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,
else
subprog->keep_fastcall_stack = 1;
subprog->fastcall_stack_off = min(subprog->fastcall_stack_off, off);
+ subprog->fastcall_stack_end = max(subprog->fastcall_stack_end, off);
}
static int mark_fastcall_patterns(struct bpf_verifier_env *env)
@@ -20846,6 +20860,7 @@ static int mark_fastcall_patterns(struct bpf_verifier_env *env)
int s, i;
for (s = 0; s < env->subprog_cnt; ++s, ++subprog) {
+ subprog->fastcall_stack_end = S16_MIN;
/* find lowest stack spill offset used in this subprog */
lowest_off = 0;
for (i = subprog->start; i < (subprog + 1)->start; ++i) {
@@ -21072,7 +21087,8 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
* for exceptional exits.
*/
if (exception_exit)
- return PROCESS_BPF_EXIT;
+ return bpf_frame_desc_record_site(env, env->cur_state, env->insn_idx) ?:
+ PROCESS_BPF_EXIT;
if (env->cur_state->curframe) {
/* exit from nested function */
@@ -23742,8 +23758,8 @@ static void print_verification_stats(struct bpf_verifier_env *env)
env->max_states_per_insn, env->total_states,
env->peak_states);
if (env->log.level & BPF_LOG_STATS)
- verbose(env, "frame descriptors: slots %u insns %u classes %u bytes %u\n",
- env->frame_desc_slots, env->frame_desc_insns,
+ verbose(env, "frame descriptors: slots %u insns %u sites %u classes %u bytes %u\n",
+ env->frame_desc_slots, env->frame_desc_insns, env->frame_desc_sites,
env->frame_desc_pool ? env->frame_desc_pool->resource_cnt : env->resource_cnt,
env->frame_desc_bytes);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 14/29] bpf: Describe held resources at throw sites
2026-11-11 12:57 ` [PATCH bpf-next v1 14/29] bpf: Describe held resources at throw sites Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
0 siblings, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 7255 bytes --]
> bpf: Describe held resources at throw sites
TL;DR:
- s16 section counts may truncate with deep RCU nesting, so unwinding at
bpf_throw() could skip bpf_rcu_read_unlock()
- Two acquisitions of one pointer may leak a reference at throw
- Per-subprog site tables sized for the whole program
- Commit message omits a relaxed rejection
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
[ ... ]
> +struct bpf_res_site {
> + u32 insn; /* the call, as the main pass named it */
> + u32 visits;
> + s16 sections[2]; /* the sections the frame is in, RCU and preemption */
> + bool sections_vary[2]; /* the visits differ: the frame counts them in a slot */
Can the section count overflow sections[] here? frame_sections() returns an
int derived from the u32 active_rcu_locks / active_preempt_locks, and
nothing bounds the nesting depth.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> + site = &env->res_sites[s];
> + for (k = 0; k < ARRAY_SIZE(site->sections); k++) {
> + int cnt = frame_sections(st, f, k);
> +
> + if (!site->visits)
> + site->sections[k] = cnt;
> + else if (site->sections[k] != cnt)
> + site->sections_vary[k] = true;
> + }
Here the int is stored into an s16 on the first visit, and later visits are
compared against the truncated value. Consider:
for (i = 0; i < 40000; i++)
bpf_rcu_read_lock();
bpf_throw(0);
The throw site is visited once, so sections[0] becomes -25536 and
sections_vary stays false. bpf_frame_desc_resolve() then reserves no
counter slot.
[ ... ]
> + d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
> + .res = res, .count = site->sections[k], .flags = BPF_FRAME_ENT_COUNT,
> + };
build_site() emits the truncated value in the s16 bpf_frame_desc_entry.count.
At run time bpf_unwind_release_desc() adds it to ctx->nr_sections[], so the
unwinder either hits WARN_ONCE for a negative count or, when the depth is
exactly 65536 (count truncates to 0 and build_site() skips the entry), calls
no bpf_rcu_read_unlock() at all. The task then leaves the program with
rcu_read_lock_nesting still raised.
Before this patch the count lived in a 64-bit hidden slot read as
(s64)*loc, so the same program was unwound correctly. The s16 fields are
unchanged at the end of the series.
Could site_visit() set sections_vary[k] when cnt does not fit in an s16, so
the counter slot is used, or should such a program be rejected? Widening
sections[] and count would also work.
[ ... ]
> @@ -1030,13 +1245,19 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
[ ... ]
> for (i = 0; i < env->res_rel_cnt; i++)
> - items[n++] = (struct res_item) { new_of[env->res_rels[i].insn], i, ITEM_RELEASE,
> - env->res_rels[i].reg };
> + if (env->res_acqs[env->res_rels[i].acq].slot)
> + items[n++] = (struct res_item) { new_of[env->res_rels[i].insn], i, ITEM_RELEASE,
> + env->res_rels[i].reg };
Can this leak a reference when two acquisitions return the same pointer?
emit_release() clears the first slot whose value equals the released
handle, and relies on every acquisition released at the insn having a slot.
Now the release item is dropped for acquisitions without a slot, while
bpf_frame_desc_resolve() sets slot per acquisition.
t1 = bpf_task_acquire(task); /* A1 */
t2 = bpf_task_acquire(task); /* A2, same pointer as t1 */
p = cond ? t1 : t2; /* path 1: A1, path 2: A2 */
bpf_task_release(p); /* insn X */
if (cond) { bpf_task_release(t2); return 0; }
bpf_throw(0); /* path 2: t1 held, only in a register */
A1 is held at the bpf_throw() site with no stack location, so A1.slot is
true. A2 is never held at a site, so A2.slot is false. At X only A1's
slot is checked. On path 2 the verifier released A2, but at run time
R1 == t1 == A1's slot, so A1's slot is cleared. bpf_unwind_release_desc()
skips an empty slot, and t1 is never dropped at bpf_throw().
Before the patch A2 had a slot too, so whichever equal slot was cleared, the
other still held the pointer. The same can happen for any pair of
acquisitions that can return the same pointer, such as bpf_obj_new() and
bpf_refcount_acquire(), or two lookups of the same socket, and in the
opposite direction (A1 without a slot, A2 with one).
Should bpf_frame_desc_resolve() give a slot to every acquisition that shares
a release insn with an acquisition that has one, iterating to a fixpoint, or
should such programs be rejected? The final version of the series still
filters release items by slot.
[ ... ]
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -2047,9 +2047,15 @@ static int jit_subprogs(struct bpf_verifier_env *env)
[ ... ]
> + func[i]->aux->frame_desc_sites = bpf_frame_desc_sites(env, i);
Is it intended that every subprog keeps a table sized for the whole
program? bpf_frame_desc_sites() counts every site with a descriptor:
for (i = 0; i < env->res_site_cnt; i++)
n += !!env->res_sites[i].desc;
...
new_of = insns_of_pass(env, &orig_len);
tab = kvzalloc(struct_size(tab, site, n), GFP_KERNEL_ACCOUNT);
but fills only the entries whose idx lies inside the subprog. The table is
freed only when tab->cnt is 0, so each subprog with at least one site keeps
8 + 12 * n bytes, where n is the site count of the whole program. That is
up to BPF_MAX_SUBPROGS times the memory the sites need, charged to the
loader's memcg for the life of the program.
insns_of_pass() also runs once per subprog, each time doing a kvcalloc() of
orig_len u32 and a pass over insn_aux_data, so load time grows as
O(subprog_cnt * prog_len).
The commit message says "each JITed function has a table of its sites".
Could the sites in [sub->start, (sub + 1)->start) be counted before the
allocation, and new_of built once in jit_subprogs()? The function is
unchanged at the end of the series.
> bpf: Describe held resources at throw sites
>
> [ ... ]
> The rejections for a program that throws do not change, with one
> exception: an instruction that acquires a resource only on some paths is
> rejected only if the resource needs a hidden slot.
Is this statement complete? bpf_frame_desc_emit() rejects a call that
acquires a resource and is followed by an unwind_goto only when 'after' is
non-zero:
if (after && idx + 1 < len && bpf_is_unwind_goto(&env->prog->insnsi[idx + 1])) {
verbose(env, "frame_desc: the call at insn %d acquires a resource and may unwind, ...
'after' now counts only ITEM_ACQUIRE items of acquisitions with a slot.
Before this patch, a call to a static subprog that returns a reference and
is followed by an unwind_goto was always rejected. Now it is accepted when
the acquisition needs no hidden slot. The relaxation looks consistent with
the comment in the code, but the commit message names only the
acquires_sometimes() rejection as changed.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 15/29] bpf: Describe handles in registers at throw sites
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (13 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 14/29] bpf: Describe held resources at throw sites Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:11 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 16/29] bpf: Describe locks " Kumar Kartikeya Dwivedi
` (14 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently a site descriptor can only give locations on the stack. But a
compiler keeps a reference that lives across calls in R6-R9 unless it
runs out of them, and leaves it in whichever register it was in at a
bpf_throw(), which does not return. Such a reference takes a hidden slot,
with a store on acquire and a compare on release.
The unwinder has the registers of the throw site from the native entry
of bpf_throw(), and recovers the callee-saved registers of each frame for
the exception callback, see bpf_throw_caller_regs(). Do the same in the
walks that release resources, so that they have the registers of the
frame being unwound:
- In the frame that called bpf_throw(), R0-R9 are the values that the
entry captured.
- When the walk moves from a frame to its caller, R6-R9 keep their
values, except for those that the prologue of the frame saved, which
are loaded from where it saved them. R0-R5 are unknown from then on.
Add BPF_FRAME_ENT_REG for a descriptor entry that gives a register, and
have the verifier record registers at a site along with stack slots:
- R6-R9 of every frame in the state;
- R0 and R2-R5 of the frame that calls bpf_throw(). R1 holds the
cookie. For this, the site is recorded before the call clobbers the
caller-saved registers.
A register is described like a stack slot: when every visit to the site
finds the handle in it. If both hold the handle, the stack slot is
preferred. Locks and stack objects are handled as before.
The verifier forgets the contents of a dead register where paths join,
as it does for stack slots. So a register that points to a resource held
by its frame is kept like such a slot: it is not cleaned, a state can
stand for another only if both have the same contents in it, and a join
keeps it only if both sides have it. Register liveness does not change.
The program no longer uses such a register, but inserted code must not
reuse it either. The expansion of a timed may_goto keeps the address of
its count in a dead register: make it pick one that no site describes.
Registers are described only on architectures with a native entry,
x86-64 and arm64.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf.h | 11 +++-
include/linux/bpf_verifier.h | 14 +++--
kernel/bpf/fixups.c | 3 +-
kernel/bpf/frame_desc.c | 103 ++++++++++++++++++++++++++++-------
kernel/bpf/helpers.c | 2 +-
kernel/bpf/join.c | 3 +
kernel/bpf/states.c | 10 +++-
kernel/bpf/unwind.c | 29 ++++++++--
kernel/bpf/verifier.c | 13 ++++-
9 files changed, 151 insertions(+), 37 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index e8eda32bc05d..b157863cb952 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1873,6 +1873,11 @@ struct bpf_resource {
* slot of the frame, or a slot of the program's own stack that holds the
* pointer whenever the frame is at the site of the descriptor; the pointer
* may then point adj bytes into the object;
+ * - with BPF_FRAME_ENT_REG a register instead of a location, which holds
+ * such a pointer whenever the frame is at the site: off is its number. The
+ * unwinder has R6 to R9 of every frame, and R0 to R5 of the one that
+ * called bpf_throw(), where the architecture captures them, see
+ * bpf_arch_throw_entry();
* - with BPF_FRAME_ENT_OBJ the resource itself, an iterator or saved IRQ
* flags the frame has on its stack whenever it is at the site;
* - with BPF_FRAME_ENT_COUNTER a hidden slot that holds a count of sections,
@@ -1885,12 +1890,14 @@ struct bpf_resource {
#define BPF_FRAME_ENT_COUNT BIT(2)
/* Released before the other entries of the frame, which it may be using. */
#define BPF_FRAME_ENT_FIRST BIT(3)
+#define BPF_FRAME_ENT_REG BIT(4)
struct bpf_frame_desc_entry {
u16 res; /* index into the program's resource classes */
u16 flags; /* BPF_FRAME_ENT_* */
union {
s16 off; /* the location, relative to the frame pointer */
+ s16 reg; /* BPF_FRAME_ENT_REG */
s16 count; /* BPF_FRAME_ENT_COUNT */
};
s16 adj;
@@ -4631,7 +4638,7 @@ void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
void bpf_prog_fill_frame_desc_sites(struct bpf_prog *prog, const u32 *offsets, void *image);
void bpf_unwind_release(const struct bpf_resource *res, void *handle);
void bpf_dynptr_file_free(void *reader);
-void bpf_unwind_release_frames(void);
+void bpf_unwind_release_frames(const struct bpf_throw_regs *regs);
void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool);
#else
static inline void
@@ -4645,7 +4652,7 @@ bpf_prog_fill_frame_desc_sites(struct bpf_prog *prog, const u32 *offsets, void *
static inline void bpf_frame_desc_pool_free(struct bpf_frame_desc_pool *pool)
{
}
-static inline void bpf_unwind_release_frames(void)
+static inline void bpf_unwind_release_frames(const struct bpf_throw_regs *regs)
{
}
#endif
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index ca50c85006d7..158e7eddc19f 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1273,8 +1273,8 @@ struct bpf_res_acq {
bool slot; /* and some site cannot say where: the handle is kept in a slot */
};
-/* How many places of its stack a site remembers a frame keeping one handle in. */
-#define BPF_RES_LOC_MAX 3
+/* How many places a site remembers a frame keeping one handle in. */
+#define BPF_RES_LOC_MAX 4
/*
* A site: a call a frame can be unwound at, bpf_throw() or the call of a
@@ -1292,7 +1292,8 @@ struct bpf_res_site {
/*
* Where the frame at a site keeps the handle of what an acquisition acquired:
- * the slots of its stack that every visit holding it found the handle in.
+ * the slots of its stack and the registers that every visit holding it found
+ * the handle in.
*/
struct bpf_res_loc {
u32 site; /* index of the site */
@@ -1301,8 +1302,9 @@ struct bpf_res_loc {
u8 cnt; /* the places left */
bool obj; /* they are the resource itself, an object on the stack */
struct {
- s16 off; /* relative to R10 */
+ s16 off; /* relative to R10, or the number of the register */
s16 adj; /* the pointer there points that many bytes into the object */
+ bool reg;
} at[BPF_RES_LOC_MAX];
};
@@ -1374,6 +1376,8 @@ struct bpf_verifier_env {
u32 frame_desc_insns; /* instructions keeping them up to date */
u32 frame_desc_bytes; /* descriptors and resource classes */
bool describe_frames; /* frame descriptors are generated: the program may throw */
+ bool describe_regs; /* and may name registers, see bpf_arch_throw_entry() */
+ u16 frame_desc_regs; /* the registers that some site names */
int exception_callback_subprog;
bool explore_alu_limits;
bool allow_ptr_leaks;
@@ -2332,6 +2336,8 @@ int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delt
int bpf_frame_desc_record_exit(struct bpf_verifier_env *env);
bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
int f, const struct bpf_reg_state *reg);
+u16 bpf_frame_desc_kept_regs(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
+ int f);
int bpf_frame_desc_record_site(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
int insn_idx);
int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 0dfb33bb8bb8..b02ba82f5d10 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -2985,7 +2985,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
if (bpf_is_may_goto_insn(insn) && timed_may_goto) {
- u16 live = env->insn_aux_data[i].live_regs_before;
+ /* a site of a program that throws reads a register the program is done with */
+ u16 live = env->insn_aux_data[i].live_regs_before | env->frame_desc_regs;
int reg, save_off = 0;
/* a register that is dead here holds the address of the count */
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index b09cd868d8d9..5cb837688f90 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -7,11 +7,16 @@
* A frame is unwound at a site: the call it has not returned from,
* bpf_throw() or the call of a subprog that may throw. The main pass visits a
* site with every state a frame can be in there, and records what the frame
- * keeps in its own stack on each visit: the iterators and saved IRQ flags it
- * holds, which are on its stack, the slots a reference it holds is spilled
- * in, and how many RCU read and preemption disabled sections it is in. What
- * every visit finds in the same place goes into the descriptor of the site,
- * and costs the program nothing.
+ * keeps where on each visit: the iterators and saved IRQ flags it holds,
+ * which are on its stack, the slots a reference it holds is spilled in and
+ * the registers that hold it, and how many RCU read and preemption disabled
+ * sections it is in. What every visit finds in the same place goes into the
+ * descriptor of the site, and costs the program nothing.
+ *
+ * The unwinder has the registers of a frame where bpf_throw() has a native
+ * entry that captures those of the throw site, see bpf_arch_throw_entry(): R6
+ * to R9 of every frame, which are callee-saved and recovered from where the
+ * frames above saved them, and R0 to R5 of the frame that throws.
*
* What the visits of some site do not agree on is written down when it is
* acquired, in a hidden stack slot. Nothing is inferred about where the
@@ -287,6 +292,25 @@ bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_v
return false;
}
+/* The registers of frame @f of @st that are kept, see bpf_frame_desc_keeps(). */
+u16 bpf_frame_desc_kept_regs(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
+ int f)
+{
+ u16 kept = 0;
+ int i;
+
+ if (!env->describe_regs)
+ return 0;
+ /* the registers a site names: R6 to R9, and all but R1 of the frame that throws */
+ for (i = BPF_REG_0; i < BPF_REG_FP; i++) {
+ if (i == BPF_REG_1 || (i < BPF_REG_6 && f != st->curframe))
+ continue;
+ if (bpf_frame_desc_keeps(env, st, f, &st->frame[f]->regs[i]))
+ kept |= BIT(i);
+ }
+ return kept;
+}
+
/*
* Is @slot the stack object of reference @id: the first slot of an iterator,
* of a ringbuf dynptr, which tracks the record's reference as its parent,
@@ -757,14 +781,26 @@ static int frame_sections(const struct bpf_verifier_state *st, int f, int kind)
}
/*
- * The places in the stack of frame @f of @st that hold the handle of @ref,
- * which the frame owns, in @loc: the stack object the reference is of, or
- * the slots a pointer to the object is spilled in. A lock has none. The slots
- * a spill is removed from again, see bpf_remove_fastcall_spills_fills(), hold
- * nothing at run time.
+ * The registers the unwinder can read of a frame: R6 to R9 are callee-saved,
+ * so it has them for a frame that is in a call as well. The others it has for
+ * the frame that calls bpf_throw() only; R1 is the cookie there.
+ */
+static const u8 site_regs[] = {
+ BPF_REG_6, BPF_REG_7, BPF_REG_8, BPF_REG_9,
+ BPF_REG_0, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5,
+};
+
+/*
+ * The places of frame @f of @st that hold the handle of @ref, which the frame
+ * owns, in @loc: the stack object the reference is of, or the slots of the
+ * stack a pointer to the object is spilled in and the registers that hold
+ * one, see site_regs; @throws when the frame calls bpf_throw(). A lock has
+ * none. The slots a spill is removed from again, see
+ * bpf_remove_fastcall_spills_fills(), hold nothing at run time.
*/
static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifier_state *st, int f,
- const struct bpf_reference_state *ref, struct bpf_res_loc *loc)
+ const struct bpf_reference_state *ref, bool throws,
+ struct bpf_res_loc *loc)
{
const struct bpf_res_acq *acq = &env->res_acqs[ref->acq - 1];
const struct bpf_func_state *frame = st->frame[f];
@@ -792,20 +828,31 @@ static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifie
continue;
}
loc->at[loc->cnt].off = spi_off(k);
+ loc->at[loc->cnt].reg = false;
loc->at[loc->cnt++].adj = adj;
if (loc->obj)
- break;
+ return;
+ }
+ if (!env->describe_regs || ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG))
+ return;
+ for (k = 0; k < (throws ? ARRAY_SIZE(site_regs) : 4) && loc->cnt < BPF_RES_LOC_MAX; k++) {
+ if (!reg_ref_off(&frame->regs[site_regs[k]], ref->id, &adj))
+ continue;
+ loc->at[loc->cnt].off = site_regs[k];
+ loc->at[loc->cnt].reg = true;
+ loc->at[loc->cnt++].adj = adj;
}
}
/*
* A throw may find frame @f of @st at the call @insn. Record what the frame
- * keeps in its stack there, for the descriptor of the site: the visits of a
- * site have to agree on it, since the site has one descriptor.
+ * keeps where there, for the descriptor of the site: the visits of a site
+ * have to agree on it, since the site has one descriptor.
*/
static int site_visit(struct bpf_verifier_env *env, const struct bpf_verifier_state *st, int f,
u32 insn)
{
+ bool throws = f == st->curframe && bpf_is_throw_kfunc(&env->prog->insnsi[insn]);
struct bpf_res_loc cur, *loc;
struct bpf_res_site *site;
u32 i, j, k, n, s;
@@ -837,7 +884,7 @@ static int site_visit(struct bpf_verifier_env *env, const struct bpf_verifier_st
if (ref->frameno != f || !ref->acq)
continue;
env->res_acqs[ref->acq - 1].held = true;
- ref_locations(env, st, f, ref, &cur);
+ ref_locations(env, st, f, ref, throws, &cur);
for (j = 0; j < env->res_loc_cnt; j++)
if (env->res_locs[j].site == s && env->res_locs[j].acq == ref->acq - 1)
break;
@@ -858,7 +905,8 @@ static int site_visit(struct bpf_verifier_env *env, const struct bpf_verifier_st
loc->visits++;
for (k = 0, n = 0; k < loc->cnt; k++) {
for (j = 0; j < cur.cnt; j++)
- if (cur.at[j].off == loc->at[k].off && cur.at[j].adj == loc->at[k].adj)
+ if (cur.at[j].off == loc->at[k].off && cur.at[j].adj == loc->at[k].adj &&
+ cur.at[j].reg == loc->at[k].reg)
break;
if (j < cur.cnt && cur.obj == loc->obj)
loc->at[n++] = loc->at[k];
@@ -872,7 +920,9 @@ static int site_visit(struct bpf_verifier_env *env, const struct bpf_verifier_st
* A throw may find the frames of @st where they are: the current one at
* @insn_idx, bpf_throw() or the call of a subprog that may throw, and each of
* its callers at the call it has not returned from. The frames that called a
- * function verified on its own were recorded when they called it.
+ * function verified on its own were recorded when they called it. The
+ * registers of the frame that calls bpf_throw() have to be those it calls it
+ * with: the state is the one in front of the call.
*/
int bpf_frame_desc_record_site(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
int insn_idx)
@@ -1030,6 +1080,18 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
if (!loc->cnt || loc->visits != env->res_sites[loc->site].visits)
env->res_acqs[loc->acq].slot = true;
}
+ /*
+ * A register that a site names holds the handle from where the program
+ * read it for the last time: the program is done with it, but nothing
+ * that is inserted later may use it, see bpf_do_misc_fixups().
+ */
+ for (i = 0; i < env->res_loc_cnt; i++) {
+ const struct bpf_res_loc *loc = &env->res_locs[i];
+
+ for (k = 0; !env->res_acqs[loc->acq].slot && k < loc->cnt; k++)
+ if (loc->at[k].reg)
+ env->frame_desc_regs |= BIT(loc->at[k].off);
+ }
for (i = 0; i < env->res_acq_cnt && !ret; i++) {
struct bpf_res_acq *a = &env->res_acqs[i];
@@ -1394,8 +1456,9 @@ static int build_slots(struct bpf_verifier_env *env, int s, const u32 *new_of)
/*
* The descriptor of site @idx: where the frame at the site keeps, in its own
- * stack, what it holds that has no slot, and how many sections it is in when
- * its subprog does not count them. A site that has nothing to say has none.
+ * stack or in a register, what it holds that has no slot, and how many
+ * sections it is in when its subprog does not count them. A site that has
+ * nothing to say has none.
*/
static int build_site(struct bpf_verifier_env *env, u32 idx, const u32 *new_of)
{
@@ -1424,6 +1487,8 @@ static int build_site(struct bpf_verifier_env *env, u32 idx, const u32 *new_of)
continue;
if (resource_goes_first(&env->resources[a->res]))
flags |= BPF_FRAME_ENT_FIRST;
+ if (loc->at[0].reg)
+ flags |= BPF_FRAME_ENT_REG;
d->ent[d->nent++] = (struct bpf_frame_desc_entry) {
.res = a->res, .off = loc->at[0].off, .adj = loc->at[0].adj, .flags = flags,
};
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 793eb457fb62..648443e8f009 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3493,7 +3493,7 @@ notrace void bpf_throw_unwind(struct bpf_throw_regs *regs, u64 cookie)
if (regs)
memcpy(ctx.saved, ®s->r[BPF_REG_6], sizeof(ctx.saved));
/* What the frames about to be discarded hold is released first. */
- bpf_unwind_release_frames();
+ bpf_unwind_release_frames(regs);
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 4ec834e31dff..180b0a96029c 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -663,6 +663,9 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
u32 cnt;
+ /* the registers both keep for a throw are joined, see bpf_frame_desc_keeps() */
+ live_regs |= bpf_frame_desc_kept_regs(env, a, fr) &
+ bpf_frame_desc_kept_regs(env, b, fr);
/* not the states that get to the head through a call of the function */
frame = a->frame[fr];
cnt = frame->loop_stack_cnt;
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 33cf3376f276..5a773bfb9e84 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -215,6 +215,8 @@ int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_s
u32 ip = bpf_frame_insn_idx(st, i);
u16 live_regs = env->insn_aux_data[ip].live_regs_before;
+ /* a throw finds what the frame holds through them, see frame_desc.c */
+ live_regs |= bpf_frame_desc_kept_regs(env, st, i);
__clean_func_state(env, st, st->frame[i], live_regs, i);
}
return 0;
@@ -691,9 +693,10 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
* the type in the current state is meaningful.
*/
static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
- struct bpf_func_state *cur, u32 insn_idx)
+ struct bpf_func_state *cur, u32 insn_idx, u16 kept)
{
- u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
+ /* and the registers that @old keeps for a throw, see bpf_frame_desc_keeps() */
+ u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before | kept;
u16 i;
if (old->callback_depth > cur->callback_depth)
@@ -781,7 +784,8 @@ bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state
insn_idx = bpf_frame_insn_idx(old, i);
if (old->frame[i]->callsite != cur->frame[i]->callsite)
return false;
- if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx))
+ if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx,
+ bpf_frame_desc_kept_regs(env, old, i)))
return false;
}
diff --git a/kernel/bpf/unwind.c b/kernel/bpf/unwind.c
index b8845d7a0220..473897c3aff7 100644
--- a/kernel/bpf/unwind.c
+++ b/kernel/bpf/unwind.c
@@ -179,6 +179,13 @@ struct bpf_unwind_ctx {
/* The sections the frames hold, per kind, summed by the first walk. */
const struct bpf_resource *section[BPF_UNWIND_NR_SECTIONS];
s64 nr_sections[BPF_UNWIND_NR_SECTIONS];
+ /*
+ * The registers of the frame the walk is at, when the entry of
+ * bpf_throw() captured those of the throw site: the callee-saved ones,
+ * and R0 to R5 while the walk is at the frame that threw.
+ */
+ const struct bpf_throw_regs *entry;
+ struct bpf_throw_regs regs;
};
/*
@@ -212,6 +219,7 @@ static void bpf_unwind_release_desc(struct bpf_unwind_ctx *ctx, const struct bpf
const struct bpf_frame_desc_entry *ent = &d->ent[i];
const struct bpf_resource *res = &pool->resources[ent->res];
u64 *loc = (u64 *)(long)(fp + ent->off);
+ u64 val;
if (ent->flags & (BPF_FRAME_ENT_COUNTER | BPF_FRAME_ENT_COUNT)) {
if (!locks || first || late)
@@ -226,9 +234,13 @@ static void bpf_unwind_release_desc(struct bpf_unwind_ctx *ctx, const struct bpf
if (ent->flags & BPF_FRAME_ENT_OBJ) {
handle = loc;
} else {
- if (!*loc)
+ /* no descriptor names a register that was not captured */
+ if ((ent->flags & BPF_FRAME_ENT_REG) && WARN_ON_ONCE(!ctx->entry))
+ continue;
+ val = ent->flags & BPF_FRAME_ENT_REG ? ctx->regs.r[ent->reg] : *loc;
+ if (!val)
continue;
- handle = (void *)(long)(*loc - ent->adj);
+ handle = (void *)(long)(val - ent->adj);
}
if (locks && late != (res->kind == BPF_RESOURCE_IRQ_FLAG &&
!arch_irqs_disabled_flags(*(unsigned long *)handle)))
@@ -290,6 +302,8 @@ static bool bpf_unwind_walker(void *cookie, u64 ip, u64 sp, u64 bp)
return !ctx->cnt;
ctx->cnt++;
bpf_unwind_release_frame(ctx, prog, ip, bp);
+ /* the walk leaves the frame for its caller, which has R6 to R9 as it left them */
+ bpf_throw_caller_regs(bpf_prog_frame_aux(prog), bp, &ctx->regs.r[BPF_REG_6]);
/* the main program is where the exception callback takes over */
return bpf_is_subprog(prog);
}
@@ -302,16 +316,21 @@ static bool bpf_unwind_walker(void *cookie, u64 ip, u64 sp, u64 bp)
* each frame's entries of the phase in the order of its descriptor; the
* frames are released as the walk passes them, so their number is not
* bounded. The sections need no walk of their own: the first one summed
- * their counts, and sections of one kind are all the same.
+ * their counts, and sections of one kind are all the same. @regs are the
+ * registers of the frame that called bpf_throw(), or NULL when the
+ * architecture does not capture them.
*/
-void bpf_unwind_release_frames(void)
+void bpf_unwind_release_frames(const struct bpf_throw_regs *regs)
{
- struct bpf_unwind_ctx ctx = {};
+ struct bpf_unwind_ctx ctx = { .entry = regs };
int s;
for (ctx.phase = 0; ctx.phase < BPF_UNWIND_NR_PHASES; ctx.phase++) {
if (ctx.phase != BPF_UNWIND_PHASE_SECTIONS) {
ctx.cnt = 0;
+ /* every walk starts at the throw site */
+ if (regs)
+ ctx.regs = *regs;
arch_bpf_stack_walk(bpf_unwind_walker, &ctx);
continue;
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index cbe99c346731..b32c5a4f844f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -12217,6 +12217,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
verbose(env, "failed to push state for global subprog exception path\n");
return PTR_ERR(branch);
}
+ /* a throw of the subprog finds the frames as they are at the call */
+ err = bpf_frame_desc_record_site(env, env->cur_state, *insn_idx);
+ if (err)
+ return err;
return process_bpf_exit_full(env, NULL, true);
}
@@ -16477,6 +16481,10 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (err < 0)
return err;
}
+ /* with the registers the frame calls it with, which the call clobbers */
+ err = bpf_frame_desc_record_site(env, env->cur_state, insn_idx);
+ if (err)
+ return err;
}
bpf_diag_record_caller_saved(env, regs);
@@ -21087,8 +21095,7 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
* for exceptional exits.
*/
if (exception_exit)
- return bpf_frame_desc_record_site(env, env->cur_state, env->insn_idx) ?:
- PROCESS_BPF_EXIT;
+ return PROCESS_BPF_EXIT;
if (env->cur_state->curframe) {
/* exit from nested function */
@@ -25710,6 +25717,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
*/
for (i = 0; bpf_jit_supports_exception_cleanup() && i < env->subprog_cnt; i++)
env->describe_frames |= env->subprog_info[i].might_throw;
+ /* the unwinder has the registers of the frames where bpf_throw() has a native entry */
+ env->describe_regs = env->describe_frames && bpf_arch_throw_entry();
ret = check_unwind_insns(env);
if (ret < 0)
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 15/29] bpf: Describe handles in registers at throw sites
2026-11-11 12:57 ` [PATCH bpf-next v1 15/29] bpf: Describe handles in registers " Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:11 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]
> bpf: Describe handles in registers at throw sites
TL;DR: Nit only: the set of registers a throw site can name is defined twice,
in ref_locations() and bpf_frame_desc_kept_regs(), so the two could drift
apart. Suggests sharing site_regs[] and naming the literal 4.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> +/* The registers of frame @f of @st that are kept, see bpf_frame_desc_keeps(). */
> +u16 bpf_frame_desc_kept_regs(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
> + int f)
> +{
> + u16 kept = 0;
> + int i;
> +
> + if (!env->describe_regs)
> + return 0;
> + /* the registers a site names: R6 to R9, and all but R1 of the frame that throws */
> + for (i = BPF_REG_0; i < BPF_REG_FP; i++) {
> + if (i == BPF_REG_1 || (i < BPF_REG_6 && f != st->curframe))
> + continue;
[ ... ]
> + for (k = 0; k < (throws ? ARRAY_SIZE(site_regs) : 4) && loc->cnt < BPF_RES_LOC_MAX; k++) {
This isn't a bug, but the set of registers a site can name is written down
twice. ref_locations() walks site_regs[] and relies on the literal 4
meaning "the first four entries are R6-R9", while
bpf_frame_desc_kept_regs() rebuilds the same set from a loop over R0-R9
with its own exclusion test.
Could bpf_frame_desc_kept_regs() walk site_regs[] with the same bound as
ref_locations(), and could the literal 4 become a named count of the
callee-saved entries, so the two definitions can't drift apart?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 15/29] bpf: Describe handles in registers at throw sites
2026-11-11 12:57 ` [PATCH bpf-next v1 15/29] bpf: Describe handles in registers " Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 16:11 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 16:11 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] bpf: failed state joins on dead registers in
bpf_states_join()
--
commit 76f36da94410f1998945cf52f9a2b12ce1afb269
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Describe handles in registers at throw sites
This commit records resource handles held in registers in site descriptors
and recovers callee-saved registers across frames during exception unwinding
so that references held in registers at throw sites do not require hidden
stack slots.
> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> index 4ec834e31dff5..180b0a96029cf 100644
> --- a/kernel/bpf/join.c
> +++ b/kernel/bpf/join.c
> @@ -663,6 +663,9 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
> u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
> u32 cnt;
>
> + /* the registers both keep for a throw are joined, see bpf_frame_desc_keeps() */
> + live_regs |= bpf_frame_desc_kept_regs(env, a, fr) &
> + bpf_frame_desc_kept_regs(env, b, fr);
[Severity: Medium]
Can adding bpf_frame_desc_kept_regs() to live_regs cause bpf_states_join()
to fail when a dead register holds resource pointers with different offsets
or IDs in states a and b?
bpf_frame_desc_kept_regs() sets BIT(reg) whenever reg points to any held
resource at any constant s16 offset. If a dead register holds p + 0 on one
iteration of a loop and p + 1 on the next, BIT(reg) is set in both masks, so
join_func() runs join_reg() on the dead register. join_reg() rejects the
mismatched pointer offsets and aborts the entire state join instead of
clearing the dead register to NOT_INIT in dst, preventing loop widening and
causing verification to fail with -E2BIG once flow_merge() reaches
BPF_FLOW_MAX_STATES. Similarly, if the dead register holds two different
held resources in a and b, join_reg() in pass 1 calls jid_note() on their
distinct IDs, poisoning js->pairs and failing join_refs_check() in pass 2.
In addition, flow_unset_differs() assumes that only live_regs_before
registers remain initialized after bpf_clean_verifier_state():
kernel/bpf/flow.c:flow_unset_differs() {
for (fr = 0; fr <= a->curframe; fr++)
for (i = BPF_REG_0; i < BPF_REG_FP; i++)
if ((a->frame[fr]->regs[i].type == NOT_INIT) !=
(b->frame[fr]->regs[i].type == NOT_INIT))
return true;
return false;
}
When a dead register is kept by bpf_frame_desc_kept_regs() in only one of
the two states, flow_unset_differs() returns true and prevents flow_merge()
and flow_end_walk() from joining them while flow_has_room() is true.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=15
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 16/29] bpf: Describe locks at throw sites
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (14 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 15/29] bpf: Describe handles in registers " Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:04 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 17/29] bpf: Apply hidden slot limits only when a slot is needed Kumar Kartikeya Dwivedi
` (13 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently no site describes a lock, so every lock that a throw finds held
is kept in a hidden slot. Its address is stored before the call that
takes the lock, and cleared before the unlock.
A lock does not move, so a site can describe it:
- The lock of a global variable, or of the only element of any other
array map that the program refers to, is at an address known at
verification time. Its resource class carries the address. The site
entry is marked BPF_FRAME_ENT_FIXED and reads nothing from the frame.
This does not apply to an inner map: the map that stands for it
during verification has no values.
- Any other lock is found through a pointer to the map value or
allocated object that contains it. Such a pointer has the same id as
the lock. The acquisition records the offset of the lock in the
object, and the site gives a stack slot or a register that holds such
a pointer, with the difference as its adjustment. A lock whose offset
does not fit in an adjustment is not found through a pointer.
A lock at a fixed address then costs the program nothing when every
visit to a site finds it held. Any other lock costs nothing when, in
addition, the frame has a pointer to the containing object in the same
location on every visit. A lock that some visits to a site find held and
others do not still takes a hidden slot.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf.h | 4 ++
include/linux/bpf_verifier.h | 2 +
kernel/bpf/frame_desc.c | 77 +++++++++++++++++++++++++++++-------
kernel/bpf/unwind.c | 2 +
4 files changed, 70 insertions(+), 15 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index b157863cb952..cbb0d7a61f98 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1860,6 +1860,7 @@ struct bpf_resource {
union {
void (*dtor)(void *);
struct btf_struct_meta *meta;
+ void *addr; /* a lock at a fixed address, in the only element of an array map */
};
};
@@ -1880,6 +1881,8 @@ struct bpf_resource {
* bpf_arch_throw_entry();
* - with BPF_FRAME_ENT_OBJ the resource itself, an iterator or saved IRQ
* flags the frame has on its stack whenever it is at the site;
+ * - with BPF_FRAME_ENT_FIXED no location: a lock at the address its class
+ * gives, which the frame holds whenever it is at the site;
* - with BPF_FRAME_ENT_COUNTER a hidden slot that holds a count of sections,
* signed: the sections the frame entered less those of its callers it left;
* - with BPF_FRAME_ENT_COUNT no location: the count of the sections the frame
@@ -1891,6 +1894,7 @@ struct bpf_resource {
/* Released before the other entries of the frame, which it may be using. */
#define BPF_FRAME_ENT_FIRST BIT(3)
#define BPF_FRAME_ENT_REG BIT(4)
+#define BPF_FRAME_ENT_FIXED BIT(5)
struct bpf_frame_desc_entry {
u16 res; /* index into the program's resource classes */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 158e7eddc19f..a7c9b113e071 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1253,6 +1253,7 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
#define BPF_RES_F_MAY_FAIL BIT(1) /* acquired only when the call returns 0 */
#define BPF_RES_F_DEREF BIT(2) /* the handle is the first word of a dynptr on the stack */
#define BPF_RES_F_CTX BIT(3) /* a __ref argument the kernel passes the main program */
+#define BPF_RES_F_NO_PTR BIT(5) /* a lock that no pointer to what it is in leads to */
/*
* A call that acquires a resource, see frame_desc.c. When a frame can be
@@ -1301,6 +1302,7 @@ struct bpf_res_loc {
u16 acq; /* index of the acquisition */
u8 cnt; /* the places left */
bool obj; /* they are the resource itself, an object on the stack */
+ bool fixed; /* it is a lock at the address of its class, and needs no place */
struct {
s16 off; /* relative to R10, or the number of the register */
s16 adj; /* the pointer there points that many bytes into the object */
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index 5cb837688f90..72dc6597321a 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -376,6 +376,42 @@ static int resource_of_kind(struct bpf_verifier_env *env, u8 kind)
return resource_intern(env, &res);
}
+/*
+ * The class of the lock that the current insn takes through @reg, a pointer
+ * to it. A lock in the only element of an array map, which every global
+ * variable is, is at a fixed address: its class says where, and a site needs
+ * no place for it. That is so for a map the program refers to itself only:
+ * what stands for the inner maps of a map has no values. Any other lock is
+ * @off bytes into what the pointer is into, a map value or an object: a site
+ * finds it through a pointer to that which the frame keeps, unless the offset
+ * is too large for a site to tell, see BPF_RES_F_NO_PTR.
+ */
+static int resource_of_lock(struct bpf_verifier_env *env, const struct bpf_reference_state *ref,
+ const struct bpf_reg_state *reg, s16 *off, u8 *flags)
+{
+ struct bpf_resource res;
+ const struct bpf_map *map;
+ u64 base;
+ u32 i;
+
+ memset(&res, 0, sizeof(res));
+ res.kind = ref->type == REF_TYPE_LOCK ? BPF_RESOURCE_SPIN_LOCK : BPF_RESOURCE_RES_LOCK;
+ if (!reg || !tnum_is_const(reg->var_off) ||
+ reg->var_off.value != (u64)(s16)reg->var_off.value) {
+ *flags |= BPF_RES_F_NO_PTR;
+ return resource_intern(env, &res);
+ }
+ *off = reg->var_off.value;
+ map = reg->type == PTR_TO_MAP_VALUE ? reg->map_ptr : NULL;
+ for (i = 0; map && i < env->used_map_cnt; i++)
+ if (env->used_maps[i] == map)
+ break;
+ if (map && i < env->used_map_cnt && map->map_type == BPF_MAP_TYPE_ARRAY &&
+ map->ops->map_direct_value_addr && !map->ops->map_direct_value_addr(map, &base, *off))
+ res.addr = (void *)(long)(base + *off);
+ return resource_intern(env, &res);
+}
+
/* The offset from R10 of the stack slot @spi of a frame. */
static s16 spi_off(int spi)
{
@@ -397,6 +433,8 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
s16 *adj, s16 *obj_off)
{
const struct bpf_func_state *frame = st->frame[st->curframe];
+ /* a handle that is in no register can only be found by the sites */
+ bool in_reg = regno >= BPF_REG_0 && regno <= BPF_REG_5;
const struct bpf_stack_state *slot;
struct bpf_resource res;
struct bpf_reg_state reg;
@@ -414,8 +452,7 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
}
*flags |= BPF_RES_F_ARG;
if (ref->type & REF_TYPE_LOCK_MASK)
- return resource_of_kind(env, ref->type == REF_TYPE_LOCK ? BPF_RESOURCE_SPIN_LOCK :
- BPF_RESOURCE_RES_LOCK);
+ return resource_of_lock(env, ref, in_reg ? &frame->regs[regno] : NULL, adj, flags);
if (!stack_object_find(st, ref, &objframe, &spi))
return -EOPNOTSUPP;
if (objframe != st->curframe)
@@ -794,9 +831,11 @@ static const u8 site_regs[] = {
* The places of frame @f of @st that hold the handle of @ref, which the frame
* owns, in @loc: the stack object the reference is of, or the slots of the
* stack a pointer to the object is spilled in and the registers that hold
- * one, see site_regs; @throws when the frame calls bpf_throw(). A lock has
- * none. The slots a spill is removed from again, see
- * bpf_remove_fastcall_spills_fills(), hold nothing at run time.
+ * one, see site_regs; @throws when the frame calls bpf_throw(). A lock at a
+ * fixed address needs no place. Any other lock is found like an object,
+ * through a pointer to what it is in, which has the id of the lock. The slots
+ * a spill is removed from again, see bpf_remove_fastcall_spills_fills(), hold
+ * nothing at run time.
*/
static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifier_state *st, int f,
const struct bpf_reference_state *ref, bool throws,
@@ -805,42 +844,50 @@ static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifie
const struct bpf_res_acq *acq = &env->res_acqs[ref->acq - 1];
const struct bpf_func_state *frame = st->frame[f];
s16 adj, lim = env->subprog_info[frame->subprogno].fastcall_stack_end;
+ bool lock = ref->type & REF_TYPE_LOCK_MASK, ptr;
const struct bpf_stack_state *slot;
int k;
+ memset(loc->at, 0, sizeof(loc->at));
loc->cnt = 0;
loc->obj = false;
- if (ref->type & REF_TYPE_LOCK_MASK)
+ loc->fixed = lock && env->resources[acq->res].addr;
+ if (loc->fixed) {
+ loc->cnt = 1;
return;
+ }
+ /* a pointer the frame keeps: to the object, or to what a lock with an id is in */
+ ptr = lock ? ref->id && !(acq->flags & BPF_RES_F_NO_PTR) :
+ ref->type == REF_TYPE_PTR && !(acq->flags & BPF_RES_F_ARG);
for (k = 0; k < bpf_stack_nr_slots(frame) && loc->cnt < BPF_RES_LOC_MAX; k++) {
slot = bpf_stack_slot(frame, k);
adj = 0;
if (spi_off(k) < lim)
break;
- if (stack_slot_is_object(slot, ref->id)) {
+ if (!lock && stack_slot_is_object(slot, ref->id)) {
/* the handle of a dynptr is its first word, as when it was created */
if (slot->slot_type[BPF_REG_SIZE - 1] != STACK_DYNPTR) {
loc->obj = true;
loc->cnt = 0;
}
- } else if (ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG) ||
- !bpf_is_spilled_reg(slot) || !reg_ref_off(&slot->spilled_ptr, ref->id, &adj)) {
+ } else if (!ptr || !bpf_is_spilled_reg(slot) ||
+ !reg_ref_off(&slot->spilled_ptr, ref->id, &adj)) {
continue;
}
loc->at[loc->cnt].off = spi_off(k);
- loc->at[loc->cnt].reg = false;
- loc->at[loc->cnt++].adj = adj;
+ /* the acquisition of a lock has how far the lock is into what it is in */
+ loc->at[loc->cnt++].adj = lock ? adj - acq->adj : adj;
if (loc->obj)
return;
}
- if (!env->describe_regs || ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG))
+ if (!env->describe_regs || !ptr)
return;
for (k = 0; k < (throws ? ARRAY_SIZE(site_regs) : 4) && loc->cnt < BPF_RES_LOC_MAX; k++) {
if (!reg_ref_off(&frame->regs[site_regs[k]], ref->id, &adj))
continue;
loc->at[loc->cnt].off = site_regs[k];
loc->at[loc->cnt].reg = true;
- loc->at[loc->cnt++].adj = adj;
+ loc->at[loc->cnt++].adj = lock ? adj - acq->adj : adj;
}
}
@@ -908,7 +955,7 @@ static int site_visit(struct bpf_verifier_env *env, const struct bpf_verifier_st
if (cur.at[j].off == loc->at[k].off && cur.at[j].adj == loc->at[k].adj &&
cur.at[j].reg == loc->at[k].reg)
break;
- if (j < cur.cnt && cur.obj == loc->obj)
+ if (j < cur.cnt && cur.obj == loc->obj && cur.fixed == loc->fixed)
loc->at[n++] = loc->at[k];
}
loc->cnt = n;
@@ -1481,7 +1528,7 @@ static int build_site(struct bpf_verifier_env *env, u32 idx, const u32 *new_of)
for (i = 0; i < env->res_loc_cnt; i++) {
const struct bpf_res_loc *loc = &env->res_locs[i];
const struct bpf_res_acq *a = &env->res_acqs[loc->acq];
- u16 flags = loc->obj ? BPF_FRAME_ENT_OBJ : 0;
+ u16 flags = loc->obj ? BPF_FRAME_ENT_OBJ : loc->fixed ? BPF_FRAME_ENT_FIXED : 0;
if (loc->site != idx || a->slot)
continue;
diff --git a/kernel/bpf/unwind.c b/kernel/bpf/unwind.c
index 473897c3aff7..d3cd95f08769 100644
--- a/kernel/bpf/unwind.c
+++ b/kernel/bpf/unwind.c
@@ -233,6 +233,8 @@ static void bpf_unwind_release_desc(struct bpf_unwind_ctx *ctx, const struct bpf
continue;
if (ent->flags & BPF_FRAME_ENT_OBJ) {
handle = loc;
+ } else if (ent->flags & BPF_FRAME_ENT_FIXED) {
+ handle = res->addr;
} else {
/* no descriptor names a register that was not captured */
if ((ent->flags & BPF_FRAME_ENT_REG) && WARN_ON_ONCE(!ctx->entry))
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 16/29] bpf: Describe locks at throw sites
2026-11-11 12:57 ` [PATCH bpf-next v1 16/29] bpf: Describe locks " Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:04 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 6268 bytes --]
> bpf: Describe locks at throw sites
TL;DR:
- Possible s16 overflow of adj - acq->adj in ref_locations() may make
bpf_throw unlock a wrong address, corrupting kernel memory
- Possible new load rejections for programs taking different locks at one lock
insn
- Global locks past 32KB may lose their address (nit)
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> +static int resource_of_lock(struct bpf_verifier_env *env, const struct bpf_reference_state *ref,
> + const struct bpf_reg_state *reg, s16 *off, u8 *flags)
> +{
[ ... ]
> + memset(&res, 0, sizeof(res));
> + res.kind = ref->type == REF_TYPE_LOCK ? BPF_RESOURCE_SPIN_LOCK : BPF_RESOURCE_RES_LOCK;
> + if (!reg || !tnum_is_const(reg->var_off) ||
> + reg->var_off.value != (u64)(s16)reg->var_off.value) {
> + *flags |= BPF_RES_F_NO_PTR;
> + return resource_intern(env, &res);
> + }
> + *off = reg->var_off.value;
> + map = reg->type == PTR_TO_MAP_VALUE ? reg->map_ptr : NULL;
> + for (i = 0; map && i < env->used_map_cnt; i++)
> + if (env->used_maps[i] == map)
> + break;
> + if (map && i < env->used_map_cnt && map->map_type == BPF_MAP_TYPE_ARRAY &&
> + map->ops->map_direct_value_addr && !map->ops->map_direct_value_addr(map, &base, *off))
> + res.addr = (void *)(long)(base + *off);
> + return resource_intern(env, &res);
> +}
This isn't a bug, but the comment above resource_of_lock() and the commit
message both say that the lock of a global variable gets a class that gives
its address. The s16 test on reg->var_off.value runs before the used_maps
and map_direct_value_addr() lookup.
Doesn't that mean a global lock more than 32767 bytes into its section, for
example a lock placed after a large array in .bss, gets BPF_RES_F_NO_PTR and
no address? array_map_direct_value_addr() takes a u32 offset and a fixed
entry needs no adjustment, so the s16 limit would not seem to apply there.
Such a lock still takes a hidden slot and its inserted instructions.
[ ... ]
> @@ -414,8 +452,7 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
> *flags |= BPF_RES_F_ARG;
> if (ref->type & REF_TYPE_LOCK_MASK)
> - return resource_of_kind(env, ref->type == REF_TYPE_LOCK ? BPF_RESOURCE_SPIN_LOCK :
> - BPF_RESOURCE_RES_LOCK);
> + return resource_of_lock(env, ref, in_reg ? &frame->regs[regno] : NULL, adj, flags);
Before this patch every lock acquisition got the same class and an adj of 0.
Now the class carries the lock address for a global or single-element array
map lock, and the acquisition records the lock offset in adj and may set
BPF_RES_F_NO_PTR.
acq_add() still requires one bpf_spin_lock()/bpf_res_spin_lock() call insn
to have the same res, flags and adj on every path:
if (a->res != res || a->flags != flags || a->adj != adj || a->obj_off != obj_off) {
verbose(env, "frame_desc: insn %d acquires resources of different types on different paths\n",
Can this reject programs that loaded before? Any program that may throw
(describe_frames is set from might_throw) is rejected when one lock insn is
reached with locks of different objects, for example a static __noinline
helper that takes &lock_a and &lock_b from two private(A)/private(B)
sections, a global lock and a hash map element, or map values whose lock
offsets differ. This happens even if no throw ever finds the lock held.
The parent commit loads such programs, and the hidden slot does not depend
on the lock class since emit_acquire_before() stores R1 whatever the lock is.
The commit message does not mention a new rejection, and says a lock not
described at a site "still takes a hidden slot".
A later patch in the series, 0ed33b5c066c ("bpf: Apply hidden slot limits
only when a slot is needed"), makes acq_add() create one acquisition per
class, so the case where no throw finds the lock held loads again. At the
end of the series, a lock insn reached with two different lock objects that
a throw finds held still gets two acquisitions that each need a slot, and
slot_can_be_kept() rejects them with the same message.
[ ... ]
> @@ -805,42 +844,50 @@ static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifie
> loc->at[loc->cnt].off = spi_off(k);
> - loc->at[loc->cnt].reg = false;
> - loc->at[loc->cnt++].adj = adj;
> + /* the acquisition of a lock has how far the lock is into what it is in */
> + loc->at[loc->cnt++].adj = lock ? adj - acq->adj : adj;
[ ... ]
> loc->at[loc->cnt].off = site_regs[k];
> loc->at[loc->cnt].reg = true;
> - loc->at[loc->cnt++].adj = adj;
> + loc->at[loc->cnt++].adj = lock ? adj - acq->adj : adj;
Can adj - acq->adj overflow the s16 stored in at[].adj?
Only the two inputs are checked to fit in s16: adj by reg_ref_off(), and
acq->adj by the (u64)(s16) test in resource_of_lock(). The difference can
be as low as -65535 and is silently truncated when stored into at[].adj and
then into bpf_frame_desc_entry::adj.
A privileged program can keep a map value pointer far below the element.
With bypass_spec_v1, adjust_ptr_min_max_vals() and sanitize_check_bounds()
accept a constant r7 += -32768 on a PTR_TO_MAP_VALUE and keep its id:
r0 = lookup();
r7 = r0;
r7 += -32768;
r1 = r0;
r1 += 4;
call bpf_spin_lock;
r1 = 0;
call bpf_throw;
Here adj = -32768 and acq->adj = 4, so -32772 is stored as 32764. Every
visit agrees on R7, so the site gets an entry with no hidden slot. Then
bpf_unwind_release_desc() computes:
handle = val - ent->adj
which is value - 65532 instead of value + 4, and bpf_unwind_release()
calls bpf_spin_unlock() on that address. Doesn't this write into unrelated
kernel memory while the real lock stays held?
Before this patch locks never used site locations, and object locations use
adj directly, so they cannot overflow. Would skipping the place when the
difference does not fit in s16 fix it? The same expression is still in
ref_locations() at the end of the series, so no later fix was found.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 16/29] bpf: Describe locks at throw sites
2026-11-11 12:57 ` [PATCH bpf-next v1 16/29] bpf: Describe locks " Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 16:04 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 16:04 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] bpf: s16 overflow in ref_locations() when computing lock offset
adjustment
--
commit b280bb740c33eccf7a76373a4566fec46d049b84
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Describe locks at throw sites
This commit records held locks in throw site descriptors either by their fixed
address for single-element array maps or relative to a stack slot or register
holding a pointer to the containing object, avoiding hidden stack slots when
every visit to the site agrees on the lock.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index 5cb837688f90c..72dc6597321a9 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -805,42 +844,50 @@ static void ref_locations(struct bpf_verifier_env *env, const struct bpf_verifie
> const struct bpf_res_acq *acq = &env->res_acqs[ref->acq - 1];
> const struct bpf_func_state *frame = st->frame[f];
> s16 adj, lim = env->subprog_info[frame->subprogno].fastcall_stack_end;
> + bool lock = ref->type & REF_TYPE_LOCK_MASK, ptr;
> const struct bpf_stack_state *slot;
> int k;
>
> + memset(loc->at, 0, sizeof(loc->at));
> loc->cnt = 0;
> loc->obj = false;
> - if (ref->type & REF_TYPE_LOCK_MASK)
> + loc->fixed = lock && env->resources[acq->res].addr;
> + if (loc->fixed) {
> + loc->cnt = 1;
> return;
> + }
> + /* a pointer the frame keeps: to the object, or to what a lock with an id is in */
> + ptr = lock ? ref->id && !(acq->flags & BPF_RES_F_NO_PTR) :
> + ref->type == REF_TYPE_PTR && !(acq->flags & BPF_RES_F_ARG);
> for (k = 0; k < bpf_stack_nr_slots(frame) && loc->cnt < BPF_RES_LOC_MAX; k++) {
> slot = bpf_stack_slot(frame, k);
> adj = 0;
> if (spi_off(k) < lim)
> break;
> - if (stack_slot_is_object(slot, ref->id)) {
> + if (!lock && stack_slot_is_object(slot, ref->id)) {
> /* the handle of a dynptr is its first word, as when it was created */
> if (slot->slot_type[BPF_REG_SIZE - 1] != STACK_DYNPTR) {
> loc->obj = true;
> loc->cnt = 0;
> }
> - } else if (ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG) ||
> - !bpf_is_spilled_reg(slot) || !reg_ref_off(&slot->spilled_ptr, ref->id, &adj)) {
> + } else if (!ptr || !bpf_is_spilled_reg(slot) ||
> + !reg_ref_off(&slot->spilled_ptr, ref->id, &adj)) {
> continue;
> }
> loc->at[loc->cnt].off = spi_off(k);
> - loc->at[loc->cnt].reg = false;
> - loc->at[loc->cnt++].adj = adj;
> + /* the acquisition of a lock has how far the lock is into what it is in */
> + loc->at[loc->cnt++].adj = lock ? adj - acq->adj : adj;
> if (loc->obj)
> return;
> }
> - if (!env->describe_regs || ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG))
> + if (!env->describe_regs || !ptr)
> return;
> for (k = 0; k < (throws ? ARRAY_SIZE(site_regs) : 4) && loc->cnt < BPF_RES_LOC_MAX; k++) {
> if (!reg_ref_off(&frame->regs[site_regs[k]], ref->id, &adj))
> continue;
> loc->at[loc->cnt].off = site_regs[k];
> loc->at[loc->cnt].reg = true;
> - loc->at[loc->cnt++].adj = adj;
> + loc->at[loc->cnt++].adj = lock ? adj - acq->adj : adj;
> }
> }
[Severity: High]
Can adj - acq->adj underflow the s16 loc->at[].adj field in both loops?
resource_of_lock() checks that acq->adj fits in s16 (0..32767) and
reg_ref_off() checks that the pointer's constant var_off fits in s16
(-32768..32767), but when the spilled or register pointer has been adjusted to
a negative offset and acq->adj > 0 such that adj - acq->adj < S16_MIN, storing
the difference into loc->at[].adj wraps by +65536. On unwind,
bpf_unwind_release_desc() computes handle = (void *)(long)(val - ent->adj) and
passes an out-of-bounds address 65536 bytes below the lock to
bpf_spin_unlock() or bpf_res_spin_unlock(), corrupting kernel memory and
leaving the lock held.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=16
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 17/29] bpf: Apply hidden slot limits only when a slot is needed
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (15 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 16/29] bpf: Describe locks " Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 16:31 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller Kumar Kartikeya Dwivedi
` (12 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
The instructions that maintain a hidden slot are emitted once per call,
and a slot holds one handle. Currently a program that throws is rejected
for anything that these instructions cannot handle, as soon as the main
pass sees it:
- an instruction that acquires resources of different types, or takes
their handle in different ways, on different paths;
- an acquisition whose handle is not in a register;
- a release whose handle is not in an argument register, is in a
register that does not point to the released resource, or is taken in
different ways on different paths;
- a frame that holds two resources acquired by the same instruction.
None of this matters for a resource that gets no hidden slot, either
because its sites describe it or because no throw finds it held. Such a
program loads if it contains no throw, and a throw elsewhere in the
program is no reason to reject it.
Record these cases during the pass, and reject them after it, only for
the resources that need a hidden slot:
- An instruction gets one acquisition for each way it acquires.
- A release records why its handle cannot be cleared from a slot.
- An acquisition records that a frame held two of its resources.
Two resources of one instruction that a throw finds held have no
location that all visits agree on, so they always need a slot. That case
loads only if no throw finds them held.
The messages do not change.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 9 ++
kernel/bpf/frame_desc.c | 182 ++++++++++++++++++++++++-----------
2 files changed, 137 insertions(+), 54 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index a7c9b113e071..edd6005c32f6 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1272,6 +1272,7 @@ struct bpf_res_acq {
s16 off; /* the slot, relative to R10 */
bool held; /* some frame holds what it acquired at a site */
bool slot; /* and some site cannot say where: the handle is kept in a slot */
+ bool multi; /* a frame held two of what it acquired at once: one slot cannot */
};
/* How many places a site remembers a frame keeping one handle in. */
@@ -1317,8 +1318,16 @@ struct bpf_res_rel {
u8 reg; /* the argument the call releases */
u8 flags; /* BPF_RES_F_DEREF */
s16 adj; /* the argument points that many bytes into the object */
+ u8 bad; /* BPF_RES_REL_*: why no insns can clear a slot here */
};
+/* The handle is in no argument register. */
+#define BPF_RES_REL_NO_REG 1
+/* The register does not point to what is released. */
+#define BPF_RES_REL_NOT_PTR 2
+/* The call takes the handle in different ways on different paths. */
+#define BPF_RES_REL_DIFFER 3
+
/* A call that enters or leaves a section, for the count of its frame, see frame_desc.c. */
struct bpf_section_insn {
u32 insn;
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index 72dc6597321a..0ef7fda410ee 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -39,7 +39,11 @@
* - A frame that returns holds nothing but a reference it returns in R0,
* which the call acquires for the caller like the call of a kfunc that
* returns one, and is in the sections it was called in.
- * - A frame holds one resource of an acquiring call at a time.
+ * - A slot holds one handle, so a frame that holds two resources of one
+ * acquiring call at once can have no slot for them.
+ *
+ * What can have no slot loads when its sites describe it or no throw finds
+ * it held, and is refused otherwise.
*
* The main pass records every call that acquires or releases, which
* acquisition a release may be of, and the visits of the sites. What needs a
@@ -443,6 +447,8 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
*adj = 0;
*obj_off = 0;
if (after) {
+ if (!in_reg)
+ return -EOPNOTSUPP;
reg = frame->regs[regno];
if (!reg_ref_off(®, ref->id, adj))
*adj = 0;
@@ -473,8 +479,11 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
}
/*
- * The acquisition of insn @insn through R@reg: the same on every visit, since
- * the insns that keep its slot are emitted once. Returns its index.
+ * The acquisition of insn @insn through R@reg, of a resource of class @res.
+ * An insn that acquires in different ways on different paths has an
+ * acquisition for each: the insns that keep a slot are emitted once per insn,
+ * so none of them may need a slot then, see bpf_frame_desc_resolve(). Returns
+ * its index.
*/
static int acq_add(struct bpf_verifier_env *env, u32 insn, u16 res, u8 reg, u8 flags, s16 adj,
s16 obj_off)
@@ -484,14 +493,9 @@ static int acq_add(struct bpf_verifier_env *env, u32 insn, u16 res, u8 reg, u8 f
for (i = 0; i < cnt; i++) {
a = &env->res_acqs[i];
- if (a->insn != insn || a->reg != reg || (a->flags ^ flags) & BPF_RES_F_CTX)
- continue;
- if (a->res != res || a->flags != flags || a->adj != adj || a->obj_off != obj_off) {
- verbose(env, "frame_desc: insn %d acquires resources of different types on different paths\n",
- insn);
- return -EINVAL;
- }
- return i;
+ if (a->insn == insn && a->reg == reg && a->res == res && a->flags == flags &&
+ a->adj == adj && a->obj_off == obj_off)
+ return i;
}
if (cnt >= U16_MAX - 1)
return -E2BIG;
@@ -507,7 +511,9 @@ static int acq_add(struct bpf_verifier_env *env, u32 insn, u16 res, u8 reg, u8 f
/*
* refs[@idx] is what acquisition @acq acquired, held by the frame that owns
- * it: its handle is in the slot of the acquisition, and the slot holds one.
+ * it. The slot of an acquisition holds one handle: a frame that holds two of
+ * what it acquired at once is fine as long as the acquisition gets no slot,
+ * see bpf_frame_desc_resolve().
*/
static int ref_set_acq(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int idx,
int acq)
@@ -518,9 +524,8 @@ static int ref_set_acq(struct bpf_verifier_env *env, struct bpf_verifier_state *
for (i = 0; i < st->acquired_refs; i++) {
if (i == idx || st->refs[i].acq != acq + 1 || st->refs[i].frameno != ref->frameno)
continue;
- verbose(env, "frame_desc: a frame holds two resources of insn %d at once, which is not supported in a program that throws\n",
- env->res_acqs[acq].insn);
- return -EINVAL;
+ env->res_acqs[acq].multi = true;
+ break;
}
ref->res = env->res_acqs[acq].res + 1;
ref->acq = acq + 1;
@@ -561,11 +566,9 @@ int bpf_frame_desc_record_acquire(struct bpf_verifier_env *env, int idx, int reg
}
if (res < 0)
return res;
- if (regno < BPF_REG_0 || regno > BPF_REG_5) {
- verbose(env, "frame_desc: insn %d takes the handle of a resource in no register\n",
- env->insn_idx);
- return -EINVAL;
- }
+ /* no insns can put it into a slot then: R10 stands for no register */
+ if (regno < BPF_REG_0 || regno > BPF_REG_5)
+ regno = BPF_REG_FP;
acq = acq_add(env, env->insn_idx, res, regno, flags, adj, obj_off);
return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
}
@@ -630,6 +633,48 @@ int bpf_frame_desc_record_ctx_arg(struct bpf_verifier_env *env, int idx, int arg
return ret < 0 ? ret : ref_set_acq(env, st, idx, ret);
}
+/*
+ * The call @insn releases through R@reg what acquisition @acq acquired: its
+ * handle leaves the slot of the acquisition in front of the call, when it has
+ * one. @bad says why no insns can do that here. The insns compare the
+ * argument the same way whatever it releases, so releases of one argument
+ * that differ in how they take the handle are bad as well. That matters for
+ * an acquisition that has a slot only, see bpf_frame_desc_resolve().
+ */
+static int rel_add(struct bpf_verifier_env *env, u32 insn, u16 acq, u8 reg, u8 flags, s16 adj,
+ u8 bad)
+{
+ u32 i, cnt = env->res_rel_cnt;
+ struct bpf_res_rel *r, *same = NULL;
+
+ for (i = 0; i < cnt; i++) {
+ r = &env->res_rels[i];
+ if (r->insn != insn || r->reg != reg)
+ continue;
+ if (r->flags != flags || r->adj != adj) {
+ if (!r->bad)
+ r->bad = BPF_RES_REL_DIFFER;
+ if (!bad)
+ bad = BPF_RES_REL_DIFFER;
+ } else if (r->acq == acq) {
+ same = r;
+ }
+ }
+ if (same) {
+ if (!same->bad)
+ same->bad = bad;
+ return 0;
+ }
+ r = kvrealloc(env->res_rels, array_size(cnt + 1, sizeof(*r)), GFP_KERNEL_ACCOUNT);
+ if (!r)
+ return -ENOMEM;
+ r[cnt] = (struct bpf_res_rel) { .insn = insn, .acq = acq, .reg = reg, .flags = flags,
+ .adj = adj, .bad = bad };
+ env->res_rels = r;
+ env->res_rel_cnt = cnt + 1;
+ return 0;
+}
+
/*
* Record that the current insn releases the resource refs[@idx], before it
* is released: its handle, in the argument the call releases, see
@@ -642,11 +687,10 @@ int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
struct bpf_verifier_state *st = env->cur_state;
const struct bpf_reference_state *ref = &st->refs[idx];
const struct bpf_func_state *frame = st->frame[st->curframe];
- u32 i, cnt = env->res_rel_cnt, insn = env->insn_idx;
+ u32 i, insn = env->insn_idx;
int regno = env->release_regno;
const struct bpf_res_acq *acq;
- struct bpf_res_rel *r;
- u8 flags = 0;
+ u8 flags = 0, bad = 0;
s16 adj = 0;
if (!env->describe_frames || st->speculative)
@@ -667,40 +711,17 @@ int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
return -EINVAL;
}
acq = &env->res_acqs[ref->acq - 1];
+ /* what insns could not compare with a slot matters when there is a slot only */
if (regno < BPF_REG_1 || regno > BPF_REG_5) {
- verbose(env, "frame_desc: insn %d releases a resource whose handle is in no argument register\n",
- insn);
- return -EINVAL;
- }
- if (acq->flags & BPF_RES_F_DEREF) {
+ regno = BPF_REG_FP;
+ bad = BPF_RES_REL_NO_REG;
+ } else if (acq->flags & BPF_RES_F_DEREF) {
flags = BPF_RES_F_DEREF;
} else if (!(acq->flags & BPF_RES_F_ARG) &&
!reg_ref_off(&frame->regs[regno], ref->id, &adj)) {
- verbose(env, "frame_desc: insn %d releases a resource R%d does not point to\n", insn,
- regno);
- return -EINVAL;
- }
- for (i = 0; i < cnt; i++) {
- r = &env->res_rels[i];
- if (r->insn != insn || r->reg != regno)
- continue;
- /* the argument is taken the same way whatever it releases */
- if (r->flags != flags || r->adj != adj) {
- verbose(env, "frame_desc: insn %d takes the handle of what it releases differently on different paths\n",
- insn);
- return -EINVAL;
- }
- if (r->acq == ref->acq - 1)
- return 0;
+ bad = BPF_RES_REL_NOT_PTR;
}
- r = kvrealloc(env->res_rels, array_size(cnt + 1, sizeof(*r)), GFP_KERNEL_ACCOUNT);
- if (!r)
- return -ENOMEM;
- r[cnt] = (struct bpf_res_rel) { .insn = insn, .acq = ref->acq - 1, .reg = regno,
- .flags = flags, .adj = adj };
- env->res_rels = r;
- env->res_rel_cnt = cnt + 1;
- return 0;
+ return rel_add(env, insn, ref->acq - 1, regno, flags, adj, bad);
}
/* The current insn enters (@delta 1) or leaves (-1) a section of @kind: counted by its frame. */
@@ -1095,6 +1116,50 @@ static int emit_acquire_after(struct bpf_insn *buf, const struct bpf_res_acq *a)
return cnt;
}
+/*
+ * Acquisition @acq needs a slot. Can the insns that keep it be emitted? They
+ * are emitted once per insn, so the insn has to acquire in one way, with the
+ * handle in a register, and every call that releases what it acquired has to
+ * take the handle in one way that insns can compare with the slot.
+ */
+static int slot_can_be_kept(struct bpf_verifier_env *env, u32 acq)
+{
+ const struct bpf_res_acq *a = &env->res_acqs[acq], *b;
+ const struct bpf_res_rel *r;
+ u32 i;
+
+ for (i = 0; i < env->res_acq_cnt; i++) {
+ b = &env->res_acqs[i];
+ if (i == acq || b->insn != a->insn || b->reg != a->reg ||
+ (a->flags ^ b->flags) & BPF_RES_F_CTX)
+ continue;
+ verbose(env, "frame_desc: insn %d acquires resources of different types on different paths\n",
+ a->insn);
+ return -EINVAL;
+ }
+ if (a->reg == BPF_REG_FP) {
+ verbose(env, "frame_desc: insn %d takes the handle of a resource in no register\n",
+ a->insn);
+ return -EINVAL;
+ }
+ for (i = 0; i < env->res_rel_cnt; i++) {
+ r = &env->res_rels[i];
+ if (r->acq != acq || !r->bad)
+ continue;
+ if (r->bad == BPF_RES_REL_NO_REG)
+ verbose(env, "frame_desc: insn %d releases a resource whose handle is in no argument register\n",
+ r->insn);
+ else if (r->bad == BPF_RES_REL_NOT_PTR)
+ verbose(env, "frame_desc: insn %d releases a resource R%d does not point to\n",
+ r->insn, r->reg);
+ else
+ verbose(env, "frame_desc: insn %d takes the handle of what it releases differently on different paths\n",
+ r->insn);
+ return -EINVAL;
+ }
+ return 0;
+}
+
/*
* Decide what needs a hidden slot, once the pass has seen every frame, and
* reserve it.
@@ -1149,6 +1214,15 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
}
if (!a->slot)
continue;
+ ret = slot_can_be_kept(env, i);
+ if (ret)
+ break;
+ if (a->multi) {
+ verbose(env, "frame_desc: a frame holds two resources of insn %d at once, which is not supported in a program that throws\n",
+ a->insn);
+ ret = -EINVAL;
+ break;
+ }
/* some visit of the insn has nothing a slot can hold where this one has */
if (!(a->flags & BPF_RES_F_CTX) && acquires_sometimes(env, new_of[a->insn], a)) {
verbose(env, "frame_desc: insn %d acquires a resource on some paths only, or one of a type that cannot be released\n",
@@ -1348,8 +1422,8 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
if (!env->res_acq_cnt && !env->section_insn_cnt)
return 0;
new_of = insns_of_pass(env, &orig_len);
- items = kvcalloc(env->res_acq_cnt + env->res_rel_cnt + env->section_insn_cnt, sizeof(*items),
- GFP_KERNEL_ACCOUNT);
+ items = kvcalloc(env->res_acq_cnt + env->res_rel_cnt + env->section_insn_cnt,
+ sizeof(*items), GFP_KERNEL_ACCOUNT);
if (!new_of || !items) {
ret = -ENOMEM;
goto out;
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 17/29] bpf: Apply hidden slot limits only when a slot is needed
2026-11-11 12:57 ` [PATCH bpf-next v1 17/29] bpf: Apply hidden slot limits only when a slot is needed Kumar Kartikeya Dwivedi
@ 2026-11-11 16:31 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 16:31 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] bpf: multi-resource check bypass for fixed locks in
bpf_frame_desc_resolve()
--
commit 744e9327531c3db28e3b5167eeaeeddb7a4859e0
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Apply hidden slot limits only when a slot is needed
This commit defers hidden stack slot capability checks from the main
verification pass to bpf_frame_desc_resolve(), rejecting incompatible
acquisitions and releases only when the resource actually requires a hidden
slot.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index 72dc6597321a9..0ef7fda410ee1 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -507,7 +511,9 @@ static int acq_add(struct bpf_verifier_env *env, u32 insn, u16 res, u8 reg, u8 f
>
> /*
> * refs[@idx] is what acquisition @acq acquired, held by the frame that owns
> - * it: its handle is in the slot of the acquisition, and the slot holds one.
> + * it. The slot of an acquisition holds one handle: a frame that holds two of
> + * what it acquired at once is fine as long as the acquisition gets no slot,
> + * see bpf_frame_desc_resolve().
> */
> static int ref_set_acq(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int idx,
> int acq)
> @@ -518,9 +524,8 @@ static int ref_set_acq(struct bpf_verifier_env *env, struct bpf_verifier_state *
> for (i = 0; i < st->acquired_refs; i++) {
> if (i == idx || st->refs[i].acq != acq + 1 || st->refs[i].frameno != ref->frameno)
> continue;
> - verbose(env, "frame_desc: a frame holds two resources of insn %d at once, which is not supported in a program that throws\n",
> - env->res_acqs[acq].insn);
> - return -EINVAL;
> + env->res_acqs[acq].multi = true;
> + break;
> }
> ref->res = env->res_acqs[acq].res + 1;
> ref->acq = acq + 1;
[ ... ]
> @@ -1149,6 +1214,15 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
> }
> if (!a->slot)
> continue;
> + ret = slot_can_be_kept(env, i);
> + if (ret)
> + break;
> + if (a->multi) {
> + verbose(env, "frame_desc: a frame holds two resources of insn %d at once, which is not supported in a program that throws\n",
> + a->insn);
> + ret = -EINVAL;
> + break;
> + }
> /* some visit of the insn has nothing a slot can hold where this one has */
> if (!(a->flags & BPF_RES_F_CTX) && acquires_sometimes(env, new_of[a->insn], a)) {
> verbose(env, "frame_desc: insn %d acquires a resource on some paths only, or one of a type that cannot be released\n",
[Severity: High]
Can a fixed-address lock bypass the a->multi check when a throw site sees
two references on one path and none on another?
For a bpf_res_spin_lock in a single-element array map, two lookups give
distinct reg->id values for the same lock address, so calling
bpf_res_spin_lock() at the same instruction with both pointers records two
references with the same ref->acq and sets a->multi = true. When that
state reaches a throw site, ref_locations() sets cur.fixed = true and
cur.cnt = 1 for both references without checking ref->id, and site_visit()
increments loc->visits once per matching reference in st->acquired_refs:
kernel/bpf/frame_desc.c:site_visit() {
...
site->visits++;
for (i = 0; i < st->acquired_refs; i++) {
...
loc = &env->res_locs[j];
loc->visits++;
}
If a second path reaches the same throw site holding neither reference,
loc->visits and site->visits are both 2 with loc->cnt == 1, so a->slot
stays false. bpf_frame_desc_resolve() then skips the a->multi check, and
build_site() emits a BPF_FRAME_ENT_FIXED descriptor entry that unlocks the
unheld lock when unwinding from that site on the second path.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=17
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (16 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 17/29] bpf: Apply hidden slot limits only when a slot is needed Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:41 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 19/29] bpf: Allow a callee to release its caller's resources Kumar Kartikeya Dwivedi
` (11 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
A hidden slot belongs to the frame that made the acquiring call.
Currently a program that throws is rejected whenever a resource ends up
held by another frame:
- a subprog returns holding anything other than a reference in R0, for
example a lock that it took for its caller;
- a subprog returns in more or fewer RCU read or preemption disabled
sections than it was called in;
- a callee creates an iterator or saves IRQ flags on its caller's stack.
These programs are rejected even if no throw can find the resource held,
although they load today. An example is one subprog that takes a lock and
another that drops it, with an assertion elsewhere in the program.
Reject them only if a hidden slot is needed:
- A resource that a subprog returns holding belongs to its caller from
then on, under the acquisition that the subprog made. That acquisition
cannot have a hidden slot. The program loads if the caller's sites
describe the resource, or if no throw finds it held.
- A stack object belongs to the frame whose stack contains it, whichever
frame created it. Its sites find it there, so it needs no slot unless
some visits to a site find it held and others do not.
- A callee that returns in different sections than it was called in
leaves the difference to its caller, whose sites give the number of
sections. The caller then cannot count its sections in a hidden slot.
A function verified on its own reports this to its caller in
bpf_flow_return().
A lock at a fixed address, an iterator and saved IRQ flags are always
where a site finds them. Subprogs that take and drop these for their
callers therefore load if every visit to a throw finds them held, or
none does.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 15 ++++
kernel/bpf/frame_desc.c | 160 +++++++++++++++++++++++------------
kernel/bpf/join.c | 2 +-
kernel/bpf/states.c | 2 +-
kernel/bpf/verifier.c | 4 +
5 files changed, 125 insertions(+), 58 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index edd6005c32f6..9a0ea25ab5c2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1112,6 +1112,12 @@ struct bpf_subprog_info {
bool stack_args_from_code: 1;
/* it releases what its caller holds, so it's verified in the context of such caller */
bool releases_inherited: 1;
+ /*
+ * The kinds of sections, a bit each, that a callee returned to it in
+ * more or fewer of than it called the callee in: a count it keeps in a
+ * slot does not follow that, see frame_desc.c.
+ */
+ u8 section_delta;
/* it loads the end of the packet from the context */
bool loads_pkt_end: 1;
/* a call of it was verified on its own: its insns are not in insns_total of the caller */
@@ -1272,6 +1278,13 @@ struct bpf_res_acq {
s16 off; /* the slot, relative to R10 */
bool held; /* some frame holds what it acquired at a site */
bool slot; /* and some site cannot say where: the handle is kept in a slot */
+ /*
+ * An insn, plus one, that leaves what it acquired with another frame
+ * than the one that ran the acquisition: the call a subprog returns
+ * holding it from, or the acquisition itself when it creates a stack
+ * object in the stack of a caller. A slot cannot follow it there.
+ */
+ u32 elsewhere;
bool multi; /* a frame held two of what it acquired at once: one slot cannot */
};
@@ -2345,6 +2358,8 @@ int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx);
int bpf_frame_desc_record_ctx_arg(struct bpf_verifier_env *env, int idx, int argno);
int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delta);
int bpf_frame_desc_record_exit(struct bpf_verifier_env *env);
+void bpf_frame_desc_record_sections(struct bpf_verifier_env *env, int subprog, bool rcu,
+ bool preempt);
bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
int f, const struct bpf_reg_state *reg);
u16 bpf_frame_desc_kept_regs(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index 0ef7fda410ee..9854be9ab478 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -36,9 +36,14 @@
*
* - A frame releases what it acquired itself. A callee cannot reach the
* slot of its caller, so it may not release what its caller holds.
- * - A frame that returns holds nothing but a reference it returns in R0,
- * which the call acquires for the caller like the call of a kfunc that
- * returns one, and is in the sections it was called in.
+ * - A slot goes with its frame. A reference a subprog returns in R0 is
+ * acquired by the call for the caller, like the call of a kfunc that
+ * returns one.
+ * Whatever else a frame comes to hold of what another frame acquired can
+ * have no slot: a lock a subprog returns holding, or a stack object a
+ * callee creates in the stack of its caller. Likewise a subprog whose
+ * callee returns in other sections than it was called in cannot count
+ * its sections in a slot.
* - A slot holds one handle, so a frame that holds two resources of one
* acquiring call at once can have no slot for them.
*
@@ -429,12 +434,13 @@ static s16 spi_off(int spi)
* front of it, the pointer the call takes to a lock or to a stack object. The
* handle of a dynptr is the record or the reader, the first word of the
* dynptr at @obj_off, so that a clone releases it as well as the dynptr it was
- * created in. -EOPNOTSUPP for a resource the unwinder cannot release, -EXDEV
- * for a stack object in the stack of another frame.
+ * created in. @owner is the frame that has a stack object in its stack, which
+ * may be a caller of the current one. -EOPNOTSUPP for a resource the unwinder
+ * cannot release.
*/
static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
const struct bpf_reference_state *ref, int regno, bool after, u8 *flags,
- s16 *adj, s16 *obj_off)
+ s16 *adj, s16 *obj_off, int *owner)
{
const struct bpf_func_state *frame = st->frame[st->curframe];
/* a handle that is in no register can only be found by the sites */
@@ -446,6 +452,7 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
*adj = 0;
*obj_off = 0;
+ *owner = st->curframe;
if (after) {
if (!in_reg)
return -EOPNOTSUPP;
@@ -461,13 +468,13 @@ static int acquired_class(struct bpf_verifier_env *env, struct bpf_verifier_stat
return resource_of_lock(env, ref, in_reg ? &frame->regs[regno] : NULL, adj, flags);
if (!stack_object_find(st, ref, &objframe, &spi))
return -EOPNOTSUPP;
- if (objframe != st->curframe)
- return -EXDEV;
+ *owner = objframe;
slot = bpf_stack_slot(st->frame[objframe], spi);
switch (slot->slot_type[BPF_REG_SIZE - 1]) {
case STACK_DYNPTR:
*flags |= BPF_RES_F_DEREF;
- *obj_off = spi_off(spi);
+ /* a slot is filled from the stack of the frame that runs the insn only */
+ *obj_off = objframe == st->curframe ? spi_off(spi) : 0;
return resource_of_kind(env, slot->spilled_ptr.dynptr.type == BPF_DYNPTR_TYPE_FILE ?
BPF_RESOURCE_DYNPTR_FILE : BPF_RESOURCE_RINGBUF);
case STACK_IRQ_FLAG:
@@ -547,30 +554,37 @@ int bpf_frame_desc_record_acquire(struct bpf_verifier_env *env, int idx, int reg
struct bpf_reference_state *ref = &st->refs[idx];
u8 flags = may_fail ? BPF_RES_F_MAY_FAIL : 0;
s16 adj, obj_off;
- int res, acq;
+ int res, acq, owner;
ref->res = 0;
ref->acq = 0;
if (!env->describe_frames || st->speculative)
return 0;
- res = acquired_class(env, st, ref, regno, after, &flags, &adj, &obj_off);
+ res = acquired_class(env, st, ref, regno, after, &flags, &adj, &obj_off, &owner);
if (res == -EOPNOTSUPP) {
/* the insns of another visit must not take this for a resource of theirs */
env->insn_aux_data[env->insn_idx].res_none = true;
return 0;
}
- if (res == -EXDEV) {
- verbose(env, "frame_desc: insn %d creates an object in the stack of another frame, which is not supported in a program that throws\n",
- env->insn_idx);
- return -EINVAL;
- }
if (res < 0)
return res;
/* no insns can put it into a slot then: R10 stands for no register */
if (regno < BPF_REG_0 || regno > BPF_REG_5)
regno = BPF_REG_FP;
acq = acq_add(env, env->insn_idx, res, regno, flags, adj, obj_off);
- return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
+ if (acq < 0)
+ return acq;
+ /*
+ * A stack object is held by the frame that has it in its stack, which
+ * its sites find it in: the slot of a callee that created it there
+ * would be gone with the callee.
+ */
+ if (owner != st->curframe) {
+ ref->frameno = owner;
+ if (!env->res_acqs[acq].elsewhere)
+ env->res_acqs[acq].elsewhere = env->insn_idx + 1;
+ }
+ return ref_set_acq(env, st, idx, acq);
}
/*
@@ -748,71 +762,89 @@ int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delt
/*
* The current frame returns to its caller. Its slots and its counts of
* sections go with the frame, and nothing moves them to the frame it returns
- * to, so it returns holding nothing but a reference in R0, which the call
- * acquires for the caller, see bpf_frame_desc_record_return(), and in as many
- * sections as it was called in. The entry counts of the sections are lowered
- * as the frame leaves its callers' sections, see release_context_state(),
- * which counts those apart.
+ * to. What it returns holding is its caller's from then on:
+ *
+ * - A reference in R0 the call acquires for the caller, like the call of a
+ * kfunc that returns one, see bpf_frame_desc_record_return().
+ * - Anything else, a lock the frame took for its caller, stays what it was
+ * acquired as, and can have no slot: the sites of the caller have to say
+ * where it is, or no throw may find it held, see
+ * bpf_frame_desc_record_return().
+ *
+ * A frame that returns in more or fewer sections than it was called in leaves
+ * the difference to its caller, whose sites say how many it is in then: a
+ * count the caller keeps in a slot would not know. The entry counts of the
+ * sections are lowered as the frame leaves its callers' sections, see
+ * release_context_state(), which counts those apart.
*/
int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
{
struct bpf_verifier_state *st = env->cur_state;
const struct bpf_func_state *frame = st->frame[st->curframe];
- int i, held = 0;
- s16 adj;
if (!env->describe_frames || st->speculative)
return 0;
- if (st->active_rcu_locks != frame->entry_rcu_locks || frame->rcu_released_below ||
- st->active_preempt_locks != frame->entry_preempt_locks || frame->preempt_released_below) {
- verbose(env, "frame_desc: a subprog returns in other sections than it was called in at insn %d, which is not supported in a program that throws\n",
- env->insn_idx);
- return -EINVAL;
- }
- for (i = 0; i < st->acquired_refs; i++) {
- const struct bpf_reference_state *ref = &st->refs[i];
-
- if (!ref->res || ref->frameno != st->curframe)
- continue;
- if (ref->type == REF_TYPE_PTR && !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
- reg_ref_off(&frame->regs[BPF_REG_0], ref->id, &adj) && !held++)
- continue;
- verbose(env, "frame_desc: a subprog returns holding what R0 does not carry at insn %d, which is not supported in a program that throws\n",
- env->insn_idx);
- return -EINVAL;
- }
+ /* a function verified on its own has its caller told in bpf_flow_return() */
+ if (st->curframe)
+ bpf_frame_desc_record_sections(env, st->frame[st->curframe - 1]->subprogno,
+ st->active_rcu_locks != frame->entry_rcu_locks +
+ frame->rcu_released_below,
+ st->active_preempt_locks != frame->entry_preempt_locks +
+ frame->preempt_released_below);
return 0;
}
/*
* A subprog called at insn @callsite returned to the current frame of @st
- * with @r0. The reference refs[@idx] it returned holding is the caller's
- * from here on, acquired by the call like the reference a kfunc returns; with
- * @idx < 0 it returned holding nothing, and what it left in R0 must not be
- * taken for a reference by the insns another visit has follow the call.
+ * with @r0, holding the reference refs[@idx], which is the caller's from here
+ * on, or nothing with @idx < 0.
+ *
+ * A reference that R0 carries is acquired by the call, like the reference a
+ * kfunc returns. Anything else stays what it was acquired as, in the subprog:
+ * the slot of that acquisition is gone with the frame of the subprog, so it
+ * can have none, see bpf_frame_desc_resolve(). And what the subprog left in
+ * R0 then must not be taken for a reference by the insns another visit has
+ * follow the call.
*/
int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
int idx, int callsite, const struct bpf_reg_state *r0)
{
struct bpf_reference_state *ref;
+ struct bpf_res_acq *held;
s16 adj = 0;
int acq;
if (!env->describe_frames || st->speculative)
return 0;
ref = idx < 0 ? NULL : &st->refs[idx];
- if (!ref || !ref->res) {
- if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
- env->insn_aux_data[callsite].res_none = true;
- return 0;
+ if (ref && !ref->res)
+ ref = NULL;
+ if (ref && ref->type == REF_TYPE_PTR && ref->id &&
+ !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
+ reg_ref_off(r0, ref->id, &adj)) {
+ acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
+ return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
}
- if (!reg_ref_off(r0, ref->id, &adj)) {
- verbose(env, "frame_desc: the subprog called at insn %d returns holding what R0 does not carry, which is not supported in a program that throws\n",
- callsite);
- return -EINVAL;
+ if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
+ env->insn_aux_data[callsite].res_none = true;
+ if (ref) {
+ held = &env->res_acqs[ref->acq - 1];
+ if (!held->elsewhere)
+ held->elsewhere = callsite + 1;
}
- acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
- return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
+ return 0;
+}
+
+/*
+ * A callee returned to @subprog in more or fewer RCU read or preemption
+ * disabled sections than it was called in, see bpf_frame_desc_record_exit().
+ */
+void bpf_frame_desc_record_sections(struct bpf_verifier_env *env, int subprog, bool rcu,
+ bool preempt)
+{
+ if (!env->describe_frames)
+ return;
+ env->subprog_info[subprog].section_delta |= (rcu ? BIT(0) : 0) | (preempt ? BIT(1) : 0);
}
/*
@@ -1020,6 +1052,8 @@ void bpf_frame_desc_reset(struct bpf_verifier_env *env)
env->section_insn_cnt = 0;
env->res_site_cnt = 0;
env->res_loc_cnt = 0;
+ for (i = 0; i < env->subprog_cnt; i++)
+ env->subprog_info[i].section_delta = 0;
for (i = 0; i < env->prog->len; i++) {
env->insn_aux_data[i].res_none = false;
env->insn_aux_data[i].res_none_r0 = false;
@@ -1217,6 +1251,13 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
ret = slot_can_be_kept(env, i);
if (ret)
break;
+ /* the slot is in the frame that runs the insn, not in the one that holds this */
+ if (a->elsewhere) {
+ verbose(env, "frame_desc: what insn %d acquires is held by another frame from insn %d on, where a throw finds it in no place its paths agree on, which is not supported in a program that throws\n",
+ a->insn, a->elsewhere - 1);
+ ret = -EINVAL;
+ break;
+ }
if (a->multi) {
verbose(env, "frame_desc: a frame holds two resources of insn %d at once, which is not supported in a program that throws\n",
a->insn);
@@ -1245,6 +1286,13 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
for (k = 0; k < ARRAY_SIZE(sub->section_off) && !ret; k++) {
if (!site->sections_vary[k] || sub->section_off[k])
continue;
+ /* the count would miss what a callee returned in and did not leave */
+ if (sub->section_delta & BIT(k)) {
+ verbose(env, "frame_desc: a subprog returns in other sections than it was called in, to one that counts its sections in a hidden slot for insn %d, which is not supported in a program that throws\n",
+ site->insn);
+ ret = -EINVAL;
+ break;
+ }
ret = bpf_reserve_stack_slots(env, sub - env->subprog_info, 1, &sub->section_off[k]);
env->frame_desc_slots++;
}
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 180b0a96029c..87d87d359312 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -463,7 +463,7 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
fa->entry_rcu_locks != fb->entry_rcu_locks ||
fa->entry_preempt_locks != fb->entry_preempt_locks)
return false;
- /* a frame that left a section of its callers hands its count over, see frame_desc.c */
+ /* the sections a frame left of those it was entered in matter to a throw, see frame_desc.c */
if (env->describe_frames && (fa->rcu_released_below != fb->rcu_released_below ||
fa->preempt_released_below != fb->preempt_released_below))
return false;
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 5a773bfb9e84..01eb8d9077ef 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -717,7 +717,7 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
old->entry_rcu_locks != cur->entry_rcu_locks ||
old->entry_preempt_locks != cur->entry_preempt_locks)
return false;
- /* a frame that left a section of its callers hands its count over, see frame_desc.c */
+ /* the sections a frame left of those it was entered in matter to a throw, see frame_desc.c */
if (env->describe_frames && (old->rcu_released_below != cur->rcu_released_below ||
old->preempt_released_below != cur->preempt_released_below))
return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b32c5a4f844f..a710da1a15c6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -12028,6 +12028,10 @@ int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *cal
}
caller->active_irq_id = fresh_id(env, idmap, caller->active_irq_id);
caller->active_lock_id = fresh_id(env, idmap, caller->active_lock_id);
+ /* the sections the function entered and did not leave are the caller's, see frame_desc.c */
+ bpf_frame_desc_record_sections(env, frame->subprogno,
+ caller->active_rcu_locks != caller_rcu,
+ caller->active_preempt_locks != caller_preempt);
/* fresh_id() had no room for an id, or there are no ids left */
if (idmap->cnt == BPF_ID_MAP_SIZE || env->id_gen >= BPF_MAX_ID) {
verbose(env, "too many ids at the call at insn %d\n", call_idx);
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller
2026-11-11 12:57 ` [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:41 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 9033 bytes --]
> bpf: Allow a callee to leave resources with its caller
TL;DR:
- Possible IRQs left disabled after bpf_throw: unwinder may restore
callee-created caller-stack IRQ flags in wrong order
- Commit message claims a lock taken in one subprog and dropped in another now
loads, but it is still rejected
- Possible spurious rejection when a subprog returns a reference while holding
a lock
> bpf: Allow a callee to leave resources with its caller
>
> [ ... ]
> These programs are rejected even if no throw can find the resource held,
> although they load today. An example is one subprog that takes a lock and
> another that drops it, with an assertion elsewhere in the program.
> [ ... ]
> A lock at a fixed address, an iterator and saved IRQ flags are always
> where a site finds them. Subprogs that take and drop these for their
> callers therefore load if every visit to a throw finds them held, or
> none does.
The commit message presents "one subprog that takes a lock and another that
drops it" as something this patch makes loadable. Is that accurate at this
commit?
After take_wlock() returns, prepare_func_exit() sets the lock reference's
frameno to the caller. When drop_wlock() then calls bpf_spin_unlock(),
process_spin_lock() -> bpf_frame_desc_record_release() still runs the
unchanged check:
if (ref->frameno != st->curframe) {
verbose(env, "frame_desc: insn %d releases what another frame holds, which is not supported in a program that throws\n",
insn);
return -EINVAL;
}
With inline verification this compares frameno 0 against curframe 1, and with
independent verification it compares -1 against curframe 0, so the program
is rejected either way. The same applies to an iterator, IRQ flags or a
dynptr that a callee creates for its caller and a different subprog later
destroys.
The lock_wrappers selftest added in d084772089f5 only seems to load after
0bbb9cb47a2c "bpf: Allow a callee to release its caller's resources", which
replaces this check with foreign_release(). Could the changelog be limited
to what this commit enables (a subprog takes a resource and leaves it with
its caller, which the caller then releases), or point to the follow-up
commit?
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -547,30 +554,37 @@ int bpf_frame_desc_record_acquire(struct bpf_verifier_env *env, int idx, int reg
[ ... ]
> - return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
> + if (acq < 0)
> + return acq;
> + /*
> + * A stack object is held by the frame that has it in its stack, which
> + * its sites find it in: the slot of a callee that created it there
> + * would be gone with the callee.
> + */
> + if (owner != st->curframe) {
> + ref->frameno = owner;
> + if (!env->res_acqs[acq].elsewhere)
> + env->res_acqs[acq].elsewhere = env->insn_idx + 1;
> + }
> + return ref_set_acq(env, st, idx, acq);
> }
Can this leave interrupts disabled after a throw? With the -EXDEV rejection
gone, a callee can now create a stack object in its caller's stack and have
it assigned to the caller (ref->frameno = owner). The unwinder still seems
to assume that whatever an outer frame holds was acquired before anything an
inner frame holds.
bpf_unwind_release_frames() walks the frames innermost first, and in the
LOCKS phase bpf_unwind_release_frame() restores IRQ flags saved with
interrupts disabled first, and the one saved with interrupts enabled last.
That is only correct if every IRQ save held by frame N-1 is older than every
save held by frame N. A callee can now save IRQ flags into its caller's
stack after saving its own:
static __noinline void callee(unsigned long *cf)
{
unsigned long mine;
bpf_local_irq_save(&mine); /* IRQs on: mine = enabled, held by frame 1 */
bpf_local_irq_save(cf); /* IRQs off: *cf = disabled, now held by frame 0 */
bpf_throw(0);
}
SEC("tc") int prog(struct __sk_buff *ctx)
{
unsigned long f;
callee(&f);
return 0;
}
callee() has a PTR_TO_STACK argument, so func_is_on_its_own() refuses it and
it is verified inline. check_kfunc_call() -> mark_stack_slot_irq_flag() ->
bpf_frame_desc_record_acquire() -> acquired_class() finds objframe 0, so
ref->frameno = 0 and elsewhere is set. At the throw, `mine` is described in
the bpf_throw site of frame 1 and `f` in the call site of frame 0. Every
visit agrees, so a->slot stays false and the new elsewhere check in
bpf_frame_desc_resolve() never fires. For bpf_throw, check_resource_leak()
passes check_lock = !bpf_jit_supports_exception_cleanup(), which is false on
x86-64 and arm64, and check_reference_leak() ignores REF_TYPE_IRQ, so the
throw is not rejected.
At runtime, in bpf_unwind_release_desc():
if (locks && late != (res->kind == BPF_RESOURCE_IRQ_FLAG &&
!arch_irqs_disabled_flags(*(unsigned long *)handle)))
continue;
bpf_unwind_release(res, handle);
Frame 1 restores `mine` in pass 1, which turns IRQs on. Frame 0 then
restores `f` in pass 0, with local_irq_restore(disabled), which turns IRQs
off again. bpf_throw_unwind() calls the exception callback without a
further fix-up, so the tc program returns to the networking stack with
interrupts disabled. The correct order is `f` first, then `mine`.
bpf_res_spin_lock_irqsave(lock, cf) after an IRQ save on the callee's own
stack hits the same problem.
The same assumption is behind resource_goes_first() ("A stack object may use
a referenced object the frame holds as well: it goes first"). A callee that
holds a file reference and creates a file dynptr reader (or an iterator built
on a reference) in its caller's stack has the reference released in its own
frame's OBJS walk before frame 0 frees the reader.
The new selftests only cover a callee that creates an iterator for its
caller (throw_iter_from_callee, iter_from_callee), and no commit later in the
series seems to change kernel/bpf/unwind.c or this ownership rule. Should a
stack object created for a caller be rejected while the creating frame holds
anything of the LOCKS phase, or should the unwinder order releases across
frames by acquisition?
[ ... ]
> @@ -748,71 +762,89 @@ int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delt
[ ... ]
> int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
> int idx, int callsite, const struct bpf_reg_state *r0)
> {
[ ... ]
> ref = idx < 0 ? NULL : &st->refs[idx];
> + if (ref && !ref->res)
> + ref = NULL;
> + if (ref && ref->type == REF_TYPE_PTR && ref->id &&
> + !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
> + reg_ref_off(r0, ref->id, &adj)) {
> + acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
> + return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
> }
> + if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
> + env->insn_aux_data[callsite].res_none = true;
Can a subprog that returns a reference in R0 while also holding a lock still
have that R0 reference kept in a hidden slot?
bpf_frame_desc_record_return() is now called once for each resource handed to
the caller. The call for the R0 reference takes the acq_add() branch above,
but the call for the lock (not REF_TYPE_PTR) falls through to the res_none
test. R0 is a non-NULL pointer on that visit, so res_none is set on the
callsite even though R0 carries the reference that the callsite's R0
acquisition records.
If a caller site later needs a slot for that acquisition,
bpf_frame_desc_resolve() calls acquires_sometimes(), which returns
aux->res_none, and the program is rejected with "insn %d acquires a resource
on some paths only, or one of a type that cannot be released", even though
the call acquires in R0 on every visit. For example, since
bpf_list_pop_front() is KF_ACQUIRE | KF_SPINLOCK_SAFE:
static __noinline struct node *pop_locked(void)
{
bpf_spin_lock(&glock);
n = bpf_list_pop_front(&ghead);
return n ? container_of(n, ...) : NULL; /* returns holding glock */
}
...
n = pop_locked();
bpf_spin_unlock(&glock);
if (n && cond) bpf_obj_drop(n);
bpf_throw(0); /* visits disagree on n: its acquisition needs a slot */
The same caller loads if pop_locked() unlocks before it returns. The
comment says "A reference that R0 carries is acquired by the call, like the
reference a kfunc returns", and a kfunc's R0 acquisition can have a slot.
This is a spurious rejection rather than a safety problem, but would setting
res_none only when no handed reference is carried in R0 avoid it? The code
is unchanged at the end of the series.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller
2026-11-11 12:57 ` [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 16:41 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 16:41 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] bpf: cross-frame unwind ordering bug in
bpf_frame_desc_record_acquire()
- [Medium] bpf: multi-resource tracking bypass in
bpf_frame_desc_record_return()
--
commit 48230df87bf3a57835f6fce92009993e3192e84d
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Allow a callee to leave resources with its caller
This commit allows BPF subprograms in programs that throw to leave held
resources, caller-stack objects, or RCU and preemption section deltas with
their caller unless a hidden stack slot is required to track them.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index 0ef7fda410ee1..9854be9ab4781 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -547,30 +554,37 @@ int bpf_frame_desc_record_acquire(struct bpf_verifier_env *env, int idx, int reg
> /* no insns can put it into a slot then: R10 stands for no register */
> if (regno < BPF_REG_0 || regno > BPF_REG_5)
> regno = BPF_REG_FP;
> acq = acq_add(env, env->insn_idx, res, regno, flags, adj, obj_off);
> - return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
> + if (acq < 0)
> + return acq;
> + /*
> + * A stack object is held by the frame that has it in its stack, which
> + * its sites find it in: the slot of a callee that created it there
> + * would be gone with the callee.
> + */
> + if (owner != st->curframe) {
> + ref->frameno = owner;
> + if (!env->res_acqs[acq].elsewhere)
> + env->res_acqs[acq].elsewhere = env->insn_idx + 1;
> + }
> + return ref_set_acq(env, st, idx, acq);
> }
[Severity: High]
Can assigning ref->frameno = owner here reverse the unwind release order
when the callee throws before returning?
In bpf_unwind_release_frame(), the ordering passes (releasing
BPF_FRAME_ENT_FIRST iterators and file dynptrs before regular objects in
BPF_UNWIND_PHASE_OBJS, and restoring disabled IRQ flags before enabled IRQ
flags in BPF_UNWIND_PHASE_LOCKS) run per frame as the unwinder walks from
the innermost callee frame to the outermost caller frame:
kernel/bpf/unwind.c:bpf_unwind_release_frame() {
...
for (pass = 0; pass < passes; pass++) {
for (first = 1; first >= 0; first--) {
bpf_unwind_release_desc(ctx, pool, d, fp, first, pass);
bpf_unwind_release_desc(ctx, pool, site, fp, first, pass);
}
}
}
If a callee acquires a referenced object (such as a cgroup via
bpf_cgroup_from_id()) in its own frame, initializes a borrowing iterator
(such as bpf_iter_css_task_new()) on the caller's stack, and throws before
returning, the callee frame is unwound first and drops the backing cgroup
reference before the caller frame unwinds and destroys the iterator,
causing a use-after-free. Similarly, if a callee saves IRQ flags on its
own stack while interrupts are enabled and then saves nested IRQ flags on
the caller's stack while interrupts are disabled before throwing, unwinding
the callee frame first enables interrupts and unwinding the caller frame
second restores the disabled state, leaving local CPU interrupts disabled.
[ ... ]
> @@ -748,71 +762,89 @@ int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delt
[ ... ]
> int bpf_frame_desc_record_return(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
> int idx, int callsite, const struct bpf_reg_state *r0)
> {
> struct bpf_reference_state *ref;
> + struct bpf_res_acq *held;
> s16 adj = 0;
> int acq;
>
> if (!env->describe_frames || st->speculative)
> return 0;
> ref = idx < 0 ? NULL : &st->refs[idx];
> - if (!ref || !ref->res) {
> - if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
> - env->insn_aux_data[callsite].res_none = true;
> - return 0;
> + if (ref && !ref->res)
> + ref = NULL;
> + if (ref && ref->type == REF_TYPE_PTR && ref->id &&
> + !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
> + reg_ref_off(r0, ref->id, &adj)) {
> + acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
> + return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
> }
> - if (!reg_ref_off(r0, ref->id, &adj)) {
> - verbose(env, "frame_desc: the subprog called at insn %d returns holding what R0 does not carry, which is not supported in a program that throws\n",
> - callsite);
> - return -EINVAL;
> + if (r0->type != SCALAR_VALUE || !tnum_equals_const(r0->var_off, 0))
> + env->insn_aux_data[callsite].res_none = true;
> + if (ref) {
> + held = &env->res_acqs[ref->acq - 1];
> + if (!held->elsewhere)
> + held->elsewhere = callsite + 1;
> }
> - acq = acq_add(env, callsite, ref->res - 1, BPF_REG_0, 0, adj, 0);
> - return acq < 0 ? acq : ref_set_acq(env, st, idx, acq);
> + return 0;
> }
[Severity: Medium]
When a subprog returns holding a non-R0 resource, does
bpf_frame_desc_record_return() miss setting held->multi if the caller frame
already holds a resource from the same acquisition?
If a caller invokes a lock-taking subprog twice, the first lock's
ref->frameno is already updated to the caller when the second call runs
ref_set_acq() in the callee frame, so ref_set_acq() does not set
env->res_acqs[acq].multi. When the second call returns, the caller frame
holds two references with the same ref->acq, and in site_visit() both
references match the same env->res_locs[j] entry and increment loc->visits
twice for a single increment of site->visits:
kernel/bpf/frame_desc.c:site_visit() {
...
for (j = 0; j < env->res_loc_cnt; j++)
if (env->res_locs[j].site == s && env->res_locs[j].acq == ref->acq - 1)
break;
if (j == env->res_loc_cnt) {
...
cur.visits = 1;
loc[j] = cur;
env->res_loc_cnt = j + 1;
continue;
}
/* the places this visit finds the handle in as well */
loc = &env->res_locs[j];
loc->visits++;
}
If the resource has loc->fixed set (such as a resilient spin lock in a
1-element array map acquired via two distinct map-lookup pointers on one
path) and another path reaches the same throw site holding neither,
site->visits (2) equals loc->visits (2) with loc->cnt == 1, so
bpf_frame_desc_resolve() leaves a->slot false, skips the a->elsewhere and
a->multi checks, and emits a site descriptor that unlocks an unheld lock if
the second path throws. Calling ref_set_acq(env, st, idx, ref->acq - 1) on
the non-R0 return path and checking a->multi even when !a->slot would catch
this.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=18
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 19/29] bpf: Allow a callee to release its caller's resources
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (17 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 18/29] bpf: Allow a callee to leave resources with its caller Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 17:02 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it Kumar Kartikeya Dwivedi
` (10 subsequent siblings)
29 siblings, 1 reply; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently a program that throws is rejected when a callee releases a
resource held by its caller. The handle could be in a hidden slot of the
caller, and the callee does not know where that slot is.
Handle the two cases separately.
If the resource has no hidden slot, nothing needs to be cleared. A site
of the caller describes the resource if every visit finds it held, and
omits it if none does.
If the resource has a hidden slot, the callee cannot clear it, but the
caller can, before the call. This is correct when the slot is not needed
from the call on:
- the resource is released on every return from the call, and
- no throw finds the caller holding it at the call. That is, the callee
does not throw before it releases the resource.
For each call to a subprog and each acquisition of the caller, record
whether a frame above the caller released the resource during the call,
and whether any return from the call found the caller still holding it.
After the pass, a resource with a hidden slot that a callee releases has
the slot cleared before every such call. The program is rejected if one
of these calls does not allow it.
A subprog that releases resources of its callers must be verified in the
context of each caller, so that a throw sees the frames below it as they
are. The verifier normally learns this when the subprog returns, but a
subprog that throws after the release never returns. So mark the
functions between the owner and the releasing frame at the release
itself.
Setting the mark invalidates what the current round of the main pass has
recorded so far: the frames of the callers at the call, still holding
their resources. So the round is run again when a mark is set, see
do_check_all().
This allows a subprog that unlocks a lock of its caller, and one that
releases a resource that the caller's sites describe.
Still rejected, for a resource that the caller keeps in a hidden slot:
- a callee that may throw before it releases the resource;
- a callee that releases the resource only on some paths;
- a callback that releases the resource, since its helper may never
call it;
- a callx to a subprog that releases the resource.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 23 +++++
kernel/bpf/frame_desc.c | 158 ++++++++++++++++++++++++++++++++---
kernel/bpf/verifier.c | 6 +-
3 files changed, 173 insertions(+), 14 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 9a0ea25ab5c2..a98023c239c4 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1278,6 +1278,7 @@ struct bpf_res_acq {
s16 off; /* the slot, relative to R10 */
bool held; /* some frame holds what it acquired at a site */
bool slot; /* and some site cannot say where: the handle is kept in a slot */
+ u32 foreign; /* an insn of another frame that releases what it acquired, plus one */
/*
* An insn, plus one, that leaves what it acquired with another frame
* than the one that ran the acquisition: the call a subprog returns
@@ -1341,6 +1342,20 @@ struct bpf_res_rel {
/* The call takes the handle in different ways on different paths. */
#define BPF_RES_REL_DIFFER 3
+/*
+ * A call of a subprog during which a frame above the caller releases what an
+ * acquisition of the caller acquired. The caller clears its slot in front of
+ * the call when no throw finds it holding the resource there and no return
+ * from the call does, see frame_desc.c.
+ */
+struct bpf_res_call {
+ u32 insn; /* the call; U32_MAX when the caller is not in the state */
+ u16 acq; /* index of the acquisition */
+ bool released;
+ bool held_after; /* some return from the call finds the caller holding it */
+ bool clear; /* decided once the pass is over */
+};
+
/* A call that enters or leaves a section, for the count of its frame, see frame_desc.c. */
struct bpf_section_insn {
u32 insn;
@@ -1390,6 +1405,8 @@ struct bpf_verifier_env {
u32 res_rel_cnt;
struct bpf_section_insn *section_insns;
u32 section_insn_cnt;
+ struct bpf_res_call *res_calls;
+ u32 res_call_cnt;
struct bpf_res_site *res_sites;
u32 res_site_cnt;
struct bpf_res_loc *res_locs;
@@ -1402,6 +1419,12 @@ struct bpf_verifier_env {
bool describe_frames; /* frame descriptors are generated: the program may throw */
bool describe_regs; /* and may name registers, see bpf_arch_throw_entry() */
u16 frame_desc_regs; /* the registers that some site names */
+ /*
+ * The round of the main pass found out that a subprog releases what
+ * its callers hold: they are recorded differently then, so the round
+ * is done again, see do_check_all().
+ */
+ bool frame_desc_redo;
int exception_callback_subprog;
bool explore_alu_limits;
bool allow_ptr_leaks;
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index 9854be9ab478..d2ad02925f06 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -34,8 +34,9 @@
* A slot belongs to one frame and holds one handle, so a program that throws
* keeps to what that can express:
*
- * - A frame releases what it acquired itself. A callee cannot reach the
- * slot of its caller, so it may not release what its caller holds.
+ * - A callee cannot reach the slot of its caller. What it releases of its
+ * caller's has no slot, or one the caller can empty in front of the
+ * call, see foreign_release().
* - A slot goes with its frame. A reference a subprog returns in R0 is
* acquired by the call for the caller, like the call of a kfunc that
* returns one.
@@ -689,12 +690,76 @@ static int rel_add(struct bpf_verifier_env *env, u32 insn, u16 acq, u8 reg, u8 f
return 0;
}
+/*
+ * What the call @insn of a subprog means for what acquisition @acq acquired
+ * is noted in the record of the two, see struct bpf_res_call. NULL when there
+ * is no memory for one.
+ */
+static struct bpf_res_call *call_note(struct bpf_verifier_env *env, u32 insn, u16 acq)
+{
+ u32 i, cnt = env->res_call_cnt;
+ struct bpf_res_call *c;
+
+ for (i = 0; i < cnt; i++)
+ if (env->res_calls[i].insn == insn && env->res_calls[i].acq == acq)
+ return &env->res_calls[i];
+ c = kvrealloc(env->res_calls, array_size(cnt + 1, sizeof(*c)), GFP_KERNEL_ACCOUNT);
+ if (!c)
+ return NULL;
+ c[cnt] = (struct bpf_res_call) { .insn = insn, .acq = acq };
+ env->res_calls = c;
+ env->res_call_cnt = cnt + 1;
+ return &c[cnt];
+}
+
+/*
+ * The current insn releases @ref, which a frame below the current one owns.
+ * Nothing tells a callee where the slot of its caller is, so it cannot clear
+ * one. There are two ways out:
+ *
+ * - What the caller holds without a slot needs none cleared: its sites say
+ * where the handle is on the visits that find it held, and say nothing on
+ * those that find it released.
+ * - The caller clears its slot in front of the call when the callee releases
+ * the resource on every path and no throw finds the caller holding it at
+ * the call, see bpf_frame_desc_resolve().
+ *
+ * The functions between the owner and the current frame are verified in the
+ * context of their callers from now on, so that every throw sees the frames
+ * below it as they are. The round of the main pass that finds one out is done
+ * again, since it recorded the frames of its callers at the call, holding
+ * what they held then.
+ */
+static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
+ const struct bpf_reference_state *ref, struct bpf_res_acq *acq)
+{
+ const struct bpf_func_state *below = st->frame[ref->frameno + 1];
+ struct bpf_subprog_info *sub;
+ struct bpf_res_call *c;
+ int f;
+
+ for (f = ref->frameno + 1; f <= st->curframe; f++) {
+ sub = &env->subprog_info[st->frame[f]->subprogno];
+ if (sub->releases_inherited)
+ continue;
+ sub->releases_inherited = true;
+ env->frame_desc_redo = true;
+ }
+ if (!acq->foreign)
+ acq->foreign = env->insn_idx + 1;
+ c = call_note(env, ref->frameno < 0 ? U32_MAX : below->callsite, ref->acq - 1);
+ if (!c)
+ return -ENOMEM;
+ c->released = true;
+ return 0;
+}
+
/*
* Record that the current insn releases the resource refs[@idx], before it
* is released: its handle, in the argument the call releases, see
* bpf_verifier_env::release_regno, leaves the slot of its acquisition in
- * front of the call. A frame releases what it acquired itself: nothing tells
- * a callee where the slot of its caller is.
+ * front of the call. That is so for what the frame holds itself: nothing
+ * tells a callee where the slot of its caller is, see foreign_release().
*/
int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
{
@@ -703,7 +768,7 @@ int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
const struct bpf_func_state *frame = st->frame[st->curframe];
u32 i, insn = env->insn_idx;
int regno = env->release_regno;
- const struct bpf_res_acq *acq;
+ struct bpf_res_acq *acq;
u8 flags = 0, bad = 0;
s16 adj = 0;
@@ -719,12 +784,9 @@ int bpf_frame_desc_record_release(struct bpf_verifier_env *env, int idx)
}
if (!ref->res)
return 0;
- if (ref->frameno != st->curframe) {
- verbose(env, "frame_desc: insn %d releases what another frame holds, which is not supported in a program that throws\n",
- insn);
- return -EINVAL;
- }
acq = &env->res_acqs[ref->acq - 1];
+ if (ref->frameno != st->curframe)
+ return foreign_release(env, st, ref, acq);
/* what insns could not compare with a slot matters when there is a slot only */
if (regno < BPF_REG_1 || regno > BPF_REG_5) {
regno = BPF_REG_FP;
@@ -781,6 +843,8 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
{
struct bpf_verifier_state *st = env->cur_state;
const struct bpf_func_state *frame = st->frame[st->curframe];
+ struct bpf_res_call *c;
+ int i;
if (!env->describe_frames || st->speculative)
return 0;
@@ -791,6 +855,17 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
frame->rcu_released_below,
st->active_preempt_locks != frame->entry_preempt_locks +
frame->preempt_released_below);
+ /* does the caller hold after the call what a callee may release in it? */
+ if (!st->curframe || !env->subprog_info[frame->subprogno].releases_inherited)
+ return 0;
+ for (i = 0; i < st->acquired_refs; i++) {
+ if (st->refs[i].frameno != st->curframe - 1 || !st->refs[i].acq)
+ continue;
+ c = call_note(env, frame->callsite, st->refs[i].acq - 1);
+ if (!c)
+ return -ENOMEM;
+ c->held_after = true;
+ }
return 0;
}
@@ -1052,6 +1127,9 @@ void bpf_frame_desc_reset(struct bpf_verifier_env *env)
env->section_insn_cnt = 0;
env->res_site_cnt = 0;
env->res_loc_cnt = 0;
+ env->res_call_cnt = 0;
+ env->frame_desc_redo = false;
+ /* that the subprogs release what their callers hold stays */
for (i = 0; i < env->subprog_cnt; i++)
env->subprog_info[i].section_delta = 0;
for (i = 0; i < env->prog->len; i++) {
@@ -1067,12 +1145,14 @@ void bpf_frame_desc_env_free(struct bpf_verifier_env *env)
kvfree(env->section_insns);
kvfree(env->res_sites);
kvfree(env->res_locs);
+ kvfree(env->res_calls);
kvfree(env->resources);
env->res_acqs = NULL;
env->res_rels = NULL;
env->section_insns = NULL;
env->res_sites = NULL;
env->res_locs = NULL;
+ env->res_calls = NULL;
env->resources = NULL;
}
@@ -1150,6 +1230,43 @@ static int emit_acquire_after(struct bpf_insn *buf, const struct bpf_res_acq *a)
return cnt;
}
+/*
+ * A frame above the one that holds what acquisition @acq acquired releases it,
+ * and the acquisition has a slot. The holder can clear the slot itself, in
+ * front of the call the release happens in, when the resource is gone after
+ * every such call and no throw finds the holder with it at the call: the slot
+ * is empty then for as long as the callee can throw after the release, and it
+ * was no use before it. The call is one of a subprog: a helper that calls a
+ * callback may not call it at all, and the holder has the resource after it
+ * then.
+ */
+static bool foreign_cleared(struct bpf_verifier_env *env, u16 acq, const u32 *new_of, u32 orig_len)
+{
+ bool any = false;
+ u32 i, j;
+
+ for (i = 0; i < env->res_call_cnt; i++) {
+ const struct bpf_res_call *c = &env->res_calls[i];
+
+ if (c->acq != acq || !c->released)
+ continue;
+ if (c->held_after || c->insn >= orig_len)
+ return false;
+ /* a helper may never call the callback that releases */
+ if (!bpf_pseudo_call(&env->prog->insnsi[new_of[c->insn]]))
+ return false;
+ for (j = 0; j < env->res_loc_cnt; j++)
+ if (env->res_locs[j].acq == acq &&
+ env->res_sites[env->res_locs[j].site].insn == c->insn)
+ return false;
+ any = true;
+ }
+ for (i = 0; any && i < env->res_call_cnt; i++)
+ if (env->res_calls[i].acq == acq && env->res_calls[i].released)
+ env->res_calls[i].clear = true;
+ return any;
+}
+
/*
* Acquisition @acq needs a slot. Can the insns that keep it be emitted? They
* are emitted once per insn, so the insn has to acquire in one way, with the
@@ -1264,6 +1381,12 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
ret = -EINVAL;
break;
}
+ if (a->foreign && !foreign_cleared(env, i, new_of, orig_len)) {
+ verbose(env, "frame_desc: insn %d releases what another frame holds in a hidden slot, which is not supported in a program that throws\n",
+ a->foreign - 1);
+ ret = -EINVAL;
+ break;
+ }
/* some visit of the insn has nothing a slot can hold where this one has */
if (!(a->flags & BPF_RES_F_CTX) && acquires_sometimes(env, new_of[a->insn], a)) {
verbose(env, "frame_desc: insn %d acquires a resource on some paths only, or one of a type that cannot be released\n",
@@ -1305,12 +1428,13 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
enum {
ITEM_SECTION,
ITEM_RELEASE,
+ ITEM_CLEAR,
ITEM_ACQUIRE,
};
struct res_item {
u32 idx; /* insn of the current image */
- u32 rec; /* which section insn, release or acquisition */
+ u32 rec; /* which section insn, release, call or acquisition */
u8 kind;
u8 reg; /* ITEM_RELEASE: the argument the call releases */
};
@@ -1391,6 +1515,11 @@ static int emit_items(struct bpf_verifier_env *env, const struct res_item *it, i
cnt += emit_release(env, buf + cnt, &it[i], k - i);
i = k - 1;
break;
+ case ITEM_CLEAR:
+ /* a callee releases what the slot holds, on every path, see foreign_cleared() */
+ off = env->res_acqs[env->res_calls[it[i].rec].acq].off;
+ buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, off, 0);
+ break;
case ITEM_ACQUIRE:
cnt += emit_acquire_before(buf + cnt, &env->res_acqs[it[i].rec]);
break;
@@ -1470,8 +1599,8 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
if (!env->res_acq_cnt && !env->section_insn_cnt)
return 0;
new_of = insns_of_pass(env, &orig_len);
- items = kvcalloc(env->res_acq_cnt + env->res_rel_cnt + env->section_insn_cnt,
- sizeof(*items), GFP_KERNEL_ACCOUNT);
+ items = kvcalloc(env->res_acq_cnt + env->res_rel_cnt + env->res_call_cnt +
+ env->section_insn_cnt, sizeof(*items), GFP_KERNEL_ACCOUNT);
if (!new_of || !items) {
ret = -ENOMEM;
goto out;
@@ -1487,6 +1616,9 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
if (env->res_acqs[env->res_rels[i].acq].slot)
items[n++] = (struct res_item) { new_of[env->res_rels[i].insn], i, ITEM_RELEASE,
env->res_rels[i].reg };
+ for (i = 0; i < env->res_call_cnt; i++)
+ if (env->res_calls[i].clear)
+ items[n++] = (struct res_item) { new_of[env->res_calls[i].insn], i, ITEM_CLEAR };
for (i = 0; i < env->res_acq_cnt; i++)
if (env->res_acqs[i].slot && !(env->res_acqs[i].flags & BPF_RES_F_CTX))
items[n++] = (struct res_item) { new_of[env->res_acqs[i].insn], i, ITEM_ACQUIRE };
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a710da1a15c6..ec9f77a63197 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11998,6 +11998,9 @@ int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *cal
if (env->log.level & BPF_LOG_LEVEL2)
verbose(env, "func#%d releases what its caller holds at insn %d\n",
callee->subprogno, call_idx);
+ /* what the round recorded of the function so far does not hold, see frame_desc.c */
+ if (env->describe_frames && !env->subprog_info[callee->subprogno].releases_inherited)
+ env->frame_desc_redo = true;
env->subprog_info[callee->subprogno].releases_inherited = true;
return 1;
}
@@ -22273,7 +22276,8 @@ static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
/* All is done again, whatever came out, see do_check_all() */
static bool round_is_dropped(struct bpf_verifier_env *env)
{
- return (env->restart && bpf_flow_loops_lost_bound(env)) || env->guards_late;
+ return (env->restart && bpf_flow_loops_lost_bound(env)) || env->guards_late ||
+ env->frame_desc_redo;
}
static int do_check(struct bpf_verifier_env *env)
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 19/29] bpf: Allow a callee to release its caller's resources
2026-11-11 12:57 ` [PATCH bpf-next v1 19/29] bpf: Allow a callee to release its caller's resources Kumar Kartikeya Dwivedi
@ 2026-11-11 17:02 ` sashiko-bot
0 siblings, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 17:02 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] bpf: missing releases_inherited mark in release_context_state()
--
commit e5be88d9a8d47ae92f3e9f98a812d289864c60bf
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Allow a callee to release its caller's resources
This commit allows a subprogram in a program that throws to release
resources acquired by its caller, clearing the caller's hidden slot before
the call when needed and marking intermediate subprograms with
releases_inherited at the release site so they are verified in the context
of their callers.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index 9854be9ab4781..d2ad02925f061 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -689,12 +690,76 @@ static int rel_add(struct bpf_verifier_env *env, u32 insn, u16 acq, u8 reg, u8 f
> return 0;
> }
>
> +/*
> + * What the call @insn of a subprog means for what acquisition @acq acquired
> + * is noted in the record of the two, see struct bpf_res_call. NULL when there
> + * is no memory for one.
> + */
> +static struct bpf_res_call *call_note(struct bpf_verifier_env *env, u32 insn, u16 acq)
> +{
> + u32 i, cnt = env->res_call_cnt;
> + struct bpf_res_call *c;
> +
> + for (i = 0; i < cnt; i++)
> + if (env->res_calls[i].insn == insn && env->res_calls[i].acq == acq)
> + return &env->res_calls[i];
> + c = kvrealloc(env->res_calls, array_size(cnt + 1, sizeof(*c)), GFP_KERNEL_ACCOUNT);
> + if (!c)
> + return NULL;
> + c[cnt] = (struct bpf_res_call) { .insn = insn, .acq = acq };
> + env->res_calls = c;
> + env->res_call_cnt = cnt + 1;
> + return &c[cnt];
> +}
> +
> +/*
> + * The current insn releases @ref, which a frame below the current one owns.
> + * Nothing tells a callee where the slot of its caller is, so it cannot clear
> + * one. There are two ways out:
> + *
> + * - What the caller holds without a slot needs none cleared: its sites say
> + * where the handle is on the visits that find it held, and say nothing on
> + * those that find it released.
> + * - The caller clears its slot in front of the call when the callee releases
> + * the resource on every path and no throw finds the caller holding it at
> + * the call, see bpf_frame_desc_resolve().
> + *
> + * The functions between the owner and the current frame are verified in the
> + * context of their callers from now on, so that every throw sees the frames
> + * below it as they are. The round of the main pass that finds one out is done
> + * again, since it recorded the frames of its callers at the call, holding
> + * what they held then.
> + */
> +static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
> + const struct bpf_reference_state *ref, struct bpf_res_acq *acq)
> +{
> + const struct bpf_func_state *below = st->frame[ref->frameno + 1];
> + struct bpf_subprog_info *sub;
> + struct bpf_res_call *c;
> + int f;
> +
> + for (f = ref->frameno + 1; f <= st->curframe; f++) {
> + sub = &env->subprog_info[st->frame[f]->subprogno];
> + if (sub->releases_inherited)
> + continue;
> + sub->releases_inherited = true;
> + env->frame_desc_redo = true;
> + }
[Severity: High]
Does release_context_state() also need to set sub->releases_inherited and
env->frame_desc_redo when a frame leaves an inherited RCU read or
preemption-disabled section?
foreign_release() marks sub->releases_inherited when a callee releases an
entry in st->refs[] so that a subprog that throws without returning is
verified in the context of its caller. However, RCU and preemption-disabled
sections are tracked via entry_rcu_locks and entry_preempt_locks rather
than st->refs[]:
kernel/bpf/verifier.c:release_context_state() {
for (fr = 0; fr <= state->curframe; fr++) {
frame = state->frame[fr];
if (frame->entry_rcu_locks > state->active_rcu_locks)
frame->rcu_released_below += frame->entry_rcu_locks - state->active_rcu_locks;
if (frame->entry_preempt_locks > state->active_preempt_locks)
frame->preempt_released_below += frame->entry_preempt_locks -
state->active_preempt_locks;
...
}
For those sections, releases_inherited is only set in bpf_flow_return() when
the callee returns. If a static subprogram calls bpf_rcu_read_unlock() on
its caller's RCU section and then calls bpf_throw() without returning, it
remains verified on its own without the caller's frames in env->cur_state.
When bpf_rcu_read_unlock() calls invalidate_rcu_protected_refs(), it does
not see STACK_ITER slots with MEM_RCU on the caller's stack, allowing an
RCU-protected iterator to outlive its RCU section across a sleepable call
before bpf_throw() unwinds the stack and invokes the iterator destructor
(use-after-free).
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=19
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (18 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 19/29] bpf: Allow a callee to release its caller's resources Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:57 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources Kumar Kartikeya Dwivedi
` (9 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently a callee can release a reference that its caller keeps in a
hidden slot only if it does not throw before the release, since the
caller clears the slot before the call. A subprog that is passed an
object, may throw, and then drops the object is rejected if its caller
keeps the object in a hidden slot.
Transfer the reference to the callee at the call:
- When the verifier sees a subprog release a reference of its caller,
it records which argument register the reference was passed in.
- A direct call to the subprog then transfers the reference that the
caller holds and passes in that register. For the caller, the call is
the release: the handle is cleared from its hidden slot, if it has
one, before the call. For the callee, its entry is the acquisition,
with the handle in the argument register. The callee holds the
reference like one it acquired itself: its sites describe it, or it
gets a hidden slot that the entry of the callee fills from the
argument.
- A callee that returns without releasing the reference gives it back
to the caller, under the caller's original acquisition.
Recording the argument changes what the current round of the main pass
has recorded so far, so the round is run again when one is found.
Only references are transferred. Other resources, such as a lock, stay
with the caller even if a pointer to them is passed.
This allows a subprog that drops an object or discards a record that it
is passed, and may throw both before and after.
Still rejected:
- a reference that a callee is passed and returns without releasing on
some paths, if the caller keeps it in a hidden slot;
- a subprog that keeps a transferred reference in a hidden slot, if
another call passes something else in that argument;
- a callx to a subprog that releases a reference that its caller keeps
in a hidden slot.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 30 +++++-
kernel/bpf/frame_desc.c | 192 ++++++++++++++++++++++++++++++-----
kernel/bpf/join.c | 6 +-
kernel/bpf/states.c | 6 +-
kernel/bpf/verifier.c | 12 +++
5 files changed, 212 insertions(+), 34 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index a98023c239c4..03290f7e42b5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -524,6 +524,13 @@ struct bpf_func_state {
*/
u32 rcu_released_below;
u32 preempt_released_below;
+ /*
+ * The ids of what the frame was called with in R1 to R5, and the
+ * acquisition, plus one, each was of in the caller when the call
+ * handed it to the frame, see bpf_frame_desc_record_call().
+ */
+ u32 entry_arg_id[5];
+ u16 entry_arg_acq[5];
/* The following fields should be last. See copy_func_state() */
/* The state of the stack. Each element of the array describes BPF_REG_SIZE
@@ -1112,6 +1119,13 @@ struct bpf_subprog_info {
bool stack_args_from_code: 1;
/* it releases what its caller holds, so it's verified in the context of such caller */
bool releases_inherited: 1;
+ /*
+ * The arguments, a bit per register, that it releases what its caller
+ * holds through: the call hands that to it, see frame_desc.c. And those
+ * of them that some call passes something else in.
+ */
+ u8 consumes;
+ u8 entry_none;
/*
* The kinds of sections, a bit each, that a callee returned to it in
* more or fewer of than it called the callee in: a count it keeps in a
@@ -1260,6 +1274,7 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
#define BPF_RES_F_DEREF BIT(2) /* the handle is the first word of a dynptr on the stack */
#define BPF_RES_F_CTX BIT(3) /* a __ref argument the kernel passes the main program */
#define BPF_RES_F_NO_PTR BIT(5) /* a lock that no pointer to what it is in leads to */
+#define BPF_RES_F_ENTRY BIT(4) /* what its caller hands a subprog in the register */
/*
* A call that acquires a resource, see frame_desc.c. When a frame can be
@@ -1269,7 +1284,11 @@ void bpf_patch_image_install(struct bpf_prog *prog, const struct bpf_patch_image
* per acquiring call.
*/
struct bpf_res_acq {
- u32 insn; /* the insn as the main pass named it; the argument with BPF_RES_F_CTX */
+ /*
+ * The insn as the main pass named it; the argument with BPF_RES_F_CTX,
+ * the first insn of the subprog with BPF_RES_F_ENTRY.
+ */
+ u32 insn;
u16 res; /* the class of what it acquires */
u8 reg; /* R0 for what the call returns, else the argument that is the handle */
u8 flags; /* BPF_RES_F_* */
@@ -1279,6 +1298,7 @@ struct bpf_res_acq {
bool held; /* some frame holds what it acquired at a site */
bool slot; /* and some site cannot say where: the handle is kept in a slot */
u32 foreign; /* an insn of another frame that releases what it acquired, plus one */
+ bool handed_back; /* a callee it was handed to returned holding it */
/*
* An insn, plus one, that leaves what it acquired with another frame
* than the one that ran the acquisition: the call a subprog returns
@@ -1344,9 +1364,9 @@ struct bpf_res_rel {
/*
* A call of a subprog during which a frame above the caller releases what an
- * acquisition of the caller acquired. The caller clears its slot in front of
- * the call when no throw finds it holding the resource there and no return
- * from the call does, see frame_desc.c.
+ * acquisition of the caller acquired, without being handed it. The caller
+ * clears its slot in front of the call when no throw finds it holding the
+ * resource there and no return from the call does, see frame_desc.c.
*/
struct bpf_res_call {
u32 insn; /* the call; U32_MAX when the caller is not in the state */
@@ -2383,6 +2403,8 @@ int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delt
int bpf_frame_desc_record_exit(struct bpf_verifier_env *env);
void bpf_frame_desc_record_sections(struct bpf_verifier_env *env, int subprog, bool rcu,
bool preempt);
+int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ struct bpf_func_state *callee, int callsite);
bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
int f, const struct bpf_reg_state *reg);
u16 bpf_frame_desc_kept_regs(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index d2ad02925f06..6679f134b14e 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -34,12 +34,15 @@
* A slot belongs to one frame and holds one handle, so a program that throws
* keeps to what that can express:
*
- * - A callee cannot reach the slot of its caller. What it releases of its
- * caller's has no slot, or one the caller can empty in front of the
- * call, see foreign_release().
+ * - A callee cannot reach the slot of its caller. A reference it releases
+ * that its caller passes it in a register is handed over to it by the
+ * call: it is the callee's from its entry, and no longer the caller's.
+ * What it releases otherwise, the lock of its caller for one, has no
+ * slot, or one the caller can empty in front of the call, see
+ * bpf_frame_desc_record_call() and foreign_release().
* - A slot goes with its frame. A reference a subprog returns in R0 is
* acquired by the call for the caller, like the call of a kfunc that
- * returns one.
+ * returns one, and one it was handed and did not release it gives back.
* Whatever else a frame comes to hold of what another frame acquired can
* have no slot: a lock a subprog returns holding, or a stack object a
* callee creates in the stack of its caller. Likewise a subprog whose
@@ -715,8 +718,13 @@ static struct bpf_res_call *call_note(struct bpf_verifier_env *env, u32 insn, u1
/*
* The current insn releases @ref, which a frame below the current one owns.
* Nothing tells a callee where the slot of its caller is, so it cannot clear
- * one. There are two ways out:
+ * one. There are three ways out:
*
+ * - The call hands what the callee releases over to it, when it is passed in
+ * a register: the callee owns it from its entry, as if it had acquired it
+ * there, see bpf_frame_desc_record_call(). Which arguments a subprog
+ * releases is found out here, and the round of the main pass is done again
+ * with that known.
* - What the caller holds without a slot needs none cleared: its sites say
* where the handle is on the visits that find it held, and say nothing on
* those that find it released.
@@ -726,9 +734,7 @@ static struct bpf_res_call *call_note(struct bpf_verifier_env *env, u32 insn, u1
*
* The functions between the owner and the current frame are verified in the
* context of their callers from now on, so that every throw sees the frames
- * below it as they are. The round of the main pass that finds one out is done
- * again, since it recorded the frames of its callers at the call, holding
- * what they held then.
+ * below it as they are.
*/
static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
const struct bpf_reference_state *ref, struct bpf_res_acq *acq)
@@ -736,7 +742,7 @@ static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifi
const struct bpf_func_state *below = st->frame[ref->frameno + 1];
struct bpf_subprog_info *sub;
struct bpf_res_call *c;
- int f;
+ int f, k;
for (f = ref->frameno + 1; f <= st->curframe; f++) {
sub = &env->subprog_info[st->frame[f]->subprogno];
@@ -745,6 +751,21 @@ static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifi
sub->releases_inherited = true;
env->frame_desc_redo = true;
}
+ /* every frame it was passed down through in a register is handed it by the one below */
+ for (f = ref->frameno + 1; f <= st->curframe; f++) {
+ if (!ref->id || ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG))
+ break;
+ for (k = BPF_REG_1; k <= BPF_REG_5; k++)
+ if (st->frame[f]->entry_arg_id[k - 1] == ref->id)
+ break;
+ if (k > BPF_REG_5)
+ break;
+ sub = &env->subprog_info[st->frame[f]->subprogno];
+ if (sub->consumes & BIT(k))
+ continue;
+ sub->consumes |= BIT(k);
+ env->frame_desc_redo = true;
+ }
if (!acq->foreign)
acq->foreign = env->insn_idx + 1;
c = call_note(env, ref->frameno < 0 ? U32_MAX : below->callsite, ref->acq - 1);
@@ -828,6 +849,10 @@ int bpf_frame_desc_record_section(struct bpf_verifier_env *env, u8 kind, s8 delt
*
* - A reference in R0 the call acquires for the caller, like the call of a
* kfunc that returns one, see bpf_frame_desc_record_return().
+ * - What the call handed the frame to release, and it did not on this path,
+ * is its caller's again, as what the caller acquired it as. The caller has
+ * emptied the slot of that, if it has one, see
+ * bpf_frame_desc_record_call().
* - Anything else, a lock the frame took for its caller, stays what it was
* acquired as, and can have no slot: the sites of the caller have to say
* where it is, or no throw may find it held, see
@@ -843,7 +868,10 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
{
struct bpf_verifier_state *st = env->cur_state;
const struct bpf_func_state *frame = st->frame[st->curframe];
+ const struct bpf_res_acq *acq;
struct bpf_res_call *c;
+ u16 back;
+ s16 adj;
int i;
if (!env->describe_frames || st->speculative)
@@ -855,6 +883,25 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
frame->rcu_released_below,
st->active_preempt_locks != frame->entry_preempt_locks +
frame->preempt_released_below);
+ for (i = 0; st->curframe && i < st->acquired_refs; i++) {
+ struct bpf_reference_state *ref = &st->refs[i];
+
+ if (!ref->res || ref->frameno != st->curframe)
+ continue;
+ acq = &env->res_acqs[ref->acq - 1];
+ if (!(acq->flags & BPF_RES_F_ENTRY))
+ continue;
+ /* the call acquires what R0 carries */
+ if (ref->type == REF_TYPE_PTR && ref->id &&
+ reg_ref_off(&frame->regs[BPF_REG_0], ref->id, &adj))
+ continue;
+ back = frame->entry_arg_acq[acq->reg - 1];
+ if (!back)
+ continue;
+ env->res_acqs[back - 1].handed_back = true;
+ ref->acq = back;
+ ref->frameno = st->curframe - 1;
+ }
/* does the caller hold after the call what a callee may release in it? */
if (!st->curframe || !env->subprog_info[frame->subprogno].releases_inherited)
return 0;
@@ -869,6 +916,71 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
return 0;
}
+/*
+ * Frame @callee of @st is entered with its arguments in place, by the call at
+ * insn @callsite of the frame below it when @st is given; a callback, a
+ * callx and a function verified on its own come with none.
+ *
+ * What a subprog releases of its caller's through an argument, see
+ * bpf_subprog_info::consumes, the call hands over to it: the reference the
+ * caller holds and passes in that register. For the caller the call releases
+ * it, so its handle leaves the slot of the caller in front of the call. For
+ * the callee its entry acquires it, with the handle in the register, and it
+ * holds it like what it acquires itself: its sites say where it keeps it, or
+ * a slot of its own does, filled at its entry.
+ *
+ * That slot is filled on every call, so every call has to pass what its
+ * caller holds, or NULL: bpf_subprog_info::entry_none has the arguments some
+ * call passes something else in.
+ */
+int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ struct bpf_func_state *callee, int callsite)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[callee->subprogno];
+ struct bpf_reference_state *ref;
+ int i, k, acq, err;
+ s16 adj;
+
+ if (!env->describe_frames || (st && st->speculative))
+ return 0;
+ for (k = BPF_REG_1; k <= BPF_REG_5; k++) {
+ struct bpf_reg_state *reg = &callee->regs[k];
+
+ callee->entry_arg_id[k - 1] = 0;
+ callee->entry_arg_acq[k - 1] = 0;
+ if (reg->type == SCALAR_VALUE || reg->type == NOT_INIT) {
+ if ((sub->consumes & BIT(k)) && !bpf_register_is_null(reg))
+ sub->entry_none |= BIT(k);
+ continue;
+ }
+ callee->entry_arg_id[k - 1] = reg->id;
+ if (!(sub->consumes & BIT(k)))
+ continue;
+ for (i = 0, ref = NULL; st && i < st->acquired_refs; i++) {
+ ref = &st->refs[i];
+ if (ref->frameno == callee->frameno - 1 && ref->acq && ref->type == REF_TYPE_PTR &&
+ !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
+ reg_ref_off(reg, ref->id, &adj))
+ break;
+ ref = NULL;
+ }
+ if (!ref) {
+ sub->entry_none |= BIT(k);
+ continue;
+ }
+ err = rel_add(env, callsite, ref->acq - 1, k, 0, adj, 0);
+ if (err)
+ return err;
+ acq = acq_add(env, sub->start, ref->res - 1, k, BPF_RES_F_ENTRY, adj, 0);
+ if (acq < 0)
+ return acq;
+ callee->entry_arg_acq[k - 1] = ref->acq;
+ ref->frameno = callee->frameno;
+ ref->acq = acq + 1;
+ }
+ return 0;
+}
+
/*
* A subprog called at insn @callsite returned to the current frame of @st
* with @r0, holding the reference refs[@idx], which is the caller's from here
@@ -1129,9 +1241,11 @@ void bpf_frame_desc_reset(struct bpf_verifier_env *env)
env->res_loc_cnt = 0;
env->res_call_cnt = 0;
env->frame_desc_redo = false;
- /* that the subprogs release what their callers hold stays */
- for (i = 0; i < env->subprog_cnt; i++)
+ /* what the subprogs release stays, what the calls of the round passed them goes */
+ for (i = 0; i < env->subprog_cnt; i++) {
+ env->subprog_info[i].entry_none = 0;
env->subprog_info[i].section_delta = 0;
+ }
for (i = 0; i < env->prog->len; i++) {
env->insn_aux_data[i].res_none = false;
env->insn_aux_data[i].res_none_r0 = false;
@@ -1232,13 +1346,13 @@ static int emit_acquire_after(struct bpf_insn *buf, const struct bpf_res_acq *a)
/*
* A frame above the one that holds what acquisition @acq acquired releases it,
- * and the acquisition has a slot. The holder can clear the slot itself, in
- * front of the call the release happens in, when the resource is gone after
- * every such call and no throw finds the holder with it at the call: the slot
- * is empty then for as long as the callee can throw after the release, and it
- * was no use before it. The call is one of a subprog: a helper that calls a
- * callback may not call it at all, and the holder has the resource after it
- * then.
+ * without being handed it, and the acquisition has a slot. The holder can
+ * clear the slot itself, in front of the call the release happens in, when
+ * the resource is gone after every such call and no throw finds the holder
+ * with it at the call: the slot is empty then for as long as the callee can
+ * throw after the release, and it was no use before it. The call is one of a
+ * subprog: a helper that calls a callback may not call it at all, and the
+ * holder has the resource after it then.
*/
static bool foreign_cleared(struct bpf_verifier_env *env, u16 acq, const u32 *new_of, u32 orig_len)
{
@@ -1282,7 +1396,7 @@ static int slot_can_be_kept(struct bpf_verifier_env *env, u32 acq)
for (i = 0; i < env->res_acq_cnt; i++) {
b = &env->res_acqs[i];
if (i == acq || b->insn != a->insn || b->reg != a->reg ||
- (a->flags ^ b->flags) & BPF_RES_F_CTX)
+ (a->flags ^ b->flags) & (BPF_RES_F_CTX | BPF_RES_F_ENTRY))
continue;
verbose(env, "frame_desc: insn %d acquires resources of different types on different paths\n",
a->insn);
@@ -1387,8 +1501,24 @@ int bpf_frame_desc_resolve(struct bpf_verifier_env *env)
ret = -EINVAL;
break;
}
+ /* the holder emptied the slot for the callee, which did not release */
+ if (a->handed_back) {
+ verbose(env, "frame_desc: what insn %d acquires is in a hidden slot, and a subprog it is passed to releases it on some paths only, which is not supported in a program that throws\n",
+ a->insn);
+ ret = -EINVAL;
+ break;
+ }
+ /* the slot of what a subprog is handed is filled on every call of it */
+ if ((a->flags & BPF_RES_F_ENTRY) &&
+ (bpf_find_containing_subprog(env, new_of[a->insn])->entry_none & BIT(a->reg))) {
+ verbose(env, "frame_desc: R%d of the subprog at insn %d is what its caller holds on some calls only, which is not supported in a program that throws\n",
+ a->reg, a->insn);
+ ret = -EINVAL;
+ break;
+ }
/* some visit of the insn has nothing a slot can hold where this one has */
- if (!(a->flags & BPF_RES_F_CTX) && acquires_sometimes(env, new_of[a->insn], a)) {
+ if (!(a->flags & (BPF_RES_F_CTX | BPF_RES_F_ENTRY)) &&
+ acquires_sometimes(env, new_of[a->insn], a)) {
verbose(env, "frame_desc: insn %d acquires a resource on some paths only, or one of a type that cannot be released\n",
a->insn);
ret = -EINVAL;
@@ -1535,10 +1665,11 @@ static int emit_items(struct bpf_verifier_env *env, const struct res_item *it, i
}
/*
- * At the entry of each subprog: its slots and counts start out empty, and
- * the main program takes the references the kernel passed it that it keeps
- * in a slot from the context, which R1 points at. The insns go in front of
- * the first one, which a jump to it skips.
+ * At the entry of each subprog: its slots and counts start out empty, but for
+ * what it is handed: the main program takes the references the kernel passed
+ * it that it keeps in a slot from the context, which R1 points at, and a
+ * subprog what its caller hands it from the argument. The insns go in front
+ * of the first one, which a jump to it skips.
*/
static int emit_entries(struct bpf_verifier_env *env)
{
@@ -1547,7 +1678,7 @@ static int emit_entries(struct bpf_verifier_env *env)
int s, cnt, ret = 0;
new_of = insns_of_pass(env, &orig_len);
- buf = kvcalloc(2 * (env->res_acq_cnt + 2), sizeof(*buf), GFP_KERNEL_ACCOUNT);
+ buf = kvcalloc(4 * (env->res_acq_cnt + 2), sizeof(*buf), GFP_KERNEL_ACCOUNT);
if (!new_of || !buf) {
ret = -ENOMEM;
goto out;
@@ -1564,6 +1695,13 @@ static int emit_entries(struct bpf_verifier_env *env)
if (a->flags & BPF_RES_F_CTX) {
buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_1, a->obj_off);
buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_AX, a->off);
+ } else if ((a->flags & BPF_RES_F_ENTRY) && a->adj) {
+ buf[cnt++] = BPF_MOV64_REG(BPF_REG_AX, a->reg);
+ buf[cnt++] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, 1);
+ buf[cnt++] = BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, a->adj);
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_AX, a->off);
+ } else if (a->flags & BPF_RES_F_ENTRY) {
+ buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, a->reg, a->off);
} else {
buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, a->off, 0);
}
@@ -1619,8 +1757,10 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
for (i = 0; i < env->res_call_cnt; i++)
if (env->res_calls[i].clear)
items[n++] = (struct res_item) { new_of[env->res_calls[i].insn], i, ITEM_CLEAR };
+ /* the entry of a subprog fills the slots of what it is handed, see emit_entries() */
for (i = 0; i < env->res_acq_cnt; i++)
- if (env->res_acqs[i].slot && !(env->res_acqs[i].flags & BPF_RES_F_CTX))
+ if (env->res_acqs[i].slot &&
+ !(env->res_acqs[i].flags & (BPF_RES_F_CTX | BPF_RES_F_ENTRY)))
items[n++] = (struct res_item) { new_of[env->res_acqs[i].insn], i, ITEM_ACQUIRE };
sort(items, n, sizeof(*items), item_cmp, NULL);
for (i = 0; i < n; i = k) {
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 87d87d359312..913025ee1f73 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -463,9 +463,11 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
fa->entry_rcu_locks != fb->entry_rcu_locks ||
fa->entry_preempt_locks != fb->entry_preempt_locks)
return false;
- /* the sections a frame left of those it was entered in matter to a throw, see frame_desc.c */
+ /* the sections a frame left and what it was handed matter to a throw, see frame_desc.c */
if (env->describe_frames && (fa->rcu_released_below != fb->rcu_released_below ||
- fa->preempt_released_below != fb->preempt_released_below))
+ fa->preempt_released_below != fb->preempt_released_below ||
+ memcmp(fa->entry_arg_acq, fb->entry_arg_acq,
+ sizeof(fa->entry_arg_acq))))
return false;
if (!join_loop_stack(fa, fb))
return false;
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 01eb8d9077ef..ecea6d762e14 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -717,9 +717,11 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
old->entry_rcu_locks != cur->entry_rcu_locks ||
old->entry_preempt_locks != cur->entry_preempt_locks)
return false;
- /* the sections a frame left of those it was entered in matter to a throw, see frame_desc.c */
+ /* the sections a frame left and what it was handed matter to a throw, see frame_desc.c */
if (env->describe_frames && (old->rcu_released_below != cur->rcu_released_below ||
- old->preempt_released_below != cur->preempt_released_below))
+ old->preempt_released_below != cur->preempt_released_below ||
+ memcmp(old->entry_arg_acq, cur->entry_arg_acq,
+ sizeof(old->entry_arg_acq))))
return false;
for (i = 0; i < MAX_BPF_REG; i++)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ec9f77a63197..1f41095f4f1f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11451,6 +11451,12 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
err = set_callee_state_cb(env, caller, callee, callsite);
if (err)
goto err_out;
+ /* a plain call hands the callee what it releases of the caller's, see frame_desc.c */
+ err = bpf_frame_desc_record_call(env, set_callee_state_cb == set_callee_state &&
+ bpf_pseudo_call(&env->prog->insnsi[callsite]) ? state : NULL,
+ callee, callsite);
+ if (err)
+ goto err_out;
/* only increment it after check_reg_arg() finished */
state->curframe++;
@@ -11859,6 +11865,12 @@ static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int i
return err;
}
}
+ /* what it is called with, should it release some of what it inherits */
+ err = bpf_frame_desc_record_call(env, NULL, callee, insn_idx);
+ if (err) {
+ bpf_free_verifier_state(&entry, false);
+ return err;
+ }
account_current_path(env);
env->subprog_info[subprog].on_its_own = true;
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it
2026-11-11 12:57 ` [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 16:57 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3681 bytes --]
> bpf: Transfer a reference to the callee that releases it
TL;DR: bpf_frame_desc_record_call() skips its speculative-state guard for
callbacks and callx (state is NULL), so a speculatively explored call may
wrongly mark an argument entry_none and cause the verifier to reject a valid
program.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index d2ad02925f06..6679f134b14e 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> +int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
> + struct bpf_func_state *callee, int callsite)
> +{
> + struct bpf_subprog_info *sub = &env->subprog_info[callee->subprogno];
> + struct bpf_reference_state *ref;
> + int i, k, acq, err;
> + s16 adj;
> +
> + if (!env->describe_frames || (st && st->speculative))
> + return 0;
setup_func_entry() passes st == NULL for every callback and every callx:
err = bpf_frame_desc_record_call(env, set_callee_state_cb == set_callee_state &&
bpf_pseudo_call(&env->prog->insnsi[callsite]) ? state : NULL,
callee, callsite);
Can the state still be speculative in those cases? push_callback_call()
builds callback_state with push_stack(), which copies cur_state including
->speculative, and check_func_callx_one() -> check_static_func_call() ->
setup_func_entry() runs on env->cur_state. Speculative states come from
sanitize_speculative_path() on the dead side of a decided branch when
!env->bypass_spec_v1, and from the code after an unwind_goto.
With st == NULL the speculative guard is skipped:
> + for (k = BPF_REG_1; k <= BPF_REG_5; k++) {
> + struct bpf_reg_state *reg = &callee->regs[k];
> +
> + callee->entry_arg_id[k - 1] = 0;
> + callee->entry_arg_acq[k - 1] = 0;
> + if (reg->type == SCALAR_VALUE || reg->type == NOT_INIT) {
> + if ((sub->consumes & BIT(k)) && !bpf_register_is_null(reg))
> + sub->entry_none |= BIT(k);
> + continue;
> + }
> + callee->entry_arg_id[k - 1] = reg->id;
> + if (!(sub->consumes & BIT(k)))
> + continue;
> + for (i = 0, ref = NULL; st && i < st->acquired_refs; i++) {
[ ... ]
> + if (!ref) {
> + sub->entry_none |= BIT(k);
> + continue;
> + }
For a consumed register holding a pointer or a non-zero scalar this sets
sub->entry_none |= BIT(k). entry_none lasts for the whole round, and
bpf_frame_desc_resolve() then rejects the program with "R%d of the subprog
at insn %d is what its caller holds on some calls only" when the subprog's
BPF_RES_F_ENTRY acquisition needs a hidden slot.
Every other recorder (bpf_frame_desc_record_acquire(), _record_release(),
_record_exit(), _record_return(), _record_site()) skips speculative states,
and so does the direct-call path of this function. Could a callback call
such as bpf_loop(..., cb, ctx, ...), or a callx of a consuming subprog,
that is reached only on a speculatively explored path wrongly mark the
argument as one that some call passes something else in, and cause a
program the non-speculative paths would accept to be rejected?
Would testing env->cur_state->speculative (or the passed state) separately
from whether the call hands anything over make this consistent with the
other recorders?
The later commit acd08268bd12 ("bpf: Support callx when a callee releases
resources") changes setup_func_entry() to pass the state for callx, so the
guard covers callx from then on. Callbacks still pass NULL at the end of
the series.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it
2026-11-11 12:57 ` [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 16:57 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 16:57 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] bpf: reference transfer via dead args in
bpf_frame_desc_record_call()
- [Medium] bpf: wrong consumed arg in foreign_release() when regs share
ref->id
--
commit f7b2132e50e4597119ba4ba7d888e76f32a3bd7e
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Transfer a reference to the callee that releases it
This commit transfers references passed in argument registers from the caller
to a subprog that releases them, tracking the acquired reference in the
callee's frame descriptor or hidden stack slot and returning it if unreleased.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index d2ad02925f061..6679f134b14ee 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -736,7 +742,7 @@ static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifi
> const struct bpf_func_state *below = st->frame[ref->frameno + 1];
> struct bpf_subprog_info *sub;
> struct bpf_res_call *c;
> - int f;
> + int f, k;
>
> for (f = ref->frameno + 1; f <= st->curframe; f++) {
> sub = &env->subprog_info[st->frame[f]->subprogno];
> @@ -745,6 +751,21 @@ static int foreign_release(struct bpf_verifier_env *env, const struct bpf_verifi
> sub->releases_inherited = true;
> env->frame_desc_redo = true;
> }
> + /* every frame it was passed down through in a register is handed it by the one below */
> + for (f = ref->frameno + 1; f <= st->curframe; f++) {
> + if (!ref->id || ref->type != REF_TYPE_PTR || (acq->flags & BPF_RES_F_ARG))
> + break;
> + for (k = BPF_REG_1; k <= BPF_REG_5; k++)
> + if (st->frame[f]->entry_arg_id[k - 1] == ref->id)
> + break;
> + if (k > BPF_REG_5)
> + break;
> + sub = &env->subprog_info[st->frame[f]->subprogno];
> + if (sub->consumes & BIT(k))
> + continue;
> + sub->consumes |= BIT(k);
> + env->frame_desc_redo = true;
> + }
[Severity: Medium]
Can foreign_release() mark the wrong argument register in sub->consumes when
two arguments at subprog entry share ref->id?
bpf_frame_desc_record_call() records reg->id in callee->entry_arg_id[k - 1]
for every non-scalar argument without checking reg_ref_off(), and
foreign_release() picks the lowest register k matching ref->id and breaks. If
a caller passes a pointer sharing ref->id in R1 that fails reg_ref_off() (such
as a bpf_skc_to_mptcp_sock() result or a variable-offset ringbuf pointer) and
passes the base reference in R2 that the subprog actually releases,
foreign_release() marks R1 instead of R2. On the next round,
bpf_frame_desc_record_call() fails reg_ref_off() on R1, sets entry_none on R1,
and never transfers R2.
Even when reg_ref_off() succeeds on both registers (for example, one call passes
(&p1->field, p1) where the subprog releases R2, and another call passes
(&p2->field, p3) while keeping p2 in a hidden slot), matching R1 first causes
the second call to transfer and hand back p2 on R1, and once both R1 and R2
are in sub->consumes, transferring p1 on R1 at the first call moves
ref->frameno to the callee so R2 finds no caller reference and sets
entry_none. bpf_frame_desc_resolve() then rejects the program.
[ ... ]
> @@ -869,6 +916,71 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
> return 0;
> }
>
> +/*
> + * Frame @callee of @st is entered with its arguments in place, by the call at
> + * insn @callsite of the frame below it when @st is given; a callback, a
> + * callx and a function verified on its own come with none.
[ ... ]
> + */
> +int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
> + struct bpf_func_state *callee, int callsite)
> +{
> + struct bpf_subprog_info *sub = &env->subprog_info[callee->subprogno];
> + struct bpf_reference_state *ref;
> + int i, k, acq, err;
> + s16 adj;
> +
> + if (!env->describe_frames || (st && st->speculative))
> + return 0;
> + for (k = BPF_REG_1; k <= BPF_REG_5; k++) {
> + struct bpf_reg_state *reg = &callee->regs[k];
> +
> + callee->entry_arg_id[k - 1] = 0;
> + callee->entry_arg_acq[k - 1] = 0;
> + if (reg->type == SCALAR_VALUE || reg->type == NOT_INIT) {
> + if ((sub->consumes & BIT(k)) && !bpf_register_is_null(reg))
> + sub->entry_none |= BIT(k);
> + continue;
> + }
> + callee->entry_arg_id[k - 1] = reg->id;
> + if (!(sub->consumes & BIT(k)))
> + continue;
> + for (i = 0, ref = NULL; st && i < st->acquired_refs; i++) {
> + ref = &st->refs[i];
> + if (ref->frameno == callee->frameno - 1 && ref->acq && ref->type == REF_TYPE_PTR &&
> + !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
> + reg_ref_off(reg, ref->id, &adj))
> + break;
> + ref = NULL;
> + }
> + if (!ref) {
> + sub->entry_none |= BIT(k);
> + continue;
> + }
> + err = rel_add(env, callsite, ref->acq - 1, k, 0, adj, 0);
> + if (err)
> + return err;
> + acq = acq_add(env, sub->start, ref->res - 1, k, BPF_RES_F_ENTRY, adj, 0);
> + if (acq < 0)
> + return acq;
> + callee->entry_arg_acq[k - 1] = ref->acq;
> + ref->frameno = callee->frameno;
> + ref->acq = acq + 1;
> + }
> + return 0;
> +}
[Severity: High]
Can bpf_frame_desc_record_call() transfer a reference through an argument
register that is dead at sub->start?
In setup_func_entry(), set_callee_state() copies R1..R5 from the caller to the
callee without filtering by env->insn_aux_data[sub->start].live_regs_before,
and bpf_frame_desc_record_call() runs before bpf_clean_verifier_state() scrubs
dead registers at sub->start. If an unused argument register Rk still holds a
copy of the caller's referenced pointer at the callsite (for example, R1 still
holds a copy of the pointer while the subprog only reads and releases R2),
foreign_release() sees entry_arg_id[k - 1] == ref->id and sets BIT(k) in
sub->consumes. On the next round, bpf_frame_desc_record_call() transfers the
reference via the dead register Rk, emitting an ITEM_RELEASE before the call
that reads Rk and an entry store at sub->start that writes Rk into the
callee's hidden slot.
Because Rk is not in live_regs_before at the callsite or sub->start and is not
in env->frame_desc_regs, bpf_do_misc_fixups() can pick Rk as a scratch
register when expanding a preceding timed_may_goto in the caller, overwriting
Rk at runtime with the per-CPU struct bpf_timed_may_goto pointer:
kernel/bpf/fixups.c:bpf_do_misc_fixups() {
...
u16 live = env->insn_aux_data[i].live_regs_before | env->frame_desc_regs;
int reg, save_off = 0;
/* a register that is dead here holds the address of the count */
for (reg = BPF_REG_0; reg <= BPF_REG_9; reg++)
if (!(live & BIT(reg)))
break;
...
}
At runtime, the caller's ITEM_RELEASE compares its hidden slot against the
clobbered Rk and fails to clear the caller's slot, while the callee's entry
stores the per-CPU timed_may_goto address into the callee's hidden slot. When
the callee later releases the real pointer via R2 and throws, the unwinder
passes the per-CPU timed_may_goto pointer from the callee's slot to the
resource destructor and then double frees the object from the caller's
uncleared slot. Checking env->insn_aux_data[sub->start].live_regs_before &
BIT(k) in bpf_frame_desc_record_call() would ignore registers that the callee
never reads.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=20
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (19 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 20/29] bpf: Transfer a reference to the callee that releases it Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 17:13 ` sashiko-bot
2026-11-11 12:57 ` [PATCH bpf-next v1 22/29] selftests/bpf: Test rejected throws that hold resources Kumar Kartikeya Dwivedi
` (8 subsequent siblings)
29 siblings, 2 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Currently a callx transfers no reference, and its caller does not clear
a hidden slot before it. A subprog that is called through a function
pointer can release a resource of its caller only if the caller keeps
the resource without a hidden slot.
A callx can call one of several subprogs, but the instructions inserted
before it run whichever subprog is called. These are the instructions
that clear the caller's hidden slot, both for a transferred reference
and for a resource that the callee releases without a transfer. They are
correct only if all the subprogs agree on what they release.
Treat a callx like a direct call, and record for each subprog it can
call which resources that subprog leaves with the caller:
- A reference that the caller passes in an argument through which the
subprog does not release. If the reference has a hidden slot and the
callx transfers it to another subprog in the same argument, the
program is rejected.
- Every resource of the caller, if the subprog releases no resources of
its callers. Such a subprog is not verified in the context of its
callers, so its return does not report that the caller still holds
the resource. Recording it at the call stops the caller from clearing
its hidden slot before the callx.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
include/linux/bpf_verifier.h | 10 ++++
kernel/bpf/frame_desc.c | 94 +++++++++++++++++++++++++++++++++---
kernel/bpf/verifier.c | 10 +++-
3 files changed, 104 insertions(+), 10 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 03290f7e42b5..67ae2631019f 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1367,10 +1367,19 @@ struct bpf_res_rel {
* acquisition of the caller acquired, without being handed it. The caller
* clears its slot in front of the call when no throw finds it holding the
* resource there and no return from the call does, see frame_desc.c.
+ *
+ * A callx calls one of several subprogs, and what is put in front of it runs
+ * whichever it calls: it has a record as well for what a subprog it calls
+ * leaves with the caller, see bpf_frame_desc_record_callx().
*/
struct bpf_res_call {
u32 insn; /* the call; U32_MAX when the caller is not in the state */
u16 acq; /* index of the acquisition */
+ /*
+ * The arguments, a bit per register, that a callx passes it in to a
+ * subprog that does not release through them.
+ */
+ u8 lent;
bool released;
bool held_after; /* some return from the call finds the caller holding it */
bool clear; /* decided once the pass is over */
@@ -2405,6 +2414,7 @@ void bpf_frame_desc_record_sections(struct bpf_verifier_env *env, int subprog, b
bool preempt);
int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct bpf_func_state *callee, int callsite);
+int bpf_frame_desc_record_callx(struct bpf_verifier_env *env, int subprog, int callsite);
bool bpf_frame_desc_keeps(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
int f, const struct bpf_reg_state *reg);
u16 bpf_frame_desc_kept_regs(const struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
index 6679f134b14e..11b2546bc179 100644
--- a/kernel/bpf/frame_desc.c
+++ b/kernel/bpf/frame_desc.c
@@ -39,7 +39,9 @@
* call: it is the callee's from its entry, and no longer the caller's.
* What it releases otherwise, the lock of its caller for one, has no
* slot, or one the caller can empty in front of the call, see
- * bpf_frame_desc_record_call() and foreign_release().
+ * bpf_frame_desc_record_call() and foreign_release(). A callx does both
+ * for every subprog it may call, so they have to agree on what they
+ * release, see bpf_frame_desc_record_callx().
* - A slot goes with its frame. A reference a subprog returns in R0 is
* acquired by the call for the caller, like the call of a kfunc that
* returns one, and one it was handed and did not release it gives back.
@@ -918,8 +920,8 @@ int bpf_frame_desc_record_exit(struct bpf_verifier_env *env)
/*
* Frame @callee of @st is entered with its arguments in place, by the call at
- * insn @callsite of the frame below it when @st is given; a callback, a
- * callx and a function verified on its own come with none.
+ * insn @callsite of the frame below it when @st is given; a callback and a
+ * function verified on its own come with none.
*
* What a subprog releases of its caller's through an argument, see
* bpf_subprog_info::consumes, the call hands over to it: the reference the
@@ -981,6 +983,56 @@ int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier
return 0;
}
+/*
+ * The callx at insn @callsite of the current frame calls @subprog, one of the
+ * subprogs it may call. What the frame puts in front of the callx runs
+ * whichever it calls: the insns that take what it hands over out of its slot,
+ * and the one that empties the slot of what a callee releases without being
+ * handed it, see foreign_cleared(). So note what this subprog leaves with the
+ * frame:
+ *
+ * - What the frame holds and passes it in an argument that it does not
+ * release through. Another subprog may be handed it in that argument, and
+ * the slot would be emptied for this one as well, see slot_can_be_kept().
+ * - All that the frame holds, when the subprog releases nothing of its
+ * callers'. It is not verified in their context then, so its return has
+ * no frame to say so, see bpf_frame_desc_record_exit().
+ */
+int bpf_frame_desc_record_callx(struct bpf_verifier_env *env, int subprog, int callsite)
+{
+ const struct bpf_subprog_info *sub = &env->subprog_info[subprog];
+ struct bpf_verifier_state *st = env->cur_state;
+ const struct bpf_func_state *frame = st->frame[st->curframe];
+ const struct bpf_reference_state *ref;
+ struct bpf_res_call *c;
+ int i, k;
+ u8 lent;
+ s16 adj;
+
+ if (!env->describe_frames || st->speculative)
+ return 0;
+ for (i = 0; i < st->acquired_refs; i++) {
+ ref = &st->refs[i];
+ if (ref->frameno != st->curframe || !ref->acq)
+ continue;
+ lent = 0;
+ for (k = BPF_REG_1; k <= BPF_REG_5; k++)
+ if (!(sub->consumes & BIT(k)) && ref->type == REF_TYPE_PTR &&
+ !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
+ reg_ref_off(&frame->regs[k], ref->id, &adj))
+ lent |= BIT(k);
+ if (!lent && sub->releases_inherited)
+ continue;
+ c = call_note(env, callsite, ref->acq - 1);
+ if (!c)
+ return -ENOMEM;
+ c->lent |= lent;
+ if (!sub->releases_inherited)
+ c->held_after = true;
+ }
+ return 0;
+}
+
/*
* A subprog called at insn @callsite returned to the current frame of @st
* with @r0, holding the reference refs[@idx], which is the caller's from here
@@ -1352,10 +1404,12 @@ static int emit_acquire_after(struct bpf_insn *buf, const struct bpf_res_acq *a)
* with it at the call: the slot is empty then for as long as the callee can
* throw after the release, and it was no use before it. The call is one of a
* subprog: a helper that calls a callback may not call it at all, and the
- * holder has the resource after it then.
+ * holder has the resource after it then. A callx is fine when that holds for
+ * every subprog it calls, see bpf_frame_desc_record_callx().
*/
static bool foreign_cleared(struct bpf_verifier_env *env, u16 acq, const u32 *new_of, u32 orig_len)
{
+ const struct bpf_insn *insn;
bool any = false;
u32 i, j;
@@ -1367,7 +1421,8 @@ static bool foreign_cleared(struct bpf_verifier_env *env, u16 acq, const u32 *ne
if (c->held_after || c->insn >= orig_len)
return false;
/* a helper may never call the callback that releases */
- if (!bpf_pseudo_call(&env->prog->insnsi[new_of[c->insn]]))
+ insn = &env->prog->insnsi[new_of[c->insn]];
+ if (!bpf_pseudo_call(insn) && !bpf_is_callx(insn))
return false;
for (j = 0; j < env->res_loc_cnt; j++)
if (env->res_locs[j].acq == acq &&
@@ -1381,11 +1436,26 @@ static bool foreign_cleared(struct bpf_verifier_env *env, u16 acq, const u32 *ne
return any;
}
+/*
+ * Does the callx @insn pass what @acq acquired in R@reg to a subprog that
+ * leaves it with its caller?
+ */
+static bool call_lends(const struct bpf_verifier_env *env, u32 insn, u16 acq, u8 reg)
+{
+ u32 i;
+
+ for (i = 0; i < env->res_call_cnt; i++)
+ if (env->res_calls[i].insn == insn && env->res_calls[i].acq == acq)
+ return env->res_calls[i].lent & BIT(reg);
+ return false;
+}
+
/*
* Acquisition @acq needs a slot. Can the insns that keep it be emitted? They
* are emitted once per insn, so the insn has to acquire in one way, with the
* handle in a register, and every call that releases what it acquired has to
- * take the handle in one way that insns can compare with the slot.
+ * take the handle in one way that insns can compare with the slot. A callx
+ * that hands it over has to do so whichever subprog it calls.
*/
static int slot_can_be_kept(struct bpf_verifier_env *env, u32 acq)
{
@@ -1409,7 +1479,14 @@ static int slot_can_be_kept(struct bpf_verifier_env *env, u32 acq)
}
for (i = 0; i < env->res_rel_cnt; i++) {
r = &env->res_rels[i];
- if (r->acq != acq || !r->bad)
+ if (r->acq != acq)
+ continue;
+ if (call_lends(env, r->insn, acq, r->reg)) {
+ verbose(env, "frame_desc: the callx at insn %d passes what R%d holds to a subprog that releases it and to one that does not, which is not supported in a program that throws\n",
+ r->insn, r->reg);
+ return -EINVAL;
+ }
+ if (!r->bad)
continue;
if (r->bad == BPF_RES_REL_NO_REG)
verbose(env, "frame_desc: insn %d releases a resource whose handle is in no argument register\n",
@@ -1778,7 +1855,8 @@ int bpf_frame_desc_emit(struct bpf_verifier_env *env)
for (k = i; k < n && items[k].idx == idx; k++)
;
- if (verifier_bug_if(idx >= len || env->prog->insnsi[idx].code != (BPF_JMP | BPF_CALL),
+ if (verifier_bug_if(idx >= len || (env->prog->insnsi[idx].code != (BPF_JMP | BPF_CALL) &&
+ !bpf_is_callx(&env->prog->insnsi[idx])),
env, "resource kept at insn %d, not a call", idx)) {
ret = -EFAULT;
break;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1f41095f4f1f..51896b200161 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11451,9 +11451,10 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
err = set_callee_state_cb(env, caller, callee, callsite);
if (err)
goto err_out;
- /* a plain call hands the callee what it releases of the caller's, see frame_desc.c */
+ /* the call of a subprog hands it what it releases of the caller's, see frame_desc.c */
err = bpf_frame_desc_record_call(env, set_callee_state_cb == set_callee_state &&
- bpf_pseudo_call(&env->prog->insnsi[callsite]) ? state : NULL,
+ (bpf_pseudo_call(&env->prog->insnsi[callsite]) ||
+ bpf_is_callx(&env->prog->insnsi[callsite])) ? state : NULL,
callee, callsite);
if (err)
goto err_out;
@@ -12280,6 +12281,11 @@ static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int *
return err;
env->subprog_info[subprog].via_callx = true;
+ /* what the insns in front of the callx do goes for every subprog it calls, see frame_desc.c */
+ err = bpf_frame_desc_record_callx(env, subprog, *insn_idx);
+ if (err)
+ return err;
+
return check_static_func_call(env, subprog, insn_idx);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources
2026-11-11 12:57 ` [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources Kumar Kartikeya Dwivedi
@ 2026-11-11 14:04 ` bot+bpf-ci
2026-11-11 17:13 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: bot+bpf-ci @ 2026-11-11 14:04 UTC (permalink / raw)
To: memxor, bpf
Cc: ast, andrii, daniel, eddyz87, emil, ihor.solodrai, kkd,
kernel-team, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2899 bytes --]
> bpf: Support callx when a callee releases resources
TL;DR: Comment and commit message only: the stated reason a subprog releasing
nothing of its callers' leaves everything with the caller (not verified in
their context) looks inaccurate; the code itself seems correct.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> +/*
> + * The callx at insn @callsite of the current frame calls @subprog, one of the
> + * subprogs it may call. What the frame puts in front of the callx runs
> + * whichever it calls: the insns that take what it hands over out of its slot,
> + * and the one that empties the slot of what a callee releases without being
> + * handed it, see foreign_cleared(). So note what this subprog leaves with the
> + * frame:
> + *
> + * - What the frame holds and passes it in an argument that it does not
> + * release through. Another subprog may be handed it in that argument, and
> + * the slot would be emptied for this one as well, see slot_can_be_kept().
> + * - All that the frame holds, when the subprog releases nothing of its
> + * callers'. It is not verified in their context then, so its return has
> + * no frame to say so, see bpf_frame_desc_record_exit().
> + */
This isn't a bug, but is the reason given for the second item accurate? The
comment above, and the commit message ("Such a subprog is not verified in
the context of its callers, so its return does not report that the caller
still holds the resource"), say a subprog that releases nothing of its
callers' is not verified in their context.
func_is_on_its_own() only looks at sub->releases_inherited to send such a
subprog to the caller's context. It still returns false, and
check_static_func_call() goes through setup_func_entry() and verifies the
callee in the caller's context, for several cases: changes_pkt_data,
has_tail_call, has_ld_abs, is_cb, speculative state,
in_rbtree_lock_required_cb(), more than BPF_ID_MAP_SIZE/2 refs, a live
PTR_TO_STACK or CONST_PTR_TO_DYNPTR argument, and arg_needs_caller().
The real reason its return reports nothing seems to be the early return in
bpf_frame_desc_record_exit():
if (!st->curframe || !env->subprog_info[frame->subprogno].releases_inherited)
return 0;
The code looks correct, since bpf_frame_desc_record_callx() sets held_after
whenever !sub->releases_inherited however the subprog is verified. Could the
comment and the commit message say something like "its return does not say
so, since bpf_frame_desc_record_exit() looks only at subprogs that release
their callers' resources"?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43027088527
^ permalink raw reply [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources
2026-11-11 12:57 ` [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources Kumar Kartikeya Dwivedi
2026-11-11 14:04 ` bot+bpf-ci
@ 2026-11-11 17:13 ` sashiko-bot
1 sibling, 0 replies; 58+ messages in thread
From: sashiko-bot @ 2026-11-11 17:13 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] bpf: resource leak via bypassed lent check in
bpf_frame_desc_record_callx()
- [Medium] bpf: hidden slot cleared without release in
bpf_frame_desc_record_callx()
--
commit 10ca45cf5c85b803527fe102c801dcc6b5d2cc99
Author: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf: Support callx when a callee releases resources
This commit extends frame descriptor tracking to support callx instructions
whose target subprogs transfer or release caller-held resources, rejecting
programs when targets disagree on whether an argument reference is consumed.
> diff --git a/kernel/bpf/frame_desc.c b/kernel/bpf/frame_desc.c
> index 6679f134b14ee..11b2546bc179f 100644
> --- a/kernel/bpf/frame_desc.c
> +++ b/kernel/bpf/frame_desc.c
[ ... ]
> @@ -981,6 +983,56 @@ int bpf_frame_desc_record_call(struct bpf_verifier_env *env, struct bpf_verifier
> return 0;
> }
>
> +/*
> + * The callx at insn @callsite of the current frame calls @subprog, one of the
> + * subprogs it may call. What the frame puts in front of the callx runs
> + * whichever it calls: the insns that take what it hands over out of its slot,
> + * and the one that empties the slot of what a callee releases without being
> + * handed it, see foreign_cleared(). So note what this subprog leaves with the
> + * frame:
> + *
> + * - What the frame holds and passes it in an argument that it does not
> + * release through. Another subprog may be handed it in that argument, and
> + * the slot would be emptied for this one as well, see slot_can_be_kept().
> + * - All that the frame holds, when the subprog releases nothing of its
> + * callers'. It is not verified in their context then, so its return has
> + * no frame to say so, see bpf_frame_desc_record_exit().
> + */
> +int bpf_frame_desc_record_callx(struct bpf_verifier_env *env, int subprog, int callsite)
> +{
> + const struct bpf_subprog_info *sub = &env->subprog_info[subprog];
> + struct bpf_verifier_state *st = env->cur_state;
> + const struct bpf_func_state *frame = st->frame[st->curframe];
> + const struct bpf_reference_state *ref;
> + struct bpf_res_call *c;
> + int i, k;
> + u8 lent;
> + s16 adj;
> +
> + if (!env->describe_frames || st->speculative)
> + return 0;
> + for (i = 0; i < st->acquired_refs; i++) {
> + ref = &st->refs[i];
> + if (ref->frameno != st->curframe || !ref->acq)
> + continue;
> + lent = 0;
> + for (k = BPF_REG_1; k <= BPF_REG_5; k++)
> + if (!(sub->consumes & BIT(k)) && ref->type == REF_TYPE_PTR &&
> + !(env->res_acqs[ref->acq - 1].flags & BPF_RES_F_ARG) &&
> + reg_ref_off(&frame->regs[k], ref->id, &adj))
> + lent |= BIT(k);
[Severity: High]
Can a caller's hidden slot still be cleared before a callx to a subprog that
does not consume Rk, leaking the reference on a later bpf_throw()?
When one target of a callx consumes Rk, emit_release() emits instructions
before the callx that compare the 64-bit runtime value of Rk with the caller's
hidden slot and zero the slot if they match. However,
bpf_frame_desc_record_callx() only sets BIT(k) in lent when reg_ref_off()
succeeds in the verifier state at the callx.
If another path reaches the same callx to invoke a non-consuming subprog while
Rk still holds the handle address at runtime, but its verifier state fails
reg_ref_off() (for instance, Rk is NOT_INIT after merging with a path that did
not set Rk, or after an earlier call to a no-op subprog cleared caller-saved
registers in the verifier while leaving Rk intact in hardware), BIT(k) is not
set in c->lent. slot_can_be_kept() then accepts the program, the pre-callx
sequence zeroes the caller's hidden slot at runtime before the non-consuming
subprog runs, and a subsequent bpf_throw() fails to release the resource.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 1f41095f4f1fa..51896b2001614 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -11451,9 +11451,10 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
> err = set_callee_state_cb(env, caller, callee, callsite);
> if (err)
> goto err_out;
> - /* a plain call hands the callee what it releases of the caller's, see frame_desc.c */
> + /* the call of a subprog hands it what it releases of the caller's, see frame_desc.c */
> err = bpf_frame_desc_record_call(env, set_callee_state_cb == set_callee_state &&
> - bpf_pseudo_call(&env->prog->insnsi[callsite]) ? state : NULL,
> + (bpf_pseudo_call(&env->prog->insnsi[callsite]) ||
> + bpf_is_callx(&env->prog->insnsi[callsite])) ? state : NULL,
> callee, callsite);
[Severity: Medium]
What happens if a callx invokes a subprog with sub->consumes & BIT(k) set on a
path where Rk holds the runtime address of a caller-held reference without
satisfying reg_ref_off()?
Because sub->consumes & BIT(k) is set, bpf_frame_desc_record_callx() skips Rk.
In bpf_frame_desc_record_call(), if one callx path passes the caller's
reference in Rk (recording rel_add() at callsite, with the callee releasing it
without needing its own hidden slot) and a second path passes the same runtime
pointer in Rk in a form that fails reg_ref_off() (such as a pointer with a
variable offset in [0, 1] that is 0 at runtime, or an untrusted pointer from
bpf_rdonly_cast()) on a branch where the callee does not release it,
bpf_frame_desc_record_call() only sets sub->entry_none |= BIT(k) and leaves
the reference with the caller.
Because the callee's BPF_RES_F_ENTRY acquisition does not need a hidden slot
of its own, bpf_frame_desc_resolve() never checks sub->entry_none, and
handed_back is not set on exit. The emit_release() instructions placed before
the callx still zero the caller's hidden slot at runtime on the second path,
leaking the reference if the caller later throws.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794399387.git.memxor@gmail.com?part=21
^ permalink raw reply [flat|nested] 58+ messages in thread
* [PATCH bpf-next v1 22/29] selftests/bpf: Test rejected throws that hold resources
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (20 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 21/29] bpf: Support callx when a callee releases resources Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 23/29] selftests/bpf: Test resource cleanup on bpf_throw() Kumar Kartikeya Dwivedi
` (7 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add a program to exceptions_fail for each rule that a program must follow
when it throws while holding resources. The programs are rejected
because:
- a callee may throw while the lock of its caller is held, and again
after it has released the lock;
- a callee is passed a record that its caller keeps in a hidden slot,
and returns without discarding it on some paths;
- a callee leaves an RCU section of a caller that counts its sections in
a hidden slot;
- an object is released while the lock inside it is held;
- an iterator or saved IRQ flags on the stack are overwritten;
- an RCU protected iterator outlives its RCU section.
The programs run on x86-64 and arm64 only, where these rules apply.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../selftests/bpf/progs/exceptions_fail.c | 214 ++++++++++++++++++
1 file changed, 214 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index f02882224a03..9b47345d5fd0 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -458,4 +458,218 @@ int reject_subprog_throw_irq_lock(void *ctx)
return 0;
}
+/*
+ * What a program that throws may not do yet: the frames keep what they hold
+ * in hidden slots of their own, see kernel/bpf/frame_desc.c.
+ */
+
+struct locked_obj {
+ struct bpf_spin_lock lock;
+ int data;
+};
+
+private(L) struct bpf_spin_lock held_lock;
+
+static __noinline void throw_around_unlock_held(int before, int after)
+{
+ if (before)
+ bpf_throw(0);
+ bpf_spin_unlock(&held_lock);
+ if (after)
+ bpf_throw(0);
+}
+
+/*
+ * The callee may throw with the lock of its caller held and after it dropped
+ * it, so the call of the caller says neither. The lock is in a slot of the
+ * caller then, which the callee cannot reach to empty.
+ */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("releases what another frame holds in a hidden slot")
+int reject_callee_releases(struct __sk_buff *ctx)
+{
+ bpf_spin_lock(&held_lock);
+ throw_around_unlock_held(ctx->len, ctx->mark);
+ return 0;
+}
+
+struct {
+ __uint(type, BPF_MAP_TYPE_RINGBUF);
+ __uint(max_entries, 4096);
+} fail_ringbuf SEC(".maps");
+
+/* Discards the record when told. */
+__used static __noinline int discard_if(void *rec, int drop)
+{
+ if (drop)
+ bpf_ringbuf_discard(rec, 0);
+ return 0;
+}
+
+/*
+ * The two paths to the throw keep the record in different registers, so it is
+ * in a slot, which the caller empties for the callee it hands the record to.
+ * The callee returns holding the record when it is not told to discard it.
+ */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("releases it on some paths only")
+__naked int reject_callee_releases_sometimes(void)
+{
+ asm volatile (
+ "r7 = *(u32 *)(r1 + %[len]);"
+ "r8 = *(u32 *)(r1 + %[mark]);"
+ "r2 = *(u32 *)(r1 + %[priority]);"
+ "*(u64 *)(r10 - 8) = r2;"
+ "r1 = %[fail_ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 4f;"
+ "r6 = r0;"
+ "r9 = 0;"
+ "r0 = 0;"
+ "if r7 == 0 goto 1f;"
+ "r9 = r6;"
+ "r6 = 0;"
+ "1: r1 = *(u64 *)(r10 - 8);"
+ "if r1 == 0 goto 2f;"
+ "r1 = 0;"
+ "call %[bpf_throw];"
+ "2: r1 = r6;"
+ "if r1 != 0 goto 3f;"
+ "r1 = r9;"
+ "3: r2 = r8;"
+ "call discard_if;"
+ "if r8 != 0 goto 4f;"
+ "r1 = r6;"
+ "if r1 != 0 goto +1;"
+ "r1 = r9;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "4: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(fail_ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len)),
+ [mark]"i"(offsetof(struct __sk_buff, mark)),
+ [priority]"i"(offsetof(struct __sk_buff, priority))
+ : __clobber_all);
+}
+
+__used static __noinline void leave_rcu(void)
+{
+ bpf_rcu_read_unlock();
+}
+
+/*
+ * The paths to the throw are in an RCU read section or not, so the frame
+ * counts its sections in a slot. The count does not follow what a callee
+ * does: the callee leaves the section it was called in.
+ */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("a subprog returns in other sections than it was called in")
+__naked int reject_callee_leaves_rcu(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r7 = *(u32 *)(r1 + %[mark]);"
+ "if r6 == 0 goto 1f;"
+ "call %[bpf_rcu_read_lock];"
+ "1: if r7 == 0 goto 2f;"
+ "r1 = 0;"
+ "call %[bpf_throw];"
+ "2: if r6 != 0 goto 3f;"
+ "call %[bpf_rcu_read_lock];"
+ "3: call leave_rcu;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len)),
+ [mark]"i"(offsetof(struct __sk_buff, mark))
+ : __clobber_all);
+}
+
+struct obj_map_value {
+ struct locked_obj __kptr *obj;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct obj_map_value);
+} obj_map SEC(".maps");
+
+/* The unwinder would unlock a lock in an object that may be freed by then. */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("releases an object whose lock is held")
+int reject_release_locked_obj(struct __sk_buff *ctx)
+{
+ struct obj_map_value *v;
+ struct locked_obj *o;
+
+ v = bpf_map_lookup_elem(&obj_map, &(int){0});
+ if (!v)
+ return 0;
+ o = bpf_obj_new(typeof(*o));
+ if (!o)
+ return 0;
+ bpf_spin_lock(&o->lock);
+ o = bpf_kptr_xchg(&v->obj, o);
+ bpf_throw(0);
+ return 0;
+}
+
+/* The unwinder restores what is at the address of the flags. */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("cannot write over saved irq flags in a program that throws")
+int reject_write_over_irq_flags(struct __sk_buff *ctx)
+{
+ unsigned long flags;
+
+ bpf_local_irq_save(&flags);
+ *(volatile unsigned long *)&flags = 0;
+ bpf_throw(0);
+ return 0;
+}
+
+/* The unwinder destroys what is at the address of the iterator. */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("cannot write over an iterator in a program that throws")
+int reject_write_over_iter(struct __sk_buff *ctx)
+{
+ struct bpf_iter_num it;
+
+ bpf_iter_num_new(&it, 0, 10);
+ *(volatile __u64 *)&it = 0x1234;
+ bpf_throw(0);
+ return 0;
+}
+
+/* Nothing may destroy the iterator once the section it is protected by ended. */
+SEC("?syscall")
+__arch_x86_64 __arch_arm64
+__failure __msg("an RCU protected iterator may not outlive its section in a program that throws")
+int reject_rcu_iter_outlives_section(void *ctx)
+{
+ struct bpf_iter_task it;
+
+ bpf_rcu_read_lock();
+ bpf_iter_task_new(&it, NULL, BPF_TASK_ITER_ALL_PROCS);
+ bpf_rcu_read_unlock();
+ bpf_throw(0);
+ return 0;
+}
+
char _license[] SEC("license") = "GPL";
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 23/29] selftests/bpf: Test resource cleanup on bpf_throw()
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (21 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 22/29] selftests/bpf: Test rejected throws that hold resources Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 24/29] selftests/bpf: Test cleanup of handles on the stack Kumar Kartikeya Dwivedi
` (6 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add exceptions_cleanup. It runs programs that throw while holding
resources, and checks which resources the unwinder released and in which
order.
Each program records the handle of every resource it acquires. Tracing
programs attached to bpf_unwind_release() record every release during an
unwind. The test compares the two.
The programs cover:
- the kinds of resources: local objects, ringbuf records, tasks, sockets
and their casts, iterators, a file dynptr, spin locks, saved IRQ
flags, RCU read and preemption disabled sections, and the referenced
argument of a struct_ops program;
- throws from the main program, from static and global subprogs,
through an indirect call, and out of a recursion, with every frame
holding resources;
- a reference that a subprog returns to its caller;
- registers that hold the handle at the throw only on some paths, or on
none;
- one releasing instruction shared by two acquisitions;
- acquisitions that can fail and leave the slot empty;
- the release order: locks and IRQ flags first, then sections, then a
destructor that sleeps.
For the struct_ops program, bpf_testmod calls test_refcounted() with a
reference to current when the struct_ops is registered.
The suite runs a second time with bpf_jit_harden set, so the test is
serial.
Add verifier_frame_desc. It checks the number of hidden slots and
inserted instructions that the verifier reports for a few programs, and
that two records reserved by the same instruction and held at once are
rejected.
Both tests run on x86-64 and arm64 only. Elsewhere a program still cannot
throw while it holds resources.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 257 ++++
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/exceptions_cleanup.c | 1200 +++++++++++++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 200 +++
.../selftests/bpf/test_kmods/bpf_testmod.c | 8 +
5 files changed, 1667 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_frame_desc.c
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
new file mode 100644
index 000000000000..b4213036cea7
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -0,0 +1,257 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "sysctl_helpers.h"
+
+#include "exceptions_cleanup.skel.h"
+
+/* The cookie the programs throw. */
+#define THROWN 0xeB9F
+
+/*
+ * What a run of a program has to show: its return value, the number of
+ * resources the program recorded acquiring, and the releases observed, in
+ * order, as a string: a digit names the acquired resource released at that
+ * point, 's' an RCU or preemption section, '?' any acquired resource not
+ * named elsewhere. @param is a choice the program reads.
+ */
+struct expect {
+ int retval;
+ int acquired;
+ const char *released;
+ __u32 param;
+};
+
+static void check_releases(struct exceptions_cleanup *skel, const struct expect *e)
+{
+ bool matched[ARRAY_SIZE(skel->bss->acquired)] = {};
+ int n = strlen(e->released), i, j;
+
+ ASSERT_EQ(skel->bss->acquired_cnt, e->acquired, "acquired");
+ if (!ASSERT_EQ(skel->bss->released_cnt, n, "released"))
+ return;
+ for (i = 0; i < n; i++) {
+ __u64 handle = skel->bss->released[i];
+ char c = e->released[i];
+
+ if (c == 's') {
+ ASSERT_TRUE(skel->bss->released_section[i], "section");
+ continue;
+ }
+ if (!ASSERT_FALSE(skel->bss->released_section[i], "not a section"))
+ continue;
+ if (c == '?') {
+ for (j = 0; j < e->acquired; j++)
+ if (!matched[j] && handle == skel->bss->acquired[j])
+ break;
+ if (ASSERT_LT(j, e->acquired, "released an acquired resource"))
+ matched[j] = true;
+ continue;
+ }
+ j = c - '0';
+ if (ASSERT_EQ(handle, skel->bss->acquired[j], "release order"))
+ matched[j] = true;
+ }
+}
+
+static void reset(struct exceptions_cleanup *skel, __u32 param)
+{
+ skel->bss->acquired_cnt = 0;
+ skel->bss->released_cnt = 0;
+ memset(skel->bss->acquired, 0, sizeof(skel->bss->acquired));
+ memset(skel->bss->released, 0, sizeof(skel->bss->released));
+ memset(skel->bss->released_section, 0, sizeof(skel->bss->released_section));
+ skel->bss->param = param;
+ skel->bss->test_pid = getpid();
+}
+
+/* Run @prog once and check what it returned and what was released against @e. */
+static void run_prog(struct exceptions_cleanup *skel, struct bpf_program *prog,
+ const struct expect *e)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, ropts,
+ .data_in = &pkt_v4,
+ .data_size_in = sizeof(pkt_v4),
+ .repeat = 1,
+ );
+ int ret;
+
+ reset(skel, e->param);
+ /* A syscall program runs on nothing: no input, and once, without a repeat count. */
+ if (bpf_program__type(prog) == BPF_PROG_TYPE_SYSCALL) {
+ ropts.data_in = NULL;
+ ropts.data_size_in = 0;
+ ropts.repeat = 0;
+ }
+ ret = bpf_prog_test_run_opts(bpf_program__fd(prog), &ropts);
+ if (!ASSERT_OK(ret, "test_run"))
+ return;
+ ASSERT_EQ(ropts.retval, e->retval, "retval");
+ check_releases(skel, e);
+}
+
+/*
+ * Register the struct_ops whose program throws: the kernel calls it with a
+ * task reference in a __ref argument as it registers, see bpf_dummy_reg().
+ */
+static void run_struct_ops(struct exceptions_cleanup *skel, const struct expect *e)
+{
+ struct bpf_link *link;
+
+ reset(skel, e->param);
+ link = bpf_map__attach_struct_ops(skel->maps.throwing_ops);
+ if (!ASSERT_OK_PTR(link, "attach_struct_ops"))
+ return;
+ check_releases(skel, e);
+ bpf_link__destroy(link);
+}
+
+/* Open a file, which runs the LSM program that throws with a file and a dynptr that reads it. */
+static void run_file(struct exceptions_cleanup *skel, const struct expect *e)
+{
+ int fd;
+
+ reset(skel, e->param);
+ skel->bss->file_armed = 1;
+ fd = open("/proc/self/exe", O_RDONLY);
+ if (!ASSERT_OK_FD(fd, "open"))
+ return;
+ close(fd);
+ ASSERT_EQ(skel->bss->file_armed, 0, "program ran");
+ check_releases(skel, e);
+}
+
+/* Run every program; @prefix names the pass in the subtest names. */
+static void run_suite(const char *prefix)
+{
+ struct exceptions_cleanup *skel;
+ int server = -1;
+ char name[64];
+
+ skel = exceptions_cleanup__open();
+ if (!ASSERT_OK_PTR(skel, "exceptions_cleanup__open"))
+ return;
+ /* registered by the subtests that run its program */
+ bpf_map__set_autoattach(skel->maps.throwing_ops, false);
+ if (!ASSERT_OK(exceptions_cleanup__load(skel), "exceptions_cleanup__load"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup__attach(skel), "attach"))
+ goto out;
+ /* The listener throw_sock_cast looks up, on an ephemeral port. */
+ server = start_server(AF_INET, SOCK_STREAM, "127.0.0.1", 0, 0);
+ if (!ASSERT_OK_FD(server, "start_server"))
+ goto out;
+ skel->bss->server_port = get_socket_local_port(server);
+
+/* Run program @pname as subtest @sname, expecting @ret, @n acquisitions and the releases @rel. */
+#define RUN_AS(pname, sname, ret, n, rel, ...) do { \
+ struct expect e = { .retval = ret, .acquired = n, .released = rel, ##__VA_ARGS__ }; \
+ \
+ snprintf(name, sizeof(name), "%s%s", prefix, sname); \
+ if (test__start_subtest(name)) \
+ run_prog(skel, skel->progs.pname, &e); \
+} while (0)
+#define RUN(pname, ret, n, rel, ...) RUN_AS(pname, #pname, ret, n, rel, ##__VA_ARGS__)
+#define THROW(pname, n, rel, ...) RUN(pname, THROWN, n, rel, ##__VA_ARGS__)
+ THROW(throw_obj_reg, 1, "0");
+ THROW(throw_obj_two, 2, "??");
+ THROW(throw_ringbuf, 1, "0");
+ THROW(throw_task, 1, "0");
+ THROW(throw_sock_cast, 1, "0");
+ THROW(throw_iter, 1, "0");
+ THROW(throw_obj_maybe, 1, "0");
+ THROW(throw_obj_caller, 1, "0");
+ THROW(throw_obj_callx_caller, 1, "0");
+ THROW(throw_obj_origin_after_guard, 1, "0");
+ THROW(throw_obj_unconditional, 1, "0");
+ THROW(throw_obj_dead_caller, 1, "0");
+ THROW(throw_obj_subprog, 1, "0");
+ THROW(throw_obj_global_caller, 1, "0");
+ /* Locks and IRQ flags go first, sections next, objects last. */
+ THROW(throw_spin_lock, 2, "10");
+ THROW(throw_rcu, 1, "s0");
+ THROW(throw_irq, 1, "0s");
+ THROW(throw_obj_after_caller_lock, 2, "01");
+ THROW(throw_sleeping_dtor_after_section, 1, "s0");
+ /* References the program keeps in different places on different paths, or nowhere. */
+ THROW(throw_reg_conflict, 1, "0");
+ THROW(throw_dead_reg, 1, "0");
+ THROW(throw_lock_two_locations, 2, "01");
+ RUN_AS(throw_shared_release, "throw_shared_release_first", THROWN, 2, "1",
+ .param = 0);
+ RUN_AS(throw_shared_release, "throw_shared_release_second", THROWN, 2, "0",
+ .param = 1);
+ RUN_AS(throw_refcount_shared_release, "throw_refcount_shared_release_first", THROWN, 1, "0",
+ .param = 0);
+ RUN_AS(throw_refcount_shared_release, "throw_refcount_shared_release_second", THROWN, 1, "0",
+ .param = 1);
+ /* A callee overwrites the caller's copy of the pointer: the object is released, once. */
+ THROW(throw_callee_clobbers_slot, 1, "0");
+ /* What is released through one call or on one path and not through another. */
+ RUN_AS(throw_irq_order, "throw_irq_order_enabled_first", THROWN, 2, "10s", .param = 1);
+ RUN_AS(throw_irq_order, "throw_irq_order_enabled_last", THROWN, 2, "01", .param = 0);
+ /* A recursive subprog holding an object per frame: each frame owns its own. */
+ THROW(throw_recursion_holding, 4, "3210");
+ /* what the frames hold at call depth two */
+ RUN_AS(throw_ringbuf_depth2_split, "throw_ringbuf_depth2_split_held", THROWN, 1, "0",
+ .param = 3);
+ RUN_AS(throw_ringbuf_depth2_split, "throw_ringbuf_depth2_split_free", THROWN, 1, "",
+ .param = 1);
+ /* a throw reached by paths of different stack depth */
+ RUN_AS(throw_slot_unallocated, "throw_slot_unallocated_held", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_slot_unallocated, "throw_slot_unallocated_free", THROWN, 0, "", .param = 1);
+ /* a reference that may be NULL when a callee throws */
+ RUN_AS(throw_task_maybe_null, "throw_task_maybe_null_null", THROWN, 1, "", .param = 1);
+ RUN_AS(throw_task_maybe_null, "throw_task_maybe_null_held", THROWN, 1, "0", .param = 3);
+ /* a reference held as a pointer into the object */
+ RUN_AS(throw_pop_null, "throw_pop_null_popped", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_pop_null, "throw_pop_null_empty", THROWN, 0, "", .param = 1);
+ /* A spill the verifier removes is not where the unwinder looks. */
+ RUN_AS(throw_fastcall_spill, "throw_fastcall_spill_throw", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_fastcall_spill, "throw_fastcall_spill_discard", 0, 1, "", .param = 1);
+ /* A reference a subprog returns is the caller's from the call on. */
+ THROW(throw_obj_returned, 1, "0");
+ /* One call that takes the object on one path and NULL on the other. */
+ RUN_AS(throw_xchg_maybe, "throw_xchg_maybe_kept", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_xchg_maybe, "throw_xchg_maybe_given", THROWN, 1, "", .param = 1);
+ /* A file and the reader of its dynptr, which goes first. */
+ snprintf(name, sizeof(name), "%sthrow_file_dynptr_held", prefix);
+ if (test__start_subtest(name))
+ run_file(skel, &(struct expect){ .acquired = 1, .released = "s0", .param = 0 });
+ snprintf(name, sizeof(name), "%sthrow_file_dynptr_released", prefix);
+ if (test__start_subtest(name))
+ run_file(skel, &(struct expect){ .acquired = 1, .released = "", .param = 1 });
+ /* A reference the kernel passed in a __ref argument, held at the throw or released before it. */
+ snprintf(name, sizeof(name), "%sthrow_ops_ref_held", prefix);
+ if (test__start_subtest(name))
+ run_struct_ops(skel, &(struct expect){ .acquired = 1, .released = "0", .param = 0 });
+ snprintf(name, sizeof(name), "%sthrow_ops_ref_released", prefix);
+ if (test__start_subtest(name))
+ run_struct_ops(skel, &(struct expect){ .acquired = 1, .released = "", .param = 1 });
+#undef THROW
+#undef RUN
+#undef RUN_AS
+out:
+ if (server >= 0)
+ close(server);
+ exceptions_cleanup__destroy(skel);
+}
+
+/* Serial, because it sets bpf_jit_harden. */
+void serial_test_exceptions_cleanup(void)
+{
+ const char *harden = "/proc/sys/net/core/bpf_jit_harden";
+ char old[16] = {};
+
+#if !defined(__x86_64__) && !defined(__aarch64__)
+ /* elsewhere a program may not throw while it holds resources */
+ test__skip();
+ return;
+#endif
+ run_suite("");
+ /* Constant blinding rewrites the program, with the insns that keep the slots in it. */
+ if (sysctl_set_or_fail(harden, old, "2"))
+ return;
+ run_suite("harden_");
+ sysctl_set_or_fail(harden, NULL, old);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 4d77b8fd4eab..466180400b15 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -58,6 +58,7 @@
#include "verifier_div_const.skel.h"
#include "verifier_div_mod_bounds.skel.h"
#include "verifier_div_overflow.skel.h"
+#include "verifier_frame_desc.skel.h"
#include "verifier_flow_keys.skel.h"
#include "verifier_global_subprogs.skel.h"
#include "verifier_global_ptr_args.skel.h"
@@ -267,6 +268,7 @@ void test_verifier_div0(void) { RUN(verifier_div0); }
void test_verifier_div_const(void) { RUN(verifier_div_const); }
void test_verifier_div_mod_bounds(void) { RUN(verifier_div_mod_bounds); }
void test_verifier_div_overflow(void) { RUN(verifier_div_overflow); }
+void test_verifier_frame_desc(void) { RUN(verifier_frame_desc); }
void test_verifier_flow_keys(void) { RUN(verifier_flow_keys); }
void test_verifier_global_subprogs(void) { RUN(verifier_global_subprogs); }
void test_verifier_global_ptr_args(void) { RUN(verifier_global_ptr_args); }
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
new file mode 100644
index 000000000000..af7c439c7e09
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -0,0 +1,1200 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include <bpf/bpf_endian.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+#include "../test_kmods/bpf_testmod.h"
+
+#define MAX_RES 8
+#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
+
+struct foo {
+ int data;
+ struct bpf_list_node node;
+};
+
+struct locked_foo {
+ int data;
+ struct bpf_spin_lock lock;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_RINGBUF);
+ __uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+private(A) struct bpf_spin_lock glock;
+
+/* Resources the program under test acquired, in order. */
+__u64 acquired[MAX_RES];
+__u32 acquired_cnt;
+/* The releases observed, in order: the handle of the resource, or none for a section. */
+__u64 released[MAX_RES];
+__u8 released_section[MAX_RES];
+__u32 released_cnt;
+/* The test process: the sections it leaves are the ones under test. */
+__u32 test_pid;
+/* A choice of the test, read by the program under test. */
+__u32 param;
+
+static __always_inline void mark_acquired(void *p)
+{
+ __u32 i = acquired_cnt;
+
+ if (i < MAX_RES) {
+ acquired[i] = (__u64)p;
+ acquired_cnt = i + 1;
+ }
+}
+
+/*
+ * Record a release: of @p, one of the resources the program acquired, or,
+ * as a @section, of what the program has no handle of: a section of the test
+ * process, or the reader of a file dynptr. The hooks fire for every unwind in
+ * the system, so releases of other programs are left out.
+ */
+static __always_inline void mark_released(void *p, bool section)
+{
+ __u32 i, n = released_cnt;
+
+ if (n >= MAX_RES)
+ return;
+ if (section) {
+ if (bpf_get_current_pid_tgid() >> 32 != test_pid)
+ return;
+ } else {
+ if (!p)
+ return;
+ for (i = 0; i < MAX_RES; i++)
+ if (i < acquired_cnt && acquired[i] == (__u64)p)
+ break;
+ if (i == MAX_RES)
+ return;
+ }
+ released[n] = section ? 0 : (__u64)p;
+ released_section[n] = section;
+ released_cnt = n + 1;
+}
+
+/*
+ * Observe every release the unwinder performs, identified like the program
+ * sees it: the handle is the pointer to an object or a record, the address of
+ * a stack object or of a lock; a section has none.
+ */
+SEC("fentry/bpf_unwind_release")
+int BPF_PROG(release_hook, struct bpf_resource *res, void *handle)
+{
+ mark_released(handle, res->kind == BPF_RESOURCE_RCU || res->kind == BPF_RESOURCE_PREEMPT ||
+ res->kind == BPF_RESOURCE_DYNPTR_FILE);
+ return 0;
+}
+
+/* An object in a callee-saved register of the throwing frame. */
+SEC("tc")
+int throw_obj_reg(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_obj_drop(f);
+ return 0;
+}
+
+/* Two objects, one spilled to the stack by the compiler. */
+SEC("tc")
+int throw_obj_two(struct __sk_buff *ctx)
+{
+ struct foo *f, *g;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ g = bpf_obj_new(typeof(*g));
+ if (!g) {
+ bpf_obj_drop(f);
+ return 1;
+ }
+ mark_acquired(f);
+ mark_acquired(g);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_obj_drop(f);
+ bpf_obj_drop(g);
+ return 0;
+}
+
+/* A ringbuf reservation. */
+SEC("tc")
+int throw_ringbuf(struct __sk_buff *ctx)
+{
+ void *p;
+
+ p = bpf_ringbuf_reserve(&ringbuf, 8, 0);
+ if (!p)
+ return 1;
+ mark_acquired(p);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_ringbuf_discard(p, 0);
+ return 0;
+}
+
+/* A task reference released through the destructor registry. */
+SEC("tc")
+int throw_task(struct __sk_buff *ctx)
+{
+ struct task_struct *t;
+
+ t = bpf_task_acquire(bpf_get_current_task_btf());
+ if (!t)
+ return 1;
+ mark_acquired(t);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_task_release(t);
+ return 0;
+}
+
+/*
+ * The task the kernel passes a struct_ops program in a __ref argument is the
+ * program's to release: the unwinder releases it when the program has not.
+ */
+SEC("struct_ops/test_refcounted")
+int BPF_PROG(throw_ops_ref, int dummy, struct task_struct *task)
+{
+ mark_acquired(task);
+ if (param)
+ bpf_task_release(task);
+ bpf_throw(0);
+ return 0;
+}
+
+SEC(".struct_ops.link")
+struct bpf_testmod_ops throwing_ops = {
+ .test_refcounted = (void *)throw_ops_ref,
+};
+
+/* Set by the test before the open of a file that the program acts on. */
+__u32 file_armed;
+
+/*
+ * The file is a referenced object, and the dynptr holds a reader of it that
+ * the program has no handle of: the unwinder frees the reader, then puts the
+ * file.
+ */
+SEC("lsm/file_open")
+int BPF_PROG(throw_file_dynptr)
+{
+ struct bpf_dynptr dynptr;
+ struct file *file;
+
+ if (!file_armed || bpf_get_current_pid_tgid() >> 32 != test_pid)
+ return 0;
+ file_armed = 0;
+ file = bpf_get_task_exe_file(bpf_get_current_task_btf());
+ if (!file)
+ return 0;
+ mark_acquired(file);
+ bpf_dynptr_from_file(file, 0, &dynptr);
+ if (param) {
+ bpf_dynptr_file_discard(&dynptr);
+ bpf_put_file(file);
+ }
+ bpf_throw(0);
+ return 0;
+}
+
+/* Port, in network order, of the listener throw_sock_cast looks up. */
+__u16 server_port;
+
+/*
+ * A socket acquired by a lookup and cast to its struct by a helper, which
+ * carries the reference under the struct's type: released as the socket.
+ */
+SEC("tc")
+int throw_sock_cast(struct __sk_buff *ctx)
+{
+ struct bpf_sock_tuple tuple = {};
+ struct bpf_sock *skc;
+ struct tcp_sock *tp;
+
+ tuple.ipv4.daddr = bpf_htonl(0x7f000001);
+ tuple.ipv4.dport = server_port;
+ skc = bpf_skc_lookup_tcp(ctx, &tuple, sizeof(tuple.ipv4), -1, 0);
+ if (!skc)
+ return 1;
+ tp = bpf_skc_to_tcp_sock(skc);
+ if (!tp) {
+ bpf_sk_release(skc);
+ return 1;
+ }
+ mark_acquired(tp);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_sk_release((struct bpf_sock *)tp);
+ return 0;
+}
+
+/* An open-coded iterator on the stack. */
+SEC("tc")
+int throw_iter(struct __sk_buff *ctx)
+{
+ struct bpf_iter_num it;
+ int *v;
+
+ bpf_iter_num_new(&it, 0, 10);
+ mark_acquired(&it);
+ while ((v = bpf_iter_num_next(&it))) {
+ if (*v == 3 && ctx->len)
+ bpf_throw(0xeB9F);
+ }
+ bpf_iter_num_destroy(&it);
+ return 0;
+}
+
+/* The object is NULL on one path and an object on the other at the throw. */
+SEC("tc")
+int throw_obj_maybe(struct __sk_buff *ctx)
+{
+ struct foo *f = NULL;
+
+ if (ctx->len > 1) {
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ }
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ if (f)
+ bpf_obj_drop(f);
+ return 0;
+}
+
+static __noinline int throw_in_subprog(struct __sk_buff *ctx, int x)
+{
+ /* Use callee-saved registers so the caller's are saved in this frame. */
+ __u64 a = x * 3, b = x * 5, c = x * 7;
+
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ return a + b + c;
+}
+
+/* The object is held by the caller of the throwing static subprog. */
+SEC("tc")
+int throw_obj_caller(struct __sk_buff *ctx)
+{
+ struct foo *f;
+ int ret;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ ret = throw_in_subprog(ctx, ctx->len);
+ bpf_obj_drop(f);
+ return ret;
+}
+
+/* The object is held by the caller across a call of the throwing subprog by its address. */
+SEC("tc")
+int throw_obj_callx_caller(struct __sk_buff *ctx)
+{
+ int (*callee)(struct __sk_buff *, int) = throw_in_subprog;
+ struct foo *f;
+ int ret;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ /* Hide the callee from the compiler, so that the call goes through the pointer: a callx. */
+ asm volatile ("" : "+r"(callee));
+ ret = callee(ctx, ctx->len);
+ bpf_obj_drop(f);
+ return ret;
+}
+
+/* Allocates at its first insns: the call that acquires is at the start of the subprog. */
+static __noinline int alloc_and_throw(void)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (f)
+ mark_acquired(f);
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/* A count the verifier does not know. */
+volatile __u32 loops = 1;
+
+/*
+ * The loop gets a may_goto of the verifier's, in front of the subprog that
+ * allocates at its first insns: what the main pass named by instruction has
+ * to be told from the subprog starts as they are after that.
+ */
+SEC("tc")
+int throw_obj_origin_after_guard(struct __sk_buff *ctx)
+{
+ __u32 i;
+
+ for (i = 0; i < loops; i++)
+ ;
+ return alloc_and_throw();
+}
+
+/* The throw is unconditional, so nothing after it keeps the object alive. */
+SEC("tc")
+int throw_obj_unconditional(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+static __noinline int throw_unconditionally(void)
+{
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/*
+ * The callee's throw is unconditional, so the caller never reads the object
+ * again after the call: the object is released from the slot of the caller's
+ * frame, which the acquisition filled.
+ */
+SEC("tc")
+int throw_obj_dead_caller(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ return throw_unconditionally();
+}
+
+static __noinline int subprog_acquire_throw(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/* The object is acquired and the throw raised in a static subprog. */
+SEC("tc")
+int throw_obj_subprog(struct __sk_buff *ctx)
+{
+ return subprog_acquire_throw(ctx) + 1;
+}
+
+__noinline int global_throw(struct __sk_buff *ctx)
+{
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/* The object is held by the caller of the throwing global subprog. */
+SEC("tc")
+int throw_obj_global_caller(struct __sk_buff *ctx)
+{
+ struct foo *f;
+ int ret;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ ret = global_throw(ctx);
+ bpf_obj_drop(f);
+ return ret;
+}
+
+/*
+ * R6 holds the ringbuf record on one path and a number on the other at the
+ * throw: the record is released from the hidden slot of the frame, which the
+ * acquisition fills and the release clears, whatever R6 holds.
+ */
+SEC("tc")
+__naked int throw_reg_conflict(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r7 = 0;"
+ "if r6 == 0 goto 1f;"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r6 = r0;"
+ "r7 = 1;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r7;"
+ "goto 2f;"
+ "1: r6 = 42;"
+ "2: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "if r7 == 0 goto 3f;"
+ "r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "3: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
+/* bpf_throw() leaves a spin lock and a section too. */
+SEC("tc")
+int throw_spin_lock(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ mark_acquired(&glock);
+ bpf_spin_lock(&glock);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_spin_unlock(&glock);
+ bpf_obj_drop(f);
+ return 0;
+}
+
+SEC("tc")
+int throw_rcu(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ bpf_rcu_read_lock();
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_rcu_read_unlock();
+ bpf_obj_drop(f);
+ return 0;
+}
+
+SEC("tc")
+int throw_irq(struct __sk_buff *ctx)
+{
+ unsigned long flags;
+
+ mark_acquired(&flags);
+ bpf_local_irq_save(&flags);
+ bpf_preempt_disable();
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_preempt_enable();
+ bpf_local_irq_restore(&flags);
+ return 0;
+}
+
+/*
+ * The record is read by nothing after its acquisition, and the register that
+ * held it is overwritten: the program keeps the pointer nowhere at the throw,
+ * and the record is released from the slot the acquisition filled.
+ */
+SEC("tc")
+__naked int throw_dead_reg(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/* Two elements, so that a lock of one is found through the pointer to it, not at a fixed address. */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __type(key, int);
+ __type(value, struct locked_foo);
+ __uint(max_entries, 2);
+} lock_map SEC(".maps");
+
+/*
+ * The lock of a map element is taken through the pointer in R7, which the
+ * path holding a record then moves to R9, leaving a number in R7: no
+ * register holds the pointer on both paths to the throw. The lock is
+ * released from the slot of the frame, which holds the address of the lock
+ * since the call that took it, and the record after the lock.
+ */
+SEC("tc")
+__naked int throw_lock_two_locations(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r1 = %[lock_map] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 3f;"
+ "r7 = r0;"
+ "r2 = r7;"
+ "r2 += %[lock_off];"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r8 = 0;"
+ "if r6 == 0 goto 1f;"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r8 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 8) = r0;"
+ "r2 = 2;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "1: r1 = r7;"
+ "r1 += %[lock_off];"
+ "call %[bpf_spin_lock];"
+ "r9 = 42;"
+ "if r8 == 0 goto 2f;"
+ "r9 = r7;"
+ "r7 = 42;"
+ "2: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "3: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(lock_map),
+ __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len)),
+ [lock_off]"i"(offsetof(struct locked_foo, lock))
+ : __clobber_all);
+}
+
+/*
+ * Two records, of which the one the test chooses is discarded at a single
+ * instruction. The paths discarding either reach the throw with the other
+ * record in its register and a dead pointer in the register of the discarded
+ * one: the frame has a slot for each record, and the discard clears the slot
+ * that holds the pointer it releases, whichever it is.
+ */
+SEC("tc")
+__naked int throw_shared_release(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r7 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r6;"
+ "*(u64 *)(r1 + 8) = r7;"
+ "r2 = 2;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r8 = %[param] ll;"
+ "r8 = *(u32 *)(r8 + 0);"
+ "r1 = r6;"
+ "if r8 == 0 goto +1;"
+ "r1 = r7;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+struct refcounted_foo {
+ int data;
+ struct bpf_list_node node;
+ struct bpf_refcount ref;
+};
+
+/*
+ * Two references to one object, its allocation and a bpf_refcount_acquire()
+ * of it, hold the same pointer, and the one the test chooses is dropped. Each
+ * has a slot of the frame, and the two hold equal handles: the drop clears
+ * one slot holding the pointer, whichever it is, and the unwinder drops the
+ * object exactly once more.
+ */
+SEC("tc")
+int throw_refcount_shared_release(struct __sk_buff *ctx)
+{
+ struct refcounted_foo *a, *b;
+
+ a = bpf_obj_new(typeof(*a));
+ if (!a)
+ return 1;
+ b = bpf_refcount_acquire(a);
+ mark_acquired(a);
+ bpf_obj_drop(param ? a : b);
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/* The node first, so that the pointer to it is the pointer to the object. */
+struct lfoo {
+ struct bpf_list_node node;
+ int data;
+};
+
+struct locked_list {
+ struct bpf_spin_lock lock;
+ struct bpf_list_head head __contains(lfoo, node);
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __type(key, int);
+ __type(value, struct locked_list);
+ __uint(max_entries, 1);
+} locked_list_map SEC(".maps");
+
+/* Pops a node under its caller's lock, and throws holding it. */
+static __noinline struct bpf_list_node *pop_and_throw(struct locked_list *l)
+{
+ struct bpf_list_node *n = bpf_list_pop_front(&l->head);
+
+ if (n)
+ mark_acquired(n);
+ bpf_throw(0xeB9F);
+ return n;
+}
+
+/*
+ * The callee's node was popped under the lock its caller holds. Locks are
+ * released before objects, whichever frame holds them: the unwinder unlocks
+ * the caller's lock, then drops the node.
+ */
+SEC("tc")
+int throw_obj_after_caller_lock(struct __sk_buff *ctx)
+{
+ struct bpf_list_node *n;
+ struct locked_list *l;
+ struct lfoo *f;
+ int key = 0;
+
+ l = bpf_map_lookup_elem(&locked_list_map, &key);
+ if (!l)
+ return 1;
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(&l->lock);
+ bpf_spin_lock(&l->lock);
+ bpf_list_push_back(&l->head, &f->node);
+ n = pop_and_throw(l);
+ if (n)
+ bpf_list_push_back(&l->head, n);
+ bpf_spin_unlock(&l->lock);
+ return 0;
+}
+
+/* Enters an RCU read section and throws in it. */
+static __noinline int rcu_and_throw(void)
+{
+ bpf_rcu_read_lock();
+ bpf_throw(0xeB9F);
+ bpf_rcu_read_unlock();
+ return 0;
+}
+
+/*
+ * The destructor of a kmem_cache iterator sleeps, and the callee throws in an
+ * RCU read section: sections are left before objects are released, whichever
+ * frame holds them, so the iterator is destroyed outside the section. Only a
+ * sleepable program may create it: a syscall program.
+ */
+SEC("syscall")
+int throw_sleeping_dtor_after_section(void *ctx)
+{
+ struct bpf_iter_kmem_cache it;
+
+ bpf_iter_kmem_cache_new(&it);
+ mark_acquired(&it);
+ rcu_and_throw();
+ bpf_iter_kmem_cache_destroy(&it);
+ return 0;
+}
+
+/*
+ * Two paths save the same two sets of IRQ flags in opposite orders and meet at
+ * the throw, kept apart by the RCU read section one of them holds: the
+ * unwinder restores the flags saved with interrupts disabled first and the
+ * ones saved with them enabled last, whichever path ran.
+ */
+SEC("tc")
+int throw_irq_order(struct __sk_buff *ctx)
+{
+ int order = param;
+ unsigned long a, b;
+
+ mark_acquired(&a);
+ mark_acquired(&b);
+ if (order) {
+ bpf_local_irq_save(&a);
+ bpf_local_irq_save(&b);
+ bpf_rcu_read_lock();
+ } else {
+ bpf_local_irq_save(&b);
+ bpf_local_irq_save(&a);
+ }
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ if (order) {
+ bpf_rcu_read_unlock();
+ bpf_local_irq_restore(&b);
+ bpf_local_irq_restore(&a);
+ } else {
+ bpf_local_irq_restore(&a);
+ bpf_local_irq_restore(&b);
+ }
+ return 0;
+}
+
+/*
+ * Allocates, recurses while holding, and throws in the innermost frame: every
+ * frame of the subprog holds an object obtained at the same instruction, each
+ * in the slot of its own frame.
+ */
+static __noinline int alloc_recurse_throw(int n)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 0;
+ mark_acquired(f);
+ if (n)
+ alloc_recurse_throw(n - 1);
+ else
+ bpf_throw(0xeB9F);
+ bpf_obj_drop(f);
+ return 0;
+}
+
+SEC("tc")
+int throw_recursion_holding(struct __sk_buff *ctx)
+{
+ return alloc_recurse_throw(3);
+}
+
+/* The object, in the stack of the frame that passes the struct to a callee. */
+struct slot_ctx {
+ struct foo *node;
+};
+
+/*
+ * Stashes the caller's object, overwrites the stack slot it came from with a
+ * number across the throw, and puts it back: the caller's stack holds no
+ * pointer to the object at the throw.
+ */
+static __noinline void clobber_slot_and_throw(struct slot_ctx *c)
+{
+ struct foo *f = c->node;
+
+ c->node = (void *)(long)param;
+ bpf_throw(0xeB9F);
+ c->node = f;
+}
+
+/*
+ * The caller keeps the object in the stack slot alone at the call, which the
+ * callee writes: the object is released from the hidden slot of the caller's
+ * frame, once, whatever the stack slot holds.
+ */
+SEC("tc")
+int throw_callee_clobbers_slot(struct __sk_buff *ctx)
+{
+ struct slot_ctx c;
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ c.node = f;
+ clobber_slot_and_throw(&c);
+ bpf_obj_drop(c.node);
+ return 0;
+}
+
+/* ---- what the frames hold at call depth two ---- */
+
+/*
+ * Reserves a record of its own and discards it before the throw when told,
+ * after it otherwise: its caller says so by a constant, and its frame holds
+ * the record at the throw through one call and not through the other.
+ */
+__used static __noinline int reserve_and_throw(const long *flag, int drop)
+{
+ void *rec;
+
+ rec = bpf_ringbuf_reserve(&ringbuf, 8, 0);
+ if (!rec)
+ return 1;
+ mark_acquired(rec);
+ if (drop)
+ bpf_ringbuf_discard(rec, 0);
+ if (*flag & 1)
+ bpf_throw(0xeB9F);
+ if (!drop)
+ bpf_ringbuf_discard(rec, 0);
+ return 0;
+}
+
+/* Two calls of reserve_and_throw() with a constant each, which clang would merge in C. */
+__naked static __noinline int call_twice(const long *flag)
+{
+ asm volatile (
+ "r6 = r1;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "r2 &= 2;"
+ "if r2 == 0 goto 1f;"
+ "r1 = r6;"
+ "r2 = 0;"
+ "call reserve_and_throw;"
+ "exit;"
+ "1: r1 = r6;"
+ "r2 = 1;"
+ "call reserve_and_throw;"
+ "exit;"
+ :
+ :
+ : __clobber_all);
+}
+
+/*
+ * The throw is at depth two, reached through either call of the frame below:
+ * the slot of the frame that throws holds the record or is empty.
+ */
+SEC("tc")
+int throw_ringbuf_depth2_split(struct __sk_buff *ctx)
+{
+ long flag = param;
+
+ return call_twice(&flag);
+}
+
+/* ---- a throw reached by paths of different stack depth ---- */
+
+/*
+ * A throw reached first by a path that never wrote fp-16, then by one that
+ * keeps the record only there: the record is in the hidden slot of the frame
+ * on the second path, and the slot is empty on the first.
+ */
+SEC("tc")
+__naked int throw_slot_unallocated(void)
+{
+ asm volatile (
+ "r6 = %[param] ll;"
+ "r6 = *(u32 *)(r6 + 0);"
+ "if r6 == 0 goto 1f;"
+ "2: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ "1: r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "*(u64 *)(r10 - 16) = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "goto 2b;"
+ "3: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/* ---- a reference that may be NULL when a callee throws ---- */
+
+static __noinline int throw_if_told(const long *flag)
+{
+ if (*flag & 1)
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/*
+ * A call that throws between the acquisition and the NULL check: a lookup
+ * that returned NULL acquired nothing, and left the slot of the frame empty.
+ */
+SEC("tc")
+int throw_task_maybe_null(struct __sk_buff *ctx)
+{
+ long flag = param;
+ struct task_struct *t;
+
+ t = bpf_task_from_pid(param & 2 ? test_pid : -1);
+ mark_acquired(t);
+ throw_if_told(&flag);
+ if (!t)
+ return 1;
+ bpf_task_release(t);
+ return 0;
+}
+
+/* ---- a reference held as a pointer into the object ---- */
+
+private(D) struct bpf_spin_lock pop_lock;
+private(D) struct bpf_list_head pop_head __contains(foo, node);
+
+/* Pushes a fresh object into the list the pop below takes it from. */
+__used static __noinline int push_one(void)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 0;
+ mark_acquired(f);
+ bpf_spin_lock(&pop_lock);
+ bpf_list_push_front(&pop_head, &f->node);
+ bpf_spin_unlock(&pop_lock);
+ return 0;
+}
+
+/*
+ * Pops a node, which points 8 bytes into its object, and keeps it in R6 on
+ * one path and a number on the other: the slot of the frame holds the object
+ * the node is in, which the throw releases. With param the list is empty and
+ * the pop returns NULL, which leaves the slot empty.
+ */
+SEC("tc")
+__naked int throw_pop_null(void)
+{
+ asm volatile (
+ "r1 = %[param] ll;"
+ "r1 = *(u32 *)(r1 + 0);"
+ "if r1 != 0 goto 1f;"
+ "call push_one;"
+ "1: r1 = %[pop_lock] ll;"
+ "call %[bpf_spin_lock];"
+ "r1 = %[pop_head] ll;"
+ "call %[bpf_list_pop_front];"
+ "r6 = r0;"
+ "r1 = %[pop_lock] ll;"
+ "call %[bpf_spin_unlock];"
+ "if r6 != 0 goto 2f;"
+ "r6 = 42;"
+ "2: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm_addr(pop_lock),
+ __imm_addr(pop_head),
+ __imm(bpf_spin_lock),
+ __imm(bpf_spin_unlock),
+ __imm(bpf_list_pop_front),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The record is spilled around a helper the JIT inlines, a spill the verifier
+ * removes again, and sits in R6 at the throw: the unwinder takes it from the
+ * slot its frame keeps it in, not from the program's stack.
+ */
+SEC("tc")
+__naked int throw_fastcall_spill(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r1 = r0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "call %[bpf_get_smp_processor_id];"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r6 = r1;"
+ "r2 = %[acquired] ll;"
+ "*(u64 *)(r2 + 0) = r6;"
+ "r2 = %[acquired_cnt] ll;"
+ "r3 = 1;"
+ "*(u32 *)(r2 + 0) = r3;"
+ "r2 = %[param] ll;"
+ "r2 = *(u32 *)(r2 + 0);"
+ "if r2 != 0 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_get_smp_processor_id),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/* ---- a reference a subprog returns ---- */
+
+static __noinline struct foo *alloc_foo(void)
+{
+ return bpf_obj_new(struct foo);
+}
+
+/*
+ * The subprog returns the object it allocated: the call acquires it for the
+ * caller, which holds it at the throw.
+ */
+SEC("tc")
+int throw_obj_returned(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = alloc_foo();
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_obj_drop(f);
+ return 0;
+}
+
+/* ---- a call that releases on some paths only ---- */
+
+struct foo_value {
+ struct foo __kptr *f;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct foo_value);
+} foo_map SEC(".maps");
+
+/*
+ * The object goes into the map when the test says so, and NULL otherwise: the
+ * slot of the object is cleared when the call took the object, and still
+ * holds it at the throw when the call took NULL.
+ */
+SEC("tc")
+int throw_xchg_maybe(struct __sk_buff *ctx)
+{
+ struct foo_value *v;
+ struct foo *f, *old;
+
+ v = bpf_map_lookup_elem(&foo_map, &(int){0});
+ if (!v)
+ return 1;
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ old = bpf_kptr_xchg(&v->f, param ? f : NULL);
+ if (old)
+ bpf_obj_drop(old);
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
new file mode 100644
index 000000000000..4a8280bc95cd
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -0,0 +1,200 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
+private(A) struct bpf_res_spin_lock rlock;
+
+/*
+ * A socket cast to its struct by a helper carries the reference of the
+ * socket a lookup acquired: the socket is in the slot the lookup filled at
+ * the throw, and the release through the cast pointer clears that slot.
+ */
+SEC("tc")
+__description("frame_desc: a throw with a socket cast to tcp_sock live")
+__arch_x86_64 __arch_arm64
+__success
+int sock_cast_at_throw(struct __sk_buff *skb)
+{
+ struct bpf_sock_tuple tuple = {};
+ struct bpf_sock *sk;
+ struct tcp_sock *tp;
+
+ sk = bpf_sk_lookup_tcp(skb, &tuple, sizeof(tuple.ipv4), -1, 0);
+ if (!sk)
+ return 0;
+ tp = bpf_skc_to_tcp_sock(sk);
+ if (!tp) {
+ bpf_sk_release(sk);
+ return 0;
+ }
+ if (skb->len > 100)
+ bpf_throw(0);
+ bpf_sk_release((struct bpf_sock *)tp);
+ return 1;
+}
+
+/* Nothing reads the socket after the lookup: it is in the slot of the frame at the throw. */
+SEC("tc")
+__description("frame_desc: a socket read by nothing but the throw")
+__arch_x86_64 __arch_arm64
+__success
+int sock_dead_at_throw(struct __sk_buff *skb)
+{
+ struct bpf_sock_tuple tuple = {};
+ struct bpf_sock *sk;
+
+ sk = bpf_sk_lookup_tcp(skb, &tuple, sizeof(tuple.ipv4), -1, 0);
+ if (!sk)
+ return 0;
+ bpf_throw(0);
+ return 1;
+}
+
+static __noinline int throw_unconditionally(void)
+{
+ bpf_throw(0);
+ return 0;
+}
+
+/*
+ * The callee's throw is unconditional, so the caller never reads the socket
+ * again after the call and keeps it nowhere in its stack: the socket is in
+ * the hidden slot of the caller's frame, which the lookup filled, and the
+ * statistics line reports the slot and the insns that keep it.
+ */
+SEC("tc")
+__description("frame_desc: a socket nothing reads in the caller of a throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 2 sites 0 classes 1")
+int sock_dead_in_caller(struct __sk_buff *skb)
+{
+ struct bpf_sock_tuple tuple = {};
+ struct bpf_sock *sk;
+
+ sk = bpf_sk_lookup_tcp(skb, &tuple, sizeof(tuple.ipv4), -1, 0);
+ if (!sk)
+ return 0;
+ return throw_unconditionally();
+}
+
+static __noinline struct bpf_sock *lookup_in_subprog(struct __sk_buff *skb)
+{
+ struct bpf_sock_tuple tuple = {};
+
+ return bpf_sk_lookup_tcp(skb, &tuple, sizeof(tuple.ipv4), -1, 0);
+}
+
+/*
+ * The socket was acquired by a subprog that has returned: the call acquired
+ * it for the caller, whose frame holds it at the throw in the slot of the
+ * call. Nothing is unwound while the subprog holds it, so the lookup in the
+ * subprog has no slot.
+ */
+SEC("tc")
+__description("frame_desc: a socket a returned subprog acquired, held at a throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 2 sites 0 classes 1")
+int sock_dead_from_subprog(struct __sk_buff *skb)
+{
+ struct bpf_sock *sk;
+
+ sk = lookup_in_subprog(skb);
+ if (!sk)
+ return 0;
+ return throw_unconditionally();
+}
+
+/*
+ * The throw is reached holding the lock, or without it when the lock failed:
+ * the address of the lock goes into the slot of the call in front of it, the
+ * slot is cleared again when the call did not return 0, and the unlock clears
+ * the slot. The statistics line reports the slot and the insns that keep it.
+ */
+SEC("tc")
+__description("frame_desc: a resilient lock that fails leaves its slot empty")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 7 sites 0 classes 1")
+int res_lock_fails_leaves_slot_empty(struct __sk_buff *skb)
+{
+ int ret;
+
+ ret = bpf_res_spin_lock(&rlock);
+ if (skb->len > 100)
+ bpf_throw(0);
+ if (!ret)
+ bpf_res_spin_unlock(&rlock);
+ return ret;
+}
+
+struct {
+ __uint(type, BPF_MAP_TYPE_RINGBUF);
+ __uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+/* Discards the record in R1. */
+__naked __noinline __used
+static long discard_record(void)
+{
+ asm volatile (
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_ringbuf_discard)
+ : __clobber_all);
+}
+
+/*
+ * A loop the verifier walks reserves a record at one insn on each of two
+ * iterations, the first kept in R7 and the second spilled: the call has one
+ * slot in the frame, which cannot hold both records.
+ */
+SEC("tc")
+__description("frame_desc: two records reserved at one insn are held at once")
+__arch_x86_64 __arch_arm64
+__failure __msg("a frame holds two resources of insn 6 at once")
+__naked int release_one_of_several(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r6 = 0;"
+ "1: r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 5f;"
+ "if r7 != 0 goto 2f;"
+ "r7 = r0;"
+ "goto 3f;"
+ "2: *(u64 *)(r10 - 8) = r0;"
+ "3: r6 += 1;"
+ "if r6 < 2 goto 1b;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 4f;"
+ "r1 = r7;"
+ "call discard_record;"
+ "4: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "5: if r7 == 0 goto 6f;"
+ "r1 = r7;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "6: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_get_prandom_u32),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index db1a7073aee3..85ca4e8b5983 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -2093,6 +2093,14 @@ static int bpf_dummy_reg(void *kdata, struct bpf_link *link)
ops->test_2(4, ops->data);
if (ops->test_trampoline_stack_args)
ops->test_trampoline_stack_args(1, 2, 3, 4, 5, 6, 7, 8, 9999);
+ /*
+ * The program releases the task, or the unwinder does when it throws:
+ * with bpf_task_release(), which puts what bpf_task_acquire() gets.
+ */
+ if (ops->test_refcounted) {
+ refcount_inc(¤t->rcu_users);
+ ops->test_refcounted(0, current);
+ }
return 0;
}
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 24/29] selftests/bpf: Test cleanup of handles on the stack
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (22 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 23/29] selftests/bpf: Test resource cleanup on bpf_throw() Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 25/29] selftests/bpf: Test cleanup of handles in registers Kumar Kartikeya Dwivedi
` (5 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add programs to exceptions_cleanup that keep their resources on their own
stack when they throw, where the unwinder reads them:
- a record spilled in the frame that throws, and in its caller;
- a popped list node, a pointer into its object, that is spilled, or
NULL;
- a record that is spilled on one path to the throw and in a register on
the other, which takes a hidden slot;
- a callee that leaves the RCU read section of its caller and throws;
- a section entered on only one path to the throw, which is counted.
Check in verifier_frame_desc how many hidden slots, inserted
instructions and site descriptors the verifier reports:
- none for a socket released before the throw;
- a site descriptor and no slot for a spilled record, in the frame that
throws and in its caller, for an iterator, and for an RCU read
section;
- the same for a record spilled in a slot that is not read again, where
two paths join before a call to a subprog that may throw;
- a slot for a record or a section that differs between the paths to
the throw.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 18 +-
.../selftests/bpf/progs/exceptions_cleanup.c | 202 +++++++++++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 240 ++++++++++++++++++
3 files changed, 459 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
index b4213036cea7..63909280e349 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -214,6 +214,19 @@ static void run_suite(const char *prefix)
/* One call that takes the object on one path and NULL on the other. */
RUN_AS(throw_xchg_maybe, "throw_xchg_maybe_kept", THROWN, 1, "0", .param = 0);
RUN_AS(throw_xchg_maybe, "throw_xchg_maybe_given", THROWN, 1, "", .param = 1);
+ /* What the frames keep in their own stack at the throw is released from there. */
+ THROW(throw_spilled, 1, "0");
+ RUN_AS(throw_spilled_caller, "throw_spilled_caller_throw", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_spilled_caller, "throw_spilled_caller_return", 0, 1, "", .param = 1);
+ RUN_AS(throw_spilled_node, "throw_spilled_node_popped", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_spilled_node, "throw_spilled_node_empty", THROWN, 0, "", .param = 1);
+ RUN_AS(throw_spilled_some_paths, "throw_spilled_some_paths_reg", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_spilled_some_paths, "throw_spilled_some_paths_stack", THROWN, 1, "0",
+ .param = 1);
+ /* The sections the frames are in, counted where the paths to a throw differ. */
+ THROW(throw_callee_left_rcu, 1, "0");
+ RUN_AS(throw_rcu_some_paths, "throw_rcu_some_paths_in", THROWN, 0, "s", .param = 1);
+ RUN_AS(throw_rcu_some_paths, "throw_rcu_some_paths_out", THROWN, 0, "", .param = 0);
/* A file and the reader of its dynptr, which goes first. */
snprintf(name, sizeof(name), "%sthrow_file_dynptr_held", prefix);
if (test__start_subtest(name))
@@ -249,7 +262,10 @@ void serial_test_exceptions_cleanup(void)
return;
#endif
run_suite("");
- /* Constant blinding rewrites the program, with the insns that keep the slots in it. */
+ /*
+ * Constant blinding rewrites the program, with the insns that keep the
+ * slots in it, and moves the calls the sites are at.
+ */
if (sysctl_set_or_fail(harden, old, "2"))
return;
run_suite("harden_");
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
index af7c439c7e09..addcc509c6c2 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -1197,4 +1197,206 @@ int throw_xchg_maybe(struct __sk_buff *ctx)
return 0;
}
+/* ---- what the frames keep in their own stack at the throw ---- */
+
+/*
+ * The record is spilled when the throw is reached, on every path to it: the
+ * unwinder takes it from the slot of the program's stack that the descriptor
+ * of the throw names, and the frame keeps no slot for it.
+ */
+SEC("tc")
+__naked int throw_spilled(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/* Throws when the test says so. */
+__used static __noinline int throw_if_param(void)
+{
+ if (!param)
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/*
+ * The record is spilled in the caller of the subprog that throws: the
+ * unwinder takes it from the caller's stack, where the descriptor of the
+ * call says it is. When the callee returns the caller discards it.
+ */
+SEC("tc")
+__naked int throw_spilled_caller(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "call throw_if_param;"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard)
+ : __clobber_all);
+}
+
+/*
+ * The node that was popped is spilled, a pointer 8 bytes into its object or
+ * NULL when the list was empty: the unwinder releases the object the spilled
+ * pointer is in, and nothing when the slot holds NULL.
+ */
+SEC("tc")
+__naked int throw_spilled_node(void)
+{
+ asm volatile (
+ "r1 = %[param] ll;"
+ "r1 = *(u32 *)(r1 + 0);"
+ "if r1 != 0 goto 1f;"
+ "call push_one;"
+ "1: r1 = %[pop_lock] ll;"
+ "call %[bpf_spin_lock];"
+ "r1 = %[pop_head] ll;"
+ "call %[bpf_list_pop_front];"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = %[pop_lock] ll;"
+ "call %[bpf_spin_unlock];"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm_addr(pop_lock),
+ __imm_addr(pop_head),
+ __imm(bpf_spin_lock),
+ __imm(bpf_spin_unlock),
+ __imm(bpf_list_pop_front),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The record is spilled on one path to the throw and in R6 on the other, and
+ * in no other register: the paths do not agree on where it is, so the frame
+ * keeps it in a slot, and it is released once on either path.
+ */
+SEC("tc")
+__naked int throw_spilled_some_paths(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "r1 = %[param] ll;"
+ "r1 = *(u32 *)(r1 + 0);"
+ "if r1 == 0 goto 1f;"
+ "*(u64 *)(r10 - 8) = r6;"
+ "r6 = 0;"
+ "1: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/* Leaves the RCU read section of its caller and throws. */
+static __noinline int rcu_unlock_and_throw(void)
+{
+ bpf_rcu_read_unlock();
+ bpf_throw(0xeB9F);
+ return 0;
+}
+
+/*
+ * The callee leaves the section its caller entered and throws: the caller is
+ * in one section at the call, the callee in one less than it was called in,
+ * and no section is left to leave.
+ */
+SEC("tc")
+int throw_callee_left_rcu(struct __sk_buff *ctx)
+{
+ struct foo *f;
+
+ f = bpf_obj_new(typeof(*f));
+ if (!f)
+ return 1;
+ mark_acquired(f);
+ bpf_rcu_read_lock();
+ rcu_unlock_and_throw();
+ bpf_obj_drop(f);
+ return 0;
+}
+
+/*
+ * The section is entered on one path to the throw only: the frame counts its
+ * sections in a slot, and the unwinder leaves as many as it counted.
+ */
+SEC("tc")
+__naked int throw_rcu_some_paths(void)
+{
+ asm volatile (
+ "r6 = %[param] ll;"
+ "r6 = *(u32 *)(r6 + 0);"
+ "if r6 == 0 goto 1f;"
+ "call %[bpf_rcu_read_lock];"
+ "1: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
index 4a8280bc95cd..00a6dce52dfb 100644
--- a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -137,6 +137,246 @@ struct {
__uint(max_entries, 4096);
} ringbuf SEC(".maps");
+/*
+ * The socket is released before the throw: no frame is unwound while it
+ * holds the socket, so nothing keeps track of it.
+ */
+SEC("tc")
+__description("frame_desc: a socket released before the throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 0 classes 1")
+int sock_released_before_throw(struct __sk_buff *skb)
+{
+ struct bpf_sock_tuple tuple = {};
+ struct bpf_sock *sk;
+
+ sk = bpf_sk_lookup_tcp(skb, &tuple, sizeof(tuple.ipv4), -1, 0);
+ if (!sk)
+ return 0;
+ bpf_sk_release(sk);
+ bpf_throw(0);
+ return 1;
+}
+
+/*
+ * The record is spilled on every path to the throw: the descriptor of the
+ * throw names the slot of the program's stack, and the frame has no hidden
+ * slot.
+ */
+SEC("tc")
+__description("frame_desc: a record spilled at the throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int record_spilled_at_throw(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The record is spilled in the caller of the subprog that throws: the
+ * descriptor is that of the call, the site the caller is unwound at.
+ */
+SEC("tc")
+__description("frame_desc: a record spilled in the caller of a throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int record_spilled_in_caller(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "call throw_unconditionally;"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve)
+ : __clobber_all);
+}
+
+/*
+ * The record is spilled on one path to the throw and in R6 on the other, and
+ * in no other register: the visits of the throw do not agree on where it is,
+ * so it is kept in a slot.
+ */
+SEC("tc")
+__description("frame_desc: a record spilled on one path to the throw keeps its slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 2 sites 0 classes 1")
+__naked int record_spilled_on_one_path(void)
+{
+ asm volatile (
+ "r7 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r0 = 0;"
+ "if r7 == 0 goto 1f;"
+ "*(u64 *)(r10 - 8) = r6;"
+ "r6 = 0;"
+ "1: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
+/* May throw, and returns when it does not. */
+__used static __noinline int throw_maybe(void)
+{
+ if (bpf_get_prandom_u32() & 1)
+ bpf_throw(0);
+ return 0;
+}
+
+/*
+ * The record is spilled in a slot nothing reads again, and two paths join
+ * between the spill and the call of a subprog that may throw. The verifier
+ * forgets what dead slots hold where paths join, so the call reads every slot
+ * of the stack: the spill is still known at the call, and its descriptor
+ * names the slot.
+ */
+SEC("tc")
+__description("frame_desc: a spill nothing reads again is found at a call that may throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5) __flag(BPF_F_TEST_STATE_FREQ)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int dead_spill_at_call(void)
+{
+ asm volatile (
+ "r8 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "*(u64 *)(r10 - 8) = r6;"
+ "if r8 == 0 goto 1f;"
+ "r7 = 0;"
+ "1: call throw_maybe;"
+ "r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
+/* An iterator is on the stack wherever it is held: the descriptor of the throw names it. */
+SEC("tc")
+__description("frame_desc: an iterator at the throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+int iter_at_throw(struct __sk_buff *skb)
+{
+ struct bpf_iter_num it;
+ int *v;
+
+ bpf_iter_num_new(&it, 0, 10);
+ while ((v = bpf_iter_num_next(&it))) {
+ if (*v == 3 && skb->len > 100)
+ bpf_throw(0);
+ }
+ bpf_iter_num_destroy(&it);
+ return 0;
+}
+
+/*
+ * BTF FUNC records are not generated for kfuncs that only inline assembly
+ * refers to, and libbpf needs them to link the programs.
+ */
+void __kfunc_btf_root(void)
+{
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+}
+
+/*
+ * Every path to the throw is in one RCU read section: the descriptor of the
+ * throw says so, and the frame does not count its sections.
+ */
+SEC("tc")
+__description("frame_desc: a section at the throw has no count")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int rcu_at_throw(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * One path to the throw is in an RCU read section and the other is not: the
+ * frame counts its sections in a slot, which starts at zero and which the
+ * lock adds one to.
+ */
+SEC("tc")
+__description("frame_desc: a section on one path to the throw is counted")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 4 sites 0 classes 1")
+__naked int rcu_on_one_path(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "if r6 == 0 goto 1f;"
+ "call %[bpf_rcu_read_lock];"
+ "1: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm(bpf_rcu_read_lock),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
/* Discards the record in R1. */
__naked __noinline __used
static long discard_record(void)
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 25/29] selftests/bpf: Test cleanup of handles in registers
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (23 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 24/29] selftests/bpf: Test cleanup of handles on the stack Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 26/29] selftests/bpf: Test callees releasing caller resources Kumar Kartikeya Dwivedi
` (4 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add programs to exceptions_cleanup that keep their resources only in
registers when they throw:
- three records, two frames up from the throw: one in a register that
the callee saves and overwrites, one in a register that no frame
touches, and one of the callee's in a register that the throwing
frame does not save;
- a record in R3 in the frame that throws;
- a popped list node, a pointer into its object, in R6, or NULL.
Check in verifier_frame_desc that the verifier reports a site descriptor
and no slot for a record in R6 or R0 at the throw, and for a record in R6
in the caller of a subprog that may throw. A record in R2 there takes a
slot.
Also check that a register that is not read again is still found at the
throw when two paths join before it.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 5 +
.../selftests/bpf/progs/exceptions_cleanup.c | 164 ++++++++++++++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 155 +++++++++++++++++
3 files changed, 324 insertions(+)
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
index 63909280e349..a88cbdafdf77 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -227,6 +227,11 @@ static void run_suite(const char *prefix)
THROW(throw_callee_left_rcu, 1, "0");
RUN_AS(throw_rcu_some_paths, "throw_rcu_some_paths_in", THROWN, 0, "s", .param = 1);
RUN_AS(throw_rcu_some_paths, "throw_rcu_some_paths_out", THROWN, 0, "", .param = 0);
+ /* What the frames keep in registers at the throw is released from them. */
+ THROW(throw_regs_callers, 3, "201");
+ THROW(throw_reg_caller_saved, 1, "0");
+ RUN_AS(throw_reg_node, "throw_reg_node_popped", THROWN, 1, "0", .param = 0);
+ RUN_AS(throw_reg_node, "throw_reg_node_empty", THROWN, 0, "", .param = 1);
/* A file and the reader of its dynptr, which goes first. */
snprintf(name, sizeof(name), "%sthrow_file_dynptr_held", prefix);
if (test__start_subtest(name))
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
index addcc509c6c2..10397d157a48 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -1399,4 +1399,168 @@ __naked int throw_rcu_some_paths(void)
: __clobber_all);
}
+/* ---- what the frames keep in registers at the throw ---- */
+
+/* Throws, and uses no register but R1. */
+__used __naked static __noinline int throw_leaf(void)
+{
+ asm volatile (
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * Holds a record of its own in R7 and overwrites R6, which its prologue has
+ * to save for its caller then, before it calls the subprog that throws.
+ */
+__used __naked static __noinline int throw_mid(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r7 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 16) = r0;"
+ "r2 = 3;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r6 = 42;"
+ "call throw_leaf;"
+ "r1 = r7;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard)
+ : __clobber_all);
+}
+
+/*
+ * Three records in registers only, two frames up from the throw. The main
+ * program has one in R6, which its callee saves and overwrites, and one in
+ * R8, which no frame touches. The callee has one in R7, which the frame that
+ * throws does not save. The unwinder has the registers of the throw, and
+ * those of each caller from where the frame it called saved them: the frame
+ * that throws holds nothing, its caller's record goes first, then the two of
+ * the main program in the order it acquired them.
+ */
+SEC("tc")
+__naked int throw_regs_callers(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r8 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r6;"
+ "*(u64 *)(r1 + 8) = r8;"
+ "r2 = 2;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "call throw_mid;"
+ "r1 = r8;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "1: r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard)
+ : __clobber_all);
+}
+
+/*
+ * The record is in R3 only when the frame throws, a register no call
+ * preserves: the unwinder has every register of the frame that throws.
+ */
+SEC("tc")
+__naked int throw_reg_caller_saved(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r3 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The node that was popped is in R6 at the throw, a pointer 8 bytes into its
+ * object or NULL when the list was empty: the unwinder releases the object
+ * the register points into, and nothing when it holds NULL.
+ */
+SEC("tc")
+__naked int throw_reg_node(void)
+{
+ asm volatile (
+ "r1 = %[param] ll;"
+ "r1 = *(u32 *)(r1 + 0);"
+ "if r1 != 0 goto 1f;"
+ "call push_one;"
+ "1: r1 = %[pop_lock] ll;"
+ "call %[bpf_spin_lock];"
+ "r1 = %[pop_head] ll;"
+ "call %[bpf_list_pop_front];"
+ "r6 = r0;"
+ "r1 = %[pop_lock] ll;"
+ "call %[bpf_spin_unlock];"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm_addr(pop_lock),
+ __imm_addr(pop_head),
+ __imm(bpf_spin_lock),
+ __imm(bpf_spin_unlock),
+ __imm(bpf_list_pop_front),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
index 00a6dce52dfb..5f2425194b1a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -299,6 +299,161 @@ __naked int dead_spill_at_call(void)
: __clobber_all);
}
+/*
+ * The record is in R6 when the frame throws: the descriptor of the throw
+ * names the register.
+ */
+SEC("tc")
+__description("frame_desc: a record in R6 at the throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int record_in_r6_at_throw(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r6 = r0;"
+ "r0 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The record is in R0 only, where the call that reserved it left it: the
+ * unwinder has every register of the frame that throws.
+ */
+SEC("tc")
+__description("frame_desc: a record in R0 at the throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int record_in_r0_at_throw(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The record is in R6 in the caller of the subprog that may throw: R6 is
+ * callee-saved, so the unwinder has it for a frame in a call, and the
+ * descriptor of the call names it.
+ */
+SEC("tc")
+__description("frame_desc: a record in R6 in the caller of a throw has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int record_in_r6_in_caller(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r6 = r0;"
+ "call throw_maybe;"
+ "r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard)
+ : __clobber_all);
+}
+
+/*
+ * The record is in R2 only in the caller of the subprog that throws, a
+ * register the call does not preserve: the unwinder has it for the frame that
+ * throws only, so the record is kept in a slot.
+ */
+SEC("tc")
+__description("frame_desc: a record in R2 in the caller of a throw keeps its slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 2 sites 0 classes 1")
+__naked int record_in_r2_in_caller(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r2 = r0;"
+ "r0 = 0;"
+ "call throw_unconditionally;"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve)
+ : __clobber_all);
+}
+
+/*
+ * The record is in R6, which nothing reads again, and two paths join in front
+ * of the throw. The verifier forgets what dead registers hold where paths
+ * join, so the throw reads the registers: the record is still known to be in
+ * R6 at the throw, and its descriptor names the register.
+ */
+SEC("tc")
+__description("frame_desc: a register nothing reads again is found at the throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5) __flag(BPF_F_TEST_STATE_FREQ)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int dead_reg_at_throw(void)
+{
+ asm volatile (
+ "r8 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r0 = 0;"
+ "if r8 == 0 goto 1f;"
+ "r7 = 0;"
+ "1: r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
/* An iterator is on the stack wherever it is held: the descriptor of the throw names it. */
SEC("tc")
__description("frame_desc: an iterator at the throw has no slot")
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 26/29] selftests/bpf: Test callees releasing caller resources
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (24 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 25/29] selftests/bpf: Test cleanup of handles in registers Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 27/29] selftests/bpf: Test resources a callee leaves with its caller Kumar Kartikeya Dwivedi
` (3 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add programs to exceptions_cleanup where a callee releases a resource of
its caller:
- the callee discards a record that the caller keeps in R6, and the
caller throws before or after the call;
- the caller passes the record without keeping a copy, and the callee
throws before or after its discard, with the record in R6;
- the same, with the record in a hidden slot of the caller, which the
call clears;
- the callee has the record in no register or stack slot when it
throws, and keeps it in a hidden slot of its own;
- the callee unlocks the lock of a map element that the caller keeps in
a hidden slot, which the caller clears before the call.
Check in verifier_frame_desc how many hidden slots, inserted instructions
and site descriptors each case takes. Also check that a lock at a fixed
address that a callee unlocks takes a site descriptor and no slot.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 30 ++
.../selftests/bpf/progs/exceptions_cleanup.c | 281 ++++++++++++++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 253 ++++++++++++++++
3 files changed, 564 insertions(+)
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
index a88cbdafdf77..55624ba4d992 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -232,6 +232,36 @@ static void run_suite(const char *prefix)
THROW(throw_reg_caller_saved, 1, "0");
RUN_AS(throw_reg_node, "throw_reg_node_popped", THROWN, 1, "0", .param = 0);
RUN_AS(throw_reg_node, "throw_reg_node_empty", THROWN, 0, "", .param = 1);
+ /* A callee releases what its caller holds: a throw finds it held, or released. */
+ RUN_AS(throw_callee_releases, "throw_callee_releases_before", THROWN, 1, "0", .param = 1);
+ RUN_AS(throw_callee_releases, "throw_callee_releases_after", THROWN, 1, "", .param = 2);
+ RUN_AS(throw_callee_releases, "throw_callee_releases_return", 0, 1, "", .param = 0);
+ /* A callee that is handed what it releases holds it from the call on. */
+ RUN_AS(throw_callee_consumes, "throw_callee_consumes_before", THROWN, 1, "0", .param = 1);
+ RUN_AS(throw_callee_consumes, "throw_callee_consumes_after", THROWN, 1, "", .param = 2);
+ RUN_AS(throw_callee_consumes, "throw_callee_consumes_return", 0, 1, "", .param = 0);
+ /* the caller keeps it in a slot, which the call empties */
+ RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_caller_r6", THROWN,
+ 1, "0", .param = 3);
+ RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_caller_r9", THROWN,
+ 1, "0", .param = 11);
+ RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_callee_r6", THROWN,
+ 1, "0", .param = 1);
+ RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_callee_r9", THROWN,
+ 1, "0", .param = 9);
+ RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_after", THROWN,
+ 1, "", .param = 10);
+ RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_return", 0,
+ 1, "", .param = 8);
+ /* the callee has it nowhere at its throw and keeps it in a slot of its own */
+ RUN_AS(throw_callee_consumes_slot, "throw_callee_consumes_slot_throw", THROWN, 1, "0",
+ .param = 1);
+ RUN_AS(throw_callee_consumes_slot, "throw_callee_consumes_slot_return", 0, 1, "",
+ .param = 0);
+ /* A lock a callee unlocks leaves the slot of the caller in front of the call. */
+ RUN_AS(throw_callee_unlocks, "throw_callee_unlocks_before", THROWN, 1, "0", .param = 1);
+ RUN_AS(throw_callee_unlocks, "throw_callee_unlocks_after", THROWN, 1, "", .param = 2);
+ RUN_AS(throw_callee_unlocks, "throw_callee_unlocks_return", 0, 1, "", .param = 0);
/* A file and the reader of its dynptr, which goes first. */
snprintf(name, sizeof(name), "%sthrow_file_dynptr_held", prefix);
if (test__start_subtest(name))
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
index 10397d157a48..6e16caee66da 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -1563,4 +1563,285 @@ __naked int throw_reg_node(void)
: __clobber_all);
}
+/* ---- a callee that releases what its caller holds ---- */
+
+/* Discards the record of its caller. */
+__used static __noinline int discard_in_callee(void *rec)
+{
+ bpf_ringbuf_discard(rec, 0);
+ return 0;
+}
+
+/*
+ * The callee discards the record its caller passes it. A throw in front of
+ * the call finds the caller holding the record, in R6, and one after the call
+ * finds it holding nothing: no slot keeps the record, so the call that hands
+ * it to the callee has none to empty.
+ */
+SEC("tc")
+__naked int throw_callee_releases(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r6 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "r7 = %[param] ll;"
+ "r7 = *(u32 *)(r7 + 0);"
+ "if r7 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "call discard_in_callee;"
+ "if r7 != 2 goto 2f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "3: exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * Keeps the record its caller hands it in R6, discards it, and throws in
+ * front of the discard or after it when the low bits of the test's choice
+ * say so.
+ */
+__used __naked static __noinline int consume_in_callee(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "r7 = %[param] ll;"
+ "r7 = *(u32 *)(r7 + 0);"
+ "r7 &= 7;"
+ "if r7 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "if r7 != 2 goto 2f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The caller hands the record to the callee that discards it, and keeps no
+ * copy. The record is the callee's from the call on: a throw of the callee in
+ * front of the discard finds it in R6 of the callee, and one after the
+ * discard finds nothing.
+ */
+SEC("tc")
+__naked int throw_callee_consumes(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r1 = r0;"
+ "r0 = 0;"
+ "call consume_in_callee;"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve)
+ : __clobber_all);
+}
+
+/*
+ * The record is in R6 on one path to the throw of the caller and in R9 on the
+ * other, so the caller keeps it in a slot. The call that hands the record to
+ * the callee empties the slot: a throw of the callee in front of its discard
+ * releases the record once, from the callee, and one after it releases
+ * nothing.
+ */
+SEC("tc")
+__naked int throw_callee_consumes_from_slot(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 4f;"
+ "r6 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "r9 = 0;"
+ "r8 = %[param] ll;"
+ "r8 = *(u32 *)(r8 + 0);"
+ "r7 = r8;"
+ "r7 &= 8;"
+ "if r7 == 0 goto 1f;"
+ "r9 = r6;"
+ "r6 = 0;"
+ "1: r8 &= 7;"
+ "if r8 != 3 goto 2f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r1 = r6;"
+ "if r1 != 0 goto 3f;"
+ "r1 = r9;"
+ "3: call consume_in_callee;"
+ "4: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * Discards the record its caller hands it. When it throws instead the record
+ * is in no register: the cookie has taken its place in R1.
+ */
+__used __naked static __noinline int consume_or_lose(void)
+{
+ asm volatile (
+ "r2 = %[param] ll;"
+ "r2 = *(u32 *)(r2 + 0);"
+ "if r2 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(param),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The callee that is handed the record has it nowhere when it throws, so it
+ * keeps it in a slot of its own, which its entry fills from the argument.
+ */
+SEC("tc")
+__naked int throw_callee_consumes_slot(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r1 = r0;"
+ "r0 = 0;"
+ "call consume_or_lose;"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_ringbuf_reserve)
+ : __clobber_all);
+}
+
+/* Unlocks the lock of the element its caller locked. */
+__used static __noinline int unlock_in_callee(struct locked_foo *foo)
+{
+ bpf_spin_unlock(&foo->lock);
+ return 0;
+}
+
+/*
+ * The lock of a map element is held at the first throw, where the frame has
+ * the pointer to the element nowhere: the lock is in a slot of the caller,
+ * which the callee that unlocks it cannot reach. The caller empties the slot
+ * in front of the call, and the throw after the call finds nothing held.
+ */
+SEC("tc")
+__naked int throw_callee_unlocks(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r1 = %[lock_map] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r2 = r6;"
+ "r2 += %[lock_off];"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r1 = r6;"
+ "r1 += %[lock_off];"
+ "call %[bpf_spin_lock];"
+ "r7 = %[param] ll;"
+ "r7 = *(u32 *)(r7 + 0);"
+ "if r7 != 1 goto 1f;"
+ "r6 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "call unlock_in_callee;"
+ "if r7 != 2 goto 2f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(lock_map),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [lock_off]"i"(offsetof(struct locked_foo, lock))
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
index 5f2425194b1a..e98d1caf27f3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -546,6 +546,259 @@ static long discard_record(void)
: __clobber_all);
}
+/*
+ * The callee discards the record its caller passes it and keeps in R6. The
+ * throw in front of the call finds the record in the register and the one
+ * after it finds nothing held: the record has no slot, so the call that hands
+ * it to the callee has none to empty.
+ */
+SEC("tc")
+__description("frame_desc: a callee discards what its caller holds without a slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int callee_releases_no_slot(void)
+{
+ asm volatile (
+ "r7 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r6 = r0;"
+ "r0 = 0;"
+ "if r7 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "call discard_record;"
+ "if r7 != 2 goto 2f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "3: exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
+/* A choice the callees below read without a call, which would clobber their argument. */
+__u32 throw_flag;
+
+/*
+ * Keeps the record its caller hands it in R6 and discards it, and may throw
+ * in front of the discard and after it.
+ */
+__used __naked static __noinline int consume_record(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "r7 = %[throw_flag] ll;"
+ "r7 = *(u32 *)(r7 + 0);"
+ "if r7 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "if r7 != 2 goto 2f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(throw_flag),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The caller hands the record to a callee that discards it and keeps no copy
+ * of it. The record is the callee's from the call on: the throw in front of
+ * the discard finds it in R6 of the callee, the one after it finds nothing,
+ * and no frame keeps a slot.
+ */
+SEC("tc")
+__description("frame_desc: a callee that is handed a record holds it in a register")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int callee_consumes_record(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r1 = r0;"
+ "r0 = 0;"
+ "call consume_record;"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve)
+ : __clobber_all);
+}
+
+/* Discards the record its caller hands it, or throws with the cookie in its place in R1. */
+__used __naked static __noinline int consume_or_lose_record(void)
+{
+ asm volatile (
+ "r2 = %[throw_flag] ll;"
+ "r2 = *(u32 *)(r2 + 0);"
+ "if r2 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(throw_flag),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
+/*
+ * The callee has the record it is handed nowhere when it throws, so it keeps
+ * it in a slot of its own: its entry stores the argument there, and the
+ * discard clears the slot.
+ */
+SEC("tc")
+__description("frame_desc: a callee that is handed a record keeps it in a slot of its own")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 4 sites 0 classes 1")
+__naked int callee_consumes_record_slot(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 1f;"
+ "r1 = r0;"
+ "r0 = 0;"
+ "call consume_or_lose_record;"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve)
+ : __clobber_all);
+}
+
+private(B) struct bpf_spin_lock slock;
+
+/* Unlocks the lock its caller took. */
+__used static __noinline int unlock_slock(void)
+{
+ bpf_spin_unlock(&slock);
+ return 0;
+}
+
+/*
+ * The lock of a global variable is at a fixed address, which the descriptor
+ * of the throw gives: it has no slot, so the callee that unlocks it has none
+ * to empty.
+ */
+SEC("tc")
+__description("frame_desc: a callee unlocks the lock its caller holds at a throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1 classes 1")
+__naked int callee_unlocks(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[slock] ll;"
+ "call %[bpf_spin_lock];"
+ "if r6 != 1 goto 1f;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: call unlock_slock;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(slock),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
+struct lock_value {
+ int data;
+ struct bpf_spin_lock lock;
+};
+
+/* Two elements, so that a lock of one is found through the pointer to it, not at a fixed address. */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __type(key, int);
+ __type(value, struct lock_value);
+ __uint(max_entries, 2);
+} lock_values SEC(".maps");
+
+/* Unlocks the lock of the value its caller locked. */
+__used static __noinline int unlock_value(struct lock_value *val)
+{
+ bpf_spin_unlock(&val->lock);
+ return 0;
+}
+
+/*
+ * The frame has the pointer to the value nowhere at the throw, so the lock is
+ * in a slot of the caller: stored in front of the lock, empty at the entry,
+ * and emptied in front of the call of the callee that unlocks, which cannot
+ * reach it.
+ */
+SEC("tc")
+__description("frame_desc: the caller empties the slot of a lock its callee unlocks")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 3 sites 0 classes 1")
+__naked int callee_unlocks_slot(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r1 = %[lock_values] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 2f;"
+ "r7 = r0;"
+ "r1 = r7;"
+ "r1 += %[lock_off];"
+ "call %[bpf_spin_lock];"
+ "if r6 != 1 goto 1f;"
+ "r7 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r7;"
+ "call unlock_value;"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(lock_values),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len)),
+ [lock_off]"i"(offsetof(struct lock_value, lock))
+ : __clobber_all);
+}
+
/*
* A loop the verifier walks reserves a record at one insn on each of two
* iterations, the first kept in R7 and the second spilled: the call has one
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 27/29] selftests/bpf: Test resources a callee leaves with its caller
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (25 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 26/29] selftests/bpf: Test callees releasing caller resources Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 28/29] selftests/bpf: Test cleanup of locks described at sites Kumar Kartikeya Dwivedi
` (2 subsequent siblings)
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add programs to exceptions_cleanup where the caller throws while holding
a resource that a callee acquired for it: an iterator that the callee
created on the caller's stack, and an RCU read section that the callee
returned in.
Check in verifier_frame_desc that nothing is kept for resources that no
throw finds held: a lock that one subprog takes and another drops, a
section that a subprog returns in, and two records that one instruction
reserves in a loop. An iterator that a callee creates on its caller's
stack takes a site descriptor and no slot.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 3 +
.../selftests/bpf/progs/exceptions_cleanup.c | 43 ++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 133 +++++++++++++++++-
3 files changed, 177 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
index 55624ba4d992..9ba2cd9c8097 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -262,6 +262,9 @@ static void run_suite(const char *prefix)
RUN_AS(throw_callee_unlocks, "throw_callee_unlocks_before", THROWN, 1, "0", .param = 1);
RUN_AS(throw_callee_unlocks, "throw_callee_unlocks_after", THROWN, 1, "", .param = 2);
RUN_AS(throw_callee_unlocks, "throw_callee_unlocks_return", 0, 1, "", .param = 0);
+ /* What a frame holds that a callee acquired for it. */
+ THROW(throw_iter_from_callee, 1, "0");
+ THROW(throw_rcu_from_callee, 0, "s");
/* A file and the reader of its dynptr, which goes first. */
snprintf(name, sizeof(name), "%sthrow_file_dynptr_held", prefix);
if (test__start_subtest(name))
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
index 6e16caee66da..8700349a8f9c 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -1844,4 +1844,47 @@ __naked int throw_callee_unlocks(void)
: __clobber_all);
}
+/* ---- what a frame holds that another frame acquired ---- */
+
+static __noinline void new_iter_in_caller(struct bpf_iter_num *it)
+{
+ bpf_iter_num_new(it, 0, 10);
+}
+
+/*
+ * The callee creates the iterator in the stack of its caller, which holds it
+ * from then on: the throw of the caller destroys it.
+ */
+SEC("tc")
+int throw_iter_from_callee(struct __sk_buff *ctx)
+{
+ struct bpf_iter_num it;
+ int *v;
+
+ new_iter_in_caller(&it);
+ mark_acquired(&it);
+ while ((v = bpf_iter_num_next(&it))) {
+ if (*v == 3 && ctx->len)
+ bpf_throw(0xeB9F);
+ }
+ bpf_iter_num_destroy(&it);
+ return 0;
+}
+
+static __noinline void enter_rcu_in_callee(void)
+{
+ bpf_rcu_read_lock();
+}
+
+/* The callee returns in the section it entered, which the throw of its caller leaves. */
+SEC("tc")
+int throw_rcu_from_callee(struct __sk_buff *ctx)
+{
+ enter_rcu_in_callee();
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_rcu_read_unlock();
+ return 0;
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
index e98d1caf27f3..0faf464d113a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -799,10 +799,139 @@ __naked int callee_unlocks_slot(void)
: __clobber_all);
}
+private(C) struct bpf_spin_lock wlock;
+
+static __noinline void take_wlock(void)
+{
+ bpf_spin_lock(&wlock);
+}
+
+static __noinline void drop_wlock(void)
+{
+ bpf_spin_unlock(&wlock);
+}
+
+/*
+ * One subprog takes the lock and returns holding it, another drops it, and no
+ * throw finds it held: nothing keeps track of it.
+ */
+SEC("tc")
+__description("frame_desc: a lock a subprog returns holding, away from the throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 0")
+int lock_wrappers(struct __sk_buff *skb)
+{
+ if (skb->len > 1000)
+ bpf_throw(0);
+ take_wlock();
+ drop_wlock();
+ return 0;
+}
+
+static __noinline void enter_rcu(void)
+{
+ bpf_rcu_read_lock();
+}
+
+/*
+ * The subprog returns in the RCU read section it entered, and no throw finds
+ * the program in it: nothing counts it.
+ */
+SEC("tc")
+__description("frame_desc: a section a subprog returns in, away from the throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 0")
+int rcu_wrapper(struct __sk_buff *skb)
+{
+ if (skb->len > 1000)
+ bpf_throw(0);
+ enter_rcu();
+ bpf_rcu_read_unlock();
+ return 0;
+}
+
+static __noinline void new_iter_for_caller(struct bpf_iter_num *it)
+{
+ bpf_iter_num_new(it, 0, 10);
+}
+
+/*
+ * The callee creates the iterator in the stack of its caller, which holds it
+ * from then on: the descriptor of the caller's throw names it there.
+ */
+SEC("tc")
+__description("frame_desc: an iterator a callee creates in the stack of its caller")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1")
+int iter_from_callee(struct __sk_buff *skb)
+{
+ struct bpf_iter_num it;
+ int *v;
+
+ new_iter_for_caller(&it);
+ while ((v = bpf_iter_num_next(&it))) {
+ if (*v == 3 && skb->len > 100)
+ bpf_throw(0);
+ }
+ bpf_iter_num_destroy(&it);
+ return 0;
+}
+
+/*
+ * A loop the verifier walks reserves a record at one insn on each of two
+ * iterations and holds both at once, where no throw finds them: one slot
+ * could not hold both, and none is needed.
+ */
+SEC("tc")
+__description("frame_desc: two records reserved at one insn, away from the throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 0")
+__naked int two_from_one_insn(void)
+{
+ asm volatile (
+ "r8 = *(u32 *)(r1 + %[len]);"
+ "if r8 < 1000 goto +2;"
+ "r1 = 0;"
+ "call %[bpf_throw];"
+ "r7 = 0;"
+ "r6 = 0;"
+ "1: r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 4f;"
+ "if r7 != 0 goto 2f;"
+ "r7 = r0;"
+ "goto 3f;"
+ "2: *(u64 *)(r10 - 8) = r0;"
+ "3: r6 += 1;"
+ "if r6 < 2 goto 1b;"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "4: if r7 == 0 goto 5f;"
+ "r1 = r7;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "5: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
/*
* A loop the verifier walks reserves a record at one insn on each of two
- * iterations, the first kept in R7 and the second spilled: the call has one
- * slot in the frame, which cannot hold both records.
+ * iterations, the first kept in R7 and the second spilled, and a throw finds
+ * both held: the call would need a slot, which cannot hold both records.
*/
SEC("tc")
__description("frame_desc: two records reserved at one insn are held at once")
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 28/29] selftests/bpf: Test cleanup of locks described at sites
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (26 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 27/29] selftests/bpf: Test resources a callee leaves with its caller Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 12:57 ` [PATCH bpf-next v1 29/29] selftests/bpf: Test callee releases through callx Kumar Kartikeya Dwivedi
2026-11-11 20:33 ` [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Alexei Starovoitov
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add programs to exceptions_cleanup that throw while holding a lock that
has no hidden slot:
- the lock of a global variable, taken by a subprog for its caller,
which is at a fixed address;
- the lock of a map element, found through the pointer to the element
that the frame keeps in R7.
Check in verifier_frame_desc that each takes a site descriptor, no slot,
and no inserted instruction.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 3 +
.../selftests/bpf/progs/exceptions_cleanup.c | 63 +++++++++++++++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 61 ++++++++++++++++++
3 files changed, 127 insertions(+)
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
index 9ba2cd9c8097..5ef5f01a6be1 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -265,6 +265,9 @@ static void run_suite(const char *prefix)
/* What a frame holds that a callee acquired for it. */
THROW(throw_iter_from_callee, 1, "0");
THROW(throw_rcu_from_callee, 0, "s");
+ /* Locks the sites name: at a fixed address, or through a pointer the frame keeps. */
+ THROW(throw_lock_from_callee, 1, "0");
+ THROW(throw_lock_in_value, 1, "0");
/* A file and the reader of its dynptr, which goes first. */
snprintf(name, sizeof(name), "%sthrow_file_dynptr_held", prefix);
if (test__start_subtest(name))
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
index 8700349a8f9c..d6bb893f9da2 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -1887,4 +1887,67 @@ int throw_rcu_from_callee(struct __sk_buff *ctx)
return 0;
}
+/* ---- locks the sites of a frame name ---- */
+
+static __noinline void take_glock(void)
+{
+ bpf_spin_lock(&glock);
+}
+
+/*
+ * A subprog takes the lock for its caller, which throws holding it: the lock
+ * of a global variable is at a fixed address, which the throw unlocks.
+ */
+SEC("tc")
+int throw_lock_from_callee(struct __sk_buff *ctx)
+{
+ mark_acquired(&glock);
+ take_glock();
+ if (ctx->len)
+ bpf_throw(0xeB9F);
+ bpf_spin_unlock(&glock);
+ return 0;
+}
+
+/*
+ * The lock of a map element is taken through the pointer in R7, which the
+ * frame still has at the throw: the lock is found through it.
+ */
+SEC("tc")
+__naked int throw_lock_in_value(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r1 = %[lock_map] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 1f;"
+ "r7 = r0;"
+ "r2 = r7;"
+ "r2 += %[lock_off];"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r1 = r7;"
+ "r1 += %[lock_off];"
+ "call %[bpf_spin_lock];"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(lock_map),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [lock_off]"i"(offsetof(struct locked_foo, lock))
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
index 0faf464d113a..e0456a929126 100644
--- a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -799,6 +799,67 @@ __naked int callee_unlocks_slot(void)
: __clobber_all);
}
+static __noinline void take_slock(void)
+{
+ bpf_spin_lock(&slock);
+}
+
+/*
+ * A subprog takes the lock and returns holding it, and the throw of its
+ * caller finds it held: the lock is at a fixed address, so no slot has to
+ * follow it out of the frame that took it.
+ */
+SEC("tc")
+__description("frame_desc: a lock a subprog returns holding, held at the throw")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1")
+int lock_wrapper_at_throw(struct __sk_buff *skb)
+{
+ take_slock();
+ if (skb->len > 100)
+ bpf_throw(0);
+ bpf_spin_unlock(&slock);
+ return 0;
+}
+
+/*
+ * The lock is in a map value the frame keeps a pointer to in R7: the
+ * descriptor of the throw names the register and how far into the value the
+ * lock is.
+ */
+SEC("tc")
+__description("frame_desc: a lock found through a pointer the frame keeps has no slot")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 0 insns 0 sites 1")
+__naked int lock_in_value_at_throw(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r1 = %[lock_values] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 1f;"
+ "r7 = r0;"
+ "r1 = r7;"
+ "r1 += %[lock_off];"
+ "call %[bpf_spin_lock];"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(lock_values),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [lock_off]"i"(offsetof(struct lock_value, lock))
+ : __clobber_all);
+}
+
private(C) struct bpf_spin_lock wlock;
static __noinline void take_wlock(void)
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* [PATCH bpf-next v1 29/29] selftests/bpf: Test callee releases through callx
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (27 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 28/29] selftests/bpf: Test cleanup of locks described at sites Kumar Kartikeya Dwivedi
@ 2026-11-11 12:57 ` Kumar Kartikeya Dwivedi
2026-11-11 20:33 ` [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Alexei Starovoitov
29 siblings, 0 replies; 58+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-11 12:57 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, Emil Tsalapatis, Ihor Solodrai, kkd,
kernel-team
Add a program to exceptions_cleanup that keeps a record in a hidden slot
and passes it to a callee through a function pointer. The caller throws
with the record in its slot, and the callee throws before and after its
discard, when the slot of the caller is already cleared.
Check in verifier_frame_desc the slots and instructions that this takes,
and that the caller clears the hidden slot of a lock before a callx to a
subprog that unlocks.
Add a program to exceptions_fail in which a callx can call one subprog
that discards the record and another that leaves it with the caller,
which keeps it in a hidden slot.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
.../bpf/prog_tests/exceptions_cleanup.c | 9 ++
.../selftests/bpf/progs/exceptions_cleanup.c | 44 ++++++++++
.../selftests/bpf/progs/exceptions_fail.c | 70 ++++++++++++++++
.../selftests/bpf/progs/verifier_frame_desc.c | 83 +++++++++++++++++++
4 files changed, 206 insertions(+)
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
index 5ef5f01a6be1..07d3e8cd7046 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -253,6 +253,15 @@ static void run_suite(const char *prefix)
1, "", .param = 10);
RUN_AS(throw_callee_consumes_from_slot, "throw_callee_consumes_from_slot_return", 0,
1, "", .param = 8);
+ /* a callx hands over like a call, and its caller empties the slot in front of it */
+ RUN_AS(throw_callx_consumes_from_slot, "throw_callx_consumes_from_slot_caller", THROWN,
+ 1, "0", .param = 3);
+ RUN_AS(throw_callx_consumes_from_slot, "throw_callx_consumes_from_slot_callee", THROWN,
+ 1, "0", .param = 1);
+ RUN_AS(throw_callx_consumes_from_slot, "throw_callx_consumes_from_slot_after", THROWN,
+ 1, "", .param = 2);
+ RUN_AS(throw_callx_consumes_from_slot, "throw_callx_consumes_from_slot_return", 0,
+ 1, "", .param = 0);
/* the callee has it nowhere at its throw and keeps it in a slot of its own */
RUN_AS(throw_callee_consumes_slot, "throw_callee_consumes_slot_throw", THROWN, 1, "0",
.param = 1);
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
index d6bb893f9da2..13cb476dffe6 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -1730,6 +1730,50 @@ __naked int throw_callee_consumes_from_slot(void)
: __clobber_all);
}
+/*
+ * The same through the address of the callee, with the record in no register
+ * at the throw of the caller: the insns in front of the callx empty the slot
+ * of the caller, so a throw of the callee in front of its discard releases
+ * the record once.
+ */
+SEC("tc")
+__naked int throw_callx_consumes_from_slot(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r1 = %[acquired] ll;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r2 = 1;"
+ "r1 = %[acquired_cnt] ll;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "r8 = %[param] ll;"
+ "r8 = *(u32 *)(r8 + 0);"
+ "if r8 != 3 goto 1f;"
+ "r6 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "r2 = %[consume_in_callee] ll;"
+ "callx r2;"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(acquired),
+ __imm_addr(acquired_cnt),
+ __imm_addr(param),
+ __imm_addr(consume_in_callee),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw)
+ : __clobber_all);
+}
+
/*
* Discards the record its caller hands it. When it throws instead the record
* is in no register: the cookie has taken its place in R1.
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index 9b47345d5fd0..630ae7b9e857 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -561,6 +561,76 @@ __naked int reject_callee_releases_sometimes(void)
: __clobber_all);
}
+/* Discards the record in R1. */
+__used __naked static __noinline int discard_rec(void)
+{
+ asm volatile (
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_ringbuf_discard)
+ : __clobber_all);
+}
+
+/* Leaves the record in R1 with its caller. */
+__used __naked static __noinline int keep_rec(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * The record is in no register at the throw, so it is in a slot. The insns in
+ * front of the callx empty the slot for the subprog that is handed the
+ * record, and would for the one that leaves it with the caller as well, which
+ * discards it after the call.
+ */
+SEC("?tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("to a subprog that releases it and to one that does not")
+__naked int reject_callx_releases_sometimes(void)
+{
+ asm volatile (
+ "r7 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[fail_ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r6 = r0;"
+ "r0 = 0;"
+ "if r7 != 3 goto 1f;"
+ "r6 = 0;"
+ "r1 = 0;"
+ "call %[bpf_throw];"
+ "1: r2 = %[discard_rec] ll;"
+ "r8 = 0;"
+ "if r7 != 1 goto 2f;"
+ "r2 = %[keep_rec] ll;"
+ "r8 = 1;"
+ "2: r1 = r6;"
+ "callx r2;"
+ "if r8 == 0 goto 3f;"
+ "r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+ "3: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(fail_ringbuf),
+ __imm_addr(discard_rec),
+ __imm_addr(keep_rec),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_ringbuf_discard),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
__used static __noinline void leave_rcu(void)
{
bpf_rcu_read_unlock();
diff --git a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
index e0456a929126..489cdd897c6c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
+++ b/tools/testing/selftests/bpf/progs/verifier_frame_desc.c
@@ -799,6 +799,89 @@ __naked int callee_unlocks_slot(void)
: __clobber_all);
}
+/*
+ * The same through the address of the callee: the callx finds the slot of the
+ * caller emptied as the call does, since the only subprog it calls unlocks.
+ */
+SEC("tc")
+__description("frame_desc: the caller empties the slot of a lock in front of a callx")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 3 sites 0 classes 1")
+__naked int callx_unlocks_slot(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r1 = %[lock_values] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 2f;"
+ "r7 = r0;"
+ "r1 = r7;"
+ "r1 += %[lock_off];"
+ "call %[bpf_spin_lock];"
+ "if r6 != 1 goto 1f;"
+ "r7 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r7;"
+ "r2 = %[unlock_value] ll;"
+ "callx r2;"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(lock_values),
+ __imm_addr(unlock_value),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_spin_lock),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len)),
+ [lock_off]"i"(offsetof(struct lock_value, lock))
+ : __clobber_all);
+}
+
+/*
+ * The caller has the record nowhere at its throw, so it keeps it in a slot:
+ * filled after the reserve, empty at the entry, and emptied in front of the
+ * callx that hands the record to the callee, which holds it in R6.
+ */
+SEC("tc")
+__description("frame_desc: a callx hands a record over from the slot of the caller")
+__arch_x86_64 __arch_arm64
+__success __log_level(5)
+__msg("frame descriptors: slots 1 insns 5 sites 1 classes 1")
+__naked int callx_consumes_record(void)
+{
+ asm volatile (
+ "r7 = *(u32 *)(r1 + %[len]);"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r0 = 0;"
+ "if r7 != 3 goto 1f;"
+ "r6 = 0;"
+ "r1 = 0xeB9F;"
+ "call %[bpf_throw];"
+ "1: r1 = r6;"
+ "r2 = %[consume_record] ll;"
+ "callx r2;"
+ "2: r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(ringbuf),
+ __imm_addr(consume_record),
+ __imm(bpf_ringbuf_reserve),
+ __imm(bpf_throw),
+ [len]"i"(offsetof(struct __sk_buff, len))
+ : __clobber_all);
+}
+
static __noinline void take_slock(void)
{
bpf_spin_lock(&slock);
--
2.53.0
^ permalink raw reply related [flat|nested] 58+ messages in thread* Re: [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux
2026-11-11 12:56 [PATCH bpf-next v1 00/29] Exceptions - Resource Cleanup - Redux Kumar Kartikeya Dwivedi
` (28 preceding siblings ...)
2026-11-11 12:57 ` [PATCH bpf-next v1 29/29] selftests/bpf: Test callee releases through callx Kumar Kartikeya Dwivedi
@ 2026-11-11 20:33 ` Alexei Starovoitov
29 siblings, 0 replies; 58+ messages in thread
From: Alexei Starovoitov @ 2026-11-11 20:33 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf
Cc: Andrii Nakryiko, Daniel Borkmann, Eduard Zingerman,
Emil Tsalapatis, Ihor Solodrai, kkd, kernel-team
On Wed, Nov 11, 2026 at 04:56 AM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> This series allows the main program and its subprogs, static and global,
> to throw while holding resources. The unwinder releases the resources of
> every frame it discards.
Agree that we need bpf_throw() from everywhere.
Below is an idea for brainstorming. No code yet.
Drop elaboration in the verifier. Pads instead of descriptors.
Keep the analysis of what a frame holds at a site.
Change what comes out of it: not tables for a C unwinder, but landing pads.
A pad is BPF insns that the verifier adds after the main pass, like guards.
Then throw, cancel and rust panic are one mechanism:
unwind_goto / unwind_exit.
1. One unwinder
site the verifier emits
----------------------------------- ---------------------------------
call bpf_throw r0 = cookie; goto pad
loop guard, recursion check may_goto pad
call / callx of a subprog that may unwind_goto pad, behind the call
unwind, no unwind_goto behind it
helper that runs a callback unwind_goto pad, behind the call
(your 'helpers have pads')
arena load / store / atomic unwind_goto pad, behind the access.
JIT points the extable entry at it
pad: release calls
r0 = 0
unwind_exit main prog: call exception_cb; exit
Every frame leaves through its epilogue. The frame at the site has R0-R9.
A frame that waits in a call has R6-R9 and its stack.
No stack walk, no asm entries, no saved register offsets, no site tables.
No void (*)(void *) wrappers: a pad calls the release kfunc with its type.
A frame of rust keeps its own pads. Nothing is skipped by construction.
Sites that need the same pad share it. Most sites hold nothing.
2. Three questions, answered apart
WHO The frame that acquired it since its entry.
releases What a callee returns with is the caller's.
Or the callee that it was MOVED into: the caller keeps no
live copy behind the call (R6-R9, stack).
Decided at the call from the state and liveness.
Nothing provisional. Nothing to settle after the pass.
WHERE is The prog's own live place, like your lattice: register,
the handle spill, stack object, pointer to what the lock is in + delta,
fixed address, load through a live pointer.
The prog has to release it later, so the handle is live.
WHETHER The pc, when all states at the site agree.
it is held Otherwise a DROP FLAG, as rustc and clang do it.
3. The drop flag
One hidden byte in the frame. The entry of the subprog zeroes the word.
flag = 1 behind the insn of THIS frame that brought the resource:
the acquiring call, the call of a subprog that returns
with it, or the entry when it was moved in
flag = 0 at the insn of THIS frame that takes it away:
the releasing call, or the call that moves it out
*(u8 *)(r10 - H) = 0
if r7 == 0 goto skip
call bpf_task_from_pid
r6 = r0
*(u8 *)(r10 - H) = 1
skip: ... arena access, loop, call foo ... -> pad
if r7 == 0 goto out
*(u8 *)(r10 - H) = 0
r1 = r6
call bpf_task_release
pad: r1 = *(u8 *)(r10 - H)
if r1 == 0 goto done
r1 = r6
if r1 == 0 goto done
call bpf_task_release
done: r0 = 0
unwind_exit
The stores are unconditional. bpf_res_spin_lock() needs a test of r0 first.
The flag has no handle in it. So there is no register to find at the
acquire, no compare chain at the release, no refresh after a call.
A flag is emitted only where the states at some site disagree.
Identity: a reference carries its origin, as in your set.
A join or a subsumption can pair two references with different origins.
Then the origins go into one set (union-find) and share a flag.
States are joined as today.
Sections become entries of refs[], so they have an origin too.
4. What I hope goes away, in your terms
tables, unwinder, phases, asm entries, pads
registers recovered from prologues
mirror flag + the prog's own place
NULL in the lattice flag. A pad tests for NULL too
entry guarded by a scalar flag
descriptor per chain of calls flag in the callee's frame:
'differs by chain' is 'held
on some paths' there
provisional listings, transfer move at the call
other origins are not joined union-find
What stays: classes, type -> release kfunc, the places, helpers have pads,
the return value of a cancelled prog, the struct_ops hook, the stack in
arena, bpf_throw() reads all registers and slots for liveness.
5. Where I'm looking for ideas.
a. Several resources of one origin at once. rbtree_search pops nodes
into an array in a loop that is walked. A flag per origin cannot say
which.
b. A callee releases what the caller keeps live, on some paths only.
try_consume(obj). During the call the pc of the callee tells.
Behind the call only the callee knows. Maybe: the callee has its own
flag and returns it in a scratch register at exit and unwind_exit.
R2 survives exit already for 16 byte returns.
I think you're doing something like that in your followup patches.
^ permalink raw reply [flat|nested] 58+ messages in thread