* [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields
@ 2026-11-16 15:26 Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze Alexei Starovoitov
` (20 more replies)
0 siblings, 21 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
In short: the verifier keeps the maps, kfuncs, subprogs and special fields
of a program in arrays of a fixed size. A safe program that needs one more
entry is rejected. With this series the arrays grow with the program. What
was quadratic or worse below a limit is linear or n log n now. Outside of
tools/ the kernel loses 300 lines.
today new patch
maps of a program 64 no limit 7
BTF objects of a program 64 no limit 7
entries of fd_array 4096 no limit 7
kfuncs of a program 256 65536 9
module BTFs of these kfuncs 256 no limit 9
tables of functions, for callx:
pointers in all maps 1048576 no limit 10
pointers in the range of one read 4096 no limit 10
functions that one read can give 1024 no limit 10
sets of functions 4096 no limit 10
subprogs 1024 no limit 16
special fields of a map value 11 no limit 17
special fields of an allocated object 11 512 17
bytes of a name in BTF 512 4096 18
insns under a jump, with those that the
verifier adds 32767 no limit 12
insns of a program that bpftool can dump 524288 no limit 5
maps of a light skeleton 64 see 7 11
programs of a light skeleton 32 see 7 11
A table of functions is an array of pointers to subprogs in a read-only
map. A program reads a pointer from it and calls it with callx.
A special field is a kptr, a bpf_spin_lock, a bpf_timer, a bpf_list_head
and the like.
1. Why?
None of these numbers protects the kernel at run time. Each is the size of
an array, or it hides an algorithm that does not scale. Such a limit is
raised when a real program hits it. Rust crates did that to two of them:
commit 70bcb39a7392 ("bpf: Bump BPF_MAX_SUBPROGS to 1024") and commit
3f4f430fa1b2 ("bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table
of functions"). Real code is near the next ones:
subprogs 777 of 1024 the program of rust-bpf [1] in the first
commit
name in BTF 429 of 512 a struct of scx_lavd in rust-bpf
maps 41 of 64 obi_continue_protocol_http() in OBI of
OpenTelemetry
And it is over the limits of a light skeleton:
runtime-security.o of Datadog 627 programs 151 maps
tracee.bpf.o of Tracee 169 programs 87 maps
The author of a program does not choose the number of its maps. Each
.data.X and .rodata.X section is a map. Each jump table that clang makes
from a switch is a map: with -mcpu=v4 bpf_generic_uprobe.o of Tetragon has
19 of them next to its 66 maps.
2. What bounds BPF_PROG_LOAD now?
size 1000000 insns of a program and 16 MB of BTF, as today.
memory memory.max of the cgroup of the task. There bpf() returns
-ENOMEM. Today the OOM killer ends the task (patch 2).
time A fatal signal ends each long loop. The freezer does not wait
either: the task freezes, then bpf() starts again (patch 1).
A large program also has to get to the end of BPF_PROG_LOAD. Today each
signal ends it with -EAGAIN, and libbpf gives up after 5 tries. A task with
a profiling timer of 10 ms never gets through a BPF_PROG_LOAD of 11 ms.
With patch 1 a signal that does not kill waits for the end.
The limits that hold at run time do not change: the size of the stack,
32 frames, 33 tail calls.
3. What had to scale?
patch question today new
8 dump: which map is in look at all maps maple tree
this ld_imm64?
9 is this kfunc known? sort at each new one hash table
10 is this set of look at all sets hash table
functions known?
13 who calls whom, over matrix of bits, lists of edges,
any number of calls? n^3 / 64, two times nodes + edges
14 what does a table of walk its pointers one node of the
functions call? at each use call graph
15 which decl tags has look at all types a list per type
this argument?
17 which special fields enter a struct for enter it once,
has this type? each member of that copy its fields
type: members^depth
18 is this name valid? read it for each read each byte
type that has it once
19 which insns dominate quadratic n log n
this insn?
Three small tools carry most of it:
bpf_table_next() an array that doubles patch 7
struct bpf_index hash table, key -> number of an entry patch 7
struct bpf_call_graph lists of edges, sorted once: callees patch 13
first, the nodes of a cycle in a row
4. Far jumps
A conditional jump has an offset of 16 bits. The verifier adds insns under
it: inlined map lookups, checks of divisions, may_goto, constant blinding.
Then a verified program fails with -ERANGE. 2979 map lookups under one
branch are enough. Patch 12 does what an assembler does:
if r1 > r2 goto far -> if r1 > r2 goto +1
goto +1
gotol far
Where all jumps fit, no jump changes. The passes in fixups.c do not compute
offsets any more: bpf_patch_list_build() does it for all of them.
5. Fixes
3 A syscall program and the target of its tail call get the same
context. Its size is compared with the first program only.
4 bpf_loop() is inlined with a call of its callback, also when the
callback is removed as dead code.
5 A program of more than 524288 insns loads and cannot be dumped.
6 btf_resolve() does not see a type that holds itself through
ptr -> typedef -> array. Only the limits of the later walks end that
loop, and patch 17 removes one of them. With patch 6 such BTF is
refused.
6. What else changes for users?
- The rules on calls look only at insns that the main pass has seen. A
call, a tail call, LD_ABS or bpf_throw() in dead code rejects nothing
(patch 13).
- A map value with 12 kptrs had no struct btf_record at all. It has one
now. So the map needs CAP_BPF, and it cannot be mmaped or
BPF_F_RDONLY_PROG (patch 17).
- -ENOMEM in front of the JIT fails BPF_PROG_LOAD. There is no fall back
to the interpreter (patch 2).
- The loader program of a light skeleton gets -EINTR from bpf_sys_bpf().
It got -EAGAIN (patch 1).
- An extension that does tail calls cannot attach to a syscall program
that does none (patch 3).
- The tags of subprogs that make calls change, and so do their names in
kallsyms (patch 16).
- x86-32, riscv32, parisc and sparc64 have no gotol in the JIT. A program
with a far jump is not JITed there, like a program that has gotol in
its source (patch 12).
7. What stays?
65536 kfuncs JITs find a kfunc by insn->off: 16 bits.
4096 bytes of a name A search by name takes types x length.
512 special fields An allocated object has 4096 bytes at most.
4096 entries of For unprivileged programs. They have 4096 insns.
fd_array
10000 chains of calls In the analysis of stack liveness.
32 levels of types In btf_resolve().
64 KB of context For the loader program of a light skeleton:
16 bytes for a map, 4 for a program.
bpf_flow_reset() It looks at all insns after each global subprog.
A fatal signal ends that loop.
The tests are in the patch that changes the behaviour (7, 10, 12, 13, 17)
and in patches 20 and 21. Where a program above an old limit can be unsafe,
an unsafe twin of it must fail.
v1 -> v2:
- Patch 1: bpf() starts again after a freeze also when the command has
BPF_COMMON_ATTRS. It returned -EINTR. Patch 21 tests it (BPF CI bot).
- Patch 10: allocate a set of functions in front of its entry in the index.
If that allocation failed in a round that is done again, the next round
found the entry and no set behind it (sashiko).
- Patch 12: the test 'far jump over a far jump' relaxed no jump: both had
an offset of 32767. Now they have 32766 and 32768: the second one is far
and takes the first one along. The test looks at the insns (BPF CI bot).
- Patch 20: live_recursion runs on x86-64 and arm64 only: other JITs refuse
the recursion with another message. The far_pad tests need the JIT
(sashiko).
[1] https://github.com/4ast/rust-bpf
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Alexei Starovoitov (21):
bpf: Stop a load only when the loader is killed or has to freeze
bpf: Fail the load when there is no memory
bpf: Give a syscall program the largest context when it does tail
calls
bpf: Do not inline bpf_loop() when its callback is dead code
bpf: Dump programs that have more than 524288 insns
bpf: Resolve BTF structs and arrays first
bpf: Remove the limits on maps, BTFs and fd_array of a program
bpf: Do not scan all maps for each insn and each subprog
bpf: Remove the limits on kfuncs of a program
bpf: Remove the limits on tables of functions
libbpf: Size the tables of the loader program from the object
bpf: Relax jumps that patches push out of range
bpf: Keep the call graph in lists of edges
bpf: Find the effects of callees on the call graph
bpf: Link the decl tags of a type
bpf: Remove BPF_MAX_SUBPROGS
bpf: Walk each type once when looking for special fields
bpf: Raise the limit on the length of BTF names
bpf: Find dominators in O(n log n)
selftests/bpf: Test programs above the old limits
selftests/bpf: Test limits of the load with raw programs
arch/sparc/net/bpf_jit_comp_64.c | 7 +-
arch/x86/net/bpf_jit_comp.c | 12 +-
include/linux/bpf.h | 5 -
include/linux/bpf_verifier.h | 162 ++-
kernel/bpf/btf.c | 503 +++----
kernel/bpf/cfg.c | 428 +++---
kernel/bpf/check_btf.c | 13 +-
kernel/bpf/const_fold.c | 8 +-
kernel/bpf/core.c | 368 +++---
kernel/bpf/diagnostics.c | 8 +-
kernel/bpf/fixups.c | 702 +++-------
kernel/bpf/flow.c | 99 +-
kernel/bpf/join.c | 2 +-
kernel/bpf/liveness.c | 66 +-
kernel/bpf/log.c | 2 +-
kernel/bpf/loops.c | 189 +--
kernel/bpf/scev.c | 32 +-
kernel/bpf/syscall.c | 126 +-
kernel/bpf/verifier.c | 1164 ++++++-----------
net/bpf/test_run.c | 20 +-
tools/lib/bpf/bpf_gen_internal.h | 3 +-
tools/lib/bpf/gen_loader.c | 157 +--
tools/lib/bpf/libbpf.c | 8 +-
tools/testing/selftests/bpf/Makefile.skel | 2 +-
.../{blinding_log.c => blinding_far_jump.c} | 38 +-
.../selftests/bpf/prog_tests/fd_array.c | 36 +-
.../selftests/bpf/prog_tests/no_limits.c | 465 +++++++
.../selftests/bpf/prog_tests/verifier.c | 4 +
.../selftests/bpf/progs/cpumask_common.h | 3 -
.../selftests/bpf/progs/cpumask_failure.c | 5 +-
.../testing/selftests/bpf/progs/many_progs.c | 36 +
.../bpf/progs/test_global_func_deep_stack.c | 12 +-
.../selftests/bpf/progs/verifier_and.c | 2 +-
.../selftests/bpf/progs/verifier_bounds.c | 14 +-
.../bpf/progs/verifier_callx_dead_cb.c | 60 +
.../selftests/bpf/progs/verifier_callx_many.c | 31 +-
.../selftests/bpf/progs/verifier_movsx.c | 4 +-
.../selftests/bpf/progs/verifier_no_limits.c | 224 ++++
.../selftests/bpf/progs/verifier_patch_list.c | 146 ++-
.../selftests/bpf/progs/verifier_tailcall.c | 4 +-
.../selftests/bpf/progs/verifier_unpriv.c | 2 +-
.../selftests/bpf/progs/verifier_unwind.c | 32 +
tools/testing/selftests/bpf/verifier/calls.c | 3 +-
43 files changed, 2924 insertions(+), 2283 deletions(-)
rename tools/testing/selftests/bpf/prog_tests/{blinding_log.c => blinding_far_jump.c} (51%)
create mode 100644 tools/testing/selftests/bpf/prog_tests/no_limits.c
create mode 100644 tools/testing/selftests/bpf/progs/many_progs.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_dead_cb.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_no_limits.c
--
2.55.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 16:15 ` sashiko-bot
2026-11-16 15:26 ` [PATCH bpf-next v2 02/21] bpf: Fail the load when there is no memory Alexei Starovoitov
` (19 subsequent siblings)
20 siblings, 1 reply; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Any pending signal stops the verifier with -EAGAIN. A loader with a
profiling timer never loads a program that takes longer than one period:
timer: |--10ms--|--10ms--|--10ms--|
load: |--------X EAGAIN
|--------X EAGAIN ... libbpf gives up after 5 tries
Use one helper for all long loops. Add it to the loops that had no
check: cfg, const_fold, stack liveness and CO-RE relocations.
event load
fatal signal stops
system or cgroup freezer stops, the task freezes, bpf() starts again
other signals goes on
The restart is not visible in user space. A loader program gets -EINTR
from bpf_sys_bpf(), it got -EAGAIN. Unprivileged loaders wait for one
lock: test the same condition when they get it, so the queue drains.
SIGSTOP, ptrace and signals with a handler wait for the end of the load.
A tool that stops and continues a task all the time is one more timer.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 16 ++++++++++++++++
kernel/bpf/cfg.c | 2 +-
kernel/bpf/check_btf.c | 4 ++--
kernel/bpf/const_fold.c | 4 ++++
kernel/bpf/fixups.c | 20 +++-----------------
kernel/bpf/flow.c | 29 +++++++++++------------------
kernel/bpf/liveness.c | 20 ++++++++++++--------
kernel/bpf/loops.c | 22 ++++------------------
kernel/bpf/scev.c | 10 ++++------
kernel/bpf/syscall.c | 9 ++++++++-
kernel/bpf/verifier.c | 14 +++++++-------
11 files changed, 72 insertions(+), 78 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 1afee19fcd25..ecb988ab6349 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -10,6 +10,22 @@
#include <linux/tnum.h>
#include <linux/cnum.h>
#include <linux/math64.h> /* for div_s64_rem() */
+#include <linux/freezer.h>
+#include <linux/sched/signal.h>
+
+/*
+ * For each loop that can run long. The caller returns -EINTR.
+ * killed the task dies
+ * freezer waits the task freezes, then bpf() starts again
+ * other signals wait for the end of the load
+ */
+static inline bool bpf_must_abort(void)
+{
+ cond_resched();
+ return task_sigpending(current) &&
+ (__fatal_signal_pending(current) || freezing(current) ||
+ (READ_ONCE(current->jobctl) & JOBCTL_TRAP_FREEZE));
+}
/*
* Mathematical modulo, the residue of 'v' modulo 'step', normalized to [0, step).
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index bbcf42eae9d9..163fdb21828f 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -686,7 +686,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
while (env->cfg.cur_stack > 0) {
int t = insn_stack[env->cfg.cur_stack - 1];
- ret = visit_insn(t, env);
+ ret = bpf_must_abort() ? -EINTR : visit_insn(t, env);
switch (ret) {
case DONE_EXPLORING:
insn_state[t] = EXPLORED;
diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c
index 8a3f2421331e..d6b930377d75 100644
--- a/kernel/bpf/check_btf.c
+++ b/kernel/bpf/check_btf.c
@@ -402,8 +402,8 @@ int bpf_check_core_relo(struct bpf_verifier_env *env,
break;
}
- err = bpf_core_apply(&ctx, &core_relo, i,
- &prog->insnsi[core_relo.insn_off / 8]);
+ err = bpf_must_abort() ? -EINTR :
+ bpf_core_apply(&ctx, &core_relo, i, &prog->insnsi[core_relo.insn_off / 8]);
if (err)
break;
bpfptr_add(&u_core_relo, rec_size);
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index 3c8c1b64088b..5869730547ae 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -278,6 +278,10 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
}
redo:
+ if (bpf_must_abort()) {
+ kvfree(ci_in);
+ return -EINTR;
+ }
changed = false;
for (i = env->cfg.cur_postorder - 1; i >= 0; i--) {
int idx = env->cfg.insn_postorder[i];
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 1d33a78d9373..735932392a87 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -6,7 +6,6 @@
#include <linux/filter.h>
#include <linux/vmalloc.h>
#include <linux/perf_event.h>
-#include <linux/sched/signal.h>
#include <net/xdp.h>
#include "disasm.h"
@@ -165,19 +164,6 @@ static int get_callee_stack_depth(struct bpf_verifier_env *env,
}
#endif
-/*
- * A post-verification rewriting pass can queue a patch for each instruction
- * of a large program. Keep the primitives that queue them killable and
- * preemptible.
- */
-static bool bpf_rewrite_must_abort(void)
-{
- if (fatal_signal_pending(current))
- return true;
- cond_resched();
- return false;
-}
-
/*
* Queue the replacement of insn 'off' by 'patch[0..len)' until
* bpf_patch_list_commit(), see __bpf_patch_list_add(). Replacing one insn by
@@ -190,7 +176,7 @@ struct bpf_insn *bpf_patch_list_add(struct bpf_verifier_env *env, u32 off,
struct bpf_patch_list *pl = &env->patch_list;
struct bpf_insn_patch *p = pl->cnt ? &pl->patches[pl->cnt - 1] : NULL;
- if (bpf_rewrite_must_abort())
+ if (bpf_must_abort())
return ERR_PTR(-EINTR);
if (verifier_bug_if(!len || off >= env->prog->len ||
@@ -221,7 +207,7 @@ int bpf_patch_list_del(struct bpf_verifier_env *env, u32 off)
{
struct bpf_patch_list *pl = &env->patch_list;
- if (bpf_rewrite_must_abort())
+ if (bpf_must_abort())
return -EINTR;
if (verifier_bug_if(off >= env->prog->len ||
@@ -240,7 +226,7 @@ int bpf_patch_list_insert(struct bpf_verifier_env *env, u32 off,
struct bpf_insn_patch *p = pl->cnt ? &pl->patches[pl->cnt - 1] : NULL;
int err;
- if (bpf_rewrite_must_abort())
+ if (bpf_must_abort())
return -EINTR;
if (verifier_bug_if(!cnt || off >= env->prog->len ||
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 5341f95c5072..fbd1178eb9f5 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -504,11 +504,10 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)
if (!loop)
continue;
/* a deep nest of loops takes long */
- if (signal_pending(current)) {
- ret = -EAGAIN;
+ if (bpf_must_abort()) {
+ ret = -EINTR;
break;
}
- cond_resched();
loop->rt_bounded = flow_loop_is_bounded(env, h);
/* also when it's bounded: the main pass may find that it's not */
flow_place_guards(env, h, seen);
@@ -884,11 +883,10 @@ int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
for (n = 0; lost; n++) {
lost = false;
/* a chain of calls takes a round for each link */
- if (signal_pending(current)) {
- err = -EAGAIN;
+ if (bpf_must_abort()) {
+ err = -EINTR;
break;
}
- cond_resched();
for (j = 0; j < cnt; j++) {
/* it has callees first */
s = env->subprog_topo_order[n & 1 ? j : cnt - 1 - j];
@@ -1068,11 +1066,10 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
again:
while (any && changed) {
changed = false;
- if (signal_pending(current)) {
- err = -EAGAIN;
+ if (bpf_must_abort()) {
+ err = -EINTR;
break;
}
- cond_resched();
for (s = 0; callx && s < env->subprog_cnt; s++)
if (env->subprog_info[s].addr_taken)
callx_regs |= need[env->subprog_info[s].start].regs;
@@ -1619,11 +1616,10 @@ static int flow_compute_rungs(struct bpf_verifier_env *env)
depth++;
if (depth >= LOOP_STACK_SIZE)
continue;
- if (signal_pending(current)) {
- err = -EAGAIN;
+ if (bpf_must_abort()) {
+ err = -EINTR;
break;
}
- cond_resched();
err = flow_compute_loop_rungs(env, at, total, h);
}
kvfree(at);
@@ -2737,12 +2733,10 @@ static int flow_merge(struct bpf_verifier_env *env, struct flow_set *set,
int prev = s->prev_insn_idx;
again:
- if (signal_pending(current)) {
- err = -EAGAIN;
+ if (bpf_must_abort()) {
+ err = -EINTR;
goto err_free;
}
- if (need_resched())
- cond_resched();
/* again after a join: an offset may not be a constant any more, a pointer may be gone */
flow_shape_ctx(&s->st, &s->shape);
@@ -3397,7 +3391,7 @@ void bpf_flow_print_way(struct bpf_verifier_env *env, u64 log_pos, int head, int
if (!cnt || bpf_vlog_stash(&env->log, log_pos, &stash))
goto out;
env->prev_linfo = NULL;
- while (cnt-- && !signal_pending(current)) {
+ while (cnt-- && !bpf_must_abort()) {
w = &way[cnt];
if (w->s) {
verbose(env, "from %d to %d:", w->s->prev_insn_idx, w->head);
@@ -3409,7 +3403,6 @@ void bpf_flow_print_way(struct bpf_verifier_env *env, u64 log_pos, int head, int
bpf_verbose_insn(env, &insns[i]);
verbose(env, "\n");
}
- cond_resched();
}
bpf_vlog_unstash(&env->log, &stash);
out:
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 06f739610c73..2c771b936d65 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -456,7 +456,7 @@ static inline bool update_insn(struct bpf_verifier_env *env, struct func_instanc
}
/* Fixed-point computation of @live_before marks */
-static void update_instance(struct bpf_verifier_env *env, struct func_instance *instance)
+static int update_instance(struct bpf_verifier_env *env, struct func_instance *instance)
{
u32 i, frame, po_start, po_end;
int *insn_postorder = env->cfg.insn_postorder;
@@ -469,6 +469,8 @@ static void update_instance(struct bpf_verifier_env *env, struct func_instance *
po_end = (subprog + 1)->postorder_start;
/* repeat until fixed point is reached */
do {
+ if (bpf_must_abort())
+ return -EINTR;
changed = false;
for (frame = 0; frame <= instance->depth; frame++) {
if (!instance->frames[frame])
@@ -482,6 +484,7 @@ static void update_instance(struct bpf_verifier_env *env, struct func_instance *
}
}
} while (changed);
+ return 0;
}
static bool is_live_before(struct func_instance *instance, u32 insn_idx, u32 frameno, u32 half_spi,
@@ -2130,6 +2133,9 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
struct bpf_iarray *succ;
int ti;
+ err = -EINTR;
+ if (bpf_must_abort())
+ goto err_free;
if (!arg_is_visited(&at_in[i][0]) && !arg_is_visited(&at_in[i][1]))
continue;
@@ -2365,8 +2371,8 @@ static int analyze_subprog(struct bpf_verifier_env *env,
return -E2BIG;
}
- if (need_resched())
- cond_resched();
+ if (bpf_must_abort())
+ return -EINTR;
/*
* When an instance is reused (must_write_initialized == true),
@@ -2524,8 +2530,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
if (err)
return err;
}
- update_instance(env, instance);
- return 0;
+ return update_instance(env, instance);
out_free:
if (prev_instance)
@@ -2872,11 +2877,10 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
changed = false;
/* a cycle of calls takes a round for each link */
- if (signal_pending(current)) {
- err = -EAGAIN;
+ if (bpf_must_abort()) {
+ err = -EINTR;
goto out;
}
- cond_resched();
for (i = 0; i < env->cfg.cur_postorder; ++i) {
int insn_idx = env->cfg.insn_postorder[i];
struct insn_live_regs *live = &state[insn_idx];
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 7de1574af386..9946146847d7 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -2,7 +2,6 @@
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/slab.h>
-#include <linux/sched/signal.h>
#include <linux/bpf_verifier.h>
static bool is_cfg_jump(struct bpf_insn *insn)
@@ -107,7 +106,6 @@ static int compute_subprog_idoms(struct bpf_verifier_env *env, struct bpf_iarray
int po_last = (subprog + 1)->postorder_start - 1;
int *idoms = env->idoms;
int po_num, pred;
- u32 work = 0;
bool changed;
idoms[start] = start;
@@ -123,11 +121,8 @@ static int compute_subprog_idoms(struct bpf_verifier_env *env, struct bpf_iarray
int new_idom = -1;
iarray_for_each(pred, preds[idx]) {
- if (++work % 1024 == 0) {
- if (signal_pending(current))
- return -EAGAIN;
- cond_resched();
- }
+ if (bpf_must_abort())
+ return -EINTR;
if (idoms[pred] == -1)
continue;
@@ -416,24 +411,15 @@ static int compute_loops_in_subprog(struct bpf_verifier_env *env, struct loops_d
int *stack = dfs->stack;
int s, h, err, cur, stack_sz;
struct bpf_iarray *succ;
- u32 i;
stack[0] = start;
state[start].traversed = true;
state[start].next_succ = 0;
dfs_pos[start] = 1;
stack_sz = 1;
- i = 0;
do {
- /*
- * The algorithm should be very fast in practice,
- * guard against pathological inputs, just in case.
- */
- if ((++i % 1024) == 0) {
- if (signal_pending(current))
- return -EAGAIN;
- cond_resched();
- }
+ if (bpf_must_abort())
+ return -EINTR;
cur = stack[stack_sz - 1];
succ = bpf_insn_successors(env, cur);
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 5648beca3931..616e22c038ed 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2664,11 +2664,10 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header, i
idx = *min_heap_peek_inline(worklist);
min_heap_pop_inline(worklist, &worklist_cb, env->cfg.postorder_nums);
start = idx;
- if (signal_pending(current)) {
- err = -EAGAIN;
+ if (bpf_must_abort()) {
+ err = -EINTR;
goto out;
}
- cond_resched();
/* join_successor() maps nested-loop entries to their representative header. */
nested_loop = idx != cur_header ? aux[idx].loop : NULL;
@@ -3265,9 +3264,8 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
idx = postorder[i];
if (!aux[idx].need_scev)
continue;
- if (signal_pending(current))
- return -EAGAIN;
- cond_resched();
+ if (bpf_must_abort())
+ return -EINTR;
header = bpf_loop_at_index(env, idx);
err = aux[idx].loop
? compute_header_scevs(env, header)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 04a3b3c80c74..70caab9648d6 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -6604,7 +6604,14 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size,
SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size,
struct bpf_common_attr __user *, uattr_common, unsigned int, size_common)
{
- return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
+ int err = __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
+
+ cmd &= ~BPF_COMMON_ATTRS;
+ /* bpf_must_abort() ended the command, nothing of it is left */
+ if (err == -EINTR && task_sigpending(current) &&
+ (cmd == BPF_PROG_LOAD || cmd == BPF_BTF_LOAD || cmd == BPF_MAP_CREATE))
+ return -ERESTARTNOINTR;
+ return err;
}
static bool syscall_prog_is_valid_access(int off, int size,
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e4fcd1c9242d..03f8b2f33596 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21790,11 +21790,8 @@ static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
struct bpf_insn_aux_data *insn_aux;
int err;
- if (signal_pending(current))
- return -EAGAIN;
-
- if (need_resched())
- cond_resched();
+ if (bpf_must_abort())
+ return -EINTR;
/* no insn makes as many ids as there are from here to BPF_INHERITED_ID */
if (env->id_gen >= BPF_MAX_ID) {
@@ -22988,8 +22985,8 @@ static int resolve_func_ptrs(struct bpf_verifier_env *env)
continue;
for (off = 0; off + sizeof(u64) <= map->value_size; off += sizeof(u64)) {
- if (!(off % SZ_64K))
- cond_resched();
+ if (bpf_must_abort())
+ return -EINTR;
val = *(u64 *)(unsigned long)(addr + off);
if (!val || val % sizeof(struct bpf_insn) ||
val / sizeof(struct bpf_insn) >= insn_cnt)
@@ -25306,6 +25303,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
/* Serialize verification of unprivileged programs. */
if (!is_priv)
mutex_lock(&bpf_verifier_lock);
+ ret = -EINTR;
+ if (bpf_must_abort())
+ goto skip_full_check;
len = env->insn_aux_data_len = env->prog->len;
env->insn_aux_data =
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 02/21] bpf: Fail the load when there is no memory
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 16:06 ` bot+bpf-ci
2026-11-16 15:26 ` [PATCH bpf-next v2 03/21] bpf: Give a syscall program the largest context when it does tail calls Alexei Starovoitov
` (18 subsequent siblings)
20 siblings, 1 reply; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier allocates with GFP_KERNEL_ACCOUNT. At the limit of the memory
cgroup this wakes up the OOM killer, and the loader dies:
memory.max = 64M, program needs 120M
before: Killed
after: bpf() = -ENOMEM
Use one mask for all allocations of the verifier. Most of the patch is:
sed -i 's/GFP_KERNEL_ACCOUNT/BPF_VERIFIER_GFP/'
The rest:
- BPF_PROG_LOAD passes the flags to bpf_prog_alloc() and friends.
Classic BPF does not, except for the patch list of constant blinding.
- -ENOMEM in front of the JIT fails the load. There is no fall back to
the interpreter. A failed bpf_prog_alloc() gives -ENOMEM, not -EINVAL.
With __GFP_RETRY_MAYFAIL kvmalloc() may reclaim and compact for a table
of up to 4 MB before it uses vmalloc().
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 4 ++
kernel/bpf/cfg.c | 14 +++----
kernel/bpf/check_btf.c | 6 +--
kernel/bpf/const_fold.c | 4 +-
kernel/bpf/core.c | 20 +++++-----
kernel/bpf/diagnostics.c | 8 ++--
kernel/bpf/fixups.c | 22 +++++------
kernel/bpf/flow.c | 70 +++++++++++++++++-----------------
kernel/bpf/join.c | 2 +-
kernel/bpf/liveness.c | 26 ++++++-------
kernel/bpf/log.c | 2 +-
kernel/bpf/loops.c | 16 ++++----
kernel/bpf/scev.c | 22 +++++------
kernel/bpf/syscall.c | 6 +--
kernel/bpf/verifier.c | 74 ++++++++++++++++++------------------
15 files changed, 150 insertions(+), 146 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index ecb988ab6349..2f15b43074d1 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -13,6 +13,10 @@
#include <linux/freezer.h>
#include <linux/sched/signal.h>
+/* No memory fails the load. It does not wake up the OOM killer. */
+#define BPF_GFP_MAYFAIL (__GFP_RETRY_MAYFAIL | __GFP_NOWARN)
+#define BPF_VERIFIER_GFP (GFP_KERNEL_ACCOUNT | BPF_GFP_MAYFAIL)
+
/*
* For each loop that can run long. The caller returns -EINTR.
* killed the task dies
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 163fdb21828f..40aafceb139e 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -188,7 +188,7 @@ struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
size_t new_size = sizeof(struct bpf_iarray) + n_elem * sizeof(old->items[0]);
struct bpf_iarray *new;
- new = kvrealloc(old, new_size, GFP_KERNEL_ACCOUNT);
+ new = kvrealloc(old, new_size, BPF_VERIFIER_GFP);
if (!new) {
/* this is what callers always want, so simplify the call site */
kvfree(old);
@@ -663,12 +663,12 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
int ex_insn_beg, i, ret = 0;
insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!insn_state)
return -ENOMEM;
insn_stack = env->cfg.insn_stack = kvzalloc_objs(int, insn_cnt,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!insn_stack) {
kvfree(insn_state);
return -ENOMEM;
@@ -771,10 +771,10 @@ int bpf_compute_postorder(struct bpf_verifier_env *env)
struct dfs_state *state = NULL;
struct bpf_iarray *succ = NULL;
- postorder_nums = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
- postorder = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
- stack = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
- state = kvzalloc_objs(struct dfs_state, env->prog->len, GFP_KERNEL_ACCOUNT);
+ postorder_nums = kvzalloc_objs(int, env->prog->len, BPF_VERIFIER_GFP);
+ postorder = kvzalloc_objs(int, env->prog->len, BPF_VERIFIER_GFP);
+ stack = kvzalloc_objs(int, env->prog->len, BPF_VERIFIER_GFP);
+ state = kvzalloc_objs(struct dfs_state, env->prog->len, BPF_VERIFIER_GFP);
if (!postorder_nums || !postorder || !stack || !state) {
kvfree(postorder_nums);
kvfree(postorder);
diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c
index d6b930377d75..77dfb45708b6 100644
--- a/kernel/bpf/check_btf.c
+++ b/kernel/bpf/check_btf.c
@@ -63,7 +63,7 @@ static int prepare_btf_func(struct bpf_verifier_env *env,
urecord = make_bpfptr(attr->func_info, uattr.is_kernel);
min_size = min_t(u32, krec_size, urec_size);
- krecord = kvcalloc(nfuncs, krec_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ krecord = kvcalloc(nfuncs, krec_size, BPF_VERIFIER_GFP);
if (!krecord)
return -ENOMEM;
@@ -164,7 +164,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
krecord = prog->aux->func_info;
info_aux = kzalloc_objs(*info_aux, nfuncs,
- GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ BPF_VERIFIER_GFP);
if (!info_aux)
return -ENOMEM;
@@ -242,7 +242,7 @@ static int check_btf_line(struct bpf_verifier_env *env,
* pass in a smaller bpf_line_info object.
*/
linfo = kvzalloc_objs(struct bpf_line_info, nr_linfo,
- GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ BPF_VERIFIER_GFP);
if (!linfo)
return -ENOMEM;
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index 5869730547ae..1fd97f624b94 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -253,7 +253,7 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
int i, r, t;
/* kvzalloc zeroes memory, so all entries start as CONST_ARG_UNVISITED (0) */
- ci_in = kvzalloc_objs(*ci_in, insn_cnt, GFP_KERNEL_ACCOUNT);
+ ci_in = kvzalloc_objs(*ci_in, insn_cnt, BPF_VERIFIER_GFP);
if (!ci_in)
return -ENOMEM;
@@ -572,7 +572,7 @@ int bpf_compute_leave_conds(struct bpf_verifier_env *env)
bool jump;
u64 any;
- env->leave_conds = kvcalloc(env->prog->len, sizeof(*p), GFP_KERNEL_ACCOUNT);
+ env->leave_conds = kvcalloc(env->prog->len, sizeof(*p), BPF_VERIFIER_GFP);
if (!env->leave_conds)
return -ENOMEM;
for (i = 0; i < env->cfg.cur_postorder; i++) {
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index b407b2393ac3..09abfed83109 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -180,7 +180,7 @@ int bpf_prog_alloc_jited_linfo(struct bpf_prog *prog)
prog->aux->jited_linfo = kvzalloc_objs(*prog->aux->jited_linfo,
prog->aux->nr_linfo,
- bpf_memcg_flags(GFP_KERNEL | __GFP_NOWARN));
+ bpf_memcg_flags(GFP_KERNEL | BPF_GFP_MAYFAIL));
if (!prog->aux->jited_linfo)
return -ENOMEM;
@@ -309,7 +309,7 @@ int bpf_prog_calc_tag(struct bpf_prog *fp)
bool was_ld_map;
u32 i;
- dst = __vmalloc(size, GFP_KERNEL_ACCOUNT);
+ dst = __vmalloc(size, BPF_VERIFIER_GFP);
if (!dst)
return -ENOMEM;
@@ -349,7 +349,7 @@ static int patch_list_reserve_patch(struct bpf_patch_list *pl)
return 0;
cap = max(pl->cap * 2, 16U);
- p = kvrealloc(pl->patches, array_size(cap, sizeof(*p)), GFP_KERNEL_ACCOUNT);
+ p = kvrealloc(pl->patches, array_size(cap, sizeof(*p)), BPF_VERIFIER_GFP);
if (!p)
return -ENOMEM;
pl->patches = p;
@@ -366,7 +366,7 @@ static int patch_list_reserve_insns(struct bpf_patch_list *pl, u32 cnt)
return 0;
cap = max3(pl->insn_cap * 2, pl->insn_cnt + cnt, 64U);
- insns = kvrealloc(pl->insns, array_size(cap, sizeof(*insns)), GFP_KERNEL_ACCOUNT);
+ insns = kvrealloc(pl->insns, array_size(cap, sizeof(*insns)), BPF_VERIFIER_GFP);
if (!insns)
return -ENOMEM;
pl->insns = insns;
@@ -619,7 +619,7 @@ static int patch_image_map_linfo(struct bpf_patch_image *img, const struct bpf_p
const struct bpf_line_info *linfo = prog->aux->linfo;
u32 i, j, pos, next, nr = prog->aux->nr_linfo;
- img->new_linfo = kvmalloc_array(nr + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
+ img->new_linfo = kvmalloc_array(nr + 1, sizeof(u32), BPF_VERIFIER_GFP);
if (!img->new_linfo)
return -ENOMEM;
img->nr_linfo = nr;
@@ -656,8 +656,8 @@ int bpf_patch_list_build(const struct bpf_patch_list *pl, const struct bpf_prog
memset(img, 0, sizeof(*img));
img->cnt = cnt;
- img->new_off = kvmalloc_array(cnt + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
- img->new_tgt = kvmalloc_array(cnt + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
+ img->new_off = kvmalloc_array(cnt + 1, sizeof(u32), BPF_VERIFIER_GFP);
+ img->new_tgt = kvmalloc_array(cnt + 1, sizeof(u32), BPF_VERIFIER_GFP);
if (!img->new_off || !img->new_tgt)
return -ENOMEM;
@@ -678,7 +678,7 @@ int bpf_patch_list_build(const struct bpf_patch_list *pl, const struct bpf_prog
return err;
}
- img->insns = kvmalloc_array(img->new_cnt, sizeof(*img->insns), GFP_KERNEL_ACCOUNT);
+ img->insns = kvmalloc_array(img->new_cnt, sizeof(*img->insns), BPF_VERIFIER_GFP);
if (!img->insns)
return -ENOMEM;
@@ -3017,10 +3017,10 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
prog = bpf_jit_blind_constants(env, prog);
/*
* Fall back to the interpreter after blinding failures, except when
- * the loader was killed.
+ * the load has to stop or has no memory.
*/
if (IS_ERR(prog)) {
- if (PTR_ERR(prog) == -EINTR)
+ if (PTR_ERR(prog) == -EINTR || (env && PTR_ERR(prog) == -ENOMEM))
return prog;
goto out_restore;
}
diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
index 3de3e1486bbd..5783fa20f568 100644
--- a/kernel/bpf/diagnostics.c
+++ b/kernel/bpf/diagnostics.c
@@ -234,7 +234,7 @@ int bpf_diag_init(struct bpf_verifier_env *env)
if (!bpf_diag_enabled(env))
return 0;
- env->diag = kzalloc_obj(struct bpf_diag, GFP_KERNEL_ACCOUNT);
+ env->diag = kzalloc_obj(struct bpf_diag, BPF_VERIFIER_GFP);
if (!env->diag)
return -ENOMEM;
@@ -266,7 +266,7 @@ static char *diag_fmt_alloc(struct bpf_verifier_env *env, size_t size)
}
capacity = max_t(size_t, BPF_DIAG_FMT_CHUNK_SIZE, size);
- chunk = kmalloc_flex(*chunk, data, capacity, GFP_KERNEL_ACCOUNT);
+ chunk = kmalloc_flex(*chunk, data, capacity, BPF_VERIFIER_GFP);
if (!chunk)
return NULL;
@@ -456,7 +456,7 @@ static void diag_append_history(struct bpf_verifier_env *env,
goto rotate;
cap = min(log->cap ? log->cap * 2 : 64, max_events);
- events = kvrealloc(log->events, array_size(cap, sizeof(*events)), GFP_KERNEL_ACCOUNT);
+ events = kvrealloc(log->events, array_size(cap, sizeof(*events)), BPF_VERIFIER_GFP);
if (!events) {
log->growth_failed = true;
goto rotate;
@@ -542,7 +542,7 @@ static void diag_vprint_indented(struct bpf_verifier_env *env, const char *fmt,
if (!bpf_diag_enabled(env))
return;
- buf = kvasprintf(GFP_KERNEL_ACCOUNT, fmt, args);
+ buf = kvasprintf(BPF_VERIFIER_GFP, fmt, args);
if (!buf) {
diag_write(env, "%s<failed to allocate diagnostic text>\n", BPF_DIAG_TEXT_INDENT);
return;
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 735932392a87..8d1dd0770711 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -507,11 +507,11 @@ int bpf_patch_list_commit(struct bpf_verifier_env *env)
err = -ENOMEM;
data = __vmalloc(array_size(img.new_cnt, sizeof(*data)),
- GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+ BPF_VERIFIER_GFP | __GFP_ZERO);
if (!data)
goto out;
- prog = bpf_prog_realloc(prog, bpf_prog_size(img.new_cnt), GFP_USER);
+ prog = bpf_prog_realloc(prog, bpf_prog_size(img.new_cnt), GFP_USER | BPF_GFP_MAYFAIL);
if (!prog)
goto out;
env->prog = prog;
@@ -651,7 +651,7 @@ int bpf_insert_loop_guards(struct bpf_verifier_env *env)
* Where the subprog has 'r0 = 0; exit', 0 if it has none. And how far that moved
* the subprog: guard_to is from the time before.
*/
- leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
+ leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), BPF_VERIFIER_GFP);
if (!leave)
return -ENOMEM;
moved = leave + env->subprog_cnt;
@@ -689,9 +689,9 @@ int bpf_insert_loop_guards(struct bpf_verifier_env *env)
}
/* where the guard of the insn goes */
- target = kvcalloc(len, sizeof(int), GFP_KERNEL_ACCOUNT);
- zero = kvcalloc(len, sizeof(u16), GFP_KERNEL_ACCOUNT);
- patch = kvcalloc(8, sizeof(*patch), GFP_KERNEL_ACCOUNT);
+ target = kvcalloc(len, sizeof(int), BPF_VERIFIER_GFP);
+ zero = kvcalloc(len, sizeof(u16), BPF_VERIFIER_GFP);
+ patch = kvcalloc(8, sizeof(*patch), BPF_VERIFIER_GFP);
if (!target || !zero || !patch) {
err = -ENOMEM;
goto out;
@@ -1259,7 +1259,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
unsigned long *removed;
int i, next, err = 0;
- removed = kvcalloc(BITS_TO_LONGS(insn_cnt), sizeof(long), GFP_KERNEL_ACCOUNT);
+ removed = kvcalloc(BITS_TO_LONGS(insn_cnt), sizeof(long), BPF_VERIFIER_GFP);
if (!removed)
return -ENOMEM;
@@ -1768,7 +1768,7 @@ static u32 *bpf_dup_subprog_starts(struct bpf_verifier_env *env)
{
u32 *starts = NULL;
- starts = kvmalloc_objs(u32, env->subprog_cnt, GFP_KERNEL_ACCOUNT);
+ starts = kvmalloc_objs(u32, env->subprog_cnt, BPF_VERIFIER_GFP);
if (starts) {
for (int i = 0; i < env->subprog_cnt; i++)
starts[i] = env->subprog_info[i].start;
@@ -1902,7 +1902,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
* subprogs don't have IDs and not reachable via prog_get_next_id
* func[i]->stats will never be accessed and stays NULL
*/
- func[i] = bpf_prog_alloc_no_stats(bpf_prog_size(len), GFP_USER);
+ func[i] = bpf_prog_alloc_no_stats(bpf_prog_size(len), GFP_USER | BPF_GFP_MAYFAIL);
if (!func[i])
goto out_free;
memcpy(func[i]->insnsi, &prog->insnsi[subprog_start],
@@ -2215,7 +2215,7 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
err = bpf_jit_subprogs(env);
if (err == 0)
return 0;
- if (err == -EFAULT || err == -EINTR)
+ if (err == -EFAULT || err == -EINTR || err == -ENOMEM)
return err;
}
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
@@ -2859,7 +2859,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
if (!tmg) {
tmg = alloc_percpu_gfp(struct bpf_timed_may_goto,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!tmg)
return -ENOMEM;
prog->aux->timed_may_goto = tmg;
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index fbd1178eb9f5..70766ee48eb4 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -204,11 +204,11 @@ static int flow_compute_order(struct bpf_verifier_env *env, struct bpf_flow *flo
int err = -ENOMEM;
/* lists of members are indexed by loop head, len is the function */
- next = kvmalloc_array(len, sizeof(int), GFP_KERNEL_ACCOUNT);
- head = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
- tail = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
- stack = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
- linked = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+ next = kvmalloc_array(len, sizeof(int), BPF_VERIFIER_GFP);
+ head = kvmalloc_array(len + 1, sizeof(int), BPF_VERIFIER_GFP);
+ tail = kvmalloc_array(len + 1, sizeof(int), BPF_VERIFIER_GFP);
+ stack = kvmalloc_array(len + 1, sizeof(int), BPF_VERIFIER_GFP);
+ linked = kvcalloc(BITS_TO_LONGS(len), sizeof(long), BPF_VERIFIER_GFP);
if (!next || !head || !tail || !stack || !linked)
goto out;
@@ -478,7 +478,7 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)
struct bpf_loop *loop;
u32 *seen;
- seen = kvcalloc(len, sizeof(*seen), GFP_KERNEL_ACCOUNT);
+ seen = kvcalloc(len, sizeof(*seen), BPF_VERIFIER_GFP);
if (!seen)
return -ENOMEM;
@@ -859,8 +859,8 @@ int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
bool lost = true;
u16 *ret;
- kinds = kvmalloc_array(len, sizeof(*kinds), GFP_KERNEL_ACCOUNT);
- ret = kvmalloc_array(env->subprog_cnt, sizeof(*ret), GFP_KERNEL_ACCOUNT);
+ kinds = kvmalloc_array(len, sizeof(*kinds), BPF_VERIFIER_GFP);
+ ret = kvmalloc_array(env->subprog_cnt, sizeof(*ret), BPF_VERIFIER_GFP);
if (!kinds || !ret) {
kvfree(kinds);
kvfree(ret);
@@ -1015,12 +1015,12 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
u16 *gen, callx_regs = 0, pair = BIT(BPF_REG_0) | BIT(BPF_REG_2);
u8 *ret, class, op;
- need = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
- gen = kvcalloc(env->prog->len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+ need = kvcalloc(env->prog->len, sizeof(*need), BPF_VERIFIER_GFP);
+ gen = kvcalloc(env->prog->len, sizeof(u16), BPF_VERIFIER_GFP);
/* the callers need what the subprog returns: r0, and r2 of a pair */
- ret = kvcalloc(env->subprog_cnt, sizeof(u8), GFP_KERNEL_ACCOUNT);
+ ret = kvcalloc(env->subprog_cnt, sizeof(u8), BPF_VERIFIER_GFP);
if (flag)
- ahead = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
+ ahead = kvcalloc(env->prog->len, sizeof(*need), BPF_VERIFIER_GFP);
*pneed = need;
if (!need || !gen || !ret || (flag && !ahead)) {
kvfree(ahead);
@@ -1485,9 +1485,9 @@ static int flow_compute_loop_rungs(struct bpf_verifier_env *env, const u32 *at,
/* for each insn, for the current insn, for each jump back */
syms = kvmalloc_array((size_t)nr + 1 + loop->backedges_cnt, n * sizeof(*syms),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
/* for each root, and the result for each place */
- roots = kvcalloc(3 * n, sizeof(*roots), GFP_KERNEL_ACCOUNT);
+ roots = kvcalloc(3 * n, sizeof(*roots), BPF_VERIFIER_GFP);
if (!syms || !roots)
goto out;
memset(syms, 0xff, (size_t)nr * n * sizeof(*syms));
@@ -1566,7 +1566,7 @@ static int flow_compute_loop_rungs(struct bpf_verifier_env *env, const u32 *at,
err = 0;
if (!cnt)
goto out;
- loop->rungs = kvcalloc(cnt, sizeof(*loop->rungs), GFP_KERNEL_ACCOUNT);
+ loop->rungs = kvcalloc(cnt, sizeof(*loop->rungs), BPF_VERIFIER_GFP);
if (!loop->rungs) {
err = -ENOMEM;
goto out;
@@ -1595,7 +1595,7 @@ static int flow_compute_rungs(struct bpf_verifier_env *env)
;
if (h == env->prog->len)
return 0;
- at = kvmalloc_array(total, sizeof(*at), GFP_KERNEL_ACCOUNT);
+ at = kvmalloc_array(total, sizeof(*at), BPF_VERIFIER_GFP);
if (!at)
return -ENOMEM;
for (k = 0; k < total; k++) {
@@ -1636,22 +1636,22 @@ int bpf_flow_prepare(struct bpf_verifier_env *env)
struct bpf_flow *flow;
u8 *preds;
- flow = kzalloc_obj(*flow, GFP_KERNEL_ACCOUNT);
+ flow = kzalloc_obj(*flow, BPF_VERIFIER_GFP);
if (!flow)
return -ENOMEM;
env->flow = flow;
INIT_LIST_HEAD(&flow->kept);
flow->pending = RB_ROOT_CACHED;
- flow->pos = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
- flow->rank = kvcalloc(env->subprog_cnt + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
- flow->point = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
- flow->keep = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
- flow->states = kvmalloc_array(len, sizeof(*flow->states), GFP_KERNEL_ACCOUNT);
- flow->enter = kvmalloc_array(len, sizeof(*flow->enter), GFP_KERNEL_ACCOUNT);
- flow->nr_states = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
- flow->funcs = kvcalloc(env->subprog_cnt + 1, sizeof(*flow->funcs), GFP_KERNEL_ACCOUNT);
- preds = kvcalloc(len, 1, GFP_KERNEL_ACCOUNT);
+ flow->pos = kvcalloc(len, sizeof(u32), BPF_VERIFIER_GFP);
+ flow->rank = kvcalloc(env->subprog_cnt + 1, sizeof(u32), BPF_VERIFIER_GFP);
+ flow->point = kvcalloc(BITS_TO_LONGS(len), sizeof(long), BPF_VERIFIER_GFP);
+ flow->keep = kvcalloc(BITS_TO_LONGS(len), sizeof(long), BPF_VERIFIER_GFP);
+ flow->states = kvmalloc_array(len, sizeof(*flow->states), BPF_VERIFIER_GFP);
+ flow->enter = kvmalloc_array(len, sizeof(*flow->enter), BPF_VERIFIER_GFP);
+ flow->nr_states = kvcalloc(len, sizeof(u32), BPF_VERIFIER_GFP);
+ flow->funcs = kvcalloc(env->subprog_cnt + 1, sizeof(*flow->funcs), BPF_VERIFIER_GFP);
+ preds = kvcalloc(len, 1, BPF_VERIFIER_GFP);
if (!flow->pos || !flow->rank || !flow->point || !flow->keep || !flow->states ||
!flow->enter ||
!flow->nr_states || !flow->funcs || !preds) {
@@ -1808,7 +1808,7 @@ bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx)
struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
const struct bpf_verifier_state *st)
{
- struct flow_kept *k = kzalloc_obj(*k, GFP_KERNEL_ACCOUNT);
+ struct flow_kept *k = kzalloc_obj(*k, BPF_VERIFIER_GFP);
if (!k)
return NULL;
@@ -1915,13 +1915,13 @@ static int flow_entry_new(struct bpf_flow *flow)
if (flow->entry_cnt + 1 >= flow->entry_cap) {
cap = max(64U, flow->entry_cap * 2);
- entries = kvrealloc(flow->entries, cap * sizeof(*entries), GFP_KERNEL_ACCOUNT);
+ entries = kvrealloc(flow->entries, cap * sizeof(*entries), BPF_VERIFIER_GFP);
if (!entries)
return -ENOMEM;
flow->entries = entries;
flow->entry_cap = cap;
}
- e = kmalloc_obj(*e, GFP_KERNEL_ACCOUNT);
+ e = kmalloc_obj(*e, BPF_VERIFIER_GFP);
if (!e)
return -ENOMEM;
e->up = e;
@@ -1941,7 +1941,7 @@ static int flow_unite(struct bpf_flow *flow, u32 a, u32 b)
return 0;
if (flow->united_cnt == flow->united_cap) {
cap = max(16U, flow->united_cap * 2);
- u = kvrealloc(flow->united, cap * sizeof(*u), GFP_KERNEL_ACCOUNT);
+ u = kvrealloc(flow->united, cap * sizeof(*u), BPF_VERIFIER_GFP);
if (!u)
return -ENOMEM;
flow->united = u;
@@ -2881,7 +2881,7 @@ static int flow_merge(struct bpf_verifier_env *env, struct flow_set *set,
static struct flow_state *flow_new_state(struct bpf_verifier_env *env, int prev_insn_idx)
{
- struct flow_state *s = kzalloc_obj(*s, GFP_KERNEL_ACCOUNT);
+ struct flow_state *s = kzalloc_obj(*s, BPF_VERIFIER_GFP);
struct bpf_flow *flow = env->flow;
if (!s)
@@ -3206,7 +3206,7 @@ int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *calle
struct flow_state *w;
int id;
- w = kzalloc_obj(*w, GFP_KERNEL_ACCOUNT);
+ w = kzalloc_obj(*w, BPF_VERIFIER_GFP);
if (!w)
return -ENOMEM;
w->st = *caller;
@@ -3271,7 +3271,7 @@ int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, i
kfree(o);
}
- x = kzalloc_obj(*x, GFP_KERNEL_ACCOUNT);
+ x = kzalloc_obj(*x, BPF_VERIFIER_GFP);
if (!x)
return -ENOMEM;
x->st = *st;
@@ -3368,9 +3368,9 @@ void bpf_flow_print_way(struct bpf_verifier_env *env, u64 log_pos, int head, int
u32 from = flow->from, cnt = 0;
unsigned long *seen;
- seen = bitmap_zalloc(flow->len, GFP_KERNEL_ACCOUNT);
+ seen = bitmap_zalloc(flow->len, BPF_VERIFIER_GFP);
way = kvmalloc_array(bitmap_weight(flow->point, flow->len), sizeof(*way),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!seen || !way)
goto out;
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 67211f329907..6a8efb671a8a 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -46,7 +46,7 @@ struct bpf_join_state {
static struct bpf_join_state *join_state(struct bpf_verifier_env *env)
{
if (!env->join) {
- env->join = kvzalloc_obj(*env->join, GFP_KERNEL_ACCOUNT);
+ env->join = kvzalloc_obj(*env->join, BPF_VERIFIER_GFP);
if (env->join)
env->join->env = env;
}
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 2c771b936d65..053b3875cde6 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -121,7 +121,7 @@ static struct func_instance *call_instance(struct bpf_verifier_env *env,
if (f)
return f;
- f = kvzalloc_obj(*f, GFP_KERNEL_ACCOUNT);
+ f = kvzalloc_obj(*f, BPF_VERIFIER_GFP);
if (!f)
return ERR_PTR(-ENOMEM);
f->callsite = lookup_key;
@@ -155,7 +155,7 @@ static struct func_instance *lookup_instance(struct bpf_verifier_env *env,
int bpf_stack_liveness_init(struct bpf_verifier_env *env)
{
- env->liveness = kvzalloc_obj(*env->liveness, GFP_KERNEL_ACCOUNT);
+ env->liveness = kvzalloc_obj(*env->liveness, BPF_VERIFIER_GFP);
if (!env->liveness)
return -ENOMEM;
hash_init(env->liveness->func_instances);
@@ -216,7 +216,7 @@ static struct frame_masks *widen_frame_masks(struct func_instance *instance,
if (old && old->words >= words)
return old;
- new = kvzalloc_flex(*new, bits, frame_mask_words(instance, words), GFP_KERNEL_ACCOUNT);
+ new = kvzalloc_flex(*new, bits, frame_mask_words(instance, words), BPF_VERIFIER_GFP);
if (!new)
return NULL;
new->words = words;
@@ -815,7 +815,7 @@ static int print_instances(struct bpf_verifier_env *env)
cnt = 0;
hash_for_each(liveness->func_instances, bkt, instance, hl_node)
cnt++;
- sorted_instances = kvmalloc_objs(*sorted_instances, cnt, GFP_KERNEL_ACCOUNT);
+ sorted_instances = kvmalloc_objs(*sorted_instances, cnt, BPF_VERIFIER_GFP);
if (!sorted_instances)
return -ENOMEM;
cnt = 0;
@@ -2081,15 +2081,15 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
bool changed;
int i, p, r, err = -ENOMEM;
- at_in = kvmalloc_objs(*at_in, len, GFP_KERNEL_ACCOUNT);
+ at_in = kvmalloc_objs(*at_in, len, BPF_VERIFIER_GFP);
if (!at_in)
goto err_free;
- at_stack_in = kvmalloc_objs(*at_stack_in, (size_t)len * nslots, GFP_KERNEL_ACCOUNT);
+ at_stack_in = kvmalloc_objs(*at_stack_in, (size_t)len * nslots, BPF_VERIFIER_GFP);
if (!at_stack_in)
goto err_free;
- at_stack_out = kvmalloc_objs(*at_stack_out, nslots, GFP_KERNEL_ACCOUNT);
+ at_stack_out = kvmalloc_objs(*at_stack_out, nslots, BPF_VERIFIER_GFP);
if (!at_stack_out)
goto err_free;
@@ -2190,7 +2190,7 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
struct spill_snapshot *snap;
kvfree(env->callsite_at_stack[idx]);
- env->callsite_at_stack[idx] = kvmalloc_flex(*snap, at, nslots, GFP_KERNEL_ACCOUNT);
+ env->callsite_at_stack[idx] = kvmalloc_flex(*snap, at, nslots, BPF_VERIFIER_GFP);
snap = env->callsite_at_stack[idx];
if (!snap) {
err = -ENOMEM;
@@ -2321,7 +2321,7 @@ static struct func_instance *fresh_instance(struct func_instance *src)
{
struct func_instance *f;
- f = kvzalloc_obj(*f, GFP_KERNEL_ACCOUNT);
+ f = kvzalloc_obj(*f, BPF_VERIFIER_GFP);
if (!f)
return ERR_PTR(-ENOMEM);
f->callsite = src->callsite;
@@ -2546,15 +2546,15 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)
struct subprog_at_info *info;
int k, err = 0;
- info = kvzalloc_objs(*info, env->subprog_cnt, GFP_KERNEL_ACCOUNT);
+ info = kvzalloc_objs(*info, env->subprog_cnt, BPF_VERIFIER_GFP);
if (!info)
return -ENOMEM;
env->callsite_at_stack = kvzalloc_objs(*env->callsite_at_stack, insn_cnt,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
env->liveness->callee_args = kvcalloc(MAX_CALL_FRAMES * MAX_AT_TRACK_REGS,
sizeof(*env->liveness->callee_args),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!env->callsite_at_stack || !env->liveness->callee_args) {
err = -ENOMEM;
insn_cnt = env->callsite_at_stack ? insn_cnt : 0;
@@ -2839,7 +2839,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
* - repeat the computation while {in,out} fields changes for
* any instruction.
*/
- state = kvzalloc_objs(*state, insn_cnt, GFP_KERNEL_ACCOUNT);
+ state = kvzalloc_objs(*state, insn_cnt, BPF_VERIFIER_GFP);
if (!state) {
err = -ENOMEM;
goto out;
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index ffbec587a916..d5132e08ecd9 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -195,7 +195,7 @@ int bpf_vlog_stash(struct bpf_verifier_log *log, u64 pos, struct bpf_vlog_stash
stash->lost_end = log->end_pos - to;
stash->len = to - from;
/* as large as the buffer at worst: fail before the OOM killer is called */
- stash->buf = kvzalloc(stash->len, GFP_KERNEL_ACCOUNT | __GFP_NOWARN | __GFP_RETRY_MAYFAIL);
+ stash->buf = kvzalloc(stash->len, BPF_VERIFIER_GFP);
if (!stash->buf)
return -ENOMEM;
/* two pieces if it wraps around */
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 9946146847d7..434c5acfc2e1 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -28,7 +28,7 @@ static struct bpf_iarray **compute_predecessors(struct bpf_verifier_env *env)
size_t sz;
void *tmp;
- num_preds = kvzalloc_objs(u32, prog->len, GFP_KERNEL_ACCOUNT);
+ num_preds = kvzalloc_objs(u32, prog->len, BPF_VERIFIER_GFP);
if (!num_preds)
return NULL;
@@ -49,7 +49,7 @@ static struct bpf_iarray **compute_predecessors(struct bpf_verifier_env *env)
i++;
}
- result = kvzalloc(sz, GFP_KERNEL_ACCOUNT);
+ result = kvzalloc(sz, BPF_VERIFIER_GFP);
if (!result) {
kvfree(num_preds);
return NULL;
@@ -151,7 +151,7 @@ int bpf_compute_idoms(struct bpf_verifier_env *env)
if (!preds)
return -ENOMEM;
- idoms = kvzalloc_objs(*idoms, len, GFP_KERNEL_ACCOUNT);
+ idoms = kvzalloc_objs(*idoms, len, BPF_VERIFIER_GFP);
if (!idoms) {
kvfree(preds);
return -ENOMEM;
@@ -212,7 +212,7 @@ static int add_exit(struct bpf_loop *loop, int from, int to)
}
if (!loop->exits) {
loop->exits = kvzalloc_objs(*loop->exits, MAX_LOOP_EXITS,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!loop->exits)
return -ENOMEM;
}
@@ -229,7 +229,7 @@ static int mark_as_header(struct bpf_verifier_env *env, int h)
struct bpf_insn_aux_data *aux = env->insn_aux_data;
if (!aux[h].loop) {
- aux[h].loop = kvzalloc_obj(struct bpf_loop, GFP_KERNEL_ACCOUNT);
+ aux[h].loop = kvzalloc_obj(struct bpf_loop, BPF_VERIFIER_GFP);
if (!aux[h].loop)
return -ENOMEM;
}
@@ -542,9 +542,9 @@ int bpf_compute_loops(struct bpf_verifier_env *env)
struct bpf_iarray *succ;
struct bpf_loop *loop;
- dfs.dfs_pos = kvzalloc_objs(int, len, GFP_KERNEL_ACCOUNT);
- dfs.state = kvzalloc_objs(struct dfs_state, len, GFP_KERNEL_ACCOUNT);
- dfs.stack = kvzalloc_objs(int, len, GFP_KERNEL_ACCOUNT);
+ dfs.dfs_pos = kvzalloc_objs(int, len, BPF_VERIFIER_GFP);
+ dfs.state = kvzalloc_objs(struct dfs_state, len, BPF_VERIFIER_GFP);
+ dfs.stack = kvzalloc_objs(int, len, BPF_VERIFIER_GFP);
if (!dfs.dfs_pos || !dfs.state || !dfs.stack) {
err = -ENOMEM;
goto out;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 616e22c038ed..5218ce1cafc1 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -275,7 +275,7 @@ static void exprs_ht_add(struct scev *scev, u32 id)
/* @cap is a power of two. No warning for a size that kvmalloc() rejects. */
static int grow_exprs(struct scev *scev, u32 cap)
{
- gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_NOWARN;
+ gfp_t gfp = BPF_VERIFIER_GFP;
void *tmp;
u32 id;
@@ -957,11 +957,11 @@ static struct env *get_loop_env(struct scev *scev, int loop_header, int insn_idx
if (e)
return e;
- e = kzalloc_obj(*e, GFP_KERNEL_ACCOUNT);
+ e = kzalloc_obj(*e, BPF_VERIFIER_GFP);
if (!e)
return NULL;
- tmp = krealloc(envs, struct_size(envs, entries, cnt + 1), GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+ tmp = krealloc(envs, struct_size(envs, entries, cnt + 1), BPF_VERIFIER_GFP | __GFP_ZERO);
if (!tmp) {
kfree(e);
return NULL;
@@ -1812,7 +1812,7 @@ static int worklist_push(struct bpf_verifier_env *env, int loop_header, int idx)
int *data;
data = kvrealloc(worklist->data, array_size(size, sizeof(*data)),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!data)
return -ENOMEM;
worklist->data = data;
@@ -2300,7 +2300,7 @@ static int grow_array(void *p, u32 cnt, u32 *cap, size_t size)
if (cnt < *cap)
return 0;
- tmp = kvrealloc(*(void **)p, array_size(new_cap, size), GFP_KERNEL_ACCOUNT);
+ tmp = kvrealloc(*(void **)p, array_size(new_cap, size), BPF_VERIFIER_GFP);
if (!tmp)
return -ENOMEM;
*(void **)p = tmp;
@@ -2567,7 +2567,7 @@ static int find_linked_regs(struct bpf_verifier_env *env, struct env *e, int h,
q->refine = test_bit(k, roots);
}
if (!envs) {
- envs = kzalloc(sizeof(*envs), GFP_KERNEL_ACCOUNT);
+ envs = kzalloc(sizeof(*envs), BPF_VERIFIER_GFP);
if (!envs)
return -ENOMEM;
scev->envs[idx] = envs;
@@ -2634,11 +2634,11 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header, i
if (!cur_loop && !can_compute_subprog_scev(env, sub))
return 0;
- cur_env = kzalloc_obj(*cur_env, GFP_KERNEL_ACCOUNT);
+ cur_env = kzalloc_obj(*cur_env, BPF_VERIFIER_GFP);
if (!cur_env)
goto nomem;
if (log_level2) {
- old_env = kzalloc_obj(*old_env, GFP_KERNEL_ACCOUNT);
+ old_env = kzalloc_obj(*old_env, BPF_VERIFIER_GFP);
if (!old_env)
goto nomem;
}
@@ -3351,7 +3351,7 @@ int bpf_init_scev(struct bpf_verifier_env *env)
struct scev *scev;
u32 r;
- scev = kzalloc_obj(struct scev, GFP_KERNEL_ACCOUNT);
+ scev = kzalloc_obj(struct scev, BPF_VERIFIER_GFP);
if (!scev)
return -ENOMEM;
env->scev = scev;
@@ -3369,9 +3369,9 @@ int bpf_init_scev(struct bpf_verifier_env *env)
set_expr_base(scev, &scev->exprs[r], r);
scev->exprs_cnt = REG_EXPR_ID(REGS_NUM);
scev->envs = kvzalloc_objs(*scev->envs, env->prog->len,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
scev->discovered = kvzalloc_objs(*scev->discovered, env->prog->len,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!scev->envs || !scev->discovered)
goto nomem;
/*
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 70caab9648d6..eda5d8ee8e3f 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3074,13 +3074,13 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
}
/* plain bpf_prog allocation */
- prog = bpf_prog_alloc(bpf_prog_size(attr->insn_cnt), GFP_USER);
+ prog = bpf_prog_alloc(bpf_prog_size(attr->insn_cnt), GFP_USER | BPF_GFP_MAYFAIL);
if (!prog) {
if (dst_prog)
bpf_prog_put(dst_prog);
if (attach_btf)
btf_put(attach_btf);
- err = -EINVAL;
+ err = -ENOMEM;
goto put_token;
}
@@ -3103,7 +3103,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
prog->aux->user = get_current_user();
prog->len = attr->insn_cnt;
- err = bpf_prog_stream_init(prog, GFP_USER);
+ err = bpf_prog_stream_init(prog, GFP_USER | BPF_GFP_MAYFAIL);
if (err)
goto free_prog;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 03f8b2f33596..d64719c85e6f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1417,7 +1417,7 @@ static void *realloc_array(void *arr, size_t old_n, size_t new_n, size_t size)
goto out;
alloc_size = kmalloc_size_roundup(size_mul(new_n, size));
- new_arr = krealloc(arr, alloc_size, GFP_KERNEL_ACCOUNT);
+ new_arr = krealloc(arr, alloc_size, BPF_VERIFIER_GFP);
if (!new_arr) {
kfree(arr);
return NULL;
@@ -1434,7 +1434,7 @@ static void *realloc_array(void *arr, size_t old_n, size_t new_n, size_t size)
static int copy_reference_state(struct bpf_verifier_state *dst, const struct bpf_verifier_state *src)
{
dst->refs = copy_array(dst->refs, src->refs, src->acquired_refs,
- sizeof(struct bpf_reference_state), GFP_KERNEL_ACCOUNT);
+ sizeof(struct bpf_reference_state), BPF_VERIFIER_GFP);
if (!dst->refs)
return -ENOMEM;
@@ -1453,7 +1453,7 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st
size_t n = bpf_stack_nr_slots(src);
dst->stack = copy_array(dst->stack, src->stack, n, sizeof(struct bpf_stack_state),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!dst->stack)
return -ENOMEM;
@@ -1464,7 +1464,7 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st
if (n) {
dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n,
sizeof(struct bpf_reg_state),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!dst->stack_arg_regs)
return -ENOMEM;
}
@@ -1765,7 +1765,7 @@ static int copy_func_state(struct bpf_func_state *dst,
/* Instruction accounting is path-local, not part of verifier state. */
dst->insns_subtotal = 0;
dst->loop_stack = copy_array(dst->loop_stack, src->loop_stack, src->loop_stack_cnt,
- sizeof(*src->loop_stack), GFP_KERNEL_ACCOUNT);
+ sizeof(*src->loop_stack), BPF_VERIFIER_GFP);
if (!dst->loop_stack)
return -ENOMEM;
return copy_stack_state(dst, src);
@@ -1797,7 +1797,7 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
for (i = 0; i <= src->curframe; i++) {
dst = dst_state->frame[i];
if (!dst) {
- dst = kzalloc_obj(*dst, GFP_KERNEL_ACCOUNT);
+ dst = kzalloc_obj(*dst, BPF_VERIFIER_GFP);
if (!dst)
return -ENOMEM;
dst_state->frame[i] = dst;
@@ -1829,7 +1829,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
struct bpf_verifier_stack_elem *elem;
int err;
- elem = kzalloc_obj(struct bpf_verifier_stack_elem, GFP_KERNEL_ACCOUNT);
+ elem = kzalloc_obj(struct bpf_verifier_stack_elem, BPF_VERIFIER_GFP);
if (!elem)
return ERR_PTR(-ENOMEM);
@@ -2544,7 +2544,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
struct bpf_verifier_stack_elem *elem;
struct bpf_func_state *frame;
- elem = kzalloc_obj(struct bpf_verifier_stack_elem, GFP_KERNEL_ACCOUNT);
+ elem = kzalloc_obj(struct bpf_verifier_stack_elem, BPF_VERIFIER_GFP);
if (!elem)
return ERR_PTR(-ENOMEM);
@@ -2558,7 +2558,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
* Initialize it similar to do_check_common().
*/
elem->st.in_sleepable = is_sleepable;
- frame = kzalloc_obj(*frame, GFP_KERNEL_ACCOUNT);
+ frame = kzalloc_obj(*frame, BPF_VERIFIER_GFP);
if (!frame)
return ERR_PTR(-ENOMEM);
init_func_state(env, frame,
@@ -3057,7 +3057,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
return -EINVAL;
}
- tab = kzalloc_obj(*tab, GFP_KERNEL_ACCOUNT);
+ tab = kzalloc_obj(*tab, BPF_VERIFIER_GFP);
if (!tab)
return -ENOMEM;
prog_aux->kfunc_tab = tab;
@@ -3075,7 +3075,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
return 0;
if (!btf_tab && offset) {
- btf_tab = kzalloc_obj(*btf_tab, GFP_KERNEL_ACCOUNT);
+ btf_tab = kzalloc_obj(*btf_tab, BPF_VERIFIER_GFP);
if (!btf_tab)
return -ENOMEM;
prog_aux->kfunc_btf_tab = btf_tab;
@@ -3133,7 +3133,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
meta.func_name = kfunc.name;
meta.kfunc_flags = kfunc.flags ? *kfunc.flags : 0;
- tab = krealloc(tab, struct_size(tab, descs, tab->nr_descs + 1), GFP_KERNEL_ACCOUNT);
+ tab = krealloc(tab, struct_size(tab, descs, tab->nr_descs + 1), BPF_VERIFIER_GFP);
if (!tab)
return -ENOMEM;
prog_aux->kfunc_tab = tab;
@@ -3369,7 +3369,7 @@ static int record_callx_edge(struct bpf_verifier_env *env, int caller, int calle
if (!env->callx_edges) {
env->callx_edges = kvcalloc(BITS_TO_LONGS(cnt * cnt), sizeof(long),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!env->callx_edges)
return -ENOMEM;
}
@@ -3464,8 +3464,8 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env)
int i, ret = 0;
u8 *color = NULL;
- color = kvzalloc_objs(*color, cnt, GFP_KERNEL_ACCOUNT);
- dfs_stack = kvmalloc_objs(*dfs_stack, cnt, GFP_KERNEL_ACCOUNT);
+ color = kvzalloc_objs(*color, cnt, BPF_VERIFIER_GFP);
+ dfs_stack = kvmalloc_objs(*dfs_stack, cnt, BPF_VERIFIER_GFP);
if (!color || !dfs_stack) {
ret = -ENOMEM;
goto out;
@@ -5567,7 +5567,7 @@ static int build_call_graph(struct bpf_verifier_env *env, struct bpf_call_graph
cg->cnt = cnt;
cg->words = words;
- cg->calls = kvcalloc((2 * cnt + 6) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+ cg->calls = kvcalloc((2 * cnt + 6) * words, sizeof(long), BPF_VERIFIER_GFP);
if (!cg->calls)
return -ENOMEM;
cg->reach = cg->calls + cnt * words;
@@ -5914,7 +5914,7 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
struct bpf_subprog_call_depth_info *dinfo;
int depth, total, tmp, s, err;
- dinfo = kvzalloc_objs(*dinfo, cg->cnt, GFP_KERNEL_ACCOUNT);
+ dinfo = kvzalloc_objs(*dinfo, cg->cnt, BPF_VERIFIER_GFP);
if (!dinfo)
return -ENOMEM;
compute_call_depth(env, cg, dinfo);
@@ -6060,7 +6060,7 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
int i, j, s, err = 0;
/* calls + s * words: the subprogs that s calls */
- calls = kvcalloc((cnt + 2) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+ calls = kvcalloc((cnt + 2) * words, sizeof(long), BPF_VERIFIER_GFP);
if (!calls)
return -ENOMEM;
taken = calls + cnt * words;
@@ -7258,11 +7258,11 @@ static int mark_reg_func_set(struct bpf_verifier_env *env, struct bpf_reg_state
verbose(env, "more than %d sets of functions\n", BPF_MAX_FUNC_SETS);
return -E2BIG;
}
- sets = krealloc_array(env->func_sets, i + 1, sizeof(*sets), GFP_KERNEL_ACCOUNT);
+ sets = krealloc_array(env->func_sets, i + 1, sizeof(*sets), BPF_VERIFIER_GFP);
if (!sets)
return -ENOMEM;
env->func_sets = sets;
- set = kzalloc_flex(*set, subprog, n, GFP_KERNEL_ACCOUNT);
+ set = kzalloc_flex(*set, subprog, n, BPF_VERIFIER_GFP);
if (!set)
return -ENOMEM;
set->cnt = n;
@@ -7330,7 +7330,7 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
return -ENOENT;
if (cnt > BPF_MAX_FUNC_PTR_RANGE)
goto overlap;
- targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), GFP_KERNEL_ACCOUNT);
+ targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), BPF_VERIFIER_GFP);
if (!targets)
return -ENOMEM;
@@ -11313,7 +11313,7 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
}
caller = state->frame[state->curframe];
- callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
+ callee = kzalloc_obj(*callee, BPF_VERIFIER_GFP);
if (!callee)
return -ENOMEM;
state->frame[state->curframe + 1] = callee;
@@ -11674,7 +11674,7 @@ static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int i
struct bpf_verifier_state entry = {};
int i, err;
- callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
+ callee = kzalloc_obj(*callee, BPF_VERIFIER_GFP);
if (!callee)
return -ENOMEM;
entry.frame[0] = callee;
@@ -12153,7 +12153,7 @@ static int check_func_callx_set(struct bpf_verifier_env *env, int regno, int *in
return PTR_ERR(branch);
continue;
}
- copy = kzalloc_obj(*copy, GFP_KERNEL_ACCOUNT);
+ copy = kzalloc_obj(*copy, BPF_VERIFIER_GFP);
if (!copy)
return -ENOMEM;
err = bpf_copy_verifier_state(copy, cur);
@@ -21367,7 +21367,7 @@ static int flow_try_walk(struct bpf_verifier_env *env, struct loop_stack_entry *
entry->walk_id = env->walk_cnt + 1;
if (is_power_of_2(entry->walk_id)) {
walk_insns = kvrealloc(env->walk_insns, 2 * entry->walk_id * sizeof(u32),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!walk_insns)
return -ENOMEM;
env->walk_insns = walk_insns;
@@ -22925,7 +22925,7 @@ static int add_func_ptr(struct bpf_verifier_env *env, struct bpf_map *map, u32 m
if (!(env->func_ptr_cnt & (env->func_ptr_cnt - 1))) {
ptrs = kvrealloc(env->func_ptrs,
array_size(max(2 * env->func_ptr_cnt, 16U), sizeof(*ptrs)),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!ptrs)
return -ENOMEM;
env->func_ptrs = ptrs;
@@ -23098,7 +23098,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
env->prev_linfo = NULL;
env->pass_cnt++;
- state = kzalloc_obj(struct bpf_verifier_state, GFP_KERNEL_ACCOUNT);
+ state = kzalloc_obj(struct bpf_verifier_state, BPF_VERIFIER_GFP);
if (!state)
return -ENOMEM;
state->curframe = 0;
@@ -23107,7 +23107,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
/* nobody knows what the callers of a global function or of an extension hold */
state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
env->no_rcu_read_lock = context == BPF_FUNC_CTX_NO_RCU_LOCK;
- state->frame[0] = kzalloc_obj(struct bpf_func_state, GFP_KERNEL_ACCOUNT);
+ state->frame[0] = kzalloc_obj(struct bpf_func_state, BPF_VERIFIER_GFP);
if (!state->frame[0]) {
kfree(state);
return -ENOMEM;
@@ -23392,7 +23392,7 @@ static void add_insns_of_callees(struct bpf_verifier_env *env)
struct bpf_insn *insn = env->prog->insnsi;
int *added, i, j, s, c;
- added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
+ added = kvcalloc(env->subprog_cnt, sizeof(*added), BPF_VERIFIER_GFP);
if (!added)
return;
/* callees first */
@@ -23447,7 +23447,7 @@ static void print_verification_stats(struct bpf_verifier_env *env)
int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
const struct bpf_ctx_arg_aux *info, u32 cnt)
{
- prog->aux->ctx_arg_info = kmemdup_array(info, cnt, sizeof(*info), GFP_KERNEL_ACCOUNT);
+ prog->aux->ctx_arg_info = kmemdup_array(info, cnt, sizeof(*info), BPF_VERIFIER_GFP);
prog->aux->ctx_arg_info_size = cnt;
return prog->aux->ctx_arg_info ? 0 : -ENOMEM;
@@ -24636,7 +24636,7 @@ static int process_fd_array_continuous(struct bpf_verifier_env *env,
return -E2BIG;
}
- env->fd_array = kvzalloc_objs(*env->fd_array, cnt, GFP_KERNEL_ACCOUNT);
+ env->fd_array = kvzalloc_objs(*env->fd_array, cnt, BPF_VERIFIER_GFP);
if (!env->fd_array)
return -ENOMEM;
env->fd_array_cnt = cnt;
@@ -24775,7 +24775,7 @@ static int add_kfunc_desc_target(struct bpf_verifier_env *env,
}
new_tab = krealloc(tab, struct_size(tab, descs, tab->nr_descs + 1),
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!new_tab)
return -ENOMEM;
tab = new_tab;
@@ -25136,7 +25136,7 @@ static int bpf_prog_verify_signature(struct bpf_verifier_env *env,
err = -E2BIG;
goto out;
}
- data = kvmalloc(data_sz, GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+ data = kvmalloc(data_sz, BPF_VERIFIER_GFP | __GFP_ZERO);
if (!data) {
err = -ENOMEM;
goto out;
@@ -25251,7 +25251,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
/* 'struct bpf_verifier_env' can be global, but since it's not small,
* allocate/free it every time bpf_check() is called
*/
- env = kvzalloc_obj(struct bpf_verifier_env, GFP_KERNEL_ACCOUNT);
+ env = kvzalloc_obj(struct bpf_verifier_env, BPF_VERIFIER_GFP);
if (!env)
return -ENOMEM;
@@ -25310,7 +25310,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
len = env->insn_aux_data_len = env->prog->len;
env->insn_aux_data =
__vmalloc(array_size(sizeof(struct bpf_insn_aux_data), len),
- GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+ BPF_VERIFIER_GFP | __GFP_ZERO);
ret = -ENOMEM;
if (!env->insn_aux_data)
goto skip_full_check;
@@ -25561,7 +25561,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
/* if program passed verifier, update used_maps in bpf_prog_info */
env->prog->aux->used_maps = kmalloc_objs(env->used_maps[0],
env->used_map_cnt,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!env->prog->aux->used_maps) {
ret = -ENOMEM;
@@ -25576,7 +25576,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
/* if program passed verifier, update used_btfs in bpf_prog_aux */
env->prog->aux->used_btfs = kmalloc_objs(env->used_btfs[0],
env->used_btf_cnt,
- GFP_KERNEL_ACCOUNT);
+ BPF_VERIFIER_GFP);
if (!env->prog->aux->used_btfs) {
ret = -ENOMEM;
goto err_release_maps;
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 03/21] bpf: Give a syscall program the largest context when it does tail calls
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 02/21] bpf: Fail the load when there is no memory Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 04/21] bpf: Do not inline bpf_loop() when its callback is dead code Alexei Starovoitov
` (17 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
BPF_PROG_RUN compares ctx_size_in with max_ctx_offset of the program that
it runs. The target of a tail call gets the same context:
first: bpf_tail_call(ctx, ...) max_ctx_offset 0, ctx_size_in 8
target: r0 = *(u32 *)(ctx + 60000) out of bounds
The target is not known before the run. Pad the copy of the context with
zeros up to U16_MAX, no program can use more. kern_sys_bpf() makes no
copy, so its caller has to pass that much. An extension cannot add tail
calls to a program that has none.
The copy is accounted now. It comes from kmalloc-cg or vmalloc, not from
the caches of memdup_user().
Fixes: 79a7f8bdb159 ("bpf: Introduce bpf_sys_bpf() helper and program type.")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/syscall.c | 7 +++++++
net/bpf/test_run.c | 20 ++++++++++++++------
2 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index eda5d8ee8e3f..844b2c2951c6 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3859,6 +3859,12 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
err = -EINVAL;
goto out_unlock;
}
+ /* The same for tail calls, see bpf_prog_test_run_syscall(). */
+ if (prog->type == BPF_PROG_TYPE_EXT && tgt_prog->type == BPF_PROG_TYPE_SYSCALL &&
+ prog->aux->tail_call_reachable && !tgt_prog->aux->tail_call_reachable) {
+ err = -EINVAL;
+ goto out_unlock;
+ }
err = bpf_link_prime(&link->link.link, &link_primer);
if (err)
@@ -6669,6 +6675,7 @@ int kern_sys_bpf(int cmd, union bpf_attr *attr, unsigned int size)
return PTR_ERR(prog);
if (attr->test.ctx_size_in < prog->aux->max_ctx_offset ||
+ (prog->aux->tail_call_reachable && attr->test.ctx_size_in < U16_MAX) ||
attr->test.ctx_size_in > U16_MAX) {
bpf_prog_put(prog);
return -EINVAL;
diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
index d6c9dc62182e..cc3236e3c0c4 100644
--- a/net/bpf/test_run.c
+++ b/net/bpf/test_run.c
@@ -1687,8 +1687,8 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog,
{
void __user *ctx_in = u64_to_user_ptr(kattr->test.ctx_in);
__u32 ctx_size_in = kattr->test.ctx_size_in;
+ u32 retval, size = ctx_size_in;
void *ctx = NULL;
- u32 retval;
int err = 0;
/* doesn't support data_in/out, ctx_out, duration, or repeat or flags */
@@ -1702,10 +1702,18 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog,
ctx_size_in > U16_MAX)
return -EINVAL;
- if (ctx_size_in) {
- ctx = memdup_user(ctx_in, ctx_size_in);
- if (IS_ERR(ctx))
- return PTR_ERR(ctx);
+ /* the target of a tail call can use that much */
+ if (prog->aux->tail_call_reachable)
+ size = max_t(u32, size, U16_MAX);
+
+ if (size) {
+ ctx = kvzalloc(size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!ctx)
+ return -ENOMEM;
+ if (copy_from_user(ctx, ctx_in, ctx_size_in)) {
+ err = -EFAULT;
+ goto out;
+ }
}
rcu_read_lock_trace();
@@ -1720,7 +1728,7 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog,
if (copy_to_user(ctx_in, ctx, ctx_size_in))
err = -EFAULT;
out:
- kfree(ctx);
+ kvfree(ctx);
return err;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 04/21] bpf: Do not inline bpf_loop() when its callback is dead code
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (2 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 03/21] bpf: Give a syscall program the largest context when it does tail calls Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 05/21] bpf: Dump programs that have more than 524288 insns Alexei Starovoitov
` (16 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The main pass does not enter the callback of bpf_loop(0, cb). When cb
comes from a table of functions, nothing keeps it and it is removed. The
inlined loop still has 'call cb'. The call goes to the next function, or
the load fails with 'verifier bug: No program to jit'.
Leave the call of the helper. It does not look at cb.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/fixups.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 8d1dd0770711..d1eecb95545f 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -3610,8 +3610,12 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env)
struct bpf_loop_inline_state *inline_state =
&env->insn_aux_data[i].loop_inline_state;
- /* the callback that takes a stack in arena is not for a call from the program */
+ /*
+ * The callback that takes a stack in arena is not for a call from the program.
+ * The callback of a loop without iterations is dead code, it goes away.
+ */
if (is_bpf_loop_call(insn) && inline_state->fit_for_inline &&
+ env->insn_aux_data[subprogs[inline_state->callback_subprogno].start].seen &&
!subprogs[inline_state->callback_subprogno].arena_stack_entry) {
stack_depth_extra = BPF_REG_SIZE * 3 + stack_depth_roundup;
err = inline_bpf_loop(env, i,
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 05/21] bpf: Dump programs that have more than 524288 insns
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (3 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 04/21] bpf: Do not inline bpf_loop() when its callback is dead code Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 06/21] bpf: Resolve BTF structs and arrays first Alexei Starovoitov
` (15 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
kmemdup() stops at 4 MB. A program that loads cannot be dumped.
Fixes: c04c0d2b968a ("bpf: increase complexity limit and maximum program size")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/syscall.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 844b2c2951c6..a1af95dd315b 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -5076,8 +5076,7 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
u8 code;
int i;
- insns = kmemdup(prog->insnsi, bpf_prog_insn_size(prog),
- GFP_USER);
+ insns = kvmemdup(prog->insnsi, bpf_prog_insn_size(prog), GFP_USER | __GFP_NOWARN);
if (!insns)
return insns;
@@ -5252,7 +5251,7 @@ static int bpf_prog_get_info_by_fd(struct file *file,
uinsns = u64_to_user_ptr(info.xlated_prog_insns);
ulen = min_t(u32, info.xlated_prog_len, ulen);
fault = copy_to_user(uinsns, insns_sanitized, ulen);
- kfree(insns_sanitized);
+ kvfree(insns_sanitized);
if (fault)
return -EFAULT;
} else {
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 06/21] bpf: Resolve BTF structs and arrays first
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (4 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 05/21] bpf: Dump programs that have more than 524288 insns Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 17:40 ` sashiko-bot
2026-11-16 15:26 ` [PATCH bpf-next v2 07/21] bpf: Remove the limits on maps, BTFs and fd_array of a program Alexei Starovoitov
` (14 subsequent siblings)
20 siblings, 1 reply; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
btf_resolve() does not see this loop:
ptr -> typedef -> array -+
^--------------+
The walk from the pointer stops at the array and marks the typedef as
resolved. Only the limits of the later walks end the loop. The walk for
special fields is going to lose its limit.
Resolve structs and arrays first. Then a type that holds itself is always
found. Such BTF does not load any more.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/btf.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 242b011ba93a..fffde271bbd7 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6015,6 +6015,22 @@ static int btf_check_all_types(struct btf_verifier_env *env)
return err;
env->phase++;
+ /*
+ * Structs and arrays go first. A walk from a pointer stops at them and hides this loop:
+ * ptr -> typedef -> array -+
+ * ^--------------+
+ */
+ for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
+ type_id = btf->start_id + i;
+ t = btf_type_by_id(btf, type_id);
+
+ if ((btf_type_is_struct(t) || btf_type_is_array(t)) &&
+ !env_type_is_resolved(env, type_id)) {
+ err = btf_resolve(env, t, type_id);
+ if (err)
+ return err;
+ }
+ }
for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
type_id = btf->start_id + i;
t = btf_type_by_id(btf, type_id);
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 07/21] bpf: Remove the limits on maps, BTFs and fd_array of a program
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (5 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 06/21] bpf: Resolve BTF structs and arrays first Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 08/21] bpf: Do not scan all maps for each insn and each subprog Alexei Starovoitov
` (13 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A program can use 64 maps, 64 BTF objects and 4096 entries of fd_array.
Each jump table and each .data.X section is a map. Nothing needs these
numbers at run time.
Add two small tools:
bpf_table_next() a table that doubles
bpf_index key -> number of an entry, a hash table
ld_imm64 map --> used_index[address of map] --> used_maps[i]
The program takes the tables from the verifier, there is no copy. The
table of fd_array grows with the entries that are read. Unprivileged
programs have 4096 insns, they keep 4096 entries.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 39 +++++-
kernel/bpf/cfg.c | 39 ++++++
kernel/bpf/core.c | 4 +-
kernel/bpf/syscall.c | 6 +-
kernel/bpf/verifier.c | 119 ++++++------------
.../selftests/bpf/prog_tests/fd_array.c | 36 ++++--
6 files changed, 137 insertions(+), 106 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 2f15b43074d1..4d91f665109d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -742,6 +742,33 @@ struct bpf_iarray {
u32 items[];
};
+/*
+ * Make room for entry 'cnt' of a table that has 'cnt' entries, return the new entry.
+ * The table doubles: 1, 2, 4, 8, ... It is not a stack: 'cnt' never goes down.
+ */
+#define bpf_table_next(tab, cnt) \
+ ({ \
+ typeof(tab) __tab = (tab); \
+ size_t __cap = 2 * (size_t)(cnt) ?: 1; \
+ if (!((cnt) & ((cnt) - 1))) \
+ __tab = kvrealloc(__tab, array_size(__cap, sizeof(*__tab)), \
+ BPF_VERIFIER_GFP); \
+ __tab ? &((tab) = __tab)[cnt] : NULL; \
+ })
+
+/* key -> number of an entry of a table */
+struct bpf_index {
+ struct bpf_index_slot {
+ u64 key;
+ u32 next_val; /* value + 1, 0 if the slot is free */
+ } *slots;
+ u32 cnt;
+ u8 bits;
+};
+
+int bpf_index_find(const struct bpf_index *index, u64 key);
+int bpf_index_add(struct bpf_index *index, u64 key, u32 val);
+
#define iarray_for_each(item, arr) \
for (int ___idx = 0; \
___idx < (arr)->cnt && ({ item = (arr)->items[___idx]; 1; }); \
@@ -970,9 +997,6 @@ struct bpf_insn_aux_data {
struct bpf_loop *loop;
};
-#define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
-#define MAX_USED_BTFS 64 /* max number of BTFs accessed by one BPF program */
-
/*
* Longest line the verifier log can carry: tmp_str_buf, see
* TMP_STR_BUF_LEN, plus its prefix.
@@ -1259,9 +1283,12 @@ struct bpf_verifier_env {
u32 *walk_insns; /* the insns that were looked at in each of them */
u32 max_arena_stack_depth; /* of all call chains */
struct bpf_verifier_state *cur_state; /* current verifier state */
- struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
- struct btf_mod_pair used_btfs[MAX_USED_BTFS]; /* array of BTF's used by BPF program */
- struct bpf_map *insn_array_maps[MAX_USED_MAPS]; /* array of INSN_ARRAY map's to be relocated */
+ struct bpf_map **used_maps; /* array of map's used by eBPF program */
+ struct btf_mod_pair *used_btfs; /* array of BTF's used by BPF program */
+ struct bpf_index used_index; /* address of a map or BTF -> entry of these two */
+ struct bpf_map **insn_array_maps; /* array of INSN_ARRAY map's to be relocated */
+ u32 max_key_size; /* of all used maps and their inner maps */
+ u32 max_value_size;
u32 used_map_cnt; /* number of used maps */
u32 used_btf_cnt; /* number of used BTF objects */
u32 insn_array_map_cnt; /* number of used maps of type BPF_MAP_TYPE_INSN_ARRAY */
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 40aafceb139e..c16b4a283dbc 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -3,6 +3,7 @@
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
#include <linux/filter.h>
+#include <linux/hash.h>
#include <linux/sort.h>
#include "diagnostics.h"
@@ -199,6 +200,44 @@ struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
return new;
}
+static struct bpf_index_slot *index_slot(const struct bpf_index *index, u64 key)
+{
+ u32 i = hash_64(key, index->bits);
+
+ while (index->slots[i].next_val && index->slots[i].key != key)
+ i = (i + 1) & (BIT(index->bits) - 1);
+ return &index->slots[i];
+}
+
+/* Returns the value of 'key', -1 if there is none */
+int bpf_index_find(const struct bpf_index *index, u64 key)
+{
+ return index->cnt ? index_slot(index, key)->next_val - 1 : -1;
+}
+
+/* 'key' is new. At most a half of the slots is taken. */
+int bpf_index_add(struct bpf_index *index, u64 key, u32 val)
+{
+ struct bpf_index old = *index;
+ u32 i;
+
+ if (2 * old.cnt >= BIT(old.bits) - 1) {
+ index->slots = kvcalloc(BIT(old.bits + 2), sizeof(*old.slots), BPF_VERIFIER_GFP);
+ if (!index->slots) {
+ index->slots = old.slots;
+ return -ENOMEM;
+ }
+ index->bits += 2;
+ for (i = 0; old.cnt && i < BIT(old.bits); i++)
+ if (old.slots[i].next_val)
+ *index_slot(index, old.slots[i].key) = old.slots[i];
+ kvfree(old.slots);
+ }
+ *index_slot(index, key) = (struct bpf_index_slot){ key, val + 1 };
+ index->cnt++;
+ return 0;
+}
+
static int copy_insn_array(struct bpf_map *map, u32 start, u32 end, u32 *items)
{
struct bpf_insn_array_value *value;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 09abfed83109..1b51f1d97efc 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3429,7 +3429,7 @@ void __bpf_free_used_maps(struct bpf_prog_aux *aux,
static void bpf_free_used_maps(struct bpf_prog_aux *aux)
{
__bpf_free_used_maps(aux, aux->used_maps, aux->used_map_cnt);
- kfree(aux->used_maps);
+ kvfree(aux->used_maps);
}
void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len)
@@ -3450,7 +3450,7 @@ void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len)
static void bpf_free_used_btfs(struct bpf_prog_aux *aux)
{
__bpf_free_used_btfs(aux->used_btfs, aux->used_btf_cnt);
- kfree(aux->used_btfs);
+ kvfree(aux->used_btfs);
}
static void bpf_prog_free_images(struct work_struct *work)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index a1af95dd315b..9b0268d6715d 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -6291,8 +6291,8 @@ static int bpf_prog_bind_map(union bpf_attr *attr)
goto out_unlock;
}
- used_maps_new = kmalloc_objs(used_maps_new[0],
- prog->aux->used_map_cnt + 1);
+ used_maps_new = kvmalloc_objs(used_maps_new[0],
+ prog->aux->used_map_cnt + 1);
if (!used_maps_new) {
ret = -ENOMEM;
goto out_unlock;
@@ -6310,7 +6310,7 @@ static int bpf_prog_bind_map(union bpf_attr *attr)
prog->aux->used_map_cnt++;
prog->aux->used_maps = used_maps_new;
- kfree(used_maps_old);
+ kvfree(used_maps_old);
out_unlock:
mutex_unlock(&prog->aux->used_maps_mutex);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d64719c85e6f..b316443548ff 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -15407,7 +15407,6 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
case ARG_PTR_TO_MAP_KEY:
case ARG_PTR_TO_MAP_VALUE: {
bool is_key = base_type(at) == ARG_PTR_TO_MAP_KEY;
- u64 val;
int i, map_reg;
for (i = 0; i < arg; i++) {
@@ -15443,23 +15442,9 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
*/
exact_size = false;
access_type |= BPF_READ;
- val = 0;
- for (i = 0; i < env->used_map_cnt; i++) {
- struct bpf_map *map = env->used_maps[i];
- u32 sz = is_key ? map->key_size : map->value_size;
-
- if (sz > val)
- val = sz;
- if (map->inner_map_meta) {
- sz = is_key ? map->inner_map_meta->key_size
- : map->inner_map_meta->value_size;
- if (sz > val)
- val = sz;
- }
- }
- if (!val)
+ size = is_key ? env->max_key_size : env->max_value_size;
+ if (!size)
return info;
- size = val;
goto out;
}
case ARG_PTR_TO_MEM:
@@ -22021,26 +22006,21 @@ static int __add_used_btf(struct bpf_verifier_env *env, struct btf *btf)
{
struct btf_mod_pair *btf_mod;
int ret = 0;
- int i;
/* check whether we recorded this BTF (and maybe module) already */
- for (i = 0; i < env->used_btf_cnt; i++)
- if (env->used_btfs[i].btf == btf)
- goto ret_put;
+ if (bpf_index_find(&env->used_index, (unsigned long)btf) >= 0)
+ goto ret_put;
if (env->signature) {
verbose(env, "signed program cannot bind any BTF\n");
ret = -EACCES;
goto ret_put;
}
- if (env->used_btf_cnt >= MAX_USED_BTFS) {
- verbose(env, "The total number of btfs per program has reached the limit of %u\n",
- MAX_USED_BTFS);
- ret = -E2BIG;
+ btf_mod = bpf_table_next(env->used_btfs, env->used_btf_cnt);
+ if (!btf_mod) {
+ ret = -ENOMEM;
goto ret_put;
}
-
- btf_mod = &env->used_btfs[env->used_btf_cnt];
btf_mod->btf = btf;
btf_mod->module = NULL;
@@ -22053,8 +22033,7 @@ static int __add_used_btf(struct bpf_verifier_env *env, struct btf *btf)
}
}
- env->used_btf_cnt++;
- return 0;
+ return bpf_index_add(&env->used_index, (unsigned long)btf, env->used_btf_cnt++);
ret_put:
/* Either error or this BTF was already added */
@@ -22347,6 +22326,8 @@ static int claim_used_maps(struct bpf_verifier_env *env)
if (map->map_type != BPF_MAP_TYPE_INSN_ARRAY)
continue;
+ if (!bpf_table_next(env->insn_array_maps, env->insn_array_map_cnt))
+ return -ENOMEM;
err = bpf_insn_array_init(map, env->prog);
if (err) {
verbose(env, "Failed to properly initialize insn array\n");
@@ -22360,12 +22341,13 @@ static int claim_used_maps(struct bpf_verifier_env *env)
static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map)
{
+ struct bpf_map *inner = map->inner_map_meta ?: map;
int i, err;
/* check whether we recorded this map already */
- for (i = 0; i < env->used_map_cnt; i++)
- if (env->used_maps[i] == map)
- return i;
+ i = bpf_index_find(&env->used_index, (unsigned long)map);
+ if (i >= 0)
+ return i;
if (env->signature &&
env->prog->aux->sig.verdict == BPF_SIG_VERIFIED) {
@@ -22373,16 +22355,17 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map)
map->name);
return -EACCES;
}
- if (env->used_map_cnt >= MAX_USED_MAPS) {
- verbose(env, "The total number of maps per program has reached the limit of %u\n",
- MAX_USED_MAPS);
- return -E2BIG;
- }
err = check_map_prog_compatibility(env, map, env->prog);
if (err)
return err;
+ if (!bpf_table_next(env->used_maps, env->used_map_cnt) ||
+ bpf_index_add(&env->used_index, (unsigned long)map, env->used_map_cnt))
+ return -ENOMEM;
+ env->max_key_size = max3(env->max_key_size, map->key_size, inner->key_size);
+ env->max_value_size = max3(env->max_value_size, map->value_size, inner->value_size);
+
if (bpf_prog_runs_sleepable(env->prog))
atomic64_inc(&map->sleepable_refcnt);
@@ -23010,12 +22993,14 @@ static void release_maps(struct bpf_verifier_env *env)
{
__bpf_free_used_maps(env->prog->aux, env->used_maps,
env->used_map_cnt);
+ kvfree(env->used_maps);
}
/* drop refcnt of maps used by the rejected program */
static void release_btfs(struct bpf_verifier_env *env)
{
__bpf_free_used_btfs(env->used_btfs, env->used_btf_cnt);
+ kvfree(env->used_btfs);
}
/* convert pseudo BPF_LD_IMM64 into generic BPF_LD_IMM64 */
@@ -24615,39 +24600,31 @@ static int add_fd_from_fd_array(struct bpf_verifier_env *env, u32 idx, int fd)
return PTR_ERR(map);
}
-/*
- * A continuous fd_array is resolved into an in-memory cache with one slot
- * per entry. The bound here is deliberately generous and not derived from
- * the per-program object limits: Duplicate entries /are/ permitted, and
- * the number of distinct maps and BTFs a program can bind is enforced when
- * each entry is resolved by __add_used_map() and __add_used_btf().
- */
-#define MAX_FD_ARRAY_CNT 4096
-
+/* A continuous fd_array is resolved into an in-memory cache with one slot per entry. */
static int process_fd_array_continuous(struct bpf_verifier_env *env,
bpfptr_t fd_array, u32 cnt)
{
int fd, ret;
u32 i;
- if (cnt > MAX_FD_ARRAY_CNT) {
- verbose(env, "fd_array has too many entries (%u, max %u)\n",
- cnt, MAX_FD_ARRAY_CNT);
+ if (!env->bpf_capable && cnt > BPF_MAXINSNS) {
+ verbose(env, "fd_array has too many entries (%u, max %u)\n", cnt, BPF_MAXINSNS);
return -E2BIG;
}
- env->fd_array = kvzalloc_objs(*env->fd_array, cnt, BPF_VERIFIER_GFP);
- if (!env->fd_array)
- return -ENOMEM;
- env->fd_array_cnt = cnt;
for (i = 0; i < cnt; i++) {
+ if (bpf_must_abort())
+ return -EINTR;
if (copy_from_bpfptr_offset(&fd, fd_array,
(size_t)i * sizeof(fd), sizeof(fd)))
return -EFAULT;
+ if (!bpf_table_next(env->fd_array, i))
+ return -ENOMEM;
ret = add_fd_from_fd_array(env, i, fd);
if (ret)
return ret;
}
+ env->fd_array_cnt = cnt;
return 0;
}
@@ -25557,35 +25534,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret)
goto err_release_maps;
- if (env->used_map_cnt) {
- /* if program passed verifier, update used_maps in bpf_prog_info */
- env->prog->aux->used_maps = kmalloc_objs(env->used_maps[0],
- env->used_map_cnt,
- BPF_VERIFIER_GFP);
-
- if (!env->prog->aux->used_maps) {
- ret = -ENOMEM;
- goto err_release_maps;
- }
-
- memcpy(env->prog->aux->used_maps, env->used_maps,
- sizeof(env->used_maps[0]) * env->used_map_cnt);
- env->prog->aux->used_map_cnt = env->used_map_cnt;
- }
- if (env->used_btf_cnt) {
- /* if program passed verifier, update used_btfs in bpf_prog_aux */
- env->prog->aux->used_btfs = kmalloc_objs(env->used_btfs[0],
- env->used_btf_cnt,
- BPF_VERIFIER_GFP);
- if (!env->prog->aux->used_btfs) {
- ret = -ENOMEM;
- goto err_release_maps;
- }
-
- memcpy(env->prog->aux->used_btfs, env->used_btfs,
- sizeof(env->used_btfs[0]) * env->used_btf_cnt);
- env->prog->aux->used_btf_cnt = env->used_btf_cnt;
- }
+ /* if program passed verifier, it gets the tables */
+ env->prog->aux->used_maps = env->used_maps;
+ env->prog->aux->used_map_cnt = env->used_map_cnt;
+ env->prog->aux->used_btfs = env->used_btfs;
+ env->prog->aux->used_btf_cnt = env->used_btf_cnt;
if (env->used_map_cnt || env->used_btf_cnt) {
/* program is valid. Convert pseudo bpf_ld_imm64 into generic
* bpf_ld_imm64 instructions
@@ -25642,6 +25595,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
vfree(env->insn_aux_data);
bpf_patch_list_free(env);
kvfree(env->fd_array);
+ kvfree(env->used_index.slots);
+ kvfree(env->insn_array_maps);
bpf_free_scev(env);
kvfree(env->walk_insns);
bpf_flow_free(env);
diff --git a/tools/testing/selftests/bpf/prog_tests/fd_array.c b/tools/testing/selftests/bpf/prog_tests/fd_array.c
index 3078d8264deb..d7459bd92998 100644
--- a/tools/testing/selftests/bpf/prog_tests/fd_array.c
+++ b/tools/testing/selftests/bpf/prog_tests/fd_array.c
@@ -376,6 +376,11 @@ static void check_fd_array_cnt__fd_array_with_trash(void)
if (!ASSERT_EQ(prog_fd, -EINVAL, "prog should have been rejected with -EINVAL"))
goto cleanup;
+ /* trash 3: the table grows with the entries that are read */
+ prog_fd = load_test_prog((int *)8, 1 << 30);
+ if (!ASSERT_EQ(prog_fd, -EFAULT, "prog should have been rejected with -EFAULT"))
+ goto cleanup;
+
/* Validate that the prog is ok if trash is removed */
Close(extra_fds[2]);
extra_fds[2] = new_btf();
@@ -394,22 +399,27 @@ static void check_fd_array_cnt__fd_array_with_trash(void)
}
/*
- * Test that a program with too big fd_array can't be loaded.
+ * Test that the numbers of maps, BTFs and entries of fd_array are not limited.
*/
-static void check_fd_array_cnt__fd_array_too_big(void)
+static void check_fd_array_cnt__fd_array_big(void)
{
- int extra_fds[65];
- int prog_fd = -1;
- int i;
-
- for (i = 0; i < 65; i++) {
- extra_fds[i] = new_map();
- if (!ASSERT_GE(extra_fds[i], 0, "new_map"))
+ static int extra_fds[5000];
+ const int objs = 300;
+ __u32 nr_map_ids = 0;
+ int prog_fd, i, j;
+
+ for (i = 0; i < objs; i++) {
+ extra_fds[i] = i % 2 ? new_map() : new_btf();
+ if (!ASSERT_GE(extra_fds[i], 0, "new_obj"))
goto cleanup_fds;
}
+ for (j = objs; j < ARRAY_SIZE(extra_fds); j++)
+ extra_fds[j] = extra_fds[j % objs];
- prog_fd = load_test_prog(extra_fds, 65);
- ASSERT_EQ(prog_fd, -E2BIG, "prog should have been rejected with -E2BIG");
+ prog_fd = load_test_prog(extra_fds, ARRAY_SIZE(extra_fds));
+ if (ASSERT_GE(prog_fd, 0, "prog should have been loaded"))
+ check_expected_map_ids(prog_fd, objs / 2 + 1, NULL, &nr_map_ids);
+ Close(prog_fd);
cleanup_fds:
while (i-- > 0)
@@ -436,6 +446,6 @@ void test_fd_array_cnt(void)
if (test__start_subtest("fd-array-trash-input"))
check_fd_array_cnt__fd_array_with_trash();
- if (test__start_subtest("fd-array-2big"))
- check_fd_array_cnt__fd_array_too_big();
+ if (test__start_subtest("fd-array-big"))
+ check_fd_array_cnt__fd_array_big();
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 08/21] bpf: Do not scan all maps for each insn and each subprog
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (6 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 07/21] bpf: Remove the limits on maps, BTFs and fd_array of a program Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 09/21] bpf: Remove the limits on kfuncs of a program Alexei Starovoitov
` (12 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The number of maps has no limit now.
- The dump of a program looks for the map of each ld_imm64. Put the
addresses of the maps into a tree.
- The JIT of each subprog needs the insn arrays only.
- The jump table of a subprog doubles.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/cfg.c | 3 ++-
kernel/bpf/fixups.c | 4 +--
kernel/bpf/syscall.c | 64 +++++++++++++++++++++-----------------------
3 files changed, 35 insertions(+), 36 deletions(-)
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index c16b4a283dbc..92dc8d0dc22a 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -366,13 +366,14 @@ static int __compute_subprog_jts(struct bpf_verifier_env *env)
}
old_cnt = subprog->jt ? subprog->jt->cnt : 0;
- jt = bpf_iarray_realloc(subprog->jt, old_cnt + jt_cur->cnt);
+ jt = bpf_iarray_realloc(subprog->jt, roundup_pow_of_two(old_cnt + jt_cur->cnt));
if (!jt) {
subprog->jt = NULL;
kvfree(jt_cur);
return -ENOMEM;
}
memcpy(jt->items + old_cnt, jt_cur->items, jt_cur->cnt << 2);
+ jt->cnt = old_cnt + jt_cur->cnt;
subprog->jt = jt;
kvfree(jt_cur);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index d1eecb95545f..20697eba9c5f 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1950,8 +1950,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->arena_stack_size = prog->aux->arena_stack_size;
func[i]->aux->arena_stack_entry = env->subprog_info[i].arena_stack_entry;
func[i]->aux->uses_arena_insns = bpf_prog_has_arena_insns(func[i]);
- func[i]->aux->used_maps = env->used_maps;
- func[i]->aux->used_map_cnt = env->used_map_cnt;
+ func[i]->aux->used_maps = env->insn_array_maps;
+ func[i]->aux->used_map_cnt = env->insn_array_map_cnt;
num_exentries = 0;
insn = func[i]->insnsi;
for (j = 0; j < func[i]->len; j++, insn++) {
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 9b0268d6715d..82c9af39e342 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -14,6 +14,7 @@
#include <linux/slab.h>
#include <linux/sched/signal.h>
#include <linux/vmalloc.h>
+#include <linux/maple_tree.h>
#include <linux/mmzone.h>
#include <linux/anon_inodes.h>
#include <linux/fdtable.h>
@@ -5015,32 +5016,23 @@ static int bpf_map_get_fd_by_id(const union bpf_attr *attr)
return fd;
}
-static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog,
- unsigned long addr, u32 *off,
- u32 *type)
+/* address of a map, addresses of its value -> map */
+static int bpf_prog_map_addrs(const struct bpf_prog *prog, struct maple_tree *mt)
{
- const struct bpf_map *map;
- int i;
+ struct bpf_map *map;
+ int i, err = 0;
+ u64 addr;
- mutex_lock(&prog->aux->used_maps_mutex);
- for (i = 0, *off = 0; i < prog->aux->used_map_cnt; i++) {
+ guard(mutex)(&prog->aux->used_maps_mutex);
+ for (i = 0; i < prog->aux->used_map_cnt && !err; i++) {
map = prog->aux->used_maps[i];
- if (map == (void *)addr) {
- *type = BPF_PSEUDO_MAP_FD;
- goto out;
- }
- if (!map->ops->map_direct_value_meta)
- continue;
- if (!map->ops->map_direct_value_meta(map, addr, off)) {
- *type = BPF_PSEUDO_MAP_VALUE;
- goto out;
- }
+ err = mtree_store(mt, (unsigned long)map, map, GFP_USER);
+ if (!err && map->ops->map_direct_value_meta &&
+ !map->ops->map_direct_value_addr(map, &addr, 0))
+ err = mtree_store_range(mt, addr, addr + map->value_size - 1, map,
+ GFP_USER);
}
- map = NULL;
-
-out:
- mutex_unlock(&prog->aux->used_maps_mutex);
- return map;
+ return err;
}
static void prepare_dump_pseudo_call(struct bpf_insn *insn)
@@ -5069,16 +5061,19 @@ static void prepare_dump_pseudo_call(struct bpf_insn *insn)
static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
const struct cred *f_cred)
{
+ struct bpf_insn *insns = NULL;
const struct bpf_map *map;
- struct bpf_insn *insns;
- u32 off, type;
+ struct maple_tree maps;
+ u32 off;
u64 imm;
u8 code;
int i;
- insns = kvmemdup(prog->insnsi, bpf_prog_insn_size(prog), GFP_USER | __GFP_NOWARN);
+ mt_init(&maps);
+ if (!bpf_prog_map_addrs(prog, &maps))
+ insns = kvmemdup(prog->insnsi, bpf_prog_insn_size(prog), GFP_USER | __GFP_NOWARN);
if (!insns)
- return insns;
+ goto out;
for (i = 0; i < prog->len; i++) {
code = insns[i].code;
@@ -5120,15 +5115,18 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
continue;
imm = ((u64)insns[i + 1].imm << 32) | (u32)insns[i].imm;
- map = bpf_map_from_imm(prog, imm, &off, &type);
- if (map) {
- insns[i].src_reg = type;
- insns[i].imm = map->id;
- insns[i + 1].imm = off;
+ map = mtree_load(&maps, imm);
+ if (!map || imm > ULONG_MAX)
continue;
- }
+ off = 0;
+ insns[i].src_reg = BPF_PSEUDO_MAP_FD;
+ if (imm != (unsigned long)map && !map->ops->map_direct_value_meta(map, imm, &off))
+ insns[i].src_reg = BPF_PSEUDO_MAP_VALUE;
+ insns[i].imm = map->id;
+ insns[i + 1].imm = off;
}
-
+out:
+ mtree_destroy(&maps);
return insns;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 09/21] bpf: Remove the limits on kfuncs of a program
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (7 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 08/21] bpf: Do not scan all maps for each insn and each subprog Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 10/21] bpf: Remove the limits on tables of functions Alexei Starovoitov
` (11 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A program can call 256 kfuncs from 256 module BTFs. Calls in dead code
count too. Each new kfunc sorts the table again.
Append to the tables and find the entries through one index:
(func_id, offset) -> descriptor
(func_id, offset, specialized) -> descriptor of the other target
(0, offset) -> BTF of the module
What stays is the width of insn->off, where JITs get the number of
the descriptor: 65536 of them.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 25 ++--
kernel/bpf/core.c | 2 +-
kernel/bpf/fixups.c | 4 +-
kernel/bpf/syscall.c | 2 +-
kernel/bpf/verifier.c | 225 ++++++++++-------------------------
5 files changed, 79 insertions(+), 179 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 4d91f665109d..508aef9bad9d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1286,6 +1286,7 @@ struct bpf_verifier_env {
struct bpf_map **used_maps; /* array of map's used by eBPF program */
struct btf_mod_pair *used_btfs; /* array of BTF's used by BPF program */
struct bpf_index used_index; /* address of a map or BTF -> entry of these two */
+ struct bpf_index kfunc_index; /* see kfunc_key() */
struct bpf_map **insn_array_maps; /* array of INSN_ARRAY map's to be relocated */
u32 max_key_size; /* of all used maps and their inner maps */
u32 max_value_size;
@@ -2195,11 +2196,6 @@ enum bpf_reg_arg_type {
DST_OP_NO_MARK /* same as above, check only, don't mark */
};
-#define MAX_KFUNC_DESCS 256
-/* Each kfunc can have its canonical and one specialized call target. */
-#define MAX_KFUNC_CALL_DESCS (MAX_KFUNC_DESCS * 2)
-static_assert(MAX_KFUNC_CALL_DESCS <= S16_MAX + 1);
-
struct bpf_kfunc_desc {
struct btf_func_model func_model;
struct bpf_func_proto proto;
@@ -2209,20 +2205,17 @@ struct bpf_kfunc_desc {
};
struct bpf_kfunc_desc_tab {
+ struct bpf_kfunc_desc *descs;
u32 nr_descs;
- u32 nr_base_descs;
- /* Sorted by func_id (BTF ID) and offset (fd_array offset) during
- * verification. JITs use the descriptor index stored in the finalized
- * call's off field. The first nr_base_descs entries are the canonical
- * descriptors used for verifier lookups. Call specialization may append
- * immutable descriptors for additional targets.
- *
- * Grown one entry at a time by bpf_add_kfunc_call() and during
- * call specialization.
- */
- struct bpf_kfunc_desc descs[];
};
+static inline void bpf_free_kfunc_tab(struct bpf_kfunc_desc_tab *tab)
+{
+ if (tab)
+ kvfree(tab->descs);
+ kfree(tab);
+}
+
/* Functions exported from verifier.c, used by fixups.c */
void bpf_mark_subprog_exc_cb(struct bpf_verifier_env *env, int subprog);
bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 1b51f1d97efc..3385df367b05 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -195,7 +195,7 @@ void bpf_prog_jit_attempt_done(struct bpf_prog *prog)
prog->aux->jited_linfo = NULL;
}
- kfree(prog->aux->kfunc_tab);
+ bpf_free_kfunc_tab(prog->aux->kfunc_tab);
prog->aux->kfunc_tab = NULL;
}
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 20697eba9c5f..e05d767ab71e 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -122,10 +122,10 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
struct bpf_kfunc_desc_tab *tab;
tab = prog->aux->kfunc_tab;
- if (insn->off < 0 || insn->off >= tab->nr_descs)
+ if ((u16)insn->off >= tab->nr_descs)
return NULL;
- res = &tab->descs[insn->off];
+ res = &tab->descs[(u16)insn->off];
if (bpf_jit_supports_far_kfunc_call()) {
if (res->func_id != insn->imm)
return NULL;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 82c9af39e342..fb2ecc9d81f5 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2489,7 +2489,7 @@ static void __bpf_prog_put_rcu_tasks(struct rcu_head *rcu)
static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred)
{
module_put(prog->aux->mod);
- kfree(prog->aux->kfunc_tab);
+ bpf_free_kfunc_tab(prog->aux->kfunc_tab);
kfree(prog->aux->ctx_arg_info);
if (prog->aux->attach_btf)
btf_put(prog->aux->attach_btf);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b316443548ff..f4e2a9f9f787 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2690,48 +2690,28 @@ static int bpf_find_exception_callback_insn_off(struct bpf_verifier_env *env)
return ret;
}
-#define MAX_KFUNC_BTFS 256
-
struct bpf_kfunc_btf {
struct btf *btf;
struct module *module;
- u16 offset;
};
struct bpf_kfunc_btf_tab {
- struct bpf_kfunc_btf descs[MAX_KFUNC_BTFS];
+ struct bpf_kfunc_btf *descs;
u32 nr_descs;
};
-static int kfunc_desc_cmp_by_id_off(const void *a, const void *b)
+/* The key of env->kfunc_index. The BTF of a module has func_id 0. */
+static u64 kfunc_key(u32 func_id, u16 offset, bool specialized)
{
- const struct bpf_kfunc_desc *d0 = a;
- const struct bpf_kfunc_desc *d1 = b;
-
- /* func_id is not greater than BTF_MAX_TYPE */
- return d0->func_id - d1->func_id ?: d0->offset - d1->offset;
-}
-
-static int kfunc_btf_cmp_by_off(const void *a, const void *b)
-{
- const struct bpf_kfunc_btf *d0 = a;
- const struct bpf_kfunc_btf *d1 = b;
-
- return d0->offset - d1->offset;
+ return func_id | (u64)offset << 32 | (u64)specialized << 48;
}
static struct bpf_kfunc_desc *
-find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)
+find_kfunc_desc(struct bpf_verifier_env *env, u32 func_id, u16 offset)
{
- struct bpf_kfunc_desc desc = {
- .func_id = func_id,
- .offset = offset,
- };
- struct bpf_kfunc_desc_tab *tab;
+ int i = bpf_index_find(&env->kfunc_index, kfunc_key(func_id, offset, false));
- tab = prog->aux->kfunc_tab;
- return bsearch(&desc, tab->descs, tab->nr_base_descs,
- sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off);
+ return func_id && i >= 0 ? &env->prog->aux->kfunc_tab->descs[i] : NULL;
}
int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
@@ -2831,50 +2811,38 @@ static struct btf *fd_array_get_btf(struct bpf_verifier_env *env, u32 idx)
static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env,
s16 offset)
{
- struct bpf_kfunc_btf kf_btf = { .offset = offset };
- struct bpf_kfunc_btf_tab *tab;
+ struct bpf_kfunc_btf_tab *tab = env->prog->aux->kfunc_btf_tab;
+ int i = bpf_index_find(&env->kfunc_index, kfunc_key(0, offset, false));
struct bpf_kfunc_btf *b;
struct module *mod;
struct btf *btf;
- tab = env->prog->aux->kfunc_btf_tab;
- b = bsearch(&kf_btf, tab->descs, tab->nr_descs,
- sizeof(tab->descs[0]), kfunc_btf_cmp_by_off);
- if (!b) {
- if (tab->nr_descs == MAX_KFUNC_BTFS) {
- verbose(env, "too many different module BTFs\n");
- return ERR_PTR(-E2BIG);
- }
-
- btf = fd_array_get_btf(env, offset);
- if (IS_ERR(btf))
- return btf;
- if (!btf_is_module(btf)) {
- verbose(env, "BTF fd for kfunc is not a module BTF\n");
- btf_put(btf);
- return ERR_PTR(-EINVAL);
- }
+ if (i >= 0)
+ return tab->descs[i].btf;
- mod = btf_try_get_module(btf);
- if (!mod) {
- btf_put(btf);
- return ERR_PTR(-ENXIO);
- }
+ b = bpf_table_next(tab->descs, tab->nr_descs);
+ if (!b)
+ return ERR_PTR(-ENOMEM);
- b = &tab->descs[tab->nr_descs++];
- b->btf = btf;
- b->module = mod;
- b->offset = offset;
+ btf = fd_array_get_btf(env, offset);
+ if (IS_ERR(btf))
+ return btf;
+ if (!btf_is_module(btf)) {
+ verbose(env, "BTF fd for kfunc is not a module BTF\n");
+ btf_put(btf);
+ return ERR_PTR(-EINVAL);
+ }
- /* sort() reorders entries by value, so b may no longer point
- * to the right entry after this
- */
- sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]),
- kfunc_btf_cmp_by_off, NULL);
- } else {
- btf = b->btf;
+ mod = btf_try_get_module(btf);
+ if (!mod) {
+ btf_put(btf);
+ return ERR_PTR(-ENXIO);
}
+ b->btf = btf;
+ b->module = mod;
+ if (bpf_index_add(&env->kfunc_index, kfunc_key(0, offset, false), tab->nr_descs++))
+ return ERR_PTR(-ENOMEM);
return btf;
}
@@ -2887,6 +2855,7 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab)
module_put(tab->descs[tab->nr_descs].module);
btf_put(tab->descs[tab->nr_descs].btf);
}
+ kvfree(tab->descs);
kfree(tab);
}
@@ -2908,22 +2877,14 @@ static struct btf *find_kfunc_desc_btf(struct bpf_verifier_env *env, s16 offset)
static struct btf *find_kfunc_desc_btf_cached(struct bpf_verifier_env *env, s16 offset)
{
- struct bpf_kfunc_btf kf_btf = { .offset = offset };
- struct bpf_kfunc_btf_tab *tab;
- struct bpf_kfunc_btf *b;
+ int i = bpf_index_find(&env->kfunc_index, kfunc_key(0, offset, false));
if (!offset)
return btf_vmlinux ?: ERR_PTR(-ENOENT);
if (offset < 0)
return ERR_PTR(-EINVAL);
- tab = env->prog->aux->kfunc_btf_tab;
- if (!tab)
- return ERR_PTR(-ENOENT);
-
- b = bsearch(&kf_btf, tab->descs, tab->nr_descs,
- sizeof(tab->descs[0]), kfunc_btf_cmp_by_off);
- return b ? b->btf : ERR_PTR(-ENOENT);
+ return i >= 0 ? env->prog->aux->kfunc_btf_tab->descs[i].btf : ERR_PTR(-ENOENT);
}
#define KF_IMPL_SUFFIX "_impl"
@@ -3081,19 +3042,9 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
prog_aux->kfunc_btf_tab = btf_tab;
}
- if (find_kfunc_desc(env->prog, func_id, offset))
+ if (find_kfunc_desc(env, func_id, offset))
return 0;
- if (tab->nr_base_descs == MAX_KFUNC_DESCS) {
- verbose(env, "too many different kernel function calls\n");
- return -E2BIG;
- }
- if (tab->nr_descs != tab->nr_base_descs) {
- verifier_bug(env, "unexpected specialized func desc found (%d descs, %d base descs)",
- tab->nr_descs, tab->nr_base_descs);
- return -EFAULT;
- }
-
err = fetch_kfunc_meta(env, func_id, offset, &kfunc);
if (err)
return err;
@@ -3133,12 +3084,9 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
meta.func_name = kfunc.name;
meta.kfunc_flags = kfunc.flags ? *kfunc.flags : 0;
- tab = krealloc(tab, struct_size(tab, descs, tab->nr_descs + 1), BPF_VERIFIER_GFP);
- if (!tab)
+ desc = bpf_table_next(tab->descs, tab->nr_descs);
+ if (!desc)
return -ENOMEM;
- prog_aux->kfunc_tab = tab;
-
- desc = &tab->descs[tab->nr_descs];
memset(desc, 0, sizeof(*desc));
err = gen_kfunc_arg_proto(env, &meta, &func_model, &desc->proto);
@@ -3149,11 +3097,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
desc->offset = offset;
desc->addr = addr;
desc->func_model = func_model;
- tab->nr_descs++;
- tab->nr_base_descs++;
- sort(tab->descs, tab->nr_base_descs, sizeof(tab->descs[0]),
- kfunc_desc_cmp_by_id_off, NULL);
- return 0;
+ return bpf_index_add(&env->kfunc_index, kfunc_key(func_id, offset, false), tab->nr_descs++);
}
static int add_subprogs(struct bpf_verifier_env *env)
@@ -3260,7 +3204,7 @@ static int add_kfuncs(struct bpf_verifier_env *env)
return -EPERM;
}
- ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
+ ret = bpf_must_abort() ? -EINTR : bpf_add_kfunc_call(env, insn->imm, insn->off);
if (ret < 0)
return ret;
}
@@ -15042,7 +14986,7 @@ bool bpf_needs_fixed_stack_off(struct bpf_verifier_env *env, int insn_idx)
if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
return false;
} else if (bpf_pseudo_kfunc_call(insn)) {
- desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+ desc = find_kfunc_desc(env, insn->imm, insn->off);
if (!desc)
return false;
fn = &desc->proto;
@@ -15520,7 +15464,7 @@ static u16 call_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn,
u16 mask = 0;
if (bpf_pseudo_kfunc_call(insn)) {
- desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+ desc = find_kfunc_desc(env, insn->imm, insn->off);
if (!desc)
return 0;
fn = &desc->proto;
@@ -15961,7 +15905,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
func_name = meta.func_name;
insn_aux = &env->insn_aux_data[insn_idx];
- desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+ desc = find_kfunc_desc(env, insn->imm, insn->off);
if (!desc) {
verifier_bug(env, "kfunc descriptor not found for func_id %u", insn->imm);
return -EFAULT;
@@ -24703,64 +24647,21 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc
return 0;
}
-static int add_kfunc_desc_target(struct bpf_verifier_env *env,
- const struct bpf_kfunc_desc *target_desc,
- u16 base_desc_idx, u16 *desc_idx)
+/* A kfunc has at most one specialized target. Returns the index of its descriptor. */
+static int add_specialized_kfunc(struct bpf_verifier_env *env, const struct bpf_kfunc_desc *desc)
{
- struct bpf_kfunc_desc desc = *target_desc;
- struct bpf_kfunc_desc_tab *new_tab;
- struct bpf_kfunc_desc_tab *tab;
- struct bpf_prog_aux *prog_aux;
- u32 i;
-
- prog_aux = env->prog->aux;
- tab = prog_aux->kfunc_tab;
-
- if (base_desc_idx >= tab->nr_base_descs) {
- verifier_bug(env, "kfunc desc_idx %d out of bounds (%d descriptors)",
- base_desc_idx, tab->nr_base_descs);
- return -EFAULT;
- }
- if (tab->descs[base_desc_idx].func_id != desc.func_id) {
- verifier_bug(env, "kfunc desc %d has invalid id (expected %d, got %d)",
- base_desc_idx, tab->descs[base_desc_idx].func_id, desc.func_id);
- return -EFAULT;
- }
- if (tab->descs[base_desc_idx].offset != desc.offset) {
- verifier_bug(env, "kfunc desc %d has invalid offset (expected %d, got %d)",
- base_desc_idx, tab->descs[base_desc_idx].offset, desc.offset);
- return -EFAULT;
- }
-
- if (tab->descs[base_desc_idx].addr == desc.addr) {
- *desc_idx = base_desc_idx;
- return 0;
- }
+ struct bpf_kfunc_desc_tab *tab = env->prog->aux->kfunc_tab;
+ u64 key = kfunc_key(desc->func_id, desc->offset, true);
+ int i = bpf_index_find(&env->kfunc_index, key);
- for (i = tab->nr_base_descs; i < tab->nr_descs; i++) {
- if (tab->descs[i].func_id == desc.func_id &&
- tab->descs[i].offset == desc.offset &&
- tab->descs[i].addr == desc.addr) {
- *desc_idx = i;
- return 0;
- }
- }
-
- if (tab->nr_descs == MAX_KFUNC_CALL_DESCS) {
- verbose(env, "too many different kernel function call targets\n");
- return -E2BIG;
- }
-
- new_tab = krealloc(tab, struct_size(tab, descs, tab->nr_descs + 1),
- BPF_VERIFIER_GFP);
- if (!new_tab)
+ if (i >= 0)
+ return verifier_bug_if(tab->descs[i].addr != desc->addr, env,
+ "two targets of kfunc %u", desc->func_id) ? -EFAULT : i;
+ if (!bpf_table_next(tab->descs, tab->nr_descs) ||
+ bpf_index_add(&env->kfunc_index, key, tab->nr_descs))
return -ENOMEM;
- tab = new_tab;
- prog_aux->kfunc_tab = tab;
-
- *desc_idx = tab->nr_descs;
- tab->descs[tab->nr_descs++] = desc;
- return 0;
+ tab->descs[tab->nr_descs] = *desc;
+ return tab->nr_descs++;
}
static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux,
@@ -24786,9 +24687,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct bpf_kfunc_desc desc_copy;
struct bpf_kfunc_desc *desc;
unsigned long call_imm;
- u16 base_desc_idx, desc_idx;
+ int err, desc_idx;
bool near_call;
- int err;
if (!insn->imm) {
verbose(env, "invalid kernel function call not eliminated in verifier pass\n");
@@ -24801,13 +24701,13 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
* __bpf_call_base, unless the JIT needs to call functions that are
* further than 32 bits away (bpf_jit_supports_far_kfunc_call()).
*/
- desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+ desc = find_kfunc_desc(env, insn->imm, insn->off);
if (!desc) {
verifier_bug(env, "kernel function descriptor not found for func_id %u",
insn->imm);
return -EFAULT;
}
- base_desc_idx = desc - env->prog->aux->kfunc_tab->descs;
+ desc_idx = desc - env->prog->aux->kfunc_tab->descs;
near_call = !bpf_jit_supports_far_kfunc_call();
desc_copy = *desc;
@@ -24827,9 +24727,15 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
insn->imm = call_imm;
}
- err = add_kfunc_desc_target(env, desc, base_desc_idx, &desc_idx);
- if (err)
- return err;
+ if (desc->addr != env->prog->aux->kfunc_tab->descs[desc_idx].addr)
+ desc_idx = add_specialized_kfunc(env, desc);
+ if (desc_idx < 0)
+ return desc_idx;
+ /* JITs find the descriptor by insn->off */
+ if (desc_idx > U16_MAX) {
+ verbose(env, "too many different kernel function calls\n");
+ return -E2BIG;
+ }
insn->off = desc_idx;
if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
@@ -25596,6 +25502,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
bpf_patch_list_free(env);
kvfree(env->fd_array);
kvfree(env->used_index.slots);
+ kvfree(env->kfunc_index.slots);
kvfree(env->insn_array_maps);
bpf_free_scev(env);
kvfree(env->walk_insns);
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 10/21] bpf: Remove the limits on tables of functions
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (8 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 09/21] bpf: Remove the limits on kfuncs of a program Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 11/21] libbpf: Size the tables of the loader program from the object Alexei Starovoitov
` (10 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
4096 sets of functions sets are in a hash table now
1024 targets of one load counted before duplicates were removed
4096 pointers near a load the scan is linear and can be killed
1M pointers in all maps the table grows
Nothing sorts the pointers: walk the maps in the order of their addresses,
and collect the targets of a load in a bitmap.
The number of a subprog in a set has 32 bits.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 5 +-
kernel/bpf/verifier.c | 132 +++++++-----------
.../selftests/bpf/progs/verifier_callx_many.c | 31 +++-
3 files changed, 82 insertions(+), 86 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 508aef9bad9d..3b40f66894d9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1332,6 +1332,7 @@ struct bpf_verifier_env {
/* What a pointer that was read from a table of functions may point to. */
struct bpf_func_set **func_sets;
u32 func_set_cnt;
+ struct bpf_index func_set_index;
bool has_callx;
/*
* Call graph edges created by callx instructions. A bitmap of
@@ -1718,8 +1719,8 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
bool bpf_map_is_rdonly(const struct bpf_map *map);
/* Sorted, no function is there twice */
struct bpf_func_set {
- u16 cnt;
- u16 subprog[] __counted_by(cnt);
+ u32 cnt;
+ u32 subprog[] __counted_by(cnt);
};
struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f4e2a9f9f787..c5a44c887ada 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -18,6 +18,7 @@
#include <linux/vmalloc.h>
#include <linux/stringify.h>
#include <linux/bsearch.h>
+#include <linux/jhash.h>
#include <linux/sort.h>
#include <linux/perf_event.h>
#include <linux/ctype.h>
@@ -6503,17 +6504,6 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
return 0;
}
-static int cmp_func_ptrs(const void *_a, const void *_b)
-{
- const struct bpf_func_ptr *a = _a, *b = _b;
-
- if (a->map != b->map)
- return a->map < b->map ? -1 : 1;
- if (a->map_off != b->map_off)
- return a->map_off < b->map_off ? -1 : 1;
- return 0;
-}
-
/* Find the first pointer to a function at or after 'off' in the value of 'map' */
static u32 func_ptr_lower_bound(struct bpf_verifier_env *env, const struct bpf_map *map, u64 off)
{
@@ -7164,14 +7154,6 @@ static void mark_reg_func_ptr(struct bpf_verifier_env *env, struct bpf_reg_state
regs[regno].subprogno = subprog;
}
-static int cmp_u16(const void *a, const void *b)
-{
- return *(const u16 *)a - *(const u16 *)b;
-}
-
-/* that many different sets at most: each load from a table looks through all of them */
-#define BPF_MAX_FUNC_SETS 4096
-
/*
* A load from a table of functions gives one register: a pointer to one of a set of functions.
* One state for each function would differ in nothing else up to the call, and there every
@@ -7179,49 +7161,40 @@ static int cmp_u16(const void *a, const void *b)
* Equal sets have the same number, so registers are compared as they are.
*/
static int mark_reg_func_set(struct bpf_verifier_env *env, struct bpf_reg_state *regs, int regno,
- u16 *subprogs, u32 cnt)
+ const u32 *subprogs, u32 n)
{
- struct bpf_func_set **sets, *set;
- u32 i, n = 0;
+ struct bpf_func_set *set;
+ u64 key;
+ int i;
- sort(subprogs, cnt, sizeof(*subprogs), cmp_u16, NULL);
- for (i = 0; i < cnt; i++)
- if (!n || subprogs[n - 1] != subprogs[i])
- subprogs[n++] = subprogs[i];
mark_reg_func_ptr(env, regs, regno, n == 1 ? subprogs[0] : 0);
if (n == 1)
return 0;
- for (i = 0; i < env->func_set_cnt; i++) {
+ /* sets with the same hash have keys in a row */
+ key = (u64)jhash2(subprogs, n, 0) << 32;
+ for (; (i = bpf_index_find(&env->func_set_index, key)) >= 0; key++) {
set = env->func_sets[i];
if (set->cnt == n && !memcmp(set->subprog, subprogs, n * sizeof(*subprogs)))
break;
}
- if (i == env->func_set_cnt) {
- if (i == BPF_MAX_FUNC_SETS) {
- verbose(env, "more than %d sets of functions\n", BPF_MAX_FUNC_SETS);
- return -E2BIG;
- }
- sets = krealloc_array(env->func_sets, i + 1, sizeof(*sets), BPF_VERIFIER_GFP);
- if (!sets)
- return -ENOMEM;
- env->func_sets = sets;
- set = kzalloc_flex(*set, subprog, n, BPF_VERIFIER_GFP);
- if (!set)
+ if (i < 0) {
+ i = env->func_set_cnt;
+ set = kvzalloc_flex(*set, subprog, n, BPF_VERIFIER_GFP);
+ /* nothing fails between the entry of the index and the set that it stands for */
+ if (!set || !bpf_table_next(env->func_sets, i) ||
+ bpf_index_add(&env->func_set_index, key, i)) {
+ kvfree(set);
return -ENOMEM;
+ }
set->cnt = n;
memcpy(set->subprog, subprogs, n * sizeof(*subprogs));
- sets[env->func_set_cnt++] = set;
+ env->func_sets[env->func_set_cnt++] = set;
}
regs[regno].func_set = i + 1;
return 0;
}
-/* a read from a table of functions yields one of that many at most */
-#define BPF_MAX_FUNC_PTR_TARGETS BPF_MAX_SUBPROGS
-/* and that many pointers at most are next to what is read at variable offset */
-#define BPF_MAX_FUNC_PTR_RANGE 4096
-
/* Count up to 'limit' offsets in [min_off, max_off] that the read is possible at */
static u32 count_read_offsets(struct tnum offs, u64 min_off, u64 max_off, u32 limit)
{
@@ -7263,7 +7236,8 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
struct tnum offs = tnum_add(reg->var_off, tnum_const(off));
struct bpf_reg_state *regs = cur_regs(env);
struct bpf_map *map = reg->map_ptr;
- u16 *targets __free(kfree) = NULL;
+ unsigned long *found __free(kvfree) = NULL;
+ u32 *targets __free(kvfree) = NULL;
struct bpf_func_ptr *ptrs;
u32 i, cnt, n = 0;
u64 o, lo, hi;
@@ -7272,13 +7246,14 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
ptrs = bpf_map_range_func_ptrs(env, map, min_off, max_off - min_off + size, &cnt);
if (!ptrs)
return -ENOENT;
- if (cnt > BPF_MAX_FUNC_PTR_RANGE)
- goto overlap;
- targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), BPF_VERIFIER_GFP);
- if (!targets)
+ targets = kvmalloc_array(min(cnt, env->subprog_cnt), sizeof(*targets), BPF_VERIFIER_GFP);
+ found = kvcalloc(BITS_TO_LONGS(env->subprog_cnt), sizeof(long), BPF_VERIFIER_GFP);
+ if (!targets || !found)
return -ENOMEM;
for (i = 0; i < cnt; i++) {
+ if (bpf_must_abort())
+ return -EINTR;
/* the offsets at which the read overlaps with the pointer */
lo = ptrs[i].map_off >= size ? ptrs[i].map_off - size + 1 : 0;
lo = max(lo, min_off);
@@ -7288,17 +7263,13 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
continue;
if (o != ptrs[i].map_off || size != sizeof(u64) || value_regno < 0)
goto overlap;
- if (n == BPF_MAX_FUNC_PTR_TARGETS) {
- verbose(env, "read from map '%s' may yield more than %d pointers to functions\n",
- map->name, BPF_MAX_FUNC_PTR_TARGETS);
- return -E2BIG;
- }
subprog = bpf_find_subprog(env, ptrs[i].xlated_off);
if (verifier_bug_if(subprog <= 0, env,
"no function at insn %u for map '%s' offset %u",
ptrs[i].xlated_off, map->name, ptrs[i].map_off))
return -EFAULT;
- targets[n++] = subprog;
+ __set_bit(subprog, found);
+ n++;
}
}
/* pointers are next to what is read, e.g. the data of an array of vtables */
@@ -7313,6 +7284,9 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
if (err)
return err;
+ n = 0;
+ for_each_set_bit(subprog, found, env->subprog_cnt)
+ targets[n++] = subprog;
return mark_reg_func_set(env, regs, value_regno, targets, n);
overlap:
@@ -22834,29 +22808,11 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
return claim_used_maps(env);
}
-/* that many pointers to functions are recognized in the maps of a program at most */
-#define BPF_MAX_FUNC_PTRS (1 << 20)
-
static int add_func_ptr(struct bpf_verifier_env *env, struct bpf_map *map, u32 map_off,
u32 xlated_off)
{
- struct bpf_func_ptr *ptrs;
-
- if (env->func_ptr_cnt == BPF_MAX_FUNC_PTRS) {
- verbose(env, "too many pointers to functions in maps, the limit is %d\n",
- BPF_MAX_FUNC_PTRS);
- return -E2BIG;
- }
-
- /* grow by doubling, the array is sorted and searched later */
- if (!(env->func_ptr_cnt & (env->func_ptr_cnt - 1))) {
- ptrs = kvrealloc(env->func_ptrs,
- array_size(max(2 * env->func_ptr_cnt, 16U), sizeof(*ptrs)),
- BPF_VERIFIER_GFP);
- if (!ptrs)
- return -ENOMEM;
- env->func_ptrs = ptrs;
- }
+ if (!bpf_table_next(env->func_ptrs, env->func_ptr_cnt))
+ return -ENOMEM;
env->func_ptrs[env->func_ptr_cnt++] = (struct bpf_func_ptr){
.map = map,
.map_off = map_off,
@@ -22866,6 +22822,11 @@ static int add_func_ptr(struct bpf_verifier_env *env, struct bpf_map *map, u32 m
return 0;
}
+static int cmp_maps(const void *a, const void *b)
+{
+ return cmp_int(*(const unsigned long *)a, *(const unsigned long *)b);
+}
+
/*
* Compilers put pointers to functions into read-only data: tables of functions,
* structures of operations, vtables, where they are mixed with other data.
@@ -22890,17 +22851,24 @@ static int add_func_ptr(struct bpf_verifier_env *env, struct bpf_map *map, u32 m
*/
static int resolve_func_ptrs(struct bpf_verifier_env *env)
{
+ struct bpf_map *map, **maps __free(kvfree) = NULL;
int insn_cnt = env->prog->len;
int i, err, subprog;
- struct bpf_map *map;
u64 addr, val;
u32 off;
- if (!env->has_callx || !env->allow_ptr_leaks)
+ if (!env->has_callx || !env->allow_ptr_leaks || !env->used_map_cnt)
return 0;
+ /* with the maps in order the pointers are in order */
+ maps = kvmalloc_objs(*maps, env->used_map_cnt, BPF_VERIFIER_GFP);
+ if (!maps)
+ return -ENOMEM;
+ memcpy(maps, env->used_maps, env->used_map_cnt * sizeof(*maps));
+ sort(maps, env->used_map_cnt, sizeof(*maps), cmp_maps, NULL);
+
for (i = 0; i < env->used_map_cnt; i++) {
- map = env->used_maps[i];
+ map = maps[i];
/* coincidences in maps that are shared with other programs don't matter */
if (READ_ONCE(map->user) != (unsigned long)env->prog->aux)
continue;
@@ -22926,9 +22894,6 @@ static int resolve_func_ptrs(struct bpf_verifier_env *env)
return err;
}
}
- if (env->func_ptr_cnt)
- sort(env->func_ptrs, env->func_ptr_cnt, sizeof(*env->func_ptrs),
- cmp_func_ptrs, NULL);
return 0;
}
@@ -25516,8 +25481,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->callx_edges);
kvfree(env->func_ptrs);
for (i = 0; i < env->func_set_cnt; i++)
- kfree(env->func_sets[i]);
- kfree(env->func_sets);
+ kvfree(env->func_sets[i]);
+ kvfree(env->func_sets);
+ kvfree(env->func_set_index.slots);
bpf_diag_free(env);
kvfree(env->idoms);
kvfree(env);
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_many.c b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
index 23c8948c24b8..804cdeb04845 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_many.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
@@ -47,7 +47,7 @@ REP400(MANY_FUNC)
#define SETS400(mask) "r7 = r6; r7 &= " #mask "; r7 += r8;" REP400(SET_LOAD)
SEC("socket")
-__failure __msg("more than 4096 sets of functions")
+__success __retval(11)
__naked void callx_rodata_4400_sets(void)
{
asm volatile (
@@ -65,6 +65,35 @@ __naked void callx_rodata_4400_sets(void)
: __clobber_all);
}
+#define TABLE_5000(name, last, mask) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ RODATA(tab, ".rept 2500; .quad many000; .quad many001; .endr;" last) \
+ "call %[bpf_get_prandom_u32];" \
+ "r0 &= 8191;" \
+ "if r0 < " #mask " goto +1;" \
+ "r0 = 0;" \
+ "r0 <<= 3;" \
+ "r2 = tab_%= ll;" \
+ "r2 += r0;" \
+ "r2 = *(u64 *)(r2 + 0);" \
+ "r1 = 10;" \
+ "callx r2;" \
+ "exit;" \
+ : \
+ : __imm(bpf_get_prandom_u32) \
+ : __clobber_all); \
+}
+
+SEC("socket")
+__success __retval(11)
+TABLE_5000(callx_rodata_5000_pointers, "", 5000)
+
+SEC("socket")
+__failure __msg("overlaps with a pointer to a function")
+TABLE_5000(callx_rodata_5000_pointers_and_data, ".quad 1;", 5001)
+
/* 3 states that have no join get to a table of 400 functions: 3 states, and not 1200 */
SEC("socket")
__success __retval(11)
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 11/21] libbpf: Size the tables of the loader program from the object
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (9 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 10/21] bpf: Remove the limits on tables of functions Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 16:06 ` bot+bpf-ci
2026-11-16 15:26 ` [PATCH bpf-next v2 12/21] bpf: Relax jumps that patches push out of range Alexei Starovoitov
` (9 subsequent siblings)
20 siblings, 1 reply; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
gen_loader mirrors limits of the kernel: 64 maps, 32 programs and 256
kfuncs. The numbers are known before the loader is made.
fd_array: [ maps | copies of maps | unused | BTFs of kfuncs ]
prog_fds: [ programs ] was on the stack of the loader
Use gotol and a register for ctx when an offset needs more than 16 bits.
That was an error, so loaders that could be made do not get gotol.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/bpf_gen_internal.h | 3 +-
tools/lib/bpf/gen_loader.c | 157 +++++++++++++------------------
tools/lib/bpf/libbpf.c | 8 +-
3 files changed, 74 insertions(+), 94 deletions(-)
diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h
index 206adf28793d..c30e05b66abc 100644
--- a/tools/lib/bpf/bpf_gen_internal.h
+++ b/tools/lib/bpf/bpf_gen_internal.h
@@ -50,6 +50,7 @@ struct bpf_gen {
struct ksym_desc *ksyms;
__u32 nr_ksyms;
int fd_array;
+ int prog_fds;
int nr_fd_array;
/*
* Maps with pointers to functions, that programs get their own copies
@@ -61,7 +62,7 @@ struct bpf_gen {
};
void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps,
- int max_func_ptr_maps);
+ int max_func_ptr_maps, int nr_kfuncs);
int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps);
void bpf_gen__free(struct bpf_gen *gen);
void bpf_gen__load_btf(struct bpf_gen *gen, const void *raw_data, __u32 raw_size);
diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c
index 555d4757a898..27b4bc9e55ac 100644
--- a/tools/lib/bpf/gen_loader.c
+++ b/tools/lib/bpf/gen_loader.c
@@ -15,11 +15,6 @@
#include "bpf_gen_internal.h"
#include "skel_internal.h"
-#define MAX_USED_MAPS 64
-#define MAX_USED_PROGS 32
-#define MAX_KFUNC_DESCS 256
-#define MAX_FD_ARRAY_SZ (MAX_USED_MAPS + MAX_KFUNC_DESCS)
-
/* The following structure describes the stack layout of the loader program.
* In addition R6 contains the pointer to context.
* R7 contains the result of the last sys_bpf command (typically error or FD).
@@ -34,7 +29,6 @@
struct loader_stack {
__u32 btf_fd;
__u32 inner_map_fd;
- __u32 prog_fd[MAX_USED_PROGS];
};
#define stack_off(field) \
@@ -47,6 +41,16 @@ static int blob_fd_array_off(struct bpf_gen *gen, int index)
return gen->fd_array + index * sizeof(int);
}
+static int blob_prog_fd_off(struct bpf_gen *gen, int index)
+{
+ return gen->prog_fds + index * sizeof(int);
+}
+
+static bool is_simm16(__s64 value)
+{
+ return value == (__s64)(__s16)value;
+}
+
static int realloc_insn_buf(struct bpf_gen *gen, __u32 size)
{
size_t off = gen->insn_cur - gen->insn_start;
@@ -113,20 +117,18 @@ static int add_data(struct bpf_gen *gen, const void *data, __u32 size);
static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off);
void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps,
- int max_func_ptr_maps)
+ int max_func_ptr_maps, int nr_kfuncs)
{
- size_t stack_sz = sizeof(struct loader_stack), nr_progs_sz;
- int i;
+ size_t stack_sz = sizeof(struct loader_stack);
+ int i, cleanup_sz;
- gen->fd_array = add_data(gen, NULL, MAX_FD_ARRAY_SZ * sizeof(int));
+ /* maps | maps with pointers to functions | unused | BTFs of kfuncs, never at index 0 */
+ gen->fd_array = add_data(gen, NULL,
+ (nr_maps + max_func_ptr_maps + 1 + nr_kfuncs) * sizeof(int));
+ gen->prog_fds = add_data(gen, NULL, nr_progs * sizeof(int));
gen->log_level = log_level;
gen->nr_obj_maps = nr_maps;
gen->max_func_ptr_maps = max_func_ptr_maps;
- if (nr_maps + max_func_ptr_maps > MAX_USED_MAPS) {
- pr_warn("Total maps exceeds %d\n", MAX_USED_MAPS);
- gen->error = -E2BIG;
- return;
- }
/* their fds are closed like the fds of the maps of the object when loading fails */
nr_maps += max_func_ptr_maps;
/* save ctx pointer into R6 */
@@ -139,27 +141,24 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps
emit(gen, BPF_MOV64_IMM(BPF_REG_3, 0));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
- /* nr_progs includes subprogs and progs that are not loaded */
- nr_progs = min(nr_progs, MAX_USED_PROGS);
- /* amount of stack actually used, only used to calculate iterations, not stack offset */
- nr_progs_sz = offsetof(struct loader_stack, prog_fd[nr_progs]);
- /* jump over cleanup code */
- emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0,
- /* size of cleanup code below (including map fd cleanup) */
- (nr_progs_sz / 4) * 3 + 2 +
- /* 6 insns for emit_sys_close_blob,
- * 6 insns for debug_regs in emit_sys_close_blob
- */
- nr_maps * (6 + (gen->log_level ? 6 : 0))));
+ /*
+ * jump over cleanup code below:
+ * 6 insns for emit_sys_close_blob,
+ * 6 insns for debug_regs in emit_sys_close_blob
+ */
+ cleanup_sz = (stack_sz / 4) * 3 + 2 + (nr_progs + nr_maps) * (6 + (gen->log_level ? 6 : 0));
+ emit(gen, is_simm16(cleanup_sz) ? BPF_JMP_A(cleanup_sz) : BPF_JMP32_A(cleanup_sz));
/* remember the label where all error branches will jump to */
gen->cleanup_label = gen->insn_cur - gen->insn_start;
/* emit cleanup code: close all temp FDs */
- for (i = 0; i < nr_progs_sz; i += 4) {
+ for (i = 0; i < stack_sz; i += 4) {
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, -stack_sz + i));
emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0, 1));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
}
+ for (i = 0; i < nr_progs; i++)
+ emit_sys_close_blob(gen, blob_prog_fd_off(gen, i));
for (i = 0; i < nr_maps; i++)
emit_sys_close_blob(gen, blob_fd_array_off(gen, i));
/* R7 contains the error code from sys_bpf. Copy it into R0 and exit. */
@@ -192,28 +191,9 @@ static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
return prev - gen->data_start;
}
-/* Get index for map_fd/btf_fd slot in reserved fd_array, or in data relative
- * to start of fd_array. Caller can decide if it is usable or not.
- */
-static int add_map_fd(struct bpf_gen *gen)
-{
- if (gen->nr_maps == MAX_USED_MAPS) {
- pr_warn("Total maps exceeds %d\n", MAX_USED_MAPS);
- gen->error = -E2BIG;
- return 0;
- }
- return gen->nr_maps++;
-}
-
static int add_kfunc_btf_fd(struct bpf_gen *gen)
{
- int cur;
-
- if (gen->nr_fd_array == MAX_KFUNC_DESCS) {
- cur = add_data(gen, NULL, sizeof(int));
- return (cur - gen->fd_array) / sizeof(int);
- }
- return MAX_USED_MAPS + gen->nr_fd_array++;
+ return gen->nr_obj_maps + gen->max_func_ptr_maps + 1 + gen->nr_fd_array++;
}
static int insn_bytes_to_bpf_size(__u32 sz)
@@ -247,18 +227,33 @@ static void move_blob2blob(struct bpf_gen *gen, int off, int size, int blob_off)
emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_1, BPF_REG_0, 0));
}
+/* Returns the register for an access to ctx + *off: R6, or tmp if *off needs more than 16 bits */
+static int emit_ctx_reg(struct bpf_gen *gen, int tmp, int *off)
+{
+ if (is_simm16(*off))
+ return BPF_REG_6;
+ emit(gen, BPF_MOV64_REG(tmp, BPF_REG_6));
+ emit(gen, BPF_ALU64_IMM(BPF_ADD, tmp, *off));
+ *off = 0;
+ return tmp;
+}
+
static void move_blob2ctx(struct bpf_gen *gen, int ctx_off, int size, int blob_off)
{
+ int ctx = emit_ctx_reg(gen, BPF_REG_2, &ctx_off);
+
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
0, 0, 0, blob_off));
emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_1, 0));
- emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_6, BPF_REG_0, ctx_off));
+ emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), ctx, BPF_REG_0, ctx_off));
}
static void move_ctx2blob(struct bpf_gen *gen, int off, int size, int ctx_off,
bool check_non_zero)
{
- emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_6, ctx_off));
+ int ctx = emit_ctx_reg(gen, BPF_REG_0, &ctx_off);
+
+ emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, ctx, ctx_off));
if (check_non_zero)
/* If value in ctx is zero don't update the blob.
* For example: when ctx->map.max_entries == 0, keep default max_entries from bpf.c
@@ -277,12 +272,6 @@ static void move_stack2blob(struct bpf_gen *gen, int off, int size, int stack_of
emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_1, BPF_REG_0, 0));
}
-static void move_stack2ctx(struct bpf_gen *gen, int ctx_off, int size, int stack_off)
-{
- emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_10, stack_off));
- emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_6, BPF_REG_0, ctx_off));
-}
-
static void emit_sys_bpf(struct bpf_gen *gen, int cmd, int attr, int attr_size)
{
emit(gen, BPF_MOV64_IMM(BPF_REG_1, cmd));
@@ -294,11 +283,6 @@ static void emit_sys_bpf(struct bpf_gen *gen, int cmd, int attr, int attr_size)
emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
}
-static bool is_simm16(__s64 value)
-{
- return value == (__s64)(__s16)value;
-}
-
static void emit_check_err(struct bpf_gen *gen)
{
__s64 off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 1;
@@ -309,7 +293,8 @@ static void emit_check_err(struct bpf_gen *gen)
if (is_simm16(off)) {
emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
} else {
- gen->error = -ERANGE;
+ emit(gen, BPF_JMP_IMM(BPF_JSGE, BPF_REG_7, 0, 1));
+ emit(gen, BPF_JMP32_A(off - 1));
}
}
@@ -401,12 +386,12 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps)
for (i = 0; i < gen->nr_func_ptr_maps; i++)
emit_sys_close_blob(gen, blob_fd_array_off(gen, gen->nr_obj_maps + i));
for (i = 0; i < gen->nr_progs; i++)
- move_stack2ctx(gen,
- sizeof(struct bpf_loader_ctx) +
- sizeof(struct bpf_map_desc) * gen->nr_maps +
- sizeof(struct bpf_prog_desc) * i +
- offsetof(struct bpf_prog_desc, prog_fd), 4,
- stack_off(prog_fd[i]));
+ move_blob2ctx(gen,
+ sizeof(struct bpf_loader_ctx) +
+ sizeof(struct bpf_map_desc) * gen->nr_maps +
+ sizeof(struct bpf_prog_desc) * i +
+ offsetof(struct bpf_prog_desc, prog_fd), 4,
+ blob_prog_fd_off(gen, i));
for (i = 0; i < gen->nr_maps; i++)
move_blob2ctx(gen,
sizeof(struct bpf_loader_ctx) +
@@ -578,8 +563,7 @@ void bpf_gen__map_create(struct bpf_gen *gen,
gen->error = -EDOM; /* internal bug */
return;
} else {
- /* add_map_fd does gen->nr_maps++ */
- idx = add_map_fd(gen);
+ idx = gen->nr_maps++;
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
0, 0, 0, blob_fd_array_off(gen, idx)));
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_7, 0));
@@ -720,11 +704,7 @@ static void emit_bpf_kallsyms_lookup_name(struct bpf_gen *gen, struct ksym_relo_
* BPF_REG_8 - pointer to instruction
*
* We need to reuse BTF fd for same symbol otherwise each relocation takes a new
- * index, while kernel limits total kfunc BTFs to 256. For duplicate symbols,
- * this would mean a new BTF fd index for each entry. By pairing symbol name
- * with index, we get the insn->imm, insn->off pairing that kernel uses for
- * kfunc_tab, which becomes the effective limit even though all of them may
- * share same index in fd_array (such that kfunc_btf_tab has 1 element).
+ * index in fd_array, which has one slot for each symbol.
*/
static void emit_relo_kfunc_btf(struct bpf_gen *gen, struct ksym_relo_desc *relo, int insn)
{
@@ -837,7 +817,8 @@ static void emit_relo_ksym_typeless(struct bpf_gen *gen,
/* skip typeless ksym_desc in fd closing loop in cleanup_relos */
kdesc->typeless = true;
emit_bpf_kallsyms_lookup_name(gen, relo);
- emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_7, -ENOENT, 1));
+ emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_7, -ENOENT, 1));
+ emit(gen, BPF_MOV64_IMM(BPF_REG_7, 0));
emit_check_err(gen);
/* store lower half of addr into insn[insn_idx].imm */
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_8, BPF_REG_9, offsetof(struct bpf_insn, imm)));
@@ -979,8 +960,7 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
emit_sys_close_blob(gen, insn);
} else if (!kdesc->is_ld64) {
emit_sys_close_blob(gen, blob_fd_array_off(gen, kdesc->off));
- if (kdesc->off < MAX_FD_ARRAY_SZ)
- gen->nr_fd_array--;
+ gen->nr_fd_array--;
}
}
if (gen->nr_ksyms) {
@@ -1136,14 +1116,10 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
gen->attach_kind = 0;
}
emit_check_err(gen);
- if (gen->nr_progs == MAX_USED_PROGS) {
- pr_warn("Total programs exceeds %d\n", MAX_USED_PROGS);
- gen->error = -E2BIG;
- return;
- }
- /* remember prog_fd in the stack, if successful */
- emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
- stack_off(prog_fd[gen->nr_progs])));
+ /* remember prog_fd in the blob, if successful */
+ emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
+ 0, 0, 0, blob_prog_fd_off(gen, gen->nr_progs)));
+ emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_7, 0));
gen->nr_progs++;
}
@@ -1159,10 +1135,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
static void emit_copy_initial_value(struct bpf_gen *gen, int map_idx, int value,
__u32 value_size, int nr_more_insns)
{
- emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
- sizeof(struct bpf_loader_ctx) +
- sizeof(struct bpf_map_desc) * map_idx +
- offsetof(struct bpf_map_desc, initial_value)));
+ int off = sizeof(struct bpf_loader_ctx) + sizeof(struct bpf_map_desc) * map_idx +
+ offsetof(struct bpf_map_desc, initial_value);
+ int ctx = emit_ctx_reg(gen, BPF_REG_3, &off);
+
+ emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, ctx, off));
emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8 + nr_more_insns));
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
0, 0, 0, value));
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 44134421bd7d..1efc0dd37b2e 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -9909,7 +9909,7 @@ static int bpf_object_load(struct bpf_object *obj, int extra_log_level, const ch
* permit cross-endian creation of "light skeleton".
*/
if (obj->gen_loader) {
- int nr_func_ptr_maps = 0, nr_progs = 0, i;
+ int nr_func_ptr_maps = 0, nr_progs = 0, nr_entries = 0, i;
bool text_has_callx = false;
/*
@@ -9929,8 +9929,10 @@ static int bpf_object_load(struct bpf_object *obj, int extra_log_level, const ch
!prog_is_subprog(obj, &obj->programs[i]) &&
(text_has_callx || prog_has_callx(&obj->programs[i])))
nr_progs++;
- bpf_gen__init(obj->gen_loader, obj->log_level | extra_log_level,
- obj->nr_programs, obj->nr_maps, nr_func_ptr_maps * nr_progs);
+ for (i = 0; i < obj->nr_programs; i++)
+ nr_entries += !prog_is_subprog(obj, &obj->programs[i]);
+ bpf_gen__init(obj->gen_loader, obj->log_level | extra_log_level, nr_entries,
+ obj->nr_maps, nr_func_ptr_maps * nr_progs, obj->nr_extern);
} else if (!is_native_endianness(obj)) {
pr_warn("object '%s': loading non-native endianness is unsupported\n", obj->name);
return libbpf_err(-LIBBPF_ERRNO__ENDIAN);
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 12/21] bpf: Relax jumps that patches push out of range
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (10 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 11/21] libbpf: Size the tables of the loader program from the object Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 13/21] bpf: Keep the call graph in lists of edges Alexei Starovoitov
` (8 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A jump has 16 bits. Patches grow the code under it: inlined lookups,
checks of divisions, may_goto, guards of loops, constant blinding.
A verified program fails with -ERANGE: 2979 lookups under one branch
are enough.
Let bpf_patch_list_build() use gotol for such a jump:
if r1 > r2 goto far -> if r1 > r2 goto +1
goto +1
gotol far
The same form works for may_goto and unwind_goto. An image where all
jumps fit does not change.
The passes do not compute offsets any more:
- the last insn of a patch stands for the patched insn and keeps
its target;
- bpf_patch_list_jump() gives a target in the old image to an insn.
may_goto_off(), bpf_adj_moved_insn() and BPF_GUARD_NEAR go away. All
guards that leave a subprog share one 'r0 = 0; exit'.
x86-32, riscv32, parisc and sparc64 have no gotol in the JIT. They fail
on the unknown opcode, as they do when gotol is in the program. There
a guard that is more than 32K insns away from the end of its subprog
had its own exit and was JITed. It is not JITed now.
A jump to itself is a jump like any other. It goes to the start of its
patch: 'nospec; goto -1' in sanitized dead code is 'nospec; goto -2' now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 11 +-
kernel/bpf/core.c | 335 +++++++-----
kernel/bpf/fixups.c | 502 +++++-------------
.../{blinding_log.c => blinding_far_jump.c} | 38 +-
.../selftests/bpf/progs/verifier_and.c | 2 +-
.../selftests/bpf/progs/verifier_bounds.c | 14 +-
.../selftests/bpf/progs/verifier_movsx.c | 4 +-
.../selftests/bpf/progs/verifier_patch_list.c | 146 ++++-
.../selftests/bpf/progs/verifier_unpriv.c | 2 +-
9 files changed, 484 insertions(+), 570 deletions(-)
rename tools/testing/selftests/bpf/prog_tests/{blinding_log.c => blinding_far_jump.c} (51%)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 3b40f66894d9..28104dae5f08 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1208,6 +1208,10 @@ struct bpf_insn_patch {
u32 len;
u32 start; /* first insn of the patch in bpf_patch_list.insns */
u32 target; /* insn of the patch that jumps to 'off' land on */
+ u32 self; /* insn of the patch that stands for insn 'off' */
+ u32 jmp; /* 1 + insn of the patch that goes to insn 'jmp_to', 0 if none */
+ u32 jmp_to;
+ u32 jmp_back; /* or to that many insns in front of the patch of it */
struct bpf_insn orig; /* the replaced insn when the patch was queued */
};
@@ -1230,7 +1234,9 @@ struct bpf_patch_image {
u32 *new_off;
u32 *new_tgt;
u32 *new_linfo;
+ u32 *relaxed; /* insn before patch_image_relax() -> insn after it */
u32 cnt;
+ u32 short_cnt; /* insns before patch_image_relax() */
u32 new_cnt;
u32 nr_linfo;
s32 grow;
@@ -1256,7 +1262,8 @@ struct bpf_insn *__bpf_patch_list_add(struct bpf_patch_list *pl,
int __bpf_patch_list_del(struct bpf_patch_list *pl, const struct bpf_prog *prog,
u32 off);
int __bpf_patch_list_insert(struct bpf_patch_list *pl, const struct bpf_prog *prog,
- u32 off, const struct bpf_insn *insns, u32 cnt);
+ u32 off, const struct bpf_insn *insns, u32 cnt, bool behind);
+void bpf_patch_list_jump(struct bpf_patch_list *pl, const struct bpf_insn *insn, u32 to, u32 back);
void bpf_patch_list_release(struct bpf_patch_list *pl);
int bpf_patch_list_build(const struct bpf_patch_list *pl, const struct bpf_prog *prog,
struct bpf_patch_image *img);
@@ -2266,6 +2273,8 @@ static inline void bpf_patch_list_free(struct bpf_verifier_env *env)
}
#endif
int bpf_patch_list_del(struct bpf_verifier_env *env, u32 off);
+int bpf_patch_list_append(struct bpf_verifier_env *env, u32 off,
+ const struct bpf_insn *insns, u32 cnt);
int bpf_patch_list_insert(struct bpf_verifier_env *env, u32 off,
const struct bpf_insn *insns, u32 cnt);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 3385df367b05..147261bf9e6d 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -374,11 +374,7 @@ static int patch_list_reserve_insns(struct bpf_patch_list *pl, u32 cnt)
return 0;
}
-/*
- * Extract the pc-relative operand of a jump, a subprog call or a ld_imm64 of a
- * subprog. gotox and a tail call pass for a jump to the next insn: nothing
- * reads their 'off'.
- */
+/* Extract the pc-relative operand of a jump, a subprog call or a ld_imm64 of a subprog. */
static bool insn_get_rel(const struct bpf_insn *insn, s64 *rel, bool *is_imm)
{
u8 code = insn->code;
@@ -389,7 +385,8 @@ static bool insn_get_rel(const struct bpf_insn *insn, s64 *rel, bool *is_imm)
return true;
}
if ((BPF_CLASS(code) != BPF_JMP && BPF_CLASS(code) != BPF_JMP32) ||
- BPF_OP(code) == BPF_CALL || BPF_OP(code) == BPF_EXIT)
+ BPF_OP(code) == BPF_CALL || BPF_OP(code) == BPF_EXIT ||
+ code == (BPF_JMP | BPF_TAIL_CALL) || code == (BPF_JMP | BPF_JA | BPF_X))
return false;
*is_imm = code == (BPF_JMP32 | BPF_JA);
@@ -397,21 +394,6 @@ static bool insn_get_rel(const struct bpf_insn *insn, s64 *rel, bool *is_imm)
return true;
}
-/* Set the pc-relative operand that insn_get_rel() extracted. */
-static int insn_set_rel(struct bpf_insn *insn, s64 rel, bool is_imm)
-{
- if (is_imm) {
- if (rel < S32_MIN || rel > S32_MAX)
- return -ERANGE;
- insn->imm = rel;
- } else {
- if (rel < S16_MIN || rel > S16_MAX)
- return -ERANGE;
- insn->off = rel;
- }
- return 0;
-}
-
/* Whether one of the 'len' insns of a patch in front of the last one jumps past it. */
static bool patch_list_jumps_past(const struct bpf_insn *insns, u32 len)
{
@@ -425,72 +407,53 @@ static bool patch_list_jumps_past(const struct bpf_insn *insns, u32 len)
return false;
}
-/*
- * The 'len' insns of a patch were moved down by 'by' insns inside it. Keep
- * what they referred to in front of the patch, the same way as
- * bpf_adj_moved_insn() does: a call to the start of the patch, the old start
- * of a subprog, stays there, a jump to itself doesn't.
- */
-static int patch_list_move_down(struct bpf_insn *insns, u32 len, u32 by)
-{
- bool is_imm, call;
- s64 rel, tgt;
- u32 k;
- int err;
-
- for (k = 0; k < len; k++) {
- if (!insn_get_rel(&insns[k], &rel, &is_imm))
- continue;
- call = bpf_pseudo_func(&insns[k]) || bpf_pseudo_call(&insns[k]);
- tgt = (s64)k + rel + 1;
- if (tgt > 0 || (tgt == 0 && !call))
- continue;
- err = insn_set_rel(&insns[k], rel - by, is_imm);
- if (err)
- return err;
- }
- return 0;
-}
+#define NO_JUMP U32_MAX
/*
- * insn is number 'idx' of the 'len' insns that replace insn 'off' of the old
- * image (an insn that isn't patched is its own patch of length 1). It lands
- * at 'pos' of the new image. 'new_off' maps the old image to the start of
- * each patch, which calls go to, 'new_tgt' to where jumps go.
+ * Where insn 'idx' of patch 'p' goes in the new image. Calls go to the start
+ * of a patch, img->new_off[], jumps to img->new_tgt[].
*/
-static int patch_list_adj_insn(struct bpf_insn *insn, const u32 *new_off,
- const u32 *new_tgt, u32 cnt, u32 off, u32 idx,
- u32 len, u32 pos)
+static int patch_insn_target(const struct bpf_patch_image *img, const struct bpf_insn_patch *p,
+ const struct bpf_insn *insn, u32 idx, u32 *pos)
{
- bool is_imm, call;
+ bool is_imm, call = bpf_pseudo_func(insn) || bpf_pseudo_call(insn);
s64 rel, tgt;
+ *pos = NO_JUMP;
if (!insn_get_rel(insn, &rel, &is_imm))
return 0;
- call = bpf_pseudo_func(insn) || bpf_pseudo_call(insn);
- /* relative to the first insn of the patch */
- tgt = (s64)idx + rel + 1;
- if (tgt >= 0 && tgt < len)
- return 0;
-
- /* the patch pushed what follows 'off' down by len - 1 */
- tgt = tgt < 0 ? off + tgt : off + tgt - len + 1;
- if (tgt < 0 || tgt > cnt)
+ if (idx + 1 == p->jmp) {
+ tgt = p->jmp_to;
+ } else if (idx == p->self) {
+ tgt = p->off + rel + 1;
+ } else {
+ tgt = idx + rel + 1;
+ if (tgt < 0)
+ return -EFAULT;
+ *pos = img->new_off[p->off] + tgt;
+ if (tgt < p->len)
+ return 0;
+ /* the patch pushed what follows 'off' down by len - 1 */
+ tgt += (s64)p->off - p->len + 1;
+ }
+ if (tgt < 0 || tgt > img->cnt)
return -EFAULT;
- rel = (s64)(call ? new_off : new_tgt)[tgt] - pos - 1;
- return insn_set_rel(insn, rel, is_imm);
+ *pos = (call ? img->new_off : img->new_tgt)[tgt];
+ if (idx + 1 == p->jmp && p->jmp_back)
+ *pos = img->new_off[tgt] - p->jmp_back;
+ return 0;
}
/*
* Queue the replacement of insn 'off' of 'prog' by 'patch[0..len)'. Patches
* are queued in increasing order of 'off' and applied together, so a pass
- * keeps addressing insns by their index in 'prog'. Jumps in the patch that
- * leave it are relative to the patch placed at 'off'. Patching 'off' again
- * replaces the last insn of its patch, where the insn being patched went, and
- * the insns in front of it must not jump past it. Jumps of the new insns that
- * leave the patch are relative to the first of them, as for a first patch.
+ * keeps addressing insns by their index in 'prog'. The last insn of the patch
+ * stands for insn 'off': a copy of a jump or a call goes where it went. The
+ * others jump inside of the patch, ahead, or see bpf_patch_list_jump().
+ * Patching 'off' again replaces the last insn of its patch, and the insns in
+ * front of it must not jump past it.
*
* Return the copy of the last insn of the patch, which stays valid until the
* next call.
@@ -505,18 +468,16 @@ struct bpf_insn *__bpf_patch_list_add(struct bpf_patch_list *pl,
int err;
if (p && off == p->off) {
- if (WARN_ON_ONCE(patch_list_jumps_past(pl->insns + p->start, p->len)))
+ if (WARN_ON_ONCE(p->self != p->len - 1 ||
+ patch_list_jumps_past(pl->insns + p->start, p->len)))
return ERR_PTR(-EFAULT);
err = patch_list_reserve_insns(pl, len - 1);
if (err)
return ERR_PTR(err);
/* the last patch is at the end of pl->insns */
memcpy(pl->insns + p->start + p->len - 1, patch, len * sizeof(*patch));
- err = patch_list_move_down(pl->insns + p->start + p->len - 1, len,
- p->len - 1);
- if (err)
- return ERR_PTR(err);
p->len += len - 1;
+ p->self = p->len - 1;
pl->insn_cnt += len - 1;
return &pl->insns[p->start + p->len - 1];
}
@@ -526,11 +487,13 @@ struct bpf_insn *__bpf_patch_list_add(struct bpf_patch_list *pl,
return ERR_PTR(err);
p = &pl->patches[pl->cnt++];
- p->off = off;
- p->len = len;
- p->target = 0;
- p->start = pl->insn_cnt;
- p->orig = prog->insnsi[off];
+ *p = (struct bpf_insn_patch){
+ .off = off,
+ .len = len,
+ .self = len - 1,
+ .start = pl->insn_cnt,
+ .orig = prog->insnsi[off],
+ };
memcpy(pl->insns + p->start, patch, len * sizeof(*patch));
pl->insn_cnt += len;
@@ -541,19 +504,11 @@ struct bpf_insn *__bpf_patch_list_add(struct bpf_patch_list *pl,
int __bpf_patch_list_del(struct bpf_patch_list *pl, const struct bpf_prog *prog,
u32 off)
{
- struct bpf_insn_patch *p;
- int err;
+ int err = patch_list_reserve_patch(pl);
- err = patch_list_reserve_patch(pl);
if (err)
return err;
-
- p = &pl->patches[pl->cnt++];
- p->off = off;
- p->len = 0;
- p->target = 0;
- p->start = pl->insn_cnt;
- p->orig = prog->insnsi[off];
+ pl->patches[pl->cnt++] = (struct bpf_insn_patch){ .off = off };
return 0;
}
@@ -564,24 +519,20 @@ int __bpf_patch_list_del(struct bpf_patch_list *pl, const struct bpf_prog *prog,
* at the first inserted insn. Inserting in front of a queued patch puts the
* insns in front of it. 'off' is the start of a subprog, so nothing falls
* through to the inserted insns.
+ * Or queue them to go 'behind' insn 'off'. That is the last patch of 'off'.
*/
int __bpf_patch_list_insert(struct bpf_patch_list *pl, const struct bpf_prog *prog,
- u32 off, const struct bpf_insn *insns, u32 cnt)
+ u32 off, const struct bpf_insn *insns, u32 cnt, bool behind)
{
struct bpf_insn_patch *p = pl->cnt ? &pl->patches[pl->cnt - 1] : NULL;
+ struct bpf_insn *insn;
int err;
if (!p || off != p->off) {
- err = patch_list_reserve_patch(pl) ?: patch_list_reserve_insns(pl, 1);
- if (err)
- return err;
- p = &pl->patches[pl->cnt++];
- p->off = off;
- p->len = 1;
- p->target = 0;
- p->start = pl->insn_cnt;
- p->orig = prog->insnsi[off];
- pl->insns[pl->insn_cnt++] = p->orig;
+ insn = __bpf_patch_list_add(pl, prog, off, &prog->insnsi[off], 1);
+ if (IS_ERR(insn))
+ return PTR_ERR(insn);
+ p = &pl->patches[pl->cnt - 1];
}
err = patch_list_reserve_insns(pl, cnt);
@@ -589,18 +540,36 @@ int __bpf_patch_list_insert(struct bpf_patch_list *pl, const struct bpf_prog *pr
return err;
/* the last patch is at the end of pl->insns */
- err = patch_list_move_down(pl->insns + p->start, p->len, cnt);
- if (err)
- return err;
- memmove(pl->insns + p->start + cnt, pl->insns + p->start,
- p->len * sizeof(*insns));
- memcpy(pl->insns + p->start, insns, cnt * sizeof(*insns));
+ insn = pl->insns + p->start;
+ if (behind) {
+ insn += p->len;
+ } else {
+ memmove(insn + cnt, insn, p->len * sizeof(*insns));
+ p->target += cnt;
+ p->self += cnt;
+ p->jmp += p->jmp ? cnt : 0;
+ }
+ memcpy(insn, insns, cnt * sizeof(*insns));
p->len += cnt;
- p->target += cnt;
pl->insn_cnt += cnt;
return 0;
}
+/*
+ * 'insn' of the last patch goes to insn 'to' of the old image, however far that is.
+ * With 'back' it goes to the last insns of the patch of insn 'to - 1'.
+ * A patch has one such jump.
+ */
+void bpf_patch_list_jump(struct bpf_patch_list *pl, const struct bpf_insn *insn, u32 to, u32 back)
+{
+ struct bpf_insn_patch *p = &pl->patches[pl->cnt - 1];
+
+ WARN_ON_ONCE(p->jmp);
+ p->jmp = insn - &pl->insns[p->start] + 1;
+ p->jmp_to = to;
+ p->jmp_back = back;
+}
+
void bpf_patch_list_release(struct bpf_patch_list *pl)
{
kvfree(pl->patches);
@@ -640,19 +609,100 @@ static int patch_image_map_linfo(struct bpf_patch_image *img, const struct bpf_p
return 0;
}
+static bool is_short_jump(const struct bpf_insn *insn)
+{
+ bool is_imm;
+ s64 rel;
+
+ return insn_get_rel(insn, &rel, &is_imm) && !is_imm;
+}
+
+/*
+ * Some jump is out of its 16-bit range. 'tgt' has the targets of all jumps.
+ *
+ * goto far -> gotol far
+ *
+ * if r1 > r2 -> if r1 > r2 goto +1 also may_goto, unwind_goto
+ * goto far goto +1
+ * gotol far
+ *
+ * That moves the targets of other jumps away. One pass is enough: take each
+ * jump for far that would be far with all jumps that it goes over relaxed.
+ */
+static int patch_image_relax(struct bpf_patch_image *img, u32 *tgt)
+{
+ u32 cnt = img->new_cnt, pos, to, lo, hi, n, *map;
+ unsigned long *far __free(kvfree) = NULL;
+ struct bpf_insn *insns = img->insns;
+
+ far = kvcalloc(BITS_TO_LONGS(cnt), sizeof(long), BPF_VERIFIER_GFP);
+ map = img->relaxed = kvmalloc_array(cnt + 1, sizeof(u32), BPF_VERIFIER_GFP);
+ if (!far || !map)
+ return -ENOMEM;
+
+ for (pos = 0, n = 0; pos <= cnt; pos++) {
+ map[pos] = pos + n;
+ n += pos < cnt && is_short_jump(&insns[pos]) ? 2 : 0;
+ }
+ for (pos = 0; pos < cnt; pos++) {
+ if (!is_short_jump(&insns[pos]))
+ continue;
+ lo = min(pos, tgt[pos]);
+ hi = max(pos + 1, tgt[pos]);
+ if (map[hi] - map[lo] <= S16_MAX)
+ continue;
+ if (insns[pos].code == (BPF_JMP | BPF_JA))
+ insns[pos] = BPF_JMP32_A(0);
+ else
+ __set_bit(pos, far);
+ }
+ for (pos = 0, n = 0; pos <= cnt; pos++) {
+ map[pos] = pos + n;
+ n += pos < cnt && test_bit(pos, far) ? 2 : 0;
+ }
+
+ /* like kvmalloc(): the size of a prog fits into 32 bits */
+ if (map[cnt] > INT_MAX / sizeof(*insns))
+ return -ENOMEM;
+ insns = kvrealloc(insns, array_size(map[cnt], sizeof(*insns)), BPF_VERIFIER_GFP);
+ if (!insns)
+ return -ENOMEM;
+ img->insns = insns;
+ for (pos = cnt; pos--;) {
+ to = map[pos];
+ insns[to] = insns[pos];
+ if (!test_bit(pos, far))
+ continue;
+ insns[to].off = 1;
+ insns[to + 1] = BPF_JMP_A(1);
+ insns[to + 2] = BPF_JMP32_A(map[tgt[pos]] - to - 3);
+ tgt[pos] = NO_JUMP;
+ }
+ for (pos = 0; pos <= img->cnt; pos++) {
+ img->new_off[pos] = map[img->new_off[pos]];
+ img->new_tgt[pos] = map[img->new_tgt[pos]];
+ }
+ img->grow += map[cnt] - cnt;
+ img->new_cnt = map[cnt];
+ return 0;
+}
+
/*
* Build the image of 'prog' with the queued patches applied in img->insns,
* without touching 'prog'. img->new_off[] maps each old insn to the start of
* its patch, img->new_tgt[] to where jumps to it land, both map a removed insn
* to the next one that stays. img->new_linfo[] is there when insns are
- * removed, see patch_image_map_linfo(). On -ERANGE, img->bad is the old insn
- * whose jump got out of range. No list has both removed and inserted insns.
+ * removed, see patch_image_map_linfo(). On -EFAULT, img->bad is the old insn
+ * whose jump is wrong. No list has both removed and inserted insns.
*/
int bpf_patch_list_build(const struct bpf_patch_list *pl, const struct bpf_prog *prog,
struct bpf_patch_image *img)
{
- u32 cnt = prog->len, i, k, n;
+ u32 cnt = prog->len, i, k, n, pos, *map;
+ u32 *tgt __free(kvfree) = NULL;
+ bool far = false;
int err;
+ s64 rel;
memset(img, 0, sizeof(*img));
img->cnt = cnt;
@@ -670,7 +720,7 @@ int bpf_patch_list_build(const struct bpf_patch_list *pl, const struct bpf_prog
img->grow += pl->patches[k++].len - 1;
}
}
- img->new_cnt = cnt + img->grow;
+ img->short_cnt = img->new_cnt = cnt + img->grow;
if (img->del && prog->aux->nr_linfo) {
err = patch_image_map_linfo(img, prog);
@@ -679,34 +729,53 @@ int bpf_patch_list_build(const struct bpf_patch_list *pl, const struct bpf_prog
}
img->insns = kvmalloc_array(img->new_cnt, sizeof(*img->insns), BPF_VERIFIER_GFP);
- if (!img->insns)
+ tgt = kvmalloc_array(img->new_cnt, sizeof(*tgt), BPF_VERIFIER_GFP);
+ if (!img->insns || !tgt)
return -ENOMEM;
- for (i = 0, k = 0; i < cnt; i++) {
+ for (i = 0, k = 0, pos = 0; i < cnt; i++) {
+ struct bpf_insn_patch none = { .off = i, .len = 1 };
const struct bpf_insn *src = &prog->insnsi[i];
- u32 len = 1, pos = img->new_off[i];
+ const struct bpf_insn_patch *p = &none;
if (k < pl->cnt && pl->patches[k].off == i) {
- src = &pl->insns[pl->patches[k].start];
- len = pl->patches[k++].len;
+ p = &pl->patches[k++];
+ src = &pl->insns[p->start];
}
- for (n = 0; n < len; n++) {
- img->insns[pos + n] = src[n];
- err = patch_list_adj_insn(&img->insns[pos + n], img->new_off,
- img->new_tgt, cnt, i, n, len, pos + n);
+ for (n = 0; n < p->len; n++, pos++) {
+ img->insns[pos] = src[n];
+ err = patch_insn_target(img, p, &src[n], n, &tgt[pos]);
if (err) {
img->bad = i;
return err;
}
+ rel = (s64)tgt[pos] - pos - 1;
+ far |= is_short_jump(&src[n]) && rel != (s16)rel;
}
}
+
+ err = far ? patch_image_relax(img, tgt) : 0;
+ if (err)
+ return err;
+
+ for (i = 0, map = img->relaxed; i < img->short_cnt; i++) {
+ if (tgt[i] == NO_JUMP)
+ continue;
+ pos = map ? map[i] : i;
+ rel = (s64)(map ? map[tgt[i]] : tgt[i]) - pos - 1;
+ if (is_short_jump(&img->insns[pos]))
+ img->insns[pos].off = rel;
+ else
+ img->insns[pos].imm = rel;
+ }
return 0;
}
void bpf_patch_image_free(struct bpf_patch_image *img)
{
kvfree(img->insns);
+ kvfree(img->relaxed);
kvfree(img->new_linfo);
kvfree(img->new_tgt);
kvfree(img->new_off);
@@ -1576,7 +1645,6 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
{
struct bpf_insn *to = to_buff;
u32 imm_rnd = get_random_u32();
- int off;
BUILD_BUG_ON(BPF_REG_PARAMS + 3 != MAX_BPF_JIT_REG);
BUILD_BUG_ON(BPF_REG_ARENA_SP + 2 != MAX_BPF_JIT_REG);
@@ -1649,15 +1717,9 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
case BPF_JMP | BPF_JSGE | BPF_K:
case BPF_JMP | BPF_JSLE | BPF_K:
case BPF_JMP | BPF_JSET | BPF_K:
- /* Accommodate for extra offset in case of a backjump. */
- off = from->off;
- if (off < 0)
- off -= 2;
- if (off < S16_MIN)
- return -ERANGE;
*to++ = BPF_ALU64_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ from->imm);
*to++ = BPF_ALU64_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
- *to++ = BPF_JMP_REG(from->code, from->dst_reg, BPF_REG_AX, off);
+ *to++ = BPF_JMP_REG(from->code, from->dst_reg, BPF_REG_AX, from->off);
break;
case BPF_JMP32 | BPF_JEQ | BPF_K:
@@ -1671,16 +1733,9 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
case BPF_JMP32 | BPF_JSGE | BPF_K:
case BPF_JMP32 | BPF_JSLE | BPF_K:
case BPF_JMP32 | BPF_JSET | BPF_K:
- /* Accommodate for extra offset in case of a backjump. */
- off = from->off;
- if (off < 0)
- off -= 2;
- if (off < S16_MIN)
- return -ERANGE;
*to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ from->imm);
*to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
- *to++ = BPF_JMP32_REG(from->code, from->dst_reg, BPF_REG_AX,
- off);
+ *to++ = BPF_JMP32_REG(from->code, from->dst_reg, BPF_REG_AX, from->off);
break;
case BPF_LD | BPF_IMM | BPF_DW:
@@ -1880,10 +1935,6 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bp
if (!rewritten)
continue;
- if (rewritten < 0) {
- err = rewritten;
- break;
- }
last = env ? bpf_patch_list_add(env, i, insn_buff, rewritten) :
__bpf_patch_list_add(&pl, clone, i, insn_buff, rewritten);
if (IS_ERR(last)) {
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index e05d767ab71e..3daa419f75a9 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -198,7 +198,6 @@ struct bpf_insn *bpf_patch_list_add(struct bpf_verifier_env *env, u32 off,
return insn;
}
- /* -ERANGE takes a second patch with a 16-bit jump back out of it: no pass has one */
return __bpf_patch_list_add(pl, env->prog, off, patch, len);
}
@@ -218,13 +217,11 @@ int bpf_patch_list_del(struct bpf_verifier_env *env, u32 off)
return __bpf_patch_list_del(pl, env->prog, off);
}
-/* Queue insns to go in front of insn 'off', see __bpf_patch_list_insert(). */
-int bpf_patch_list_insert(struct bpf_verifier_env *env, u32 off,
- const struct bpf_insn *insns, u32 cnt)
+static int patch_list_insert(struct bpf_verifier_env *env, u32 off,
+ const struct bpf_insn *insns, u32 cnt, bool behind)
{
struct bpf_patch_list *pl = &env->patch_list;
struct bpf_insn_patch *p = pl->cnt ? &pl->patches[pl->cnt - 1] : NULL;
- int err;
if (bpf_must_abort())
return -EINTR;
@@ -234,11 +231,21 @@ int bpf_patch_list_insert(struct bpf_verifier_env *env, u32 off,
env, "insns inserted at %u out of order", off))
return -EFAULT;
- err = __bpf_patch_list_insert(pl, env->prog, off, insns, cnt);
- if (err == -ERANGE)
- verbose(env, "insn %d cannot be patched due to 16-bit range\n",
- env->insn_aux_data[off].orig_idx);
- return err;
+ return __bpf_patch_list_insert(pl, env->prog, off, insns, cnt, behind);
+}
+
+/* Queue insns to go in front of insn 'off', see __bpf_patch_list_insert(). */
+int bpf_patch_list_insert(struct bpf_verifier_env *env, u32 off,
+ const struct bpf_insn *insns, u32 cnt)
+{
+ return patch_list_insert(env, off, insns, cnt, false);
+}
+
+/* Queue insns to go behind insn 'off', which does not fall through. They are not dead code. */
+int bpf_patch_list_append(struct bpf_verifier_env *env, u32 off,
+ const struct bpf_insn *insns, u32 cnt)
+{
+ return patch_list_insert(env, off, insns, cnt, true);
}
void bpf_patch_list_free(struct bpf_verifier_env *env)
@@ -255,20 +262,24 @@ static void patch_list_adj_aux(struct bpf_prog *prog, struct bpf_insn_patch *p,
const struct bpf_insn_aux_data *old,
struct bpf_insn_aux_data *slot)
{
- struct bpf_insn_aux_data *last = slot + p->len - 1;
+ struct bpf_insn_aux_data *last = slot + p->self;
struct bpf_insn *orig = &p->orig;
u32 i;
+ memset(slot, 0, sizeof(*slot) * p->len);
*last = *old;
/* Recompute zext_dst: the last insn of the patch need not be the patched one. */
- last->zext_dst = bpf_insn_def32(prog, &patch[p->len - 1]) >= 0;
- memset(slot, 0, sizeof(*slot) * (p->len - 1));
+ last->zext_dst = bpf_insn_def32(prog, &patch[p->self]) >= 0;
- for (i = 0; i < p->len - 1; i++) {
+ for (i = 0; i < p->len; i++) {
+ if (i == p->self)
+ continue;
/* Expand the seen count of the patched insn to the patch. */
- slot[i].seen = last->seen;
+ slot[i].seen = last->seen ?: i > p->self;
/* messages name the insn of the original program */
slot[i].orig_idx = last->orig_idx;
+ /* for may_goto in front of the insn, see bpf_insert_loop_guards() */
+ slot[i].live_regs_before = last->live_regs_before;
slot[i].zext_dst = bpf_insn_def32(prog, &patch[i]) >= 0;
/*
* A copy of the patched insn keeps its marking. Any other
@@ -288,12 +299,12 @@ static void patch_list_adj_aux(struct bpf_prog *prog, struct bpf_insn_patch *p,
* systematically mark it for non-stack access if it is not the patched
* insn, otherwise keep the original marking.
*/
- if (bpf_is_mem_insn(&patch[p->len - 1]) &&
- memcmp(&patch[p->len - 1], orig, sizeof(*orig)))
+ if (bpf_is_mem_insn(&patch[p->self]) &&
+ memcmp(&patch[p->self], orig, sizeof(*orig)))
last->non_stack_access = true;
/* indirect jumps to the replaced insn land on p->target */
- if (last->indirect_target && p->target != p->len - 1) {
+ if (last->indirect_target && p->target != p->self) {
slot[p->target].indirect_target = 1;
last->indirect_target = 0;
}
@@ -325,6 +336,28 @@ static void patch_list_drop_removed(struct bpf_verifier_env *env)
}
}
+/*
+ * Make room for the insns that patch_image_relax() added.
+ * A jump to dead code is dead code, see bpf_opt_hard_wire_dead_code_branches().
+ */
+static void patch_list_relax_aux(const struct bpf_patch_image *img, struct bpf_insn_aux_data *data)
+{
+ const u32 *map = img->relaxed;
+ u32 i, k, to;
+
+ for (i = img->short_cnt; i--;)
+ data[map[i]] = data[i];
+ for (i = 0; i < img->short_cnt; i++) {
+ for (k = map[i] + 1; k < map[i + 1]; k++) {
+ to = k + 1 + bpf_jmp_offset(&img->insns[k]);
+ data[k] = (struct bpf_insn_aux_data){
+ .orig_idx = data[map[i]].orig_idx,
+ .seen = data[map[i]].seen ? data[to].seen : 0,
+ };
+ }
+ }
+}
+
/* Fill 'data', the aux data of the new image, from the aux data of the old one. */
static void patch_list_remap_aux(struct bpf_verifier_env *env,
const struct bpf_patch_image *img,
@@ -332,22 +365,24 @@ static void patch_list_remap_aux(struct bpf_verifier_env *env,
{
struct bpf_patch_list *pl = &env->patch_list;
struct bpf_insn_patch *p;
- u32 i, k, end;
+ u32 i, k, end, pos;
- for (i = 0, k = 0; i < img->cnt; i = end) {
+ for (i = 0, k = 0, pos = 0; i < img->cnt; i = end) {
if (k < pl->cnt && pl->patches[k].off == i) {
p = &pl->patches[k++];
if (p->len)
patch_list_adj_aux(env->prog, p, &pl->insns[p->start],
- &env->insn_aux_data[i],
- &data[img->new_off[i]]);
+ &env->insn_aux_data[i], &data[pos]);
+ pos += p->len;
end = i + 1;
continue;
}
end = k < pl->cnt ? pl->patches[k].off : img->cnt;
- memcpy(data + img->new_off[i], env->insn_aux_data + i,
- sizeof(*data) * (end - i));
+ memcpy(data + pos, env->insn_aux_data + i, sizeof(*data) * (end - i));
+ pos += end - i;
}
+ if (img->relaxed)
+ patch_list_relax_aux(img, data);
vfree(env->insn_aux_data);
env->insn_aux_data = data;
env->insn_aux_data_len = img->new_cnt;
@@ -494,11 +529,6 @@ int bpf_patch_list_commit(struct bpf_verifier_env *env)
goto out;
err = bpf_patch_list_build(pl, prog, &img);
- if (err == -ERANGE) {
- verbose(env, "insn %d cannot be patched due to 16-bit range\n",
- env->insn_aux_data[img.bad].orig_idx);
- goto out;
- }
if (verifier_bug_if(err == -EFAULT, env, "insn %u jumps out of the program",
img.bad))
goto out;
@@ -535,52 +565,6 @@ int bpf_patch_list_commit(struct bpf_verifier_env *env)
return err;
}
-/*
- * insn was moved down by delta insns inside its own patch. Operands relative
- * to the pc that point in front of the old position did not move with it.
- */
-static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
-{
- u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
- s64 off = insn->imm, off_min = S32_MIN;
- bool is_imm = true;
-
- if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn)) {
- /* subprog that started at the old position starts with the patch */
- if (off >= 0)
- return 0;
- } else if ((class == BPF_JMP || class == BPF_JMP32) &&
- op != BPF_CALL && op != BPF_EXIT) {
- if (insn->code != (BPF_JMP32 | BPF_JA)) {
- off = insn->off;
- off_min = S16_MIN;
- is_imm = false;
- }
- /* jump to itself stays */
- if (off >= -1)
- return 0;
- } else {
- return 0;
- }
-
- off -= delta;
- if (off < off_min)
- return -ERANGE;
- if (is_imm)
- insn->imm = off;
- else
- insn->off = off;
- return 0;
-}
-
-static bool jumps_to_itself(const struct bpf_insn *insn)
-{
- u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
-
- return (class == BPF_JMP || class == BPF_JMP32) && op != BPF_CALL && op != BPF_EXIT &&
- insn->code != (BPF_JMP32 | BPF_JA) && insn->off == -1;
-}
-
/* How a subprog leaves when it is out of budget, see push_budget_exit() */
static struct bpf_insn budget_exit_insn(struct bpf_verifier_env *env, struct bpf_subprog_info *sub)
{
@@ -591,205 +575,62 @@ static struct bpf_insn budget_exit_insn(struct bpf_verifier_env *env, struct bpf
return BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0);
}
-/*
- * may_goto has 16 bits for the offset, and what is between it and its target is going
- * to grow: may_goto itself becomes 10 insns, 14 when it has to save a register.
- */
-#define BPF_GUARD_NEAR 2048
-
-static bool guard_leaves_subprog(const struct bpf_insn_aux_data *aux)
-{
- return aux->loop_guard && !aux->guard_no_unwind;
-}
-
-/*
- * 'r0 = 0; exit' behind the last insn of the subprog, for all its guards that are near.
- * Returns the number of insns that are going to be added.
- */
-static int append_budget_exit(struct bpf_verifier_env *env, int s)
-{
- struct bpf_subprog_info *sub = &env->subprog_info[s];
- int last = (sub + 1)->start - 1, n = 0, i;
- struct bpf_insn buf[4], *insn;
-
- for (i = max_t(int, sub->start, last + 1 - BPF_GUARD_NEAR); i <= last; i++)
- if (guard_leaves_subprog(&env->insn_aux_data[i]))
- break;
- if (i > last)
- return 0;
-
- buf[n++] = env->prog->insnsi[last];
- buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
- if (bpf_ret_reg_pair(env, s))
- buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
- buf[n++] = budget_exit_insn(env, sub);
- insn = bpf_patch_list_add(env, last, buf, n);
- return IS_ERR(insn) ? PTR_ERR(insn) : n - 1;
-}
-
/*
* Add may_goto in front of the insns that the main pass marked, see
* check_loop_guard(). It verified the edge from the insn to where may_goto goes.
+ *
+ * may_goto to or may_goto leave see push_budget_exit()
+ * insn insn
+ * ...
+ * last insn of the subprog
+ * leave: r0 = 0
+ * exit
+ *
+ * bpf_do_misc_fixups() keeps the address of the count of may_goto in a register that is
+ * dead at the may_goto. What is live in front of the insn is live in front of its guard.
+ * Nothing else is:
+ * - The guard of a jump goes where the jump goes.
+ * - What is live where the guard at the head of a loop goes is live at the head,
+ * see bpf_head_guard_target().
+ * - On the way out of the subprog R0, and R2 of a pair, are set to 0.
*/
int bpf_insert_loop_guards(struct bpf_verifier_env *env)
{
+ struct bpf_insn patch[2] = { BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 0, 0) };
+ struct bpf_insn leave[3] = { BPF_MOV64_IMM(BPF_REG_0, 0) };
struct bpf_insn_aux_data *aux = env->insn_aux_data;
- int len = env->prog->len, i, r, s, to, n, jmp, err = 0;
- struct bpf_insn *patch, *insn;
- int *target, *leave, *moved;
- bool indirect;
- u16 *zero;
- s64 off;
-
- for (i = 0; i < len; i++)
- if (aux[i].loop_guard)
- break;
- if (i == len)
- return 0;
-
- /*
- * Where the subprog has 'r0 = 0; exit', 0 if it has none. And how far that moved
- * the subprog: guard_to is from the time before.
- */
- leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), BPF_VERIFIER_GFP);
- if (!leave)
- return -ENOMEM;
- moved = leave + env->subprog_cnt;
- for (s = 0; s < env->subprog_cnt && !err; s++) {
- leave[s] = append_budget_exit(env, s);
- if (leave[s] < 0)
- err = leave[s];
- }
- /* A guard cannot jump into a patch that is in the list: 'r0 = 0' has no index yet. */
- err = err ?: bpf_patch_list_commit(env);
- if (err) {
- kvfree(leave);
- return err;
- }
- aux = env->insn_aux_data;
- len = env->prog->len;
- for (s = 0, n = 0; s < env->subprog_cnt; s++) {
- moved[s] = n;
- n += leave[s];
- if (!leave[s])
- continue;
- i = env->subprog_info[s + 1].start;
- leave[s] = i - leave[s];
- /*
- * The list gives what is known about an insn to the last insn of its patch,
- * but for indirect_target. Here the original insn is the first one.
- */
- indirect = aux[leave[s] - 1].indirect_target;
- swap(aux[leave[s] - 1], aux[i - 1]);
- aux[leave[s] - 1].indirect_target = indirect;
- aux[i - 1].indirect_target = false;
- /* the last insn may be dead code */
- while (--i >= leave[s])
- aux[i].seen = 1;
- }
-
- /* where the guard of the insn goes */
- target = kvcalloc(len, sizeof(int), BPF_VERIFIER_GFP);
- zero = kvcalloc(len, sizeof(u16), BPF_VERIFIER_GFP);
- patch = kvcalloc(8, sizeof(*patch), BPF_VERIFIER_GFP);
- if (!target || !zero || !patch) {
- err = -ENOMEM;
- goto out;
- }
- for (i = 0; i < len; i++) {
- s = bpf_find_containing_subprog(env, i) - env->subprog_info;
- target[i] = aux[i].loop_guard ? aux[i].guard_to + moved[s] : -1;
- if (guard_leaves_subprog(&aux[i])) {
- /* see push_budget_exit() */
- if (leave[s] && leave[s] - i <= BPF_GUARD_NEAR) {
- target[i] = leave[s];
- } else {
- target[i] = -2;
- zero[i] = BIT(BPF_REG_0);
- if (bpf_ret_reg_pair(env, s))
- zero[i] |= BIT(BPF_REG_2);
- }
- }
- }
-
- for (i = 0; i < len; i++) {
- to = target[i];
- if (to == -1)
- continue;
- /*
- * may_goto to or may_goto +1
- * insn goto +N
- * r0 = 0, ...
- * exit
- * insn
- */
- n = 0;
- if (zero[i]) {
- patch[n++] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0);
- patch[n++] = BPF_JMP_A(hweight16(zero[i]) + 1);
- for (r = BPF_REG_0; r < BPF_REG_FP; r++)
- if (zero[i] & BIT(r))
- patch[n++] = BPF_MOV64_IMM(r, 0);
- }
- jmp = n++;
- /* What is behind moves with this patch. The other patches are not there yet. */
- if (to == -2)
- off = 0;
- else if (to > i)
- off = to - i + n - jmp - 1;
- else
- off = to - (i + jmp) - 1;
- if (off > S16_MAX || off < S16_MIN) {
- verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
- err = -ERANGE;
- break;
- }
- patch[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
- if (to == -2)
- patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env, i));
- patch[n] = env->prog->insnsi[i];
- /* A jump to itself is a back edge. It goes through may_goto like the others. */
- if (jumps_to_itself(&patch[n]))
- patch[n].off -= n;
- else
- err = bpf_adj_moved_insn(&patch[n], n);
- if (err) {
- verbose(env, "insn %d cannot be patched due to 16-bit range\n", i);
- break;
- }
- n++;
- insn = bpf_patch_list_add(env, i, patch, n);
- if (IS_ERR(insn)) {
- err = PTR_ERR(insn);
- break;
+ struct bpf_subprog_info *sub;
+ struct bpf_insn *insn;
+ int i, s, n, err;
+ bool leaves;
+
+ for (s = 0; s < env->subprog_cnt; s++) {
+ sub = &env->subprog_info[s];
+ n = bpf_ret_reg_pair(env, s) ? 3 : 2;
+ leaves = false;
+ for (i = sub->start; i < sub[1].start; i++) {
+ if (!aux[i].loop_guard)
+ continue;
+ /* A jump to itself is a back edge. It goes through may_goto too. */
+ patch[1] = env->prog->insnsi[i];
+ insn = bpf_patch_list_add(env, i, patch, 2);
+ if (IS_ERR(insn))
+ return PTR_ERR(insn);
+ if (aux[i].guard_no_unwind)
+ bpf_patch_list_jump(&env->patch_list, insn - 1, aux[i].guard_to, 0);
+ else
+ bpf_patch_list_jump(&env->patch_list, insn - 1, sub[1].start, n);
+ leaves |= !aux[i].guard_no_unwind;
}
- }
- err = err ?: bpf_patch_list_commit(env);
- if (err)
- goto out;
- /*
- * bpf_do_misc_fixups() keeps the address of the count of may_goto in a register that is
- * dead at the may_goto. A new insn has no live_regs_before: all registers would be dead.
- * What is live in front of the insn is live in front of its guard. Nothing else is:
- * - The guard of a jump goes where the jump goes.
- * - What is live where the guard at the head of a loop goes is live at the head,
- * see bpf_head_guard_target().
- * - On the way out of the subprog R0, and R2 of a pair, are set to 0.
- */
- aux = env->insn_aux_data;
- for (i = 0, n = 0; i < len; i++) {
- if (target[i] == -1)
+ if (!leaves)
continue;
- r = zero[i] ? 3 + hweight16(zero[i]) : 1;
- aux[i + n].live_regs_before = aux[i + n + r].live_regs_before;
- n += r;
+ leave[1] = BPF_MOV64_IMM(BPF_REG_2, 0);
+ leave[n - 1] = budget_exit_insn(env, sub);
+ err = bpf_patch_list_append(env, i - 1, leave, n);
+ if (err)
+ return err;
}
-out:
- kvfree(leave);
- kvfree(target);
- kvfree(zero);
- kvfree(patch);
- return err;
+ return bpf_patch_list_commit(env);
}
/*
@@ -1530,13 +1371,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
struct bpf_insn *patch = insn_buf;
*patch++ = BPF_ST_NOSPEC();
- *patch = *insn;
- ret = bpf_adj_moved_insn(patch++, 1);
- if (ret) {
- verbose(env, "insn %d cannot be patched due to 16-bit range\n",
- env->insn_aux_data[i].orig_idx);
- return ret;
- }
+ *patch++ = *insn;
cnt = patch - insn_buf;
insn = bpf_patch_list_add(env, i, insn_buf, cnt);
if (IS_ERR(insn))
@@ -2262,12 +2097,10 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
return err;
}
-/* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */
static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len)
{
struct bpf_subprog_info *info = env->subprog_info;
int err, cnt = env->subprog_cnt;
- struct bpf_insn *last;
struct bpf_prog *prog;
/* We only reserve one slot for hidden subprogs in subprog_info. */
@@ -2275,19 +2108,13 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
verifier_bug(env, "only one hidden subprog supported");
return -EFAULT;
}
- /*
- * We're not patching any existing instruction, just appending the new
- * ones for the hidden subprog.
- */
- last = bpf_patch_list_add(env, env->prog->len - 1, patch, len);
- if (IS_ERR(last))
- return PTR_ERR(last);
- err = bpf_patch_list_commit(env);
+ err = bpf_patch_list_append(env, env->prog->len - 1, patch, len) ?:
+ bpf_patch_list_commit(env);
if (err)
return err;
prog = env->prog;
info[cnt + 1].start = info[cnt].start;
- info[cnt].start = prog->len - len + 1;
+ info[cnt].start = prog->len - len;
/* The patch ends with the exit. */
info[cnt].exit_idx = prog->len - 1;
env->subprog_cnt++;
@@ -2295,44 +2122,17 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat
return 0;
}
-/*
- * Expand may_goto: load the count from the stack, jump to the target of
- * may_goto when it is zero, then the tail that updates the count.
- * 'off' is the offset of may_goto, 'reach' how far it jumps once the patches
- * queued so far are applied, see may_goto_off().
- * Use gotol when 'reach' is too far for 16-bit offset of a conditional jump.
- */
-static int may_goto_expand(struct bpf_insn *insn_buf, int off, int reach,
- int stack_off, const struct bpf_insn *tail, int tail_cnt)
-{
- int cnt = 0;
-
- /* Forward jump has to step over the tail */
- off = off >= 0 ? off + tail_cnt : off - 1;
- reach = reach >= 0 ? reach + tail_cnt : reach - 1;
-
- insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_10, stack_off);
- if (reach == (s16)reach) {
- insn_buf[cnt++] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, off);
- } else {
- insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_AX, 0, 1);
- insn_buf[cnt++] = BPF_JMP32_A(off >= 0 ? off : off - 1);
- }
- memcpy(insn_buf + cnt, tail, tail_cnt * sizeof(*tail));
- return cnt + tail_cnt;
-}
-
/*
* Timed may_goto. All may_goto of a prog use one count and one timestamp: per-cpu memory of the
* prog, set in the first insns of the main prog, and by the callers of async callbacks.
* @reg holds the address. It is a register that is dead at the may_goto, or R0 that is saved at
- * @save_off in the stack. See may_goto_expand() for @off and @reach.
+ * @save_off in the stack. @jmp is the insn that goes where may_goto goes.
*/
-static int may_goto_expand_timed(struct bpf_insn *insn_buf, int off, int reach,
+static int may_goto_expand_timed(struct bpf_insn *insn_buf, int *jmp,
struct bpf_timed_may_goto __percpu *tmg, int reg, int save_off)
{
struct bpf_insn ld_tmg[] = { BPF_LD_IMM64(reg, (__force unsigned long)tmg) };
- int cnt = 0, jmp, tail_cnt = save_off ? 6 : 5;
+ int cnt = 0;
if (save_off)
insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_10, reg, save_off);
@@ -2343,19 +2143,9 @@ static int may_goto_expand_timed(struct bpf_insn *insn_buf, int off, int reach,
if (save_off) {
insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_AX, 0, 2);
insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, reg, BPF_REG_10, save_off);
- jmp = cnt++;
- insn_buf[jmp] = BPF_JMP32_A(off >= 0 ? off + tail_cnt : off - jmp);
- } else {
- jmp = cnt;
- off = off >= 0 ? off + tail_cnt : off - jmp;
- reach = reach >= 0 ? reach + tail_cnt : reach - jmp;
- if (reach == (s16)reach) {
- insn_buf[cnt++] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, off);
- } else {
- insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_AX, 0, 1);
- insn_buf[cnt++] = BPF_JMP32_A(off >= 0 ? off : off - 1);
- }
}
+ *jmp = cnt;
+ insn_buf[cnt++] = save_off ? BPF_JMP32_A(0) : BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, 0);
insn_buf[cnt++] = BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, 1);
insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_AX, 0, 2);
/* AX is the argument and the return value */
@@ -2367,27 +2157,6 @@ static int may_goto_expand_timed(struct bpf_insn *insn_buf, int off, int reach,
return cnt;
}
-/*
- * The offset of may_goto at 'idx' once the patches queued in front are applied.
- * The patches that a forward may_goto jumps over aren't queued yet, so its
- * offset stays as it is.
- */
-static int may_goto_off(struct bpf_verifier_env *env, const struct bpf_insn *insn,
- u32 idx)
-{
- struct bpf_patch_list *pl = &env->patch_list;
- u32 tgt = idx + insn->off + 1;
- int off = insn->off;
- u32 k = pl->cnt;
-
- if (off >= 0)
- return off;
- /* all queued patches are in front of idx */
- while (k > 0 && pl->patches[k - 1].off >= tgt)
- off -= pl->patches[--k].len - 1;
- return off;
-}
-
#ifdef CONFIG_64BIT
struct bpf_div_magic {
u64 m; /* multiplier, sign extended for signed division */
@@ -2599,7 +2368,6 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
if (env->seen_exception && !env->exception_callback_subprog) {
struct bpf_insn *patch = insn_buf;
- *patch++ = env->prog->insnsi[insn_cnt - 1];
*patch++ = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
*patch++ = BPF_EXIT_INSN();
ret = add_hidden_subprog(env, insn_buf, patch - insn_buf);
@@ -2846,7 +2614,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
if (bpf_is_may_goto_insn(insn) && timed_may_goto) {
u16 live = env->insn_aux_data[i].live_regs_before;
- int reg, save_off = 0;
+ int reg, jmp, save_off = 0, to = i + insn->off + 1;
/* a register that is dead here holds the address of the count */
for (reg = BPF_REG_0; reg <= BPF_REG_9; reg++)
@@ -2865,28 +2633,26 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
prog->aux->timed_may_goto = tmg;
}
prog->jit_required = 1;
- cnt = may_goto_expand_timed(insn_buf, insn->off, may_goto_off(env, insn, i),
- tmg, reg, save_off);
+ cnt = may_goto_expand_timed(insn_buf, &jmp, tmg, reg, save_off);
insn = bpf_patch_list_add(env, i, insn_buf, cnt);
if (IS_ERR(insn))
return PTR_ERR(insn);
+ bpf_patch_list_jump(&env->patch_list, insn - (cnt - 1 - jmp), to, 0);
goto next_insn;
} else if (bpf_is_may_goto_insn(insn)) {
- int stack_off = -stack_depth - 8;
- struct bpf_insn tail[] = {
- BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, 1),
- BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_AX, stack_off),
- };
+ int stack_off = -stack_depth - 8, to = i + insn->off + 1;
stack_depth_extra = 8;
- cnt = may_goto_expand(insn_buf, insn->off,
- may_goto_off(env, insn, i),
- stack_off, tail, ARRAY_SIZE(tail));
+ insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_AX, BPF_REG_10, stack_off);
+ insn_buf[1] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, 0);
+ insn_buf[2] = BPF_ALU64_IMM(BPF_SUB, BPF_REG_AX, 1);
+ insn_buf[3] = BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_AX, stack_off);
- insn = bpf_patch_list_add(env, i, insn_buf, cnt);
+ insn = bpf_patch_list_add(env, i, insn_buf, 4);
if (IS_ERR(insn))
return PTR_ERR(insn);
+ bpf_patch_list_jump(&env->patch_list, insn - 2, to, 0);
goto next_insn;
}
@@ -3521,9 +3287,7 @@ static int inline_bpf_loop(struct bpf_verifier_env *env, int position,
int reg_loop_ctx = BPF_REG_8;
struct bpf_insn *insn_buf = env->insn_buf;
- u32 callback_start;
struct bpf_insn *last;
- s32 callback_offset;
u32 cnt = 0;
/* This represents an inlined version of bpf_iter.c:bpf_loop,
@@ -3565,19 +3329,13 @@ static int inline_bpf_loop(struct bpf_verifier_env *env, int position,
insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_10, r7_offset);
insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_8, BPF_REG_10, r8_offset);
- /*
- * Note: insn_buf[12] is the BPF_CALL_REL instruction. Jumps and calls
- * out of a patch are relative to the patch placed at 'position': its
- * 'cnt' insns push what is behind 'position' down by cnt - 1.
- */
- callback_start = env->subprog_info[callback_subprogno].start;
- callback_offset = callback_start - position;
- if (callback_start > position)
- callback_offset += cnt - 1;
- insn_buf[12].imm = callback_offset - 12 - 1;
-
last = bpf_patch_list_add(env, position, insn_buf, cnt);
- return PTR_ERR_OR_ZERO(last);
+ if (IS_ERR(last))
+ return PTR_ERR(last);
+ /* insn_buf[12] is the BPF_CALL_REL instruction */
+ bpf_patch_list_jump(&env->patch_list, last - (cnt - 1 - 12),
+ env->subprog_info[callback_subprogno].start, 0);
+ return 0;
}
static bool is_bpf_loop_call(struct bpf_insn *insn)
diff --git a/tools/testing/selftests/bpf/prog_tests/blinding_log.c b/tools/testing/selftests/bpf/prog_tests/blinding_far_jump.c
similarity index 51%
rename from tools/testing/selftests/bpf/prog_tests/blinding_log.c
rename to tools/testing/selftests/bpf/prog_tests/blinding_far_jump.c
index 658a8494dae5..0b3e09864b32 100644
--- a/tools/testing/selftests/bpf/prog_tests/blinding_log.c
+++ b/tools/testing/selftests/bpf/prog_tests/blinding_far_jump.c
@@ -1,24 +1,23 @@
// SPDX-License-Identifier: GPL-2.0
-/* The verifier log of a prog whose constants are blinded */
+/* A far jump in a prog whose constants are blinded */
#include <test_progs.h>
#include <linux/filter.h>
#include "sysctl_helpers.h"
#define NR_CONSTS 11000
-#define LOG_SIZE 65536
/*
* A prog of one function gets its constants blinded when its runtime is
* selected. Each 'r0 += 1' becomes 3 insns, which pushes the jump over them
* out of its 16-bit range. Serial, because it sets bpf_jit_harden.
*/
-void serial_test_blinding_log(void)
+void serial_test_blinding_far_jump(void)
{
const char *harden = "/proc/sys/net/core/bpf_jit_harden";
- LIBBPF_OPTS(bpf_prog_load_opts, opts);
+ char old[16] = {}, pkt[64] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = pkt, .data_size_in = sizeof(pkt));
struct bpf_insn *insns;
- char old[16] = {}, *log;
int i, cnt = 0, fd;
if (!is_jit_enabled()) {
@@ -28,32 +27,25 @@ void serial_test_blinding_log(void)
}
insns = calloc(NR_CONSTS + 3, sizeof(*insns));
- log = calloc(1, LOG_SIZE);
- if (!ASSERT_OK_PTR(insns, "insns") || !ASSERT_OK_PTR(log, "log"))
- goto out;
+ if (!ASSERT_OK_PTR(insns, "insns"))
+ return;
- insns[cnt++] = BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32);
- insns[cnt++] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, NR_CONSTS);
+ /* mark is 0 */
+ insns[cnt++] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, offsetof(struct __sk_buff, mark));
+ insns[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, NR_CONSTS);
for (i = 0; i < NR_CONSTS; i++)
insns[cnt++] = BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1);
insns[cnt++] = BPF_EXIT_INSN();
- opts.log_buf = log;
- opts.log_size = LOG_SIZE;
- /* log stats only: level 1 logs each insn, which wraps the buffer */
- opts.log_level = 4;
if (sysctl_set_or_fail(harden, old, "2"))
goto out;
- fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, cnt, &opts);
+ fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, cnt, NULL);
sysctl_set_or_fail(harden, NULL, old);
- /* don't mind whether the interpreter takes the prog or the load fails */
- if (fd >= 0)
- close(fd);
-
- /* expect the message about the jump in the log and in log_true_size */
- ASSERT_HAS_SUBSTR(log, "cannot be patched due to 16-bit range", "log");
- ASSERT_EQ(opts.log_true_size, strlen(log) + 1, "log_true_size");
+ if (!ASSERT_OK_FD(fd, "load"))
+ goto out;
+ if (ASSERT_OK(bpf_prog_test_run_opts(fd, &opts), "run"))
+ ASSERT_EQ(opts.retval, NR_CONSTS, "retval");
+ close(fd);
out:
free(insns);
- free(log);
}
diff --git a/tools/testing/selftests/bpf/progs/verifier_and.c b/tools/testing/selftests/bpf/progs/verifier_and.c
index fb64ea9928f1..8ef5f87a9766 100644
--- a/tools/testing/selftests/bpf/progs/verifier_and.c
+++ b/tools/testing/selftests/bpf/progs/verifier_and.c
@@ -90,7 +90,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if w0 < 0x1 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R1 !read_ok'` */
-__xlated_unpriv("goto pc-1") /* `r1 = *(u32*)(r1 + 512)`, sanitized dead code */
+__xlated_unpriv("goto pc-2") /* `r1 = *(u32*)(r1 + 512)`, sanitized dead code */
__xlated_unpriv("r0 = 0")
#endif
__naked void known_subreg_with_unknown_reg(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c
index b39275b92e6a..9dc23611332e 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c
@@ -626,7 +626,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 invalid mem access 'scalar'` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("exit")
#endif
__naked void _32bit_and_64bit_arithmetic_test1(void)
@@ -655,7 +655,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 invalid mem access 'scalar'` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("exit")
#endif
__naked void _32bit_and_64bit_arithmetic_test2(void)
@@ -709,7 +709,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if r1 != 0x0 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside of the allowed memory range` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("r0 = 0")
#endif
__naked void reg_0_reg_xor_1(void)
@@ -742,7 +742,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if w1 != 0x0 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside of the allowed memory range` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("r0 = 0")
#endif
__naked void reg32_0_reg32_xor_1(void)
@@ -775,7 +775,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if r1 > 0x0 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside of the allowed memory range` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("r0 = 0")
#endif
__naked void reg_2_reg_xor_3(void)
@@ -862,7 +862,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if r1 >= 0x0 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside of the allowed memory range` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("r0 = 0")
#endif
__naked void reg_0_reg_xor_3(void)
@@ -896,7 +896,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if w1 >= 0x0 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside of the allowed memory range` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("r0 = 0")
#endif
__naked void reg32_0_reg32_xor_3(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_movsx.c b/tools/testing/selftests/bpf/progs/verifier_movsx.c
index 6f90dac78fcd..44730109db5d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_movsx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_movsx.c
@@ -437,7 +437,7 @@ __success_unpriv
__xlated_unpriv("w0 = 0")
__xlated_unpriv("exit")
__xlated_unpriv("nospec") /* inserted to prevent `frame pointer is read only` */
-__xlated_unpriv("goto pc-1")
+__xlated_unpriv("goto pc-2")
#endif
__naked void mov64sx_s32_varoff_2(void)
{
@@ -465,7 +465,7 @@ __success_unpriv
__xlated_unpriv("w0 = 0")
__xlated_unpriv("exit")
__xlated_unpriv("nospec") /* inserted to prevent `frame pointer is read only` */
-__xlated_unpriv("goto pc-1")
+__xlated_unpriv("goto pc-2")
#endif
__naked void mov64sx_s32_varoff_3(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_patch_list.c b/tools/testing/selftests/bpf/progs/verifier_patch_list.c
index aada57330ff2..80d08d93b625 100644
--- a/tools/testing/selftests/bpf/progs/verifier_patch_list.c
+++ b/tools/testing/selftests/bpf/progs/verifier_patch_list.c
@@ -126,29 +126,132 @@ __naked void patch_list_may_goto(void)
: __clobber_all);
}
-/*
- * The conditional jump skips 8200 divisions, which their patches turn into
- * 32800 insns, too far for its 16-bit offset.
- */
-SEC("raw_tp")
-__description("patch list: jump pushed out of range by patches")
-__failure __msg("insn 2 cannot be patched due to 16-bit range")
-__naked void patch_list_jump_out_of_range(void)
+/* the JITs that support gotol */
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
+ defined(__TARGET_ARCH_powerpc) || defined(__TARGET_ARCH_s390) || \
+ defined(__TARGET_ARCH_loongarch) || \
+ (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)
+
+/* The patches turn 8200 divisions into 32800 insns, too far for a 16-bit offset. */
+#define DIVS ".rept 8200; r7 /= r6; .endr;"
+
+/* len is not 0 and mark is 0, which the verifier doesn't know */
+#define FAR_JUMP(name, field) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r6 = *(u32 *)(r1 + %[off]);" \
+ "r7 = 1;" \
+ "r0 = 1;" \
+ "if r6 == 0 goto 0f;" \
+ DIVS \
+ "r0 = 2;" \
+"0: exit;" \
+ : \
+ : __imm_const(off, offsetof(struct __sk_buff, field)) \
+ : __clobber_all); \
+}
+
+SEC("tc")
+__description("patch list: far jump, taken")
+__success __retval(1)
+__arch_x86_64
+__arch_arm64
+__xlated("3: if r6 == 0x0 goto pc+1")
+__xlated("4: goto pc+1")
+__xlated("5: gotol pc+32801")
+FAR_JUMP(patch_list_far_jump_taken, mark)
+
+SEC("tc")
+__description("patch list: far jump, not taken")
+__success __retval(2)
+FAR_JUMP(patch_list_far_jump_not_taken, len)
+
+/* The first jump is in range, 32766, until the second one, 32768, gets two more insns. */
+SEC("tc")
+__description("patch list: far jump over a far jump")
+__success __retval(1)
+__arch_x86_64
+__arch_arm64
+__xlated("3: if r6 == 0x0 goto pc+1")
+__xlated("4: goto pc+1")
+__xlated("5: gotol pc+32768")
+__xlated("6: if r6 == 0x1 goto pc+1")
+__xlated("7: goto pc+1")
+__xlated("8: gotol pc+32768")
+__naked void patch_list_far_jump_over_far_jump(void)
{
- asm volatile (" \
- call %[bpf_get_prandom_u32]; \
- r1 = r0; \
- if r0 == 0 goto 0f; \
- .rept 8200; \
- r0 /= r1; \
- .endr; \
-0: r0 = 0; \
- exit; \
-" :
- : __imm(bpf_get_prandom_u32)
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[mark]);"
+ "r7 = 1;"
+ "r0 = 1;"
+ "if r6 == 0 goto 0f;"
+ "if r6 == 1 goto 1f;"
+ ".rept 8191; r7 /= r6; .endr;"
+ "r0 = 2;"
+"0: r7 = 0;"
+ "r0 = 1;"
+ "exit;"
+"1: r0 = 3;"
+ "exit;"
+ :
+ : __imm_const(mark, offsetof(struct __sk_buff, mark))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("patch list: far goto, far jump back, far may_goto")
+__success __retval(3)
+__naked void patch_list_far_loop(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + %[len]);"
+ "r7 = 1;"
+ "r0 = 0;"
+ "goto 1f;"
+"0: .8byte %[may_goto];"
+ DIVS
+ "r0 += 1;"
+"1: if r0 < 3 goto 0b;"
+ "exit;"
+ :
+ : __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 8202, 0))
: __clobber_all);
}
+/* The loop has no bound. The verifier adds may_goto at its head, the exit is far. */
+#define FAR_GUARD(name, off, expect) \
+__naked __noinline __used static unsigned long name##_func(void) \
+{ \
+ asm volatile ( \
+"0: call %[bpf_get_prandom_u32];" \
+ "if r0 != 1 goto +1;" \
+ "goto 0b;" \
+ ".rept 33000; r3 = 0; .endr;" \
+ "if r0 != 2 goto +1;" \
+ ".8byte %[back];" \
+ "r1 = %[far_data] ll;" \
+ "r0 = *(u8 *)(r1 + " #off ");" \
+ "exit;" \
+ : \
+ : __imm(bpf_get_prandom_u32), \
+ __imm_addr(far_data), \
+ __imm_insn(back, BPF_JMP32_A(-33005)) \
+ : __clobber_all); \
+} \
+SEC("tc") expect \
+__naked void name(void) \
+{ \
+ asm volatile ("call " #name "_func; exit;" ::: __clobber_all); \
+}
+
+char far_data[8] SEC(".data.far") = { [7] = 9 };
+
+FAR_GUARD(patch_list_far_guard, 7, __success __retval(9))
+FAR_GUARD(patch_list_far_guard_bad, 8, __failure __msg("invalid access to map value"))
+#endif
+
/* the JITs that support gotox, as in verifier_gotox.c */
#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
defined(__TARGET_ARCH_powerpc) || \
@@ -220,8 +323,9 @@ __arch_arm64
__success
__xlated("12: if r6 != 0x0 goto pc-3")
__xlated("...")
-__xlated("32783: if r13 != 0x0 goto pc+1")
-__xlated("32784: gotol pc-32773")
+__xlated("32783: if r13 == 0x0 goto pc+1")
+__xlated("32784: goto pc+1")
+__xlated("32785: gotol pc-32774")
__naked void patch_list_may_goto_gotol(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
index 62f998216d4b..7406e8a7d180 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
@@ -615,7 +615,7 @@ __retval(0)
#ifdef SPEC_V1
__xlated_unpriv("if r0 == 0x0 goto pc+2")
__xlated_unpriv("nospec") /* inserted to prevent `R7 invalid mem access 'scalar'` */
-__xlated_unpriv("goto pc-1") /* sanitized dead code */
+__xlated_unpriv("goto pc-2") /* sanitized dead code */
__xlated_unpriv("exit")
#endif
__naked void alu32_mov_u32_const(void)
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 13/21] bpf: Keep the call graph in lists of edges
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (11 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 12/21] bpf: Relax jumps that patches push out of range Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 14/21] bpf: Find the effects of callees on the call graph Alexei Starovoitov
` (7 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The call graph is a matrix of bits. Its closure takes O(n^3 / 64) and
runs twice. The passes over it repeat until nothing changes.
Keep the edges in lists and sort the graph once, with cycles in a row:
main -> f1 -> f2 -> f3 order: f3 | f2 f1 | main
^ |
+-----+
Marks go to callers or to callees in one pass over that order. callx
calls one extra node, which calls all functions whose address is taken.
After the main pass only the insns that were seen count. A call, a tail
call, LD_ABS or bpf_throw() in dead code rejects nothing. The load of
an address is not a call: the main pass records who calls a callback.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 40 +-
kernel/bpf/cfg.c | 128 +++++
kernel/bpf/verifier.c | 485 +++++++-----------
.../bpf/progs/test_global_func_deep_stack.c | 12 +-
.../selftests/bpf/progs/verifier_tailcall.c | 4 +-
tools/testing/selftests/bpf/verifier/calls.c | 3 +-
6 files changed, 357 insertions(+), 315 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 28104dae5f08..7bea740eeaee 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -769,6 +769,32 @@ struct bpf_index {
int bpf_index_find(const struct bpf_index *index, u64 key);
int bpf_index_add(struct bpf_index *index, u64 key, u32 val);
+/* A graph of calls. The nodes are subprogs, and behind them what stands for a set of subprogs. */
+struct bpf_call_graph {
+ u32 cnt; /* nodes */
+ u32 edge_cnt;
+ u32 *callee; /* node s calls callee[first[s]] .. callee[first[s + 1] - 1] */
+ u32 *first;
+ /* see bpf_call_graph_sort() */
+ u32 *postorder;
+ u32 *order;
+ u32 *cycle;
+ u16 *marks; /* of the user of the graph */
+ u32 next; /* nodes that have 'first' */
+};
+
+#define bpf_for_each_callee(cg, s, j) \
+ for (u32 ___e = (cg)->first[s]; \
+ ___e < (cg)->first[(s) + 1] && ({ j = (cg)->callee[___e]; 1; }); ___e++)
+
+int bpf_call_graph_init(struct bpf_call_graph *cg, u32 cnt);
+int bpf_call_graph_add(struct bpf_call_graph *cg, u32 caller, u32 callee);
+void bpf_call_graph_sort(struct bpf_call_graph *cg, u16 skip);
+bool bpf_call_graph_recursive(const struct bpf_call_graph *cg, u32 s);
+void bpf_call_graph_mark_callees(struct bpf_call_graph *cg, u16 mask);
+void bpf_call_graph_mark_callers(struct bpf_call_graph *cg, u16 mask, u16 skip);
+void bpf_call_graph_free(struct bpf_call_graph *cg);
+
#define iarray_for_each(item, arr) \
for (int ___idx = 0; \
___idx < (arr)->cnt && ({ item = (arr)->items[___idx]; 1; }); \
@@ -1088,6 +1114,7 @@ struct bpf_subprog_info {
* U32_MAX when it has none.
*/
u32 exit_idx;
+ u32 callx_edge; /* 1 + the first of its env->callx_edges, 0 if none */
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
u16 stack_depth; /* max. stack depth used by this function */
u16 stack_extra;
@@ -1341,13 +1368,12 @@ struct bpf_verifier_env {
u32 func_set_cnt;
struct bpf_index func_set_index;
bool has_callx;
- /*
- * Call graph edges created by callx instructions. A bitmap of
- * subprog_cnt * subprog_cnt bits, where bit (caller * subprog_cnt + callee)
- * is set when the main verification pass sees 'caller' calling 'callee'
- * via callx. Allocated when the first such edge is recorded.
- */
- unsigned long *callx_edges;
+ /* What callx and helpers call, a list for each caller, see record_callx_edge() */
+ struct bpf_callx_edge {
+ u32 callee;
+ u32 next; /* 1 + the next edge of the caller, 0 if none */
+ } *callx_edges;
+ struct bpf_index callx_index;
struct bpf_idmap idmap_scratch;
struct {
int *insn_state;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 92dc8d0dc22a..07412c0b9d30 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -238,6 +238,134 @@ int bpf_index_add(struct bpf_index *index, u64 key, u32 val)
return 0;
}
+int bpf_call_graph_init(struct bpf_call_graph *cg, u32 cnt)
+{
+ *cg = (struct bpf_call_graph){ .cnt = cnt };
+ /* the last 4 parts are for bpf_call_graph_sort() */
+ cg->first = kvcalloc(8 * (size_t)cnt + 1, sizeof(u32), BPF_VERIFIER_GFP);
+ cg->marks = kvcalloc(cnt, sizeof(u16), BPF_VERIFIER_GFP);
+ if (!cg->first || !cg->marks)
+ return -ENOMEM;
+ cg->postorder = cg->first + cnt + 1;
+ cg->order = cg->postorder + cnt;
+ cg->cycle = cg->order + cnt;
+ return 0;
+}
+
+void bpf_call_graph_free(struct bpf_call_graph *cg)
+{
+ kvfree(cg->first);
+ kvfree(cg->callee);
+ kvfree(cg->marks);
+}
+
+/* Callers come in increasing order */
+int bpf_call_graph_add(struct bpf_call_graph *cg, u32 caller, u32 callee)
+{
+ if (WARN_ON_ONCE(callee >= cg->cnt))
+ return -EFAULT;
+ if (!bpf_table_next(cg->callee, cg->edge_cnt))
+ return -ENOMEM;
+ while (cg->next <= caller)
+ cg->first[cg->next++] = cg->edge_cnt;
+ cg->callee[cg->edge_cnt++] = callee;
+ return 0;
+}
+
+/*
+ * Tarjan's algorithm. Calls of the nodes that have a mark in 'skip' do not count.
+ *
+ * main -> f1 -> f2 -> f3 postorder: f3 f2 f1 main
+ * ^ | order: f3 f2 f1 main callees first, a cycle in a row
+ * +-----+ cycle: 0 1 1 3 where the cycle starts in 'order'
+ */
+void bpf_call_graph_sort(struct bpf_call_graph *cg, u16 skip)
+{
+ u32 cnt = cg->cnt, *low = cg->cycle + cnt, *edge = low + cnt, *path = edge + cnt;
+ u32 *open = path + cnt, depth = 0, top = 0, num = 0, done = 0, post = 0, i, s, j;
+
+ while (cg->next <= cnt)
+ cg->first[cg->next++] = cg->edge_cnt;
+ /* low: 0 - not visited, U32_MAX - is in 'order' */
+ memset(low, 0, cnt * sizeof(*low));
+ for (i = 0; i < cnt; i++) {
+ if (!low[i])
+ path[depth++] = i;
+ while (depth) {
+ s = path[depth - 1];
+ if (!low[s]) {
+ /* until the node is in 'order': its number */
+ low[s] = cg->cycle[s] = ++num;
+ edge[s] = cg->first[s];
+ open[top++] = s;
+ }
+ if (edge[s] < cg->first[s + 1]) {
+ j = cg->callee[edge[s]];
+ if (cg->marks[j] & skip) {
+ edge[s]++;
+ } else if (!low[j]) {
+ path[depth++] = j;
+ } else {
+ low[s] = min(low[s], low[j]);
+ edge[s]++;
+ }
+ continue;
+ }
+ depth--;
+ cg->postorder[post++] = s;
+ if (low[s] != cg->cycle[s])
+ continue;
+ for (j = done; low[s] != U32_MAX; done++) {
+ cg->order[done] = open[--top];
+ cg->cycle[open[top]] = j;
+ low[open[top]] = U32_MAX;
+ }
+ }
+ }
+}
+
+bool bpf_call_graph_recursive(const struct bpf_call_graph *cg, u32 s)
+{
+ u32 j;
+
+ bpf_for_each_callee(cg, s, j)
+ if (cg->cycle[j] == cg->cycle[s])
+ return true;
+ return false;
+}
+
+/* What a node has of 'mask', all nodes that it gets to have */
+void bpf_call_graph_mark_callees(struct bpf_call_graph *cg, u16 mask)
+{
+ u32 start, end, i, j;
+ u16 m;
+
+ for (end = cg->cnt; end; end = start) {
+ start = cg->cycle[cg->order[end - 1]];
+ for (i = start, m = 0; i < end; i++)
+ m |= cg->marks[cg->order[i]] & mask;
+ for (i = start; i < end; i++)
+ bpf_for_each_callee(cg, cg->order[i], j)
+ cg->marks[j] |= m;
+ }
+}
+
+/* What a node has of 'mask', all nodes that get to it have. 'skip' is that of the sort. */
+void bpf_call_graph_mark_callers(struct bpf_call_graph *cg, u16 mask, u16 skip)
+{
+ u32 start, end, j;
+ u16 m;
+
+ for (start = 0; start < cg->cnt; start = end) {
+ for (end = start, m = 0; end < cg->cnt && cg->cycle[cg->order[end]] == start; end++)
+ bpf_for_each_callee(cg, cg->order[end], j)
+ if (!(cg->marks[j] & skip))
+ m |= cg->marks[j] & mask;
+ for (j = start; j < end; j++)
+ cg->marks[cg->order[j]] |= m;
+ }
+}
+
static int copy_insn_array(struct bpf_map *map, u32 start, u32 end, u32 *items)
{
struct bpf_insn_array_value *value;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c5a44c887ada..5fe15a4b3bd8 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3213,7 +3213,8 @@ static int add_kfuncs(struct bpf_verifier_env *env)
return 0;
}
-static void find_subprog_properties(struct bpf_verifier_env *env)
+/* Before the main pass all insns count, after it those that were 'seen' */
+static void find_subprog_properties(struct bpf_verifier_env *env, bool seen)
{
struct bpf_subprog_info *subprog = env->subprog_info;
struct bpf_insn *insn = env->prog->insnsi;
@@ -3222,10 +3223,15 @@ static void find_subprog_properties(struct bpf_verifier_env *env)
for (cur_subprog = 0; cur_subprog < env->subprog_cnt; cur_subprog++) {
int i;
+ subprog[cur_subprog].has_tail_call = false;
+ subprog[cur_subprog].tail_call_reachable = false;
+ subprog[cur_subprog].has_ld_abs = false;
for (i = subprog[cur_subprog].start;
i < subprog[cur_subprog + 1].start; i++) {
u8 code = insn[i].code;
+ if (seen && !env->insn_aux_data[i].seen)
+ continue;
if (code == (BPF_JMP | BPF_CALL) &&
insn[i].src_reg == 0 &&
insn[i].imm == BPF_FUNC_tail_call) {
@@ -3310,33 +3316,18 @@ static int check_subprogs(struct bpf_verifier_env *env)
*/
static int record_callx_edge(struct bpf_verifier_env *env, int caller, int callee)
{
- u32 cnt = env->subprog_cnt;
+ u32 *first = &env->subprog_info[caller].callx_edge, cnt = env->callx_index.cnt;
+ u64 key = (u64)caller << 32 | callee;
- if (!env->callx_edges) {
- env->callx_edges = kvcalloc(BITS_TO_LONGS(cnt * cnt), sizeof(long),
- BPF_VERIFIER_GFP);
- if (!env->callx_edges)
- return -ENOMEM;
- }
- __set_bit(caller * cnt + callee, env->callx_edges);
+ if (bpf_index_find(&env->callx_index, key) >= 0)
+ return 0;
+ if (!bpf_table_next(env->callx_edges, cnt) || bpf_index_add(&env->callx_index, key, cnt))
+ return -ENOMEM;
+ env->callx_edges[cnt] = (struct bpf_callx_edge){ callee, *first };
+ *first = cnt + 1;
return 0;
}
-/*
- * Return the first subprog with the number >= 'from' that 'caller' calls
- * via callx, or -1 when there is none.
- */
-static int next_callx_callee(struct bpf_verifier_env *env, int caller, int from)
-{
- u32 cnt = env->subprog_cnt;
- unsigned long bit, end = (caller + 1) * cnt;
-
- if (!env->callx_edges || from >= cnt)
- return -1;
- bit = find_next_bit(env->callx_edges, end, caller * cnt + from);
- return bit < end ? bit - caller * cnt : -1;
-}
-
/*
* 'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit.
* JITs don't emit a branch for it. unwind_exit finds the pad by its return
@@ -3403,93 +3394,40 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env)
struct bpf_subprog_info *si = env->subprog_info;
int *insn_postorder = env->cfg.insn_postorder;
struct bpf_insn *insn = env->prog->insnsi;
- int cnt = env->subprog_cnt;
- int *dfs_stack = NULL;
- int top = 0, order = 0;
- int i, ret = 0;
- u8 *color = NULL;
-
- color = kvzalloc_objs(*color, cnt, BPF_VERIFIER_GFP);
- dfs_stack = kvmalloc_objs(*dfs_stack, cnt, BPF_VERIFIER_GFP);
- if (!color || !dfs_stack) {
- ret = -ENOMEM;
- goto out;
- }
+ int cnt = env->subprog_cnt, i, j, idx, ret;
+ struct bpf_call_graph cg;
- /*
- * DFS post-order traversal.
- * Color values: 0 = unvisited, 1 = on stack, 2 = done.
- */
+ ret = bpf_call_graph_init(&cg, cnt);
for (i = 0; i < cnt; i++) {
- if (color[i])
- continue;
- color[i] = 1;
- dfs_stack[top++] = i;
-
- while (top > 0) {
- int cur = dfs_stack[top - 1];
- int po_start = si[cur].postorder_start;
- int po_end = si[cur + 1].postorder_start;
- bool pushed = false;
- int j;
-
- for (j = po_start; j < po_end; j++) {
- int idx = insn_postorder[j];
- int callee;
-
- if (bpf_is_callx(&insn[idx])) {
- /* find a callee that is not explored yet */
- callee = -1;
- do {
- callee = next_callx_callee(env, cur, callee + 1);
- } while (callee >= 0 && color[callee] == 2);
- if (callee < 0)
- continue;
- } else if (bpf_pseudo_call(&insn[idx]) || bpf_pseudo_func(&insn[idx])) {
- callee = bpf_find_subprog(env, idx + insn[idx].imm + 1);
- if (callee < 0) {
- ret = -EFAULT;
- goto out;
- }
- } else {
- continue;
- }
- if (color[callee] == 2)
- continue;
- /*
- * The call that is its own target never returns. It is
- * also what libbpf leaves in place of a call that it
- * could not resolve.
- */
- if (bpf_pseudo_call(&insn[idx]) && insn[idx].imm == -1) {
- verbose(env, "call at insn %d calls itself\n", idx);
- ret = -EINVAL;
- goto out;
- }
- /* recursion: the order within the cycle is arbitrary */
- if (color[callee] == 1)
- continue;
- color[callee] = 1;
- dfs_stack[top++] = callee;
- pushed = true;
+ for (j = si[i].postorder_start; j < si[i + 1].postorder_start && !ret; j++) {
+ idx = insn_postorder[j];
+ if (!bpf_pseudo_call(&insn[idx]) && !bpf_pseudo_func(&insn[idx]))
+ continue;
+ /*
+ * The call that is its own target never returns. It is
+ * also what libbpf leaves in place of a call that it
+ * could not resolve.
+ */
+ if (bpf_pseudo_call(&insn[idx]) && insn[idx].imm == -1) {
+ verbose(env, "call at insn %d calls itself\n", idx);
+ ret = -EINVAL;
break;
}
-
- if (!pushed) {
- color[cur] = 2;
- env->subprog_topo_order[order++] = cur;
- top--;
- }
+ ret = bpf_call_graph_add(&cg, i,
+ bpf_find_subprog(env, idx + insn[idx].imm + 1));
}
}
+ if (ret)
+ goto out;
+ bpf_call_graph_sort(&cg, 0);
+ memcpy(env->subprog_topo_order, cg.postorder, cnt * sizeof(u32));
if (env->log.level & BPF_LOG_LEVEL2)
for (i = 0; i < cnt; i++)
verbose(env, "topo_order[%d] = %s\n",
i, bpf_subprog_name(env, env->subprog_topo_order[i]));
out:
- kvfree(dfs_stack);
- kvfree(color);
+ bpf_call_graph_free(&cg);
return ret;
}
@@ -5447,122 +5385,75 @@ static int round_up_stack_depth(struct bpf_verifier_env *env, int stack_depth)
return round_up(max_t(u32, stack_depth, 1), 32);
}
-/*
- * A set of subprogs is a bitmap. The call graph has a set for every subprog:
- * calls + s * BITS_TO_LONGS(cnt) are the subprogs that s calls. Make it
- * the subprogs that s gets to, through any number of calls.
- */
-static void close_call_graph(unsigned long *calls, int cnt)
-{
- int words = BITS_TO_LONGS(cnt);
- int j, s;
+enum {
+ CG_LIVE = BIT(0), /* what the kernel calls and what that gets to */
+ CG_IN_CB = BIT(1), /* callbacks and what they get to */
+ CG_IN_ASYNC_CB = BIT(2), /* async callbacks and what they get to */
+ CG_IN_RECURSION = BIT(3), /* recursive subprogs and what they get to */
+ CG_TAIL_CALL = BIT(4), /* subprogs that get to a tail call */
+ CG_ARENA_SP = BIT(5), /* subprogs that get to r12 */
+ CG_CB = BIT(6),
+ CG_GLOBAL = BIT(7),
+};
- for (j = 0; j < cnt; j++)
- for (s = 0; s < cnt; s++)
- if (test_bit(j, calls + s * words))
- bitmap_or(calls + s * words, calls + s * words,
- calls + j * words, cnt);
-}
+#define for_each_marked_subprog(cg, s, mark) \
+ for (s = 0; s < (cg)->cnt; s++) if (!((cg)->marks[s] & (mark))) {} else
/*
* Who calls whom. It's known after the main verification pass, which finds out
- * what callx and the helpers that take callbacks call.
+ * what callx and the helpers that take callbacks call, and what is dead code.
*
* What follows for a subprog from its callers, or from what it calls, is computed
* over this graph and not by a walk of one chain of calls after another: the
* number of chains grows exponentially, and there is no end to them when subprogs
* call themselves.
*/
-struct bpf_call_graph {
- int cnt; /* subprogs */
- int words; /* longs in a set of subprogs */
- unsigned long *calls; /* calls + s * words: the subprogs that s calls */
- unsigned long *reach; /* reach + s * words: the subprogs that s gets to */
- unsigned long *live; /* what the kernel calls and what that gets to */
- unsigned long *in_cb; /* callbacks and what they get to */
- unsigned long *in_async_cb; /* async callbacks and what they get to */
- unsigned long *in_recursion; /* recursive subprogs and what they get to */
- unsigned long *tail_calls; /* subprogs that have a tail call */
- unsigned long *arena_sp; /* subprogs that count on r12 of the caller */
-};
-
-static unsigned long *subprog_calls(struct bpf_call_graph *cg, int s)
-{
- return cg->calls + s * cg->words;
-}
-
-static unsigned long *subprog_reach(struct bpf_call_graph *cg, int s)
-{
- return cg->reach + s * cg->words;
-}
-
-/* add the subprog and what it gets to */
-static void add_reach(struct bpf_call_graph *cg, unsigned long *set, int s)
-{
- __set_bit(s, set);
- bitmap_or(set, set, subprog_reach(cg, s), cg->cnt);
-}
-
static int build_call_graph(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
{
struct bpf_subprog_info *si = env->subprog_info;
struct bpf_insn *insn = env->prog->insnsi;
- int cnt = env->subprog_cnt, words = BITS_TO_LONGS(cnt);
- int i, j, s;
-
- cg->cnt = cnt;
- cg->words = words;
- cg->calls = kvcalloc((2 * cnt + 6) * words, sizeof(long), BPF_VERIFIER_GFP);
- if (!cg->calls)
- return -ENOMEM;
- cg->reach = cg->calls + cnt * words;
- cg->live = cg->reach + cnt * words;
- cg->in_cb = cg->live + words;
- cg->in_async_cb = cg->in_cb + words;
- cg->in_recursion = cg->in_async_cb + words;
- cg->tail_calls = cg->in_recursion + words;
- cg->arena_sp = cg->tail_calls + words;
+ int cnt = env->subprog_cnt, i, s, err;
+ err = bpf_call_graph_init(cg, cnt);
for (s = 0; s < cnt; s++) {
- for (i = si[s].start; i < si[s + 1].start; i++) {
- /* the subprog that takes the address of a subprog counts as its caller */
- if (!bpf_pseudo_call(insn + i) && !bpf_pseudo_func(insn + i))
- continue;
- j = bpf_find_subprog(env, i + insn[i].imm + 1);
- if (verifier_bug_if(j < 0, env, "callee not found at insn %d",
- i + insn[i].imm + 1))
- return -EFAULT;
- /*
- * async callbacks don't increase bpf prog stack size
- * unless called directly
- */
- if (si[j].is_async_cb && !bpf_pseudo_call(insn + i))
- continue;
- __set_bit(j, subprog_calls(cg, s));
- }
- /* the callees of callx and the callbacks of helpers, see record_callx_edge() */
- for (j = next_callx_callee(env, s, 0); j >= 0; j = next_callx_callee(env, s, j + 1))
- __set_bit(j, subprog_calls(cg, s));
+ for (i = si[s].start; i < si[s + 1].start && !err; i++)
+ if (bpf_pseudo_call(insn + i) && env->insn_aux_data[i].seen)
+ err = bpf_call_graph_add(cg, s, bpf_find_subprog(env,
+ i + insn[i].imm + 1));
+ /* the callees of callx and the callbacks of helpers */
+ for (i = si[s].callx_edge; i && !err; i = env->callx_edges[i - 1].next)
+ err = bpf_call_graph_add(cg, s, env->callx_edges[i - 1].callee);
}
- memcpy(cg->reach, cg->calls, cnt * words * sizeof(long));
- close_call_graph(cg->reach, cnt);
+ if (err)
+ return err;
+ bpf_call_graph_sort(cg, 0);
for (s = 0; s < cnt; s++)
if (!s || si[s].is_async_cb)
- add_reach(cg, cg->live, s);
- for_each_set_bit(s, cg->live, cnt) {
- if (si[s].is_cb)
- add_reach(cg, cg->in_cb, s);
- if (si[s].is_async_cb)
- add_reach(cg, cg->in_async_cb, s);
- if (test_bit(s, subprog_reach(cg, s)))
- add_reach(cg, cg->in_recursion, s);
- if (si[s].has_tail_call)
- __set_bit(s, cg->tail_calls);
- }
+ cg->marks[s] = CG_LIVE;
+ bpf_call_graph_mark_callees(cg, CG_LIVE);
+ for_each_marked_subprog(cg, s, CG_LIVE)
+ cg->marks[s] |= (si[s].is_cb ? CG_CB | CG_IN_CB : 0) |
+ (si[s].is_async_cb ? CG_IN_ASYNC_CB : 0) |
+ (bpf_call_graph_recursive(cg, s) ? CG_IN_RECURSION : 0) |
+ (si[s].has_tail_call ? CG_TAIL_CALL : 0) |
+ (si[s].uses_arena_sp ? CG_ARENA_SP : 0) |
+ (bpf_subprog_is_global(env, s) ? CG_GLOBAL : 0);
+ bpf_call_graph_mark_callees(cg, CG_IN_CB | CG_IN_ASYNC_CB | CG_IN_RECURSION);
+ bpf_call_graph_mark_callers(cg, CG_TAIL_CALL, 0);
return 0;
}
+static bool subprog_calls(struct bpf_call_graph *cg, u32 caller, u32 callee)
+{
+ u32 j;
+
+ bpf_for_each_callee(cg, caller, j)
+ if (j == callee)
+ return true;
+ return false;
+}
+
/* the insn of subprog 'caller' that is the edge to subprog 'callee' */
static int find_call_insn(struct bpf_verifier_env *env, int caller, int callee)
{
@@ -5570,13 +5461,9 @@ static int find_call_insn(struct bpf_verifier_env *env, int caller, int callee)
struct bpf_insn *insn = env->prog->insnsi;
int i, end = si[caller + 1].start;
- for (i = si[caller].start; i < end; i++) {
- if (!bpf_pseudo_call(insn + i) &&
- (!bpf_pseudo_func(insn + i) || si[callee].is_async_cb))
- continue;
- if (i + insn[i].imm + 1 == si[callee].start)
+ for (i = si[caller].start; i < end; i++)
+ if (bpf_pseudo_call(insn + i) && i + insn[i].imm + 1 == si[callee].start)
return i;
- }
/* It's not recorded which callx or helper calls what. Take the first one. */
for (i = si[caller].start; i < end - 1; i++)
if (bpf_is_callx(insn + i) || bpf_calls_callback(env, i))
@@ -5589,8 +5476,8 @@ static int check_exception_cb(struct bpf_verifier_env *env, struct bpf_call_grap
{
int cb = env->exception_callback_subprog, s;
- for_each_set_bit(s, cg->live, cg->cnt) {
- if (!cb || !test_bit(cb, subprog_calls(cg, s)))
+ for_each_marked_subprog(cg, s, CG_LIVE) {
+ if (!cb || !subprog_calls(cg, s, cb))
continue;
verbose(env, "insn %d cannot call exception cb directly\n",
find_call_insn(env, s, cb));
@@ -5611,8 +5498,8 @@ static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *
{
int j, s;
- for_each_set_bit(s, cg->live, cg->cnt) {
- if (!test_bit(s, subprog_reach(cg, s))) {
+ for_each_marked_subprog(cg, s, CG_LIVE) {
+ if (!bpf_call_graph_recursive(cg, s)) {
env->subprog_info[s].recursive = false;
continue;
}
@@ -5641,9 +5528,8 @@ static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *
return -EOPNOTSUPP;
}
/* the last call of the cycle */
- for_each_set_bit(j, subprog_reach(cg, s), cg->cnt)
- if (test_bit(s, subprog_calls(cg, j)))
- break;
+ for (j = 0; cg->cycle[j] != cg->cycle[s] || !subprog_calls(cg, j, s); j++)
+ ;
verbose(env, "recursive call from %s() to %s()\n",
bpf_subprog_name(env, j), bpf_subprog_name(env, s));
bpf_diag_program_structure(
@@ -5662,13 +5548,13 @@ static int check_callbacks(struct bpf_verifier_env *env, struct bpf_call_graph *
struct bpf_insn *insn = env->prog->insnsi;
int i, s;
- for_each_set_bit(s, cg->in_cb, cg->cnt) {
+ for_each_marked_subprog(cg, s, CG_IN_CB) {
if (si[s].has_ld_abs) {
verbose(env, "cannot use BPF_LD_[ABS|IND] within callback\n");
return -EINVAL;
}
for (i = si[s].start; i < si[s + 1].start; i++) {
- if (!bpf_is_throw_kfunc(insn + i))
+ if (!bpf_is_throw_kfunc(insn + i) || !env->insn_aux_data[i].seen)
continue;
verbose(env,
"bpf_throw kfunc (insn %d) cannot be called from callback subprog %d\n",
@@ -5689,9 +5575,8 @@ static int check_tail_calls(struct bpf_verifier_env *env, struct bpf_call_graph
struct bpf_subprog_info *si = env->subprog_info;
int s;
- for_each_set_bit(s, cg->live, cg->cnt) {
- if (!si[s].has_tail_call &&
- !bitmap_intersects(subprog_reach(cg, s), cg->tail_calls, cg->cnt))
+ for_each_marked_subprog(cg, s, CG_LIVE) {
+ if (!(cg->marks[s] & CG_TAIL_CALL))
continue;
if (si[s].is_cb) {
verbose(env, "cannot tail call within callback\n");
@@ -5723,32 +5608,18 @@ static int check_tail_calls(struct bpf_verifier_env *env, struct bpf_call_graph
*/
static void mark_arena_stack_entries(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
{
- struct bpf_subprog_info *si = env->subprog_info;
- bool changed;
int s;
- do {
- changed = false;
- for_each_set_bit(s, cg->live, cg->cnt) {
- if (test_bit(s, cg->arena_sp) || si[s].arena_stack_entry)
- continue;
- if (!si[s].uses_arena_sp &&
- !bitmap_intersects(subprog_calls(cg, s), cg->arena_sp, cg->cnt))
- continue;
- if (si[s].is_cb) {
- si[s].arena_stack_entry = true;
- continue;
- }
- __set_bit(s, cg->arena_sp);
- changed = true;
- }
- } while (changed);
+ bpf_call_graph_sort(cg, CG_CB);
+ bpf_call_graph_mark_callers(cg, CG_ARENA_SP, CG_CB);
+ for_each_marked_subprog(cg, s, CG_CB)
+ env->subprog_info[s].arena_stack_entry = cg->marks[s] & CG_ARENA_SP;
}
/* temporary state used for call frame depth calculation */
struct bpf_subprog_call_depth_info {
- int caller; /* the caller that has the most on the stack. -1: the kernel */
- int depth; /* the most that the frames of the callers take. -1: not known yet */
+ int calls; /* the number of frames that take the most on the stack */
+ int depth; /* the most that the frames of the callers take */
int arena_depth; /* the same in the stack in arena */
int frame; /* the most consecutive static call stack frames on top of stack */
};
@@ -5763,53 +5634,67 @@ static int subprog_stack_size(struct bpf_verifier_env *env, int s)
return round_up_stack_depth(env, sub->stack_depth);
}
+static void max_call_depth(struct bpf_subprog_call_depth_info *to,
+ const struct bpf_subprog_call_depth_info *from)
+{
+ if (from->depth > to->depth || !to->calls) {
+ to->depth = max(to->depth, from->depth);
+ to->calls = from->calls;
+ }
+ to->arena_depth = max(to->arena_depth, from->arena_depth);
+}
+
/*
* A subprog is entered with no more than what one of its callers is entered with plus
* the frame of that caller. Not when both are recursive: the callee checks the stack at run
- * time. A cycle adds nothing, so this comes to an end.
+ * time. A cycle adds nothing: all subprogs on it have the same.
+ * The count of frames starts anew in a global subprog.
*/
static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph *cg,
struct bpf_subprog_call_depth_info *dinfo)
{
+ struct bpf_subprog_call_depth_info in, out, *callee;
struct bpf_subprog_info *si = env->subprog_info;
- struct bpf_subprog_call_depth_info *callee;
- int depth, arena_depth, frame, j, s;
- bool changed;
-
- for (s = 0; s < cg->cnt; s++) {
- dinfo[s].caller = -1;
- dinfo[s].depth = !s || si[s].is_async_cb ? 0 : -1;
- }
- do {
- cond_resched();
- changed = false;
- for_each_set_bit(s, cg->live, cg->cnt) {
- if (dinfo[s].depth < 0)
+ u32 start, end, i, j, s;
+ int frame;
+
+ memset(dinfo, 0, cg->cnt * sizeof(*dinfo));
+ bpf_call_graph_sort(cg, 0);
+ for (end = cg->cnt; end; end = start) {
+ start = cg->cycle[cg->order[end - 1]];
+ for (i = start, in = dinfo[cg->order[i]]; i < end; i++)
+ max_call_depth(&in, &dinfo[cg->order[i]]);
+ for (i = start; i < end; i++) {
+ s = cg->order[i];
+ if (!(cg->marks[s] & CG_LIVE))
continue;
- for_each_set_bit(j, subprog_calls(cg, s), cg->cnt) {
- callee = &dinfo[j];
- depth = dinfo[s].depth;
- arena_depth = dinfo[s].arena_depth;
+ bpf_for_each_callee(cg, s, j) {
+ out = in;
+ out.calls++;
if (!si[s].recursive || !si[j].recursive) {
- depth += subprog_stack_size(env, s);
- arena_depth += si[s].arena_stack_depth;
+ out.depth += subprog_stack_size(env, s);
+ out.arena_depth += si[s].arena_stack_depth;
}
- frame = dinfo[s].frame + !si[j].recursive;
- if (bpf_subprog_is_global(env, j))
- frame = 0;
- if (depth <= callee->depth && arena_depth <= callee->arena_depth &&
- frame <= callee->frame)
+ max_call_depth(&dinfo[j], &out);
+ }
+ }
+ }
+
+ bpf_call_graph_sort(cg, CG_GLOBAL);
+ for (end = cg->cnt; end; end = start) {
+ start = cg->cycle[cg->order[end - 1]];
+ for (i = start, frame = 0; i < end; i++)
+ frame = max(frame, dinfo[cg->order[i]].frame);
+ for (i = start; i < end; i++) {
+ bpf_for_each_callee(cg, cg->order[i], j) {
+ callee = &dinfo[j];
+ if (!(cg->marks[cg->order[i]] & CG_LIVE) ||
+ (cg->marks[j] & CG_GLOBAL))
continue;
- if (depth > callee->depth) {
- callee->depth = depth;
- callee->caller = s;
- }
- callee->arena_depth = max(callee->arena_depth, arena_depth);
- callee->frame = max(callee->frame, frame);
- changed = true;
+ callee->frame = max(callee->frame, frame + !si[j].recursive);
}
}
- } while (changed);
+ }
}
/*
@@ -5827,7 +5712,7 @@ static int pad_recursive_frames(struct bpf_verifier_env *env, struct bpf_call_gr
u32 arena_frame = 0, depth = 0, room, frame;
int s;
- for_each_set_bit(s, cg->live, cg->cnt) {
+ for_each_marked_subprog(cg, s, CG_LIVE) {
depth = max_t(u32, depth, dinfo[s].arena_depth + si[s].arena_stack_depth);
if (si[s].recursive)
arena_frame = max(arena_frame, si[s].arena_stack_depth);
@@ -5839,7 +5724,7 @@ static int pad_recursive_frames(struct bpf_verifier_env *env, struct bpf_call_gr
room = BPF_ARENA_STACK_MAX - depth - 2 * arena_frame;
env->max_arena_stack_depth = BPF_ARENA_STACK_MAX;
- for_each_set_bit(s, cg->live, cg->cnt) {
+ for_each_marked_subprog(cg, s, CG_LIVE) {
if (!si[s].recursive)
continue;
frame = DIV_ROUND_UP(si[s].arena_stack_depth * (THREAD_SIZE / 2), room);
@@ -5857,7 +5742,7 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
{
struct bpf_subprog_info *si = env->subprog_info;
struct bpf_subprog_call_depth_info *dinfo;
- int depth, total, tmp, s, err;
+ int depth, s, err;
dinfo = kvzalloc_objs(*dinfo, cg->cnt, BPF_VERIFIER_GFP);
if (!dinfo)
@@ -5869,7 +5754,7 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
if (err)
compute_call_depth(env, cg, dinfo);
- for_each_set_bit(s, cg->live, cg->cnt) {
+ for_each_marked_subprog(cg, s, CG_LIVE) {
/* A caller of it is over a limit. That one is reported, not what it calls. */
if (dinfo[s].frame > MAX_CALL_FRAMES || dinfo[s].depth > env->stack_limit ||
dinfo[s].arena_depth > BPF_ARENA_STACK_MAX)
@@ -5924,7 +5809,7 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
*/
depth = dinfo[s].depth;
/* nor of the frames of recursive subprogs, see bpf_insert_recursion_checks() */
- si[s].in_recursion = test_bit(s, cg->in_recursion);
+ si[s].in_recursion = cg->marks[s] & CG_IN_RECURSION;
if (si[s].in_recursion)
depth += THREAD_SIZE / 2;
if (s && si[s].has_tail_call && depth >= 256) {
@@ -5949,12 +5834,8 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
if (depth > env->max_stack_depth)
env->max_stack_depth = depth;
if (depth > env->stack_limit) {
- total = 0;
- for (tmp = s; tmp >= 0; tmp = dinfo[tmp].caller)
- total++;
-
verbose(env, "combined stack size of %d calls is %d. Too large\n",
- total, depth);
+ dinfo[s].calls + 1, depth);
goto out;
}
}
@@ -5999,22 +5880,16 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
{
struct bpf_subprog_info *si = env->subprog_info;
struct bpf_insn *insn = env->prog->insnsi;
- int cnt = env->subprog_cnt, words = BITS_TO_LONGS(cnt);
- unsigned long *calls, *taken, *callx;
+ int cnt = env->subprog_cnt, i, j, s, err;
+ struct bpf_call_graph cg;
bool plain_callx = false;
- int i, j, s, err = 0;
-
- /* calls + s * words: the subprogs that s calls */
- calls = kvcalloc((cnt + 2) * words, sizeof(long), BPF_VERIFIER_GFP);
- if (!calls)
- return -ENOMEM;
- taken = calls + cnt * words;
- callx = taken + words;
+ /* callx calls node 'cnt', which calls each subprog whose address is taken */
+ err = bpf_call_graph_init(&cg, cnt + 1);
for (s = 0; s < cnt; s++) {
- for (i = si[s].start; i < si[s + 1].start; i++) {
+ for (i = si[s].start; i < si[s + 1].start && !err; i++) {
if (bpf_is_callx(&insn[i])) {
- __set_bit(s, callx);
+ err = bpf_call_graph_add(&cg, s, cnt);
if (!bpf_is_unwind_goto(&insn[i + 1]))
plain_callx = true;
}
@@ -6028,8 +5903,8 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
*/
if (!j && bpf_pseudo_call(&insn[i])) {
verbose(env, "recursive call of the main program\n");
- kvfree(calls);
- return -EINVAL;
+ err = -EINVAL;
+ goto out;
}
/* the helper that calls a callback doesn't expect it to unwind */
if (bpf_pseudo_func(&insn[i]) || !bpf_is_unwind_goto(&insn[i + 1]))
@@ -6040,9 +5915,9 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
*/
if (bpf_pseudo_call(&insn[i]) ||
!func_is_kept_for_later(env, i, si[s + 1].start))
- __set_bit(j, calls + s * words);
+ err = bpf_call_graph_add(&cg, s, j);
if (bpf_pseudo_func(&insn[i]))
- __set_bit(j, taken);
+ si[j].addr_taken = true;
}
if (bpf_is_ldimm64(&insn[i]))
i++;
@@ -6051,19 +5926,20 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
for (i = 0; i < env->func_ptr_cnt; i++) {
j = bpf_find_subprog(env, env->func_ptrs[i].xlated_off);
if (j >= 0)
- __set_bit(j, taken);
+ si[j].addr_taken = true;
}
- for_each_set_bit(s, callx, cnt)
- bitmap_or(calls + s * words, calls + s * words, taken, cnt);
- for_each_set_bit(s, taken, cnt)
- si[s].addr_taken = true;
+ for (s = 0; s < cnt && !err; s++)
+ if (si[s].addr_taken)
+ err = bpf_call_graph_add(&cg, cnt, s);
+ if (err)
+ goto out;
/*
* Callers of a global subprog look at can_unwind before the subprog is verified.
* The main prog returns to the kernel.
*/
for (s = 0; s < cnt; s++) {
- if (plain_callx && test_bit(s, taken))
+ if (plain_callx && si[s].addr_taken)
si[s].plain_call = true;
si[s].bail_unwinds = si[s].can_unwind ||
(s && !si[s].plain_call && !bpf_subprog_is_global(env, s) &&
@@ -6071,10 +5947,10 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
!bpf_prog_is_offloaded(env->prog->aux));
}
- close_call_graph(calls, cnt);
+ bpf_call_graph_sort(&cg, 0);
for (s = 0; s < cnt; s++) {
- if (!test_bit(s, calls + s * words))
+ if (!bpf_call_graph_recursive(&cg, s))
continue;
err = -EINVAL;
if (!s) {
@@ -6107,7 +5983,8 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
si[s].recursive = true;
err = 0;
}
- kvfree(calls);
+out:
+ bpf_call_graph_free(&cg);
return err;
}
@@ -6115,9 +5992,10 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
{
enum priv_stack_mode priv_stack_mode = PRIV_STACK_UNKNOWN;
struct bpf_subprog_info *si = env->subprog_info;
- struct bpf_call_graph cg = {};
+ struct bpf_call_graph cg;
int s, ret;
+ find_subprog_properties(env, true);
for (int i = 0; i < env->subprog_cnt; i++) {
if (si[i].has_tail_call) {
/* the stack in arena wouldn't be returned */
@@ -6146,7 +6024,7 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
goto out;
mark_arena_stack_entries(env, &cg);
- for_each_set_bit(s, cg.live, cg.cnt) {
+ for_each_marked_subprog(&cg, s, CG_LIVE) {
/*
* All async_cb subprogs use normal kernel stack. If a particular
* subprog appears in both main prog and async_cb subtree, that
@@ -6154,7 +6032,7 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
* The private stack has room for one frame of a subprog.
* x86-64 uses R9 for both private stack frame pointer and arg6.
*/
- if (priv_stack_mode != PRIV_STACK_ADAPTIVE || test_bit(s, cg.in_async_cb) ||
+ if (priv_stack_mode != PRIV_STACK_ADAPTIVE || (cg.marks[s] & CG_IN_ASYNC_CB) ||
si[s].recursive || (IS_ENABLED(CONFIG_X86_64) && si[s].stack_arg_cnt)) {
si[s].priv_stack_mode = NO_PRIV_STACK;
continue;
@@ -6172,7 +6050,7 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
ret = check_call_depth(env, &cg);
out:
- kvfree(cg.calls);
+ bpf_call_graph_free(&cg);
return ret;
}
@@ -25200,7 +25078,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
- find_subprog_properties(env);
+ find_subprog_properties(env, false);
/* Validate BTF before reporting subprogram layout errors. */
ret = bpf_check_btf_info(env, attr, uattr);
@@ -25479,6 +25357,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->succ);
kvfree(env->gotox_tmp_buf);
kvfree(env->callx_edges);
+ kvfree(env->callx_index.slots);
kvfree(env->func_ptrs);
for (i = 0; i < env->func_set_cnt; i++)
kvfree(env->func_sets[i]);
diff --git a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
index 11af79aa2ced..2a763f6db693 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
@@ -125,7 +125,8 @@ int global_func_many_chains(void *ctx)
/*
* r1 is the number of calls that follow, so main -> fs15 .. fs0 and main -> fs31 .. fs16
- * are executed: 17 frames each. main -> fs31 .. fs0 is in the call graph: 33 frames.
+ * are executed: 17 frames each. With main -> fs16 -> fs15 no call is dead code, and
+ * main -> fs31 .. fs0 is in the call graph: 33 frames.
* libbpf puts fs15() .. fs0() in front of their second caller fs16().
*/
#undef F_0
@@ -138,6 +139,15 @@ SEC("syscall")
__failure __msg("the call stack of 32 frames is too deep !")
__load_if_JITed()
__naked int static_func_deep_stack_second_caller(void)
+{
+ asm volatile ("r1 = 15; call fs15; r1 = 15; call fs31; r1 = 1; call fs16; exit;"
+ ::: __clobber_all);
+}
+
+SEC("syscall")
+__success
+__load_if_JITed()
+__naked int static_func_deep_stack_dead_call(void)
{
asm volatile ("r1 = 15; call fs15; r1 = 15; call fs31; exit;" ::: __clobber_all);
}
diff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
index 81be4ef8c321..a272795781e4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
@@ -137,8 +137,8 @@ int exception_cb_tail_call(__u64 cookie)
}
SEC("tc")
-__description("tail call in an async callback that calls nothing")
-__failure __msg("cannot tail call within callback")
+__description("dead tail call in an async callback")
+__success
__attribute__((btf_decl_tag("exception_callback:exception_cb_tail_call")))
__naked void tail_call_in_async_callback(void)
{
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index bb2b9e8c71df..5357d667fa04 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -1526,8 +1526,7 @@
},
.prog_type = BPF_PROG_TYPE_XDP,
.flags = F_NEEDS_JIT_ENABLED,
- .errstr = "call stack",
- .result = REJECT,
+ .result = ACCEPT,
},
{
"calls: spill into caller stack frame",
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 14/21] bpf: Find the effects of callees on the call graph
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (12 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 13/21] bpf: Keep the call graph in lists of edges Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 15/21] bpf: Link the decl tags of a type Alexei Starovoitov
` (6 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Each load of the address of a table of functions walks all its pointers,
again after each function that the walk finds, and again in each round
of close_callee_effects(). 1000 loads of a table of 128K pointers take
a minute.
Make the map one node of the call graph:
f1: r1 = map ll --+
+--> node of the map --> all functions in the map
f2: r1 = map ll --+
The functions that maps point to are roots of the walk of the cfg, like
the exception callback.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/cfg.c | 240 +++++++++++++++--------------------------------
1 file changed, 75 insertions(+), 165 deletions(-)
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 07412c0b9d30..6bab58797685 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -52,54 +52,6 @@ enum {
BRANCH = 2,
};
-static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off)
-{
- struct bpf_subprog_info *subprog;
-
- subprog = bpf_find_containing_subprog(env, off);
- subprog->changes_pkt_data = true;
-}
-
-static void mark_subprog_might_sleep(struct bpf_verifier_env *env, int off)
-{
- struct bpf_subprog_info *subprog;
-
- subprog = bpf_find_containing_subprog(env, off);
- subprog->might_sleep = true;
-}
-
-static void mark_subprog_might_throw(struct bpf_verifier_env *env, int off)
-{
- struct bpf_subprog_info *subprog;
-
- subprog = bpf_find_containing_subprog(env, off);
- subprog->might_throw = true;
-}
-
-/*
- * 't' is an index of a call-site.
- * 'w' is a callee entry point.
- * Eventually this function would be called when env->cfg.insn_state[w] == EXPLORED.
- * DFS traversal order guarantees that callee's effect marks are correct at that
- * moment, unless the callee calls one of its callers: see close_callee_effects().
- * Returns true if the marks of the caller changed.
- */
-static bool merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
-{
- struct bpf_subprog_info *caller, *callee;
-
- caller = bpf_find_containing_subprog(env, t);
- callee = bpf_find_containing_subprog(env, w);
- if (caller->changes_pkt_data >= callee->changes_pkt_data &&
- caller->might_sleep >= callee->might_sleep &&
- caller->might_throw >= callee->might_throw)
- return false;
- caller->changes_pkt_data |= callee->changes_pkt_data;
- caller->might_sleep |= callee->might_sleep;
- caller->might_throw |= callee->might_throw;
- return true;
-}
-
enum {
DONE_EXPLORING = 0,
KEEP_EXPLORING = 1,
@@ -169,19 +121,13 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
bool visit_callee)
{
int ret, insn_sz;
- int w;
insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
if (ret)
return ret;
- if (visit_callee) {
- w = t + insns[t].imm + 1;
- merge_callee_effects(env, t, w);
- ret = push_insn(t, w, BRANCH, env);
- }
- return ret;
+ return visit_callee ? push_insn(t, t + insns[t].imm + 1, BRANCH, env) : 0;
}
struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
@@ -628,98 +574,63 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
return keep_exploring ? KEEP_EXPLORING : DONE_EXPLORING;
}
-/*
- * Return pointers to functions in the read-only map that ld_imm64 instruction
- * 't' loads the address of, or of its value, if there are any.
- */
-static struct bpf_func_ptr *insn_func_ptrs(struct bpf_verifier_env *env, int t, u32 *cnt)
-{
- struct bpf_insn *insn = &env->prog->insnsi[t];
-
- *cnt = 0;
- if (!env->func_ptr_cnt || !bpf_is_ldimm64(insn))
- return NULL;
- if (insn->src_reg != BPF_PSEUDO_MAP_VALUE &&
- insn->src_reg != BPF_PSEUDO_MAP_IDX_VALUE &&
- insn->src_reg != BPF_PSEUDO_MAP_FD &&
- insn->src_reg != BPF_PSEUDO_MAP_IDX)
- return NULL;
-
- return bpf_map_func_ptrs(env, env->used_maps[env->insn_aux_data[t].map_index], cnt);
-}
-
-/*
- * ld_imm64 that loads the address of a map that has pointers to functions
- * is similar to ld_imm64 with BPF_PSEUDO_FUNC that loads the address of one
- * function: any of them may be read from the map and called via callx later.
- * Treat it as a call of all of them.
- */
-static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
- struct bpf_func_ptr *ptrs, u32 cnt)
-{
- int *insn_stack = env->cfg.insn_stack;
- int *insn_state = env->cfg.insn_state;
- int ret, w;
- u32 i;
-
- ret = push_insn(t, t + 2, FALLTHROUGH, env);
- if (ret)
- return ret;
-
- /*
- * Explore one function at a time like push_insn() does, so that
- * DISCOVERED function is a caller of 't' and not the one that waits in
- * insn_stack[], which merge_callee_effects() relies on.
- */
- for (i = 0; i < cnt; i++) {
- w = ptrs[i].xlated_off;
- if (insn_state[w])
- continue;
-
- if (env->cfg.cur_stack >= env->prog->len)
- return -E2BIG;
-
- insn_stack[env->cfg.cur_stack++] = w;
- insn_state[w] = DISCOVERED;
- return KEEP_EXPLORING;
- }
-
- /* all functions are explored, their effects are known */
- for (i = 0; i < cnt; i++) {
- w = ptrs[i].xlated_off;
- merge_callee_effects(env, t, w);
-
- /* the same mark as push_insn() leaves on a branch target */
- mark_jump_target(env, w);
- }
- return DONE_EXPLORING;
-}
+enum {
+ CHANGES_PKT_DATA = BIT(0),
+ MIGHT_SLEEP = BIT(1),
+ MIGHT_THROW = BIT(2),
+};
/*
- * A subprog that calls one of its callers is EXPLORED before that caller is: it
- * took the marks that the caller had so far, and so did everybody who called it
- * since. Merge again at every call until nothing changes.
+ * A subprog does what its callees do. The load of the address of a function counts as a call.
+ * So does the load of the address of a map with pointers to functions:
+ *
+ * f1: r1 = map ll --+
+ * +--> node of the map --> all functions in the map
+ * f2: r1 = map ll --+
*/
-static void close_callee_effects(struct bpf_verifier_env *env)
+static int merge_callee_effects(struct bpf_verifier_env *env)
{
+ struct bpf_subprog_info *si = env->subprog_info;
struct bpf_insn *insn = env->prog->insnsi;
- struct bpf_func_ptr *ptrs;
- bool changed;
- u32 cnt, j;
- int t;
-
- do {
- changed = false;
- for (t = 0; t < env->prog->len; t++) {
+ u32 cnt = env->subprog_cnt, n, s, t;
+ struct bpf_call_graph cg;
+ struct bpf_func_ptr *ptr;
+ int err;
+
+ err = bpf_call_graph_init(&cg, cnt + env->used_map_cnt);
+ for (s = 0; s < cnt && !err; s++) {
+ cg.marks[s] = (si[s].changes_pkt_data ? CHANGES_PKT_DATA : 0) |
+ (si[s].might_sleep ? MIGHT_SLEEP : 0) |
+ (si[s].might_throw ? MIGHT_THROW : 0);
+ for (t = si[s].start; t < si[s + 1].start && !err; t++) {
if (bpf_pseudo_call(&insn[t]) || bpf_pseudo_func(&insn[t]))
- changed |= merge_callee_effects(env, t, t + insn[t].imm + 1);
- ptrs = insn_func_ptrs(env, t, &cnt);
- for (j = 0; j < cnt; j++)
- changed |= merge_callee_effects(env, t, ptrs[j].xlated_off);
- if (bpf_is_ldimm64(&insn[t]))
- t++;
+ err = bpf_call_graph_add(&cg, s, bpf_find_subprog(env,
+ t + insn[t].imm + 1));
+ else if (bpf_is_ldimm64(&insn[t]) && insn[t].src_reg &&
+ insn[t].src_reg != BPF_PSEUDO_BTF_ID)
+ err = bpf_call_graph_add(&cg, s,
+ cnt + env->insn_aux_data[t].map_index);
}
- } while (changed);
+ }
+ for (s = 0; s < env->used_map_cnt; s++) {
+ ptr = bpf_map_func_ptrs(env, env->used_maps[s], &n);
+ for (; n-- && !err; ptr++)
+ err = bpf_must_abort() ? -EINTR :
+ bpf_call_graph_add(&cg, cnt + s,
+ bpf_find_subprog(env, ptr->xlated_off));
+ }
+ if (err)
+ goto out;
+ bpf_call_graph_sort(&cg, 0);
+ bpf_call_graph_mark_callers(&cg, U16_MAX, 0);
+ for (s = 0; s < cnt; s++) {
+ si[s].changes_pkt_data = cg.marks[s] & CHANGES_PKT_DATA;
+ si[s].might_sleep = cg.marks[s] & MIGHT_SLEEP;
+ si[s].might_throw = cg.marks[s] & MIGHT_THROW;
+ }
+out:
+ bpf_call_graph_free(&cg);
+ return err;
}
/* Visits the instruction at index t and returns one of the following:
@@ -730,17 +641,12 @@ static void close_callee_effects(struct bpf_verifier_env *env)
static int visit_insn(int t, struct bpf_verifier_env *env)
{
struct bpf_insn *insns = env->prog->insnsi, *insn = &insns[t];
- struct bpf_func_ptr *ptrs;
+ struct bpf_subprog_info *sub;
int ret, off, insn_sz;
- u32 cnt;
if (bpf_pseudo_func(insn))
return visit_func_call_insn(t, insns, env, true);
- ptrs = insn_func_ptrs(env, t, &cnt);
- if (ptrs)
- return visit_func_ptrs_insn(t, env, ptrs, cnt);
-
/* All non-branch instructions have a single fall-through edge. */
if (BPF_CLASS(insn->code) != BPF_JMP &&
BPF_CLASS(insn->code) != BPF_JMP32) {
@@ -760,6 +666,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
*/
if (bpf_is_sync_callback_calling_insn(insn))
mark_calls_callback(env, t);
+ sub = bpf_find_containing_subprog(env, t);
if (bpf_helper_call(insn)) {
const struct bpf_func_proto *fp;
@@ -770,11 +677,11 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
* elimination.
*/
if (ret == 0 && fp->might_sleep)
- mark_subprog_might_sleep(env, t);
+ sub->might_sleep = true;
/* nobody knows yet whether the dynptr is that of an skb */
if (bpf_helper_changes_pkt_data(insn->imm) ||
insn->imm == BPF_FUNC_dynptr_write)
- mark_subprog_changes_pkt_data(env, t);
+ sub->changes_pkt_data = true;
} else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
struct bpf_call_arg_meta meta;
@@ -785,11 +692,11 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
* dead code elimination.
*/
if (ret == 0 && bpf_is_kfunc_sleepable(&meta))
- mark_subprog_might_sleep(env, t);
+ sub->might_sleep = true;
if (ret == 0 && bpf_is_kfunc_pkt_changing(&meta))
- mark_subprog_changes_pkt_data(env, t);
+ sub->changes_pkt_data = true;
if (ret == 0 && bpf_is_throw_kfunc(insn))
- mark_subprog_might_throw(env, t);
+ sub->might_throw = true;
}
return visit_func_call_insn(t, insns, env, insn->src_reg == BPF_PSEUDO_CALL);
@@ -828,7 +735,8 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
{
int insn_cnt = env->prog->len;
int *insn_stack, *insn_state;
- int ex_insn_beg, i, ret = 0;
+ int i, root, ret = 0;
+ u32 ptr = 0;
insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
BPF_VERIFIER_GFP);
@@ -842,15 +750,12 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
return -ENOMEM;
}
- ex_insn_beg = env->exception_callback_subprog
- ? env->subprog_info[env->exception_callback_subprog].start
- : 0;
-
- insn_state[0] = DISCOVERED; /* mark 1st insn as discovered */
- insn_stack[0] = 0; /* 0 is the first instruction */
+ root = 0; /* 0 is the first instruction */
+walk_cfg:
+ insn_state[root] = DISCOVERED;
+ insn_stack[0] = root;
env->cfg.cur_stack = 1;
-walk_cfg:
while (env->cfg.cur_stack > 0) {
int t = insn_stack[env->cfg.cur_stack - 1];
@@ -877,11 +782,16 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
goto err_free;
}
- if (ex_insn_beg && insn_state[ex_insn_beg] != EXPLORED) {
- insn_state[ex_insn_beg] = DISCOVERED;
- insn_stack[0] = ex_insn_beg;
- env->cfg.cur_stack = 1;
+ root = env->subprog_info[env->exception_callback_subprog].start;
+ if (insn_state[root] != EXPLORED)
goto walk_cfg;
+
+ /* callx can call what the maps point to */
+ while (ptr < env->func_ptr_cnt) {
+ root = env->func_ptrs[ptr++].xlated_off;
+ mark_jump_target(env, root);
+ if (insn_state[root] != EXPLORED)
+ goto walk_cfg;
}
for (i = 0; i < insn_cnt; i++) {
@@ -910,8 +820,8 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
i++; /* skip second half of ldimm64 */
}
}
- ret = 0; /* cfg looks good */
- close_callee_effects(env);
+ /* cfg looks good */
+ ret = merge_callee_effects(env);
env->prog->aux->changes_pkt_data = env->subprog_info[0].changes_pkt_data;
env->prog->aux->might_sleep = env->subprog_info[0].might_sleep;
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 15/21] bpf: Link the decl tags of a type
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (13 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 14/21] bpf: Find the effects of callees on the call graph Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 16/21] bpf: Remove BPF_MAX_SUBPROGS Alexei Starovoitov
` (5 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The search for a tag looks at all types, for each argument of each
subprog. That is quadratic when the number of subprogs has no limit.
Link the tags when the BTF is loaded:
next_tag[func] -> tag -> tag -> 0
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/btf.c | 45 +++++++++++++++++++++++++++++++++++++++------
1 file changed, 39 insertions(+), 6 deletions(-)
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index fffde271bbd7..9bb98a1c1072 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -281,6 +281,7 @@ struct btf {
char name[MODULE_NAME_LEN];
bool kernel_btf;
__u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
+ u32 *next_tag; /* see btf_link_decl_tags() */
};
enum verifier_phase {
@@ -1907,6 +1908,7 @@ static void btf_free(struct btf *btf)
if (!btf_is_vmlinux(btf))
kvfree(btf->data);
kvfree(btf->base_id_map);
+ kvfree(btf->next_tag);
kfree(btf);
}
@@ -3636,13 +3638,45 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
return BTF_FIELD_FOUND;
}
+/* For BTF of programs: next_tag[type] -> its first tag, next_tag[tag] -> its next tag, ... 0 */
+static int btf_link_decl_tags(struct btf *btf)
+{
+ u32 i;
+
+ btf->next_tag = kvcalloc(btf->nr_types, sizeof(u32), GFP_KERNEL | __GFP_NOWARN);
+ if (!btf->next_tag)
+ return -ENOMEM;
+ for (i = btf->nr_types; --i;) {
+ if (!btf_type_is_decl_tag(btf->types[i]))
+ continue;
+ btf->next_tag[i] = btf->next_tag[btf->types[i]->type];
+ btf->next_tag[btf->types[i]->type] = i;
+ }
+ return 0;
+}
+
+static int btf_type_ptr_cmp(const void *a, const void *b)
+{
+ return cmp_int(*(void **)a, *(void **)b);
+}
+
+/* last_id is 0 for the first tag */
int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key, int last_id)
{
+ int i = last_id ?: btf_named_start_id(btf, false) - 1, n = btf_nr_types(btf);
int len = strlen(tag_key);
- int i, n;
+ struct btf_type **p;
+
+ if (btf->next_tag && !last_id) {
+ /* the types are in the order of their ids in btf->data */
+ p = bsearch(&pt, btf->types + 1, n - 1, sizeof(*p), btf_type_ptr_cmp);
+ if (!p)
+ return -ENOENT;
+ i = p - btf->types;
+ }
- for (i = last_id + 1, n = btf_nr_types(btf); i < n; i++) {
+ while ((i = btf->next_tag ? btf->next_tag[i] ?: n : i + 1) < n) {
const struct btf_type *t = btf_type_by_id(btf, i);
if (!btf_type_is_decl_tag(t))
@@ -3665,8 +3699,7 @@ const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type
const struct btf_type *t;
int len, id;
- id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,
- btf_named_start_id(btf, false) - 1);
+ id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, 0);
if (id < 0)
return ERR_PTR(id);
@@ -6519,7 +6552,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr,
if (err)
goto errout;
- err = btf_parse_type_sec(env);
+ err = btf_parse_type_sec(env) ?: btf_link_decl_tags(btf);
if (err)
goto errout;
@@ -8677,8 +8710,8 @@ static int btf_scan_decl_tags(struct bpf_verifier_env *env,
const struct btf_type *fn_t,
u32 arg_idx, bool is_global, u32 *tags)
{
- int id = btf_named_start_id(btf, false) - 1;
const char tag_key[] = "arg:";
+ int id = 0;
static const struct {
const char *tag_value;
enum btf_arg_tag arg_tag;
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 16/21] bpf: Remove BPF_MAX_SUBPROGS
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (14 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 15/21] bpf: Link the decl tags of a type Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 17/21] bpf: Walk each type once when looking for special fields Alexei Starovoitov
` (4 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Mark the starts of subprogs in a bitmap, count them, then allocate
subprog_info. 180 KB of zeros go away from each load.
The number of the callee was in the 16 bits of insn->off. Put it into imm
of the copy of the call that the JIT gets:
main image: call pc+N is not changed, nothing to undo
func[i]: call subprog# JIT: bpf_jit_get_func_addr()
x86 and sparc use that function now, like the other JITs. The dump of the
program finds the number with a binary search.
Other changes:
- The tags of subprogs that make calls change, and so do their names in
kallsyms.
- The loops of the JIT over subprogs can be stopped.
- The limit of 10000 in stack liveness counts chains of calls only.
- Liveness of registers visits callees first. Else a chain of N calls
takes N rounds.
What stays: bpf_flow_reset() looks at all insns after each global
function. That loop can be stopped between two functions.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/sparc/net/bpf_jit_comp_64.c | 7 +-
arch/x86/net/bpf_jit_comp.c | 12 ++-
include/linux/bpf_verifier.h | 7 +-
kernel/bpf/cfg.c | 2 +-
kernel/bpf/check_btf.c | 5 +-
kernel/bpf/core.c | 7 +-
kernel/bpf/fixups.c | 152 ++++++++++++-------------------
kernel/bpf/liveness.c | 20 +++-
kernel/bpf/syscall.c | 25 ++++-
kernel/bpf/verifier.c | 76 ++++++++--------
10 files changed, 151 insertions(+), 162 deletions(-)
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
index 52f145e3843c..a2172100dba1 100644
--- a/arch/sparc/net/bpf_jit_comp_64.c
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -1213,8 +1213,13 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx)
/* function call */
case BPF_JMP | BPF_CALL:
{
- u8 *func = ((u8 *)__bpf_call_base) + imm;
+ bool fixed;
+ u64 func;
+ int err;
+ err = bpf_jit_get_func_addr(ctx->prog, insn, ctx->prog->aux->func, &func, &fixed);
+ if (err)
+ return err;
ctx->saw_call = true;
emit_call((u32 *)func, ctx);
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 5bc744c42f7f..8b5772262e13 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3323,16 +3323,22 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* call */
case BPF_JMP | BPF_CALL: {
const struct btf_func_model *fm = NULL;
- bool arena_sp_args = false;
+ bool arena_sp_args = false, fixed;
+ u64 addr;
- func = (u8 *) __bpf_call_base + imm32;
+ /* subprogs have addresses in the last pass */
+ err = bpf_jit_get_func_addr(bpf_prog, insn, bpf_prog->aux->func, &addr,
+ &fixed);
+ if (err)
+ return err;
+ func = (u8 *)addr;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
u8 *tcc_load = prog;
LOAD_TAIL_CALL_CNT_PTR(stack_depth);
ip += prog - tcc_load;
}
- if (!imm32)
+ if (fixed && !imm32)
return -EINVAL;
if (arena_sp && call_clobbers_arena_sp(insn)) {
u8 *start = prog;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7bea740eeaee..a11cbf75a8fc 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -901,7 +901,6 @@ struct bpf_insn_aux_data {
union {
enum bpf_reg_type ptr_type; /* pointer type for load/store insns */
struct bpf_map_ptr_state map_ptr_state;
- s32 call_imm; /* saved imm field of call insn */
u32 alu_limit; /* limit for add/sub register with pointer */
struct {
u32 map_index; /* index into used_maps[] */
@@ -1078,8 +1077,6 @@ struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
u32 size);
int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log);
-#define BPF_MAX_SUBPROGS 1024
-
/*
* A pointer to a static subprog in the value of a frozen read-only array map:
* a 64-bit value that is the offset in bytes of the first instruction of
@@ -1354,9 +1351,9 @@ struct bpf_verifier_env {
struct bpf_verifier_log log;
struct bpf_diag *diag;
struct bpf_func_proto bpf_subprog_scratch;
- struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */
+ struct bpf_subprog_info *subprog_info;
/* subprog indices sorted in topological order: leaves first, callers last */
- int subprog_topo_order[BPF_MAX_SUBPROGS + 2];
+ int *subprog_topo_order;
/*
* Pointers to static subprogs found in frozen read-only maps of the
* program, see resolve_func_ptrs(). Sorted by map and map_off.
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 6bab58797685..71ec6063dce6 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -478,7 +478,7 @@ void bpf_free_subprog_jts(struct bpf_verifier_env *env)
{
int i;
- for (i = 0; i < ARRAY_SIZE(env->subprog_info); i++) {
+ for (i = 0; i < env->subprog_cnt; i++) {
kvfree(env->subprog_info[i].jt);
env->subprog_info[i].jt = NULL;
}
diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c
index 77dfb45708b6..79e079e907c9 100644
--- a/kernel/bpf/check_btf.c
+++ b/kernel/bpf/check_btf.c
@@ -163,8 +163,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
urecord = make_bpfptr(attr->func_info, uattr.is_kernel);
krecord = prog->aux->func_info;
- info_aux = kzalloc_objs(*info_aux, nfuncs,
- BPF_VERIFIER_GFP);
+ info_aux = kvzalloc_objs(*info_aux, nfuncs, BPF_VERIFIER_GFP);
if (!info_aux)
return -ENOMEM;
@@ -207,7 +206,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
return 0;
err_free:
- kfree(info_aux);
+ kvfree(info_aux);
return ret;
}
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 147261bf9e6d..2f0c9e246876 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1595,7 +1595,6 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,
const struct bpf_insn *insn, bool extra_pass,
u64 *func_addr, bool *func_addr_fixed)
{
- s16 off = insn->off;
s32 imm = insn->imm;
u8 *addr;
int err;
@@ -1609,8 +1608,8 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,
if (!extra_pass)
addr = NULL;
else if (prog->aux->func &&
- off >= 0 && off < prog->aux->real_func_cnt)
- addr = (u8 *)prog->aux->func[off]->bpf_func;
+ imm >= 0 && imm < prog->aux->real_func_cnt)
+ addr = (u8 *)prog->aux->func[imm]->bpf_func;
else
return -EINVAL;
} else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
@@ -3519,7 +3518,7 @@ static void bpf_prog_free_images(struct work_struct *work)
bpf_jit_free(aux->func[i]);
}
if (aux->real_func_cnt) {
- kfree(aux->func);
+ kvfree(aux->func);
bpf_prog_unlock_free(aux->prog);
} else {
bpf_jit_free(aux->prog);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 3daa419f75a9..4d9b67981520 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1672,6 +1672,11 @@ static int resolve_func_ptrs(struct bpf_verifier_env *env, struct bpf_prog *prog
return 0;
}
+static int insn_callee(struct bpf_verifier_env *env, int idx)
+{
+ return bpf_find_subprog(env, idx + env->prog->insnsi[idx].imm + 1);
+}
+
static int jit_subprogs(struct bpf_verifier_env *env)
{
struct bpf_prog *prog = env->prog, **func, *tmp;
@@ -1681,57 +1686,21 @@ static int jit_subprogs(struct bpf_verifier_env *env)
void *old_bpf_func;
int err, num_exentries;
- for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
- if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
- continue;
-
- /* Upon error here we cannot fall back to interpreter but
- * need a hard reject of the program. Thus -EFAULT is
- * propagated in any case.
- */
- subprog = bpf_find_subprog(env, i + insn->imm + 1);
- if (verifier_bug_if(subprog < 0, env, "No program to jit at insn %d",
- i + insn->imm + 1))
- return -EFAULT;
- /* temporarily remember subprog id inside insn instead of
- * aux_data, since next loop will split up all insns into funcs
- */
- insn->off = subprog;
- /* remember original imm in case JIT fails and fallback
- * to interpreter will be needed
- */
- env->insn_aux_data[i].call_imm = insn->imm;
- /* point imm to __bpf_call_base+1 from JITs point of view */
- insn->imm = 1;
- if (bpf_pseudo_func(insn)) {
-#if defined(MODULES_VADDR)
- u64 addr = MODULES_VADDR;
-#else
- u64 addr = VMALLOC_START;
-#endif
- /* jit (e.g. x86_64) may emit fewer instructions
- * if it learns a u32 imm is the same as a u64 imm.
- * Set close enough to possible prog address.
- */
- insn[0].imm = (u32)addr;
- insn[1].imm = addr >> 32;
- }
- }
-
err = bpf_prog_alloc_jited_linfo(prog);
if (err)
- goto out_undo_insn;
+ goto out_done;
err = -ENOMEM;
- func = kzalloc_objs(prog, env->subprog_cnt);
+ func = kvzalloc_objs(prog, env->subprog_cnt, BPF_VERIFIER_GFP);
if (!func)
- goto out_undo_insn;
+ goto out_done;
for (i = 0; i < env->subprog_cnt; i++) {
subprog_start = subprog_end;
subprog_end = env->subprog_info[i + 1].start;
len = subprog_end - subprog_start;
+ err = -ENOMEM;
/* bpf_prog_run() doesn't call subprogs directly,
* hence main prog stats include the runtime of subprogs.
* subprogs don't have IDs and not reachable via prog_get_next_id
@@ -1744,6 +1713,32 @@ static int jit_subprogs(struct bpf_verifier_env *env)
len * sizeof(struct bpf_insn));
func[i]->type = prog->type;
func[i]->len = len;
+ for (j = 0, insn = func[i]->insnsi; j < len; j++, insn++) {
+ if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
+ continue;
+ /* a hard reject, there is no fall back to the interpreter */
+ subprog = insn_callee(env, subprog_start + j);
+ if (verifier_bug_if(subprog < 0, env, "No program to jit at insn %d",
+ subprog_start + j)) {
+ err = -EFAULT;
+ goto out_free;
+ }
+ /* JITs take the address from aux->func[] in the last pass */
+ insn->imm = subprog;
+ if (bpf_pseudo_func(insn)) {
+#if defined(MODULES_VADDR)
+ u64 addr = MODULES_VADDR;
+#else
+ u64 addr = VMALLOC_START;
+#endif
+ /* jit (e.g. x86_64) may emit fewer instructions
+ * if it learns a u32 imm is the same as a u64 imm.
+ * Set close enough to possible prog address.
+ */
+ insn[0].imm = (u32)addr;
+ insn[1].imm = addr >> 32;
+ }
+ }
if (bpf_prog_calc_tag(func[i]))
goto out_free;
func[i]->is_func = 1;
@@ -1821,7 +1816,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
err = -ENOTSUPP;
goto out_free;
}
- cond_resched();
+ err = -EINTR;
+ if (bpf_must_abort())
+ goto out_free;
}
/* at this point all bpf functions were successfully JITed
@@ -1831,28 +1828,17 @@ static int jit_subprogs(struct bpf_verifier_env *env)
for (i = 0; i < env->subprog_cnt; i++) {
insn = func[i]->insnsi;
for (j = 0; j < func[i]->len; j++, insn++) {
- if (bpf_pseudo_func(insn)) {
- subprog = insn->off;
- insn[0].imm = (u32)(long)func[subprog]->bpf_func;
- insn[1].imm = ((u64)(long)func[subprog]->bpf_func) >> 32;
- continue;
- }
- if (!bpf_pseudo_call(insn))
+ if (!bpf_pseudo_func(insn))
continue;
- subprog = insn->off;
- insn->imm = BPF_CALL_IMM(func[subprog]->bpf_func);
+ subprog = insn_callee(env, func[i]->aux->subprog_start + j);
+ insn[0].imm = (u32)(long)func[subprog]->bpf_func;
+ insn[1].imm = ((u64)(long)func[subprog]->bpf_func) >> 32;
}
/* we use the aux data to keep a list of the start addresses
* of the JITed images for each function in the program
*
- * for some architectures, such as powerpc64, the imm field
- * might not be large enough to hold the offset of the start
- * address of the callee's JITed image from __bpf_call_base
- *
- * in such cases, we can lookup the start address of a callee
- * by using its subprog id, available from the off field of
- * the call instruction, as an index for this list
+ * the imm field of a call has the subprog id, an index for this list
*/
func[i]->aux->func = func;
func[i]->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt;
@@ -1866,7 +1852,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
err = -ENOTSUPP;
goto out_free;
}
- cond_resched();
+ err = -EINTR;
+ if (bpf_must_abort())
+ goto out_free;
}
/* the addresses of all functions are final */
@@ -1900,19 +1888,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
* prog consistent for later dump requests, so they can
* later look the same as if they were interpreted only.
*/
- for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
- if (bpf_pseudo_func(insn)) {
- insn[0].imm = env->insn_aux_data[i].call_imm;
- insn[1].imm = insn->off;
- insn->off = 0;
- continue;
- }
- if (!bpf_pseudo_call(insn))
- continue;
- insn->imm = env->insn_aux_data[i].call_imm;
- subprog = bpf_find_subprog(env, i + insn->imm + 1);
- insn->off = subprog;
- }
+ for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++)
+ if (bpf_pseudo_func(insn))
+ insn[1].imm = insn_callee(env, i);
prog->jited = 1;
prog->bpf_func = func[0]->bpf_func;
@@ -1946,17 +1924,16 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->poke_tab = NULL;
bpf_jit_free(func[i]);
}
- kfree(func);
-out_undo_insn:
+ kvfree(func);
+out_done:
bpf_prog_jit_attempt_done(prog);
return err;
}
int bpf_jit_subprogs(struct bpf_verifier_env *env)
{
- int err, i;
+ int err;
bool blinded = false;
- struct bpf_insn *insn;
struct bpf_prog *prog, *orig_prog;
u32 *orig_subprog_starts;
@@ -1992,28 +1969,11 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env)
return 0;
out_jit_err:
- if (blinded) {
- bpf_jit_prog_release_other(orig_prog, prog);
- /* roll back to the clean original prog */
- prog = env->prog = orig_prog;
- goto out_restore;
- } else {
- if (err != -EFAULT) {
- /*
- * We will fall back to interpreter mode when err is not -EFAULT, before
- * that, insn->off and insn->imm should be restored to their original
- * values since they were modified by jit_subprogs.
- */
- for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
- if (!bpf_pseudo_call(insn))
- continue;
- insn->off = 0;
- insn->imm = env->insn_aux_data[i].call_imm;
- }
- }
+ if (!blinded)
goto out_cleanup;
- }
-
+ bpf_jit_prog_release_other(orig_prog, prog);
+ /* roll back to the clean original prog */
+ prog = env->prog = orig_prog;
out_restore:
bpf_restore_subprog_starts(env, orig_subprog_starts);
kvfree(orig_subprog_starts);
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 053b3875cde6..cfa416f76209 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2365,7 +2365,8 @@ static int analyze_subprog(struct bpf_verifier_env *env,
struct func_instance *prev_instance = NULL;
int j, err;
- if (++env->liveness->subprog_calls > 10000) {
+ /* a subprog on its own is analyzed once, chains of calls multiply */
+ if (depth && ++env->liveness->subprog_calls > 10000) {
verbose(env, "liveness analysis exceeded complexity limit (%d calls)\n",
env->liveness->subprog_calls);
return -E2BIG;
@@ -2817,7 +2818,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
struct bpf_insn *insns = env->prog->insnsi;
struct insn_live_regs *state;
int insn_cnt = env->prog->len;
- int err = 0, i, subprog, start, end;
+ int err = 0, i, n = 0, subprog, start, end, *order;
bool changed, ret_reg_pair;
/* Use the following algorithm:
@@ -2840,7 +2841,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
* any instruction.
*/
state = kvzalloc_objs(*state, insn_cnt, BPF_VERIFIER_GFP);
- if (!state) {
+ order = kvzalloc_objs(*order, insn_cnt, BPF_VERIFIER_GFP);
+ if (!state || !order) {
err = -ENOMEM;
goto out;
}
@@ -2854,6 +2856,15 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
}
+ /* callees first, so a chain of calls takes one round */
+ for (i = 0; i < env->subprog_cnt; i++) {
+ subprog = env->subprog_topo_order[i];
+ start = env->subprog_info[subprog].postorder_start;
+ end = env->subprog_info[subprog + 1].postorder_start;
+ while (start < end)
+ order[n++] = env->cfg.insn_postorder[start++];
+ }
+
changed = true;
while (changed) {
u32 args = mask_widen(GENMASK(BPF_REG_5, BPF_REG_1)), callx_use = 0, declared;
@@ -2882,7 +2893,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
goto out;
}
for (i = 0; i < env->cfg.cur_postorder; ++i) {
- int insn_idx = env->cfg.insn_postorder[i];
+ int insn_idx = order[i];
struct insn_live_regs *live = &state[insn_idx];
struct bpf_iarray *succ;
u32 new_out = 0;
@@ -2943,6 +2954,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
*/
err = bpf_compute_subprog_arg_access(env);
out:
+ kvfree(order);
kvfree(state);
return err;
}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index fb2ecc9d81f5..237c47139e2b 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2465,7 +2465,7 @@ static void __bpf_prog_put_rcu(struct rcu_head *rcu)
kvfree(aux->jited_linfo);
kvfree(aux->linfo);
kvfree(aux->func_info);
- kfree(aux->func_info_aux);
+ kvfree(aux->func_info_aux);
free_uid(aux->user);
security_bpf_prog_free(aux->prog);
bpf_prog_free(aux->prog);
@@ -5035,7 +5035,22 @@ static int bpf_prog_map_addrs(const struct bpf_prog *prog, struct maple_tree *mt
return err;
}
-static void prepare_dump_pseudo_call(struct bpf_insn *insn)
+/* the id of the subprog that starts at insn 'start' */
+static u32 bpf_prog_find_func(const struct bpf_prog *prog, u32 start)
+{
+ u32 l = 0, r = prog->aux->real_func_cnt, m;
+
+ while (r - l > 1) {
+ m = (l + r) / 2;
+ if (prog->aux->func[m]->aux->subprog_start <= start)
+ l = m;
+ else
+ r = m;
+ }
+ return l;
+}
+
+static void prepare_dump_pseudo_call(const struct bpf_prog *prog, struct bpf_insn *insn, u32 idx)
{
s32 call_off = insn->imm;
@@ -5044,12 +5059,12 @@ static void prepare_dump_pseudo_call(struct bpf_insn *insn)
* 1. For interpreter (BPF_CALL_ARGS): insn->off is the index of
* interpreters_args array, so here using bpf_call_args_imm()
* to get the real address offset.
- * 2. For JIT (BPF_CALL): insn->off is the subprog id.
+ * 2. For JIT (BPF_CALL): the subprog id.
*/
if (insn->code == (BPF_JMP | BPF_CALL_ARGS))
insn->imm = bpf_call_args_imm(insn->off);
else
- insn->imm = insn->off;
+ insn->imm = bpf_prog_find_func(prog, idx + call_off + 1);
/* Avoid dumping a truncated and misleading pc-relative offset. */
if (call_off > S16_MAX || call_off < S16_MIN)
@@ -5093,7 +5108,7 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
code == (BPF_JMP | BPF_CALL_ARGS)) {
/* Restore the legacy xlated dump layout. */
if (insns[i].src_reg == BPF_PSEUDO_CALL)
- prepare_dump_pseudo_call(&insns[i]);
+ prepare_dump_pseudo_call(prog, &insns[i], i);
if (code == (BPF_JMP | BPF_CALL_ARGS))
insns[i].code = BPF_JMP | BPF_CALL;
if (!bpf_dump_raw_ok(f_cred))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5fe15a4b3bd8..dde8de6b116e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2602,31 +2602,14 @@ int bpf_find_subprog(struct bpf_verifier_env *env, int off)
return p - env->subprog_info;
}
-static int add_subprog(struct bpf_verifier_env *env, int off)
+static int add_subprog(struct bpf_verifier_env *env, unsigned long *starts, int off)
{
- struct bpf_subprog_info *subprog = env->subprog_info;
- int insn_cnt = env->prog->len;
- int i;
-
- if (off >= insn_cnt || off < 0) {
+ if (off >= env->prog->len || off < 0) {
verbose(env, "call to invalid destination\n");
return -EINVAL;
}
- i = bpf_find_subprog(env, off);
- if (i >= 0)
- return i;
- if (env->subprog_cnt >= BPF_MAX_SUBPROGS) {
- verbose(env, "too many subprograms\n");
- return -E2BIG;
- }
- /*
- * Determine subprog starts. The end is one before the next starts.
- * The array stays sorted: subprogs that start after 'off' move up.
- */
- for (i = env->subprog_cnt++; i > 0 && subprog[i - 1].start > off; i--)
- subprog[i] = subprog[i - 1];
- subprog[i] = (struct bpf_subprog_info){ .start = off, .exit_idx = U32_MAX };
- return i;
+ __set_bit(off, starts);
+ return 0;
}
static int bpf_find_exception_callback_insn_off(struct bpf_verifier_env *env)
@@ -3103,15 +3086,17 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
static int add_subprogs(struct bpf_verifier_env *env)
{
- struct bpf_subprog_info *subprog = env->subprog_info;
- int i, ret, insn_cnt = env->prog->len, ex_cb_insn;
+ int i, cnt, ret, insn_cnt = env->prog->len, ex_cb_insn;
+ unsigned long *starts __free(kvfree) = NULL;
struct bpf_insn *insn = env->prog->insnsi;
const char *operation, *suggestion;
+ struct bpf_subprog_info *subprog;
+ starts = kvcalloc(BITS_TO_LONGS(insn_cnt), sizeof(long), BPF_VERIFIER_GFP);
+ if (!starts)
+ return -ENOMEM;
/* Add entry function. */
- ret = add_subprog(env, 0);
- if (ret)
- return ret;
+ __set_bit(0, starts);
for (i = 0; i < insn_cnt; i++, insn++) {
if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
@@ -3133,7 +3118,7 @@ static int add_subprogs(struct bpf_verifier_env *env)
return -EPERM;
}
- ret = add_subprog(env, i + insn->imm + 1);
+ ret = add_subprog(env, starts, i + insn->imm + 1);
if (ret < 0)
return ret;
}
@@ -3148,7 +3133,7 @@ static int add_subprogs(struct bpf_verifier_env *env)
struct bpf_prog_aux *aux = env->prog->aux;
for (i = 1; i < aux->func_info_cnt; i++) {
- ret = add_subprog(env, aux->func_info[i].insn_off);
+ ret = add_subprog(env, starts, aux->func_info[i].insn_off);
if (ret < 0)
return ret;
}
@@ -3162,23 +3147,32 @@ static int add_subprogs(struct bpf_verifier_env *env)
/* If ex_cb_insn > 0, this means that the main program has a subprog
* marked using BTF decl tag to serve as the exception callback.
*/
- if (ex_cb_insn) {
- ret = add_subprog(env, ex_cb_insn);
- if (ret < 0)
- return ret;
- for (i = 1; i < env->subprog_cnt; i++) {
- if (env->subprog_info[i].start != ex_cb_insn)
- continue;
- env->exception_callback_subprog = i;
- bpf_mark_subprog_exc_cb(env, i);
- break;
- }
+ ret = add_subprog(env, starts, ex_cb_insn);
+ if (ret < 0)
+ return ret;
+
+ /* + 2 for the fake and the hidden subprogs */
+ cnt = bitmap_weight(starts, insn_cnt);
+ subprog = env->subprog_info = kvcalloc(cnt + 2, sizeof(*subprog), BPF_VERIFIER_GFP);
+ env->subprog_topo_order = kvcalloc(cnt, sizeof(int), BPF_VERIFIER_GFP);
+ if (!subprog || !env->subprog_topo_order)
+ return -ENOMEM;
+
+ /* Determine subprog starts. The end is one before the next starts. */
+ for_each_set_bit(i, starts, insn_cnt) {
+ subprog[env->subprog_cnt].start = i;
+ subprog[env->subprog_cnt++].exit_idx = U32_MAX;
}
/* Add a fake 'exit' subprog which could simplify subprog iteration
* logic. 'subprog_cnt' should not be increased.
*/
- subprog[env->subprog_cnt].start = insn_cnt;
+ subprog[cnt].start = insn_cnt;
+
+ if (ex_cb_insn) {
+ env->exception_callback_subprog = bpf_find_subprog(env, ex_cb_insn);
+ bpf_mark_subprog_exc_cb(env, env->exception_callback_subprog);
+ }
if (env->log.level & BPF_LOG_LEVEL2)
for (i = 0; i < env->subprog_cnt; i++)
@@ -25365,6 +25359,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->func_set_index.slots);
bpf_diag_free(env);
kvfree(env->idoms);
+ kvfree(env->subprog_info);
+ kvfree(env->subprog_topo_order);
kvfree(env);
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 17/21] bpf: Walk each type once when looking for special fields
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (15 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 16/21] bpf: Remove BPF_MAX_SUBPROGS Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 18/21] bpf: Raise the limit on the length of BTF names Alexei Starovoitov
` (3 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The walk enters a struct again for each member of that type. With three
members on each level it takes 3^depth steps, in recursion, and cannot be
killed. Only the limit of 32 levels ends it.
Enter a struct once and copy the fields that were found:
struct in { kptr p; };
struct out { struct in a; long x; struct in b[2]; };
enter 'in' copy from a, twice
The walk has its own stack, so the depth is free. The table of fields
grows, so BTF_FIELDS_MAX goes away. Fields are found in order and do not
overlap: look them up with a binary search.
A map value with 12 kptrs had no record at all. It has one now, so the
rules for special fields apply: the map needs CAP_BPF, and it cannot be
mmaped or BPF_F_RDONLY_PROG.
The size of a record is bounded by the size of what it describes. For a
map it is value_size. An allocated object has 4096 bytes, that is 512
fields. A struct with more gets no record when the BTF is loaded.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 5 -
include/linux/bpf_verifier.h | 15 +
kernel/bpf/btf.c | 334 ++++++++----------
kernel/bpf/syscall.c | 6 +-
kernel/bpf/verifier.c | 67 ++--
.../selftests/bpf/progs/cpumask_common.h | 3 -
.../selftests/bpf/progs/cpumask_failure.c | 5 +-
7 files changed, 197 insertions(+), 238 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 16dd9b3d6f33..a5ad08b7937c 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -193,11 +193,6 @@ struct bpf_map_ops {
const struct bpf_iter_seq_info *iter_seq_info;
};
-enum {
- /* Support at most 11 fields in a BTF type */
- BTF_FIELDS_MAX = 11,
-};
-
enum btf_field_type {
BPF_SPIN_LOCK = (1 << 0),
BPF_TIMER = (1 << 1),
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index a11cbf75a8fc..f45c18b3cfa2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1975,6 +1975,21 @@ static inline int bpf_head_guard_target(const struct bpf_verifier_env *env, int
}
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+/* The first field that ends behind 'off'. The fields are sorted and do not overlap. */
+static inline u32 btf_record_first_field(const struct btf_record *rec, s64 off)
+{
+ u32 l = 0, r = rec->cnt, m;
+
+ while (l < r) {
+ m = (l + r) / 2;
+ if (rec->fields[m].offset + rec->fields[m].size <= off)
+ l = m + 1;
+ else
+ r = m;
+ }
+ return l;
+}
+
/* Kinds of member a by-value struct or union may be composed of. */
enum btf_member_kind {
BTF_MEMBER_SCALAR = BIT(0), /* an int or an enum */
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 9bb98a1c1072..f97b8e754863 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -3836,106 +3836,106 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_
return type;
}
-/* Repeat a number of fields for a specified number of times.
+/*
+ * A walk over the members of a type and of the structs in it. A struct is entered once.
+ * The next member of that type gets a copy of the fields that were found:
*
- * Copy the fields starting from the first field and repeat them for
- * repeat_cnt times. The fields are repeated by adding the offset of each
- * field with
- * (i + 1) * elem_size
- * where i is the repeat index and elem_size is the size of an element.
+ * struct in { kptr p; };
+ * struct out { struct in a; long x; struct in b[2]; };
+ * enter 'in' copy from a, twice
*/
-static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
- u32 field_cnt, u32 repeat_cnt, u32 elem_size)
-{
- u32 i, j, total_cnt, total_repeats;
- u32 cur;
-
- /* Ensure not repeating fields that should not be repeated. */
- for (i = 0; i < field_cnt; i++) {
- switch (info[i].type) {
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- case BPF_LIST_HEAD:
- case BPF_RB_ROOT:
- break;
- default:
- return -EINVAL;
- }
- }
+struct btf_field_walk {
+ struct btf_field_info *info;
+ u32 cnt;
+ u32 field_mask;
+ u32 seen_mask;
+ struct btf_field_scope {
+ const struct btf_type *t;
+ u32 off; /* in the outer type */
+ u32 nelems;
+ u32 first; /* its fields in 'info' */
+ u32 cnt;
+ u32 next; /* member to look at */
+ u32 parent;
+ } *scopes;
+ u32 scope; /* the current one */
+ struct bpf_index index; /* address of the type -> scope */
+};
- if (check_add_overflow(repeat_cnt, 1, &total_repeats) ||
- check_mul_overflow(field_cnt, total_repeats, &total_cnt) ||
- total_cnt > (u32)info_cnt)
- return -E2BIG;
+/* An allocated object has 4096 bytes at most */
+#define BTF_OBJ_FIELDS_MAX (SZ_4K / sizeof(u64))
- cur = field_cnt;
- for (i = 0; i < repeat_cnt; i++) {
- memcpy(&info[cur], &info[0], field_cnt * sizeof(info[0]));
- for (j = 0; j < field_cnt; j++)
- info[cur++].off += (i + 1) * elem_size;
- }
+static int btf_add_field(struct btf_field_walk *w, struct btf_field_info info)
+{
+ struct btf_field_info *last = w->cnt ? &w->info[w->cnt - 1] : NULL;
+ if (last && info.off < last->off + btf_field_type_size(last->type))
+ return -EEXIST;
+ if ((w->field_mask & BPF_GRAPH_NODE) && w->cnt == BTF_OBJ_FIELDS_MAX)
+ return -EFBIG;
+ if (bpf_must_abort())
+ return -EINTR;
+ if (!bpf_table_next(w->info, w->cnt))
+ return -ENOMEM;
+ w->info[w->cnt++] = info;
return 0;
}
-static int btf_find_struct_field(const struct btf *btf,
- const struct btf_type *t, u32 field_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level, u32 *seen_mask);
-
-/* Find special fields in the struct type of a field.
- *
- * This function is used to find fields of special types that is not a
- * global variable or a direct field of a struct type. It also handles the
- * repetition if it is the element type of an array.
- */
-static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
- u32 off, u32 nelems,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt, u32 level, u32 *seen_mask)
+/* Copy 'cnt' fields from 'first' on to 'nelems' elements. The first copy is 'by' bytes off. */
+static int btf_repeat_fields(struct btf_field_walk *w, u32 first, u32 cnt, u32 by,
+ u32 nelems, u32 elem_size, bool array)
{
- int ret, err, i;
-
- level++;
- if (level >= MAX_RESOLVE_DEPTH)
- return -E2BIG;
-
- ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level, seen_mask);
+ struct btf_field_info info;
+ u32 i, j;
+ int err;
- if (ret <= 0)
- return ret;
+ for (i = 0; i < nelems && cnt; i++, by += elem_size) {
+ for (j = first; j < first + cnt; j++) {
+ info = w->info[j];
+ info.off += by;
+ if (array && !(info.type & (BPF_KPTR | BPF_UPTR | BPF_GRAPH_ROOT)))
+ return -EINVAL;
+ if (info.type & w->seen_mask)
+ return -E2BIG;
+ err = btf_add_field(w, info);
+ if (err)
+ return err;
+ }
+ }
+ return 0;
+}
- /* Shift the offsets of the nested struct fields to the offsets
- * related to the container.
- */
- for (i = 0; i < ret; i++)
- info[i].off += off;
+static int btf_enter_struct(struct btf_field_walk *w, const struct btf_type *t, u32 off, u32 nelems)
+{
+ int i = bpf_index_find(&w->index, (unsigned long)t);
+ struct btf_field_scope *s;
- if (nelems > 1) {
- err = btf_repeat_fields(info, info_cnt, ret, nelems - 1, t->size);
- if (err == 0)
- ret *= nelems;
- else
- ret = err;
+ if (i >= 0) {
+ s = &w->scopes[i];
+ return btf_repeat_fields(w, s->first, s->cnt, off - s->off, nelems, t->size,
+ nelems > 1);
}
-
- return ret;
+ i = w->index.cnt;
+ s = bpf_table_next(w->scopes, i);
+ if (!s || bpf_index_add(&w->index, (unsigned long)t, i))
+ return -ENOMEM;
+ *s = (struct btf_field_scope){
+ .t = t, .off = off, .nelems = nelems, .first = w->cnt, .parent = w->scope,
+ };
+ w->scope = i;
+ return 0;
}
static int btf_find_field_one(const struct btf *btf,
const struct btf_type *var,
const struct btf_type *var_type,
int var_idx,
- u32 off, u32 expected_size,
- u32 field_mask, u32 *seen_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level)
+ u32 base, u32 off, u32 expected_size,
+ struct btf_field_walk *w)
{
int ret, align, sz, field_type;
- struct btf_field_info tmp;
const struct btf_array *array;
+ struct btf_field_info info;
u32 i, nelems = 1;
/* Walk into array types to find the element type and the number of
@@ -3952,15 +3952,13 @@ static int btf_find_field_one(const struct btf *btf,
return 0;
field_type = btf_get_field_type(btf, var_type,
- field_mask, seen_mask, &align, &sz);
+ w->field_mask, &w->seen_mask, &align, &sz);
/* Look into variables of struct types */
if (!field_type && __btf_type_is_struct(var_type)) {
sz = var_type->size;
if (expected_size && expected_size != sz * nelems)
return 0;
- ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
- &info[0], info_cnt, level, seen_mask);
- return ret;
+ return btf_enter_struct(w, var_type, base + off, nelems);
}
if (field_type == 0)
@@ -3972,6 +3970,7 @@ static int btf_find_field_one(const struct btf *btf,
return 0;
if (off % align)
return 0;
+ off += base;
switch (field_type) {
case BPF_SPIN_LOCK:
@@ -3983,8 +3982,7 @@ static int btf_find_field_one(const struct btf *btf,
case BPF_REFCOUNT:
case BPF_TASK_WORK:
case BPF_RCU_HEAD:
- ret = btf_find_struct(btf, var_type, off, sz, field_type,
- info_cnt ? &info[0] : &tmp);
+ ret = btf_find_struct(btf, var_type, off, sz, field_type, &info);
if (ret < 0)
return ret;
break;
@@ -3992,17 +3990,14 @@ static int btf_find_field_one(const struct btf *btf,
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
- ret = btf_find_kptr(btf, var_type, off, sz,
- info_cnt ? &info[0] : &tmp, field_mask);
+ ret = btf_find_kptr(btf, var_type, off, sz, &info, w->field_mask);
if (ret < 0)
return ret;
break;
case BPF_LIST_HEAD:
case BPF_RB_ROOT:
ret = btf_find_graph_root(btf, var, var_type,
- var_idx, off, sz,
- info_cnt ? &info[0] : &tmp,
- field_type);
+ var_idx, off, sz, &info, field_type);
if (ret < 0)
return ret;
break;
@@ -4012,81 +4007,51 @@ static int btf_find_field_one(const struct btf *btf,
if (ret == BTF_FIELD_IGNORE)
return 0;
- if (!info_cnt)
- return -E2BIG;
- if (nelems > 1) {
- ret = btf_repeat_fields(info, info_cnt, 1, nelems - 1, sz);
- if (ret < 0)
- return ret;
- }
- return nelems;
+ return btf_add_field(w, info) ?:
+ btf_repeat_fields(w, w->cnt - 1, 1, sz, nelems - 1, sz, true);
}
-static int btf_find_struct_field(const struct btf *btf,
- const struct btf_type *t, u32 field_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level, u32 *seen_mask)
+static int btf_find_field(const struct btf *btf, const struct btf_type *t,
+ struct btf_field_walk *w)
{
- int ret, idx = 0;
+ const struct btf_var_secinfo *vsi;
const struct btf_member *member;
+ struct btf_field_scope *s;
+ const struct btf_type *var;
u32 i, off;
+ int err;
- for_each_member(i, t, member) {
- const struct btf_type *member_type = btf_type_by_id(btf,
- member->type);
-
- off = __btf_member_bit_offset(t, member);
- if (off % 8)
- /* valid C code cannot generate such BTF */
- return -EINVAL;
- off /= 8;
-
- ret = btf_find_field_one(btf, t, member_type, i,
- off, 0,
- field_mask, seen_mask,
- &info[idx], info_cnt - idx, level);
- if (ret < 0)
- return ret;
- idx += ret;
- }
- return idx;
-}
-
-static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt, u32 level, u32 *seen_mask)
-{
- int ret, idx = 0;
- const struct btf_var_secinfo *vsi;
- u32 i, off;
+ if (!__btf_type_is_struct(t) && !btf_type_is_datasec(t))
+ return -EINVAL;
- for_each_vsi(i, t, vsi) {
- const struct btf_type *var = btf_type_by_id(btf, vsi->type);
- const struct btf_type *var_type = btf_type_by_id(btf, var->type);
-
- off = vsi->offset;
- ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
- field_mask, seen_mask,
- &info[idx], info_cnt - idx,
- level);
- if (ret < 0)
- return ret;
- idx += ret;
+ for (err = btf_enter_struct(w, t, 0, 1); !err;) {
+ if (bpf_must_abort())
+ return -EINTR;
+ s = &w->scopes[w->scope];
+ i = s->next++;
+ if (i == btf_type_vlen(s->t)) {
+ if (!w->scope)
+ break;
+ s->cnt = w->cnt - s->first;
+ w->scope = s->parent;
+ err = btf_repeat_fields(w, s->first, s->cnt, s->t->size, s->nelems - 1,
+ s->t->size, true);
+ } else if (btf_type_is_datasec(s->t)) {
+ vsi = btf_type_var_secinfo(s->t) + i;
+ var = btf_type_by_id(btf, vsi->type);
+ err = btf_find_field_one(btf, var, btf_type_by_id(btf, var->type), -1,
+ 0, vsi->offset, vsi->size, w);
+ } else {
+ member = btf_type_member(s->t) + i;
+ off = __btf_member_bit_offset(s->t, member);
+ if (off % 8)
+ /* valid C code cannot generate such BTF */
+ return -EINVAL;
+ err = btf_find_field_one(btf, s->t, btf_type_by_id(btf, member->type), i,
+ s->off, off / 8, 0, w);
+ }
}
- return idx;
-}
-
-static int btf_find_field(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt)
-{
- u32 seen_mask = 0;
-
- if (__btf_type_is_struct(t))
- return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0, &seen_mask);
- else if (btf_type_is_datasec(t))
- return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0, &seen_mask);
- return -EINVAL;
+ return err;
}
/* Callers have to ensure the life cycle of btf if it is program BTF */
@@ -4238,37 +4203,29 @@ static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field,
__alignof__(struct bpf_rb_node));
}
-static int btf_field_cmp(const void *_a, const void *_b, const void *priv)
-{
- const struct btf_field *a = (const struct btf_field *)_a;
- const struct btf_field *b = (const struct btf_field *)_b;
-
- if (a->offset < b->offset)
- return -1;
- else if (a->offset > b->offset)
- return 1;
- return 0;
-}
-
struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
u32 field_mask, u32 value_size)
{
- struct btf_field_info info_arr[BTF_FIELDS_MAX];
- u32 next_off = 0, field_type_size;
+ struct btf_field_info *info_arr __free(kvfree) = NULL;
+ struct btf_field_walk w = { .field_mask = field_mask };
+ u32 field_type_size;
struct btf_record *rec;
int ret, i, cnt;
- ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr));
+ ret = btf_find_field(btf, t, &w);
+ kvfree(w.scopes);
+ kvfree(w.index.slots);
+ info_arr = w.info;
if (ret < 0)
return ERR_PTR(ret);
- if (!ret)
+ if (!w.cnt)
return NULL;
- cnt = ret;
+ cnt = w.cnt;
/* This needs to be kzalloc to zero out padding and unused fields, see
* comment in btf_record_equal.
*/
- rec = kzalloc_flex(*rec, fields, cnt, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ rec = kvzalloc_flex(*rec, fields, cnt, BPF_VERIFIER_GFP);
if (!rec)
return ERR_PTR(-ENOMEM);
@@ -4286,11 +4243,10 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
ret = -EFAULT;
goto end;
}
- if (info_arr[i].off < next_off) {
- ret = -EEXIST;
+ if (bpf_must_abort()) {
+ ret = -EINTR;
goto end;
}
- next_off = info_arr[i].off + field_type_size;
rec->field_mask |= info_arr[i].type;
rec->fields[i].offset = info_arr[i].off;
@@ -4379,9 +4335,6 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
goto end;
}
- sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp,
- NULL, rec);
-
return rec;
end:
btf_record_free(rec);
@@ -6369,6 +6322,10 @@ btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf)
ret = -EINVAL;
goto free_aof;
}
+ if (bpf_must_abort()) {
+ ret = -EINTR;
+ goto free_aof;
+ }
ret = btf_find_kptr(btf, t, 0, 0, &tmp, BPF_KPTR);
if (ret != BTF_FIELD_FOUND)
@@ -6426,6 +6383,9 @@ btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf)
record = btf_parse_fields(btf, t, BPF_SPIN_LOCK | BPF_RES_SPIN_LOCK | BPF_LIST_HEAD | BPF_LIST_NODE |
BPF_RB_ROOT | BPF_RB_NODE | BPF_REFCOUNT |
BPF_KPTR, t->size);
+ /* It cannot be allocated. A map has its own record. */
+ if (record == ERR_PTR(-EFBIG))
+ continue;
/* The record cannot be unset, treat it as an error if so */
if (IS_ERR_OR_NULL(record)) {
ret = PTR_ERR_OR_ZERO(record) ?: -EFAULT;
@@ -8170,15 +8130,11 @@ int btf_struct_access(struct bpf_verifier_log *log,
if (!meta)
break;
rec = meta->record;
- for (i = 0; i < rec->cnt; i++) {
- struct btf_field *field = &rec->fields[i];
- u32 offset = field->offset;
- if (off < offset + field->size && offset < off + size) {
- bpf_log(log,
- "direct access to %s is disallowed\n",
- btf_field_type_name(field->type));
- return -EACCES;
- }
+ i = btf_record_first_field(rec, off);
+ if (i < rec->cnt && rec->fields[i].offset < off + size) {
+ bpf_log(log, "direct access to %s is disallowed\n",
+ btf_field_type_name(rec->fields[i].type));
+ return -EACCES;
}
break;
}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 237c47139e2b..ec8e78792911 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -703,7 +703,7 @@ void btf_record_free(struct btf_record *rec)
continue;
}
}
- kfree(rec);
+ kvfree(rec);
}
void bpf_map_free_record(struct bpf_map *map)
@@ -721,7 +721,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec)
if (IS_ERR_OR_NULL(rec))
return NULL;
size = struct_size(rec, fields, rec->cnt);
- new_rec = kmemdup(rec, size, GFP_KERNEL | __GFP_NOWARN);
+ new_rec = kvmemdup(rec, size, BPF_VERIFIER_GFP);
if (!new_rec)
return ERR_PTR(-ENOMEM);
/* Do a deep copy of the btf_record */
@@ -1293,6 +1293,8 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
BPF_RB_ROOT | BPF_REFCOUNT | BPF_WORKQUEUE | BPF_UPTR |
BPF_TASK_WORK | BPF_RCU_HEAD,
map->value_size);
+ if (map->record == ERR_PTR(-ENOMEM) || map->record == ERR_PTR(-EINTR))
+ return PTR_ERR(map->record);
if (!IS_ERR_OR_NULL(map->record)) {
int i;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index dde8de6b116e..1a8e5a108a68 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4839,48 +4839,43 @@ static int check_map_access(struct bpf_verifier_env *env, struct bpf_reg_state *
if (IS_ERR_OR_NULL(map->record))
return 0;
rec = map->record;
- for (i = 0; i < rec->cnt; i++) {
+ /* If any part of a field can be touched by load/store, reject this program. */
+ for (i = btf_record_first_field(rec, reg_smin(reg) + off);
+ i < rec->cnt && rec->fields[i].offset < reg_umax(reg) + off + size; i++) {
struct btf_field *field = &rec->fields[i];
u32 p = field->offset;
- /* If any part of a field can be touched by load/store, reject
- * this program. To check that [x1, x2) overlaps with [y1, y2),
- * it is sufficient to check x1 < y2 && y1 < x2.
- */
- if (reg_smin(reg) + off < p + field->size &&
- p < reg_umax(reg) + off + size) {
- switch (field->type) {
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- if (src != ACCESS_DIRECT) {
- verbose(env, "%s cannot be accessed indirectly by helper\n",
- btf_field_type_name(field->type));
- return -EACCES;
- }
- if (!tnum_is_const(reg->var_off)) {
- verbose(env, "%s access cannot have variable offset\n",
- btf_field_type_name(field->type));
- return -EACCES;
- }
- if (p != off + reg->var_off.value) {
- verbose(env, "%s access misaligned expected=%u off=%llu\n",
- btf_field_type_name(field->type),
- p, off + reg->var_off.value);
- return -EACCES;
- }
- if (size != bpf_size_to_bytes(BPF_DW)) {
- verbose(env, "%s access size must be BPF_DW\n",
- btf_field_type_name(field->type));
- return -EACCES;
- }
- break;
- default:
- verbose(env, "%s cannot be accessed directly by load/store\n",
+ switch (field->type) {
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ if (src != ACCESS_DIRECT) {
+ verbose(env, "%s cannot be accessed indirectly by helper\n",
btf_field_type_name(field->type));
return -EACCES;
}
+ if (!tnum_is_const(reg->var_off)) {
+ verbose(env, "%s access cannot have variable offset\n",
+ btf_field_type_name(field->type));
+ return -EACCES;
+ }
+ if (p != off + reg->var_off.value) {
+ verbose(env, "%s access misaligned expected=%u off=%llu\n",
+ btf_field_type_name(field->type),
+ p, off + reg->var_off.value);
+ return -EACCES;
+ }
+ if (size != bpf_size_to_bytes(BPF_DW)) {
+ verbose(env, "%s access size must be BPF_DW\n",
+ btf_field_type_name(field->type));
+ return -EACCES;
+ }
+ break;
+ default:
+ verbose(env, "%s cannot be accessed directly by load/store\n",
+ btf_field_type_name(field->type));
+ return -EACCES;
}
}
return 0;
diff --git a/tools/testing/selftests/bpf/progs/cpumask_common.h b/tools/testing/selftests/bpf/progs/cpumask_common.h
index 8fe01308d210..03edad35aa13 100644
--- a/tools/testing/selftests/bpf/progs/cpumask_common.h
+++ b/tools/testing/selftests/bpf/progs/cpumask_common.h
@@ -7,9 +7,6 @@
#include "errno.h"
#include <stdbool.h>
-/* Should use BTF_FIELDS_MAX, but it is not always available in vmlinux.h,
- * so use the hard-coded number as a workaround.
- */
#define CPUMASK_KPTR_FIELDS_MAX 11
int err;
diff --git a/tools/testing/selftests/bpf/progs/cpumask_failure.c b/tools/testing/selftests/bpf/progs/cpumask_failure.c
index 2a58209eb8be..47967f960bbc 100644
--- a/tools/testing/selftests/bpf/progs/cpumask_failure.c
+++ b/tools/testing/selftests/bpf/progs/cpumask_failure.c
@@ -15,7 +15,6 @@ struct kptr_nested_array_2 {
};
struct kptr_nested_array_1 {
- /* Make btf_parse_fields() in map_create() return -E2BIG */
struct kptr_nested_array_2 d_2[CPUMASK_KPTR_FIELDS_MAX + 1];
};
@@ -213,8 +212,8 @@ int BPF_PROG(test_global_mask_rcu_no_null_check, struct task_struct *task, u64 c
}
SEC("tp_btf/task_newtask")
-__failure __msg("has no valid kptr")
-int BPF_PROG(test_invalid_nested_array, struct task_struct *task, u64 clone_flags)
+__success
+int BPF_PROG(test_big_nested_array, struct task_struct *task, u64 clone_flags)
{
struct bpf_cpumask *local, *prev;
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 18/21] bpf: Raise the limit on the length of BTF names
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (16 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 17/21] bpf: Walk each type once when looking for special fields Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 19/21] bpf: Find dominators in O(n log n) Alexei Starovoitov
` (2 subsequent siblings)
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A name of 513 bytes refuses the whole BTF. Mangled names can be longer.
The users that copy a name use snprintf(). Raise the limit to 4096. It
cannot go away: a search by name takes types x length, and so does the
log. The search can be stopped now.
Many types can point into one long string. Look at each byte once and
keep a bit for each offset:
strings: f o o \0 b \7 r \0
names: 1 1 1 0 0 0 1 0
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/btf.c | 108 +++++++++++++++++++++++------------------------
1 file changed, 52 insertions(+), 56 deletions(-)
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index f97b8e754863..e95ca39f990c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -309,6 +309,9 @@ enum resolve_mode {
*/
};
+/* It bounds the time of the loops that compare or print names */
+#define BTF_MAX_NAME_LEN 4096
+
#define MAX_RESOLVE_DEPTH 32
struct btf_sec_info {
@@ -320,6 +323,7 @@ struct btf_verifier_env {
struct btf *btf;
u8 *visit_states;
struct resolve_vertex stack[MAX_RESOLVE_DEPTH];
+ unsigned long *names; /* offsets of the strings that can be names */
struct bpf_verifier_log log;
u32 log_type_id;
u32 top_stack;
@@ -685,10 +689,15 @@ s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)
return id;
}
} else {
+ /* only the tags in BTF of a program have long names */
+ bool slow = !btf->kernel_btf && strlen(name) > KSYM_NAME_LEN;
+
for (id = btf_start_id(btf); id < total; id++) {
t = btf_type_by_id(btf, id);
if (BTF_INFO_KIND(t->info) != kind)
continue;
+ if (slow && bpf_must_abort())
+ return -EINTR;
tname = btf_name_by_offset(btf, t->name_off);
if (strcmp(tname, name) == 0)
return id;
@@ -915,49 +924,6 @@ const char *btf_str_by_offset(const struct btf *btf, u32 offset)
return NULL;
}
-/* Names in BTF of Rust are not C identifiers. Allow any printable character */
-static bool btf_name_valid_identifier(const struct btf *btf, u32 offset)
-{
- /* offset must be valid */
- const char *src = btf_str_by_offset(btf, offset);
- const char *src_limit;
-
- if (!isprint(*src))
- return false;
-
- /* set a limit on identifier length */
- src_limit = src + KSYM_NAME_LEN;
- src++;
- while (*src && src < src_limit) {
- if (!isprint(*src))
- return false;
- src++;
- }
-
- return !*src;
-}
-
-/* Allow any printable character in DATASEC names */
-static bool btf_name_valid_section(const struct btf *btf, u32 offset)
-{
- /* offset must be valid */
- const char *src = btf_str_by_offset(btf, offset);
- const char *src_limit;
-
- if (!*src)
- return false;
-
- /* set a limit on identifier length */
- src_limit = src + KSYM_NAME_LEN;
- while (*src && src < src_limit) {
- if (!isprint(*src))
- return false;
- src++;
- }
-
- return !*src;
-}
-
static const char *__btf_name_by_offset(const struct btf *btf, u32 offset)
{
const char *name;
@@ -1993,9 +1959,21 @@ static int env_resolve_init(struct btf_verifier_env *env)
static void btf_verifier_env_free(struct btf_verifier_env *env)
{
kvfree(env->visit_states);
+ kvfree(env->names);
kfree(env);
}
+/* Names in BTF of Rust are not C identifiers. Allow any printable character */
+static bool btf_name_valid(const struct btf_verifier_env *env, u32 offset)
+{
+ const struct btf *btf = env->btf;
+
+ /* offset must be valid */
+ if (offset < btf->start_str_off)
+ return *btf_str_by_offset(btf, offset);
+ return test_bit(offset - btf->start_str_off, env->names);
+}
+
static bool env_type_is_resolve_sink(const struct btf_verifier_env *env,
const struct btf_type *next_type)
{
@@ -2768,7 +2746,7 @@ static int btf_ref_type_check_meta(struct btf_verifier_env *env,
*/
if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF) {
if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -3030,7 +3008,7 @@ static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
/* fwd type must have a valid name */
if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -3353,7 +3331,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env,
/* struct type either no name or a valid one */
if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -3371,7 +3349,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env,
/* struct member either no name or a valid one */
if (member->name_off &&
- !btf_name_valid_identifier(btf, member->name_off)) {
+ !btf_name_valid(env, member->name_off)) {
btf_verifier_log_member(env, t, member, "Invalid name");
return -EINVAL;
}
@@ -4162,6 +4140,8 @@ static int btf_parse_graph_root(const struct btf *btf,
* verify its type.
*/
for_each_member(i, t, member) {
+ if (bpf_must_abort())
+ return -EINTR;
if (strcmp(info->graph_root.node_name,
__btf_name_by_offset(btf, member->name_off)))
continue;
@@ -4640,7 +4620,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env,
/* enum type either no name or a valid one */
if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -4656,7 +4636,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env,
/* enum member must have a valid name */
if (!enums[i].name_off ||
- !btf_name_valid_identifier(btf, enums[i].name_off)) {
+ !btf_name_valid(env, enums[i].name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -4748,7 +4728,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
/* enum type either no name or a valid one */
if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -4764,7 +4744,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
/* enum member must have a valid name */
if (!enums[i].name_off ||
- !btf_name_valid_identifier(btf, enums[i].name_off)) {
+ !btf_name_valid(env, enums[i].name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -5054,7 +5034,7 @@ static s32 btf_locsec_check_meta(struct btf_verifier_env *env,
}
if (!t->name_off ||
- !btf_name_valid_section(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -5246,7 +5226,7 @@ static s32 btf_func_check_meta(struct btf_verifier_env *env,
u32 meta_left)
{
if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -5315,7 +5295,7 @@ static s32 btf_var_check_meta(struct btf_verifier_env *env,
}
if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -5381,7 +5361,7 @@ static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
}
if (!t->name_off ||
- !btf_name_valid_section(env->btf, t->name_off)) {
+ !btf_name_valid(env, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
@@ -5768,7 +5748,7 @@ static int btf_func_proto_check(struct btf_verifier_env *env,
if (args[i].name_off &&
(!btf_name_offset_valid(btf, args[i].name_off) ||
- !btf_name_valid_identifier(btf, args[i].name_off))) {
+ !btf_name_valid(env, args[i].name_off))) {
btf_verifier_log_type(env, t,
"Invalid arg#%u", i + 1);
return -EINVAL;
@@ -6067,6 +6047,7 @@ static int btf_parse_str_sec(struct btf_verifier_env *env)
const struct btf_header *hdr;
struct btf *btf = env->btf;
const char *start, *end;
+ u32 i, len = 0;
hdr = &btf->hdr;
start = btf->nohdr_data + hdr->str_off;
@@ -6091,6 +6072,21 @@ static int btf_parse_str_sec(struct btf_verifier_env *env)
return -EINVAL;
}
+ /* many names can share one long string: look at each byte once */
+ kvfree(env->names);
+ env->names = kvcalloc(BITS_TO_LONGS(hdr->str_len), sizeof(long),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!env->names)
+ return -ENOMEM;
+ for (i = hdr->str_len; i--;) {
+ if (!start[i])
+ len = 0;
+ else if (!isprint(start[i]))
+ len = BTF_MAX_NAME_LEN;
+ else if (len++ < BTF_MAX_NAME_LEN)
+ __set_bit(i, env->names);
+ }
+
return 0;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 19/21] bpf: Find dominators in O(n log n)
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (17 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 18/21] bpf: Raise the limit on the length of BTF names Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 20/21] selftests/bpf: Test programs above the old limits Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 21/21] selftests/bpf: Test limits of the load with raw programs Alexei Starovoitov
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The intersection walks up the dominator tree for each predecessor. Many
edges to one insn make it quadratic:
if ... goto out 30000 of them take 13 s, 100000 do not finish
if ... goto out
...
out:
Use the algorithm of Lengauer and Tarjan, with path compression.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/loops.c | 157 ++++++++++++++++++++++++++++-----------------
1 file changed, 98 insertions(+), 59 deletions(-)
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 434c5acfc2e1..110197685051 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -96,78 +96,117 @@ static int idoms_intersect(struct bpf_verifier_env *env, int a, int b)
return a;
}
-/* See "A Simple, Fast Dominance Algorithm" by Cooper et al. for details. */
-static int compute_subprog_idoms(struct bpf_verifier_env *env, struct bpf_iarray **preds,
- int subprog_idx)
-{
- struct bpf_subprog_info *subprog = &env->subprog_info[subprog_idx];
- int start = subprog->start;
- int po_first = subprog->postorder_start;
- int po_last = (subprog + 1)->postorder_start - 1;
- int *idoms = env->idoms;
- int po_num, pred;
- bool changed;
+/* An insn. The nodes are in the order of a depth-first walk, node 0 is none. */
+struct idom_node {
+ u32 insn;
+ u32 parent; /* in the walk */
+ u32 semi; /* semidominator */
+ u32 idom;
+ u32 up; /* a parent, or a parent of it, once the node is done */
+ u32 min; /* the node with the lowest 'semi' on the way to 'up' */
+ u32 bucket; /* list of the nodes whose 'semi' this node is */
+ u32 next; /* in that list. Before that the next successor for the walk */
+};
- idoms[start] = start;
- changed = true;
- do {
- changed = false;
- /*
- * Iterate in reverse postorder, skipping the subprog entry:
- * insn_postorder[po_last] is the subprogram entry.
- */
- for (po_num = po_last - 1; po_num >= po_first; po_num--) {
- int idx = env->cfg.insn_postorder[po_num];
- int new_idom = -1;
-
- iarray_for_each(pred, preds[idx]) {
- if (bpf_must_abort())
- return -EINTR;
-
- if (idoms[pred] == -1)
- continue;
- if (new_idom == -1)
- new_idom = pred;
- else
- new_idom = idoms_intersect(env, pred, new_idom);
- }
- if (new_idom != -1 && idoms[idx] != new_idom) {
- idoms[idx] = new_idom;
- changed = true;
- }
- }
- } while (changed);
- idoms[start] = -1;
- return 0;
+/* The node with the lowest 'semi' on the way up from node 'id'. Makes the way short. */
+static u32 idom_eval(struct idom_node *nodes, u32 *path, u32 id)
+{
+ struct idom_node *node, *up;
+ u32 cnt = 0, cur;
+
+ for (cur = id; nodes[nodes[cur].up].up; cur = nodes[cur].up)
+ path[cnt++] = cur;
+ while (cnt--) {
+ node = &nodes[path[cnt]];
+ up = &nodes[node->up];
+ if (nodes[up->min].semi < nodes[node->min].semi)
+ node->min = up->min;
+ node->up = up->up;
+ }
+ return nodes[id].min;
}
+/*
+ * See "A Fast Algorithm for Finding Dominators in a Flowgraph" by Lengauer and Tarjan.
+ *
+ * 1 -> 2 -> 3 -> 4 walk: parent[4] = 3
+ * | ^ from the end: semi[4] = 1
+ * +-> 5 ---------+ from the top: idom[4] = 1
+ */
int bpf_compute_idoms(struct bpf_verifier_env *env)
{
- struct bpf_iarray **preds;
- u32 len = env->prog->len;
- int *idoms, i, err = 0;
+ u32 len = env->prog->len, cnt = 0, i, insn, id, cur, best, *num, *path;
+ struct bpf_iarray **preds, *succ;
+ struct idom_node *nodes, *node;
+ int err = -ENOMEM;
preds = compute_predecessors(env);
- if (!preds)
- return -ENOMEM;
+ env->idoms = kvmalloc_objs(*env->idoms, len, BPF_VERIFIER_GFP);
+ nodes = kvzalloc_objs(*nodes, len + 1, BPF_VERIFIER_GFP);
+ /* insn -> node */
+ num = kvzalloc_objs(*num, 2 * (size_t)len, BPF_VERIFIER_GFP);
+ path = num + len;
+ if (!preds || !env->idoms || !nodes || !num)
+ goto out;
+ memset(env->idoms, 0xff, len * sizeof(*env->idoms));
- idoms = kvzalloc_objs(*idoms, len, BPF_VERIFIER_GFP);
- if (!idoms) {
- kvfree(preds);
- return -ENOMEM;
+ err = -EINTR;
+ for (i = 0; i < env->subprog_cnt; i++) {
+ insn = env->subprog_info[i].start;
+ for (cur = 0;; insn = succ->items[nodes[cur].next++]) {
+ if (!num[insn]) {
+ num[insn] = ++cnt;
+ nodes[cnt] = (struct idom_node){ insn, cur, cnt, .min = cnt };
+ cur = cnt;
+ }
+ if (bpf_must_abort())
+ goto out;
+ /* go back to a node that has a successor left */
+ for (; cur; cur = nodes[cur].parent) {
+ succ = bpf_insn_successors(env, nodes[cur].insn);
+ if (nodes[cur].next < succ->cnt)
+ break;
+ }
+ if (!cur)
+ break;
+ }
}
- env->idoms = idoms;
- for (i = 0; i < len; i++)
- idoms[i] = -1;
-
- for (i = 0; i < env->subprog_cnt; i++) {
- err = compute_subprog_idoms(env, preds, i);
- if (err)
- break;
+ for (id = cnt; id; id--) {
+ node = &nodes[id];
+ if (!node->parent)
+ continue;
+ iarray_for_each(insn, preds[node->insn]) {
+ if (bpf_must_abort())
+ goto out;
+ if (!num[insn])
+ continue;
+ best = idom_eval(nodes, path, num[insn]);
+ node->semi = min(node->semi, nodes[best].semi);
+ }
+ node->next = nodes[node->semi].bucket;
+ nodes[node->semi].bucket = id;
+ node->up = node->parent;
+ for (cur = nodes[node->parent].bucket; cur; cur = nodes[cur].next) {
+ best = idom_eval(nodes, path, cur);
+ nodes[cur].idom = nodes[best].semi < nodes[cur].semi ? best : node->parent;
+ }
+ nodes[node->parent].bucket = 0;
}
+ for (id = 1; id <= cnt; id++) {
+ node = &nodes[id];
+ if (!node->parent)
+ continue;
+ if (node->idom != node->semi)
+ node->idom = nodes[node->idom].idom;
+ env->idoms[node->insn] = nodes[node->idom].insn;
+ }
+ err = 0;
+out:
kvfree(preds);
+ kvfree(nodes);
+ kvfree(num);
return err;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 20/21] selftests/bpf: Test programs above the old limits
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (18 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 19/21] bpf: Find dominators in O(n log n) Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 21/21] selftests/bpf: Test limits of the load with raw programs Alexei Starovoitov
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
100 maps, 1100 subprogs, calls in dead code, 1000 kptrs, far cleanup
pads, bpf_loop() with a dead callback. Unsafe twins must fail.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 4 +
.../bpf/progs/verifier_callx_dead_cb.c | 60 +++++
.../selftests/bpf/progs/verifier_no_limits.c | 224 ++++++++++++++++++
.../selftests/bpf/progs/verifier_unwind.c | 32 +++
4 files changed, 320 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_dead_cb.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_no_limits.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 4d77b8fd4eab..57ff2578a991 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -39,6 +39,7 @@
#include "verifier_call_large_imm.skel.h"
#include "verifier_calls_in_a_row.skel.h"
#include "verifier_callx.skel.h"
+#include "verifier_callx_dead_cb.skel.h"
#include "verifier_callx_many.skel.h"
#include "verifier_callx_rodata.skel.h"
#include "verifier_cfg.skel.h"
@@ -110,6 +111,7 @@
#include "verifier_netfilter_retcode.skel.h"
#include "verifier_bpf_fastcall.skel.h"
#include "verifier_or_jmp32_k.skel.h"
+#include "verifier_no_limits.skel.h"
#include "verifier_patch_list.skel.h"
#include "verifier_percpu_addr.skel.h"
#include "verifier_precision.skel.h"
@@ -248,6 +250,7 @@ void test_verifier_btf_unreliable_prog(void) { RUN(verifier_btf_unreliable_prog
void test_verifier_call_large_imm(void) { RUN(verifier_call_large_imm); }
void test_verifier_calls_in_a_row(void) { RUN(verifier_calls_in_a_row); }
void test_verifier_callx(void) { RUN_JITED(verifier_callx); }
+void test_verifier_callx_dead_cb(void) { RUN_JITED(verifier_callx_dead_cb); }
void test_verifier_callx_many(void) { RUN_JITED(verifier_callx_many); }
void test_verifier_callx_rodata(void) { RUN_JITED(verifier_callx_rodata); }
void test_verifier_cfg(void) { RUN(verifier_cfg); }
@@ -317,6 +320,7 @@ void test_verifier_mul(void) { RUN(verifier_mul); }
void test_verifier_netfilter_ctx(void) { RUN(verifier_netfilter_ctx); }
void test_verifier_netfilter_retcode(void) { RUN(verifier_netfilter_retcode); }
void test_verifier_bpf_fastcall(void) { RUN(verifier_bpf_fastcall); }
+void test_verifier_no_limits(void) { RUN_JITED(verifier_no_limits); }
void test_verifier_or_jmp32_k(void) { RUN(verifier_or_jmp32_k); }
/*
* Run serially with bpf_jit_harden at 0: blinding changes the xlated insns that
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_dead_cb.c b/tools/testing/selftests/bpf/progs/verifier_callx_dead_cb.c
new file mode 100644
index 000000000000..a0873095566b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_dead_cb.c
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-2.0
+/* libbpf appends all functions of the table to the prog, hence a file of its own. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#ifdef CAN_USE_CALLX
+
+__naked __noinline __used static int live_func(void)
+{
+ asm volatile ("r0 = 3; exit;");
+}
+
+__naked __noinline __used static int dead_cb(void)
+{
+ asm volatile ("r0 = 0; exit;");
+}
+
+/* The loop has no iterations. The callback is the last function and it is removed. */
+SEC("socket")
+__success __retval(3)
+__naked void loop_with_dead_callback(void)
+{
+ asm volatile (
+ ".pushsection .rodata,\"a\",@progbits;"
+ ".balign 8;"
+"funcs_%=:"
+ ".quad live_func;"
+ ".quad dead_cb;"
+ ".type funcs_%=, @object;"
+ ".size funcs_%=, .-funcs_%=;"
+ ".popsection;"
+ "r6 = funcs_%= ll;"
+ "r1 = 0;"
+ "r2 = *(u64 *)(r6 + 8);"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r2 = *(u64 *)(r6 + 0);"
+ "callx r2;"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all);
+}
+
+#else
+
+SEC("socket")
+__skip("callx needs arch JIT and compiler support")
+__success
+int dummy(void *ctx)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_no_limits.c b/tools/testing/selftests/bpf/progs/verifier_no_limits.c
new file mode 100644
index 000000000000..a38d75293238
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_no_limits.c
@@ -0,0 +1,224 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Programs above the limits that the verifier used to have, and unsafe twins of them */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+#define REP10(M, p) M(p##0) M(p##1) M(p##2) M(p##3) M(p##4) M(p##5) M(p##6) M(p##7) M(p##8) M(p##9)
+#define REP100(M, p) REP10(M, p##0) REP10(M, p##1) REP10(M, p##2) REP10(M, p##3) REP10(M, p##4) \
+ REP10(M, p##5) REP10(M, p##6) REP10(M, p##7) REP10(M, p##8) REP10(M, p##9)
+
+/* 100 maps */
+
+#define MAP(n) char map##n[8] SEC(".data.map" #n) = { 1 };
+#define MAP_READ(n) "r1 = map" #n " ll; r1 = *(u8 *)(r1 + 0); r0 += r1;"
+REP100(MAP, 1)
+
+SEC("?tc")
+__success __retval(100)
+__naked void maps_100(void)
+{
+ asm volatile ("r0 = 0;" REP100(MAP_READ, 1) "exit;" ::: __clobber_all);
+}
+
+SEC("?tc")
+__failure __msg("invalid access to map value, value_size=8 off=8 size=1")
+__naked void maps_100_bad(void)
+{
+ asm volatile ("r0 = 0;" REP100(MAP_READ, 1) "r1 = map199 ll; r0 = *(u8 *)(r1 + 8); exit;"
+ ::: __clobber_all);
+}
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 2);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+} jmp_table SEC(".maps");
+
+/* 1100 subprogs */
+
+#define REP1100(M) REP100(M, 10) REP100(M, 11) REP100(M, 12) REP100(M, 13) REP100(M, 14) \
+ REP100(M, 15) REP100(M, 16) REP100(M, 17) REP100(M, 18) REP100(M, 19) \
+ REP100(M, 20)
+#define FUNC(n) __naked __noinline __used static void func##n(void) \
+ { asm volatile ("r0 = r1; r0 += 1; r2 = 0; exit;"); }
+#define FUNC_CALL(n) "r1 = r0; call func" #n ";"
+REP1100(FUNC)
+
+SEC("?tc")
+__success __retval(1100)
+__naked void subprogs_1100(void)
+{
+ asm volatile ("r0 = 0;" REP1100(FUNC_CALL) "exit;" ::: __clobber_all);
+}
+
+SEC("?tc")
+__failure __msg("invalid mem access 'scalar'")
+__naked void subprogs_1100_bad(void)
+{
+ asm volatile ("r0 = 0;" REP1100(FUNC_CALL) "r0 = *(u8 *)(r0 + 0); exit;" ::: __clobber_all);
+}
+
+/* Calls in dead code do not count. Only the main pass knows that (x & 7) > 7 is false. */
+
+#define DEAD_IF(limit) \
+ "call %[bpf_get_prandom_u32];" \
+ "r0 &= 7;" \
+ "if r0 <= " #limit " goto 1f;"
+
+#define RECURSION(name, limit) \
+__naked __noinline __used static void name##_func(void) \
+{ \
+ asm volatile ( \
+ DEAD_IF(limit) \
+ "call " #name "_func;" \
+"1: r0 = 0;" \
+ "exit;" \
+ :: __imm(bpf_get_prandom_u32) : __clobber_all); \
+} \
+__naked void name(void) \
+{ \
+ asm volatile ("call " #name "_func; exit;" ::: __clobber_all); \
+}
+
+void dead_recursion(void) SEC("?raw_tp") __success;
+RECURSION(dead_recursion, 7)
+
+void live_recursion(void) SEC("?raw_tp") __arch_x86_64 __arch_arm64
+__failure __msg("in a tracing program");
+RECURSION(live_recursion, 6)
+
+#define TAIL_CALL_IN_CB(name, limit) \
+__naked __noinline __used static void name##_cb(void) \
+{ \
+ asm volatile ( \
+ "r6 = *(u64 *)(r2 + 0);" \
+ DEAD_IF(limit) \
+ "r1 = r6;" \
+ "r2 = %[jmp_table] ll;" \
+ "r3 = 0;" \
+ "call %[bpf_tail_call];" \
+"1: r0 = 1;" \
+ "exit;" \
+ :: __imm(bpf_get_prandom_u32), __imm(bpf_tail_call), \
+ __imm_addr(jmp_table) : __clobber_all); \
+} \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "*(u64 *)(r10 - 8) = r1;" \
+ "r1 = 1;" \
+ "r2 = " #name "_cb ll;" \
+ "r3 = r10;" \
+ "r3 += -8;" \
+ "r4 = 0;" \
+ "call %[bpf_loop];" \
+ "exit;" \
+ :: __imm(bpf_loop) : __clobber_all); \
+}
+
+void dead_tail_call_in_cb(void) SEC("?tc") __success __retval(1);
+TAIL_CALL_IN_CB(dead_tail_call_in_cb, 7)
+
+void live_tail_call_in_cb(void) SEC("?tc") __failure __msg("tail call");
+TAIL_CALL_IN_CB(live_tail_call_in_cb, 6)
+
+/* Many edges to one insn: the dominators take linear time */
+SEC("?tc")
+__success __retval(0)
+__naked void goto_out_30000(void)
+{
+ asm volatile (
+ "r6 = *(u32 *)(r1 + 0);"
+ ".rept 30000;"
+ "if r6 == 1 goto 1f;"
+ ".endr;"
+"1: r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* GCC loses the tag of 'struct task_struct __kptr *task[1000]' */
+struct one_kptr {
+ struct task_struct __kptr *p;
+};
+
+/* 1000 kptrs in a map value. The struct is too big for an allocated object. */
+
+struct many_kptrs {
+ long x;
+ struct task_struct __kptr *first;
+ struct one_kptr task[1000];
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct many_kptrs);
+} kptr_map SEC(".maps");
+
+SEC("?tc")
+__success __retval(0)
+int kptrs_1000(void *ctx)
+{
+ struct many_kptrs *v = bpf_map_lookup_elem(&kptr_map, &(int){ 0 });
+ struct task_struct *t;
+
+ if (!v)
+ return 1;
+ t = bpf_kptr_xchg(&v->task[999].p, NULL);
+ if (t)
+ bpf_task_release(t);
+ return v->x;
+}
+
+SEC("?tc")
+__failure __msg("kptr access misaligned")
+int kptrs_1000_bad(void *ctx)
+{
+ struct many_kptrs *v = bpf_map_lookup_elem(&kptr_map, &(int){ 0 });
+
+ if (!v)
+ return 1;
+ *(volatile int *)((void *)&v->task[999].p + 4) = 1;
+ return 0;
+}
+
+SEC("?tc")
+__success __retval(1)
+int kptrs_1000_obj_new(void *ctx)
+{
+ struct many_kptrs *p = bpf_obj_new(typeof(*p));
+
+ if (!p)
+ return 1;
+ bpf_obj_drop(p);
+ return 0;
+}
+
+/* 500 kptrs in an allocated object */
+
+struct obj_kptrs {
+ struct task_struct __kptr *first;
+ struct one_kptr task[499];
+};
+
+SEC("?tc")
+__failure __msg("direct access to kptr is disallowed")
+int kptrs_500_obj_bad(void *ctx)
+{
+ struct obj_kptrs *p = bpf_obj_new(typeof(*p));
+
+ if (!p)
+ return 1;
+ *(volatile long *)&p->task[498].p = 1;
+ bpf_obj_drop(p);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_unwind.c b/tools/testing/selftests/bpf/progs/verifier_unwind.c
index 5a21345c34d4..596e5e0301e5 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unwind.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unwind.c
@@ -69,6 +69,38 @@ static unsigned long may_unwind(void)
);
}
+/* The patches of the divisions push the pad out of the 16-bit range */
+#define FAR_PAD(name, sub) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r6 = *(u32 *)(r1 + 0);" \
+ "r1 = r6;" \
+ "r1 -= " #sub ";" \
+ "call may_unwind;" \
+ unwind_goto(8202) \
+ "r7 = 1;" \
+ ".rept 8200; r7 /= r6; .endr;" \
+ "exit;" \
+ "r0 += 100;" \
+ "exit;" \
+ : \
+ : __imm_unwind_goto(8202) \
+ : __clobber_all); \
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__success __retval(7)
+__load_if_JITed()
+FAR_PAD(far_pad_returns, 64)
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__success __retval(105)
+__load_if_JITed()
+FAR_PAD(far_pad_unwinds, 0)
+
/* owns nothing: its pad only carries on */
__naked __noinline __used
static unsigned long passes_on(void)
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH bpf-next v2 21/21] selftests/bpf: Test limits of the load with raw programs
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
` (19 preceding siblings ...)
2026-11-16 15:26 ` [PATCH bpf-next v2 20/21] selftests/bpf: Test programs above the old limits Alexei Starovoitov
@ 2026-11-16 15:26 ` Alexei Starovoitov
20 siblings, 0 replies; 26+ messages in thread
From: Alexei Starovoitov @ 2026-11-16 15:26 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A load under a timer signal, under the freezer and in a small memory
cgroup, the context of a syscall program, 33000 subprogs, a dump of more
than 4 MB, 300 kfuncs, loops of BTF types, long BTF names, a light
skeleton with 600 maps and 601 programs.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile.skel | 2 +-
.../selftests/bpf/prog_tests/no_limits.c | 465 ++++++++++++++++++
.../testing/selftests/bpf/progs/many_progs.c | 36 ++
3 files changed, 502 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/no_limits.c
create mode 100644 tools/testing/selftests/bpf/progs/many_progs.c
diff --git a/tools/testing/selftests/bpf/Makefile.skel b/tools/testing/selftests/bpf/Makefile.skel
index b8eaecc7f429..45e4ce5dca4d 100644
--- a/tools/testing/selftests/bpf/Makefile.skel
+++ b/tools/testing/selftests/bpf/Makefile.skel
@@ -34,7 +34,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \
core_kern.c core_kern_overflow.c test_ringbuf.c \
test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \
- test_ringbuf_overwrite.c callx_rodata.c
+ test_ringbuf_overwrite.c callx_rodata.c many_progs.c
LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
diff --git a/tools/testing/selftests/bpf/prog_tests/no_limits.c b/tools/testing/selftests/bpf/prog_tests/no_limits.c
new file mode 100644
index 000000000000..b949d1db23e6
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/no_limits.c
@@ -0,0 +1,465 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Limits of the load that an ELF file cannot show, or shows at a high price */
+
+#include <test_progs.h>
+#include <bpf/btf.h>
+#include <sys/param.h>
+#include <sys/resource.h>
+#include <sys/time.h>
+#include "cgroup_helpers.h"
+#include "many_progs.lskel.h"
+
+static int load(struct bpf_insn *insns, int cnt)
+{
+ return bpf_prog_load(BPF_PROG_TYPE_SCHED_CLS, NULL, "GPL", insns, cnt, NULL);
+}
+
+static struct bpf_insn *straight_code(int cnt)
+{
+ struct bpf_insn *insns = calloc(cnt, sizeof(*insns));
+ int i;
+
+ if (!ASSERT_OK_PTR(insns, "insns"))
+ return NULL;
+ for (i = 0; i < cnt - 1; i++)
+ insns[i] = BPF_MOV64_IMM(BPF_REG_0, i);
+ insns[i] = BPF_EXIT_INSN();
+ return insns;
+}
+
+static void on_alarm(int sig)
+{
+}
+
+/* A signal that does not kill does not stop the load */
+static void test_signals(void)
+{
+ struct itimerval on = { { 0, 200 }, { 0, 200 } }, off = {};
+ struct sigaction sa = { .sa_handler = on_alarm }, old;
+ struct bpf_insn *insns = straight_code(20000);
+ int fd;
+
+ if (!insns)
+ return;
+ sigaction(SIGALRM, &sa, &old);
+ setitimer(ITIMER_REAL, &on, NULL);
+ fd = load(insns, 20000);
+ setitimer(ITIMER_REAL, &off, NULL);
+ sigaction(SIGALRM, &old, NULL);
+ if (ASSERT_OK_FD(fd, "load"))
+ close(fd);
+ free(insns);
+}
+
+static int exit_code(pid_t pid)
+{
+ int status;
+
+ if (!ASSERT_EQ(waitpid(pid, &status, 0), pid, "wait") ||
+ !ASSERT_TRUE(WIFEXITED(status), "exited"))
+ return -1;
+ return WEXITSTATUS(status);
+}
+
+/* The freezer does not wait for the end of the load. The load starts again after it. */
+static void test_freezer(int flag)
+{
+ struct bpf_insn *insns = straight_code(200000);
+ struct bpf_common_attr common = {};
+ union bpf_attr attr = {
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .insns = ptr_to_u64(insns),
+ .insn_cnt = 200000,
+ .license = ptr_to_u64("GPL"),
+ };
+ int cg = -1, pipes[2] = { -1, -1 }, fd;
+ char buf[64] = "";
+ pid_t pid;
+
+ if (!insns || !ASSERT_OK(setup_cgroup_environment(), "setup") ||
+ !ASSERT_OK(pipe(pipes), "pipe"))
+ goto out;
+ cg = create_and_get_cgroup("/no_limits");
+ if (!ASSERT_OK_FD(cg, "cgroup"))
+ goto out;
+ pid = fork();
+ if (!pid) {
+ /* the fd that the prog gets */
+ if (join_parent_cgroup("/no_limits"))
+ _exit(1);
+ fd = dup(pipes[0]);
+ close(fd);
+ if (write(pipes[1], &fd, sizeof(fd)) != sizeof(fd))
+ _exit(2);
+ _exit(syscall(__NR_bpf, BPF_PROG_LOAD | flag, &attr, sizeof(attr), &common,
+ sizeof(common)) != fd ? 3 : 0);
+ }
+ close(pipes[1]);
+ if (ASSERT_EQ(read(pipes[0], &fd, sizeof(fd)), sizeof(fd), "read")) {
+ usleep(10000);
+ ASSERT_OK(write_cgroup_file("/no_limits", "cgroup.freeze", "1"), "freeze");
+ while (!strstr(buf, "frozen 1") &&
+ !read_cgroup_file("/no_limits", "cgroup.events", buf, sizeof(buf)))
+ usleep(1000);
+ snprintf(buf, sizeof(buf), "/proc/%d/fd/%d", pid, fd);
+ ASSERT_ERR(faccessat(AT_FDCWD, buf, F_OK, AT_SYMLINK_NOFOLLOW), "no prog yet");
+ ASSERT_OK(write_cgroup_file("/no_limits", "cgroup.freeze", "0"), "thaw");
+ }
+ ASSERT_OK(exit_code(pid), "load");
+out:
+ close(pipes[0]);
+ if (cg >= 0)
+ close(cg);
+ cleanup_cgroup_environment();
+ free(insns);
+}
+
+/* 4 * funcs + 1 insns */
+static struct bpf_insn *many_funcs(int funcs)
+{
+ struct bpf_insn *insns = calloc(4 * funcs + 1, sizeof(*insns)), *f;
+ int i;
+
+ if (!ASSERT_OK_PTR(insns, "insns"))
+ return NULL;
+ for (i = 0, f = insns + funcs + 1; i < funcs; i++) {
+ insns[i] = BPF_CALL_REL(f - &insns[i] - 1);
+ *f++ = BPF_MOV64_IMM(BPF_REG_0, i);
+ *f++ = BPF_MOV64_IMM(BPF_REG_2, 0);
+ *f++ = BPF_EXIT_INSN();
+ }
+ insns[funcs] = BPF_EXIT_INSN();
+ return insns;
+}
+
+static int load_in_memcg(struct bpf_insn *insns, int cnt)
+{
+ pid_t pid = fork();
+ int fd;
+
+ if (pid)
+ return exit_code(pid);
+ fd = join_parent_cgroup("/no_limits") ? -EINVAL : load(insns, cnt);
+ _exit(fd < 0 ? -fd : 0);
+}
+
+/* The load fails. The OOM killer leaves the loader alone. */
+static void test_memcg(void)
+{
+ struct bpf_insn *insns = straight_code(200000), *funcs = many_funcs(2000);
+ int cg = -1, mb, err;
+ char max[16];
+
+ if (!insns || !funcs || !ASSERT_OK(setup_cgroup_environment(), "setup"))
+ goto out;
+ cg = create_and_get_cgroup("/no_limits");
+ if (!ASSERT_OK_FD(cg, "cgroup"))
+ goto out;
+ if (write_cgroup_file("/no_limits", "memory.max", "16M")) {
+ test__skip();
+ goto out;
+ }
+ ASSERT_EQ(load_in_memcg(insns, 200000), ENOMEM, "errno");
+
+ /* some of these end in the JIT of the subprogs */
+ for (mb = 4; mb <= 24; mb += 2) {
+ snprintf(max, sizeof(max), "%dM", mb);
+ ASSERT_OK(write_cgroup_file("/no_limits", "memory.max", max), "max");
+ err = load_in_memcg(funcs, 4 * 2000 + 1);
+ ASSERT_TRUE(!err || err == ENOMEM, "errno");
+ }
+out:
+ if (cg >= 0)
+ close(cg);
+ cleanup_cgroup_environment();
+ free(insns);
+ free(funcs);
+}
+
+static int run(int fd)
+{
+ char pkt[64] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = pkt, .data_size_in = sizeof(pkt));
+
+ return bpf_prog_test_run_opts(fd, &opts) ?: opts.retval;
+}
+
+/* The target of a tail call reads the context at 'off'. The run passes 8 bytes. */
+static void check_syscall_target(int off, int expected)
+{
+ LIBBPF_OPTS(bpf_prog_load_opts, opts, .prog_flags = BPF_F_SLEEPABLE);
+ int map = bpf_map_create(BPF_MAP_TYPE_PROG_ARRAY, NULL, 4, 4, 1, NULL);
+ struct bpf_insn first[] = {
+ BPF_LD_MAP_FD(BPF_REG_2, map),
+ BPF_MOV64_IMM(BPF_REG_3, 0),
+ BPF_EMIT_CALL(BPF_FUNC_tail_call),
+ BPF_MOV64_IMM(BPF_REG_0, 3),
+ BPF_EXIT_INSN(),
+ }, target[] = {
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, off),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0),
+ BPF_EXIT_INSN(),
+ };
+ int ctx[2] = { 5, 7 }, key = 0, fd, target_fd;
+ LIBBPF_OPTS(bpf_test_run_opts, run_opts, .ctx_in = ctx, .ctx_size_in = sizeof(ctx));
+
+ fd = bpf_prog_load(BPF_PROG_TYPE_SYSCALL, NULL, "GPL", first, ARRAY_SIZE(first), &opts);
+ target_fd = bpf_prog_load(BPF_PROG_TYPE_SYSCALL, NULL, "GPL", target, ARRAY_SIZE(target),
+ &opts);
+ if (ASSERT_OK_FD(fd, "load") && ASSERT_OK_FD(target_fd, "target"))
+ ASSERT_EQ(bpf_map_update_elem(map, &key, &target_fd, 0) ?:
+ bpf_prog_test_run_opts(fd, &run_opts) ?: (int)run_opts.retval,
+ expected, "result");
+ close(fd);
+ close(target_fd);
+ close(map);
+}
+
+static void test_syscall_ctx(void)
+{
+ int off;
+
+ check_syscall_target(4, 7);
+ for (off = 8; off < 65532; off += 4096)
+ check_syscall_target(off, 0);
+}
+
+/* The number of a subprog does not fit into s16 */
+static void test_subprogs(void)
+{
+ const int funcs = 33000, cnt = 4 * funcs + 1;
+ struct bpf_insn *insns = many_funcs(funcs);
+ struct bpf_prog_info info = {};
+ __u32 len = sizeof(info);
+ int fd;
+
+ if (!insns)
+ return;
+ fd = load(insns, cnt);
+ if (!ASSERT_OK_FD(fd, "load"))
+ goto out;
+ ASSERT_EQ(run(fd), funcs - 1, "retval");
+ info.xlated_prog_len = cnt * sizeof(*insns);
+ info.xlated_prog_insns = ptr_to_u64(insns);
+ if (ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &len), "dump"))
+ ASSERT_EQ(insns[funcs - 1].imm, funcs, "subprog");
+ close(fd);
+out:
+ free(insns);
+}
+
+/* A division takes 4 insns. The dump has more than 4 MB. */
+static void test_dump(void)
+{
+ const int cnt = 140000;
+ struct bpf_insn *insns = straight_code(4 * cnt);
+ struct bpf_prog_info info = {};
+ __u32 len = sizeof(info);
+ int fd, i;
+
+ if (!insns)
+ return;
+ insns[0] = BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32);
+ insns[1] = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
+ for (i = 2; i < cnt - 1; i++)
+ insns[i] = BPF_ALU64_REG(BPF_DIV, BPF_REG_0, BPF_REG_1);
+ insns[i] = BPF_EXIT_INSN();
+
+ fd = load(insns, cnt);
+ if (ASSERT_OK_FD(fd, "load")) {
+ info.xlated_prog_len = 4 * cnt * sizeof(*insns);
+ info.xlated_prog_insns = ptr_to_u64(insns);
+ if (ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &len), "dump"))
+ ASSERT_GT(info.xlated_prog_len, 4 << 20, "len");
+ close(fd);
+ }
+ free(insns);
+}
+
+/* Calls of all kfuncs that the verifier can describe, in dead code */
+static void test_kfuncs(void)
+{
+ struct btf *btf = btf__load_vmlinux_btf();
+ struct bpf_insn insns[1024], one[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0),
+ BPF_EXIT_INSN(),
+ };
+ const struct btf_type *t;
+ int id, fd, cnt = 2;
+
+ if (!ASSERT_OK_PTR(btf, "btf"))
+ return;
+ for (id = 1; id < btf__type_cnt(btf) && cnt < 302; id++) {
+ t = btf__type_by_id(btf, id);
+ if (!btf_is_func(t) || strncmp(btf__name_by_offset(btf, t->name_off), "bpf_", 4))
+ continue;
+ one[2].imm = id;
+ fd = load(one, ARRAY_SIZE(one));
+ if (fd < 0)
+ continue;
+ close(fd);
+ insns[cnt++] = one[2];
+ }
+ btf__free(btf);
+ if (cnt < 302) {
+ test__skip();
+ return;
+ }
+ insns[0] = one[0];
+ insns[1] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, cnt - 2);
+ insns[cnt++] = one[3];
+ fd = load(insns, cnt);
+ if (ASSERT_OK_FD(fd, "load"))
+ close(fd);
+}
+
+/* Type 1 is int, type 'value' is the value of a map */
+static void check_btf(struct btf *btf, int value, int expected)
+{
+ LIBBPF_OPTS(bpf_map_create_opts, opts, .btf_key_type_id = 1, .btf_value_type_id = value);
+ int fd;
+
+ if (ASSERT_EQ(btf__load_into_kernel(btf), expected, "load") && !expected) {
+ opts.btf_fd = btf__fd(btf);
+ fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, NULL, 4, 4, 1, &opts);
+ if (ASSERT_OK_FD(fd, "map"))
+ close(fd);
+ }
+ btf__free(btf);
+}
+
+static struct btf *new_btf(void)
+{
+ struct btf *btf = btf__new_empty();
+
+ btf__add_int(btf, "int", 4, BTF_INT_SIGNED);
+ return btf;
+}
+
+enum chain { STRUCTS, ARRAYS, TYPEDEFS, DAG };
+
+/* Type 2 holds type 3, ... The last one holds type 'last'. */
+static struct btf *btf_chain(enum chain kind, int cnt, int last)
+{
+ struct btf *btf = new_btf();
+ int i, next;
+
+ for (i = 2; i < cnt + 2; i++) {
+ next = i == cnt + 1 ? last : i + 1;
+ if (kind == ARRAYS) {
+ btf__add_array(btf, 1, next, 1);
+ } else if (kind == TYPEDEFS) {
+ btf__add_typedef(btf, "t", next);
+ } else {
+ btf__add_struct(btf, "s", 4);
+ btf__add_field(btf, "a", next, 0, 0);
+ if (kind != DAG)
+ continue;
+ btf__add_field(btf, "b", next, 0, 0);
+ btf__add_field(btf, "c", next, 0, 0);
+ }
+ }
+ return btf;
+}
+
+static void test_btf(void)
+{
+ static char name[4098];
+ struct btf *btf;
+ enum chain kind;
+ int i;
+
+ for (kind = STRUCTS; kind <= DAG; kind++) {
+ check_btf(btf_chain(kind, 30, 1), 2, 0);
+ check_btf(btf_chain(kind, 30, 2), 2, -EEXIST);
+ }
+
+ /* 1M kptrs in a struct that nobody can allocate */
+ btf = new_btf();
+ btf__add_struct(btf, "task_struct", 4);
+ btf__add_field(btf, "a", 1, 0, 0);
+ btf__add_type_tag(btf, "kptr", 2);
+ btf__add_ptr(btf, 3);
+ btf__add_array(btf, 1, 4, 1000000);
+ for (i = 0; i < 100; i++) {
+ btf__add_struct(btf, "s", 8000008);
+ btf__add_field(btf, "p", 4, 0, 0);
+ btf__add_field(btf, "a", 5, 64, 0);
+ }
+ check_btf(btf, 1, 0);
+
+ /* ptr -> typedef -> struct -> array -> typedef */
+ btf = new_btf();
+ btf__add_ptr(btf, 3);
+ btf__add_typedef(btf, "t", 4);
+ btf__add_struct(btf, "s", 4);
+ btf__add_field(btf, "a", 5, 0, 0);
+ btf__add_array(btf, 1, 3, 1);
+ check_btf(btf, 4, -EEXIST);
+
+ /* ptr -> typedef -> array -> typedef */
+ btf = new_btf();
+ btf__add_ptr(btf, 3);
+ btf__add_typedef(btf, "t", 4);
+ btf__add_array(btf, 1, 3, 1);
+ check_btf(btf, 4, -EEXIST);
+
+ memset(name, 'x', 4097);
+ btf = new_btf();
+ btf__add_typedef(btf, name, 1);
+ check_btf(btf, 2, -EINVAL);
+
+ name[4096] = 0;
+ btf = new_btf();
+ btf__add_typedef(btf, name, 1);
+ check_btf(btf, 2, 0);
+
+ name[4000] = 7;
+ btf = new_btf();
+ btf__add_typedef(btf, name, 1);
+ check_btf(btf, 2, -EINVAL);
+}
+
+/* 600 maps and 601 programs */
+static void test_lskel(void)
+{
+ struct rlimit old, new = { 4096, 4096 };
+ struct many_progs_lskel *skel;
+
+ getrlimit(RLIMIT_NOFILE, &old);
+ new.rlim_max = MAX(new.rlim_max, old.rlim_max);
+ setrlimit(RLIMIT_NOFILE, &new);
+ skel = many_progs_lskel__open_and_load();
+ setrlimit(RLIMIT_NOFILE, &old);
+ if (!ASSERT_OK_PTR(skel, "load"))
+ return;
+ ASSERT_EQ(run(skel->progs.prog699.prog_fd), 699, "retval");
+ ASSERT_EQ(run(skel->progs.last.prog_fd), 2, "weak ksym");
+ many_progs_lskel__destroy(skel);
+}
+
+void serial_test_no_limits(void)
+{
+ if (test__start_subtest("signals"))
+ test_signals();
+ if (test__start_subtest("freezer"))
+ test_freezer(0);
+ if (test__start_subtest("freezer_common_attrs"))
+ test_freezer(BPF_COMMON_ATTRS);
+ if (test__start_subtest("memcg"))
+ test_memcg();
+ if (test__start_subtest("syscall_ctx"))
+ test_syscall_ctx();
+ if (test__start_subtest("subprogs"))
+ test_subprogs();
+ if (test__start_subtest("dump"))
+ test_dump();
+ if (test__start_subtest("kfuncs"))
+ test_kfuncs();
+ if (test__start_subtest("btf"))
+ test_btf();
+ if (test__start_subtest("lskel"))
+ test_lskel();
+}
diff --git a/tools/testing/selftests/bpf/progs/many_progs.c b/tools/testing/selftests/bpf/progs/many_progs.c
new file mode 100644
index 000000000000..060f7b265258
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/many_progs.c
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+#define REP10(M, p) M(p##0) M(p##1) M(p##2) M(p##3) M(p##4) M(p##5) M(p##6) M(p##7) M(p##8) M(p##9)
+#define REP100(M, p) REP10(M, p##0) REP10(M, p##1) REP10(M, p##2) REP10(M, p##3) REP10(M, p##4) \
+ REP10(M, p##5) REP10(M, p##6) REP10(M, p##7) REP10(M, p##8) REP10(M, p##9)
+
+#define PROG(n) \
+struct { \
+ __uint(type, BPF_MAP_TYPE_ARRAY); \
+ __uint(max_entries, 1); \
+ __type(key, int); \
+ __type(value, int); \
+} map##n SEC(".maps"); \
+ \
+SEC("tc") \
+int prog##n(void *ctx) \
+{ \
+ int *v = bpf_map_lookup_elem(&map##n, &(int){ 0 }); \
+ \
+ return v ? *v + n : 0; \
+}
+
+REP100(PROG, 1) REP100(PROG, 2) REP100(PROG, 3) REP100(PROG, 4) REP100(PROG, 5) REP100(PROG, 6)
+
+extern const void no_such_symbol __ksym __weak;
+
+/* The loader is more than 32768 insns away from its cleanup code */
+SEC("tc")
+int last(void *ctx)
+{
+ return &no_such_symbol ? 1 : 2;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH bpf-next v2 02/21] bpf: Fail the load when there is no memory
2026-11-16 15:26 ` [PATCH bpf-next v2 02/21] bpf: Fail the load when there is no memory Alexei Starovoitov
@ 2026-11-16 16:06 ` bot+bpf-ci
0 siblings, 0 replies; 26+ messages in thread
From: bot+bpf-ci @ 2026-11-16 16:06 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4961 bytes --]
> bpf: Fail the load when there is no memory
TL;DR:
- Unconverted CO-RE (bpf_core_apply) allocations may still wake the memcg OOM
killer during load
- Constant-blinding allocations lack a mayfail flag, so ENOMEM-fails-load may
still hit global OOM
- Commit message overclaims "all allocations"
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
[ ... ]
> @@ -3017,10 +3017,10 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
> prog = bpf_jit_blind_constants(env, prog);
> /*
> * Fall back to the interpreter after blinding failures, except when
> - * the loader was killed.
> + * the load has to stop or has no memory.
> */
> if (IS_ERR(prog)) {
> - if (PTR_ERR(prog) == -EINTR)
> + if (PTR_ERR(prog) == -EINTR || (env && PTR_ERR(prog) == -ENOMEM))
> return prog;
> goto out_restore;
> }
The commit message says "Use one mask for all allocations of the verifier",
and this hunk makes -ENOMEM from bpf_jit_blind_constants() fatal for the
load. Are the allocations inside bpf_jit_blind_constants() covered by that
rule?
bpf_jit_blind_constants():
linfo_unblinded = kvmalloc_array(nr_linfo, sizeof(*linfo_unblinded),
GFP_USER);
...
clone = bpf_prog_clone_create(prog, GFP_USER);
bpf_prog_clone_create():
gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | gfp_extra_flags;
fp = __vmalloc(fp_other->pages * PAGE_SIZE, gfp_flags);
Neither call has __GFP_RETRY_MAYFAIL or __GFP_NOWARN, so under global
memory pressure they can still reach the OOM killer or print an allocation
failure warning before returning the -ENOMEM that now fails the load.
The path is bpf_check() -> __bpf_prog_select_runtime(env, ...) ->
bpf_prog_jit_compile() -> bpf_jit_blind_constants() whenever blinding is
requested (bpf_jit_harden). bpf_jit_subprogs() reaches the same
allocations for programs with subprogs. These allocations are not
memcg-charged (bpf_prog_clone_create() does not use bpf_memcg_flags()), so
the memcg case from the commit message is not hit.
Other verifier-path allocations that keep a plain mask are
bpf_jit_add_poke_descriptor() (krealloc_array(..., GFP_KERNEL), called from
bpf_do_misc_fixups()) and the func[] array in jit_subprogs(). The func[]
array is converted later in the series by 8aac70a8f47f ("bpf: Remove
BPF_MAX_SUBPROGS"); bpf_jit_blind_constants() is not converted anywhere in
the series.
Should the env path of bpf_jit_blind_constants() pass
GFP_USER | BPF_GFP_MAYFAIL, as bpf_patch_list_commit() already does for
bpf_prog_realloc()? Otherwise the commit message could limit its claim to
the GFP_KERNEL_ACCOUNT sites.
> The verifier allocates with GFP_KERNEL_ACCOUNT. At the limit of the memory
> cgroup this wakes up the OOM killer, and the loader dies:
> memory.max = 64M, program needs 120M
> before: Killed
> after: bpf() = -ENOMEM
> Use one mask for all allocations of the verifier. Most of the patch is:
> sed -i 's/GFP_KERNEL_ACCOUNT/BPF_VERIFIER_GFP/'
Can a CO-RE load at memory.max still be OOM-killed from inside the
verifier? The patch does not convert the GFP_KERNEL_ACCOUNT allocations
reached through:
bpf_prog_load() -> bpf_check() -> bpf_check_core_relo() ->
bpf_core_apply()
kernel/bpf/btf.c:bpf_core_apply() {
...
specs = kzalloc_objs(*specs, 3, GFP_KERNEL_ACCOUNT);
...
cands.cands = kzalloc_objs(*cands.cands, cc->cnt, GFP_KERNEL_ACCOUNT);
...
}
bpf_core_find_cands() also reaches populate_cand_cache() (kmemdup() and
kmemdup_nul() with GFP_KERNEL_ACCOUNT) and bpf_core_add_cands() (kmalloc()
with GFP_KERNEL_ACCOUNT).
bpf_check_core_relo() runs for every load with attr->core_relo_cnt != 0,
which includes light skeletons and loader programs. try_charge_memcg()
only skips mem_cgroup_oom() for __GFP_NORETRY or __GFP_RETRY_MAYFAIL:
if (gfp_mask & __GFP_RETRY_MAYFAIL)
goto nomem;
None of these masks carries __GFP_RETRY_MAYFAIL, so the memcg OOM killer
can still be woken from the verifier, which is the "Killed" outcome the
commit message says is removed. The tree at the end of the series still
has GFP_KERNEL_ACCOUNT in bpf_core_apply().
Separately, bpf_prog_verify_signature() converts the "data" kvmalloc() but
leaves sig = kvmemdup_bpfptr(usig, attr->signature_size), which allocates
with GFP_USER | __GFP_NOWARN. That is not memcg-accounted, so it cannot
cause the memcg kill, but it is another verifier allocation that does not
use the one mask.
Should the bpf_core_apply() and candidate cache allocations be converted to
BPF_VERIFIER_GFP, or should the commit message be narrowed so it does not
claim all allocations of the verifier?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43899902250
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH bpf-next v2 11/21] libbpf: Size the tables of the loader program from the object
2026-11-16 15:26 ` [PATCH bpf-next v2 11/21] libbpf: Size the tables of the loader program from the object Alexei Starovoitov
@ 2026-11-16 16:06 ` bot+bpf-ci
0 siblings, 0 replies; 26+ messages in thread
From: bot+bpf-ci @ 2026-11-16 16:06 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3598 bytes --]
> libbpf: Size the tables of the loader program from the object
TL;DR: Larger per-program code in the generated loader may push branch offsets
past 16 bits, so gotol is emitted; kernels older than v6.4 reject it, so
previously loadable light skeletons may now fail to load there.
> gen_loader mirrors limits of the kernel: 64 maps, 32 programs and 256
> kfuncs. The numbers are known before the loader is made.
>
> fd_array: [ maps | copies of maps | unused | BTFs of kfuncs ]
> prog_fds: [ programs ] was on the stack of the loader
>
> Use gotol and a register for ctx when an offset needs more than 16 bits.
> That was an error, so loaders that could be made do not get gotol.
Is that last sentence accurate for loaders close to the old 16-bit limit?
The prog fd slots moved from the stack into the blob, which makes the
cleanup block and the per-program fd store longer. Every backward branch
to cleanup_label from emit_check_err() has to cross that code.
> diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c
> --- a/tools/lib/bpf/gen_loader.c
> +++ b/tools/lib/bpf/gen_loader.c
[ ... ]
> @@ -139,27 +141,24 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps
> [ ... ]
> for (i = 0; i < stack_sz; i += 4) {
> emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, -stack_sz + i));
> emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0, 1));
> emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
> }
> + for (i = 0; i < nr_progs; i++)
> + emit_sys_close_blob(gen, blob_prog_fd_off(gen, i));
> for (i = 0; i < nr_maps; i++)
> emit_sys_close_blob(gen, blob_fd_array_off(gen, i));
Previously this was 3 insns per stack prog_fd slot. Now each program costs
6 insns here (12 with log_level).
[ ... ]
> @@ -294,7 +283,8 @@ static void emit_check_err(struct bpf_gen *gen)
> if (is_simm16(off)) {
> emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
> } else {
> - gen->error = -ERANGE;
> + emit(gen, BPF_JMP_IMM(BPF_JSGE, BPF_REG_7, 0, 1));
> + emit(gen, BPF_JMP32_A(off - 1));
> }
[ ... ]
> @@ -1136,14 +1116,10 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
> [ ... ]
> + /* remember prog_fd in the blob, if successful */
> + emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
> + 0, 0, 0, blob_prog_fd_off(gen, gen->nr_progs)));
> + emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_7, 0));
> gen->nr_progs++;
The store of each prog fd grew from 1 insn (BPF_STX_MEM to the stack) to 3
insns here. emit_relo_ksym_typeless() also adds one insn per typeless ksym.
Take an object with N entry programs and no subprograms, where N <= 32 was
allowed before. The backward distance from the last emit_check_err() grows
by roughly N * (3 + 2) insns, or N * (9 + 2) with log_level. For N = 32
that is about 160 to 350 insns.
A loader whose last check was within that margin below 32768 insns used to
be generated with plain 16-bit BPF_JSLT branches. Now is_simm16(off) fails
and emit_check_err() emits BPF_JMP32_A (gotol) instead, which kernels older
than v6.4 reject in do_check() ("BPF_JA uses reserved fields").
So a light skeleton that a previous libbpf could build for those kernels
could now fail to load there. Should the commit message be reworded, or
should the per-program growth be kept out of the backward-branch distance?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/43899902250
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze
2026-11-16 15:26 ` [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze Alexei Starovoitov
@ 2026-11-16 16:15 ` sashiko-bot
0 siblings, 0 replies; 26+ messages in thread
From: sashiko-bot @ 2026-11-16 16:15 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] bpf: bpf_log_attr_finalize() overwrites -EINTR on
bpf_must_abort()
- [Medium] bpf: stale map->user state across restart on blinding abort
in bpf_check()
--
commit 595bfc06524fa376713bc6415b648b32cc04152d
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Stop a load only when the loader is killed or has to freeze
This commit replaces signal_pending() and fatal_signal_pending() checks across
verifier passes with a unified bpf_must_abort() helper that aborts with -EINTR
only when the loader has a fatal signal or needs to freeze, and restarts
BPF_PROG_LOAD, BPF_BTF_LOAD, and BPF_MAP_CREATE via -ERESTARTNOINTR.
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index 1d33a78d93739..735932392a87f 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
[ ... ]
> @@ -190,7 +176,7 @@ struct bpf_insn *bpf_patch_list_add(struct bpf_verifier_env *env, u32 off,
> struct bpf_patch_list *pl = &env->patch_list;
> struct bpf_insn_patch *p = pl->cnt ? &pl->patches[pl->cnt - 1] : NULL;
>
> - if (bpf_rewrite_must_abort())
> + if (bpf_must_abort())
> return ERR_PTR(-EINTR);
[Severity: Medium]
Can aborting on a non-fatal freezer request here leave mutated map state
behind when BPF_PROG_LOAD restarts?
For a single-subprog program, bpf_check() runs bpf_fixup_call_args() ->
bpf_jit_subprogs() -> resolve_func_ptrs(), which sets BPF_MAP_USER_PATCHED on
ptr->map->user and zeroes dead function-pointer slots in the frozen read-only
map, and then populates env->prog->aux->used_maps before calling
__bpf_prog_select_runtime():
kernel/bpf/verifier.c:bpf_check() {
...
env->prog = __bpf_prog_select_runtime(env, env->prog, &ret);
err_release_maps:
err = bpf_log_attr_finalize(attr_log, &env->log);
if (err)
ret = err;
if (ret)
release_insn_arrays(env);
if (!env->prog->aux->used_maps)
release_maps(env);
...
}
If constant blinding inside __bpf_prog_select_runtime() ->
bpf_prog_jit_compile() -> bpf_jit_blind_constants() hits bpf_must_abort() in
bpf_patch_list_add(), it returns -EINTR after the map was already modified and
with env->prog->aux->used_maps already set. Synchronous release_maps() is
skipped, __bpf_free_used_maps() is deferred to a workqueue (and its
cmpxchg(&map->user, (unsigned long)aux, 0) cannot clear BPF_MAP_USER_PATCHED),
and when the thawed task restarts BPF_PROG_LOAD via -ERESTARTNOINTR,
bpf_map_claim() either fails with -EBUSY or races with the deferred map
release work.
[ ... ]
> diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> index 04a3b3c80c74e..70caab9648d67 100644
> --- a/kernel/bpf/syscall.c
> +++ b/kernel/bpf/syscall.c
> @@ -6604,7 +6604,14 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size,
> SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size,
> struct bpf_common_attr __user *, uattr_common, unsigned int, size_common)
> {
> - return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
> + int err = __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
> +
> + cmd &= ~BPF_COMMON_ATTRS;
> + /* bpf_must_abort() ended the command, nothing of it is left */
> + if (err == -EINTR && task_sigpending(current) &&
> + (cmd == BPF_PROG_LOAD || cmd == BPF_BTF_LOAD || cmd == BPF_MAP_CREATE))
> + return -ERESTARTNOINTR;
> + return err;
> }
[Severity: Medium]
Does bpf_log_attr_finalize() overwrite -EINTR with -ENOSPC and prevent the
syscall from restarting after freezing?
When bpf_check() aborts with ret = -EINTR, its cleanup path still finalizes
the verifier log:
kernel/bpf/verifier.c:bpf_check() {
...
err_release_maps:
err = bpf_log_attr_finalize(attr_log, &env->log);
if (err)
ret = err;
...
}
If the user supplied a log buffer and log->len_max > log->len_total (for
example, a rolling log buffer, or under BPF_LOG_LEVEL1 where do_check() calls
bpf_flow_print_way() on err == -EINTR and do_check_common() skips
bpf_vlog_reset()), bpf_log_attr_finalize() returns -ENOSPC and clobbers
ret = -EINTR. SYSCALL_DEFINE5(bpf) then sees err == -ENOSPC instead of
-EINTR and returns -ENOSPC to user space after thawing instead of
-ERESTARTNOINTR.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116152626.4032587-1-alexei.starovoitov@gmail.com?part=1
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH bpf-next v2 06/21] bpf: Resolve BTF structs and arrays first
2026-11-16 15:26 ` [PATCH bpf-next v2 06/21] bpf: Resolve BTF structs and arrays first Alexei Starovoitov
@ 2026-11-16 17:40 ` sashiko-bot
0 siblings, 0 replies; 26+ messages in thread
From: sashiko-bot @ 2026-11-16 17:40 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] bpf: missing Fixes trailer for BTF loop detection in
btf_check_all_types()
--
commit f2b6216605cc6af96cb5995fe77081d986ad71b1
Author: Alexei Starovoitov <ast@kernel.org>
> bpf: Resolve BTF structs and arrays first
>
> btf_resolve() does not see this loop:
>
> ptr -> typedef -> array -+
> ^--------------+
>
> The walk from the pointer stops at the array and marks the typedef as
> resolved. Only the limits of the later walks end the loop. The walk for
> special fields is going to lose its limit.
>
> Resolve structs and arrays first. Then a type that holds itself is always
> found. Such BTF does not load any more.
>
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
[Severity: Low]
This fixes a bug in BTF type resolution where resolving a pointer first
marks intermediate modifiers as resolved and hides self-referential loops
through structs or arrays. Should this include:
Fixes: eb3f595dab40 ("bpf: btf: Validate type reference")
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116152626.4032587-1-alexei.starovoitov@gmail.com?part=6
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2026-11-16 17:40 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 15:26 [PATCH bpf-next v2 00/21] bpf: Remove the limits on maps, kfuncs, subprogs and special fields Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 01/21] bpf: Stop a load only when the loader is killed or has to freeze Alexei Starovoitov
2026-11-16 16:15 ` sashiko-bot
2026-11-16 15:26 ` [PATCH bpf-next v2 02/21] bpf: Fail the load when there is no memory Alexei Starovoitov
2026-11-16 16:06 ` bot+bpf-ci
2026-11-16 15:26 ` [PATCH bpf-next v2 03/21] bpf: Give a syscall program the largest context when it does tail calls Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 04/21] bpf: Do not inline bpf_loop() when its callback is dead code Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 05/21] bpf: Dump programs that have more than 524288 insns Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 06/21] bpf: Resolve BTF structs and arrays first Alexei Starovoitov
2026-11-16 17:40 ` sashiko-bot
2026-11-16 15:26 ` [PATCH bpf-next v2 07/21] bpf: Remove the limits on maps, BTFs and fd_array of a program Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 08/21] bpf: Do not scan all maps for each insn and each subprog Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 09/21] bpf: Remove the limits on kfuncs of a program Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 10/21] bpf: Remove the limits on tables of functions Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 11/21] libbpf: Size the tables of the loader program from the object Alexei Starovoitov
2026-11-16 16:06 ` bot+bpf-ci
2026-11-16 15:26 ` [PATCH bpf-next v2 12/21] bpf: Relax jumps that patches push out of range Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 13/21] bpf: Keep the call graph in lists of edges Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 14/21] bpf: Find the effects of callees on the call graph Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 15/21] bpf: Link the decl tags of a type Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 16/21] bpf: Remove BPF_MAX_SUBPROGS Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 17/21] bpf: Walk each type once when looking for special fields Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 18/21] bpf: Raise the limit on the length of BTF names Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 19/21] bpf: Find dominators in O(n log n) Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 20/21] selftests/bpf: Test programs above the old limits Alexei Starovoitov
2026-11-16 15:26 ` [PATCH bpf-next v2 21/21] selftests/bpf: Test limits of the load with raw programs Alexei Starovoitov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox