* [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits
@ 2026-11-03 23:40 Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 01/37] bpf: Keep the range of a packet pointer when a number >= 0 is added Alexei Starovoitov
` (36 more replies)
0 siblings, 37 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
This series lets the verifier of today accept more of the code that
compilers make. The main pass does not change: it follows each path of
a prog and prunes, as before.
precision A packet pointer keeps its range when the prog adds a scalar
that is not negative. A shift, a division and a remainder by
a range have bounds.
loops Scalar evolution counts more forms of 'for' and 'while', and
it has a formula for more of the registers that a loop
changes. The verifier verifies such a loop one time.
subprogs The verifier takes the arguments of a static subprog from
its code and not from BTF. A trampoline on such a subprog
keeps what the code takes, or the attach is refused.
limits 24 arguments of a static subprog, 1024 subprogs.
The patches
===========
patches 1-2 a packet pointer keeps its range when the prog adds
a scalar >= 0
patches 3-5 trampolines on static subprogs that BTF does not describe
patches 6-7 stack arguments of a static subprog from its code
patches 8-21 scalar evolution counts more loops: partial spills,
a bound in .rodata, ARG_ANYTHING, sums of constants,
ranges, more registers with a formula, over 256 exits
patches 22-25 a 32-bit counter on the stack, several back edges
patches 26-27 bounds for a shift, a division and a remainder by a range
patches 28-31 more forms of the jump that counts a loop
patch 32 more targets of a call through a table of functions
patches 33-35 24 arguments of static subprogs
patches 36-37 1024 subprogs
The tests of a patch are in the patch behind it, or in the last patch of
its group. Patches 32 and 36 raise a limit and have no test of their own.
Patches 3, 4 and 8 are fixes.
Precision
=========
if (pkt + 54 > pkt_end || off > 38)
return 0;
*(u64 *)(pkt + off)
Today the verifier rejects the load: the range of pkt is 54 bytes, but the
sum has no range. With patch 1 the sum keeps the range, and the prog can
read the 16 bytes that are in the packet for each value of off.
cpu = hash % nr_cpus;
bit = 1 << order;
Today each result is an unknown scalar, unless the second operand is
a constant. With patch 26 the result has bounds that come from the ranges
of the two operands.
Loops
=====
Scalar evolution examines each loop before the main pass of the verifier.
For 'for (i = 0; i < 10; i++)' it calculates the number of iterations. If
it also has a formula for each live register that the loop changes, then
the verifier gives these registers their ranges over all iterations and
verifies the body one time. The verifier unrolls all other loops: it
verifies them one iteration at a time.
loop --> is there a count? --> does each live register --> verified
| have a formula? one time
| no | no
v v
unrolled unrolled
Patch 8 is a fix: the second question got a wrong 'yes' for a number in
a part of a stack slot. The other patches let more loops pass:
patch question the loop
----- -------- --------------------------------------------------------
10 count The bound is in .rodata.
12 count The counter is in the stack, and a helper gets the
address of a stack slot as a number.
14 count 'r += 1; r += 1', 'r -= 1', '(u32)r' made of two shifts
16 count The first value or the bound is a range.
18 formula A number that only loses bits, a pointer to arena,
a register that gets the same constant each time
20 count The loop has more than 256 exits.
22 formula The counter is an 'int' that lives on the stack.
24 count 'continue', or a copy of the test at the end of each
path: several back edges
28 count The latch is false in the first iteration.
29 count 'for (i = 63; i >= 0; i--)', 'while (n--)'
30 count The latch tests a flag that a jump sets.
The latch is the jump that closes the loop.
The verifier relies on the count: nothing stops a counted loop at run time,
and the ranges of the variables come from the count. It does not rely on
the formulas of patch 18: it compares the state that returns to the loop
head with the state at the head.
The verifier verifies a counted loop with ranges and never goes back to one
iteration at a time. Thus each patch that counts more loops can lose a prog
that needs exact values in each iteration. The logs describe these progs.
None of them is in the corpora below.
Static subprogs and limits
==========================
BTF of a static subprog describes the source. It does not describe what the
compiler passes. rustc adds a hidden pointer to the result, and it passes
a pair in two registers. Then a function with 4 parameters takes
6 arguments.
patch the static subprog the verifier
----- ---------------------------------- ------------------------------
3 has unreliable BTF and returns in refuses fexit, fmod_ret and
R0:R2 fsession: they keep only R0
4 reads a register of r1-r5 that BTF marks BTF as unreliable: then
does not have a trampoline keeps r1-r5
6 loads more stack arguments than accepts the prog, and refuses
BTF has to attach a trampoline
limit today with the series
----------------------------------------------- ----- ---------------
arguments of a static subprog 12 24
subprogs 256 1024
functions that a load from a table of functions 64 1024
can give
The limits of kfuncs, trampolines and global subprogs do not change.
The limits are what the progs of https://github.com/4ast/rust-bpf need.
These progs do not load yet: they need more changes of the verifier.
Veristat changes
================
A: bpf-next d4bf9e34e299. B: this series. The two runs use the same
objects. The selftests that the series adds or changes are not counted:
they test the verifier itself.
Insns is the number of insns that the verifier processes. It is not the
size of a prog.
Progs is the number of progs that load in the two runs.
Corpus Progs Insns (A) Insns (B) Insns (DIFF)
--------------------- ----- --------- --------- ----------------
selftests 3573 6993083 6574720 -418363 (-5.98%)
selftests, no_alu32 3430 6287968 6044572 -243396 (-3.87%)
selftests, cpuv4 3444 5120894 5117161 -3733 (-0.07%)
selftests, bpf_gcc 3211 1520384 1438463 -81921 (-5.39%)
37 builds of Cilium 5189 42046146 42046146 +0 (+0.00%)
sched-ext/scx 493 2971692 2851736 -119956 (-4.04%)
scx, as BPF CI has it 428 1321821 1315591 -6230 (-0.47%)
tools/sched_ext 240 118790 118790 +0 (+0.00%)
public progs of Meta 511 10383121 10360656 -22465 (-0.22%)
libarena is in the first row only.
Verdict changes
---------------
No prog is lost. 5 progs load only in B:
scx_pandemonium.bpf.o pandemonium_enqueue
scx_pandemonium.bpf.o pandemonium_select_cpu
bpf_gcc/profiler1.bpf.o raw_tracepoint__sched_process_exit
bpf_gcc/profiler2.bpf.o raw_tracepoint__sched_process_exit
bpf_gcc/profiler3.bpf.o raw_tracepoint__sched_process_exit
scx_pandemonium, patch 8: a counted loop also changes a 32-bit number in
the stack. The verifier took the slot for an unknown scalar and did not
unroll the loop. But the number differs each time at the loop head, so the
verifier cannot prune there: "The sequence of 8193 jumps is too complex".
profiler, patch 24: gcc moves a pointer through a map value and ends the
loop with 'if r7 != r4', where r4 is the pointer to the end. The verifier
does not decide this jump and goes on behind the last element: "invalid
access to map value". The loop has two back edges. With a count the pointer
has its range.
Regressions
-----------
Progs that load in the two runs. The larger Insns is 1,000 or more, and B
needs at least 5% more insns. The list is complete.
File Program Insns (A) Insns (B) Insns (DIFF)
-------------------------------- ----------------- --------- --------- ----------------
lib_alloc_selftests_noasan.bpf.o alloc_selftest 23569 29898 +6329 (+26.85%)
str_map_test.bpf.o test_str_match 859 1038 +179 (+20.84%)
ci/scx_flow.bpf.o flow_dispatch 147592 166903 +19311 (+13.08%)
libarena_asan.bpf.o test_buddy_create 4563 4865 +302 (+6.62%)
libarena_asan.bpf.o arena_buddy_reset 4597 4900 +303 (+6.59%)
scx_pandemonium.bpf.o pandemonium_tick 1044 1103 +59 (+5.65%)
alloc_selftest, test_str_match, flow_dispatch and pandemonium_tick shift,
divide or take a remainder by a range. The result was an unknown scalar.
With patch 26 it has bounds, so states that were equal differ now, and the
verifier prunes fewer paths. The same bounds decide jumps in other progs:
see scx_rusty and scx_flow below.
In the two progs of libarena loops that had no count are counted now.
Top wins (30 of 89)
-------------------
Progs that load in the two runs. The larger Insns is 1,000 or more, and B
needs at least 5% fewer insns. A prog that is in several builds is listed
one time: the build with the most insns in A.
no_alu32/, bpf_gcc/: the build of the selftests. ci/: scx, as BPF CI has
it. Ranked by percentage change, then by the change of Insns, file,
program.
File Program Insns (A) Insns (B) Insns (DIFF)
-------------------------------------- --------------------------------------- --------- --------- -----------------
no_alu32/timer_start_delete_race.bpf.o start_timer 46008 32 -45976 (-99.93%)
no_alu32/timer_start_delete_race.bpf.o delete_elem 20004 15 -19989 (-99.93%)
no_alu32/access_map_in_map.bpf.o access_map_in_array 24606 43 -24563 (-99.83%)
no_alu32/access_map_in_map.bpf.o access_map_in_htab 24606 43 -24563 (-99.83%)
no_alu32/access_map_in_map.bpf.o sleepable_access_map_in_array 24606 43 -24563 (-99.83%)
no_alu32/access_map_in_map.bpf.o sleepable_access_map_in_htab 24606 43 -24563 (-99.83%)
no_alu32/linked_list.bpf.o clear_global_array_list 15384 66 -15318 (-99.57%)
no_alu32/linked_list.bpf.o clear_global_nested_list 5634 25 -5609 (-99.56%)
no_alu32/linked_list.bpf.o clear_global_list 4866 22 -4844 (-99.55%)
no_alu32/linked_list.bpf.o clear_map_list 4877 33 -4844 (-99.32%)
no_alu32/linked_list.bpf.o clear_inner_map_list 4885 41 -4844 (-99.16%)
no_alu32/rhash.bpf.o test_rhash_update_elements 8866 103 -8763 (-98.84%)
no_alu32/timer.bpf.o test1 5011 263 -4748 (-94.75%)
bpf_gcc/bpf_iter_unix.bpf.o dump_unix 5591 302 -5289 (-94.60%)
libarena_asan.bpf.o test_buddy_alloc_multiple 242319 17506 -224813 (-92.78%)
bpf_gcc/bpf_iter_task_stack.bpf.o dump_task_stack 7245 636 -6609 (-91.22%)
verity_enforce.bpf.o bpfj_verity_mmap_file 6313 876 -5437 (-86.12%)
no_alu32/verifier_arena_large.bpf.o big_alloc2 3231 487 -2744 (-84.93%)
bpf_gcc/rbtree_search_kptr.bpf.o rbtree_search_kptr_ref 1085 241 -844 (-77.79%)
libarena.bpf.o test_rbtree_add_remove_circular_reverse 13930 6349 -7581 (-54.42%)
bpf_gcc/profiler1.bpf.o kprobe__proc_sys_write 2607 1250 -1357 (-52.05%)
no_alu32/linked_list.bpf.o inner_map_list_push_pop_multiple 14553 7229 -7324 (-50.33%)
no_alu32/linked_list.bpf.o map_list_push_pop_multiple 14744 7420 -7324 (-49.67%)
no_alu32/linked_list.bpf.o global_list_push_pop_multiple 13931 7201 -6730 (-48.31%)
bpf_gcc/profiler1.bpf.o tracepoint__syscalls__sys_enter_kill 24035 12812 -11223 (-46.69%)
ci/scx_rusty.bpf.o rusty_stopping 21750 11901 -9849 (-45.28%)
bpf_gcc/profiler1.bpf.o raw_tracepoint__sched_process_exec 3080 1727 -1353 (-43.93%)
libarena.bpf.o test_rbtree_insert_many_noalloc 11478 6741 -4737 (-41.27%)
libarena.bpf.o test_rbtree_insert_many_update 11478 6741 -4737 (-41.27%)
scx_flow.bpf.o flow_select_cpu 145475 89352 -56123 (-38.58%)
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Alexei Starovoitov (37):
bpf: Keep the range of a packet pointer when a number >= 0 is added
selftests/bpf: Test range of a packet pointer after a number is added
bpf: Refuse fexit on an unreliable static subprog that returns in
R0:R2
bpf: Don't trust BTF of a static subprog that reads more registers
selftests/bpf: Test trampolines on subprogs that take more than BTF
says
bpf: Count stack arguments of a static subprog from its code
selftests/bpf: Test static subprogs that don't take what BTF says
bpf: Don't treat a partial spill as widened in scalar evolution
selftests/bpf: Test loops that change a partial spill
bpf: Let scalar evolution see loads from read-only maps
selftests/bpf: Test scalar evolution with a loop bound in .rodata
bpf: Helpers don't write through ARG_ANYTHING
selftests/bpf: Test scalar evolution with stack address as
ARG_ANYTHING
bpf: Fold sums of constants and two shifts by 32 in scalar evolution
selftests/bpf: Test sums of constants and shifts by 32 in scalar
evolution
bpf: Count loops whose first value or bound is a range
selftests/bpf: Test loops whose first value or bound is a range
bpf: Describe more registers of a counted loop
selftests/bpf: Test more registers of a counted loop
bpf: Compute scev for a loop that has more than 256 exits
selftests/bpf: Test a loop with 301 exits
bpf: Describe a 32-bit counter that lives on the stack
selftests/bpf: Test a 32-bit counter that lives on the stack
bpf: Count a loop with several back edges
selftests/bpf: Test loops with several back edges
bpf: Compute bounds for a shift, a division and a remainder by a range
selftests/bpf: Test a shift, a division and a remainder by a range
bpf: Count a loop whose latch is false in the first iteration
bpf: Count '!=' across 0 in 32 bits and loops that go down
bpf: Count a loop whose latch tests a flag that a jump sets
selftests/bpf: Test more latches of counted loops
bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of
functions
bpf: Keep the frame of analyze_subprog() small
bpf: Allow 24 arguments of static subprogs
selftests/bpf: Test a static subprog with 24 arguments
bpf: Bump BPF_MAX_SUBPROGS to 1024
selftests/bpf: Make room in veristat for the stats of 1024 subprogs
include/linux/bpf.h | 19 +-
include/linux/bpf_verifier.h | 40 +-
kernel/bpf/backtrack.c | 2 +-
kernel/bpf/btf.c | 18 +-
kernel/bpf/const_fold.c | 11 +-
kernel/bpf/fixups.c | 2 +-
kernel/bpf/liveness.c | 34 +-
kernel/bpf/loops.c | 105 +-
kernel/bpf/scev.c | 1041 +++++++++++++++--
kernel/bpf/verifier.c | 263 +++--
.../selftests/bpf/prog_tests/tracing_struct.c | 30 +-
.../selftests/bpf/prog_tests/verifier.c | 4 +
.../bpf/progs/tracing_struct_int128_tgt.c | 22 +-
.../bpf/progs/tracing_struct_int128_tracer.c | 6 +-
.../selftests/bpf/progs/verifier_alu_range.c | 168 +++
.../bpf/progs/verifier_direct_packet_access.c | 100 ++
.../selftests/bpf/progs/verifier_loop_latch.c | 165 +++
.../selftests/bpf/progs/verifier_precision.c | 11 +-
.../selftests/bpf/progs/verifier_scev.c | 692 ++++++++++-
.../selftests/bpf/progs/verifier_stack_arg.c | 325 ++++-
.../bpf/progs/verifier_stack_arg_order.c | 8 +-
tools/testing/selftests/bpf/veristat.c | 7 +-
22 files changed, 2749 insertions(+), 324 deletions(-)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_alu_range.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_latch.c
--
2.55.0
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH bpf-next 01/37] bpf: Keep the range of a packet pointer when a number >= 0 is added
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 02/37] selftests/bpf: Test range of a packet pointer after a number " Alexei Starovoitov
` (35 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
After a prog compares 'pkt + 54' with pkt_end, the verifier knows that
54 bytes are in the packet: the range of pkt (r=54 in the log). The range
stays when the prog adds a constant. But the verifier clears it when the
prog adds a scalar that is not a constant, although it has an upper bound:
if (pkt + 54 > pkt_end || off > 38)
return 0;
*(u64 *)(pkt + off) rejected: pkt has r=54, the sum has r=0
Keep the range when the scalar cannot be negative. The verifier accepts a
load if umax + insn->off + size <= range, where umax is the largest offset
of the pointer. Thus it accepts range - umax bytes after the pointer. A
scalar in [0, N] adds N to umax, and it moves the pointer by N at most:
pkt pkt + off pkt + 38 pkt + 54
| | | |
the sum its umax its range
|<-- 16 bytes ->| accepted
|<------- 54 - off bytes -------->| in the packet
Clear the range, as before, when umax of the sum is not below it. Such a
range permits only a load with a negative insn->off, which stays rejected.
It would make states different that regsafe() can prune today.
reg_bounds_sync() of the sum stays: the alignment check and the check of
the smallest offset use what var_off and the step tell about the sum. But
it can make umax smaller. Then make the range smaller by the same number,
with bpf_trim_pkt_range() of commit 88e2c896d859 ("bpf: Keep the valid
bytes of a packet pointer in scalar evolution"). This is sufficient because
the sum gets a new id, as before: no other pointer shares its range.
'(pkt + x) - y' can keep a range now, and BPF_SUB does not sync a result
that keeps a range. A prog that needs what the sync learns from var_off
about this difference (smallest offset, alignment) is rejected now. To sync
and trim the results of BPF_SUB like a sum would also change 'pkt - y',
which is not synced today: valid progs that load today are rejected then.
Sums of paths that checked different lengths now differ in the range, and
regsafe() prunes fewer of them. veristat shows the same verdicts and insns
for Cilium, Katran, sched-ext/scx and the selftests. These progs check only
the bytes in front of a variable offset: no sum keeps a range.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 24 +++++++++++++++++++-----
1 file changed, 19 insertions(+), 5 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 62f26149c3ae..fa53ee735762 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16358,6 +16358,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
const char *reason;
int ret, bounds_ret;
bool to_scalar;
+ s64 smax;
dst_reg = ®s[dst];
/*
@@ -16513,12 +16514,18 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
if (!known)
dst_reg->id = ++env->id_gen;
/*
- * Clear range for unknown addends since we can't know
- * where the pkt pointer ended up. Also clear AT_PKT_END /
+ * The range is what umax of the pointer may get to, see
+ * check_packet_access(). umax of the addend was added to it
+ * and the pointer moved by that much at most, so the range
+ * holds. Clear it if the addend may be negative: umax says
+ * nothing then. Clear it if umax got to it: it permits no
+ * byte at or after the pointer, and it would only keep equal
+ * states apart in regsafe(). Also clear AT_PKT_END /
* BEYOND_PKT_END from prior comparison as any pointer
* arithmetic invalidates them.
*/
- if (!known || dst_reg->range < 0)
+ if (dst_reg->range < 0 ||
+ (!known && (smin_val < 0 || reg_umax(dst_reg) >= dst_reg->range)))
memset(&dst_reg->raw, 0, sizeof(dst_reg->raw));
}
break;
@@ -16611,11 +16618,18 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
* A packet pointer that keeps its id or range is checked against a
* range set from the checked pointer's umax, so var_off must not tighten
* its umax. r32 must still match var_off for reg_bounds_sanity_check().
+ * A sum with a new id is different: progs rely on what var_off and the
+ * step tell about it (alignment, umin). No other pointer has its id, so
+ * it is enough that its range loses what its umax loses.
*/
- if (reg_is_pkt_pointer(dst_reg) && (known || dst_reg->range > 0))
+ if (reg_is_pkt_pointer(dst_reg) &&
+ (known || (opcode == BPF_SUB && dst_reg->range > 0))) {
__update_reg32_bounds(dst_reg);
- else
+ } else {
+ smax = reg_smax(dst_reg);
reg_bounds_sync(dst_reg);
+ bpf_trim_pkt_range(dst_reg, smax);
+ }
bounds_ret = sanitize_check_bounds(env, insn, dst_reg);
if (bounds_ret == -EACCES)
return bounds_ret;
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 02/37] selftests/bpf: Test range of a packet pointer after a number is added
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 01/37] bpf: Keep the range of a packet pointer when a number >= 0 is added Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 03/37] bpf: Refuse fexit on an unreliable static subprog that returns in R0:R2 Alexei Starovoitov
` (34 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
In each test the prog checks 54 bytes of the packet. Then it adds a number
that is not a constant to the pointer:
- [0, 38]: the range stays, 16 bytes after the sum can be read.
- [-4, 4], or [1, 54] that can get to the range: the range is cleared.
- The sum has a new id. A comparison of 'sum + 100' with pkt_end must not
give 107 bytes to the old pointer.
- var_off makes umax of a sum smaller by 2. The range must follow, or the
prog reads 2 bytes after the packet.
- 'pkt + 4' plus a loop counter 4, 12 .. 36 is a multiple of 8. Only the
step of the counter shows that. The prog loads with
BPF_F_STRICT_ALIGNMENT.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/progs/verifier_direct_packet_access.c | 100 ++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
index 139ff019d87d..6c1d2014b95b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
@@ -1032,6 +1032,106 @@ l0_%=: r0 = 0; \
: __clobber_all);
}
+/* r2 = pkt, r3 = pkt_end, r4 = unknown. 54 bytes from r2 are in the packet. */
+#define PKT_54_BYTES(insns) \
+ asm volatile (" \
+ r2 = *(u32*)(r1 + %[__sk_buff_data]); \
+ r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
+ r4 = *(u32*)(r1 + %[__sk_buff_mark]); \
+ r5 = r2; \
+ r5 += 54; \
+ if r5 > r3 goto l0_%=; \
+ " insns " \
+l0_%=: r0 = 0; \
+ exit; \
+" : \
+ : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), \
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), \
+ __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) \
+ : __clobber_all)
+
+SEC("tc")
+__description("direct packet access: checked pointer plus non-negative unknown keeps range")
+__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT)
+__naked void pkt_add_unknown_keeps_range(void)
+{
+ PKT_54_BYTES("if r4 > 38 goto l0_%=;"
+ "r2 += r4;"
+ "r0 = *(u64*)(r2 + 8);");
+}
+
+SEC("tc")
+__description("direct packet access: checked pointer plus unknown that may be negative")
+__failure __msg("invalid access to packet, off=16 size=1, R2(id={{[0-9]+}},off=16,r=0)")
+__naked void pkt_add_negative_clears_range(void)
+{
+ PKT_54_BYTES("if r4 > 8 goto l0_%=;"
+ "r4 += -4;"
+ "r2 += 20;"
+ "r2 += r4;"
+ "r0 = *(u8*)(r2 + 0);");
+}
+
+/* The byte is in the packet, but a range that umax gets to is not kept */
+SEC("tc")
+__description("direct packet access: checked pointer plus unknown that can get to the range")
+__failure __msg("invalid access to packet, off=0 size=1, R2(id={{[0-9]+}},off=0,r=0)")
+__naked void pkt_add_up_to_range_clears_range(void)
+{
+ PKT_54_BYTES("if r4 < 1 goto l0_%=;"
+ "if r4 > 54 goto l0_%=;"
+ "r2 += r4;"
+ "r0 = *(u8*)(r2 - 1);");
+}
+
+/* If r2 kept the id of r6, the check of r2 + 100 would give 107 bytes to r6 */
+SEC("tc")
+__description("direct packet access: sum that keeps range is checked, old pointer is used")
+__failure __msg("invalid access to packet, off=106 size=1, R6(id=0,off=106,r=54)")
+__naked void pkt_add_unknown_gets_new_id(void)
+{
+ PKT_54_BYTES("r6 = r2;"
+ "r4 &= 7;"
+ "r2 += r4;"
+ "r5 = r2; r5 += 100;"
+ "if r5 > r3 goto l0_%=;"
+ "r0 = *(u8*)(r6 + 106);");
+}
+
+/*
+ * r4 is 0, 8 .. 48, but its umax is 50. 16 bytes from r2 = pkt + r4 are in the packet: r=66.
+ * Then 0 or 8 is added: 8 bytes are left. var_off makes umax 56, not 58, so r=64, not 66.
+ */
+SEC("tc")
+__description("direct packet access: var_off lowers umax of a sum that keeps range")
+__failure __msg("invalid access to packet, off=64 size=1, R2(id={{[0-9]+}},off=64,r=64)")
+__naked void pkt_add_unknown_var_off(void)
+{
+ PKT_54_BYTES("r4 &= 0x38;"
+ "if r4 > 50 goto l0_%=;"
+ "r2 += r4;"
+ "r5 = r2; r5 += 16;"
+ "if r5 > r3 goto l0_%=;"
+ "r4 &= 8;"
+ "r2 += r4;"
+ "r0 = *(u8*)(r2 + 7);"
+ "r0 = *(u8*)(r2 + 8);");
+}
+
+/* r4 is 4, 12 .. 36. Only its step says that pkt + 4 + r4 is a multiple of 8. */
+SEC("tc")
+__description("direct packet access: checked pointer plus loop counter, alignment")
+__success __retval(0) __flag(BPF_F_STRICT_ALIGNMENT)
+__naked void pkt_add_counter_alignment(void)
+{
+ PKT_54_BYTES("r4 = 4;"
+ "l1_%=: r5 = r2; r5 += 4;"
+ "r5 += r4;"
+ "r0 = *(u64*)(r5 + 6);"
+ "r4 += 8;"
+ "if r4 < 40 goto l1_%=;");
+}
+
SEC("tc")
__description("direct packet access: no pruning of a path whose checks prove fewer bytes")
__failure __msg("invalid access to packet, off=255 size=8")
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 03/37] bpf: Refuse fexit on an unreliable static subprog that returns in R0:R2
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 01/37] bpf: Keep the range of a packet pointer when a number >= 0 is added Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 02/37] selftests/bpf: Test range of a packet pointer after a number " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 04/37] bpf: Don't trust BTF of a static subprog that reads more registers Alexei Starovoitov
` (33 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A trampoline keeps only R0 of the value that its target returns. Thus
fexit, fmod_ret and fsession cannot attach to a function that returns more
than 8 bytes. The check reads the size from the function model.
When BTF of a static subprog is unreliable, the model is not made from BTF.
It has 5 u64 arguments and a u64 return value. The check passes, and the
caller of the subprog gets R2 as the trampoline left it, where the verifier
saw the upper half of the result.
BTF is unreliable because of the arguments. The verifier still takes the
return type from BTF, and it verifies the subprog and its callers with a
result in R0:R2. Keep that decision in bpf_func_info_aux, and refuse the
attach when it is set.
fentry does not touch the result. It can still attach.
Fixes: 0630ad00d96d ("bpf: Add verifier support for 16-byte returns in R0: R2")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 2 ++
kernel/bpf/verifier.c | 5 ++++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5413fa5d2b2b..ef94a168c902 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1763,6 +1763,8 @@ struct bpf_subprog_arg_info {
struct bpf_func_info_aux {
u16 linkage;
bool unreliable;
+ /* The function returns in R0:R2 */
+ bool ret_reg_pair;
bool called[BPF_FUNC_CTX_CNT];
bool verified[BPF_FUNC_CTX_CNT];
bool called_with_pkt_ptrs;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fa53ee735762..dcfced2573ee 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -465,6 +465,7 @@ static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
return -EFAULT;
if (ret_regs_cnt(size) > 1) {
subprog_info(env, subprog)->ret_reg_pair = true;
+ subprog_aux(env, subprog)->ret_reg_pair = true;
env->prog->jit_required = 1;
}
}
@@ -22990,7 +22991,9 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
return ret;
}
- if (tgt_info->fmodel.ret_size > 8 &&
+ /* The model for unreliable BTF has a u64. The function still returns R0:R2. */
+ if ((tgt_info->fmodel.ret_size > 8 ||
+ (tgt_prog && tgt_prog->aux->func_info_aux[subprog].ret_reg_pair)) &&
attach_uses_trampoline_retval(prog->expected_attach_type)) {
bpf_log(log,
"Attach to function %s with a >8 byte return value is not supported for this attach type\n",
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 04/37] bpf: Don't trust BTF of a static subprog that reads more registers
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (2 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 03/37] bpf: Refuse fexit on an unreliable static subprog that returns in R0:R2 Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 05/37] selftests/bpf: Test trampolines on subprogs that take more than BTF says Alexei Starovoitov
` (32 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A trampoline saves and restores the argument registers that the BTF
prototype of its target has. The prog that is attached can clobber all
other registers.
BTF of a static subprog describes the source. It does not describe what the
compiler passes: the code can take more arguments than the prototype has
parameters. rustc, for example, adds a hidden pointer to the result.
Thus the verifier compares BTF with the registers at each call of a static
subprog. When they disagree it marks BTF as unreliable, and a trampoline
keeps all of r1-r5. But the verifier examines only the registers that BTF
has. A subprog that reads more of r1-r5 keeps reliable BTF:
BTF: f(a) the trampoline keeps r1
code: reads r1 and r2 the caller's r2 is lost
With fentry, fexit or fsession attached, the subprog runs with a value in
r2 that the verifier did not see.
Compare in the other direction too: BTF is unreliable when a register of
r1-r5 that BTF does not have is live at the entry of the subprog. A prog
can still attach. It gets 5 u64 arguments instead of the arguments of BTF,
and fexit gets the result behind them.
Liveness errs on the safe side. One static function of the C schedulers in
sched-ext/scx gets unreliable BTF though its BTF is right: a call of a
helper that the kernel does not have counts as a read of r1-r5.
There is one exception. Liveness misses r3-r5 of bpf_trace_printk() and the
flags of bpf_get_branch_snapshot(), which their protos do not declare. A
subprog that takes them from its caller keeps reliable BTF, and it prints
or passes a wrong number under a tracer. The helpers are safe with any
value there: the verifier never checks these registers.
Fixes: 8c1b6e69dcc1 ("bpf: Compare BTF types of functions arguments with actual types")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index dcfced2573ee..d95f86b62e99 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11325,6 +11325,20 @@ static void gen_subprog_arg_proto(const struct bpf_subprog_info *sub, const stru
}
}
+/*
+ * The argument slots that the code of a static subprog takes, whatever its BTF says.
+ * Liveness errs on the safe side: a callx in the subprog counts as a read of r1-r5.
+ * But it misses r3-r5 of bpf_trace_printk() and the flags of bpf_get_branch_snapshot(), which
+ * their protos do not declare. Such a subprog prints or passes a wrong number under a tracer.
+ * The helpers are safe with any value there: the verifier never checks these registers.
+ */
+static u32 code_arg_slots(struct bpf_verifier_env *env, struct bpf_subprog_info *sub)
+{
+ u32 live_regs = env->insn_aux_data[sub->start].live_regs_before;
+
+ return fls((live_regs & GENMASK(BPF_REG_5, BPF_REG_1)) >> BPF_REG_1);
+}
+
static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, struct btf *btf)
{
struct bpf_subprog_info *sub = subprog_info(env, subprog);
@@ -11361,10 +11375,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
meta.func_proto = func_proto;
/*
- * Check if all register arguments declared in BTF are initialized.
- * If they aren't the BTF is unreliable.
+ * Check if all register arguments declared in BTF are initialized,
+ * and if the code takes no other arguments: a trampoline keeps only
+ * what BTF declares. If not, the BTF is unreliable.
*/
if (!bpf_subprog_is_global(env, subprog)) {
+ if (code_arg_slots(env, sub) > sub->arg_slot_cnt)
+ return -EINVAL;
for (i = 0; i < min_t(u32, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); i++)
if (caller->regs[BPF_REG_1 + i].type == NOT_INIT)
return -EINVAL;
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 05/37] selftests/bpf: Test trampolines on subprogs that take more than BTF says
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (3 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 04/37] bpf: Don't trust BTF of a static subprog that reads more registers Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
` (31 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
callee_takes_r5() has one parameter in BTF. Its code also reads r5, and it
returns in r0 and r2.
test attached expected
--------------------- -------- ------------------------------------
int128_tgt_prog fentry the subprog gets r5 of its caller
tgt_prog_returns_pair fexit the verifier refuses to load the prog
The fentry prog reads r5 from its context. That loads only when the
trampoline has 5 u64 arguments. Then it clobbers r1-r5, so that the result
does not depend on what the kernel leaves in these registers.
test_tgt_prog_unreliable() gets the prog, the target and the message as
arguments, so that both refusals share it.
The two cases need no stack arguments. They still run only where the
compiler and the JIT have them, because they share that function and the
block of the static subprogs with the cases that need them.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/tracing_struct.c | 27 +++++++++++++------
.../bpf/progs/tracing_struct_int128_tgt.c | 15 ++++++++++-
.../bpf/progs/tracing_struct_int128_tracer.c | 6 ++++-
3 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
index de358f335d98..ab74ba077041 100644
--- a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
+++ b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
@@ -198,6 +198,10 @@ static void test_int128_tgt_prog(void)
"callee_split");
err = err ?: bpf_program__set_attach_target(tracer->progs.split_exit, fd,
"callee_split");
+ /* The trampoline has to keep r5, which BTF doesn't have. */
+ bpf_program__set_autoload(tracer->progs.unreliable_entry, true);
+ err = err ?: bpf_program__set_attach_target(tracer->progs.unreliable_entry, fd,
+ "callee_takes_r5");
} else {
bpf_program__set_autoload(tracer->progs.stack_entry, false);
bpf_program__set_autoload(tracer->progs.stack_exit, false);
@@ -239,6 +243,8 @@ static void test_int128_tgt_prog(void)
ASSERT_EQ(tracer->bss->split_i, 10, "split:i");
ASSERT_EQ(tgt->bss->ret_split, 246, "split ret");
ASSERT_EQ(tracer->bss->split_ret, 246, "split fexit ret");
+ ASSERT_EQ(tracer->bss->unreliable_r5, 2, "takes more:r5");
+ ASSERT_EQ(tgt->bss->ret_takes_more, 11, "takes more ret");
}
out:
@@ -252,14 +258,15 @@ static void test_int128_tgt_prog(void)
/*
* A static function whose BTF is unreliable gets a trampoline that only knows
* of the register arguments, so one taking stack arguments must be refused.
+ * It only knows of r0 too, so fexit to one returning in r0 and r2 must be refused.
*/
-static void test_tgt_prog_unreliable(void)
+static void test_tgt_prog_unreliable(const char *name, const char *func, const char *msg)
{
#if defined(__x86_64__) || defined(__aarch64__)
struct tracing_struct_int128_tracer *tracer = NULL;
struct tracing_struct_int128_tgt *tgt;
+ struct bpf_program *prog, *entry;
char log_buf[4096] = {};
- struct bpf_program *prog;
int err, fd;
tgt = tracing_struct_int128_tgt__open_and_load();
@@ -277,16 +284,16 @@ static void test_tgt_prog_unreliable(void)
bpf_object__for_each_program(prog, tracer->obj)
bpf_program__set_autoload(prog, false);
- bpf_program__set_autoload(tracer->progs.unreliable_entry, true);
- bpf_program__set_log_buf(tracer->progs.unreliable_entry, log_buf, sizeof(log_buf));
- err = bpf_program__set_attach_target(tracer->progs.unreliable_entry, fd,
- "callee_unreliable");
+ entry = bpf_object__find_program_by_name(tracer->obj, name);
+ bpf_program__set_autoload(entry, true);
+ bpf_program__set_log_buf(entry, log_buf, sizeof(log_buf));
+ err = bpf_program__set_attach_target(entry, fd, func);
if (!ASSERT_OK(err, "set_attach_target"))
goto out;
err = tracing_struct_int128_tracer__load(tracer);
ASSERT_ERR(err, "tracer__load");
- ASSERT_HAS_SUBSTR(log_buf, "unreliable BTF and stack arguments", "log");
+ ASSERT_HAS_SUBSTR(log_buf, msg, "log");
out:
tracing_struct_int128_tracer__destroy(tracer);
@@ -389,7 +396,11 @@ void test_tracing_struct(void)
if (test__start_subtest("int128_tgt_prog"))
test_int128_tgt_prog();
if (test__start_subtest("tgt_prog_unreliable"))
- test_tgt_prog_unreliable();
+ test_tgt_prog_unreliable("unreliable_entry", "callee_unreliable",
+ "unreliable BTF and stack arguments");
+ if (test__start_subtest("tgt_prog_returns_pair"))
+ test_tgt_prog_unreliable("odd_exit", "callee_takes_r5",
+ "with a >8 byte return value");
if (test__start_subtest("stack_arg_tailcall"))
test_stack_arg_tailcall();
if (test__start_subtest("union_args"))
diff --git a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
index 2561422c25dd..2e2ddd1b52df 100644
--- a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
+++ b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
@@ -2,13 +2,14 @@
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
struct int128_arg {
__int128 v;
};
__u64 base;
-__u64 ret_odd, ret_stack, ret_split, ret_unreliable;
+__u64 ret_odd, ret_stack, ret_split, ret_unreliable, ret_takes_more;
#if defined(__clang__)
@@ -68,6 +69,17 @@ static __noinline __u64 callee_unreliable(__u64 *p, __u64 b, __u64 c, __u64 d,
return *p + b * 2 + c * 3 + d * 4 + e * 5 + f * 6;
}
+/* BTF has 'a'. The code also takes r5. It returns in r0 and r2. */
+static __naked __noinline __used __int128 callee_takes_r5(__u64 a)
+{
+ asm volatile ("r0 = r1; r0 += r5; r2 = r5; exit;");
+}
+
+static __naked __noinline __u64 call_takes_more(void)
+{
+ asm volatile ("r1 = 7; r5 = 2; call callee_takes_r5; r0 += r2; exit;");
+}
+
#else
const volatile bool has_stack_arg = false;
@@ -93,6 +105,7 @@ int run(void *ctx)
ret_unreliable = callee_unreliable(&x, base + 2, base + 3,
base + 4, base + 5, base + 6);
}
+ ret_takes_more = call_takes_more();
#endif
#endif
return 0;
diff --git a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c
index fc81c2eea4a8..6a1d5ce0c171 100644
--- a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c
+++ b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c
@@ -67,10 +67,14 @@ int split_exit(unsigned long long *ctx)
return 0;
}
-/* Refused: the trampoline would lose the stack argument. */
+long unreliable_r5;
+
+/* For static functions with unreliable BTF: the arguments are r1-r5. It clobbers them. */
SEC("?fentry/callee_unreliable")
int unreliable_entry(unsigned long long *ctx)
{
+ unreliable_r5 = ctx[4];
+ asm volatile ("r1 = 0; r2 = 0; r3 = 0; r4 = 0; r5 = 0;" ::: "r1", "r2", "r3", "r4", "r5");
return 0;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (4 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 05/37] selftests/bpf: Test trampolines on subprogs that take more than BTF says Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-04 0:14 ` sashiko-bot
2026-11-03 23:40 ` [PATCH bpf-next 07/37] selftests/bpf: Test static subprogs that don't take what BTF says Alexei Starovoitov
` (30 subsequent siblings)
36 siblings, 1 reply; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The caller stores the 6th and subsequent arguments of a subprog through
r11, and the subprog loads them through r11. The verifier takes the number
of these stack arguments from the BTF prototype of the subprog.
But BTF describes the source. It does not describe what the compiler passes
to a static function. rustc returns Option<i32> through a hidden pointer,
and it passes Option<i32> in two registers:
fn pick_idle_cpu(p: Task, prev_cpu: i32, this_cpu: Option<i32>,
wake_flags: u64) -> Option<i32>
BTF: p, prev_cpu, this_cpu, wake_flags 4
code: result, p, prev_cpu, this_cpu (2 registers), wake_flags 6
invalid read from stack arg off 8 depth 0
31 of the 225 progs of the sched-ext schedulers in rust-bpf have such a
function.
For a static subprog use the code: the number of its stack arguments is the
highest slot that it loads through r11 on any path, dead code included. The
verifier verifies a static subprog with the state of its caller, so a load
gives the value that the caller stored. At each call and callx the caller
must have stored all slots up to that number. The error is -EFAULT, as
today with reliable BTF. Two tests of verifier_stack_arg_order.c now fail
at the call: update their messages. Global subprogs do not change: BTF is
their only source.
A trampoline keeps only what BTF has. Thus BTF is unreliable when the code
loads more stack arguments, and then the attach is refused.
BTF alone does not reject a static subprog any more. The JIT is asked at a
store through r11, not when BTF has 6 parameters. More than 12 parameters,
or a slot of BTF that is neither stored nor loaded, make BTF unreliable.
The caller can store the slots that a callee loads, and the slots of the
prototype of a callee that it has stored when the verifier reaches the
call. A store to another slot is an error as before. Thus a callee that
ignores its last stack argument must have it in BTF. LLVM loads all stack
arguments at function entry, and it stores them in front of the call.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 2 +
include/linux/bpf_verifier.h | 9 ++
kernel/bpf/btf.c | 18 ++-
kernel/bpf/fixups.c | 2 +-
kernel/bpf/verifier.c | 106 ++++++++++--------
.../bpf/progs/verifier_stack_arg_order.c | 8 +-
6 files changed, 81 insertions(+), 64 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index ef94a168c902..ca8992702dbd 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1765,6 +1765,8 @@ struct bpf_func_info_aux {
bool unreliable;
/* The function returns in R0:R2 */
bool ret_reg_pair;
+ /* The code of a static function loads stack arguments */
+ bool stack_args;
bool called[BPF_FUNC_CTX_CNT];
bool verified[BPF_FUNC_CTX_CNT];
bool called_with_pkt_ptrs;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 4a514f27f239..581458927512 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -984,12 +984,21 @@ struct bpf_subprog_info {
enum priv_stack_mode priv_stack_mode;
struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+ /*
+ * Static subprog: the number of arguments on the stack is the highest slot that
+ * the code loads. BTF describes the source. The compiler may pass something else:
+ * a hidden pointer to the result, a pair in two registers, no dead arguments.
+ */
+ bool stack_args_from_code: 1;
+ u8 code_stack_args:4;
u16 stack_arg_cnt; /* incoming + max outgoing */
u16 max_out_stack_arg_cnt;
};
static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
{
+ if (sub->stack_args_from_code)
+ return sub->code_stack_args;
if (sub->arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS)
return sub->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS;
return 0;
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 8bf39098ba97..228b93d4f8c9 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -8742,8 +8742,7 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
}
static int btf_check_arg_slots(struct bpf_verifier_log *log, const char *tname,
- bool is_global, u32 slot_cnt,
- struct bpf_subprog_info *sub)
+ bool is_global, u32 slot_cnt)
{
if (slot_cnt <= MAX_BPF_FUNC_REG_ARGS)
return 0;
@@ -8755,12 +8754,7 @@ static int btf_check_arg_slots(struct bpf_verifier_log *log, const char *tname,
tname, slot_cnt, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}
- if (!bpf_jit_supports_stack_args()) {
- bpf_log(log, "JIT does not support function %s() with %d argument slots\n",
- tname, slot_cnt);
- return -EFAULT;
- }
- sub->stack_arg_cnt = slot_cnt - MAX_BPF_FUNC_REG_ARGS;
+ /* The code of a static function tells which stack arguments it takes */
return 0;
}
@@ -8827,13 +8821,15 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
}
args = (const struct btf_param *)(t + 1);
nargs = btf_type_vlen(t);
- sub->arg_slot_cnt = nargs;
+ sub->arg_slot_cnt = min_t(u32, nargs, MAX_BPF_FUNC_ARGS);
if (nargs > MAX_BPF_FUNC_ARGS) {
+ if (!is_global) /* not fatal for static funcs */
+ return -EINVAL;
bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
MAX_BPF_FUNC_ARGS, tname, nargs);
return -EFAULT;
}
- err = btf_check_arg_slots(log, tname, is_global, nargs, sub);
+ err = btf_check_arg_slots(log, tname, is_global, nargs);
if (err)
return err;
@@ -9022,7 +9018,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return -EINVAL;
}
- err = btf_check_arg_slots(log, tname, is_global, slots_used, sub);
+ err = btf_check_arg_slots(log, tname, is_global, slots_used);
if (err)
return err;
sub->arg_slot_cnt = slots_used;
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 6b719a28be68..e8442d39d8a8 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1677,7 +1677,7 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
return -EINVAL;
}
for (i = 0; i < env->subprog_cnt; i++) {
- if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
+ if (env->subprog_info[i].stack_arg_cnt) {
verbose(env, "stack args are not supported in non-JITed programs\n");
return -EINVAL;
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d95f86b62e99..b480c5c242af 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4639,6 +4639,12 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
struct bpf_reg_state *arg;
int err;
+ /* BTF is not needed for stack arguments, so ask here. There is no load without a store. */
+ if (!bpf_jit_supports_stack_args()) {
+ verbose(env, "JIT does not support stack arguments\n");
+ return -EOPNOTSUPP;
+ }
+
if (spi >= max_stack_arg_regs) {
verbose(env, "stack arg write offset %d exceeds max %d stack args\n",
off, max_stack_arg_regs);
@@ -11336,6 +11342,8 @@ static u32 code_arg_slots(struct bpf_verifier_env *env, struct bpf_subprog_info
{
u32 live_regs = env->insn_aux_data[sub->start].live_regs_before;
+ if (bpf_in_stack_arg_cnt(sub))
+ return MAX_BPF_FUNC_REG_ARGS + bpf_in_stack_arg_cnt(sub);
return fls((live_regs & GENMASK(BPF_REG_5, BPF_REG_1)) >> BPF_REG_1);
}
@@ -11346,7 +11354,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
const struct btf_type *func, *func_proto;
struct bpf_call_arg_meta meta;
struct bpf_func_proto *fn;
- int ret, err;
+ int ret;
u32 i;
memset(&meta, 0, sizeof(meta));
@@ -11355,16 +11363,8 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
meta.func_name = bpf_subprog_name(env, subprog);
ret = btf_prepare_func_args(env, subprog);
- if (ret) {
- if (bpf_in_stack_arg_cnt(sub) > 0) {
- err = check_outgoing_stack_args(env, caller, sub->arg_slot_cnt,
- bpf_subprog_name(env, subprog),
- NULL, NULL);
- if (err)
- return err;
- }
+ if (ret)
return ret;
- }
func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
func_proto = btf_type_by_id(btf, func->type);
@@ -11375,7 +11375,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
meta.func_proto = func_proto;
/*
- * Check if all register arguments declared in BTF are initialized,
+ * Check if all arguments declared in BTF are initialized,
* and if the code takes no other arguments: a trampoline keeps only
* what BTF declares. If not, the BTF is unreliable.
*/
@@ -11385,6 +11385,10 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
for (i = 0; i < min_t(u32, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); i++)
if (caller->regs[BPF_REG_1 + i].type == NOT_INIT)
return -EINVAL;
+ /* check_func_args() reports a missing slot that the code loads: it has the name */
+ for (i = max(i, code_arg_slots(env, sub)); i < sub->arg_slot_cnt; i++)
+ if (!bpf_get_spilled_stack_arg(i - MAX_BPF_FUNC_REG_ARGS, caller))
+ return -EINVAL;
}
return check_func_args(env, &meta, env->insn_idx);
}
@@ -11525,8 +11529,8 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
int *insn_idx)
{
+ struct bpf_subprog_info *caller_info, *callee_info = &env->subprog_info[subprog];
struct bpf_verifier_state *state = env->cur_state;
- struct bpf_subprog_info *caller_info;
u16 callee_incoming, stack_arg_cnt;
struct bpf_func_state *caller;
int err;
@@ -11537,13 +11541,24 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
/*
* Track caller's total stack arg count (incoming + max outgoing).
* This is needed so the JIT knows how much stack arg space to allocate.
+ * The callee can load less than its prototype has, and the caller can still store
+ * all of it. Count the slots of the prototype only as far as the caller has stored
+ * them: compilers store in front of the call. bpf_fixup_call_args() rejects the rest.
*/
caller_info = &env->subprog_info[caller->subprogno];
- callee_incoming = bpf_in_stack_arg_cnt(&env->subprog_info[subprog]);
- stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming;
+ callee_incoming = bpf_in_stack_arg_cnt(callee_info);
+ stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) +
+ max_t(int, callee_incoming,
+ min_t(int, caller_info->max_out_stack_arg_cnt,
+ callee_info->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS));
if (stack_arg_cnt > caller_info->stack_arg_cnt)
caller_info->stack_arg_cnt = stack_arg_cnt;
+ err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS + callee_incoming,
+ bpf_subprog_name(env, subprog), NULL, NULL);
+ if (err)
+ return err;
+
/*
* For regular function entry setup new frame and continue
* from that frame.
@@ -15213,12 +15228,29 @@ static bool try_btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
/* Prepare argument metadata for liveness; the main program is handled separately. */
static void prepare_subprog_args(struct bpf_verifier_env *env)
{
- int i;
+ struct bpf_subprog_info *sub;
+ struct bpf_insn *insn;
+ int i, j, cnt;
for (i = 1; i < env->subprog_cnt; i++) {
+ sub = &env->subprog_info[i];
if (!try_btf_prepare_func_args(env, i) && bpf_subprog_is_global(env, i))
/* Keep all argument registers live if the prototype is unavailable. */
- env->subprog_info[i].arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
+ sub->arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
+ if (bpf_subprog_is_global(env, i))
+ continue;
+ cnt = 0;
+ for (j = sub->start; j < (sub + 1)->start; j++) {
+ insn = &env->prog->insnsi[j];
+ if (is_stack_arg_ldx(insn))
+ cnt = max(cnt, insn->off / BPF_REG_SIZE);
+ }
+ sub->stack_args_from_code = true;
+ sub->code_stack_args = min(cnt, MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS);
+ /* incoming + max outgoing. Calls add the latter. */
+ sub->stack_arg_cnt = sub->code_stack_args;
+ if (env->prog->aux->func_info_aux)
+ env->prog->aux->func_info_aux[i].stack_args = sub->code_stack_args;
}
}
@@ -22062,7 +22094,6 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
sub->args[0].arg_type != ARG_PTR_TO_CTX) {
env->prog->aux->func_info_aux[0].unreliable = true;
sub->arg_slot_cnt = 1;
- sub->stack_arg_cnt = 0;
}
}
@@ -22583,33 +22614,6 @@ static int flatten_bpf_func_model(struct bpf_verifier_log *log, const char *tnam
return 0;
}
-/*
- * The verifier lets a static function use stack arguments counted from its
- * BTF, even if that BTF turns out unreliable. A trampoline on the function is
- * then given only MAX_BPF_FUNC_REG_ARGS u64 arguments and would lose the
- * stack arguments, so such a function cannot be attached to.
- */
-static int check_unreliable_func_args(struct bpf_verifier_log *log, struct btf *btf,
- const struct btf_type *func_proto, const char *tname)
-{
- struct btf_func_model m;
- u32 i, nslots = 0;
-
- if (btf_distill_func_proto(NULL, btf, func_proto, tname, &m)) {
- /* at least a slot for each argument */
- nslots = btf_type_vlen(func_proto);
- } else {
- for (i = 0; i < m.nr_args; i++)
- nslots += btf_func_model_arg_slots(&m, i);
- }
- if (nslots <= MAX_BPF_FUNC_REG_ARGS)
- return 0;
-
- bpf_log(log, "Cannot attach to static function %s: unreliable BTF and stack arguments\n",
- tname);
- return -EINVAL;
-}
-
/* Functions may share a type id. The first of them is attached to. */
static int find_tgt_subprog(const struct bpf_prog *tgt_prog, u32 btf_id)
{
@@ -22991,10 +22995,16 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
return -EINVAL;
if (tgt_prog && conservative) {
- if (subprog > 0) {
- ret = check_unreliable_func_args(log, btf, t, tname);
- if (ret)
- return ret;
+ /*
+ * The verifier lets a static function use the stack arguments that
+ * its code loads, whatever its BTF says. The trampoline is given
+ * MAX_BPF_FUNC_REG_ARGS u64 arguments and would lose them.
+ */
+ if (tgt_prog->aux->func_info_aux[subprog].stack_args) {
+ bpf_log(log,
+ "Cannot attach to static function %s: unreliable BTF and stack arguments\n",
+ tname);
+ return -EINVAL;
}
t = NULL;
}
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
index 5bfcd300c7a0..cd5a77dd467b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
@@ -127,8 +127,8 @@ __naked void stack_arg_pruning_load_after_call(void)
/*
* "bad_ptr": the first arg is 'long *', which is not a recognized pointer
* type for static subprogs (not ctx, dynptr, or tagged). btf_prepare_func_args()
- * sets arg_slot_cnt = 7 / stack_arg_cnt = 2, then fails with -EINVAL. The
- * subprog is marked unreliable but the call still proceeds for static subprogs.
+ * fails with -EINVAL. The subprog is marked unreliable but the call still
+ * proceeds for static subprogs. The two loads through r11 make 2 stack arguments.
*/
__noinline __used __naked
static void subprog_bad_ptr_7args(long *a, int b, int c, int d, int e, int f, int g)
@@ -182,7 +182,7 @@ __naked void stack_arg_read_without_write_1(void)
SEC("tc")
__description("stack_arg: read with not-initialized caller write")
__failure
-__msg("invalid read from uninitialized stack arg off 8")
+__msg("callee expects 7 args, stack arg1 is not initialized")
__btf_func_path("btf__verifier_stack_arg_order.bpf.o")
__naked void stack_arg_read_without_write_2(void)
{
@@ -204,7 +204,7 @@ __naked void stack_arg_read_without_write_2(void)
SEC("tc")
__description("stack_arg: unreliable callee with too few stack args")
__failure
-__msg("invalid read from uninitialized stack arg off 16")
+__msg("callee expects 7 args, stack arg2 is not initialized")
__btf_func_path("btf__verifier_stack_arg_order.bpf.o")
__naked void stack_arg_read_with_short_bridge_write(void)
{
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 07/37] selftests/bpf: Test static subprogs that don't take what BTF says
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (5 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 08/37] bpf: Don't treat a partial spill as widened in scalar evolution Alexei Starovoitov
` (29 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The compiler can pass more or fewer arguments to a static subprog than its
BTF prototype has parameters.
test the subprog result
------------------------------- ---------------------------- ----------
more_than_prototype 1 in BTF, loads the 6th returns 7
more_than_prototype_no_store the same, the caller stores reject
nothing
less_than_prototype 7 in BTF, reads r1-r5, the returns 15
caller stores nothing
stack_arg_write_prototype_slots the same, the caller stores returns 15
the 6th and the 7th
r11_no_jit_support a store through r11, the JIT reject
has no stack arguments
global_more_than_prototype global, 1 in BTF, loads the reject
6th
tgt_prog_takes_6th 1 in BTF, loads the 6th, refuse
fentry attaches to it
more_than_prototype also examines the x86 code of the subprog. The subprog
passes no stack arguments to other functions. Thus the JIT must not move
rsp in it.
less_than_prototype has no insn that uses r11. It has a tail call, which a
prog with stack arguments cannot have. Thus it runs only with a JIT that
mixes tail calls and subprogs, also where that JIT has no stack arguments.
more_than_prototype and stack_arg_write_prototype_slots store through r11.
The interpreter has no stack arguments: they load only when the JIT is on.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/tracing_struct.c | 3 +
.../bpf/progs/tracing_struct_int128_tgt.c | 9 +-
.../selftests/bpf/progs/verifier_stack_arg.c | 102 ++++++++++++++++++
3 files changed, 113 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
index ab74ba077041..a08b480eca73 100644
--- a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
+++ b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
@@ -398,6 +398,9 @@ void test_tracing_struct(void)
if (test__start_subtest("tgt_prog_unreliable"))
test_tgt_prog_unreliable("unreliable_entry", "callee_unreliable",
"unreliable BTF and stack arguments");
+ if (test__start_subtest("tgt_prog_takes_6th"))
+ test_tgt_prog_unreliable("unreliable_entry", "callee_takes_6th",
+ "unreliable BTF and stack arguments");
if (test__start_subtest("tgt_prog_returns_pair"))
test_tgt_prog_unreliable("odd_exit", "callee_takes_r5",
"with a >8 byte return value");
diff --git a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
index 2e2ddd1b52df..6a8079fe809b 100644
--- a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
+++ b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
@@ -69,6 +69,12 @@ static __noinline __u64 callee_unreliable(__u64 *p, __u64 b, __u64 c, __u64 d,
return *p + b * 2 + c * 3 + d * 4 + e * 5 + f * 6;
}
+/* BTF has 'a'. The code also takes the 6th argument, on the stack. */
+static __naked __noinline __used __u64 callee_takes_6th(__u64 a)
+{
+ asm volatile ("r0 = *(u64 *)(r11 + 8); r0 += r1; exit;");
+}
+
/* BTF has 'a'. The code also takes r5. It returns in r0 and r2. */
static __naked __noinline __used __int128 callee_takes_r5(__u64 a)
{
@@ -77,7 +83,8 @@ static __naked __noinline __used __int128 callee_takes_r5(__u64 a)
static __naked __noinline __u64 call_takes_more(void)
{
- asm volatile ("r1 = 7; r5 = 2; call callee_takes_r5; r0 += r2; exit;");
+ asm volatile ("r1 = 1; *(u64 *)(r11 - 8) = 6; call callee_takes_6th;"
+ "r1 = r0; r5 = 2; call callee_takes_r5; r0 += r2; exit;");
}
#else
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index abf319cb8514..d1fd5a402565 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -3,6 +3,7 @@
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
#include "bpf_misc.h"
struct {
@@ -12,6 +13,40 @@ struct {
__type(value, long long);
} map_hash_8b SEC(".maps");
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+/* BTF has what the source says. The compiler dropped the arguments that are not used. */
+static __naked __noinline __used
+long takes_5_says_7(long a, long b, long c, long d, long e, long f, long g)
+{
+ asm volatile ("r0 = r1; r0 += r2; r0 += r3; r0 += r4; r0 += r5; exit;" ::: __clobber_all);
+}
+
+#define CALL_TAKES_5_SAYS_7(stores) \
+ "r1 = 1; r2 = 2; r3 = 3; r4 = 4; r5 = 5;" stores "call takes_5_says_7; exit;"
+
+/* No insn uses r11: the JIT needs no stack arguments, and a tail call is allowed */
+SEC("tc")
+__description("stack_arg: static subprog takes less than its prototype says")
+__success __retval(15)
+__load_if_JITed()
+__arch_x86_64 __arch_arm64 __arch_riscv64 __arch_s390x __arch_loongarch
+__naked void less_than_prototype(void)
+{
+ asm volatile ("r2 = %[jmp_table] ll; r3 = 0; call %[bpf_tail_call];"
+ CALL_TAKES_5_SAYS_7("")
+ :: __imm_addr(jmp_table), __imm(bpf_tail_call) : __clobber_all);
+}
+
+#endif
+
#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && \
defined(__BPF_FEATURE_STACK_ARGUMENT)
@@ -94,6 +129,52 @@ __naked void stack_arg_read_uninitialized(void)
);
}
+/* BTF has 'a'. The code also takes the 6th argument. It passes none: rsp stays where it is. */
+static __naked __noinline __used long takes_6th_says_1(long a)
+{
+ asm volatile ("r0 = *(u64 *)(r11 + 8); r0 += r1; exit;" ::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: static subprog takes more than its prototype says")
+__success __retval(7)
+__load_if_JITed()
+__arch_x86_64
+__jited("func #1")
+__jited("...")
+__jited(" movq %r9, %rax")
+__jited(" addq %rdi, %rax")
+__jited(" leave")
+__arch_arm64 __arch_riscv64
+__naked void more_than_prototype(void)
+{
+ asm volatile ("r1 = 1; *(u64 *)(r11 - 8) = 6; call takes_6th_says_1; exit;"
+ ::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: static subprog takes more than its prototype says, no store")
+__failure __msg("callee expects 6 args, stack arg1 is not initialized")
+__naked void more_than_prototype_no_store(void)
+{
+ asm volatile ("r1 = 1; call takes_6th_says_1; exit;" ::: __clobber_all);
+}
+
+/* BTF is the only source for a global subprog: it has no stack arguments. */
+__noinline __used long global_loads_6th(long a)
+{
+ asm volatile ("%0 = *(u64 *)(r11 + 8);" : "=r"(a));
+ return a;
+}
+
+SEC("tc")
+__description("stack_arg: global subprog loads the 6th argument")
+__failure __msg("invalid read from stack arg off 8 depth 0")
+__naked void global_more_than_prototype(void)
+{
+ asm volatile ("r1 = 1; call global_loads_6th; exit;" ::: __clobber_all);
+}
+
SEC("tc")
__description("stack_arg: gap at offset -8, only wrote -16")
__failure
@@ -405,6 +486,16 @@ __naked void stack_arg_write_unused_slot(void)
);
}
+SEC("tc")
+__description("stack_arg: write the slots of the prototype, the callee loads none")
+__success __retval(15)
+__load_if_JITed()
+__naked void stack_arg_write_prototype_slots(void)
+{
+ asm volatile (CALL_TAKES_5_SAYS_7("*(u64 *)(r11 - 8) = 6; *(u64 *)(r11 - 16) = 7;")
+ ::: __clobber_all);
+}
+
SEC("tc")
__description("stack_arg: sequential calls reuse slots")
__failure
@@ -669,6 +760,17 @@ __naked void stack_arg_precision_fallback(void)
#else
+/* No BTF tells about this store. The insn is what JIT has to deal with. */
+SEC("tc")
+__description("stack_arg: r11 where JIT has no stack arguments")
+__failure __msg("JIT does not support stack arguments")
+__arch_s390x __arch_loongarch
+__naked void r11_no_jit_support(void)
+{
+ asm volatile (".8byte %[st_r11]; r0 = 0; exit;"
+ :: __imm_insn(st_r11, BPF_ST_MEM(BPF_DW, 11, -8, 1)) : __clobber_all);
+}
+
SEC("socket")
__description("stack_arg is not supported by compiler or jit, use a dummy test")
__skip("stack_arg is not supported by compiler or jit")
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 08/37] bpf: Don't treat a partial spill as widened in scalar evolution
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (6 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 07/37] selftests/bpf: Test static subprogs that don't take what BTF says Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 09/37] selftests/bpf: Test loops that change a partial spill Alexei Starovoitov
` (28 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
When scalar evolution knows the number of iterations of a loop, the
verifier gives each live register its range over all iterations and
verifies the body one time. If a live register has no formula, the verifier
verifies the loop one iteration at a time.
An unknown scalar needs no formula. A partial spill counts as one, because
4 spilled bytes do not tell what the 8 bytes of the slot are. But the state
has the value of the 4 bytes, and nothing widens it:
int cnt = 0; /* 4 bytes in the stack */
for (i = 0; i < 10; i++)
cnt++;
BPF program is too large. Processed 1000001 insn
cnt is 0, 1, 2, ... at the head, so the verifier cannot prune there. i is a
range, so 'i < 10' does not end the loop either.
Treat such a slot like a register. It has no formula if the state knows
something about the spilled bytes, they are live at the loop head, and the
next value is made from a partial spill that the loop changes or is '?':
can't widen *fp-8, expr is (spill32 (zext32 (+ *fp-8 1)))
A flag or a copy of the counter is not made from the slot: no change there.
'flags |= 1', 'sum += x' and a slot that a callee or a helper writes are
unrolled again, as they are when the number is in a register.
On big-endian a narrow load from the stack was opaque. It is the high bits
of the slot, (>> *fp-8 32): now the expression tells what it is made from.
Nothing changes for a number that is loaded through a pointer to the slot,
for a slot of the caller that a loop in a callee changes, and for a slot
that is unknown at the entry and gets a value in the loop.
clamp_callee_state_pruning and clamp_nested_loop_state_pruning need a loop
that is not unrolled: they use bytes that are zero at first now.
2 progs of scx_pandemonium that count in a 32-bit slot load again. Other
progs of scx have such a loop and load before and after. The verifier
unrolls it now: arena_selftest_btree needs about 18K insns and not 9.6K.
No prog of Cilium, scx, the selftests or Meta's public progs is lost.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 61 ++++++++++++++++++-
.../selftests/bpf/progs/verifier_scev.c | 36 +++++------
2 files changed, 76 insertions(+), 21 deletions(-)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 06dee7e421ea..0f46a85f7648 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -616,9 +616,10 @@ static int mk_fill(struct scev *scev, int id, u8 code)
bool sx = BPF_MODE(code) == BPF_MEMSX;
#ifdef __BIG_ENDIAN
- /* A narrow fill from a DW spill reads high bits on big-endian. */
+ /* A narrow fill reads the high bits of the slot on big-endian. */
if (BPF_SIZE(code) != BPF_DW)
- return OPAQUE_EXPR_ID;
+ return expr2(scev, sx ? BPF_ARSH : BPF_RSH, id,
+ imm_expr(scev, 64 - BITS_PER_BYTE * bpf_size_to_bytes(BPF_SIZE(code))));
#endif
switch (BPF_SIZE(code)) {
@@ -2369,6 +2370,48 @@ static bool is_unbound_scev_reg(struct bpf_verifier_env *env, const struct bpf_f
reg->r64.size == U64_MAX && reg->r32.size == U32_MAX;
}
+/*
+ * scev_reg_for_read() has scev->unknown_reg for a partial spill, because a part does not tell
+ * what the 8 bytes of the slot are. But the state has the value of the spilled bytes.
+ * Does it know something about them? If not, they are their own widening too.
+ */
+static bool is_known_partial_spill(const struct bpf_func_state *st, u32 r)
+{
+ const struct bpf_stack_state *stack = scev_regno_to_stack(st, r);
+ const struct bpf_reg_state *reg;
+ u64 max;
+
+ if (!stack || !bpf_is_spilled_reg(stack) || bpf_spill_size(stack) == BPF_REG_SIZE)
+ return false;
+ reg = &stack->spilled_ptr;
+ max = GENMASK_ULL(bpf_spill_size(stack) * BITS_PER_BYTE - 1, 0);
+ return (reg->var_off.mask & max) != max || reg->r64.size < max || reg->r32.size < max ||
+ reg->step > 1;
+}
+
+/*
+ * Is @id made from a partial spill that the loop changes? '?' can be made from anything.
+ * @self is the slot whose next value is @id, as long as only (any ...) is above @id:
+ * (any self ...) tells that one of the paths does not change the slot.
+ */
+static bool made_from_partial_spill(struct scev *scev, const struct bpf_func_state *st,
+ const struct env *header_env, u32 id, int self, u32 lvl)
+{
+ const struct expr *e = &scev->exprs[id];
+ u32 i;
+
+ if (lvl >= EXPR_STACK_DEPTH || e->op == UNKNOWN)
+ return true;
+ if (e->op == REG)
+ return (int)e->params[0] != self && is_known_partial_spill(st, e->params[0]) &&
+ header_env->reg2expr[e->params[0]] != id;
+ for (i = 0; i < op_params_num(e->op); i++)
+ if (made_from_partial_spill(scev, st, header_env, e->params[i],
+ e->op == ANY ? self : -1, lvl + 1))
+ return true;
+ return false;
+}
+
/* Main logic in verifier.c forbids varying offsets for certain register types. */
static bool is_widenable_reg_type(const struct bpf_reg_state *reg)
{
@@ -2700,8 +2743,20 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
* However, that would require tracking STACK_MISC writes in the
* liveness analysis. Not implemented yet.
*/
- if (is_unbound_scev_reg(env, cur_func, r))
+ if (is_unbound_scev_reg(env, cur_func, r)) {
+ /*
+ * Nothing widens the value that the state has for a partial spill. A
+ * value that is made from itself can be a new one in each iteration:
+ * treat it like a register that has no formula. It's in the half
+ * of the slot that is farther from fp.
+ */
+ if (is_known_partial_spill(cur_func, r) &&
+ bpf_stack_slot_alive(env, st->curframe, (r - MAX_BPF_REG) * 2 + 1) &&
+ made_from_partial_spill(scev, cur_func, header_env,
+ header_env->reg2expr[r], r, 0))
+ goto cant_widen;
continue;
+ }
r_scev = header_env->reg2scev[r];
reg = scev_reg_for_linear(cur_func, r);
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 14e224ae4efb..c891c9b8dfbc 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -520,15 +520,15 @@ __msg("r1 = load_acquire((u8 *)(r10 -8)) {{.*}}; r1 1 -> (zext8 -8)")
__msg("r1 = load_acquire((u16 *)(r10 -8)) {{.*}}; r1 1 -> (zext16 -8)")
__msg("r1 = load_acquire((u32 *)(r10 -8)) {{.*}}; r1 1 -> (zext32 -8)")
#else
-__msg("r1 = *(u32 *)(r10 -8) {{.*}}; r1 1 -> _")
-__msg("r1 = *(u16 *)(r10 -8) {{.*}}; r1 1 -> _")
-__msg("r1 = *(u8 *)(r10 -8) {{.*}}; r1 1 -> _")
-__msg("r1 = *(s32 *)(r10 -8) {{.*}}; r1 1 -> _")
-__msg("r1 = *(s16 *)(r10 -8) {{.*}}; r1 1 -> _")
-__msg("r1 = *(s8 *)(r10 -8) {{.*}}; r1 1 -> _")
-__msg("r1 = load_acquire((u8 *)(r10 -8)) {{.*}}; r1 1 -> _")
-__msg("r1 = load_acquire((u16 *)(r10 -8)) {{.*}}; r1 1 -> _")
-__msg("r1 = load_acquire((u32 *)(r10 -8)) {{.*}}; r1 1 -> _")
+__msg("r1 = *(u32 *)(r10 -8) {{.*}}; r1 1 -> (>> -8 32)")
+__msg("r1 = *(u16 *)(r10 -8) {{.*}}; r1 1 -> (>> -8 48)")
+__msg("r1 = *(u8 *)(r10 -8) {{.*}}; r1 1 -> (>> -8 56)")
+__msg("r1 = *(s32 *)(r10 -8) {{.*}}; r1 1 -> (s>> -8 32)")
+__msg("r1 = *(s16 *)(r10 -8) {{.*}}; r1 1 -> (s>> -8 48)")
+__msg("r1 = *(s8 *)(r10 -8) {{.*}}; r1 1 -> (s>> -8 56)")
+__msg("r1 = load_acquire((u8 *)(r10 -8)) {{.*}}; r1 1 -> (>> -8 56)")
+__msg("r1 = load_acquire((u16 *)(r10 -8)) {{.*}}; r1 1 -> (>> -8 48)")
+__msg("r1 = load_acquire((u32 *)(r10 -8)) {{.*}}; r1 1 -> (>> -8 32)")
#endif
__msg("r2 = load_acquire((u64 *)(r10 -8)) {{.*}}; r2 2 -> -8")
__msg("store_release((u8 *)(r10 -8), r1) {{.*}}; *fp-8 -8 -> (spill8 1)")
@@ -2646,7 +2646,7 @@ __naked __noinline __used static int scrub_slot_callee(void)
}
/*
- * The callee turns the caller's fp-8 into misc, so the states at the call site
+ * The callee turns the caller's fp-4 from zero into misc, so the states at the call site
* differ between the first and the second iteration, while the states inside
* the callee are identical. Verification fails with an infinite loop error if
* states.c:current_loop() does not search upper frames for active loops.
@@ -2661,21 +2661,21 @@ __msg("12: safe")
__naked void clamp_callee_state_pruning(void)
{
asm volatile (" \
- w1 = 1; \
- *(u32 *)(r10 - 8) = r1; \
+ r1 = 0; \
+ *(u32 *)(r10 - 4) = r1; \
r6 = 0; \
1: r1 = r10; \
r1 += -8; \
call scrub_slot_callee; \
r6 += 1; \
if r6 < 10 goto 1b; \
- r0 = *(u32 *)(r10 - 8); \
+ r0 = *(u32 *)(r10 - 4); \
exit; \
" ::: __clobber_all);
}
/*
- * The inner loop turns fp-8 from a spill into misc. On the next outer
+ * The inner loop turns the byte at fp-4 from zero into misc. On the next outer
* iteration, the states inside the inner loop match its first invocation.
*/
SEC("socket")
@@ -2688,16 +2688,16 @@ __msg("loop header at 3, clamping r6 to 0..9 step 1")
__naked void clamp_nested_loop_state_pruning(void)
{
asm volatile (" \
- w1 = 1; \
- *(u32 *)(r10 - 8) = r1; \
+ r1 = 0; \
+ *(u16 *)(r10 - 4) = r1; \
r6 = 0; \
1: r7 = 0; \
-2: *(u32 *)(r10 - 4) = 1; \
+2: *(u8 *)(r10 - 4) = 1; \
r7 += 1; \
if r7 < 3 goto 2b; \
r6 += 1; \
if r6 < 10 goto 1b; \
- r0 = *(u32 *)(r10 - 8); \
+ r0 = *(u16 *)(r10 - 4); \
exit; \
" ::: __clobber_all);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 09/37] selftests/bpf: Test loops that change a partial spill
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (7 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 08/37] bpf: Don't treat a partial spill as widened in scalar evolution Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 10/37] bpf: Let scalar evolution see loads from read-only maps Alexei Starovoitov
` (27 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
- 4 counted loops make a number of 2 or 4 bytes in the stack from itself:
'if (...) cnt++', in a nested loop, two slots that feed each other. The
verifier did not get to the end of these loops. It unrolls them now.
- 5 loops with 1M iterations have a partial spill that does no harm: a
flag, a copy of the counter, a copy of a partial spill that does not
change, a number that the loop writes before it reads it, 32 bits about
which nothing is known. The verifier does not unroll them, as before.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 59 +++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index c891c9b8dfbc..682f22d5fe1f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -2732,6 +2732,65 @@ __naked void clamp_nested_unrolled_loop_state_pruning(void)
" ::: __clobber_all);
}
+/*
+ * A number in a part of the stack slot fp-8. There is no formula for it, but the state has its
+ * value. If the loop makes the next value from it, each iteration has a new one: the loop is
+ * verified one iteration at a time. If not, the body is verified at most two times.
+ */
+#define PARTIAL_SPILL_TEST(name, sz, count, msg, first, body) \
+SEC("socket") \
+__success __log_level(2) \
+__msg("loop header at {{[0-9]+}}, header_count is " #count) \
+__msg("loop header at {{[0-9]+}}, " msg) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "call %[bpf_get_prandom_u32];" \
+ "w0 = w0;" \
+ "r1 = 0;" \
+ "r6 = 0;" \
+ first \
+"1:" body \
+ "r6 += 1;" \
+ "if r6 < " #count " goto 1b;" \
+ "r0 = *(" #sz " *)(r10 - 8);" \
+ "exit;" \
+ :: __imm(bpf_get_prandom_u32) : __clobber_all); \
+}
+
+#define SPILL32_0 "*(u32 *)(r10 - 8) = r1;"
+#define SPILL32_INC "r1 = *(u32 *)(r10 - 8); w1 += 1; *(u32 *)(r10 - 8) = r1;"
+#define NOT_UNROLLED "widening r6 to 0..0xf423f step 1"
+
+/* 'if (...) cnt++' */
+PARTIAL_SPILL_TEST(partial_spill32_inc, u32, 10, "can't widen *fp-8, expr is (any *fp-8 (spill32 ",
+ SPILL32_0, "if r6 & 1 goto 2f;" SPILL32_INC "2:")
+PARTIAL_SPILL_TEST(partial_spill16_inc, u16, 10, "can't widen *fp-8, expr is (spill16 ",
+ "*(u16 *)(r10 - 8) = r1;", "r1 = *(u16 *)(r10 - 8); w1 += 1; *(u16 *)(r10 - 8) = r1;")
+/* A loop in the loop does it */
+PARTIAL_SPILL_TEST(partial_spill32_inc_in_nested_loop, u32, 10, "can't widen *fp-8, expr is ?",
+ SPILL32_0, "r7 = 0; 2:" SPILL32_INC "r7 += 1; if r7 < 3 goto 2b;")
+/* Two slots feed each other */
+PARTIAL_SPILL_TEST(partial_spill32_two_slots, u32, 10, "can't widen *fp-8, expr is (spill32 ",
+ SPILL32_0 "*(u32 *)(r10 - 16) = r1;",
+ "r2 = *(u32 *)(r10 - 16);" SPILL32_INC "*(u32 *)(r10 - 16) = r1; *(u32 *)(r10 - 8) = r2;")
+/* 'if (...) found = 1' */
+PARTIAL_SPILL_TEST(partial_spill32_flag, u32, 1000000, NOT_UNROLLED,
+ SPILL32_0, "if r6 != 77 goto 2f; r1 = 1; *(u32 *)(r10 - 8) = r1; 2:")
+/* 'prev = last; last = i' */
+PARTIAL_SPILL_TEST(partial_spill32_copy_of_counter, u32, 1000000, NOT_UNROLLED,
+ SPILL32_0, "r7 = *(u32 *)(r10 - 8); *(u32 *)(r10 - 8) = r6;")
+/* A copy of a partial spill that the loop does not change */
+PARTIAL_SPILL_TEST(partial_spill32_copy_of_invariant, u32, 1000000, NOT_UNROLLED,
+ SPILL32_0 "r1 = 5; *(u32 *)(r10 - 16) = r1;",
+ "r7 = *(u32 *)(r10 - 8); r1 = *(u32 *)(r10 - 16); *(u32 *)(r10 - 8) = r1;")
+/* The loop writes the partial spill before it reads it. Only the other half of the slot is live. */
+PARTIAL_SPILL_TEST(partial_spill32_dead_at_the_head, u32, 1000000, NOT_UNROLLED,
+ SPILL32_0, "call %[bpf_get_prandom_u32]; *(u32 *)(r10 - 8) = r0; r1 = *(u32 *)(r10 - 4);")
+/* Nothing is known about the 32 bits in the partial spill: they are their own widening */
+PARTIAL_SPILL_TEST(partial_spill32_unknown, u32, 1000000, NOT_UNROLLED,
+ "*(u32 *)(r10 - 8) = r0;", SPILL32_INC)
+
/*
* r7 starts as 1 + 6*k and invariant r8 as 1 + 9*k (0 <= k <= 3).
* The (any r7 r8) union must preserve base 1 and step gcd(6, 9) = 3.
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 10/37] bpf: Let scalar evolution see loads from read-only maps
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (8 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 09/37] selftests/bpf: Test loops that change a partial spill Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 11/37] selftests/bpf: Test scalar evolution with a loop bound in .rodata Alexei Starovoitov
` (26 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
User space often sets the bound of a loop in .rodata before it loads the
prog. The variable is volatile, thus the prog loads it in each iteration:
const volatile int nr_cpus;
for (i = 0; i < nr_cpus; i++)
Scalar evolution calculates how many times a loop runs. Then the verifier
does not have to verify the loop one iteration at a time. But scalar
evolution does not examine maps. For it the result of the load is unknown,
and it cannot calculate the number of iterations:
loop header at 3, non-linear latch at 5
The map is read-only, thus the load always gives the same constant.
bpf_compute_const_regs() runs before scalar evolution and already knows
this constant. Use it. insn_aux_data of an insn describes the registers
before this insn. Thus the result of the load is in insn_aux_data of the
next insn.
loop header at 3, header_count is 10
A loop with a large bound in .rodata reached the limit of 1M insns. Now it
loads, as it does with the same bound in a register.
A counted loop is verified with ranges, and there is no way back to one
iteration at a time: a prog that needs the exact counter in each iteration
(an offset into ctx, two counters with a constant sum) is rejected now, as
it is with the same bound in the insn.
7 loops of scx_cake get a count. Each of them changes one more register
that has no formula, thus the verifier still unrolls them. No prog of
Cilium, scx, the selftests or the public progs of Meta changes.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 0f46a85f7648..8521c34899b0 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -770,6 +770,15 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
switch (mode) {
case BPF_MEM:
case BPF_MEMSX:
+ /*
+ * for (i = 0; i < nr_cpus; i++) with "const volatile int nr_cpus":
+ * bpf_compute_const_regs() knows what is loaded from a read-only map.
+ */
+ if (insn->src_reg != BPF_REG_FP && idx + 1 < env->prog->len &&
+ (env->insn_aux_data[idx + 1].const_reg_mask & BIT(dst))) {
+ imm = env->insn_aux_data[idx + 1].const_reg_vals[dst];
+ return replace_reg(scev, e, dst, imm_expr(scev, imm));
+ }
return maybe_load_fp(scev, e, insn);
default:
goto mark_dst_unknown;
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 11/37] selftests/bpf: Test scalar evolution with a loop bound in .rodata
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (9 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 10/37] bpf: Let scalar evolution see loads from read-only maps Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 12/37] bpf: Helpers don't write through ARG_ANYTHING Alexei Starovoitov
` (25 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The prog loads the bound of the loop in each iteration, as clang does for a
'const volatile' variable. A second jump limits the loop to 16 iterations.
- The bound is 10 and in .rodata. It gives the count:
loop header at 3, header_count is [0..10]
- The bound is in .data. The prog and user space can write to this map,
thus the bound must not count: the second jump does.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 28 +++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 682f22d5fe1f..f33e5b51f7e9 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1528,6 +1528,34 @@ __naked void eval_expr_unallocated_stack_bound(void)
" ::: __clobber_all);
}
+const volatile __u32 rodata_bound = 10;
+volatile __u32 data_bound = 10;
+
+/*
+ * for (i = 0; i < bound; i++): the bound is loaded in each iteration. It gives the count when the
+ * map is read-only. If not, the second jump does.
+ */
+#define BOUND_IN_MAP_TEST(name, bound, count) \
+SEC("socket") \
+__success __retval(10) __log_level(2) \
+__msg("loop header at 3, header_count is [0.." #count "]") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r6 = 0;" \
+ "r7 = %[" #bound "] ll;" \
+"1: r6 += 1;" \
+ "r1 = *(u32 *)(r7 + 0);" \
+ "if r6 >= r1 goto 2f;" \
+ "if r6 < 16 goto 1b;" \
+"2: r0 = r6;" \
+ "exit;" \
+ :: __imm_addr(bound) : __clobber_all); \
+}
+
+BOUND_IN_MAP_TEST(bound_in_rodata, rodata_bound, 10)
+BOUND_IN_MAP_TEST(bound_in_data, data_bound, 16)
+
SEC("xdp")
__success
__log_level(2)
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 12/37] bpf: Helpers don't write through ARG_ANYTHING
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (10 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 11/37] selftests/bpf: Test scalar evolution with a loop bound in .rodata Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 13/37] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING Alexei Starovoitov
` (24 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A loop can keep its counter in a stack slot. Scalar evolution calculates
how many times such a loop runs only if no insn in the loop can write to
the slot through a pointer. For a call of a helper,
bpf_helper_stack_access_bytes() tells what the helper does through each
argument. For an argument type that it does not list, it assumes a read and
a write of any size. ARG_ANYTHING is such a type.
strobemeta_nounroll1 calls bpf_probe_read_user_str() in a loop. r3 is the
address to read from, and its type is ARG_ANYTHING. The liveness analysis
cannot tell if r3 points to the stack. Thus it assumes that the call can
write to all stack slots. One of them is the counter of the outer loop:
loop header at 244, can't compute iterations count
ARG_ANYTHING is a number that the prog selects freely. check_func_arg()
does not examine it, and the main pass keeps what it knows about the stack
over the call. Scalar evolution does not have to assume more.
Clear may_write for ARG_ANYTHING. Only scalar evolution uses may_write.
Keep may_read, because bpf_probe_read_kernel() can read the slot.
loop header at 244, header_count is 13
The outer loops of strobemeta_nounroll1, strobemeta_nounroll2 and
strobemeta_subprogs get a count. Each of them changes one more register
that has no formula, thus the verifier still unrolls them.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b480c5c242af..06e72aaf6769 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -15104,6 +15104,14 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
* analyzed separately.
*/
return (struct arg_access_info) {};
+ case ARG_ANYTHING:
+ /*
+ * A number that the program is free to choose. bpf_probe_read_kernel() and
+ * the like may read what it points to. check_func_arg() does not look at
+ * the number: do_check() keeps what it knows about the stack over the call.
+ */
+ info.may_write = false;
+ return info;
default:
return info;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 13/37] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (11 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 12/37] bpf: Helpers don't write through ARG_ANYTHING Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 14/37] bpf: Fold sums of constants and two shifts by 32 in scalar evolution Alexei Starovoitov
` (23 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The loop keeps its counter in fp-8. In each iteration it calls
bpf_probe_read_kernel() to read from the address of fp-16. This argument is
ARG_ANYTHING. The verifier assumed that the helper can write through it,
and thus can change the counter.
The test expects that scalar evolution calculates the number of iterations:
loop header at 2, header_count is 10
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 27 +++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index f33e5b51f7e9..495792b734f5 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1471,6 +1471,33 @@ __naked void widen_subprog_probe_read(void)
::: __clobber_all);
}
+/*
+ * The address of a stack slot is passed as a number. The helper may read what is there. It doesn't
+ * write, so the counter in the slot next to it is intact.
+ */
+SEC("tracepoint")
+__success __log_level(2)
+__msg("loop header at 2, header_count is 10")
+__naked void stack_address_as_a_number(void)
+{
+ asm volatile (" \
+ *(u64 *)(r10 - 8) = 0; \
+ *(u64 *)(r10 - 16) = 0; \
+1: r1 = r10; \
+ r1 += -24; \
+ r2 = 8; \
+ r3 = r10; \
+ r3 += -16; \
+ call %[bpf_probe_read_kernel]; \
+ r1 = *(u64 *)(r10 - 8); \
+ r1 += 1; \
+ *(u64 *)(r10 - 8) = r1; \
+ if r1 < 10 goto 1b; \
+ r0 = 0; \
+ exit; \
+" :: __imm(bpf_probe_read_kernel) : __clobber_all);
+}
+
/* An induction variable whose range overflows s64 is widened to the full range. */
SEC("socket")
__success
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 14/37] bpf: Fold sums of constants and two shifts by 32 in scalar evolution
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (12 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 13/37] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 15/37] selftests/bpf: Test sums of constants and " Alexei Starovoitov
` (22 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution finds the step of a loop variable when the expression for
one iteration is (+ r imm). Compilers also make other forms of it:
insns in the loop expression it is
----------------------- ---------------------- -------------
r6 += 1; r6 += 1 (+ (+ r6 1) 1) (+ r6 2)
r6 -= 1 (- r6 1) (+ r6 -1)
r6 <<= 32; r6 >>= 32 (>> (<< r6 32) 32) (zext32 r6)
The last one is '(u32)i' for cpu v1 and v2, which have no 32-bit insns.
With these forms the variable has no formula and the loop is not counted.
Each one is an identity for 64-bit numbers that wrap around. The first two
are rules of simplify(). The last one is in transfer(), which sees the
insns: an expression does not tell the width of a shift. (>> x 32) is also
'w6 >>= w2' with 32 in w2, which shifts by 0.
A counted loop is verified with ranges, and there is no way back to one
iteration at a time: a prog with 'r6 -= 1' that needs the exact value of a
variable in each iteration, or after the loop, is rejected now, as it is
with 'r6 += -1'.
state_loop_first_last_equal in verifier_precision examines precision
backtracking in a loop that is not counted. 'r0 += 1; r0 += 1' is a step of
2 now. Replace it with 'r0 += 2; r0 |= 0', which has no formula. expr_chain
needs a deep expression: shift by 16.
The verifier processes 3.8 % fewer insns for the selftests that are built
without 32-bit insns (start_timer: 46008 -> 32), and 5.3 % fewer for those
that gcc builds. No prog is lost.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 26 ++++++++++++++++++-
.../selftests/bpf/progs/verifier_precision.c | 11 ++++----
.../selftests/bpf/progs/verifier_scev.c | 4 +--
3 files changed, 33 insertions(+), 8 deletions(-)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 8521c34899b0..8662205a3b59 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -666,7 +666,7 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
u8 mode = BPF_MODE(insn->code);
u32 dst = insn->dst_reg;
u32 src = insn->src_reg;
- u32 op, sext;
+ u32 op, sext, l, r;
bool need_bswap;
int i, id;
s64 imm;
@@ -726,6 +726,18 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
else
goto mark_dst_unknown;
+ /*
+ * 'r1 <<= 32; r1 >>= 32' is (u32)r1 for cpu v1 and v2. Only here both
+ * shifts are known to be 64-bit ones: the result of a 32-bit shift is
+ * (zext32 ...), and (>> x 32) is 'w1 >>= w2' with 32 in w2 as well,
+ * which shifts by 0.
+ */
+ if (class == BPF_ALU64 && op == BPF_RSH && id >= 0 &&
+ is_imm(scev, id, &imm) && imm == 32 &&
+ is_binop(scev, BPF_LSH, reg2expr[dst], &l, &r) &&
+ is_imm(scev, r, &imm) && imm == 32)
+ return replace_reg(scev, e, dst, expr1(scev, ZEXT32, l));
+
id = expr2(scev, op, reg2expr[dst], id);
if (class == BPF_ALU)
@@ -1485,6 +1497,18 @@ static int simplify(struct scev *scev, u32 id, void *priv)
int a, b, l, r, t, base, slope;
s64 imm1, imm2;
+ /* (- x imm) -> (+ x -imm) */
+ if (is_binop(scev, BPF_SUB, id, &l, &r) &&
+ is_imm(scev, r, &imm1) && imm1 != S64_MIN)
+ return expr2(scev, BPF_ADD, l, imm_expr(scev, -imm1));
+
+ /* (+ (+ x imm) imm) -> (+ x imm) */
+ if (is_add(scev, id, &l, &r) &&
+ is_imm(scev, r, &imm2) &&
+ is_add(scev, l, &a, &b) &&
+ is_imm(scev, b, &imm1))
+ return expr2(scev, BPF_ADD, a, imm_expr(scev, (s64)((u64)imm1 + (u64)imm2)));
+
/* (+ (linear base slope) imm) -> (linear (+ base imm) slope) */
if (is_add(scev, id, &l, &r) &&
is_linear(scev, l, &base, &slope) &&
diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c
index f4459561bf39..71ac80a984d8 100644
--- a/tools/testing/selftests/bpf/progs/verifier_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_precision.c
@@ -159,11 +159,11 @@ __success __log_level(2)
* This is quite fragile: if verifier checkpointing heuristic changes, this
* might need adjusting.
*/
-__msg("2: (07) r0 += 1 ; R0=6")
+__msg("2: (47) r0 |= 0 ; R0=6")
__msg("3: (35) if r0 >= 0xa goto pc+1")
__msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (07) r0 += 1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 1")
+__msg("mark_precise: frame0: regs=r0 stack= before 2: (47) r0 |= 0")
+__msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 2")
__msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4")
__msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1")
__msg("mark_precise: frame0: parent state regs= stack=: R0=P4")
@@ -173,8 +173,9 @@ __naked int state_loop_first_last_equal(void)
asm volatile (
"r0 = 0;"
"l0_%=:"
- "r0 += 1;"
- "r0 += 1;"
+ "r0 += 2;"
+ /* scalar evolution has no formula for it: the loop is not counted */
+ "r0 |= 0;"
/* every few iterations we'll have a checkpoint here with
* first_idx == last_idx, potentially confusing precision
* backtracking logic
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 495792b734f5..e6251c21b474 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -298,8 +298,8 @@ __naked void expr_chain(void)
1: \
if r6 == 2 goto 2f; \
r6 += 1; \
- r6 <<= 32; \
- r6 >>= 32; \
+ r6 <<= 16; \
+ r6 >>= 16; \
w6 += 1; \
r6 = -r6; \
w6 = -w6; \
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 15/37] selftests/bpf: Test sums of constants and shifts by 32 in scalar evolution
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (13 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 14/37] bpf: Fold sums of constants and two shifts by 32 in scalar evolution Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range Alexei Starovoitov
` (21 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
- Two additions, a subtraction, two 64-bit shifts by 32. The tests expect
one step and the count, and run the progs.
- 5 pairs of shifts that are not a zero extension: one of the two is a
32-bit shift by a register that has 32 in it, which shifts by 0, one of
the counts is not 32, the shift to the right is arithmetic. If scalar
evolution takes them for a zero extension, then a read of a register
that is not set becomes dead code and the progs load. They must not
load.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 53 +++++++++++++++++++
1 file changed, 53 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index e6251c21b474..60b823876eaa 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1621,6 +1621,59 @@ __naked void latch_base_differs_from_step(void)
" ::: __clobber_all);
}
+/* Additions and subtractions of constants are one step. Two 64-bit shifts by 32 are (u32)r6. */
+#define ONE_STEP_TEST(name, first, body, jump, expr, ret) \
+SEC("socket") \
+__success __retval(ret) __log_level(2) \
+__msg(" r6=" expr) \
+__msg("loop header at 1, header_count is 8 ") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r6 = " #first ";" \
+"1:" body \
+ "if r6 " jump " goto 1b;" \
+ "r0 = r6;" \
+ "exit;" \
+ ::: __clobber_all); \
+}
+
+ONE_STEP_TEST(add_add, 0, "r6 += 1; r6 += 1;", "< 16", "(+ r6 2) / (linear r6 2)", 16)
+ONE_STEP_TEST(sub, 16, "r6 -= 2;", "> 0", "(+ r6 -2) / (linear r6 -2)", 0)
+ONE_STEP_TEST(two_shifts_by_32, 0, "r6 += 2; r6 <<= 32; r6 >>= 32;", "< 16",
+ "(zext32 (+ r6 2)) / (zext32 (linear r6 2))", 16)
+
+/*
+ * @shifts are not (u32)r7: a 32-bit shift by a register that has 32 in it shifts by 32 & 31 = 0.
+ * If they are taken for (u32)r7 then r7 is in [first, first + 7] and the read of r9 is dead code.
+ */
+#define NO_ZEXT32_TEST(name, first, shifts) \
+SEC("socket") \
+__failure __msg("R9 !read_ok") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r7 = " #first ";" \
+ "r8 = 0;" \
+"1: r1 = r7;" \
+ "r1 -= " #first ";" \
+ "if r1 < 8 goto 2f;" \
+ "r0 = *(u64 *)(r9 + 0);" \
+"2: r7 += 1;" \
+ shifts \
+ "r8 += 1;" \
+ "if r8 < 8 goto 1b;" \
+ "r0 = 0;" \
+ "exit;" \
+ ::: __clobber_all); \
+}
+
+NO_ZEXT32_TEST(rsh32_by_reg_32, 100, "r7 <<= 32; r2 = 32; w7 >>= w2;")
+NO_ZEXT32_TEST(lsh32_by_reg_32, 100, "r2 = 32; w7 <<= w2; r7 >>= 32;")
+NO_ZEXT32_TEST(lsh_by_16, 100, "r7 <<= 16; r7 >>= 32;")
+NO_ZEXT32_TEST(rsh_by_33, 100, "r7 <<= 32; r7 >>= 33;")
+NO_ZEXT32_TEST(arsh_by_32, 0x7ffffffe, "r7 <<= 32; r7 s>>= 32;")
+
SEC("xdp")
__success
__log_level(2)
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (14 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 15/37] selftests/bpf: Test sums of constants and " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-04 0:21 ` sashiko-bot
2026-11-04 7:26 ` Eduard Zingerman
2026-11-03 23:40 ` [PATCH bpf-next 17/37] selftests/bpf: Test " Alexei Starovoitov
` (20 subsequent siblings)
36 siblings, 2 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution calculates the number of iterations of a loop only if the
first value of the variable and the bound in the latch, the jump that
closes the loop, are constants:
for (i = 0; i < 10; i++) header_count is 10
for (i = 0; i < (n & 15); i++) can't compute iterations count
The verifier unrolls the second loop, and it rejects the same loop with
'n & 0x7fffffff': the sequence of jumps is too complex.
The count grows when the first value and the bound move apart. Calculate it
for the two values that are the furthest apart: an upper bound for all.
for (i = 0..16; i < 17; i++) header_count is [0..17]
- There is no count when the value that ends the loop can wrap around. It
is less than a step behind the bound, whatever the first value is.
- '!=' has a count only for a step of 1 or -1, and only if each first
value is in front of each bound.
- The ranges are those of the comparison: unsigned, signed, 32 bits.
Pointers into one object have the order of signed offsets: fp-1 is below
fp+0. Nothing limits the offset of a pointer that is compared only, and
base + offset must not wrap around: there is a count only for offsets
within +-2^29, the limit of reg_not_null(). Constants keep their own
calculation, with unsigned offsets.
- 'first value + step * count' is too wide for the variable that the latch
tests. At the loop head it has its value at the entry, or a value that
the latch let through plus a step: 0..16 in the example, not 0..32.
A counted loop is verified with ranges, and there is no way back to one
iteration at a time. Progs that need exact values are rejected now: a
second variable in a loop that tests at the top gets one step too many,
'n - i - 1' is not known to be positive, an offset into ctx is not a
constant. What compilers make of 'for (i = 0; i < n; i++)' loads. No prog
of Cilium, scx, the selftests or the public progs of Meta is lost.
'i < (n & 0x7fffffff)' loads now and runs as long as 'i < 0x7fffffff',
which loads today.
no_widen_small_spill_counter and uninit_slot_counter count from a stack
slot with unknown contents: they have an upper bound now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 10 +-
kernel/bpf/scev.c | 353 +++++++++++++++++-
kernel/bpf/verifier.c | 5 +-
.../selftests/bpf/progs/verifier_scev.c | 6 +-
4 files changed, 355 insertions(+), 19 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 581458927512..7825a3bc0205 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -348,6 +348,12 @@ struct bpf_loop_iters {
u32 min_header_count; /* min number of times header is executed */
u32 max_header_count; /* max number of times header is executed */
unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+ /*
+ * The variable that the latch tests, or -1, and what it can be at the
+ * loop header: what it is at the entry or what the latch lets through.
+ */
+ int iv_reg;
+ struct cnum64 iv_range;
};
struct loop_stack_entry {
@@ -356,6 +362,8 @@ struct loop_stack_entry {
u32 max_header_count;
u32 loop_id:31;
u32 terminates:1;
+ int iv_reg;
+ struct cnum64 iv_range;
};
#define LOOP_STACK_SIZE 16
@@ -2146,6 +2154,6 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters);
int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 insn_idx,
- struct bpf_verifier_state *entry_state, u32 max_header_count);
+ const struct loop_stack_entry *entry);
#endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 8662205a3b59..eac792dc7b10 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2035,11 +2035,20 @@ static int match_linear_latch(struct bpf_verifier_env *env,
return false;
}
+/* What is known about a scalar or about the offset of a pointer. */
struct scev_value {
- u64 value; /* scalar value or a pointer offset */
+ u64 umin, umax;
+ s64 smin, smax;
+ u32 u32_min, u32_max;
+ s32 s32_min, s32_max;
int ptr_reg; /* SCEV register identifying the pointer origin, or -1. */
};
+static bool scev_value_is_const(const struct scev_value *v)
+{
+ return v->umin == v->umax;
+}
+
static void log_scev_value(struct bpf_verifier_env *env, struct bpf_func_state *st,
const struct scev_value *v)
{
@@ -2049,8 +2058,10 @@ static void log_scev_value(struct bpf_verifier_env *env, struct bpf_func_state *
reg = scev_reg_for_read(env, st, v->ptr_reg);
log_reg(env, v->ptr_reg);
bpf_log(&env->log, " (%s)", reg ? reg_type_str(env, reg->type) : "unavailable");
+ } else if (scev_value_is_const(v)) {
+ bpf_log(&env->log, "scalar %llu", v->umin);
} else {
- bpf_log(&env->log, "scalar %llu", v->value);
+ bpf_log(&env->log, "scalar %llu..%llu", v->umin, v->umax);
}
}
@@ -2091,13 +2102,38 @@ static bool scev_value_add(struct bpf_verifier_env *env, struct bpf_func_state *
const struct scev_value *a, const struct scev_value *b,
struct scev_value *result)
{
+ bool min_wraps, max_wraps;
+
if (a->ptr_reg >= 0 && b->ptr_reg >= 0) {
if (env->log.level & BPF_LOG_LEVEL2)
log_incompatible_scev_values(env, "can't add two pointers", st, a, b);
return false;
}
- result->value = a->value + b->value;
result->ptr_reg = a->ptr_reg >= 0 ? a->ptr_reg : b->ptr_reg;
+
+ /* when both ends wrap around it is a range still */
+ min_wraps = check_add_overflow(a->umin, b->umin, &result->umin);
+ max_wraps = check_add_overflow(a->umax, b->umax, &result->umax);
+ if (min_wraps != max_wraps) {
+ result->umin = 0;
+ result->umax = U64_MAX;
+ }
+ if (check_add_overflow(a->smin, b->smin, &result->smin) ||
+ check_add_overflow(a->smax, b->smax, &result->smax)) {
+ result->smin = S64_MIN;
+ result->smax = S64_MAX;
+ }
+ min_wraps = check_add_overflow(a->u32_min, b->u32_min, &result->u32_min);
+ max_wraps = check_add_overflow(a->u32_max, b->u32_max, &result->u32_max);
+ if (min_wraps != max_wraps) {
+ result->u32_min = 0;
+ result->u32_max = U32_MAX;
+ }
+ if (check_add_overflow(a->s32_min, b->s32_min, &result->s32_min) ||
+ check_add_overflow(a->s32_max, b->s32_max, &result->s32_max)) {
+ result->s32_min = S32_MIN;
+ result->s32_max = S32_MAX;
+ }
return true;
}
@@ -2111,13 +2147,23 @@ static bool eval_expr(struct bpf_verifier_env *env, struct scev *scev,
if (is_reg(scev, id, ®no)) {
reg = scev_reg_for_read(env, st, regno);
- if (!reg || reg->type == NOT_INIT || !tnum_is_const(reg->var_off))
+ if (!reg || reg->type == NOT_INIT)
return false;
- result->value = reg->var_off.value;
+ result->umin = reg_umin(reg);
+ result->umax = reg_umax(reg);
+ result->smin = reg_smin(reg);
+ result->smax = reg_smax(reg);
+ result->u32_min = reg_u32_min(reg);
+ result->u32_max = reg_u32_max(reg);
+ result->s32_min = reg_s32_min(reg);
+ result->s32_max = reg_s32_max(reg);
result->ptr_reg = reg->type == SCALAR_VALUE ? -1 : (int)regno;
return true;
} else if (is_imm(scev, id, &imm)) {
- result->value = imm;
+ result->umin = result->umax = imm;
+ result->smin = result->smax = imm;
+ result->u32_min = result->u32_max = imm;
+ result->s32_min = result->s32_max = imm;
result->ptr_reg = -1;
return true;
} else if (is_add(scev, id, &l, &r) &&
@@ -2259,6 +2305,252 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
return true;
}
+/*
+ * The first value of the induction variable at the latch and the bound it is
+ * compared with, as ranges of unsigned numbers. For a signed comparison the sign
+ * bit is flipped: that keeps the order and the differences.
+ * [lo, hi] is what the variable can be without wrapping around.
+ */
+struct latch_domain {
+ u64 imin, imax;
+ u64 bmin, bmax;
+ u64 lo, hi;
+};
+
+static void latch_domain(const struct linear_latch *latch, const struct scev_value *initial,
+ const struct scev_value *bound, bool sign, struct latch_domain *d)
+{
+ u64 bias;
+
+ if (latch->jmp32) {
+ bias = sign ? 1U << 31 : 0;
+ d->imin = (sign ? (u32)initial->s32_min : initial->u32_min) ^ bias;
+ d->imax = (sign ? (u32)initial->s32_max : initial->u32_max) ^ bias;
+ d->bmin = (sign ? (u32)bound->s32_min : bound->u32_min) ^ bias;
+ d->bmax = (sign ? (u32)bound->s32_max : bound->u32_max) ^ bias;
+ d->lo = 0;
+ d->hi = U32_MAX;
+ return;
+ }
+ bias = sign ? 1ULL << 63 : 0;
+ d->bmin = (sign ? (u64)bound->smin : bound->umin) ^ bias;
+ d->bmax = (sign ? (u64)bound->smax : bound->umax) ^ bias;
+ if (latch->zext32) {
+ /* zero extended, so it is not negative */
+ d->imin = initial->u32_min ^ bias;
+ d->imax = initial->u32_max ^ bias;
+ d->lo = bias;
+ d->hi = bias + U32_MAX;
+ return;
+ }
+ d->imin = (sign ? (u64)initial->smin : initial->umin) ^ bias;
+ d->imax = (sign ? (u64)initial->smax : initial->umax) ^ bias;
+ d->lo = 0;
+ d->hi = U64_MAX;
+ /* It's base + offset that must not wrap around. The limit is that of reg_not_null(). */
+ if (initial->ptr_reg >= 0) {
+ d->lo = bias - BPF_MAX_VAR_OFF;
+ d->hi = bias + BPF_MAX_VAR_OFF;
+ }
+}
+
+/*
+ * '!=' with a step of one ends when the variable gets to the bound, given that
+ * it is on the right side of the bound to begin with.
+ */
+static bool count_ne_ranges(const struct latch_domain *d, bool down, u64 *cnt)
+{
+ if (d->bmin < d->lo || d->bmax > d->hi)
+ return false;
+ if (down ? d->imin < d->bmax : d->imax > d->bmin)
+ return false;
+ *cnt = down ? d->imax - d->bmin : d->bmax - d->imin;
+ return true;
+}
+
+static bool domain_add(const struct latch_domain *d, u64 v, s64 delta, u64 *res)
+{
+ if (delta >= 0 ? (u64)delta > d->hi - v : -(u64)delta > v - d->lo)
+ return false;
+ *res = v + delta;
+ return true;
+}
+
+/*
+ * The number of iterations depends on the first value, so 'first + slope * count'
+ * is far off when the first value is a range. There is a better bound for
+ * the variable that the latch tests. At the loop header it is
+ * - what it is at the entry, or
+ * - a value that the latch let through, @last is the last of them, plus what
+ * is added between the latch and the header.
+ */
+static void latch_iv_range(struct bpf_verifier_env *env, struct bpf_func_state *st,
+ const struct linear_latch *latch, const struct latch_domain *d,
+ bool sign, bool down, u64 step, u64 last, struct bpf_loop_iters *iters)
+{
+ struct scev *scev = env->scev;
+ struct scev_value off, reg_val;
+ u64 bias, min, max, next;
+ u32 l, r, regno;
+ s64 c = 0;
+
+ l = latch->base_expr;
+ if (!is_reg(scev, l, ®no)) {
+ if (!is_add(scev, latch->base_expr, &l, &r) || !is_reg(scev, l, ®no) ||
+ !eval_expr(env, scev, st, r, &off) || !scev_value_is_const(&off) ||
+ off.ptr_reg >= 0)
+ return;
+ c = latch->jmp32 || latch->zext32 ? (s32)off.smin : off.smin;
+ if (c == S64_MIN)
+ return;
+ }
+ /* the lower half says nothing about a value that does not fit in it */
+ if (latch->jmp32 &&
+ (!latch->zext32 || !eval_expr(env, scev, st, l, ®_val) || reg_val.umax > U32_MAX))
+ return;
+ if (last < d->lo || last > d->hi ||
+ !domain_add(d, d->imin, -c, &min) ||
+ !domain_add(d, d->imax, -c, &max) ||
+ !domain_add(d, last, -c, &next) ||
+ !domain_add(d, next, down ? -(s64)step : (s64)step, &next))
+ return;
+ if (down)
+ min = min(min, next);
+ else
+ max = max(max, next);
+
+ if (latch->jmp32) {
+ bias = sign ? 1U << 31 : 0;
+ min ^= bias;
+ max ^= bias;
+ /* negative as s32 is large as zero extended u64 */
+ if (min > max)
+ return;
+ iters->iv_range = cnum64_from_urange(min, max);
+ } else if (sign) {
+ bias = 1ULL << 63;
+ iters->iv_range = cnum64_from_srange(min ^ bias, max ^ bias);
+ } else {
+ iters->iv_range = cnum64_from_urange(min, max);
+ }
+ iters->iv_reg = regno;
+}
+
+/*
+ * The first value or the bound is not a constant. The count is for the pair that is
+ * the furthest apart.
+ */
+static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_func_state *st,
+ const struct linear_latch *latch,
+ const struct scev_value *initial,
+ const struct scev_value *step_val,
+ const struct scev_value *bound,
+ struct bpf_loop_iters *iters, u64 *result)
+{
+ struct scev *scev = env->scev;
+ bool sign = is_signed_op(latch->op), down;
+ struct scev_value reg_val;
+ struct latch_domain d;
+ /* the last value that the latch lets through */
+ u64 step, cnt, last;
+ u32 l, r;
+
+ /*
+ * The jump compares addresses, the ranges are those of the offsets from one base.
+ * fp-1 is below fp+0: whatever the op is, addresses have the order of signed offsets,
+ * as long as base + offset does not wrap around.
+ */
+ if (initial->ptr_reg >= 0)
+ sign = true;
+ /*
+ * (zext32 (linear ...)) is zext32() of something in all iterations but
+ * the first one at the loop header, where it is what the register has.
+ */
+ if (latch->zext32 && !latch->jmp32) {
+ l = latch->base_expr;
+ if (!is_reg(scev, l, &r) && !is_add(scev, latch->base_expr, &l, &r))
+ return false;
+ if (!eval_expr(env, scev, st, l, ®_val) || reg_val.umax > U32_MAX)
+ return false;
+ }
+
+ step = latch->jmp32 || latch->zext32 ? (u64)(s64)(s32)step_val->umin : step_val->umin;
+ if (!step)
+ return false;
+ down = (s64)step < 0;
+ if (down)
+ step = -step;
+
+ latch_domain(latch, initial, bound, sign, &d);
+ /* nothing limits the offset of a pointer that is compared only */
+ if (initial->ptr_reg >= 0 &&
+ (d.imin < d.lo || d.imax > d.hi || d.bmin < d.lo || d.bmax > d.hi))
+ return false;
+
+ /*
+ * The value that ends the loop must not wrap around. It is less than a step
+ * away from the bound whatever the first value is.
+ */
+ switch (latch->op) {
+ case BPF_JLT:
+ case BPF_JSLT:
+ if (down || d.imin >= d.bmax ||
+ check_add_overflow(div64_u64(d.bmax - d.imin - 1, step), 1, &cnt) ||
+ d.bmax > d.hi || step - 1 > d.hi - d.bmax)
+ return false;
+ last = d.bmax - 1;
+ break;
+ case BPF_JLE:
+ case BPF_JSLE:
+ if (down || d.imin > d.bmax ||
+ check_add_overflow(div64_u64(d.bmax - d.imin, step), 1, &cnt) ||
+ d.bmax > d.hi || step > d.hi - d.bmax)
+ return false;
+ last = d.bmax;
+ break;
+ case BPF_JGT:
+ case BPF_JSGT:
+ if (!down || d.imax <= d.bmin ||
+ check_add_overflow(div64_u64(d.imax - d.bmin - 1, step), 1, &cnt) ||
+ d.bmin < d.lo || step - 1 > d.bmin - d.lo)
+ return false;
+ last = d.bmin + 1;
+ break;
+ case BPF_JGE:
+ case BPF_JSGE:
+ if (!down || d.imax < d.bmin ||
+ check_add_overflow(div64_u64(d.imax - d.bmin, step), 1, &cnt) ||
+ d.bmin < d.lo || step > d.bmin - d.lo)
+ return false;
+ last = d.bmin;
+ break;
+ case BPF_JNE:
+ if (step != 1)
+ return false;
+ if (!count_ne_ranges(&d, down, &cnt)) {
+ sign = true;
+ latch_domain(latch, initial, bound, true, &d);
+ if (!count_ne_ranges(&d, down, &cnt))
+ return false;
+ }
+ if (!cnt)
+ return false;
+ last = down ? d.bmin + 1 : d.bmax - 1;
+ break;
+ default:
+ return false;
+ }
+
+ /*
+ * A 32-bit latch sees the lower half of the step. The register at the header can be
+ * a 64-bit variable that moves by all of it: the range is for a step that is the same.
+ */
+ if (step_val->umin == (down ? -step : step))
+ latch_iv_range(env, st, latch, &d, sign, down, step, last, iters);
+ *result = cnt;
+ return true;
+}
+
/*
* Loop with post-condition:
*
@@ -2295,8 +2587,11 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
struct scev *scev = env->scev;
struct scev_value initial_val, step_val, bound_val;
u64 step, bound, initial, total_change, backedge_taken_count;
+ bool konst;
u8 op = latch->op;
+ iters->iv_reg = -1;
+
if (!eval_expr(env, scev, st, latch->base_expr, &initial_val) ||
!eval_expr(env, scev, st, latch->step_expr, &step_val) ||
!eval_expr(env, scev, st, latch->bound_expr, &bound_val))
@@ -2313,9 +2608,22 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
if (!scev_value_compatible(env, st, &initial_val, &bound_val, latch))
return false;
- initial = initial_val.value;
- step = step_val.value;
- bound = bound_val.value;
+ if (!scev_value_is_const(&step_val))
+ return false;
+ konst = scev_value_is_const(&initial_val) && scev_value_is_const(&bound_val);
+ if (!konst) {
+ if (!compute_backedge_taken_max(env, st, latch, &initial_val, &step_val, &bound_val,
+ iters, &backedge_taken_count))
+ return false;
+ /* as below */
+ if (backedge_taken_count >= U32_MAX - 1)
+ return false;
+ goto counted;
+ }
+
+ initial = initial_val.umin;
+ step = step_val.umin;
+ bound = bound_val.umin;
/*
* The code below only handles mixed 64-bit/32-bit comparisons/recurrences
@@ -2336,9 +2644,9 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
step = (s64)(s32)(u32)step;
}
}
- if (latch->zext32 ^ latch->jmp32 && (initial != initial_val.value ||
- bound != bound_val.value ||
- step != step_val.value)) {
+ if (latch->zext32 ^ latch->jmp32 && (initial != initial_val.umin ||
+ bound != bound_val.umin ||
+ step != step_val.umin)) {
if (env->log.level & BPF_LOG_LEVEL2)
bpf_log(&env->log,
"scev: mixed-width latch has operands outside 32-bit ranges\n");
@@ -2368,6 +2676,7 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
}
}
+counted:
/* The header executes once on entry and once for each taken backedge, hence +1. */
iters->max_header_count = backedge_taken_count + 1;
/*
@@ -2377,7 +2686,7 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
* In such case, conservatively assume that non-latch exit can happen
* at any iteration, thus setting minimal number of iterations as 0.
*/
- iters->min_header_count = loop->exits_cnt == 1 ? iters->max_header_count : 0;
+ iters->min_header_count = konst && loop->exits_cnt == 1 ? iters->max_header_count : 0;
return true;
}
@@ -2626,6 +2935,14 @@ static void linear_bounds(struct bpf_reg_state *reg, u32 max_header_count,
out->r32 = r32;
}
+static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
+{
+ struct cnum64 r64 = cnum64_intersect(bounds->r64, iv_range);
+
+ if (!cnum64_is_empty(r64))
+ bounds->r64 = r64;
+}
+
static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 id,
unsigned long *precise_regs)
{
@@ -2720,6 +3037,8 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
}
if (!found || tmp.max_header_count < iters->max_header_count) {
iters->max_header_count = tmp.max_header_count;
+ iters->iv_reg = tmp.iv_reg;
+ iters->iv_range = tmp.iv_range;
best = latch;
}
iters->min_header_count = found ? min(iters->min_header_count, tmp.min_header_count)
@@ -2925,6 +3244,8 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
if (err)
return err;
linear_bounds(reg, iters->max_header_count, &coefs, &bounds);
+ if (r == iters->iv_reg)
+ iv_bounds(&bounds, iters->iv_range);
if (log->level & BPF_LOG_LEVEL2) {
bpf_log(log, "loop header at %d, widening ", insn_idx);
log_reg(env, r);
@@ -2965,9 +3286,11 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
}
int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur_func_state, u32 insn_idx,
- struct bpf_verifier_state *entry_state, u32 max_header_count)
+ const struct loop_stack_entry *entry)
{
+ struct bpf_verifier_state *entry_state = entry->entry_state;
struct bpf_func_state *entry_st = entry_state->frame[entry_state->curframe];
+ u32 max_header_count = entry->max_header_count;
struct bpf_verifier_log *log = &env->log;
struct bpf_reg_state *reg, *entry_reg;
struct scev *scev = env->scev;
@@ -3005,6 +3328,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
continue;
linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
+ if (r == entry->iv_reg)
+ iv_bounds(&bounds, entry->iv_range);
if (log->level & BPF_LOG_LEVEL2) {
bpf_log(log, "loop header at %d, clamping ", insn_idx);
log_reg(env, r);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 06e72aaf6769..94d9a065e62d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -20641,8 +20641,7 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
if (!entry->terminates)
return 0;
- return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry->entry_state,
- entry->max_header_count);
+ return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry);
}
static int handle_loop_entry_exit(struct bpf_verifier_env *env)
@@ -20688,6 +20687,8 @@ static int handle_loop_entry_exit(struct bpf_verifier_env *env)
entry->min_header_count = iters.min_header_count;
entry->max_header_count = iters.max_header_count;
+ entry->iv_reg = iters.iv_reg;
+ entry->iv_range = iters.iv_range;
entry->entry_state = cur->parent;
entry->terminates = true;
err = bpf_widen_scev_regs(env, cur, cur->parent, &iters);
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 60b823876eaa..6b39c76fd37e 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1188,7 +1188,8 @@ __success
__log_level(2)
__msg("scev at header 2:")
__msg_next(" *fp-8=(+ *fp-8 1) / (linear *fp-8 1)")
-__msg("loop header at 2, can't compute iterations count")
+/* whatever is in the other bytes of the slot */
+__msg("loop header at 2, header_count is [0..266] ")
__not_msg("widening")
__naked void no_widen_small_spill_counter(void)
{
@@ -2526,7 +2527,8 @@ __naked void sibling_inner_loops(void)
SEC("socket")
__success
__log_level(2)
-__msg("loop header at 0, can't compute iterations count")
+/* whatever is in the slot */
+__msg("loop header at 0, header_count is [0..11] ")
__naked void uninit_slot_counter(void)
{
asm volatile (" \
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 17/37] selftests/bpf: Test loops whose first value or bound is a range
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (15 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 18/37] bpf: Describe more registers of a counted loop Alexei Starovoitov
` (19 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
RANGE_TEST() makes a loop from a first value, a step and a bound. An atomic
add hides the first value and the bound from the verifier. Jumps put them
into ranges.
- 11 progs with a count: each kind of comparison, the jump before and
after the addition, 64 and 32 bits. Each prog runs with the two values
that are the furthest apart and returns how many times it was at the
loop head. This number must be the upper bound that the verifier
calculates. The tests also expect the range of the variable at the loop
head, at the entry and at the back edge.
- A loop with up to 2^31 iterations, which is too long to unroll.
- 3 progs where 'first value + step' wraps around for a part of the range.
- 13 progs without a count: the value that ends the loop can wrap around,
the variable moves away from the bound, '!=' can step over the bound or
start behind it, a 32-bit variable enters the loop with more than 32
bits.
- 3 progs compare pointers to the stack. fp-1 is below fp+0, but -1 is the
largest offset as an unsigned number. A pointer that has an offset of up
to 2^62, as the first value or as the bound, gives no count.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 157 ++++++++++++++++++
1 file changed, 157 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 6b39c76fd37e..6777d545ec1d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1012,6 +1012,131 @@ POST_COND_TEST32(post32_slt_wrap, "s<", INT32_MAX - 3, 2, INT3
POST_COND_TEST32(post32_sgt_wrap, "s>", (__u32)INT32_MIN + 3, -2u, (__u32)INT32_MIN, "can't compute iterations count")
POST_COND_TEST32(post32_lt_wrap, "<", -4, 2, -1, "can't compute iterations count")
+/*
+ * The first value of the counter or the bound of the loop is a range: the verifier sees only the
+ * jumps that put r6 into [first_min, first_max] and r7 into [bound_min, bound_max]. An atomic add
+ * hides what they are at run time: the two values that are the furthest apart.
+ * r9 counts how many times the header is executed, and ends the loops that have no count.
+ */
+#define RANGE_TEST(name, sign, first_min, first_max, _step, bound_min, bound_max, loop, expect) \
+SEC("socket") \
+__success __log_level(2) \
+expect \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r1 = %[first] ll;" \
+ "*(u64 *)(r10 - 8) = 0;" \
+ "lock *(u64 *)(r10 - 8) += r1;" \
+ "r6 = *(u64 *)(r10 - 8);" \
+ "r1 = %[bound] ll;" \
+ "*(u64 *)(r10 - 8) = 0;" \
+ "lock *(u64 *)(r10 - 8) += r1;" \
+ "r7 = *(u64 *)(r10 - 8);" \
+ "r9 = 0;" \
+ "r1 = %[fmin] ll; if r6 " sign "< r1 goto 2f;" \
+ "r1 = %[fmax] ll; if r6 " sign "> r1 goto 2f;" \
+ "r1 = %[bmin] ll; if r7 " sign "< r1 goto 2f;" \
+ "r1 = %[bmax] ll; if r7 " sign "> r1 goto 2f;" \
+ loop \
+ "2: r0 = r9;" \
+ "exit;" \
+ : \
+ : __imm_const(first, (__s64)(_step) > 0 ? first_min : first_max), \
+ __imm_const(bound, (__s64)(_step) > 0 ? bound_max : bound_min), \
+ __imm_const(fmin, first_min), __imm_const(fmax, first_max), \
+ __imm_const(bmin, bound_min), __imm_const(bmax, bound_max), \
+ __imm_const(step, _step) \
+ : __clobber_all); \
+}
+
+#define LOOP_HEAD "1: r8 = %[step] ll;" LIMIT_ITERATIONS
+#define PRE(exit_op) LOOP_HEAD "if r6 " exit_op " r7 goto 2f; r6 += r8; goto 1b;"
+#define POST(add, jump) LOOP_HEAD add "; if " jump " goto 1b;"
+
+/* 'count' is what the verifier has and what r9 is at run time. 'range' is r6 at the header. */
+#define COUNTED(count, range) \
+ __retval(count) \
+ __msg("loop header at {{[0-9]+}}, header_count is [0.." #count "] ") \
+ __msg("loop header at {{[0-9]+}}, widening r6 to " range " ") \
+ __msg("loop header at {{[0-9]+}}, clamping r6 to " range " ")
+#define NO_COUNT \
+ __msg("loop header at {{[0-9]+}}, can't compute iterations count") \
+ __not_msg("header_count is")
+
+#define M64 ITER_U64_MAX
+#define M32 ITER_U32_MAX
+/*
+ * Example: range_lt
+ *
+ * u64 r6 = 1..4, r7 = 8..12;
+ * while (r6 < r7) // r6 ∈ [1,3,...,11,13]
+ * r6 += 2;
+ */
+/* name sign first step bound loop expected */
+RANGE_TEST(range_lt, "", 1, 4, 2, 8, 12, PRE(">="), COUNTED(7, "1..13"))
+RANGE_TEST(range_le, "", 0, 2, 2, 6, 9, PRE(">"), COUNTED(6, "0..11"))
+RANGE_TEST(range_gt, "", 10, 14, -2, 3, 5, PRE("<="), COUNTED(7, "2..14"))
+RANGE_TEST(range_ge, "", 10, 14, -2, 4, 5, PRE("<"), COUNTED(7, "2..14"))
+RANGE_TEST(range_slt, "s", -5, 5, 2, 5, 7, PRE("s>="), COUNTED(7, "-5..8"))
+RANGE_TEST(range_ne_up, "", 0, 3, 1, 3, 6, PRE("=="), COUNTED(7, "0..6"))
+RANGE_TEST(range_ne_down, "", 8, 10, -1, 4, 8, PRE("=="), COUNTED(7, "4..10"))
+/* the order of unsigned numbers says nothing about -1..3, the order of signed numbers does */
+RANGE_TEST(range_ne_signed, "s", -3, -1, 1, -1, 3, PRE("=="), COUNTED(7, "-3..3"))
+/* The jump tests r6 after the addition. At the header r6 is a step behind. */
+RANGE_TEST(range_post_lt, "", 3, 10, 2, 14, 15, POST("r6 += r8", "r6 < r7"),
+ COUNTED(6, "3..14"))
+/* 32 bits. r6 is negative at the entry and zero extended after that. */
+RANGE_TEST(range32_slt, "s", -5, 5, 2, 7, 9, POST("w6 += w8", "w6 s< w7"),
+ COUNTED(7, "-5..0xffffffff"))
+/* Only the jump has 32 bits. The low half says nothing about r6: it's 'first + step * count'. */
+RANGE_TEST(range32_jump, "", 3, 10, 2, 13, 15, POST("r6 += r8", "w6 < w7"),
+ COUNTED(6, "3..20"))
+
+#define COUNTED_2G(range) \
+ __msg("loop header at {{[0-9]+}}, header_count is [0..2147483{{[0-9]+}}] ") \
+ __msg("loop header at {{[0-9]+}}, widening r6 to " range " ")
+/* Too long to unroll: only the count ends the loop for the verifier */
+RANGE_TEST(range_long, "", 0, 0, 1, 0, 0x7fffffff,
+ "1: r8 = %[step] ll; if r6 >= r7 goto 2f; r6 += r8; goto 1b;", COUNTED_2G("0..0x7fffffff"))
+/* 'first + step' wraps around for a part of the range: the loop starts anywhere */
+RANGE_TEST(range_first_wraps_s64, "s", ITER_S64_MAX - 5, ITER_S64_MAX - 1, 2,
+ ITER_S64_MAX - 1, ITER_S64_MAX - 1, POST("r6 += r8", "r6 s< r7"), NO_COUNT)
+RANGE_TEST(range_first_wraps_u32, "", 0, 3, -2, 5, 5, POST("w6 += w8", "w6 > w7"),
+ COUNTED_2G("0..0xffffffff"))
+RANGE_TEST(range_first_wraps_s32, "", INT32_MAX - 5, INT32_MAX - 1, 2, INT32_MAX - 1, INT32_MAX - 1,
+ POST("w6 += w8", "w6 s< w7"), COUNTED_2G("0..0xffffffff"))
+
+/*
+ * The value that ends the loop can wrap around.
+ *
+ * Example: range_lt_wrap
+ *
+ * u64 M = U64_MAX, r6 = M-9..M-5, r7 = M-3..M;
+ * while (r6 < r7) // M-1 < M, and the next value is 1
+ * r6 += 2;
+ */
+RANGE_TEST(range_lt_wrap, "", M64 - 9, M64 - 5, 2, M64 - 3, M64, PRE(">="), NO_COUNT)
+RANGE_TEST(range_le_wrap, "", M64 - 9, M64 - 5, 2, M64 - 3, M64 - 1, PRE(">"), NO_COUNT)
+RANGE_TEST(range_gt_wrap, "", 5, 9, -2, 0, 3, PRE("<="), NO_COUNT)
+RANGE_TEST(range_ge_wrap, "", 5, 9, -2, 1, 3, PRE("<"), NO_COUNT)
+/* a 32-bit addition wraps around at 2^32, also under a 64-bit jump */
+RANGE_TEST(range32_add_wrap, "", M32 - 9, M32 - 5, 2, M32 - 3, M32,
+ POST("w6 += w8", "r6 < r7"), NO_COUNT)
+RANGE_TEST(range32_ne_far, "", M32 - 5, M32 - 3, 1, M32 - 1, M32 + 3,
+ POST("w6 += w8", "r6 != r7"), NO_COUNT)
+/* the first jump tests what enters the loop, and that is not zero extended */
+RANGE_TEST(range32_first_is_wide, "", 0, M32 + 9, 2, 15, 20,
+ LOOP_HEAD "if r6 >= r7 goto 2f; w6 += w8; goto 1b;", NO_COUNT)
+/* The counter moves away from the exit. */
+RANGE_TEST(range_lt_dir, "", 3, 10, -2, 15, 20, PRE(">="), NO_COUNT)
+RANGE_TEST(range_le_dir, "", 3, 10, -2, 15, 20, PRE(">"), NO_COUNT)
+RANGE_TEST(range_gt_dir, "", 10, 20, 2, 3, 5, PRE("<="), NO_COUNT)
+RANGE_TEST(range_ge_dir, "", 10, 20, 2, 3, 5, PRE("<"), NO_COUNT)
+/* '!=' can step over the bound, or start behind it. */
+RANGE_TEST(range_ne_step2, "", 0, 3, 2, 3, 8, PRE("=="), NO_COUNT)
+RANGE_TEST(range_ne_behind, "", 0, 4, 1, 3, 8, PRE("=="), NO_COUNT)
+
SEC("xdp")
__success
__log_level(2)
@@ -3360,6 +3485,38 @@ __naked void ptr_stack_other_frame(void)
" ::: __clobber_all);
}
+/*
+ * r6 is fp-63 + r0. The jump compares addresses. fp-1 is below fp+0: they have the order of signed
+ * offsets, as long as base + offset does not wrap around. Nothing limits the offset of a pointer
+ * that is compared only: fp-63 + 0..2^62 can be a small address.
+ */
+#define PTR_RANGE_TEST(name, offset, first, step, op, bound, expect) \
+SEC("socket") \
+__success __log_level(2) \
+expect \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "call %[bpf_get_prandom_u32];" \
+ offset \
+ "r6 = r10;" \
+ "r6 += -63;" \
+ "r6 += r0;" \
+ "r7 = r10;" \
+ "r9 = 0;" \
+"1:" LIMIT_ITERATIONS \
+ first " += " #step ";" \
+ "if " first " " op " " bound " goto 1b;" \
+"2: r0 = 0;" \
+ "exit;" \
+ :: __imm(bpf_get_prandom_u32) : __clobber_all); \
+}
+
+PTR_RANGE_TEST(ptr_first_is_below_or_at_fp, "r0 &= 63;", "r6", 1, "<=", "r7",
+ __msg("loop header at {{[0-9]+}}, header_count is [0..64] "))
+PTR_RANGE_TEST(ptr_first_offset_is_wide, "r0 >>= 2;", "r6", 1, "<", "r7", NO_COUNT)
+PTR_RANGE_TEST(ptr_bound_offset_is_wide, "r0 >>= 2;", "r7", -1, ">", "r6", NO_COUNT)
+
SEC("xdp")
__success
__log_level(2)
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 18/37] bpf: Describe more registers of a counted loop
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (16 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 17/37] selftests/bpf: Test " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 19/37] selftests/bpf: Test " Alexei Starovoitov
` (18 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier verifies the body of a counted loop one time only if scalar
evolution has a formula for each live register that the loop changes. One
register without a formula makes the verifier unroll the loop.
what the loop does with a live register the register at the head
--- ------------------------------------------ ------------------------
adds a constant range from the count
sets it to one of a few invariant values union of the values
new only clears bits of [0, max]: no change
'idx &= ~(1 << order)', zero extension
new moves a pointer to arena no change
new sets it to the same constant each time no change
something else no formula
In the new rows the state that enters the loop covers all iterations:
- A number that loses bits does not grow, and a bit that is 0 stays 0. The
rule is only for the range [0, max]: 64 or [64, 127] can become 0. The
register is not equal to its copies any more, thus it loses its id.
- The verifier does not track the offset of a pointer to arena. So
bpf_clamp_scev_regs() must not give it a range at the back edge: it now
sets a range only for the types that get one at the entry.
- bpf_compute_const_regs() knows the registers that have one constant at
an insn on all paths: 'r7 &= 31' with 17 in r7. It recorded constants
that fit in 32 bits. Record that there is a constant for the others too.
The verifier does not rely on the new rows. It compares the state that
returns to the loop head with the state at the head. If they do not agree
it verifies the body again from the state that returned. The counter has
its range then: a loop that makes a number of a pointer to arena and
changes that number in each iteration no longer loads.
'x >> e', 'x / e' and 'x % e' get no rule: iterations one by one tell more
about a number that is halved 8 times. That is true for '&' too: a prog
that needs 'x &= 15' to be below 16 from the second iteration on is
rejected now. But '&' is what the loops in real progs do.
The verifier processes about 2 % fewer insns for the selftests. The progs
that change use arena: big_alloc2 goes from 2956 to 216, some get more
insns, at most 270. 3 progs of bpfjailer need 17 % fewer insns. No prog of
Cilium, scx, the selftests or the public progs of Meta is lost.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 4 ++
kernel/bpf/const_fold.c | 11 ++--
kernel/bpf/scev.c | 109 ++++++++++++++++++++++++++++++++++-
3 files changed, 119 insertions(+), 5 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7825a3bc0205..724f3b60130f 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -348,6 +348,8 @@ struct bpf_loop_iters {
u32 min_header_count; /* min number of times header is executed */
u32 max_header_count; /* max number of times header is executed */
unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+ /* [0, max] at the entry, and the loop only clears bits */
+ unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
/*
* The variable that the latch tests, or -1, and what it can be at the
* loop header: what it is at the entry or what the latch lets through.
@@ -851,12 +853,14 @@ struct bpf_insn_aux_data {
/*
* Bitmask of R0-R9 that hold known values at this instruction.
* const_reg_mask: scalar constants that fit in 32 bits.
+ * const_reg_wide_mask: scalar constants that don't, there is no val.
* const_reg_map_mask: map pointers, val is map_index into used_maps[].
* const_reg_subprog_mask: subprog pointers, val is subprog number.
* const_reg_vals[i] holds the 32-bit value for register i.
* Populated by compute_const_regs() pre-pass.
*/
u16 const_reg_mask;
+ u16 const_reg_wide_mask;
u16 const_reg_map_mask;
u16 const_reg_subprog_mask;
u32 const_reg_vals[10];
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index f9edfde965dc..ce1bd72cb6c8 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -8,7 +8,7 @@
* instruction. Tracks 64-bit constant values in R0-R9 through the program,
* using a fixed-point iteration in reverse postorder. Records which registers
* hold known constants and their values in
- * env->insn_aux_data[].{const_reg_mask, const_reg_vals}.
+ * env->insn_aux_data[].{const_reg_mask, const_reg_wide_mask, const_reg_vals}.
*/
enum const_arg_state {
@@ -276,9 +276,9 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
if (changed)
goto redo;
- /* Save computed constants into insn_aux[] if they fit into 32-bit */
+ /* Save computed constants into insn_aux[]: the value if it fits into 32-bit */
for (i = 0; i < insn_cnt; i++) {
- u16 mask = 0, map_mask = 0, subprog_mask = 0;
+ u16 mask = 0, wide_mask = 0, map_mask = 0, subprog_mask = 0;
struct bpf_insn_aux_data *aux = &insn_aux[i];
struct const_arg_info *ci = ci_in[i];
@@ -289,8 +289,10 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
case CONST_ARG_CONST: {
u64 val = c->val;
- if (val != (u32)val)
+ if (val != (u32)val) {
+ wide_mask |= BIT(r);
break;
+ }
mask |= BIT(r);
aux->const_reg_vals[r] = val;
break;
@@ -308,6 +310,7 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
}
}
aux->const_reg_mask = mask;
+ aux->const_reg_wide_mask = wide_mask;
aux->const_reg_map_mask = map_mask;
aux->const_reg_subprog_mask = subprog_mask;
}
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index eac792dc7b10..0d49c9021e30 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2754,6 +2754,21 @@ static bool made_from_partial_spill(struct scev *scev, const struct bpf_func_sta
return false;
}
+/*
+ * bpf_compute_const_regs() has @r as a constant at @insn_idx, the head of a loop: it's the same in
+ * every iteration. do_check() takes a value that a range adds to it, or to what is computed
+ * from it, for a bug.
+ * That pass knows no types: arithmetic on a pointer to arena runs in 32 bits, and a pointer and a
+ * number can have one constant. As for a pointer to arena below, this is not relied on.
+ */
+static bool is_const_scev_reg(struct bpf_verifier_env *env, int insn_idx, u32 r)
+{
+ struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
+
+ return r < ARRAY_SIZE(aux->const_reg_vals) &&
+ ((aux->const_reg_mask | aux->const_reg_wide_mask) & BIT(r));
+}
+
/* Main logic in verifier.c forbids varying offsets for certain register types. */
static bool is_widenable_reg_type(const struct bpf_reg_state *reg)
{
@@ -2967,6 +2982,45 @@ static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state
return 0;
}
+/*
+ * Does @id have no bit that register @r doesn't have at the head of the loop? 'x & e' and the
+ * low bits of x lose bits only. 'x >> e', 'x / e' and 'x % e' are not greater than x as well, but
+ * iterations one by one tell more about them: a number that is halved 8 times.
+ */
+static bool expr_loses_bits_only(struct scev *scev, u32 id, u32 r, int depth)
+{
+ struct expr *e = &scev->exprs[id];
+ u32 regno;
+
+ if (is_reg(scev, id, ®no))
+ return regno == r;
+ if (depth > 8)
+ return false;
+ switch (e->op) {
+ case BPF_AND:
+ return expr_loses_bits_only(scev, e->params[0], r, depth + 1) ||
+ expr_loses_bits_only(scev, e->params[1], r, depth + 1);
+ case ANY:
+ return expr_loses_bits_only(scev, e->params[0], r, depth + 1) &&
+ expr_loses_bits_only(scev, e->params[1], r, depth + 1);
+ case ZEXT8:
+ case ZEXT16:
+ case ZEXT32:
+ return expr_loses_bits_only(scev, e->params[0], r, depth + 1);
+ default:
+ return false;
+ }
+}
+
+/* The ranges of @reg are [0, max]. So no bit is known to be 1. */
+static bool reg_is_zero_to_max(const struct bpf_reg_state *reg)
+{
+ struct cnum64 r64 = cnum64_from_urange(0, reg_umax(reg));
+
+ return cnum64_is_subset(reg->r64, r64) &&
+ cnum32_is_subset(reg->r32, cnum32_from_cnum64(r64)) && reg->step <= 1;
+}
+
int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct bpf_loop_iters *iters)
{
@@ -3077,6 +3131,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
if (err)
return err;
+ bitmap_zero(iters->mask_regs, REGS_NUM);
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
struct bpf_reg_state *reg;
@@ -3110,8 +3165,19 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
continue;
}
+ if (is_const_scev_reg(env, insn_idx, r))
+ continue;
+
r_scev = header_env->reg2scev[r];
reg = scev_reg_for_linear(cur_func, r);
+ /*
+ * The offset of a pointer to arena is not tracked: what enters the loop stands
+ * for all iterations. If the loop makes something else of it the state that
+ * returns to the head is not covered by the state at the head, and the body is
+ * verified again from that state.
+ */
+ if (reg && reg->type == PTR_TO_ARENA)
+ continue;
/* If SCEV for r is (linear <reg> <slope>) */
if (reg && is_simple_linear(scev, r_scev, &coefs) && is_widenable_reg_type(reg)) {
/* 32-bit ALU operations on pointers convert them to scalars */
@@ -3137,6 +3203,18 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
}
continue;
}
+
+ /*
+ * [0, max] that loses bits is in [0, max], a bit that is 0 stays 0: what enters
+ * the loop stands for all iterations as it is. 64 or [64, 127] does not: it
+ * can become 0.
+ */
+ if (reg && reg->type == SCALAR_VALUE && !test_bit(r, loop->store_base_regs) &&
+ reg_is_zero_to_max(reg) &&
+ expr_loses_bits_only(scev, header_env->reg2expr[r], r, 0)) {
+ __set_bit(r, iters->mask_regs);
+ continue;
+ }
cant_widen:
if (log->level & BPF_LOG_LEVEL2) {
r_expr = header_env->reg2expr[r];
@@ -3234,8 +3312,29 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
- if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r))
+ if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r) ||
+ is_const_scev_reg(env, insn_idx, r))
+ continue;
+
+ reg = scev_reg_for_linear(cur_func, r);
+ if (reg && reg->type == PTR_TO_ARENA)
+ continue;
+
+ if (test_bit(r, iters->mask_regs)) {
+ /* The range stays. It's no longer equal to the registers with its id. */
+ err = scev_reg_for_write(env, cur_func, r, ®);
+ if (err)
+ return err;
+ if (log->level & BPF_LOG_LEVEL2) {
+ bpf_log(log, "loop header at %d, ", insn_idx);
+ log_reg(env, r);
+ bpf_log(log, " stays in 0..%llu, it loses bits only\n",
+ reg_umax(reg));
+ }
+ scratch_widened_reg_id(env, reg);
+ mark_scev_reg_scratched(env, r);
continue;
+ }
r_expr = header_env->reg2scev[r];
/* If SCEV for r is (linear <reg> <slope>)*/
@@ -3327,6 +3426,14 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
if (reg->type != entry_reg->type)
continue;
+ /*
+ * bpf_compute_loop_iters() gives no range to a pointer to arena that moves, and
+ * it does not look at dead registers. Set a range only where it would.
+ */
+ if (!is_widenable_reg_type(entry_reg) ||
+ (coefs.zext32 && entry_reg->type != SCALAR_VALUE))
+ continue;
+
linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
if (r == entry->iv_reg)
iv_bounds(&bounds, entry->iv_range);
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 19/37] selftests/bpf: Test more registers of a counted loop
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (17 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 18/37] bpf: Describe more registers of a counted loop Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 20/37] bpf: Compute scev for a loop that has more than 256 exits Alexei Starovoitov
` (17 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
- A register in [0, max] loses bits: '&' with the register on the left and
on the right, a zero extension on one of two paths, a register that is 0
or 64. The tests expect 'it loses bits only'.
- No rule, the verifier unrolls the loop: a register that enters the loop
as [64, 127], shifts to the right and to the left.
- A register has the same constant in each iteration, in 32 or in 64 bits.
- A pointer to arena moves in a loop with 100 iterations.
- The loop makes a number of a pointer to arena. The prog must not load.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 85 +++++++++++++++++++
1 file changed, 85 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 6777d545ec1d..c91d178a1aed 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -3063,6 +3063,91 @@ __naked void conditional_scalar_assignment_gcd(void)
: __clobber_all);
}
+/*
+ * Registers that have no linear formula, and the state that enters the loop stands for all
+ * iterations all the same. r7 is what @init makes of an unknown number, r8 is unknown.
+ */
+#define NOT_LINEAR_TEST(name, init, op, expect) \
+SEC("socket") \
+__success __log_level(2) \
+expect \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "call %[bpf_get_prandom_u32];" \
+ "r8 = r0;" \
+ "call %[bpf_get_prandom_u32];" \
+ "r7 = r0;" \
+ init \
+ "r6 = 0;" \
+"1:" op \
+ "r6 += 1;" \
+ "if r6 < 8 goto 1b;" \
+ "r0 = r7;" \
+ "r0 &= 1;" \
+ "exit;" \
+ :: __imm(bpf_get_prandom_u32) : __clobber_all); \
+}
+
+/* [0, max] that loses bits is in [0, max], and a bit that is 0 stays 0 */
+#define LOSES_BITS(max) __msg("r7 stays in 0.." #max ", it loses bits only")
+#define UNROLLED(expr) __msg("can't widen r7, expr is " expr) __not_msg("it loses bits only")
+
+NOT_LINEAR_TEST(and_clears_a_bit, "r7 &= 255;", "r1 = 1; r1 <<= r6; r1 ^= -1; r7 &= r1;",
+ LOSES_BITS(255))
+NOT_LINEAR_TEST(and_on_the_right, "r7 &= 255;", "r1 = r8; r1 &= r7; r7 = r1;", LOSES_BITS(255))
+NOT_LINEAR_TEST(and_0_or_64, "r7 &= 64;", "r7 &= r8;", LOSES_BITS(64))
+NOT_LINEAR_TEST(zext32_or_nothing, "r7 >>= 8;", "if r8 == 0 goto +1; w7 = w7;",
+ LOSES_BITS(72057594037927935))
+/* 64..127 can become 0 */
+NOT_LINEAR_TEST(and_64_to_127, "r7 &= 63; r7 |= 64;", "r7 &= 127;", UNROLLED("(& r7 127)"))
+/* '>>' does not make a number greater, but iterations one by one tell more: 0 after 8 shifts */
+NOT_LINEAR_TEST(rsh_on_one_path, "r7 &= 255;", "if r8 == 0 goto +1; r7 >>= 1;",
+ UNROLLED("(any r7 (>> r7 1))"))
+NOT_LINEAR_TEST(rsh_then_and, "r7 &= 255;", "r7 >>= 1; r7 &= r8;", UNROLLED("(& (>> r7 1) r8)"))
+NOT_LINEAR_TEST(lsh_grows, "r7 &= 15;", "r7 <<= 1;", UNROLLED("(<< r7 1)"))
+
+/* bpf_compute_const_regs() knows that r7 is the same constant in each iteration */
+#define NOT_CHANGED __msg("widening r6 to 0..7")
+NOT_LINEAR_TEST(same_constant, "r7 = 17;", "r7 &= 31;", NOT_CHANGED)
+NOT_LINEAR_TEST(same_wide_constant, "r7 = 0xffffffff00000011 ll;", "r7 &= -239;", NOT_CHANGED)
+
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+/* The offset of a pointer to arena is not tracked */
+#define ARENA_TEST(name, op, expect) \
+SEC("syscall") \
+__log_level(2) \
+expect \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r1 = %[arena] ll;" \
+ "r6 = 0;" \
+ "r6 = addr_space_cast(r6, 0, 1);" \
+ "r7 = 0;" \
+"1: *(u64 *)(r6 + 0) = r7;" \
+ op \
+ "r7 += 1;" \
+ "if r7 < 100 goto 1b;" \
+ "r0 = 0;" \
+ "exit;" \
+ :: __imm_addr(arena) : __clobber_all); \
+}
+
+#define NOT_UNROLLED100 __msg("loop header at 5, widening r7 to 0..99 step 1")
+/* A loop that moves the pointer is not unrolled */
+ARENA_TEST(arena_pointer_moves, "r6 += 8;", __success NOT_UNROLLED100)
+/* The loop makes a number of it: the state that returns to the loop header tells */
+ARENA_TEST(arena_pointer_becomes_a_number, "if r7 != 5 goto +1; r6 = 4096;",
+ __failure NOT_UNROLLED100 __msg("R6 invalid mem access 'scalar'"))
+#endif
+
/*
* r7 = ctx->data;
* r8 = r7
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 20/37] bpf: Compute scev for a loop that has more than 256 exits
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (18 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 19/37] selftests/bpf: Test " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 21/37] selftests/bpf: Test a loop with 301 exits Alexei Starovoitov
` (16 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
clang unrolls the loop of pyperf600 in part: 4 iterations with 150 copies
of the body. Each copy has two 'break' and a 'return', thus the loop has
451 exits. The list of exits of a loop holds 256. When the list is full
scalar evolution gives up on the loop:
loop header at 164, unsupported loop: too many exits
The complete list is for the loop around. Scalar evolution takes a nested
loop for one insn, and the exits are the successors of this insn. No result
for the loop itself depends on the list: the number of iterations comes
from the jumps that dominate the back edge. mark_latches() prepares the
exits for the loop around, and compute_max_iters() asks if there is only
one exit.
Keep the two apart:
exits_overflow the list of this loop is cut
nested_exits_overflow the list of a loop in this one is cut, and the
list of this loop can lack exits too
Only the second one makes scalar evolution give up.
loop header at 164, header_count is [0..4]
A loop around a loop with more than 256 exits is not analysed, as before.
pyperf600 and pyperf180 have 5 such loops each. They get a count. Each of
them changes one more register that has no formula, thus the verifier still
unrolls them.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 3 +++
kernel/bpf/loops.c | 6 ++++--
kernel/bpf/scev.c | 15 +++++++++++----
3 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 724f3b60130f..95e0d7face11 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -746,7 +746,10 @@ struct bpf_loop {
int exits_cnt;
bool irreducible;
bool backedges_overflow;
+ /* @exits is cut at MAX_LOOP_EXITS. Only a loop around this one needs all of them. */
bool exits_overflow;
+ /* @exits of a loop in this one is cut. Then @exits of this loop can lack exits too. */
+ bool nested_exits_overflow;
/*
* Loop-entry registers used to compute stack addresses for loads,
* stores and calls requiring fixed stack offsets, in this and nested
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 2568079c2945..e584549bd208 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -555,7 +555,7 @@ int bpf_compute_loops(struct bpf_verifier_env *env)
* s: ...
* }
*/
- /* Enclosing loops inherit the overflow, see below */
+ /* Enclosing loops get nested_exits_overflow, see below */
if (aux[t].loop->exits_overflow)
break;
err = add_exit(aux[t].loop, i, s);
@@ -572,7 +572,7 @@ int bpf_compute_loops(struct bpf_verifier_env *env)
if (!loop || !loop->exits_overflow)
continue;
for (t = aux[i].loop_header; t >= 0; t = aux[t].loop_header)
- aux[t].loop->exits_overflow = true;
+ aux[t].loop->nested_exits_overflow = true;
}
if (env->log.level & BPF_LOG_LEVEL2) {
@@ -587,6 +587,8 @@ int bpf_compute_loops(struct bpf_verifier_env *env)
bpf_log(log, ", irreducible");
if (loop->exits_overflow)
bpf_log(log, ", too many exits");
+ if (loop->nested_exits_overflow)
+ bpf_log(log, ", too many exits of a nested loop");
bpf_log(log, "\n");
for (j = 0; j < loop->backedges_cnt; j++) {
backedge = &loop->backedges[j];
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 0d49c9021e30..e0d71df9c0b4 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1260,10 +1260,16 @@ static int join_successor(struct bpf_verifier_env *env, int cur_header, int succ
return 0;
}
-/* Unsupported loops get only an unknown header environment. */
+/*
+ * Unsupported loops get only an unknown header environment.
+ * A nested loop is processed as a single instruction, its exits are the successors: all of
+ * them are needed. No result for the loop itself depends on the list of its own exits:
+ * mark_latches() prepares them for the loop around, and compute_max_iters() asks if there is
+ * only one, which a list that is cut does not have.
+ */
static bool can_compute_loop_scev(const struct bpf_loop *loop)
{
- return !loop->irreducible && !loop->exits_overflow;
+ return !loop->irreducible && !loop->nested_exits_overflow;
}
static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
@@ -3050,12 +3056,13 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
*/
loop = aux[insn_idx].loop;
if (!loop || loop->irreducible || loop->backedges_cnt != 1 || loop->backedges_overflow ||
- loop->exits_overflow) {
+ loop->nested_exits_overflow) {
if (log->level & BPF_LOG_LEVEL2)
bpf_log(log, "loop header at %d, unsupported loop:%s%s%s\n", insn_idx,
!loop || loop->irreducible ? " irreducible" : "",
loop && loop->backedges_cnt > 1 ? " multiple backedges" : "",
- loop && loop->exits_overflow ? " too many exits" : "");
+ loop && loop->nested_exits_overflow ?
+ " too many exits of a nested loop" : "");
return 0;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 21/37] selftests/bpf: Test a loop with 301 exits
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (19 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 20/37] bpf: Compute scev for a loop that has more than 256 exits Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 22/37] bpf: Describe a 32-bit counter that lives on the stack Alexei Starovoitov
` (15 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A loop with 4 iterations and 301 exits is in another loop. The inner loop
has a count. The outer loop is not analysed:
loop header at 1, unsupported loop: too many exits of a nested loop
loop header at 2, header_count is [0..4]
Without "bpf: Compute scev for a loop that has more than 256 exits" the
inner loop is 'unsupported loop: too many exits' too.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 38 +++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index c91d178a1aed..4b9a094328de 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -2238,6 +2238,44 @@ __naked void one_backedge_two_exits(void)
: __clobber_all);
}
+__naked __noinline __used static int returns_zero(void)
+{
+ asm volatile (" \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/*
+ * A loop that the compiler unrolls in part has an exit for every 'break' and 'return' in every
+ * copy of its body. How many there are says nothing about the number of iterations.
+ * Scalar evolution takes a nested loop for one insn, and its exits are the successors of this insn.
+ * The list of exits is not complete: the loop around has no formulas.
+ */
+SEC("xdp")
+__success __log_level(2)
+__msg("loop at 1, too many exits, too many exits of a nested loop")
+__msg("loop at 2, nested in 1, too many exits")
+__msg("loop header at 1, unsupported loop: too many exits of a nested loop")
+__msg("loop header at 2, header_count is [0..4] ")
+__naked void loop_around_301_exits(void)
+{
+ asm volatile (" \
+ r7 = 0; \
+3: r6 = 0; \
+1: call returns_zero; \
+ .rept 300; \
+ if r0 == 7 goto 2f; \
+ .endr; \
+ r6 += 1; \
+ if r6 != 4 goto 1b; \
+ r7 += 1; \
+ if r7 != 4 goto 3b; \
+2: r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
SEC("xdp")
__success
__log_level(2)
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 22/37] bpf: Describe a 32-bit counter that lives on the stack
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (20 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 21/37] selftests/bpf: Test a loop with 301 exits Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 23/37] selftests/bpf: Test " Alexei Starovoitov
` (14 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A compiler that has no free register for 'int i' keeps the counter of a
loop in a stack slot:
1: r2 = *(u32 *)(r10 - 40)
w2 += 1
*(u32 *)(r10 - 40) = r2
if w2 != 200 goto 1b
Scalar evolution has a formula for a counter in a register and for a
counter in a 64-bit slot. It has no formula for this one. Thus the loop has
no count, and the verifier unrolls it:
*fp-40=(spill32 (zext32 (+ *fp-40 1))) / ?
loop header at 2, non-linear latch at 5
A 32-bit store writes the low half of the slot, and a 32-bit load reads it
back. Only the low half of the sum gets to the slot. Thus the low half is
'first value + step * k' modulo 2^32 in iteration k. That is the formula of
a register that is computed in 32 bits:
*fp-40=(spill32 (+ *fp-40 1)) / (spill32 (linear (zext32 *fp-40) 1))
loop header at 2, header_count is 200
loop header at 2, widening *fp-40 to 0..199 step 1
- (spill32 x) says nothing about the other half of the slot. The verifier
changes the range of the 32-bit number only. The slot must hold such a
number when the loop starts.
- The last store to the slot in an iteration must be the 32-bit store of
'load + constant'. There is no formula if a store of a different size,
a store to the other half or a store through a pointer comes after it.
- A jump often tests a second load of the counter. With the formula of the
loop head in it that is deeper than 8 levels: simplify it first.
- On big endian this is the high half of the slot. It has no formula.
The counter has a range now, and the verifier never goes back to one
iteration at a time. A loop that passes '&i' as the key of a lookup in an
array still needs no NULL check: the range of the key is sufficient since
commit c1e03c8479e3 ("bpf: A lookup in an array is not NULL when the
largest key is below max_entries").
5 progs of scx, libarena and bpfjailer have such a loop. They need up to
92 % fewer insns (239986 -> 18034).
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 188 ++++++++++++++++++++++++++++++++++++++++------
1 file changed, 164 insertions(+), 24 deletions(-)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index e0d71df9c0b4..b41c2d4d98b4 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -239,6 +239,16 @@ static bool is_zext32(struct scev *scev, int id, u32 *left)
return is_unop(scev, ZEXT32, id, left);
}
+/* The low half of the slot is the low half of @left. The other half is not known. */
+static bool is_spill32(struct scev *scev, int id, u32 *left)
+{
+#ifdef __BIG_ENDIAN
+ /* '*(u32 *)(r10 - 8) = rX' writes the high half, see mk_fill() */
+ return false;
+#endif
+ return is_unop(scev, SPILL32, id, left);
+}
+
static bool is_any(struct scev *scev, int id, u32 *left, u32 *right)
{
return is_binop(scev, ANY, id, left, right);
@@ -257,6 +267,14 @@ static bool is_linear_or_linear32(struct scev *scev, int id, u32 *base, u32 *slo
return is_linear(scev, *zext32 ? t : id, base, slope);
}
+/* What a register is at the entry of the loop, or the low half of that. */
+static bool is_reg_or_low32(struct scev *scev, int id, u32 *reg)
+{
+ u32 t;
+
+ return is_reg(scev, id, reg) || (is_zext32(scev, id, &t) && is_reg(scev, t, reg));
+}
+
static bool is_opaque(struct scev *scev, int id)
{
return is_op(scev, id, OPAQUE);
@@ -1546,6 +1564,26 @@ static int simplify(struct scev *scev, u32 id, void *priv)
if (is_zext32(scev, id, &a) && is_zext32(scev, a, &b))
return a;
+ /*
+ * '*(u32 *)(r10 - 8) = rX' stores the low half of rX:
+ * (spill32 (zext32 $a)) -> (spill32 $a)
+ * (spill32 (+ (zext32 $a) $b)) -> (spill32 (+ $a $b))
+ */
+ if (is_spill32(scev, id, &t) && is_zext32(scev, t, &a))
+ return expr1(scev, SPILL32, a);
+ if (is_spill32(scev, id, &t) && is_add(scev, t, &l, &b) && is_zext32(scev, l, &a))
+ return expr1(scev, SPILL32, expr2(scev, BPF_ADD, a, b));
+
+ /*
+ * 'rX = *(u32 *)(r10 - 8)' loads what was stored:
+ * (zext32 (spill32 $a)) -> (zext32 $a)
+ * (zext32 (+ (spill32 $a) $b)) -> (zext32 (+ $a $b))
+ */
+ if (is_zext32(scev, id, &t) && is_spill32(scev, t, &a))
+ return expr1(scev, ZEXT32, a);
+ if (is_zext32(scev, id, &t) && is_add(scev, t, &l, &b) && is_spill32(scev, l, &a))
+ return expr1(scev, ZEXT32, expr2(scev, BPF_ADD, a, b));
+
return id;
}
@@ -1598,6 +1636,22 @@ static int compute_header_scevs(struct bpf_verifier_env *env, struct env *header
header_env->reg2scev[ra] = id;
continue;
}
+ /*
+ * rA = (spill32 (+ rA IMM)) -> (spill32 (linear (zext32 rA) IMM))
+ * The low half of the slot counts. Nothing is known about the other half.
+ */
+ if (is_spill32(scev, ra_expr, &t) &&
+ is_add(scev, t, &l, &r) &&
+ is_reg(scev, l, &rb) &&
+ is_imm(scev, r, &imm) &&
+ ra == rb) {
+ id = expr2(scev, LINEAR_SCEV, expr1(scev, ZEXT32, l), r);
+ id = expr1(scev, SPILL32, id);
+ if (id < 0)
+ return id;
+ header_env->reg2scev[ra] = id;
+ continue;
+ }
/* rA = rA -> rA*/
if (is_reg(scev, ra_expr, &rb) && ra == rb) {
header_env->reg2scev[ra] = ra_expr;
@@ -1632,7 +1686,13 @@ static int compute_insn_scevs(struct bpf_verifier_env *env, struct env *eheader,
int id, reg;
for (reg = 0; reg < REGS_NUM; reg++) {
- id = einsn->reg2expr[reg];
+ /*
+ * Simplify first. 'store, load, test' of a 32-bit counter on the stack with the
+ * equation of the header in it is deeper than EXPR_STACK_DEPTH.
+ */
+ id = transform_expr(scev, einsn->reg2expr[reg], NULL, simplify);
+ if (id < 0)
+ return id;
id = transform_expr_once(scev, 0, id, eheader->reg2scev, instantiate_header_scevs);
if (id < 0)
return id;
@@ -1827,6 +1887,16 @@ static struct bpf_reg_state *scev_reg_for_linear(struct bpf_func_state *st, u32
return &stack->spilled_ptr;
}
+/* '*(u32 *)(r10 - 8) = rX' left a number in the low half of the slot. */
+static struct bpf_reg_state *scev_reg_for_spill32(struct bpf_func_state *st, u32 r)
+{
+ struct bpf_stack_state *stack = scev_regno_to_stack(st, r);
+
+ if (!stack || bpf_spill_size(stack) != 4 || stack->spilled_ptr.type != SCALAR_VALUE)
+ return NULL;
+ return &stack->spilled_ptr;
+}
+
/*
* Return a borrowed value; never use it as a widening destination.
* NULL denotes no readable value.
@@ -2010,6 +2080,7 @@ static int match_linear_latch(struct bpf_verifier_env *env,
* base_expr |
* - (linear (+ register imm) step) |
* ^----------------------------'
+ * 'register' is (zext32 *fp-8) for a 32-bit counter on the stack.
*/
if (!is_linear_or_linear32(scev, dst_reg_scev,
&latch->base_expr, &latch->step_expr, &latch->zext32))
@@ -2029,12 +2100,12 @@ static int match_linear_latch(struct bpf_verifier_env *env,
return false;
/* (linear register step) */
- if (is_reg(scev, latch->base_expr, &t))
+ if (is_reg_or_low32(scev, latch->base_expr, &t))
return true;
/* (linear (+ register imm) step) */
if (is_add(scev, latch->base_expr, &l, &r) &&
- is_reg(scev, l, &t) &&
+ is_reg_or_low32(scev, l, &t) &&
loop_invariant(scev, r))
return true;
@@ -2165,6 +2236,16 @@ static bool eval_expr(struct bpf_verifier_env *env, struct scev *scev,
result->s32_max = reg_s32_max(reg);
result->ptr_reg = reg->type == SCALAR_VALUE ? -1 : (int)regno;
return true;
+ } else if (is_zext32(scev, id, &l) && is_reg(scev, l, ®no)) {
+ reg = scev_reg_for_spill32(st, regno);
+ if (!reg)
+ return false;
+ result->umin = result->smin = result->u32_min = reg_u32_min(reg);
+ result->umax = result->smax = result->u32_max = reg_u32_max(reg);
+ result->s32_min = reg_s32_min(reg);
+ result->s32_max = reg_s32_max(reg);
+ result->ptr_reg = -1;
+ return true;
} else if (is_imm(scev, id, &imm)) {
result->umin = result->umax = imm;
result->smin = result->smax = imm;
@@ -2401,8 +2482,8 @@ static void latch_iv_range(struct bpf_verifier_env *env, struct bpf_func_state *
s64 c = 0;
l = latch->base_expr;
- if (!is_reg(scev, l, ®no)) {
- if (!is_add(scev, latch->base_expr, &l, &r) || !is_reg(scev, l, ®no) ||
+ if (!is_reg_or_low32(scev, l, ®no)) {
+ if (!is_add(scev, latch->base_expr, &l, &r) || !is_reg_or_low32(scev, l, ®no) ||
!eval_expr(env, scev, st, r, &off) || !scev_value_is_const(&off) ||
off.ptr_reg >= 0)
return;
@@ -2474,7 +2555,7 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
*/
if (latch->zext32 && !latch->jmp32) {
l = latch->base_expr;
- if (!is_reg(scev, l, &r) && !is_add(scev, latch->base_expr, &l, &r))
+ if (!is_reg_or_low32(scev, l, &r) && !is_add(scev, latch->base_expr, &l, &r))
return false;
if (!eval_expr(env, scev, st, l, ®_val) || reg_val.umax > U32_MAX)
return false;
@@ -2881,18 +2962,47 @@ struct linear_coefs {
u32 base_reg;
s64 slope;
bool zext32;
+ /* the low half of a stack slot */
+ bool spill32;
};
+static bool is_simple_slope(struct scev *scev, u32 id, s64 *slope)
+{
+ return is_imm(scev, id, slope) &&
+ *slope <= S16_MAX &&
+ *slope >= S16_MIN &&
+ *slope != 0;
+}
+
static bool is_simple_linear(struct scev *scev, u32 id, struct linear_coefs *out)
{
u32 base, slope;
+ out->spill32 = false;
return is_linear_or_linear32(scev, id, &base, &slope, &out->zext32) &&
is_reg(scev, base, &out->base_reg) &&
- is_imm(scev, slope, &out->slope) &&
- out->slope <= S16_MAX &&
- out->slope >= S16_MIN &&
- out->slope != 0;
+ is_simple_slope(scev, slope, &out->slope);
+}
+
+/*
+ * A 32-bit counter that lives on the stack: the SCEV of slot @r is
+ * (spill32 (linear (zext32 @r) slope)) and there is a 32-bit number in the slot in @st.
+ * Return what is known about that number.
+ */
+static struct bpf_reg_state *spill32_counter(struct scev *scev, struct bpf_func_state *st,
+ u32 id, u32 r, struct linear_coefs *out)
+{
+ u32 t, base, slope;
+
+ if (!is_spill32(scev, id, &t) ||
+ !is_linear(scev, t, &base, &slope) ||
+ !is_zext32(scev, base, &t) ||
+ !is_reg(scev, t, &out->base_reg) || out->base_reg != r ||
+ !is_simple_slope(scev, slope, &out->slope))
+ return NULL;
+ out->zext32 = true;
+ out->spill32 = true;
+ return scev_reg_for_spill32(st, r);
}
static struct cnum64 cnum64_zext32(struct cnum32 r32)
@@ -2906,13 +3016,22 @@ static void linear_bounds(struct bpf_reg_state *reg, u32 max_header_count,
{
s64 slope = coefs->slope;
s64 slope_abs = slope < 0 ? -slope : slope;
- s64 min_val = reg_smin(reg);
- s64 max_val = reg_smax(reg);
+ struct bpf_reg_state low;
+ s64 min_val, max_val;
struct cnum64 r64;
struct cnum32 r32;
s64 total_change;
bool overflow;
+ if (coefs->spill32) {
+ /* what 'rX = *(u32 *)(r10 - 8)' loads */
+ low = *reg;
+ low.var_off = tnum_subreg(reg->var_off);
+ low.r64 = cnum64_zext32(reg->r32);
+ reg = &low;
+ }
+ min_val = reg_smin(reg);
+ max_val = reg_smax(reg);
overflow = check_mul_overflow(slope, (s64)max_header_count - 1, &total_change) ||
(slope > 0 && check_add_overflow(max_val, total_change, &max_val)) ||
(slope < 0 && check_add_overflow(min_val, total_change, &min_val));
@@ -2981,7 +3100,7 @@ static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state
if (!(order & PRE) || !is_reg(scev, id, &r))
continue;
/* Synthetic values have no spilled register to mark precise. */
- reg = scev_reg_for_linear(st, r);
+ reg = scev_reg_for_linear(st, r) ?: scev_reg_for_spill32(st, r);
if (reg && reg->type == SCALAR_VALUE)
__set_bit(r, precise_regs);
}
@@ -3147,6 +3266,17 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
if (!scev_reg_alive(env, st, r))
continue;
+
+ r_scev = header_env->reg2scev[r];
+ if (spill32_counter(scev, cur_func, r_scev, r, &coefs)) {
+ /* There is no exact offset to lose if nothing is known about the number */
+ if (test_bit(r, loop->store_base_regs) &&
+ is_known_partial_spill(cur_func, r)) {
+ spill_base = true;
+ goto cant_widen;
+ }
+ continue;
+ }
/*
* A heuristic:
* Ignore SCEVs computed for unbound values and treat those as widened.
@@ -3175,7 +3305,6 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
if (is_const_scev_reg(env, insn_idx, r))
continue;
- r_scev = header_env->reg2scev[r];
reg = scev_reg_for_linear(cur_func, r);
/*
* The offset of a pointer to arena is not tracked: what enters the loop stands
@@ -3299,7 +3428,7 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
struct bpf_func_state *entry_func = loop_entry->frame[loop_entry->curframe];
struct bpf_verifier_log *log = &env->log;
struct scev *scev = env->scev;
- struct bpf_reg_state *reg;
+ struct bpf_reg_state *reg, *spill32;
struct env *header_env;
struct linear_coefs coefs;
struct bounds bounds;
@@ -3319,8 +3448,13 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
- if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r) ||
- is_const_scev_reg(env, insn_idx, r))
+ if (!scev_reg_alive(env, st, r))
+ continue;
+
+ r_expr = header_env->reg2scev[r];
+ spill32 = spill32_counter(scev, cur_func, r_expr, r, &coefs);
+ if (!spill32 && (is_unbound_scev_reg(env, cur_func, r) ||
+ is_const_scev_reg(env, insn_idx, r)))
continue;
reg = scev_reg_for_linear(cur_func, r);
@@ -3343,10 +3477,11 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
continue;
}
- r_expr = header_env->reg2scev[r];
/* If SCEV for r is (linear <reg> <slope>)*/
- if (is_simple_linear(scev, r_expr, &coefs) && coefs.base_reg == r) {
- err = scev_reg_for_write(env, cur_func, r, ®);
+ if (spill32 || (is_simple_linear(scev, r_expr, &coefs) && coefs.base_reg == r)) {
+ /* the other half of the slot stays as it is */
+ reg = spill32;
+ err = reg ? 0 : scev_reg_for_write(env, cur_func, r, ®);
if (err)
return err;
linear_bounds(reg, iters->max_header_count, &coefs, &bounds);
@@ -3403,7 +3538,7 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
struct linear_coefs coefs;
struct env *header_env;
struct bounds bounds;
- u32 r;
+ u32 r, r_scev;
int err;
if (entry_state->curframe != cur_func_state->frameno) {
@@ -3414,9 +3549,14 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
+ r_scev = header_env->reg2scev[r];
+ entry_reg = spill32_counter(scev, entry_st, r_scev, r, &coefs);
+ reg = spill32_counter(scev, cur_func_state, r_scev, r, &coefs);
+ if (entry_reg && reg)
+ goto clamp;
+
/* If SCEV for r is (linear <reg> <slope>)*/
- if (!is_simple_linear(scev, header_env->reg2scev[r], &coefs) ||
- coefs.base_reg != r)
+ if (!is_simple_linear(scev, r_scev, &coefs) || coefs.base_reg != r)
continue;
entry_reg = scev_reg_for_linear(entry_st, r);
@@ -3440,7 +3580,7 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
if (!is_widenable_reg_type(entry_reg) ||
(coefs.zext32 && entry_reg->type != SCALAR_VALUE))
continue;
-
+clamp:
linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
if (r == entry->iv_reg)
iv_bounds(&bounds, entry->iv_range);
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 23/37] selftests/bpf: Test a 32-bit counter that lives on the stack
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (21 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 22/37] bpf: Describe a 32-bit counter that lives on the stack Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 24/37] bpf: Count a loop with several back edges Alexei Starovoitov
` (13 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The tests are rows of RANGE_TEST() and PARTIAL_SPILL_TEST(). The counter is
in '*(u32 *)(r10 - 8)'.
- 5 loops with a count: a register with more than 32 bits that is stored
at first, a second load for the jump, 64-bit addition and two shifts, a
first value that is a range with the jump after and in front of the
addition. Each prog runs and returns the number of iterations. It must
be the count of the verifier. The tests also expect the range of the
slot.
- Two first values, 2 and 0, meet in front of the loop. The verifier must
count the loop two times.
- The counter makes the address of a spill: the loop is not summarized. It
is summarized if nothing is known about the number.
- No formula: a step above S16_MAX, 64 bits in the slot when the loop
starts, a load that extends the sign, a sum that is made from a
different slot.
The 8 tests that expect a formula are for little endian.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 66 +++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 4b9a094328de..20141f45f28b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -3064,6 +3064,72 @@ PARTIAL_SPILL_TEST(partial_spill32_dead_at_the_head, u32, 1000000, NOT_UNROLLED,
PARTIAL_SPILL_TEST(partial_spill32_unknown, u32, 1000000, NOT_UNROLLED,
"*(u32 *)(r10 - 8) = r0;", SPILL32_INC)
+/*
+ * 'u32 i' in fp-8 is a counter: SPILL32_INC is 'i++'. It has a formula. The rows are those of
+ * RANGE_TEST: r6 is the first value, r7 is the bound, r9 is the number of iterations.
+ */
+#define SPILL32_LOOP(body) "*(u32 *)(r10 - 8) = r6; 1:" LIMIT_ITERATIONS body
+#define SPILL32_LOAD "r1 = *(u32 *)(r10 - 8);"
+#define SPILL32_COUNTED(count, range) \
+ __retval(count) \
+ __msg("loop header at {{[0-9]+}}, header_count is [0.." #count "] ") \
+ __msg("loop header at {{[0-9]+}}, widening *fp-8 to " range " ") \
+ __msg("loop header at {{[0-9]+}}, clamping *fp-8 to " range " ")
+
+/* On big-endian '*(u32 *)(r10 - 8)' is the high half of the slot: there is no formula for it. */
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+/* Only the low half of the register that is stored at first gets to the slot */
+RANGE_TEST(spill32_inc, "", 7ULL << 32, 7ULL << 32, 1, 5, 5,
+ SPILL32_LOOP(SPILL32_INC "if r1 != r7 goto 1b;"),
+ __msg(" *fp-8=(spill32 (+ *fp-8 1)) / (spill32 (linear (zext32 *fp-8) 1))")
+ SPILL32_COUNTED(5, "0..4"))
+/* The jump tests a second load of the counter */
+RANGE_TEST(spill32_second_load, "", 0, 0, 1, 5, 5,
+ SPILL32_LOOP(SPILL32_INC SPILL32_LOAD "if w1 != w7 goto 1b;"),
+ SPILL32_COUNTED(5, "0..4"))
+/* What a compiler makes without 32-bit insns */
+RANGE_TEST(spill32_add64, "", 0, 0, 1, 5, 5,
+ SPILL32_LOOP(SPILL32_LOAD "r1 += 1; *(u32 *)(r10 - 8) = r1; r1 <<= 32; r1 >>= 32;"
+ "if r1 != r7 goto 1b;"), SPILL32_COUNTED(5, "0..4"))
+/* 'first + step * count' is 0..8. The latch lets 5 through. */
+RANGE_TEST(spill32_first_is_a_range, "", 0, 3, 1, 6, 6,
+ SPILL32_LOOP(SPILL32_INC "if w1 < w7 goto 1b;"), SPILL32_COUNTED(6, "0..5"))
+RANGE_TEST(spill32_first_is_a_range_pre, "", 0, 3, 1, 6, 6,
+ SPILL32_LOOP(SPILL32_LOAD "if r1 >= r7 goto 2f;" SPILL32_INC "goto 1b;"),
+ SPILL32_COUNTED(7, "0..6"))
+/* Explore with 2 in the slot first, with 0 next. The second state is not equal to the first. */
+RANGE_TEST(spill32_first_is_precise, "", 0, 0, 1, 0, 1,
+ "if r7 == 0 goto +1; r6 = 2;" SPILL32_LOOP(SPILL32_INC "if w1 != 5 goto 1b;"),
+ __flag(BPF_F_TEST_STATE_FREQ)
+ __msg("header_count is [0..3] ") __msg("header_count is [0..5] "))
+/* The counter makes the address of a spill. Such a loop is not summarized, as with a register. */
+PARTIAL_SPILL_TEST(spill32_is_a_stack_offset, u32, 4,
+ "can't widen *fp-8, expr is (spill32 (+ *fp-8 8)), requires exact stack-offset tracking",
+ SPILL32_0, SPILL32_LOAD "r2 = r10; r2 += -48; r2 += r1; *(u64 *)(r2 + 0) = r6;"
+ "w1 += 8; *(u32 *)(r10 - 8) = r1;")
+/* But there is no exact address to lose when nothing is known about the number */
+PARTIAL_SPILL_TEST(spill32_unknown_is_a_stack_offset, u32, 1000000,
+ "widening *fp-8 to 0..0xffffffff step 1",
+ "*(u32 *)(r10 - 8) = r0;", SPILL32_LOAD "r2 = r1; r2 &= 24; r3 = r10; r3 += -48; r3 += r2;"
+ "*(u64 *)(r3 + 0) = r6; w1 += 8; *(u32 *)(r10 - 8) = r1;")
+#endif
+
+/* A step above S16_MAX has no range, as with a register */
+PARTIAL_SPILL_TEST(spill32_large_step, u32, 4, "can't widen *fp-8, expr is (spill32 ",
+ SPILL32_0, SPILL32_LOAD "w1 += 0x8000; *(u32 *)(r10 - 8) = r1;")
+/* No formula: the slot holds 64 bits when the loop starts, */
+RANGE_TEST(spill32_first_is_64bit, "", 0, 0, 1, 5, 5,
+ "*(u64 *)(r10 - 8) = r6; 1:" LIMIT_ITERATIONS SPILL32_INC "if w1 != w7 goto 1b;",
+ NO_COUNT)
+/* the load extends the sign, */
+RANGE_TEST(spill32_sext_load, "", 0, 0, 1, 5, 5,
+ SPILL32_LOOP(SPILL32_INC "r1 = *(s32 *)(r10 - 8); if r1 != r7 goto 1b;"), NO_COUNT)
+/* the sum is made from a different slot. */
+RANGE_TEST(spill32_other_slot, "", 0, 0, 1, 5, 5,
+ "*(u32 *)(r10 - 16) = r6;"
+ SPILL32_LOOP(SPILL32_LOAD "if w1 == w7 goto 2f; r1 = *(u32 *)(r10 - 16); w1 += 1;"
+ "*(u32 *)(r10 - 8) = r1; goto 1b;"), NO_COUNT)
+
/*
* r7 starts as 1 + 6*k and invariant r8 as 1 + 9*k (0 <= k <= 3).
* The (any r7 r8) union must preserve base 1 and step gcd(6, 9) = 3.
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 24/37] bpf: Count a loop with several back edges
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (22 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 23/37] selftests/bpf: Test " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 25/37] selftests/bpf: Test loops " Alexei Starovoitov
` (12 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
'continue', or a compiler that copies the end of the loop body, gives a
loop more than one back edge:
1: r6 += 1
if r0 == 0 goto 2f
if r6 < 10 goto 1b back edge 1
goto 3f
2: if r6 < 20 goto 1b back edge 2
3:
loop header at 1, unsupported loop: multiple backedges
Scalar evolution joins what all back edges bring to the loop head. A
register is linear there only if each path adds the same step. Then it is
'first value + step * k' in iteration k, whatever back edges were taken.
Thus a jump that leaves the loop and dominates a back edge, a latch, tells
in which iterations this back edge can be taken:
back edge 1 iterations 0 .. 8
back edge 2 iterations 0 .. 18
The loop ends, at the latest, in an iteration in which no back edge can be
taken. Count each back edge as before, with its latch that has the smallest
count, and take the largest count: 'header_count is [0..20]'.
With one back edge the loop ends when the latch is false. Here the loop can
go on through a different back edge and get to this latch again. The latch
must still be false in the iteration of the largest count:
- '<' and the like stay false until the variable wraps around. For a
pointer that is base + offset: the limit is an offset of 2^29.
- '!=' is false in one iteration only. Its count must be the largest one,
and its first value and bound must be constants.
- When these latches are the same test, the first iteration in which it is
false is the last one, as with one back edge.
If not, or if a back edge has no latch with a count, there is no count.
The other latches of a back edge are not tried.
These loops get ranges now, and the verifier never goes back to one
iteration at a time. It rejects a prog that needs the exact values of one
path, like the same loop with one back edge and 'break'. No prog of scx,
the selftests, Cilium or Meta is lost or needs more insns.
sched_process_exit of profiler1, 2 and 3, built by gcc, loads now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 9 +-
kernel/bpf/scev.c | 189 +++++++++++++-----
.../selftests/bpf/progs/verifier_scev.c | 2 +-
3 files changed, 140 insertions(+), 60 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 95e0d7face11..9a0ed5610650 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -351,8 +351,9 @@ struct bpf_loop_iters {
/* [0, max] at the entry, and the loop only clears bits */
unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
/*
- * The variable that the latch tests, or -1, and what it can be at the
- * loop header: what it is at the entry or what the latch lets through.
+ * The variable that the latches of all back edges test, or -1, and what
+ * it can be at the loop header: what it is at the entry or what one of
+ * the latches lets through.
*/
int iv_reg;
struct cnum64 iv_range;
@@ -726,8 +727,8 @@ struct bpf_backedge {
* Conditional jumps within the loop that dominate 'from' and have
* a successor outside the loop, nearest first. latches[0] is the
* latch; the remaining entries are additional exit conditions.
- * Each usable condition gives an upper bound on the iteration count,
- * so keeping only a subset is sound.
+ * Each usable condition tells iterations in which the back edge is
+ * not taken, so keeping only a subset is sound.
*/
int latches[MAX_LATCHES];
u8 latches_cnt;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index b41c2d4d98b4..fd6abc41b946 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2638,6 +2638,31 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
return true;
}
+/*
+ * @latch is false in iteration @cnt, counted from 0. That is the end of a loop with one back
+ * edge. A loop with several back edges may go on through another one and get to @latch
+ * again. Return the last iteration in which @latch is still false:
+ * - '<', '>' and the like stay false until the variable leaves [lo, hi] of latch_domain():
+ * a number wraps around, the offset of a pointer gets too large to tell,
+ * - '!=' is false in one iteration. Which one is known for constants only.
+ */
+static u64 latch_false_until(const struct linear_latch *latch, const struct scev_value *initial,
+ const struct scev_value *step_val, const struct scev_value *bound,
+ u64 cnt)
+{
+ bool sign = is_signed_op(latch->op) || initial->ptr_reg >= 0;
+ struct latch_domain d;
+ u64 step;
+
+ if (latch->op == BPF_JNE)
+ return scev_value_is_const(initial) && scev_value_is_const(bound) ? cnt : 0;
+
+ step = latch->jmp32 || latch->zext32 ? (u64)(s64)(s32)step_val->umin : step_val->umin;
+ latch_domain(latch, initial, bound, sign, &d);
+ /* The first value is in [lo, hi]. do_check() rejects a pointer with a larger offset. */
+ return (s64)step < 0 ? div64_u64(d.imin - d.lo, -step) : div64_u64(d.hi - d.imax, step);
+}
+
/*
* Loop with post-condition:
*
@@ -2667,7 +2692,7 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
static bool compute_max_iters(struct bpf_verifier_env *env,
struct bpf_func_state *st,
struct linear_latch *latch,
- struct bpf_loop_iters *iters)
+ struct bpf_loop_iters *iters, u64 *false_until)
{
struct bpf_insn_aux_data *aux = env->insn_aux_data;
struct bpf_loop *loop = aux[bpf_loop_at_index(env, latch->insn_idx)].loop;
@@ -2764,11 +2789,12 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
}
counted:
+ *false_until = latch_false_until(latch, &initial_val, &step_val, &bound_val,
+ backedge_taken_count);
/* The header executes once on entry and once for each taken backedge, hence +1. */
iters->max_header_count = backedge_taken_count + 1;
/*
* The latch is a conditional jump with one jump target exiting the loop.
- * Linear latch is matched only if the loop has a single backedge.
* The loop still, however can have multiple exits.
* In such case, conservatively assume that non-latch exit can happen
* at any iteration, thus setting minimal number of iterations as 0.
@@ -3146,57 +3172,32 @@ static bool reg_is_zero_to_max(const struct bpf_reg_state *reg)
cnum32_is_subset(reg->r32, cnum32_from_cnum64(r64)) && reg->step <= 1;
}
-int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
- struct bpf_loop_iters *iters)
+/*
+ * A jump that leaves the loop and dominates @backedge tells in which iterations @backedge can
+ * be taken: in none from the count of the jump to @false_until. That bounds the iterations of
+ * the loop only if it has no other back edge. Keep the jump with the smallest count.
+ * Return 1 if there is one.
+ */
+static int count_backedge(struct bpf_verifier_env *env, struct bpf_func_state *st, int insn_idx,
+ struct bpf_backedge *backedge, struct bpf_loop_iters *iters,
+ struct linear_latch *best, u64 *false_until)
{
- struct bpf_func_state *cur_func = st->frame[st->curframe];
- struct bpf_insn_aux_data *aux = env->insn_aux_data;
struct bpf_verifier_log *log = &env->log;
- struct scev *scev = env->scev;
- struct env *header_env;
- struct linear_latch latch, best;
- struct linear_coefs coefs;
- struct bpf_backedge *backedge;
struct bpf_loop_iters tmp;
- struct bpf_loop *loop;
- int insn_idx = st->insn_idx;
+ struct linear_latch latch;
bool found = false;
int linear_latch;
int latch_idx;
- int err, j;
- u32 r;
-
- if (!env->allow_uninit_stack)
- return 0;
-
- /*
- * If insn_idx is a loop header for a reducible loop with a single backedge.
- * loop is NULL for secondary entries to irreducible loops.
- */
- loop = aux[insn_idx].loop;
- if (!loop || loop->irreducible || loop->backedges_cnt != 1 || loop->backedges_overflow ||
- loop->nested_exits_overflow) {
- if (log->level & BPF_LOG_LEVEL2)
- bpf_log(log, "loop header at %d, unsupported loop:%s%s%s\n", insn_idx,
- !loop || loop->irreducible ? " irreducible" : "",
- loop && loop->backedges_cnt > 1 ? " multiple backedges" : "",
- loop && loop->nested_exits_overflow ?
- " too many exits of a nested loop" : "");
- return 0;
- }
+ u64 until;
+ int j;
/* If this backedge has a latch */
- backedge = &loop->backedges[0];
if (!backedge->latches_cnt) {
if (log->level & BPF_LOG_LEVEL2)
bpf_log(log, "loop header at %d, latch not identified\n", insn_idx);
return 0;
}
- /*
- * Every condition dominating the backedge bounds the number of
- * iterations from above, keep the smallest bound.
- */
for (j = 0; j < backedge->latches_cnt; j++) {
latch_idx = backedge->latches[j];
linear_latch = match_linear_latch(env, insn_idx, latch_idx, &latch);
@@ -3208,7 +3209,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
insn_idx, j ? "exit" : "latch", latch_idx);
continue;
}
- if (!compute_max_iters(env, cur_func, &latch, &tmp)) {
+ if (!compute_max_iters(env, st, &latch, &tmp, &until)) {
if (log->level & BPF_LOG_LEVEL2)
bpf_log(log,
"loop header at %d, can't compute iterations count at %d\n",
@@ -3219,18 +3220,107 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
iters->max_header_count = tmp.max_header_count;
iters->iv_reg = tmp.iv_reg;
iters->iv_range = tmp.iv_range;
- best = latch;
+ *false_until = until;
+ *best = latch;
}
iters->min_header_count = found ? min(iters->min_header_count, tmp.min_header_count)
: tmp.min_header_count;
found = true;
}
- if (!found) {
+ if (!found && (log->level & BPF_LOG_LEVEL2))
+ bpf_log(log, "loop header at %d, can't compute iterations count\n", insn_idx);
+ return found;
+}
+
+/* Two jumps that test the same thing: in each iteration both are true or both are false. */
+static bool same_latch(const struct linear_latch *a, const struct linear_latch *b)
+{
+ return a->base_expr == b->base_expr && a->step_expr == b->step_expr &&
+ a->bound_expr == b->bound_expr && a->op == b->op &&
+ a->zext32 == b->zext32 && a->jmp32 == b->jmp32;
+}
+
+int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ struct bpf_loop_iters *iters)
+{
+ struct bpf_func_state *cur_func = st->frame[st->curframe];
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ struct bpf_verifier_log *log = &env->log;
+ struct scev *scev = env->scev;
+ struct env *header_env;
+ struct linear_latch latch, first;
+ struct linear_coefs coefs;
+ struct bpf_loop_iters tmp;
+ struct bpf_loop *loop;
+ int insn_idx = st->insn_idx;
+ u64 until, false_until;
+ bool same = true;
+ int err, i;
+ u32 r;
+
+ if (!env->allow_uninit_stack)
+ return 0;
+
+ /*
+ * If insn_idx is a loop header for a reducible loop.
+ * loop is NULL for secondary entries to irreducible loops.
+ */
+ loop = aux[insn_idx].loop;
+ if (!loop || loop->irreducible || loop->backedges_overflow ||
+ loop->nested_exits_overflow) {
if (log->level & BPF_LOG_LEVEL2)
- bpf_log(log, "loop header at %d, can't compute iterations count\n", insn_idx);
+ bpf_log(log, "loop header at %d, unsupported loop:%s%s%s\n", insn_idx,
+ !loop || loop->irreducible ? " irreducible" : "",
+ loop && loop->backedges_overflow ? " too many backedges" : "",
+ loop && loop->nested_exits_overflow ?
+ " too many exits of a nested loop" : "");
+ return 0;
+ }
+
+ /*
+ * A register that is linear at the header is 'first value + slope * k' in iteration k
+ * whatever back edges were taken. So the jumps that dominate a back edge say in
+ * which iterations it can be taken. The loop ends, at the latest, in an iteration
+ * in which no back edge can be taken.
+ */
+ bitmap_zero(iters->precise_regs, REGS_NUM);
+ for (i = 0; i < loop->backedges_cnt; i++) {
+ err = count_backedge(env, cur_func, insn_idx, &loop->backedges[i],
+ i ? &tmp : iters, &latch, &until);
+ if (err <= 0)
+ return err;
+ err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs) ?:
+ mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs) ?:
+ mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
+ if (err)
+ return err;
+ if (!i) {
+ false_until = until;
+ first = latch;
+ continue;
+ }
+ same &= same_latch(&first, &latch);
+ false_until = min(false_until, until);
+ iters->max_header_count = max(iters->max_header_count, tmp.max_header_count);
+ iters->min_header_count = min(iters->min_header_count, tmp.min_header_count);
+ /* at the header the variable is what one of the latches lets through */
+ if (iters->iv_reg >= 0 && iters->iv_reg == tmp.iv_reg)
+ iters->iv_range = cnum64_union(iters->iv_range, tmp.iv_range);
+ else
+ iters->iv_reg = -1;
+ }
+ /*
+ * When the latches that were kept are the same test, the first iteration in which it's
+ * false is the last one. Otherwise take the largest count and make sure that the latches
+ * which became false earlier are false still.
+ */
+ if (!same && iters->max_header_count - 1 > false_until) {
+ if (log->level & BPF_LOG_LEVEL2)
+ bpf_log(log,
+ "loop header at %d, a latch may be true again in iteration %u\n",
+ insn_idx, iters->max_header_count - 1);
return 0;
}
- latch = best;
if (iters->max_header_count == 0) {
if (log->level & BPF_LOG_LEVEL2)
@@ -3365,17 +3455,6 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
return 0;
}
- bitmap_zero(iters->precise_regs, REGS_NUM);
- err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs);
- if (err)
- return err;
- err = mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs);
- if (err)
- return err;
- err = mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
- if (err)
- return err;
-
return 1;
}
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 20141f45f28b..bb4ce07de1e5 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -729,7 +729,7 @@ __msg("scev at header 3:")
__msg_next(" r6=(any r6 (+ r6 1)) / ?")
__msg_next(" r7=(any (+ r7 1) 0) / ?")
__msg_next(" r8=(+ r8 1) / (linear r8 1)")
-__msg("loop header at 3, unsupported loop: multiple backedges")
+__msg("loop header at 3, can't compute iterations count")
__naked void nested_loops_shared_header(void)
{
asm volatile (" \
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 25/37] selftests/bpf: Test loops with several back edges
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (23 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 24/37] bpf: Count a loop with several back edges Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 26/37] bpf: Compute bounds for a shift, a division and a remainder by a range Alexei Starovoitov
` (11 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The tests are rows of RANGE_TEST(). BACKEDGES() makes a loop with two back
edges. The verifier does not know which one is taken.
- 2 loops whose latches are still false in the last iteration: '<' with
'!=', a variable that wraps around directly after the last iteration.
The progs run along the longest path and return the number of
iterations. It must be the upper bound of the verifier.
- 4 loops with a latch that can be true again: 'r6 != 3' behind 3, a
variable that wraps around downwards and upwards, a pointer that gets
to the limit of its offset.
- The same '!=' with a range in front of both back edges has a count. It
has none if the bound, the variable, the op or the width differ.
- No count: a back edge without a count, 17 back edges.
- The first value is a range: the variable has what one of three latches
lets through, and nothing is known if the latches test two variables.
- Two bounds, 3 and 5, meet in front of the loop, for each of the two
latches. The verifier must count the loop two times.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 86 +++++++++++++++++++
1 file changed, 86 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index bb4ce07de1e5..f3908f37ca74 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1137,6 +1137,92 @@ RANGE_TEST(range_ge_dir, "", 10, 20, 2, 3, 5, PRE("<"), NO_COUNT)
RANGE_TEST(range_ne_step2, "", 0, 3, 2, 3, 8, PRE("=="), NO_COUNT)
RANGE_TEST(range_ne_behind, "", 0, 4, 1, 3, 8, PRE("=="), NO_COUNT)
+/*
+ * Two back edges:
+ *
+ * do {
+ * add;
+ * if (!other) {
+ * if (jump1) continue;
+ * break;
+ * }
+ * } while (jump0);
+ *
+ * The verifier doesn't know which one is taken. 'r7 == 0' is false at run time: r7 is 0..1 and 1.
+ */
+#define BACKEDGES(add, other, jump1, jump0) \
+ "1:" LIMIT_ITERATIONS add ";" \
+ "if " other " goto 3f; if " jump1 " goto 1b; goto 2f; 3: if " jump0 " goto 1b;"
+/* The loop can go on through the other back edge and get to a latch that was false */
+#define AGAIN(iteration) \
+ __msg("loop header at {{[0-9]+}}, a latch may be true again in iteration " #iteration) \
+ __not_msg("header_count is")
+
+/* The largest count. '!=' is false in one iteration: it has to be the last one. */
+RANGE_TEST(backedges_ne_is_last, "", 0, 0, 1, 0, 1,
+ BACKEDGES("r6 += 1", "r7 == 0", "r6 != 6", "r6 < 3"), COUNTED(6, "0..5"))
+/* Only the latch with the smallest count of a back edge is looked at: 'r6 != 3', not 'r6 < 100' */
+#define AND_R8(add, count, jump0) \
+ "r8 = 0;" BACKEDGES(add "; r8 += 1", "r7 == 0", "r8 < " #count, jump0)
+RANGE_TEST(backedges_ne_is_not_last, "", 0, 0, 1, 0, 1,
+ AND_R8("r6 += 1", 5, "r6 >= 100 goto 2f; if r6 != 3"), AGAIN(4))
+/*
+ * w6 is 31, 23, 15, 7 in iterations 0..3, and M32 in iteration 4. 'w6 > 15' is false in iterations
+ * 2 and 3.
+ */
+RANGE_TEST(backedges_end_then_wrap, "", 39, 39, 1, 0, 1,
+ AND_R8("w5 = -8; w6 += w5", 4, "w6 > 15"), __retval(4) __msg("header_count is [0..4] "))
+RANGE_TEST(backedges_wrap_then_end, "", 39, 39, 1, 0, 1,
+ AND_R8("w5 = -8; w6 += w5", 5, "w6 > 15"), AGAIN(4))
+/* The same upwards. The largest first value wraps around first. */
+RANGE_TEST(backedges_wrap_up_then_end, "", M32 - 55, M32 - 39, 1, 0, 1,
+ AND_R8("w6 += 8", 5, "w6 < -16"), AGAIN(4))
+/* It's base + offset that wraps around when the variable is a pointer */
+RANGE_TEST(backedges_pointer_wraps, "", 0, 0, 1, 0, 1, "r4 = r10; r4 += -8; r5 = r10; r5 += -512;"
+ AND_R8("r5 += 8", 100000000, "r5 < r4"), AGAIN(99999999))
+
+/* The same test in front of each back edge: the first iteration in which it's false is the last */
+#define TWO_TESTS(jump1, jump0) \
+ "r5 = 0; r8 = 0;" BACKEDGES("r5 += 1; r8 += 1", "r9 & 1", jump1, jump0)
+RANGE_TEST(backedges_same_test, "", 1, 6, 1, 1, 6, TWO_TESTS("r8 != r7", "r8 != r7"),
+ __retval(6) __msg("header_count is [0..6] "))
+RANGE_TEST(backedges_other_bound, "", 1, 6, 1, 1, 6, TWO_TESTS("r8 != r7", "r8 != r6"), AGAIN(5))
+RANGE_TEST(backedges_other_first, "", 1, 6, 1, 1, 6, TWO_TESTS("r8 != r7", "r5 != r7"), AGAIN(5))
+RANGE_TEST(backedges_other_op, "", 1, 6, 1, 1, 6, TWO_TESTS("r8 != r7", "r8 < r7"), AGAIN(5))
+RANGE_TEST(backedges_other_width, "", 1, 6, 1, 1, 6, TWO_TESTS("r8 != r7", "w8 != w7"), AGAIN(5))
+
+/* A back edge has no count */
+RANGE_TEST(backedges_no_count, "", 0, 0, 1, 0, 1,
+ BACKEDGES("r6 += 1", "r7 == 0", "r6 < 3", "r7 == 0"), NO_COUNT)
+/* More than 16 back edges */
+#define X4(s) s s s s
+RANGE_TEST(backedges_17, "", 0, 0, 1, 0, 1,
+ "1:" LIMIT_ITERATIONS "if r6 >= 3 goto 2f; r6 += 1;" X4(X4("if r7 == 9 goto 1b;"))
+ "goto 1b;",
+ __msg("unsupported loop: too many backedges") __not_msg("header_count is"))
+
+/*
+ * The first value is 0..3: 'first + step * count' is 0..8. At the header r6 is its first value or
+ * what one of the three latches lets through.
+ */
+RANGE_TEST(backedges_first_is_a_range, "", 0, 3, 1, 0, 2,
+ "1:" LIMIT_ITERATIONS "r6 += 1; if r7 == 0 goto 3f; if r7 == 2 goto 4f;"
+ "if r6 < 3 goto 1b; goto 2f; 4: if r6 < 6 goto 1b; goto 2f; 3: if r6 < 4 goto 1b;",
+ COUNTED(6, "0..5"))
+/* The latches test different variables: they tell nothing about r6 */
+RANGE_TEST(backedges_two_variables, "", 0, 3, 1, 0, 1,
+ BACKEDGES("r6 += 1; r7 += 1", "r9 & 1", "r6 < 3", "r7 < 6"),
+ __msg("header_count is [0..6] ") __msg("widening r6 to 0..8 "))
+/* Explore with a bound of 3 first, of 5 next. The second state is not equal to the first. */
+#define PRECISE_BOUND(reg) \
+ "r4 = 2; r5 = 2; " reg " = 5; if r7 == 0 goto +1; " reg " = 3; r0 = 0;" \
+ BACKEDGES("r6 += 1", "r9 & 1", "r6 < r4", "r6 < r5")
+#define EXPLORED_TWICE \
+ __flag(BPF_F_TEST_STATE_FREQ) \
+ __msg("header_count is [0..3] ") __msg("header_count is [0..5] ")
+RANGE_TEST(backedges_bound1_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r4"), EXPLORED_TWICE)
+RANGE_TEST(backedges_bound0_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r5"), EXPLORED_TWICE)
+
SEC("xdp")
__success
__log_level(2)
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 26/37] bpf: Compute bounds for a shift, a division and a remainder by a range
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (24 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 25/37] selftests/bpf: Test loops " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 27/37] selftests/bpf: Test " Alexei Starovoitov
` (10 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The result of x >> n, x << n, x s>> n, x / n and x % n is an unknown scalar
unless n is a known constant. Real code has n in a range: x >> (n & 31),
1 << order, hash % nr_cpus. A prog that uses such a result as an index is
rejected, unless the verifier sees each value of n as a constant in a state
of its own.
Compute the range of the result from the ranges of the two operands. With
x = [xmin, xmax] and n = [a, b], unsigned:
x >> n [xmin >> b, xmax >> a]
x << n [xmin << a, xmax << b] if no bit can be shifted out
x s>> n a larger count brings a signed bound closer to 0, or to -1
x / n [xmin / b, xmax / a]. [0, xmax] if a is 0: x / 0 is 0
x % n [0, min(xmax, b - 1)]. x if xmax < a.
[0, xmax] if a is 0: x % 0 is x
The functions for << and >> have taken the smallest and the largest count
since commit f1174f77b50c ("bpf/verifier: rework value tracking"). Commit
bb7f0f989ca7 ("bpf: fix integer overflows") left them with constants only.
What they do with var_off holds for a constant only: a bit of x can end up
in more than one place. So var_off of x is forgotten first. Then
tnum_lshift() and tnum_rshift() by the smallest count give the bits that
are 0 for each count.
As before nothing is known if the count can be 64 (32 for a 32-bit insn) or
more, and about a signed division or remainder by a range. Nothing changes
for a constant.
The bounds decide jumps, and states that were equal can differ now, so the
verifier prunes fewer paths in some progs. In sched-ext/scx 26 progs
change, with 2.6 % fewer insns in total: flow_select_cpu 145475 -> 89352,
alloc_selftest 23734 -> 29898.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 85 ++++++++++++++++++++++---------------------
1 file changed, 44 insertions(+), 41 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 94d9a065e62d..93ca0126a310 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16815,10 +16815,11 @@ static void scalar_min_max_shmul(struct bpf_reg_state *dst_reg,
static void scalar32_min_max_udiv(struct bpf_reg_state *dst_reg,
struct bpf_reg_state *src_reg)
{
- u32 src_val = reg_u32_min(src_reg); /* non-zero, const divisor */
+ u32 src_min = reg_u32_min(src_reg), src_max = reg_u32_max(src_reg); /* src_max isn't 0 */
- reg_set_urange32(dst_reg, reg_u32_min(dst_reg) / src_val,
- reg_u32_max(dst_reg) / src_val);
+ /* The largest divisor gives the smallest result, but for x / 0 = 0 */
+ reg_set_urange32(dst_reg, src_min ? reg_u32_min(dst_reg) / src_max : 0,
+ reg_u32_max(dst_reg) / (src_min ?: 1));
/* Reset other ranges/tnum to unbounded/unknown. */
reset_reg64_and_tnum(dst_reg);
@@ -16827,10 +16828,11 @@ static void scalar32_min_max_udiv(struct bpf_reg_state *dst_reg,
static void scalar_min_max_udiv(struct bpf_reg_state *dst_reg,
struct bpf_reg_state *src_reg)
{
- u64 src_val = reg_umin(src_reg); /* non-zero, const divisor */
+ u64 src_min = reg_umin(src_reg), src_max = reg_umax(src_reg); /* src_max isn't 0 */
- reg_set_urange64(dst_reg, div64_u64(reg_umin(dst_reg), src_val),
- div64_u64(reg_umax(dst_reg), src_val));
+ /* The largest divisor gives the smallest result, but for x / 0 = 0 */
+ reg_set_urange64(dst_reg, src_min ? div64_u64(reg_umin(dst_reg), src_max) : 0,
+ div64_u64(reg_umax(dst_reg), src_min ?: 1));
/* Reset other ranges/tnum to unbounded/unknown. */
reset_reg32_and_tnum(dst_reg);
@@ -16911,17 +16913,17 @@ static void scalar_min_max_sdiv(struct bpf_reg_state *dst_reg,
static void scalar32_min_max_umod(struct bpf_reg_state *dst_reg,
struct bpf_reg_state *src_reg)
{
- u32 src_val = reg_u32_min(src_reg); /* non-zero, const divisor */
- u32 res_max = src_val - 1;
+ u32 src_min = reg_u32_min(src_reg), src_max = reg_u32_max(src_reg); /* src_max isn't 0 */
/*
- * If dst_umax <= res_max, the result remains unchanged.
- * e.g., [2, 5] % 10 = [2, 5].
+ * If dst_umax < src_min, the result remains unchanged.
+ * e.g., [2, 5] % [10, 20] = [2, 5].
*/
- if (reg_u32_max(dst_reg) <= res_max)
+ if (reg_u32_max(dst_reg) < src_min)
return;
- reg_set_urange32(dst_reg, 0, min(reg_u32_max(dst_reg), res_max));
+ /* The result is below the divisor and not above x, and x % 0 = x */
+ reg_set_urange32(dst_reg, 0, min(reg_u32_max(dst_reg), src_min ? src_max - 1 : U32_MAX));
/* Reset other ranges/tnum to unbounded/unknown. */
reset_reg64_and_tnum(dst_reg);
@@ -16930,17 +16932,17 @@ static void scalar32_min_max_umod(struct bpf_reg_state *dst_reg,
static void scalar_min_max_umod(struct bpf_reg_state *dst_reg,
struct bpf_reg_state *src_reg)
{
- u64 src_val = reg_umin(src_reg); /* non-zero, const divisor */
- u64 res_max = src_val - 1;
+ u64 src_min = reg_umin(src_reg), src_max = reg_umax(src_reg); /* src_max isn't 0 */
/*
- * If dst_umax <= res_max, the result remains unchanged.
- * e.g., [2, 5] % 10 = [2, 5].
+ * If dst_umax < src_min, the result remains unchanged.
+ * e.g., [2, 5] % [10, 20] = [2, 5].
*/
- if (reg_umax(dst_reg) <= res_max)
+ if (reg_umax(dst_reg) < src_min)
return;
- reg_set_urange64(dst_reg, 0, min(reg_umax(dst_reg), res_max));
+ /* The result is below the divisor and not above x, and x % 0 = x */
+ reg_set_urange64(dst_reg, 0, min(reg_umax(dst_reg), src_min ? src_max - 1 : U64_MAX));
/* Reset other ranges/tnum to unbounded/unknown. */
reset_reg32_and_tnum(dst_reg);
@@ -17284,16 +17286,16 @@ static void scalar_min_max_rsh(struct bpf_reg_state *dst_reg,
static void scalar32_min_max_arsh(struct bpf_reg_state *dst_reg,
struct bpf_reg_state *src_reg)
{
- u64 umin_val = reg_u32_min(src_reg);
+ u32 umin_val = reg_u32_min(src_reg), umax_val = reg_u32_max(src_reg);
+ s32 smin = reg_s32_min(dst_reg), smax = reg_s32_max(dst_reg);
- /* Upon reaching here, src_known is true and
- * umax_val is equal to umin_val.
+ /*
+ * A larger count brings the result closer to 0, or to -1.
* Blow away the dst_reg umin_value/umax_value and rely on
* dst_reg var_off to refine the result.
*/
- reg_set_srange32(dst_reg,
- (u32)(((s32)reg_s32_min(dst_reg)) >> umin_val),
- (u32)(((s32)reg_s32_max(dst_reg)) >> umin_val));
+ reg_set_srange32(dst_reg, smin >> (smin < 0 ? umin_val : umax_val),
+ smax >> (smax < 0 ? umax_val : umin_val));
dst_reg->var_off = tnum_arshift(tnum_subreg(dst_reg->var_off), umin_val, 32);
@@ -17304,13 +17306,12 @@ static void scalar32_min_max_arsh(struct bpf_reg_state *dst_reg,
static void scalar_min_max_arsh(struct bpf_reg_state *dst_reg,
struct bpf_reg_state *src_reg)
{
- u64 umin_val = reg_umin(src_reg);
+ u64 umin_val = reg_umin(src_reg), umax_val = reg_umax(src_reg);
+ s64 smin = reg_smin(dst_reg), smax = reg_smax(dst_reg);
- /* Upon reaching here, src_known is true and umax_val is equal
- * to umin_val.
- */
- reg_set_srange64(dst_reg, reg_smin(dst_reg) >> umin_val,
- reg_smax(dst_reg) >> umin_val);
+ /* A larger count brings the result closer to 0, or to -1 */
+ reg_set_srange64(dst_reg, smin >> (smin < 0 ? umin_val : umax_val),
+ smax >> (smax < 0 ? umax_val : umin_val));
dst_reg->var_off = tnum_arshift(dst_reg->var_off, umin_val, 64);
@@ -17401,22 +17402,19 @@ static bool is_safe_to_compute_dst_reg_range(struct bpf_insn *insn,
case BPF_END:
return true;
- /*
- * Division and modulo operators range is only safe to compute when the
- * divisor is a constant.
- */
+ /* The range of a signed division or modulo is computed for a constant divisor only */
case BPF_DIV:
case BPF_MOD:
- return src_is_const;
+ return src_is_const || !insn->off;
- /* Shift operators range is only computable if shift dimension operand
- * is a constant. Shifts greater than 31 or 63 are undefined. This
- * includes shifts by a negative number.
+ /*
+ * Shifts greater than 31 or 63 are undefined. This includes shifts by
+ * a negative number.
*/
case BPF_LSH:
case BPF_RSH:
case BPF_ARSH:
- return (src_is_const && reg_umax(src_reg) < insn_bitness);
+ return reg_umax(src_reg) < insn_bitness;
default:
return false;
}
@@ -17485,6 +17483,11 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
return sanitize_err(env, insn, ret);
}
+ /* The count of a shift is a range: a bit of dst_reg can end up in more than one place */
+ if ((opcode == BPF_LSH || opcode == BPF_RSH || opcode == BPF_ARSH) &&
+ reg_umin(&src_reg) != reg_umax(&src_reg))
+ dst_reg->var_off = tnum_unknown;
+
/* Calculate sign/unsigned bounds and tnum for alu32 and alu64 bit ops.
* There are two classes of instructions: The first class we track both
* alu32 and alu64 sign/unsigned bounds independently this provides the
@@ -17547,7 +17550,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
break;
case BPF_DIV:
/* BPF div specification: x / 0 = 0 */
- if ((alu32 && reg_u32_min(&src_reg) == 0) || (!alu32 && reg_umin(&src_reg) == 0)) {
+ if ((alu32 && reg_u32_max(&src_reg) == 0) || (!alu32 && reg_umax(&src_reg) == 0)) {
___mark_reg_known(dst_reg, 0);
break;
}
@@ -17565,7 +17568,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
break;
case BPF_MOD:
/* BPF mod specification: x % 0 = x */
- if ((alu32 && reg_u32_min(&src_reg) == 0) || (!alu32 && reg_umin(&src_reg) == 0))
+ if ((alu32 && reg_u32_max(&src_reg) == 0) || (!alu32 && reg_umax(&src_reg) == 0))
break;
if (alu32)
if (off == 1)
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 27/37] selftests/bpf: Test a shift, a division and a remainder by a range
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (25 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 26/37] bpf: Compute bounds for a shift, a division and a remainder by a range Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 28/37] bpf: Count a loop whose latch is false in the first iteration Alexei Starovoitov
` (9 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
For each op one prog needs both bounds of the result, in 64 and in 32 bits,
and expects the register as the log has it. Next to it are progs that get
to a bad read at run time if a bound is off by one, a bit is shifted out,
or the divisor is 0. The verifier must reject them. One prog has what stays
unknown: a count of 64 or 32, a signed division.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_alu_range.c | 168 ++++++++++++++++++
2 files changed, 170 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_alu_range.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 99f0551c8b59..a771dfbaeb3f 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -9,6 +9,7 @@
#include "cb_refs.skel.h"
#include "verifier_aggregate_arg.skel.h"
#include "verifier_aggregate_ret.skel.h"
+#include "verifier_alu_range.skel.h"
#include "verifier_align.skel.h"
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
@@ -204,6 +205,7 @@ void test_cb_refs(void) { RUN_TESTS(cb_refs); }
void test_verifier_aggregate_arg(void) { RUN_TESTS(verifier_aggregate_arg); }
void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret); }
+void test_verifier_alu_range(void) { RUN(verifier_alu_range); }
void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_alu_range.c b/tools/testing/selftests/bpf/progs/verifier_alu_range.c
new file mode 100644
index 000000000000..7bd13812941c
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_alu_range.c
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The count of a shift, or a divisor, is a range and not a constant:
+ * x >> [a, b] [xmin >> b, xmax >> a]
+ * x << [a, b] [xmin << a, xmax << b], if no bit is shifted out
+ * x s>> [a, b] a larger count brings x closer to 0, or to -1
+ * x / [a, b] [xmin / b, xmax / a], and x / 0 = 0
+ * x % [a, b] [0, b - 1], x if it is below a, and x % 0 = x
+ * Each program that has to be rejected gets to the bad read at run time.
+ */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#define OK(text) SEC("socket") __description("alu range: " text) __success __log_level(2)
+#define BAD(text) SEC("socket") __description("alu range: " text) __failure \
+ __msg("invalid read from stack")
+
+/* @reg is lo .. lo + mask */
+#define RANGE(reg, lo, mask) \
+ "call %[bpf_get_prandom_u32];" \
+ #reg " = r0;" \
+ #reg " &= " #mask ";" \
+ #reg " += " #lo ";"
+
+/* The read is not allowed */
+#define UNLESS(cond) "if " cond " goto +1; r0 = *(u64 *)(r10 + 8);"
+
+/* r6 is the second operand. r7 and r8 are x. */
+#define ALU(src_lo, src_mask, x_lo, x_mask, insns) \
+ asm volatile ( \
+ RANGE(r6, src_lo, src_mask) \
+ RANGE(r7, x_lo, x_mask) \
+ "r8 = r7;" \
+ insns \
+ "r0 = 0;" \
+ "exit;" \
+ :: __imm(bpf_get_prandom_u32) : __clobber_all)
+
+OK("[128, 255] >> [2, 5]")
+__msg("r7 >>= r6 {{.*}} R7=scalar(smin=umin=smin32=umin32=4,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))")
+__msg("w8 >>= w6 {{.*}} R8=scalar(smin=umin=smin32=umin32=4,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))")
+__naked void rsh(void)
+{
+ ALU(2, 3, 128, 127, "r7 >>= r6; w8 >>= w6;"
+ UNLESS("r7 >= 4") UNLESS("r7 <= 63") UNLESS("w8 >= 4") UNLESS("w8 <= 63"));
+}
+
+BAD("255 >> 2 is 63")
+__naked void rsh_max(void)
+{
+ ALU(2, 3, 128, 127, "r7 >>= r6;" UNLESS("r7 <= 62"));
+}
+
+OK("[3, 10] << [1, 4]")
+__msg("r7 <<= r6 {{.*}} R7=scalar(smin=umin=smin32=umin32=6,smax=umax=smax32=umax32=160,var_off=(0x0; 0xfe))")
+__msg("w8 <<= w6 {{.*}} R8=scalar(smin=umin=smin32=umin32=6,smax=umax=smax32=umax32=160,var_off=(0x0; 0xfe))")
+__naked void lsh(void)
+{
+ ALU(1, 3, 3, 7, "r7 <<= r6; w8 <<= w6;"
+ UNLESS("r7 >= 6") UNLESS("r7 <= 160") UNLESS("w8 >= 6") UNLESS("w8 <= 160"));
+}
+
+BAD("10 << 4 is 160, 32 bits")
+__naked void lsh32_max(void)
+{
+ ALU(1, 3, 3, 7, "w7 <<= w6;" UNLESS("w7 <= 159"));
+}
+
+BAD("4 << 62 is 0")
+__naked void lsh_overflow(void)
+{
+ ALU(59, 3, 4, 1, "r7 <<= r6;" UNLESS("r7 != 0"));
+}
+
+BAD("4 << 30 is 0, 32 bits")
+__naked void lsh32_overflow(void)
+{
+ ALU(27, 3, 4, 1, "w7 <<= w6;" UNLESS("w7 != 0"));
+}
+
+OK("[-128, -65] s>> [2, 5], [-128, 127] s>> [2, 5]")
+__msg("r7 s>>= r6 {{.*}} R7=scalar(smin=smin32=-32,smax=smax32=-3,")
+__msg("w8 s>>= w6 {{.*}} R8=scalar(smin=0,smax=umax=0xffffffff,smin32=-32,smax32=31,")
+__naked void arsh(void)
+{
+ ALU(2, 3, -128, 63, "r7 s>>= r6;" UNLESS("r7 s>= -32") UNLESS("r7 s<= -3")
+ RANGE(r8, -128, 255) "w8 s>>= w6;" UNLESS("w8 s>= -32") UNLESS("w8 s<= 31"));
+}
+
+BAD("-65 s>> 5 is -3")
+__naked void arsh_max(void)
+{
+ ALU(2, 3, -128, 63, "r7 s>>= r6;" UNLESS("r7 s<= -4"));
+}
+
+OK("[128, 255] / [2, 5], [128, 255] / [0, 3]")
+__msg("r7 /= r6 {{.*}} R7=scalar(smin=umin=smin32=umin32=25,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))")
+__msg("w8 /= w6 {{.*}} R8=scalar(smin=umin=smin32=umin32=25,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))")
+__msg("r9 /= r6 {{.*}} R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))")
+__naked void div(void)
+{
+ ALU(2, 3, 128, 127, "r9 = r7; r7 /= r6; w8 /= w6;"
+ UNLESS("r7 >= 25") UNLESS("r7 <= 127") UNLESS("w8 >= 25") UNLESS("w8 <= 127")
+ "r6 += -2; r9 /= r6;" UNLESS("r9 <= 255"));
+}
+
+BAD("128 / 5 is 25, 32 bits")
+__naked void div32_min(void)
+{
+ ALU(2, 3, 128, 127, "w7 /= w6;" UNLESS("w7 >= 26"));
+}
+
+BAD("x / 0 is 0")
+__naked void div_by_zero(void)
+{
+ ALU(0, 3, 128, 127, "r7 /= r6;" UNLESS("r7 >= 42"));
+}
+
+OK("[128, 255] % [2, 5], [128, 255] % [256, 259], [128, 255] % [0, 3]")
+__msg("r7 %= r6 {{.*}} R7=scalar(smin=smin32=0,smax=umax=smax32=umax32=4,var_off=(0x0; 0x7))")
+__msg("w8 %= w6 {{.*}} R8=scalar(smin=smin32=0,smax=umax=smax32=umax32=4,var_off=(0x0; 0x7))")
+__msg("r9 %= r6 {{.*}} R9=scalar(smin=umin=smin32=umin32=128,smax=umax=smax32=umax32=255,var_off=(0x80; 0x7f))")
+__msg("w9 %= w6 {{.*}} R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))")
+__naked void mod(void)
+{
+ ALU(2, 3, 128, 127, "r9 = r7; r7 %%= r6; w8 %%= w6;" UNLESS("r7 <= 4") UNLESS("w8 <= 4")
+ "r6 += 254; r9 %%= r6;" UNLESS("r9 >= 128") UNLESS("r9 <= 255")
+ "r6 += -256; w9 %%= w6;" UNLESS("w9 <= 255"));
+}
+
+BAD("129 % 5 is 4")
+__naked void mod_max(void)
+{
+ ALU(2, 3, 128, 127, "r7 %%= r6;" UNLESS("r7 <= 3"));
+}
+
+BAD("255 % 255 is 0")
+__naked void mod_same(void)
+{
+ ALU(255, 3, 128, 127, "r7 %%= r6;" UNLESS("r7 >= 128"));
+}
+
+BAD("x % 0 is x, 32 bits")
+__naked void mod32_by_zero(void)
+{
+ ALU(0, 3, 128, 127, "w7 %%= w6;" UNLESS("w7 <= 2"));
+}
+
+/* The cpu takes the count modulo 64 or 32, and a signed division can change the sign */
+OK("not known: a count of 64 or 32, a signed division")
+__msg("r7 >>= r6 {{.*}} R7=scalar()")
+__msg("w8 >>= w6 {{.*}} R8=scalar()")
+__msg("r7 s/= r6 {{.*}} R7=scalar()")
+__msg("w8 s%= w6 {{.*}} R8=scalar()")
+__naked void not_known(void)
+{
+ ALU(63, 1, 128, 127, "r7 >>= r6; r6 += -32; w8 >>= w6;"
+ RANGE(r7, 128, 127) "r8 = r7; r7 s/= r6; w8 s%%= w6;");
+}
+
+BAD("x >> 64 is x")
+__naked void rsh_64(void)
+{
+ ALU(63, 1, 128, 127, "r7 >>= r6;" UNLESS("r7 == 0"));
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 28/37] bpf: Count a loop whose latch is false in the first iteration
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (26 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 27/37] selftests/bpf: Test " Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down Alexei Starovoitov
` (8 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
for (i = first; i < 64; i++) {
if (i >= n)
break;
sum += a[i];
}
With first = 63 the latch 'i + 1 < 64' is false in the first iteration.
compute_backedge_taken_count() has no count for that: "the main pass would
do a single iteration anyway". That was true while the latch was the only
jump with a count. Now scalar evolution asks each jump that leaves the loop
and dominates the back edge, and the smallest count wins. Here only
'i >= n' is left, and n is not known: 4 billion iterations. The pointer
that moves with i leaves the array, and the verifier rejects the prog.
The back edge is not taken. That is a count of 0: for constants, and for
ranges in which no first value gets through the latch.
- The operands of the latch can be pointers. Then the numbers are offsets
from one base, and addresses have the order of signed offsets: one byte
in front of a map value is below a byte in it. latch_false_at_once()
uses this order. The formulas for constants still compare unsigned
offsets. They have no count for an offset that gets to the other side of
0.
- A loop with several back edges can go on through a different back edge
and get to this latch again. A latch that is false at once can move
towards the values for which it is true: 'i < 10' with i that goes down
from 20. Thus it counts as false in the first iteration only, and
latch_false_until() is not asked.
The 9 tests of verifier_scev.c with a latch that is false at once expect a
count of 1 now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 84 ++++++++++++++-----
.../selftests/bpf/progs/verifier_scev.c | 18 ++--
2 files changed, 72 insertions(+), 30 deletions(-)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index fd6abc41b946..9ddaae03ac35 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2274,11 +2274,49 @@ static bool is_signed_op(u8 op)
}
}
-static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound, u64 *result)
+/*
+ * Is the latch false in the first iteration, whatever the first value and the bound are in
+ * their ranges? For a signed comparison the sign bit is flipped, see struct latch_domain.
+ */
+static bool latch_false_at_once(u8 op, u64 imin, u64 imax, u64 bmin, u64 bmax)
+{
+ switch (op) {
+ case BPF_JLT:
+ case BPF_JSLT:
+ return imin >= bmax;
+ case BPF_JLE:
+ case BPF_JSLE:
+ return imin > bmax;
+ case BPF_JGT:
+ case BPF_JSGT:
+ return imax <= bmin;
+ case BPF_JGE:
+ case BPF_JSGE:
+ return imax < bmin;
+ case BPF_JNE:
+ return imin == bmax && imax == bmin;
+ default:
+ return false;
+ }
+}
+
+static bool compute_backedge_taken_count(u8 op, bool ptrs, u64 initial, u64 step, u64 bound,
+ u64 *result)
{
- u64 rem, diff, backedge_taken_count;
+ u64 rem, diff, backedge_taken_count, bias = is_signed_op(op) || ptrs ? 1ULL << 63 : 0;
bool up, down;
+ /*
+ * The back edge is not taken. That is a count, and a better one than that of
+ * another exit, which may be true for a long time.
+ * @ptrs: the numbers are offsets from one base. Addresses have the order of signed
+ * offsets, see compute_backedge_taken_max(). The formulas below give up on an offset
+ * that gets to the other side of 0.
+ */
+ if (latch_false_at_once(op, initial ^ bias, initial ^ bias, bound ^ bias, bound ^ bias)) {
+ *result = 0;
+ return true;
+ }
if (step == 0)
return false;
down = (s64)step < 0;
@@ -2293,13 +2331,10 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
* The backedge_taken_count is the size of this set.
*/
if (/*
- * If the condition is initially true, decreasing values won't
- * change that without underflow. If the condition is initially false,
- * the main pass would do a single iteration anyway.
+ * The condition is initially true, decreasing values won't
+ * change that without underflow.
*/
down ||
- /* Is the latch condition false on the first iteration? */
- initial >= bound ||
/* Count iterations satisfying the strict inequality, including i=0. */
check_add_overflow(div64_u64(diff - 1, step), 1, &backedge_taken_count) ||
/*
@@ -2311,7 +2346,7 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
break;
case BPF_JLE:
/* E.g.: 3 + 2*i ≤ 7 ⇔ 2*i ≤ 4 ⇔ i ∈ [0,1,2]. */
- if (down || initial > bound ||
+ if (down ||
/* Count iterations satisfying the non-strict inequality. */
check_add_overflow(div64_u64(diff, step), 1, &backedge_taken_count) ||
backedge_taken_count > div64_u64(U64_MAX - initial, step))
@@ -2319,7 +2354,7 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
break;
case BPF_JGT:
/* E.g.: 7 - 2*i > 3 ⇔ 2*i < 4 ⇔ i ∈ [0,1]. */
- if (up || initial <= bound ||
+ if (up ||
check_add_overflow(div64_u64(-diff - 1, -step), 1, &backedge_taken_count) ||
/*
* The last iteration underflows the counter if:
@@ -2333,13 +2368,13 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
break;
case BPF_JGE:
/* E.g.: 7 - 2*i ≥ 3 ⇔ 2*i ≤ 4 ⇔ i ∈ [0,1,2]. */
- if (up || initial < bound ||
+ if (up ||
check_add_overflow(div64_u64(-diff, -step), 1, &backedge_taken_count) ||
backedge_taken_count > div64_u64(initial, -step))
return false;
break;
case BPF_JSLT:
- if (down || (s64)initial >= (s64)bound ||
+ if (down ||
check_add_overflow(div64_u64(diff - 1, step), 1, &backedge_taken_count) ||
/*
* Check if the last iteration overflows the signed counter:
@@ -2349,13 +2384,13 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
return false;
break;
case BPF_JSLE:
- if (down || (s64)initial > (s64)bound ||
+ if (down ||
check_add_overflow(div64_u64(diff, step), 1, &backedge_taken_count) ||
backedge_taken_count > div64_u64((u64)S64_MAX - initial, step))
return false;
break;
case BPF_JSGT:
- if (up || (s64)initial <= (s64)bound ||
+ if (up ||
check_add_overflow(div64_u64(-diff - 1, -step), 1, &backedge_taken_count) ||
/*
* The last iteration underflows the signed counter if:
@@ -2368,7 +2403,7 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
return false;
break;
case BPF_JSGE:
- if (up || (s64)initial < (s64)bound ||
+ if (up ||
check_add_overflow(div64_u64(-diff, -step), 1, &backedge_taken_count) ||
backedge_taken_count > div64_u64(initial - (u64)S64_MIN, -step))
return false;
@@ -2381,7 +2416,7 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
diff = down ? -diff : diff;
step = down ? -step : step;
backedge_taken_count = div64_u64_rem(diff, step, &rem);
- if (diff == 0 || rem != 0)
+ if (rem != 0)
return false;
break;
default:
@@ -2573,6 +2608,10 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
if (initial->ptr_reg >= 0 &&
(d.imin < d.lo || d.imax > d.hi || d.bmin < d.lo || d.bmax > d.hi))
return false;
+ if (latch_false_at_once(latch->op, d.imin, d.imax, d.bmin, d.bmax)) {
+ *result = 0;
+ return true;
+ }
/*
* The value that ends the loop must not wrap around. It is less than a step
@@ -2581,7 +2620,7 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
switch (latch->op) {
case BPF_JLT:
case BPF_JSLT:
- if (down || d.imin >= d.bmax ||
+ if (down ||
check_add_overflow(div64_u64(d.bmax - d.imin - 1, step), 1, &cnt) ||
d.bmax > d.hi || step - 1 > d.hi - d.bmax)
return false;
@@ -2589,7 +2628,7 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
break;
case BPF_JLE:
case BPF_JSLE:
- if (down || d.imin > d.bmax ||
+ if (down ||
check_add_overflow(div64_u64(d.bmax - d.imin, step), 1, &cnt) ||
d.bmax > d.hi || step > d.hi - d.bmax)
return false;
@@ -2597,7 +2636,7 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
break;
case BPF_JGT:
case BPF_JSGT:
- if (!down || d.imax <= d.bmin ||
+ if (!down ||
check_add_overflow(div64_u64(d.imax - d.bmin - 1, step), 1, &cnt) ||
d.bmin < d.lo || step - 1 > d.bmin - d.lo)
return false;
@@ -2605,7 +2644,7 @@ static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_
break;
case BPF_JGE:
case BPF_JSGE:
- if (!down || d.imax < d.bmin ||
+ if (!down ||
check_add_overflow(div64_u64(d.imax - d.bmin, step), 1, &cnt) ||
d.bmin < d.lo || step > d.bmin - d.lo)
return false;
@@ -2766,7 +2805,8 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
}
- if (!compute_backedge_taken_count(op, initial, step, bound, &backedge_taken_count))
+ if (!compute_backedge_taken_count(op, initial_val.ptr_reg >= 0, initial, step, bound,
+ &backedge_taken_count))
return false;
/*
@@ -2789,7 +2829,9 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
}
counted:
- *false_until = latch_false_until(latch, &initial_val, &step_val, &bound_val,
+ /* A latch that is false at once may be on its way to the values for which it's true. */
+ *false_until = !backedge_taken_count ? 0 :
+ latch_false_until(latch, &initial_val, &step_val, &bound_val,
backedge_taken_count);
/* The header executes once on entry and once for each taken backedge, hence +1. */
iters->max_header_count = backedge_taken_count + 1;
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index f3908f37ca74..790cbfe1f2e4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -858,15 +858,15 @@ PRE__COND_TEST (pre_gt_sign, "<=", ITER_S64_MIN + 1, -1, ITER_S
* r6 += 2; // unreachable
* }
*/
-PRE__COND_TEST (pre_lt_false, ">=", 7, 2, 7, "can't compute iterations count")
-PRE__COND_TEST (pre_le_false, ">", 8, 2, 7, "can't compute iterations count")
-PRE__COND_TEST (pre_gt_false, "<=", 3, -2, 3, "can't compute iterations count")
-PRE__COND_TEST (pre_ge_false, "<", 2, -2, 3, "can't compute iterations count")
-PRE__COND_TEST (pre_slt_false, "s>=", 1, 2, 1, "can't compute iterations count")
-PRE__COND_TEST (pre_sle_false, "s>", 2, 2, 1, "can't compute iterations count")
-PRE__COND_TEST (pre_sgt_false, "s<=", -1, -2, -1, "can't compute iterations count")
-PRE__COND_TEST (pre_sge_false, "s<", -2, -2, -1, "can't compute iterations count")
-PRE__COND_TEST (pre_ne_false, "==", 7, 2, 7, "can't compute iterations count")
+PRE__COND_TEST (pre_lt_false, ">=", 7, 2, 7, "header_count is 1 ")
+PRE__COND_TEST (pre_le_false, ">", 8, 2, 7, "header_count is 1 ")
+PRE__COND_TEST (pre_gt_false, "<=", 3, -2, 3, "header_count is 1 ")
+PRE__COND_TEST (pre_ge_false, "<", 2, -2, 3, "header_count is 1 ")
+PRE__COND_TEST (pre_slt_false, "s>=", 1, 2, 1, "header_count is 1 ")
+PRE__COND_TEST (pre_sle_false, "s>", 2, 2, 1, "header_count is 1 ")
+PRE__COND_TEST (pre_sgt_false, "s<=", -1, -2, -1, "header_count is 1 ")
+PRE__COND_TEST (pre_sge_false, "s<", -2, -2, -1, "header_count is 1 ")
+PRE__COND_TEST (pre_ne_false, "==", 7, 2, 7, "header_count is 1 ")
/*
* The counter moves away from the exit; bound fallback verification.
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (27 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 28/37] bpf: Count a loop whose latch is false in the first iteration Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-04 0:26 ` sashiko-bot
2026-11-03 23:40 ` [PATCH bpf-next 30/37] bpf: Count a loop whose latch tests a flag that a jump sets Alexei Starovoitov
` (7 subsequent siblings)
36 siblings, 1 reply; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Compilers make these latches of loops that end at 0:
for (i = 63; i >= 0; i--) w6 += -1; if w6 != -1 goto l
n = 64; while (n--) w6 += 1; if w6 != 0 goto l
for (j = n - 1; j >= 0 && j < 128; j--) w6 += -1; if w6 < 128 goto l
None of them has a count. In each the variable goes across 0, and scalar
evolution looks at it as an unsigned number. As a signed number it is in
order, and the signed order has formulas.
'!=': the count for constants is calculated in 64 bits from operands that
are zero extended. From 62 down to 0xffffffff is a distance of almost 2^64.
When the first value is on the wrong side of the bound as an unsigned
number, take the operands as signed 32-bit numbers: 7 goes down to -1, -8
goes up to 0. The formula for ranges does that already. From 4 up to 1 is
wrong in both orders: there is still no count.
'<': the compiler makes one unsigned test of 'j >= 0 && j < 128', and '<'
has a count only for a variable that goes up. A variable that goes down
stays below the bound. The loop ends when it gets below 0, which is a large
unsigned number. So count the latch as 'j >= 0', signed, if the bound is
not negative as a signed number. A 64-bit jump on a 32-bit counter compares
all 64 bits of the bound: there the bound has to fit in 31 bits. 'j >= 0'
has a formula for constants and for ranges. Each value that the unsigned
test lets through is not negative. Thus the count is not too small,
whatever the first value is.
Only the count takes the signed form. In a loop with several back edges
same_latch() compares the jumps as the prog has them.
The verifier no longer verifies these loops one iteration at a time.
A compiler zero extends a 32-bit counter that goes down in the loop, not in
front of it. If the first value comes from a 64-bit register and only
a 32-bit jump limits it, nothing is known about j at the loop head:
for (u32 j = bpf_get_prandom_u32(); j < 128; j--)
An access at index j is rejected now. No prog of Cilium, scx, the selftests
or the public progs of Meta is lost.
pre_lt_dir, pre_le_dir, range_lt_dir and range_le_dir of verifier_scev.c go
down under an unsigned test. They have a count now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 34 ++++++++++++++++---
.../selftests/bpf/progs/verifier_scev.c | 22 +++++++++---
2 files changed, 47 insertions(+), 9 deletions(-)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 9ddaae03ac35..70d15449eebe 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2738,8 +2738,9 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
struct scev *scev = env->scev;
struct scev_value initial_val, step_val, bound_val;
u64 step, bound, initial, total_change, backedge_taken_count;
- bool konst;
+ struct linear_latch counted;
u8 op = latch->op;
+ bool konst, sign32 = latch->jmp32 && is_signed_op(op);
iters->iv_reg = -1;
@@ -2761,6 +2762,23 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
if (!scev_value_is_const(&step_val))
return false;
+ /*
+ * 'i >= 0 && i < 128' is one unsigned test. A counter that goes down stays below 128.
+ * The loop ends when it gets below 0. A 64-bit jump compares all 64 bits of the bound.
+ * The latch of the caller stays as it is: same_latch() compares what the prog tests.
+ */
+ if ((op == BPF_JLT || op == BPF_JLE) && initial_val.ptr_reg < 0 &&
+ (latch->jmp32 || latch->zext32 ?
+ (s32)step_val.umin < 0 && initial_val.u32_min < bound_val.u32_max &&
+ (latch->jmp32 ? bound_val.u32_max : bound_val.umax) <= S32_MAX :
+ (s64)step_val.umin < 0 && initial_val.umin < bound_val.umax &&
+ bound_val.umax <= S64_MAX)) {
+ counted = *latch;
+ latch = &counted;
+ op = latch->op = BPF_JSGE;
+ bound_val = (struct scev_value){ .ptr_reg = -1 };
+ sign32 = true;
+ }
konst = scev_value_is_const(&initial_val) && scev_value_is_const(&bound_val);
if (!konst) {
if (!compute_backedge_taken_max(env, st, latch, &initial_val, &step_val, &bound_val,
@@ -2785,7 +2803,7 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
* With the above constraint doing `l: w0 += 1; if r0 s< 256 goto l;`
* is equivalent to `l: w0 += 1; if r0 < 256 goto l;`
*/
- if (latch->jmp32 && is_signed_op(op)) {
+ if (sign32) {
initial = (s64)(s32)(u32)initial;
bound = (s64)(s32)(u32)bound;
step = (s64)(s32)(u32)step;
@@ -2803,7 +2821,15 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
"scev: mixed-width latch has operands outside 32-bit ranges\n");
return false;
}
-
+ /*
+ * 7 goes down to -1, -8 goes up to 0: on the wrong side of the bound as unsigned
+ * numbers, on the right side as signed ones. From 4 up to 1 is wrong both ways.
+ */
+ if (op == BPF_JNE && latch->zext32 && ((s64)step < 0 ? initial < bound : initial > bound)) {
+ initial = (s64)(s32)(u32)initial;
+ bound = (s64)(s32)(u32)bound;
+ sign32 = true;
+ }
if (!compute_backedge_taken_count(op, initial_val.ptr_reg >= 0, initial, step, bound,
&backedge_taken_count))
@@ -2819,7 +2845,7 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
if (latch->zext32 || latch->jmp32) {
/* total_change is guaranteed not to overflow at this point */
total_change = initial + step * backedge_taken_count;
- if (latch->jmp32 && is_signed_op(op)) {
+ if (sign32) {
if ((s64)total_change < S32_MIN || (s64)total_change > S32_MAX)
return false;
} else {
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 790cbfe1f2e4..5cc886b92913 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -869,7 +869,7 @@ PRE__COND_TEST (pre_sge_false, "s<", -2, -2,
PRE__COND_TEST (pre_ne_false, "==", 7, 2, 7, "header_count is 1 ")
/*
- * The counter moves away from the exit; bound fallback verification.
+ * An unsigned counter goes down and leaves the range below 0.
*
* Example: pre_lt_dir
*
@@ -878,8 +878,19 @@ PRE__COND_TEST (pre_ne_false, "==", 7, 2,
* r6 -= 2; // r6 ∈ [3,1,U64_MAX]
* }
*/
-PRE__COND_TEST2(pre_lt_dir, ">=", 5, -2, 7, "can't compute iterations count")
-PRE__COND_TEST2(pre_le_dir, ">", 5, -2, 7, "can't compute iterations count")
+PRE__COND_TEST2(pre_lt_dir, ">=", 5, -2, 7, "header_count is [0..4] ")
+PRE__COND_TEST2(pre_le_dir, ">", 5, -2, 7, "header_count is [0..4] ")
+
+/*
+ * The counter moves away from the exit; bound fallback verification.
+ *
+ * Example: pre_gt_dir
+ *
+ * u64 r6 = 5, r9 = 0;
+ * while (++r9 < 8 && r6 > 3) { // r6 ∈ [5,7,...,19]
+ * r6 += 2; // r6 ∈ [7,9,...,19]
+ * }
+ */
PRE__COND_TEST2(pre_gt_dir, "<=", 5, 2, 3, "can't compute iterations count")
PRE__COND_TEST2(pre_ge_dir, "<", 5, 2, 3, "can't compute iterations count")
PRE__COND_TEST2(pre_slt_dir, "s>=", -1, -2, 1, "can't compute iterations count")
@@ -1128,9 +1139,10 @@ RANGE_TEST(range32_ne_far, "", M32 - 5, M32 - 3, 1, M32 - 1, M32 + 3,
/* the first jump tests what enters the loop, and that is not zero extended */
RANGE_TEST(range32_first_is_wide, "", 0, M32 + 9, 2, 15, 20,
LOOP_HEAD "if r6 >= r7 goto 2f; w6 += w8; goto 1b;", NO_COUNT)
+/* An unsigned counter goes down and leaves the range below 0. */
+RANGE_TEST(range_lt_dir, "", 3, 10, -2, 15, 20, PRE(">="), COUNTED(7, "-2..10"))
+RANGE_TEST(range_le_dir, "", 3, 10, -2, 15, 20, PRE(">"), COUNTED(7, "-2..10"))
/* The counter moves away from the exit. */
-RANGE_TEST(range_lt_dir, "", 3, 10, -2, 15, 20, PRE(">="), NO_COUNT)
-RANGE_TEST(range_le_dir, "", 3, 10, -2, 15, 20, PRE(">"), NO_COUNT)
RANGE_TEST(range_gt_dir, "", 10, 20, 2, 3, 5, PRE("<="), NO_COUNT)
RANGE_TEST(range_ge_dir, "", 10, 20, 2, 3, 5, PRE("<"), NO_COUNT)
/* '!=' can step over the bound, or start behind it. */
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 30/37] bpf: Count a loop whose latch tests a flag that a jump sets
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (28 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 31/37] selftests/bpf: Test more latches of counted loops Alexei Starovoitov
` (6 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A compiler makes a flag of 'while (n--)', 'for (i = n; i-- > 0;)' and
'for (i = n - 1; i >= 0; i--)' when the body of the loop is not small:
w6 += 1
w2 = 1
if w6 == 0 goto l
w2 = 0
l: ...
if w2 != 1 goto head
The jump that leaves the loop looks at 1 or 0. The jump that counts stays
in the loop on both edges, so scalar evolution does not ask it for a count.
bpf_edge_leaves_loop() follows an edge of a jump. It goes through the
constants that the insns next to the jump set, plain insns, calls, 'goto',
and jumps that compare such a constant with a constant. If that way leaves
the loop, the jump is a condition for the back edge like a jump that leaves
the loop itself.
- There is one way to go from the edge, so no other path has to be
examined.
- What is set in front of the jump counts only as far back as nothing else
leads to the insns in between.
- A call keeps r6 - r9. r0 - r5 are not known after it.
- latches[0] stays a jump that leaves the loop itself.
bpf_jump_leaves_loop() tells whether a jump leaves a loop itself.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 3 ++
kernel/bpf/loops.c | 99 +++++++++++++++++++++++++++++++++---
kernel/bpf/scev.c | 5 +-
3 files changed, 95 insertions(+), 12 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 9a0ed5610650..dfecdfac75ff 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -727,6 +727,7 @@ struct bpf_backedge {
* Conditional jumps within the loop that dominate 'from' and have
* a successor outside the loop, nearest first. latches[0] is the
* latch; the remaining entries are additional exit conditions.
+ * One of them may leave the loop by way of a flag, see bpf_edge_leaves_loop().
* Each usable condition tells iterations in which the back edge is
* not taken, so keeping only a subset is sound.
*/
@@ -1504,6 +1505,8 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
int mark_chain_precision(struct bpf_verifier_env *env, int regno);
int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
+bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
int bpf_split_cur_state(struct bpf_verifier_env *env);
int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index e584549bd208..cc15712f70d4 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -304,28 +304,95 @@ int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx)
return aux[idx].loop ? idx : aux[idx].loop_header;
}
+/* @known: the registers that hold the constant @val behind @insn. False: a jump or the like. */
+static bool track_consts(const struct bpf_insn *insn, u16 *known, s32 *val)
+{
+ u8 class = BPF_CLASS(insn->code);
+
+ if (class == BPF_ST || (class == BPF_STX && BPF_MODE(insn->code) == BPF_MEM))
+ return true;
+ /* r6 - r9 are what they were when the call returns */
+ if (insn->code == (BPF_JMP | BPF_CALL)) {
+ *known &= ~GENMASK(BPF_REG_5, BPF_REG_0);
+ return true;
+ }
+ if (class != BPF_ALU && class != BPF_ALU64 && class != BPF_LDX)
+ return false;
+ *known &= ~BIT(insn->dst_reg);
+ if (class != BPF_LDX && BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_K) {
+ *known |= BIT(insn->dst_reg);
+ val[insn->dst_reg] = insn->imm;
+ }
+ return true;
+}
+
+/*
+ * Does the program leave the loop at @h when the conditional jump at @n goes on at @t?
+ * It does when @t is not in the loop. A compiler also makes
+ * w2 = 1; if w1 == 0 goto l; w2 = 0; l: ...; if w2 != 1 goto head;
+ * of 'while (n--)'. So follow the constants that are set next to the jump for as long as there
+ * is one way to go.
+ */
+bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t)
+{
+ struct bpf_insn *insns = env->prog->insnsi, *insn;
+ s32 val[MAX_BPF_REG];
+ u16 known = 0;
+ int i, l;
+ u8 op;
+
+ /* the insns in front of the jump, for as long as nothing else leads to the next one */
+ for (i = n; i && i > n - 4 && !env->insn_aux_data[i].loop && env->idoms[i] == i - 1; i--)
+ ;
+ for (; i < n; i++)
+ if (!track_consts(&insns[i], &known, val))
+ known = 0;
+ for (i = 0; i < 16; i++, t++) {
+ l = bpf_loop_at_index(env, t);
+ if (l != h && !bpf_is_nested_loop(env, l, h))
+ return true;
+ insn = &insns[t];
+ op = BPF_OP(insn->code);
+ if (t == h) {
+ return false;
+ } else if (insn->code == (BPF_JMP | BPF_JA)) {
+ t += insn->off;
+ } else if (is_cond_jmp_insn(insn)) {
+ /* a 32-bit and a 64-bit jump agree on a number that is not negative */
+ if (!(known & BIT(insn->dst_reg)) || val[insn->dst_reg] < 0 ||
+ BPF_SRC(insn->code) != BPF_K || (op != BPF_JEQ && op != BPF_JNE))
+ return false;
+ if ((val[insn->dst_reg] == insn->imm) == (op == BPF_JEQ))
+ t += insn->off;
+ } else if (!track_consts(insn, &known, val)) {
+ return false;
+ }
+ }
+ return false;
+}
+
/*
- * Collect conditional jumps within n's loop that dominate 'n' and have
- * a successor outside the loop, nearest first.
+ * Collect conditional jumps within n's loop that dominate 'n' and leave
+ * the loop on one edge, see bpf_edge_leaves_loop(), nearest first.
*/
static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
struct bpf_backedge *backedge)
{
struct bpf_insn *insns = env->prog->insnsi;
bool t_exits_loop, f_exits_loop;
- int n_loop, t_loop, f_loop;
int *idoms = env->idoms;
+ int n_loop;
n_loop = bpf_loop_at_index(env, n);
if (idoms_intersect(env, n, n_loop) != n_loop)
return;
while (n >= 0) {
if (is_cond_jmp_insn(&insns[n]) && bpf_loop_at_index(env, n) == n_loop) {
- t_loop = bpf_loop_at_index(env, n + insns[n].off + 1);
- f_loop = bpf_loop_at_index(env, n + 1);
- t_exits_loop = t_loop != n_loop && !bpf_is_nested_loop(env, t_loop, n_loop);
- f_exits_loop = f_loop != n_loop && !bpf_is_nested_loop(env, f_loop, n_loop);
- if (t_exits_loop || f_exits_loop) {
+ t_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + insns[n].off + 1);
+ f_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + 1);
+ /* latches[0] is the latch: it has to leave the loop itself */
+ if ((t_exits_loop || f_exits_loop) &&
+ (backedge->latches_cnt || bpf_jump_leaves_loop(env, n_loop, n))) {
if (backedge->latches_cnt == MAX_LATCHES)
return;
backedge->latches[backedge->latches_cnt++] = n;
@@ -457,6 +524,22 @@ bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int oute
return false;
}
+/* Whether the conditional jump at @n, in the loop @h or in a loop nested in it, leaves @h */
+bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[n];
+ u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+ int t, f;
+
+ if ((class != BPF_JMP && class != BPF_JMP32) || op == BPF_JA || op == BPF_CALL ||
+ op == BPF_EXIT || op == BPF_JCOND)
+ return false;
+ t = bpf_loop_at_index(env, n + insn->off + 1);
+ f = bpf_loop_at_index(env, n + 1);
+ return (t != h && !bpf_is_nested_loop(env, t, h)) ||
+ (f != h && !bpf_is_nested_loop(env, f, h));
+}
+
int bpf_compute_loops(struct bpf_verifier_env *env)
{
int i, j, k, s, t, iloop, sloop, err = 0, len = env->prog->len;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 70d15449eebe..a5fd080a3bc0 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2025,7 +2025,6 @@ static int match_linear_latch(struct bpf_verifier_env *env,
struct bpf_insn *insn = &env->prog->insnsi[latch_idx];
struct scev *scev = env->scev;
struct env *latch_env;
- u32 true_branch_loop;
u32 true_branch_tgt;
u32 src_reg_scev;
u32 dst_reg_scev;
@@ -2036,9 +2035,7 @@ static int match_linear_latch(struct bpf_verifier_env *env,
latch->jmp32 = BPF_CLASS(insn->code) == BPF_JMP32;
/* Flip the condition if true branch jumps out of the loop. */
true_branch_tgt = latch_idx + bpf_jmp_offset(insn) + 1;
- true_branch_loop = bpf_loop_at_index(env, true_branch_tgt);
- if (true_branch_loop != header &&
- !bpf_is_nested_loop(env, true_branch_loop, header))
+ if (bpf_edge_leaves_loop(env, header, latch_idx, true_branch_tgt))
op = bpf_rev_opcode(op);
switch (op) {
case BPF_JSLT:
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 31/37] selftests/bpf: Test more latches of counted loops
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (29 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 30/37] bpf: Count a loop whose latch tests a flag that a jump sets Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
` (5 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
verifier_loop_latch.c has the latches in the forms that a compiler makes.
Each iteration stores a byte to a map value of 2048 bytes and moves the
pointer by one. The progs that load expect the count in the log. In the
progs that the verifier must reject, a store is behind the value at run
time.
- The latch is false in the first iteration, for a constant and for all of
a range, next to a second exit with a count of 4 billion. The same with
two iterations. A latch that compares pointers, also with the first one
in front of the value.
- '!=' in 32 bits, down to -1 and up to 0: 2048 iterations, and one too
many.
- One unsigned test on a counter that goes down: in 32 and 64 bits, from a
constant and from a range, and one iteration too many. Two loops that do
not end below 0: a 64-bit jump on a 32-bit counter with a bound above
32 bits, and a second back edge under a signed 64-bit jump.
- The latch tests a flag: alone, with a store next to it, with a call
between the two jumps, and one iteration too many. In 5 progs the loop
goes on when the counter gets to 0: a call overwrites the flag, the flag
is set again, is changed, is set again on one of two paths, or there is
a second way to the jump that counts.
verifier_scev.c:
- pre_lt_dir_high goes down under an unsigned test and starts above
S64_MAX, where the rule does not hold.
- 5 rows for '!=' across 0. A step that misses the bound and the way from
4 up to 1 have no count.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_loop_latch.c | 165 ++++++++++++++++++
.../selftests/bpf/progs/verifier_scev.c | 9 +
3 files changed, 176 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_latch.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index a771dfbaeb3f..574b06c383fa 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -78,6 +78,7 @@
#include "verifier_liveness_exp.skel.h"
#include "verifier_load_acquire.skel.h"
#include "verifier_loop_hierarchy.skel.h"
+#include "verifier_loop_latch.skel.h"
#include "verifier_loops1.skel.h"
#include "verifier_lwt.skel.h"
#include "verifier_map_in_map.skel.h"
@@ -272,6 +273,7 @@ void test_verifier_linked_scalars(void) { RUN(verifier_linked_scalars); }
void test_verifier_live_stack(void) { RUN(verifier_live_stack); }
void test_verifier_liveness_exp(void) { RUN(verifier_liveness_exp); }
void test_verifier_loop_hierarchy(void) { RUN(verifier_loop_hierarchy); }
+void test_verifier_loop_latch(void) { RUN(verifier_loop_latch); }
void test_verifier_loops1(void) { RUN(verifier_loops1); }
void test_verifier_lwt(void) { RUN(verifier_lwt); }
void test_verifier_map_in_map(void) { RUN(verifier_map_in_map); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
new file mode 100644
index 000000000000..1fcefce9ab01
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -0,0 +1,165 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The jumps that tell how many iterations a loop has, in the forms that a compiler makes of
+ * for (i = first; i < N; i++) from the last index
+ * for (i = N - 1; i >= 0; i--) while (n--) for (j = N - 1; j >= 0 && j < N; j--)
+ *
+ * Each iteration stores a byte and moves the pointer by one. The value has NR bytes. The programs
+ * that load expect the count in the log. In those that have to be rejected a store is behind the
+ * value at run time.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#define NR 2048
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, char[NR]);
+} map SEC(".maps");
+
+#define OK(text, count) \
+ SEC("socket") __description("latch: " text) __success __log_level(2) \
+ __msg("header_count is " count)
+#define BAD(text) \
+ SEC("socket") __description("latch: " text) __failure __msg("invalid access to map value")
+/* the pointer goes on and on */
+#define NO_COUNT(text) \
+ SEC("socket") __description("latch: " text) __failure
+
+/* r6: the value, r7: a number that is not known. @init, then the loop. */
+#define LOOP(name, init, body) \
+__naked void name(void) \
+{ \
+ asm volatile (" \
+ r1 = 0; \
+ *(u32 *)(r10 - 4) = r1; \
+ r2 = r10; \
+ r2 += -4; \
+ r1 = %[map] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto 9f; \
+ r6 = r0; \
+ call %[bpf_get_prandom_u32]; \
+ w7 = w0; \
+" init " \
+1: r1 = 0; \
+ *(u8 *)(r6 + 0) = r1; \
+ r6 += 1; \
+" body " \
+9: r0 = 0; \
+ exit; \
+" : \
+ : __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+ : __clobber_all); \
+}
+
+/*
+ * From the last index the latch is false in the first iteration. That is a count. The other way
+ * out of the loop has one too: 4 billion.
+ */
+#define UP_TO_8 "if r8 >= r7 goto 9f; r8 += 1; if r8 < 8 goto 1b;"
+
+OK("false at once, and another exit", "[0..1]")
+LOOP(false_at_once, "r8 = 7; r6 += 2047;", UP_TO_8)
+
+BAD("two iterations, and another exit")
+LOOP(false_in_second, "r8 = 6; r6 += 2047;", UP_TO_8)
+
+OK("false at once for all of a range", "[0..1]")
+LOOP(false_at_once_range, "r8 = r7; r8 &= 3; r8 += 7; r6 += 2047;", UP_TO_8)
+
+BAD("false at once for all of a range but one")
+LOOP(false_at_once_range_bad, "r8 = r7; r8 &= 3; r8 += 6; r6 += 2047;", UP_TO_8)
+
+/*
+ * The latch compares pointers. One byte in front of the value is less than a byte in it,
+ * although -1 is a large unsigned offset: the latch is true in the first iteration.
+ */
+#define PTR_UP(off) "r8 += 1; r1 = r8; r1 += " #off "; if r1 < r9 goto 1b;"
+
+OK("pointers", "8")
+LOOP(ptr_up, "r8 = r6; r9 = r6; r9 += 8; r6 += 2040;", PTR_UP(0))
+
+BAD("pointers, the first one is in front of the value")
+LOOP(ptr_up_from_below, "r8 = r6; r9 = r6; r9 += 7; r6 += 2040;", PTR_UP(-2))
+
+/* an int goes down to -1, or up from -N to 0: the distance is a 32-bit number */
+#define DOWN_TO_M1 "w8 += -1; if w8 != -1 goto 1b;"
+#define UP_TO_0 "w8 += 1; if w8 != 0 goto 1b;"
+
+OK("down to -1 in 32 bits", "2048")
+LOOP(down_to_m1, "w8 = 2047;", DOWN_TO_M1)
+
+BAD("down to -1 in 32 bits, one too many")
+LOOP(down_to_m1_bad, "w8 = 2048;", DOWN_TO_M1)
+
+OK("up to 0 in 32 bits", "2048")
+LOOP(up_to_0, "w8 = -2048;", UP_TO_0)
+
+BAD("up to 0 in 32 bits, one too many")
+LOOP(up_to_0_bad, "w8 = -2049;", UP_TO_0)
+
+/* 'j >= 0 && j < N' is one unsigned test */
+OK("down, one unsigned test", "2048")
+LOOP(down_unsigned, "w8 = 2047;", "w8 += -1; if w8 < 2048 goto 1b;")
+
+BAD("down, one unsigned test, one too many")
+LOOP(down_unsigned_bad, "w8 = 2048;", "w8 += -1; if w8 < 2049 goto 1b;")
+
+OK("down, one unsigned test, 64 bits", "2048")
+LOOP(down_unsigned64, "r8 = 2047;", "r8 += -1; if r8 <= 2047 goto 1b;")
+
+OK("down from a range, one unsigned test", "[0..2048]")
+LOOP(down_unsigned_range, "if r7 > 2047 goto 9f;", "w7 += -1; if w7 < 2048 goto 1b;")
+
+BAD("down from a range, one unsigned test, one too many")
+LOOP(down_unsigned_range_bad, "if r7 > 2048 goto 9f;", "w7 += -1; if w7 < 2049 goto 1b;")
+
+/* below 0 a 32-bit counter is 0xffffffff for a 64-bit jump */
+BAD("down, a 64-bit test with a bound above 32 bits")
+LOOP(down_unsigned_wide_bound, "w8 = 2047; r9 = 0x100000800 ll;", "w8 += -1; if r8 < r9 goto 1b;")
+
+BAD("down, an unsigned and a signed 64-bit test")
+LOOP(down_two_tests, "w8 = 4; r6 += 2043;",
+ "w8 += -1; if r8 < 5 goto 2f; if r8 s>= 5 goto 1b; goto 9f; 2: if r8 < 5 goto 1b;")
+
+/* 'while (n--)': the jump that counts sets a flag, the jump that leaves the loop tests it */
+#define FLAG(mid) "w8 += 1; w2 = 1; if w8 == 0 goto 2f; w2 = 0; 2: " mid " if w2 != 1 goto 1b;"
+
+OK("a flag", "2048")
+LOOP(flag, "w8 = -2048;", FLAG("r3 = r6;"))
+
+BAD("a flag, one too many")
+LOOP(flag_bad, "w8 = -2049;", FLAG("r3 = r6;"))
+
+OK("a flag, a store behind it", "2048")
+LOOP(flag_store, "w8 = -2048;",
+ "w8 += 1; w2 = 1; *(u32 *)(r10 - 4) = r8; if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
+
+OK("a flag, a call behind it", "2048")
+LOOP(flag_call, "w8 = -2048;",
+ "w8 += 1; w9 = 1; if w8 == 0 goto 2f; w9 = 0; 2: call %[bpf_get_prandom_u32]; if w9 != 1 goto 1b;")
+
+/* in these the loop goes on when the counter gets to 0 */
+NO_COUNT("a flag that a call overwrites")
+LOOP(flag_call_r0, "w8 = -2048;",
+ "w8 += 1; w0 = 1; if w8 == 0 goto 2f; w0 = 0; 2: call %[bpf_get_prandom_u32]; if w0 != 1 goto 1b;")
+
+NO_COUNT("a flag that is set again")
+LOOP(flag_set_again, "w8 = -2048;", FLAG("w2 = 0;"))
+
+NO_COUNT("a flag that is changed")
+LOOP(flag_changed, "w8 = -2048;", FLAG("w2 &= w7;"))
+
+NO_COUNT("a flag that is set again on one path")
+LOOP(flag_other_jump, "w8 = -2048;", FLAG("if r7 != 0 goto 4f; w2 = 0; 4:"))
+
+NO_COUNT("a flag, another way to the jump that counts")
+LOOP(flag_two_ways, "w8 = -2048;",
+ "w8 += 1; w2 = 0; if r7 != 0 goto 3f; w2 = 1; 3: if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 5cc886b92913..07c356fde68d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -877,9 +877,12 @@ PRE__COND_TEST (pre_ne_false, "==", 7, 2,
* while (++r9 <= 8 && r6 < 7) { // r6 ∈ [5,3,1,U64_MAX]
* r6 -= 2; // r6 ∈ [3,1,U64_MAX]
* }
+ *
+ * From above S64_MAX it's a long way down.
*/
PRE__COND_TEST2(pre_lt_dir, ">=", 5, -2, 7, "header_count is [0..4] ")
PRE__COND_TEST2(pre_le_dir, ">", 5, -2, 7, "header_count is [0..4] ")
+PRE__COND_TEST2(pre_lt_dir_high, ">=", ITER_S64_MIN + 5, -2, ITER_S64_MIN + 7, "can't compute iterations count")
/*
* The counter moves away from the exit; bound fallback verification.
@@ -1011,6 +1014,12 @@ POST_COND_TEST32(post32_le, "<=", -9, 2,
POST_COND_TEST32(post32_gt, ">", -1, -2u, -7, "header_count is [0..3] ")
POST_COND_TEST32(post32_ge, ">=", -1, -2u, -7, "header_count is [0..4] ")
POST_COND_TEST32(post32_ne, "!=", -1, -2u, -7, "header_count is [0..3] ")
+/* across 0: in order as signed numbers. From 4 up to 1 goes all the way around. */
+POST_COND_TEST32(post32_ne_to_m1, "!=", 3, -1u, -1, "header_count is [0..4] ")
+POST_COND_TEST32(post32_ne_to_0, "!=", -4, 1, 0, "header_count is [0..4] ")
+POST_COND_TEST32(post32_ne_to_m1_2, "!=", 3, -2u, -1, "header_count is [0..2] ")
+POST_COND_TEST32(post32_ne_miss, "!=", 4, -2u, -1, "can't compute iterations count")
+POST_COND_TEST32(post32_ne_around, "!=", 4, 1, 1, "can't compute iterations count")
/* An exit exactly at S32_MAX is accepted as well. */
POST_COND_TEST32(post32_slt_max, "s<", INT32_MAX - 6, 2, INT32_MAX, "header_count is [0..3] ")
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (30 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 31/37] selftests/bpf: Test more latches of counted loops Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-04 0:12 ` sashiko-bot
2026-11-03 23:40 ` [PATCH bpf-next 33/37] bpf: Keep the frame of analyze_subprog() small Alexei Starovoitov
` (4 subsequent siblings)
36 siblings, 1 reply; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
'tbl[i]()' loads a pointer from a table of functions in .rodata. When i is
not a constant, the verifier continues with one state for each pointer that
the load can give. It rejects a load that can give more than 64 pointers.
Rust needs more. In one prog of rust-bpf, fmt() of a trait object has 245
implementations.
The limit of 64 is arbitrary. Raise it to BPF_MAX_SUBPROGS. A table that
contains each function one time cannot have more entries. The array of
targets is now too large for the kernel stack, thus allocate it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 93ca0126a310..d503dc7cea1f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -7068,7 +7068,7 @@ static void mark_reg_func_ptr(struct bpf_verifier_env *env, struct bpf_reg_state
}
/* a read from a table of functions branches into that many states at most */
-#define BPF_MAX_FUNC_PTR_TARGETS 64
+#define BPF_MAX_FUNC_PTR_TARGETS BPF_MAX_SUBPROGS
/* and that many pointers at most are next to what is read at variable offset */
#define BPF_MAX_FUNC_PTR_RANGE 4096
@@ -7114,17 +7114,20 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
struct bpf_reg_state *regs = cur_regs(env);
struct bpf_map *map = reg->map_ptr;
struct bpf_verifier_state *branch;
- int subprog, targets[BPF_MAX_FUNC_PTR_TARGETS];
+ u16 *targets __free(kfree) = NULL;
struct bpf_func_ptr *ptrs;
u32 i, cnt, n = 0;
u64 o, lo, hi;
- int err;
+ int err, subprog;
ptrs = bpf_map_range_func_ptrs(env, map, min_off, max_off - min_off + size, &cnt);
if (!ptrs)
return -ENOENT;
if (cnt > BPF_MAX_FUNC_PTR_RANGE)
goto overlap;
+ targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), GFP_KERNEL_ACCOUNT);
+ if (!targets)
+ return -ENOMEM;
for (i = 0; i < cnt; i++) {
/* the offsets at which the read overlaps with the pointer */
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 33/37] bpf: Keep the frame of analyze_subprog() small
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (31 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 34/37] bpf: Allow 24 arguments of static subprogs Alexei Starovoitov
` (3 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
analyze_subprog() follows each chain of calls that passes a pointer to the
stack, and calls itself for each frame of the chain. Its own frame has
callee_args[], with one entry for each register and each stack argument.
"bpf: Allow 24 arguments of static subprogs" makes the array larger.
Move the array out of the frame. bpf_compute_subprog_arg_access() allocates
MAX_AT_TRACK_REGS entries for each depth one time, and frees them at its
end.
compute_subprog_args() has two more arrays of this kind, at_out[] and
at_stack_arg_entry[]. It has one caller, thus a compiler can inline it into
analyze_subprog(). Mark it noinline_for_stack. With clang 18 on arm64 and
24 arguments the frame of analyze_subprog() is 256 bytes and not 944 bytes.
The 16 frames of the longest chain take 4K of the 16K of the kernel stack
and not 14.75K.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/liveness.c | 34 ++++++++++++++++++++++++----------
1 file changed, 24 insertions(+), 10 deletions(-)
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index f51445ead384..bffa490318ce 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -70,10 +70,14 @@ struct live_stack_query {
u32 insn_idx;
};
+struct arg_track;
+
struct bpf_liveness {
DECLARE_HASHTABLE(func_instances, 8); /* maps (depth, callsite) to func_instance */
struct live_stack_query live_stack_query; /* cache to avoid repetitive ht lookups */
u32 subprog_calls; /* analyze_subprog() invocations */
+ /* MAX_AT_TRACK_REGS for every depth: analyze_subprog() calls itself, keep its frame small */
+ struct arg_track *callee_args;
};
/*
@@ -838,7 +842,7 @@ struct spill_snapshot {
/*
* Combined register + stack arg tracking: R0-R10 at indices 0-10,
- * outgoing stack arg slots at indices MAX_BPF_REG..MAX_BPF_REG+6.
+ * outgoing stack arg slots at indices MAX_BPF_REG..MAX_AT_TRACK_REGS-1.
*/
#define MAX_AT_TRACK_REGS (MAX_BPF_REG + MAX_STACK_ARG_SLOTS)
@@ -1970,12 +1974,14 @@ static void record_stack_ptrs(struct bpf_verifier_env *env, int idx, struct arg_
* @callee_entry: pre-populated entry state for R1-R5 and stack args
* NULL for main (subprog 0).
* @info: stores at_in, len for debug printing.
+ *
+ * Not inlined: at_out[] and at_stack_arg_entry[] would be in each frame of analyze_subprog().
*/
-static int compute_subprog_args(struct bpf_verifier_env *env,
- struct subprog_at_info *info,
- struct arg_track *callee_entry,
- struct func_instance *instance,
- u32 *callsites)
+static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
+ struct subprog_at_info *info,
+ struct arg_track *callee_entry,
+ struct func_instance *instance,
+ u32 *callsites)
{
int subprog = instance->subprog;
struct bpf_insn *insns = env->prog->insnsi;
@@ -2308,7 +2314,8 @@ static int analyze_subprog(struct bpf_verifier_env *env,
/* For each reachable call site in the subprog, recurse into callees */
for (int p = po_start; p < po_end; p++) {
int idx = env->cfg.insn_postorder[p];
- struct arg_track callee_args[MAX_AT_TRACK_REGS] = {};
+ struct arg_track *callee_args = env->liveness->callee_args +
+ depth * MAX_AT_TRACK_REGS;
struct arg_track none = { .frame = ARG_NONE };
struct bpf_insn *insn = &insns[idx];
struct func_instance *callee_instance;
@@ -2316,6 +2323,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
int caller_reg, cb_callee_reg;
j = idx - start; /* relative index within this subprog */
+ memset(callee_args, 0, MAX_AT_TRACK_REGS * sizeof(*callee_args));
if (bpf_pseudo_call(insn)) {
target = idx + insn->imm + 1;
@@ -2438,9 +2446,13 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)
env->callsite_at_stack = kvzalloc_objs(*env->callsite_at_stack, insn_cnt,
GFP_KERNEL_ACCOUNT);
- if (!env->callsite_at_stack) {
- kvfree(info);
- return -ENOMEM;
+ env->liveness->callee_args = kvcalloc(MAX_CALL_FRAMES * MAX_AT_TRACK_REGS,
+ sizeof(*env->liveness->callee_args),
+ GFP_KERNEL_ACCOUNT);
+ if (!env->callsite_at_stack || !env->liveness->callee_args) {
+ err = -ENOMEM;
+ insn_cnt = env->callsite_at_stack ? insn_cnt : 0;
+ goto out;
}
/*
@@ -2478,6 +2490,8 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)
kvfree(env->callsite_at_stack[k]);
kvfree(env->callsite_at_stack);
env->callsite_at_stack = NULL;
+ kvfree(env->liveness->callee_args);
+ env->liveness->callee_args = NULL;
for (k = 0; k < env->subprog_cnt; k++)
kvfree(info[k].at_in);
kvfree(info);
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 34/37] bpf: Allow 24 arguments of static subprogs
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (32 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 33/37] bpf: Keep the frame of analyze_subprog() small Alexei Starovoitov
@ 2026-11-03 23:40 ` Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 35/37] selftests/bpf: Test a static subprog with 24 arguments Alexei Starovoitov
` (2 subsequent siblings)
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:40 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A BPF function takes 12 arguments at most: 5 in registers and 7 in stack
slots. Rust needs more, because it passes a slice or a trait object in two
slots. For a struct with 5 fields derive(Debug) calls
core::fmt::Formatter::debug_struct_field5_finish(), which takes up to
23 slots.
The limit of 12 is for calls in which the kernel takes part. Only BPF code
calls a static subprog. Thus a static subprog can have a different limit:
MAX_BPF_STATIC_FUNC_ARGS.
callee arguments
------------------- ----------------------------------------
kfunc, trampoline 12, no change
global subprog 5, no change
static subprog 12 -> 24: 5 registers and 19 stack slots
The verifier counts the stack arguments of a static subprog from its code,
and JITs get the number of stack slots from the verifier. Thus JITs do not
change. The field that counts these slots, and the mask of stack arguments
in struct backtrack_state, get wider.
BTF still describes 12 parameters at most. The verifier marks BTF of a
static subprog with more parameters as unreliable and continues, as before.
The test that stores behind the last slot now stores to slot 20:
stack arg write offset -160 exceeds max 19 stack args
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 15 +++++++++++----
include/linux/bpf_verifier.h | 6 +++---
kernel/bpf/backtrack.c | 2 +-
kernel/bpf/verifier.c | 4 ++--
.../selftests/bpf/progs/verifier_stack_arg.c | 6 +++---
5 files changed, 20 insertions(+), 13 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index ca8992702dbd..7e022591dc23 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1008,13 +1008,20 @@ static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*
- * Also reuse this macro for maximum number of arguments a BPF function
- * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
- * stack arg slots. The JIT may map some stack arg slots to registers based
- * on the native calling convention (e.g., arg 6 to R9 on x86-64).
+ * Also reuse this macro for maximum number of arguments a kfunc can have and
+ * BTF of a BPF function can describe. Args 1-5 are passed in registers, args
+ * 6-12 via stack arg slots. The JIT may map some stack arg slots to registers
+ * based on the native calling convention (e.g., arg 6 to R9 on x86-64).
*/
#define MAX_BPF_FUNC_ARGS 12
+/*
+ * A static BPF function is called by BPF code only, with what the compiler
+ * makes of its arguments. A slice or a trait object of Rust takes two slots. The
+ * function that derive(Debug) calls for a struct with five fields takes 23.
+ */
+#define MAX_BPF_STATIC_FUNC_ARGS 24
+
/* The maximum number of arguments passed through registers
* a single function may have.
*/
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index dfecdfac75ff..54d45407ac1d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -487,7 +487,7 @@ static_assert(MAX_CALL_FRAMES <= (1 << 4));
static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
/* Maximum number of bpf_reg_state objects that can exist at once */
-#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define MAX_STACK_ARG_SLOTS (MAX_BPF_STATIC_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
MAX_CALL_FRAMES)
@@ -1007,7 +1007,7 @@ struct bpf_subprog_info {
* a hidden pointer to the result, a pair in two registers, no dead arguments.
*/
bool stack_args_from_code: 1;
- u8 code_stack_args:4;
+ u8 code_stack_args;
u16 stack_arg_cnt; /* incoming + max outgoing */
u16 max_out_stack_arg_cnt;
};
@@ -1029,7 +1029,7 @@ struct backtrack_state {
u32 frame;
u32 reg_masks[MAX_CALL_FRAMES];
unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
- u8 stack_arg_masks[MAX_CALL_FRAMES];
+ u32 stack_arg_masks[MAX_CALL_FRAMES];
};
struct bpf_id_pair {
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 2d6535f271a6..6d448f503de9 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -223,7 +223,7 @@ static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
return bt_frame_stack_mask(bt, bt->frame);
}
-static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
+static inline u32 bt_stack_arg_mask(struct backtrack_state *bt)
{
return bt->stack_arg_masks[bt->frame];
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d503dc7cea1f..563313809078 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4633,7 +4633,7 @@ static int check_stack_write(struct bpf_verifier_env *env,
static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
int off, struct bpf_reg_state *value_reg)
{
- int max_stack_arg_regs = MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS;
+ int max_stack_arg_regs = MAX_STACK_ARG_SLOTS;
struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
int spi = -off / BPF_REG_SIZE - 1;
struct bpf_reg_state *arg;
@@ -15257,7 +15257,7 @@ static void prepare_subprog_args(struct bpf_verifier_env *env)
cnt = max(cnt, insn->off / BPF_REG_SIZE);
}
sub->stack_args_from_code = true;
- sub->code_stack_args = min(cnt, MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS);
+ sub->code_stack_args = min(cnt, MAX_STACK_ARG_SLOTS);
/* incoming + max outgoing. Calls add the latter. */
sub->stack_arg_cnt = sub->code_stack_args;
if (env->prog->aux->func_info_aux)
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index d1fd5a402565..c54a9b80ced4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -443,7 +443,7 @@ __naked void stack_arg_shared_store(void)
SEC("tc")
__description("stack_arg: write beyond max outgoing depth")
__failure
-__msg("stack arg write offset -80 exceeds max 7 stack args")
+__msg("stack arg write offset -160 exceeds max 19 stack args")
__naked void stack_arg_write_beyond_max(void)
{
asm volatile (
@@ -452,8 +452,8 @@ __naked void stack_arg_write_beyond_max(void)
"r3 = 3;"
"r4 = 4;"
"r5 = 5;"
- /* Write to offset -80, way beyond any callee's needs */
- "*(u64 *)(r11 - 80) = 99;"
+ /* Write to offset -160, way beyond any callee's needs */
+ "*(u64 *)(r11 - 160) = 99;"
"*(u64 *)(r11 - 16) = 20;"
"*(u64 *)(r11 - 8) = 10;"
"call subprog_7args;"
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 35/37] selftests/bpf: Test a static subprog with 24 arguments
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (33 preceding siblings ...)
2026-11-03 23:40 ` [PATCH bpf-next 34/37] bpf: Allow 24 arguments of static subprogs Alexei Starovoitov
@ 2026-11-03 23:41 ` Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 36/37] bpf: Bump BPF_MAX_SUBPROGS to 1024 Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 37/37] selftests/bpf: Make room in veristat for the stats of 1024 subprogs Alexei Starovoitov
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:41 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog passes 1 .. 24 to a static subprog, in r1 - r5 and in all 19 stack
slots. The subprog adds them up. The test runs the prog and expects 300.
Thus it also shows that the JIT has room for each slot. The subprog is in
asm and its prototype has no parameters: the verifier sees in the code what
it takes.
A second prog passes 17 slices of dynptrs in stack slots. Each slice has
two ids: its own and the id of its dynptr. The subprog returns the last
one, a slice of a ringbuf dynptr. The prog submits the dynptr and then
writes through the slice. The verifier must reject the write: what the
subprog returns is still the slice of that dynptr, whatever the number of
ids in front of it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_stack_arg.c | 217 ++++++++++++++++++
1 file changed, 217 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index c54a9b80ced4..146cdf81bbc4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -13,6 +13,11 @@ struct {
__type(value, long long);
} map_hash_8b SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_RINGBUF);
+ __uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
#if defined(__BPF_FEATURE_STACK_ARGUMENT)
struct {
@@ -440,6 +445,218 @@ __naked void stack_arg_shared_store(void)
);
}
+/* r1 - r5 and all 19 slots. Its prototype has no parameters: what it takes is seen in its code. */
+__naked __noinline __used
+static unsigned long subprog_24args(void)
+{
+ asm volatile (
+ "r0 = r1;"
+ "r0 += r2;"
+ "r0 += r3;"
+ "r0 += r4;"
+ "r0 += r5;"
+ "r1 = *(u64 *)(r11 + 8);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 16);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 24);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 32);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 40);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 48);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 56);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 64);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 72);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 80);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 88);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 96);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 104);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 112);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 120);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 128);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 136);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 144);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 152);"
+ "r0 += r1;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: static subprog with 24 args")
+__success __retval(300)
+__naked void stack_arg_24args(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 152) = 24;"
+ "*(u64 *)(r11 - 144) = 23;"
+ "*(u64 *)(r11 - 136) = 22;"
+ "*(u64 *)(r11 - 128) = 21;"
+ "*(u64 *)(r11 - 120) = 20;"
+ "*(u64 *)(r11 - 112) = 19;"
+ "*(u64 *)(r11 - 104) = 18;"
+ "*(u64 *)(r11 - 96) = 17;"
+ "*(u64 *)(r11 - 88) = 16;"
+ "*(u64 *)(r11 - 80) = 15;"
+ "*(u64 *)(r11 - 72) = 14;"
+ "*(u64 *)(r11 - 64) = 13;"
+ "*(u64 *)(r11 - 56) = 12;"
+ "*(u64 *)(r11 - 48) = 11;"
+ "*(u64 *)(r11 - 40) = 10;"
+ "*(u64 *)(r11 - 32) = 9;"
+ "*(u64 *)(r11 - 24) = 8;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call subprog_24args;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long returns_arg22(void)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r11 + 136);"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/* A dynptr and a slice of it, which has two ids: its own and the one of the dynptr */
+#define SLICE(dynptr, slice) \
+ "r1 = r6;" \
+ "r2 = 8;" \
+ "r3 = 0;" \
+ "r4 = r10;" \
+ "r4 += -" #dynptr ";" \
+ "call %[bpf_dynptr_from_mem];" \
+ "r1 = r10;" \
+ "r1 += -" #dynptr ";" \
+ "r2 = 0;" \
+ "r3 = 8;" \
+ "call %[bpf_dynptr_data];" \
+ "*(u64 *)(r10 - " #slice ") = r0;"
+
+#define SLICE_ARG(slice, arg) \
+ "r1 = *(u64 *)(r10 - " #slice ");" \
+ "*(u64 *)(r11 - " #arg ") = r1;"
+
+/*
+ * The 16 slices in front of it have 32 ids. What the subprog returns is still the slice of
+ * the dynptr that is submitted.
+ */
+SEC("tc")
+__description("stack_arg: ids of the last of 17 slices")
+__failure __msg("R6 invalid mem access 'scalar'")
+__naked void stack_arg_ids_of_17_slices(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 408) = r1;"
+ "r2 = r10;"
+ "r2 += -408;"
+ "r1 = %[map_hash_8b] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ SLICE(16, 280)
+ SLICE(32, 288)
+ SLICE(48, 296)
+ SLICE(64, 304)
+ SLICE(80, 312)
+ SLICE(96, 320)
+ SLICE(112, 328)
+ SLICE(128, 336)
+ SLICE(144, 344)
+ SLICE(160, 352)
+ SLICE(176, 360)
+ SLICE(192, 368)
+ SLICE(208, 376)
+ SLICE(224, 384)
+ SLICE(240, 392)
+ SLICE(256, 400)
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "r4 = r10;"
+ "r4 += -272;"
+ "call %[bpf_ringbuf_reserve_dynptr];"
+ "r1 = r10;"
+ "r1 += -272;"
+ "r2 = 0;"
+ "r3 = 8;"
+ "call %[bpf_dynptr_data];"
+ "if r0 == 0 goto 1f;"
+ "*(u64 *)(r11 - 136) = r0;"
+ SLICE_ARG(400, 128)
+ SLICE_ARG(392, 120)
+ SLICE_ARG(384, 112)
+ SLICE_ARG(376, 104)
+ SLICE_ARG(368, 96)
+ SLICE_ARG(360, 88)
+ SLICE_ARG(352, 80)
+ SLICE_ARG(344, 72)
+ SLICE_ARG(336, 64)
+ SLICE_ARG(328, 56)
+ SLICE_ARG(320, 48)
+ SLICE_ARG(312, 40)
+ SLICE_ARG(304, 32)
+ SLICE_ARG(296, 24)
+ SLICE_ARG(288, 16)
+ SLICE_ARG(280, 8)
+ "call returns_arg22;"
+ "r6 = r0;"
+ "r1 = r10;"
+ "r1 += -272;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_submit_dynptr];"
+ "r1 = 0;"
+ "*(u64 *)(r6 + 0) = r1;"
+ "r0 = 0;"
+ "exit;"
+ "1:"
+ "r1 = r10;"
+ "r1 += -272;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard_dynptr];"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_map_lookup_elem),
+ __imm(bpf_dynptr_from_mem),
+ __imm(bpf_dynptr_data),
+ __imm(bpf_ringbuf_reserve_dynptr),
+ __imm(bpf_ringbuf_submit_dynptr),
+ __imm(bpf_ringbuf_discard_dynptr),
+ __imm_addr(map_hash_8b),
+ __imm_addr(ringbuf)
+ : __clobber_all
+ );
+}
+
SEC("tc")
__description("stack_arg: write beyond max outgoing depth")
__failure
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 36/37] bpf: Bump BPF_MAX_SUBPROGS to 1024
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (34 preceding siblings ...)
2026-11-03 23:41 ` [PATCH bpf-next 35/37] selftests/bpf: Test a static subprog with 24 arguments Alexei Starovoitov
@ 2026-11-03 23:41 ` Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 37/37] selftests/bpf: Make room in veristat for the stats of 1024 subprogs Alexei Starovoitov
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:41 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog that uses Rust crates can have more than 256 functions that the
compiler does not inline. The verifier rejects a prog with 777 subprogs:
too many subprograms
Bump the limit to 1024.
Two arrays in struct bpf_verifier_env grow: subprog_info[] and
subprog_topo_order[]. The verifier allocates the struct with kvzalloc(),
thus the size is not a problem. A load from a table of functions can give
BPF_MAX_SUBPROGS different pointers. This limit grows too.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 54d45407ac1d..1871d0c109d6 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -936,7 +936,7 @@ struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
u32 size);
int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log);
-#define BPF_MAX_SUBPROGS 256
+#define BPF_MAX_SUBPROGS 1024
/*
* A pointer to a static subprog in the value of a frozen read-only array map:
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [PATCH bpf-next 37/37] selftests/bpf: Make room in veristat for the stats of 1024 subprogs
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
` (35 preceding siblings ...)
2026-11-03 23:41 ` [PATCH bpf-next 36/37] bpf: Bump BPF_MAX_SUBPROGS to 1024 Alexei Starovoitov
@ 2026-11-03 23:41 ` Alexei Starovoitov
36 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-03 23:41 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
veristat reads the statistics at the end of the verifier log. They have one
line for each subprog. Two sizes in veristat are too small now that a prog
can have 1024 subprogs.
size old new problem with the old size
---------------------- ---- ----- ----------------------------------
log buffer 64K 1M 777 subprogs make 82K of stats.
The load fails with ENOSPC.
lines that veristat 300 1100 With more than 296 subprogs it
parses, from the end does not find 'verification time'.
The duration is 0.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/veristat.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c
index 9cfc9b4b41c1..6b295e7d6f32 100644
--- a/tools/testing/selftests/bpf/veristat.c
+++ b/tools/testing/selftests/bpf/veristat.c
@@ -1013,10 +1013,11 @@ static void free_verif_stats(struct verif_stats *stats, size_t stat_cnt)
free(stats);
}
-static char verif_log_buf[64 * 1024];
+/* the stats have a line for every subprog, with its name */
+static char verif_log_buf[1024 * 1024];
-/* Keep room for all 256 subprogram records and trailing statistics. */
-#define MAX_PARSED_LOG_LINES 300
+/* Keep room for all 1024 subprogram records and trailing statistics. */
+#define MAX_PARSED_LOG_LINES 1100
static int parse_verif_log(char * const buf, size_t buf_sz, struct verif_stats *s)
{
--
2.55.0
^ permalink raw reply related [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
2026-11-03 23:40 ` [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
@ 2026-11-04 0:12 ` sashiko-bot
2026-11-04 1:06 ` Alexei Starovoitov
0 siblings, 1 reply; 47+ messages in thread
From: sashiko-bot @ 2026-11-04 0:12 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: __free(kfree) variable declared as NULL at top of check_func_ptr_read()
--
commit 6a9289c982c1e888c820e75173835d8cb92fc098
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
This commit raises BPF_MAX_FUNC_PTR_TARGETS from 64 to BPF_MAX_SUBPROGS and
allocates the targets array dynamically in check_func_ptr_read() instead of
keeping it on the stack.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 93ca0126a3108..d503dc7cea1f8 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -7114,17 +7114,20 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
> struct bpf_reg_state *regs = cur_regs(env);
> struct bpf_map *map = reg->map_ptr;
> struct bpf_verifier_state *branch;
> - int subprog, targets[BPF_MAX_FUNC_PTR_TARGETS];
> + u16 *targets __free(kfree) = NULL;
> struct bpf_func_ptr *ptrs;
> u32 i, cnt, n = 0;
> u64 o, lo, hi;
> - int err;
> + int err, subprog;
>
> ptrs = bpf_map_range_func_ptrs(env, map, min_off, max_off - min_off + size, &cnt);
> if (!ptrs)
> return -ENOENT;
> if (cnt > BPF_MAX_FUNC_PTR_RANGE)
> goto overlap;
> + targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), GFP_KERNEL_ACCOUNT);
> + if (!targets)
> + return -ENOMEM;
[Severity: Low]
This isn't a bug, but in check_func_ptr_read() in kernel/bpf/verifier.c,
targets is declared with __free(kfree) = NULL at the top of the function
and then assigned after a goto overlap check, while the rest of the
function continues to use goto overlap for error handling.
Per the cleanup.h conventions, should targets be defined and initialized
in a single statement at the point of allocation, and should mixing
__free() with goto-based error handling in the same function be avoided?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103234104.208239-1-alexei.starovoitov@gmail.com?part=32
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code
2026-11-03 23:40 ` [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
@ 2026-11-04 0:14 ` sashiko-bot
2026-11-04 1:05 ` Alexei Starovoitov
0 siblings, 1 reply; 47+ messages in thread
From: sashiko-bot @ 2026-11-04 0:14 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: undercounted arg_slot_cnt on early return in btf_prepare_func_args()
--
commit 7a7c085f34a6a2a56b7b82b03b95e750d7742fd9
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Count stack arguments of a static subprog from its code
This commit changes the verifier to determine the number of stack arguments
of a static subprogram from the highest stack argument slot loaded through
r11 in its instructions rather than solely from its BTF prototype. It also
updates caller stack argument accounting and trampoline attachment checks
accordingly.
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 8bf39098ba97a..228b93d4f8c94 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
[ ... ]
> @@ -8827,13 +8821,15 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
> }
> args = (const struct btf_param *)(t + 1);
> nargs = btf_type_vlen(t);
> - sub->arg_slot_cnt = nargs;
> + sub->arg_slot_cnt = min_t(u32, nargs, MAX_BPF_FUNC_ARGS);
> if (nargs > MAX_BPF_FUNC_ARGS) {
> + if (!is_global) /* not fatal for static funcs */
> + return -EINVAL;
> bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
> MAX_BPF_FUNC_ARGS, tname, nargs);
> return -EFAULT;
> }
> - err = btf_check_arg_slots(log, tname, is_global, nargs, sub);
> + err = btf_check_arg_slots(log, tname, is_global, nargs);
> if (err)
> return err;
>
> @@ -9022,7 +9018,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
> return -EINVAL;
> }
>
> - err = btf_check_arg_slots(log, tname, is_global, slots_used, sub);
> + err = btf_check_arg_slots(log, tname, is_global, slots_used);
> if (err)
> return err;
> sub->arg_slot_cnt = slots_used;
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index d95f86b62e99f..b480c5c242af8 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -11537,13 +11541,24 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
> /*
> * Track caller's total stack arg count (incoming + max outgoing).
> * This is needed so the JIT knows how much stack arg space to allocate.
> + * The callee can load less than its prototype has, and the caller can still store
> + * all of it. Count the slots of the prototype only as far as the caller has stored
> + * them: compilers store in front of the call. bpf_fixup_call_args() rejects the rest.
> */
> caller_info = &env->subprog_info[caller->subprogno];
> - callee_incoming = bpf_in_stack_arg_cnt(&env->subprog_info[subprog]);
> - stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming;
> + callee_incoming = bpf_in_stack_arg_cnt(callee_info);
> + stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) +
> + max_t(int, callee_incoming,
> + min_t(int, caller_info->max_out_stack_arg_cnt,
> + callee_info->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS));
> if (stack_arg_cnt > caller_info->stack_arg_cnt)
> caller_info->stack_arg_cnt = stack_arg_cnt;
[Severity: Medium]
Can btf_prepare_func_args() leave sub->arg_slot_cnt undercounted when it
returns early on a static subprogram with multi-slot parameters?
In kernel/bpf/btf.c, btf_prepare_func_args() initially sets:
nargs = btf_type_vlen(t);
sub->arg_slot_cnt = min_t(u32, nargs, MAX_BPF_FUNC_ARGS);
which is the BTF parameter count rather than the 8-byte slot count. It only
updates sub->arg_slot_cnt to slots_used at the end of the function if the
return type and all parameters are accepted:
err = btf_check_arg_slots(log, tname, is_global, slots_used);
if (err)
return err;
sub->arg_slot_cnt = slots_used;
If btf_prepare_func_args() returns an error early for a static subprogram
(for example, when btf_validate_return_type() rejects a pointer return type,
when the parameter loop encounters an untagged pointer parameter, or when
slots_used exceeds MAX_BPF_FUNC_ARGS at too_many_slots while nargs is at
most MAX_BPF_FUNC_ARGS), sub->arg_slot_cnt remains set to nargs instead of
the prototype's 8-byte slot count.
When that static subprogram also has multi-slot parameters (such as 9-to-16
byte structs passed by value or 128-bit integers) and ignores a trailing
stack argument slot, check_static_func_call() in kernel/bpf/verifier.c
computes:
callee_info->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS
using the parameter count rather than the slot count. Because this
undercounts the prototype's stack argument slots, wouldn't
bpf_fixup_call_args() reject valid caller stores to those trailing stack
argument slots?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103234104.208239-1-alexei.starovoitov@gmail.com?part=6
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range
2026-11-03 23:40 ` [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range Alexei Starovoitov
@ 2026-11-04 0:21 ` sashiko-bot
2026-11-04 1:05 ` Alexei Starovoitov
2026-11-04 7:26 ` Eduard Zingerman
1 sibling, 1 reply; 47+ messages in thread
From: sashiko-bot @ 2026-11-04 0:21 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: iv_bounds() leaves bounds->r32 un-narrowed before log_bounds()
--
commit 2f96c458caf20b2679374f41e2e5031f644f1eb1
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Count loops whose first value or bound is a range
This commit extends BPF scalar evolution (SCEV) to compute upper bounds on
loop iteration counts when the initial value or latch bound is a range rather
than a constant, and narrows the induction variable range at the loop header
using iv_bounds().
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 8662205a3b594..eac792dc7b103 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
[ ... ]
> @@ -2626,6 +2935,14 @@ static void linear_bounds(struct bpf_reg_state *reg, u32 max_header_count,
> out->r32 = r32;
> }
>
> +static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
> +{
> + struct cnum64 r64 = cnum64_intersect(bounds->r64, iv_range);
> +
> + if (!cnum64_is_empty(r64))
> + bounds->r64 = r64;
> +}
> +
[ ... ]
> @@ -2925,6 +3244,8 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
> if (err)
> return err;
> linear_bounds(reg, iters->max_header_count, &coefs, &bounds);
> + if (r == iters->iv_reg)
> + iv_bounds(&bounds, iters->iv_range);
> if (log->level & BPF_LOG_LEVEL2) {
> bpf_log(log, "loop header at %d, widening ", insn_idx);
> log_reg(env, r);
[ ... ]
> @@ -3005,6 +3328,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
> continue;
>
> linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
> + if (r == entry->iv_reg)
> + iv_bounds(&bounds, entry->iv_range);
> if (log->level & BPF_LOG_LEVEL2) {
> bpf_log(log, "loop header at %d, clamping ", insn_idx);
> log_reg(env, r);
[Severity: Low]
Should iv_bounds() also narrow bounds->r32 when bounds->r64 is narrowed?
When widening or clamping a 32-bit induction variable (coefs.zext32 is true)
with a range initial value or bound, linear_bounds() populates both
bounds.r64 and bounds.r32 with the wider range computed from
max_header_count (for example, 0..32 when initial is 0..16 and bound is 17).
Calling iv_bounds() then intersects bounds->r64 with iv_range (narrowing
bounds.r64 to 0..16), but leaves bounds->r32 untouched at 0..32.
Although bpf_set_reg_range() later synchronizes reg->r32 from reg->r64 via
reg_bounds_sync(), log_bounds() is called in bpf_widen_scev_regs() and
bpf_clamp_scev_regs() before bpf_set_reg_range(). Because bounds->r32.size
is less than U32_MAX and cnum32_smax(bounds->r32) differs from
cnum64_smax(bounds->r64), log_bounds() at BPF_LOG_LEVEL2 prints the stale
un-narrowed 32-bit subregister range (for example, "0..16 (s32 0..32) step 1").
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103234104.208239-1-alexei.starovoitov@gmail.com?part=16
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down
2026-11-03 23:40 ` [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down Alexei Starovoitov
@ 2026-11-04 0:26 ` sashiko-bot
2026-11-04 1:06 ` Alexei Starovoitov
0 siblings, 1 reply; 47+ messages in thread
From: sashiko-bot @ 2026-11-04 0:26 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] bpf: 64-bit JLT/JLE on 32-bit counter in compute_max_iters() lacks jmp32
- [Medium] bpf: compute_max_iters() zeroes bound_val before checking konst
- [Low] bpf: compute_max_iters() misses downward BPF_JLE when initial equals bound
--
commit 08a1779026bb125181a7c2d2c0167baa7768249b
Author: Alexei Starovoitov <ast@kernel.org>
bpf: Count '!=' across 0 in 32 bits and loops that go down
This commit teaches BPF scalar evolution to compute iteration counts for
32-bitBPF_JNE loops that cross zero and for downward loops tested with
unsigned BPF_JLT or BPF_JLE latches by rewriting the latter to signed
BPF_JSGE comparisons against zero.
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 9ddaae03ac351..70d15449eebe1 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
> @@ -2738,8 +2738,9 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
> struct scev *scev = env->scev;
> struct scev_value initial_val, step_val, bound_val;
> u64 step, bound, initial, total_change, backedge_taken_count;
> - bool konst;
> + struct linear_latch counted;
> u8 op = latch->op;
> + bool konst, sign32 = latch->jmp32 && is_signed_op(op);
>
> iters->iv_reg = -1;
>
> @@ -2761,6 +2762,23 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
>
> if (!scev_value_is_const(&step_val))
> return false;
> + /*
> + * 'i >= 0 && i < 128' is one unsigned test. A counter that goes down stays below 128.
> + * The loop ends when it gets below 0. A 64-bit jump compares all 64 bits of the bound.
> + * The latch of the caller stays as it is: same_latch() compares what the prog tests.
> + */
> + if ((op == BPF_JLT || op == BPF_JLE) && initial_val.ptr_reg < 0 &&
> + (latch->jmp32 || latch->zext32 ?
> + (s32)step_val.umin < 0 && initial_val.u32_min < bound_val.u32_max &&
> + (latch->jmp32 ? bound_val.u32_max : bound_val.umax) <= S32_MAX :
> + (s64)step_val.umin < 0 && initial_val.umin < bound_val.umax &&
> + bound_val.umax <= S64_MAX)) {
[Severity: Low]
In compute_max_iters() in kernel/bpf/scev.c, both BPF_JLT and BPF_JLE are
checked with strict inequality between the minimum initial value and maximum
bound (initial_val.u32_min < bound_val.u32_max or
initial_val.umin < bound_val.umax).
When op == BPF_JLE and a downward counter starts at a value equal to the
bound (initial_val.u32_min == bound_val.u32_max or
initial_val.umin == bound_val.umax), this condition evaluates to false and
the latch is not rewritten to BPF_JSGE. Downward BPF_JLE is then rejected by
compute_backedge_taken_count() and compute_backedge_taken_max().
Should the comparison allow equality when op == BPF_JLE?
> + counted = *latch;
> + latch = &counted;
> + op = latch->op = BPF_JSGE;
> + bound_val = (struct scev_value){ .ptr_reg = -1 };
> + sign32 = true;
> + }
[Severity: Medium]
When a 64-bit unsigned BPF_JLT or BPF_JLE jump tests a 32-bit downward
counter (latch->zext32 is true and latch->jmp32 is false) with
bound_val.umax <= S32_MAX, compute_max_iters() in kernel/bpf/scev.c rewrites
counted.op to BPF_JSGE with bound_val = 0 and sets sign32 = true, but leaves
counted.jmp32 as false.
When compute_max_iters() subsequently calls compute_backedge_taken_max() or
latch_false_until(), both pass latch to latch_domain() with sign = true,
latch->jmp32 = false, and latch->zext32 = true:
kernel/bpf/scev.c:latch_domain() {
...
bias = sign ? 1ULL << 63 : 0;
d->bmin = (sign ? (u64)bound->smin : bound->umin) ^ bias;
d->bmax = (sign ? (u64)bound->smax : bound->umax) ^ bias;
if (latch->zext32) {
/* zero extended, so it is not negative */
d->imin = initial->u32_min ^ bias;
d->imax = initial->u32_max ^ bias;
d->lo = bias;
d->hi = bias + U32_MAX;
return;
}
}
Because bound is 0, latch_domain() sets d->bmin = bias and d->lo = bias, and
uses u32_min/u32_max instead of s32_min/s32_max.
In the non-constant case (!konst), compute_backedge_taken_max() then checks:
case BPF_JGE:
case BPF_JSGE:
if (!down ||
check_add_overflow(div64_u64(d.imax - d.bmin, step), 1, &cnt) ||
d.bmin < d.lo || step > d.bmin - d.lo)
return false;
Since d.bmin - d.lo is 0 and step >= 1, step > d.bmin - d.lo is always true
and compute_backedge_taken_max() unconditionally returns false.
In the constant case (konst), latch_false_until() computes
div64_u64(d.imin - d.lo, -step), which evaluates to initial_val.u32_min / -step
(i.e. backedge_taken_count - 1, the distance to 0 rather than the distance to
S32_MIN), causing bpf_compute_loop_iters() to reject multi-backedge loops
when another backedge has a different latch.
Should counted.jmp32 also be set to latch->jmp32 || latch->zext32 when
rewriting the latch to BPF_JSGE?
> konst = scev_value_is_const(&initial_val) && scev_value_is_const(&bound_val);
[Severity: Medium]
In compute_max_iters() in kernel/bpf/scev.c, bound_val is overwritten with
zero before konst is computed from scev_value_is_const(&initial_val) &&
scev_value_is_const(&bound_val).
If a downward BPF_JLT or BPF_JLE loop has a constant initial_val and a
non-constant range bound_val (for example initial_val = 10 and
bound_val in [5, 20]), scev_value_is_const(&bound_val) checks the overwritten
zero bound_val and evaluates konst to true.
As a result, compute_max_iters() skips compute_backedge_taken_max() (leaving
iters->iv_reg as -1) and sets:
iters->min_header_count = konst && loop->exits_cnt == 1 ? iters->max_header_count : 0;
This sets min_header_count to max_header_count even when the original
bound_val range includes values at or below initial_val that cause the loop
to exit on the first iteration. It also forces mixed-width latches
(!latch->zext32 && latch->jmp32) with non-zero upper 32 bits in initial_val
onto the constant path where they are rejected.
Should konst be evaluated before bound_val is overwritten?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103234104.208239-1-alexei.starovoitov@gmail.com?part=29
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code
2026-11-04 0:14 ` sashiko-bot
@ 2026-11-04 1:05 ` Alexei Starovoitov
0 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:05 UTC (permalink / raw)
To: sashiko-reviews, bpf
On Tue, Nov 03, 2026 at 04:14 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
>> + stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) +
>> + max_t(int, callee_incoming,
>> + min_t(int, caller_info->max_out_stack_arg_cnt,
>> + callee_info->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS));
>
> [Severity: Medium]
> Can btf_prepare_func_args() leave sub->arg_slot_cnt undercounted when it
> returns early on a static subprogram with multi-slot parameters?
[...]
> using the parameter count rather than the slot count. Because this
> undercounts the prototype's stack argument slots, wouldn't
> bpf_fixup_call_args() reject valid caller stores to those trailing stack
> argument slots?
Not from this patch.
bpf-next has arg_slot_cnt == nargs after an early return too, and
bpf_fixup_call_args() rejects the same store with the same message.
bpf-next also rejects the load of such a slot in the callee.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range
2026-11-04 0:21 ` sashiko-bot
@ 2026-11-04 1:05 ` Alexei Starovoitov
0 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:05 UTC (permalink / raw)
To: sashiko-reviews, bpf
On Tue, Nov 03, 2026 at 04:21 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
> - [Low] bpf: iv_bounds() leaves bounds->r32 un-narrowed before log_bounds()
Not worth a respin.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down
2026-11-04 0:26 ` sashiko-bot
@ 2026-11-04 1:06 ` Alexei Starovoitov
0 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:06 UTC (permalink / raw)
To: sashiko-reviews, bpf
On Tue, Nov 03, 2026 at 04:26 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
>> + if ((op == BPF_JLT || op == BPF_JLE) && initial_val.ptr_reg < 0 &&
>> + (latch->jmp32 || latch->zext32 ?
>> + (s32)step_val.umin < 0 && initial_val.u32_min < bound_val.u32_max &&
>
> Should the comparison allow equality when op == BPF_JLE?
Not a bug. Such a loop has no count, as before the patch.
The verifier walks it one iteration at a time.
>> + op = latch->op = BPF_JSGE;
>> + bound_val = (struct scev_value){ .ptr_reg = -1 };
>> + sign32 = true;
>> + }
>
> Should counted.jmp32 also be set to latch->jmp32 || latch->zext32 when
> rewriting the latch to BPF_JSGE?
Not a bug. It's a limitation.
A 32-bit counter under a 64-bit jump has a count when the first value is
a constant. With a range there is no count, as before the patch.
false_until is too small there. That's the safe side: no count.
>> konst = scev_value_is_const(&initial_val) && scev_value_is_const(&bound_val);
>
> This sets min_header_count to max_header_count even when the original
> bound_val range includes values at or below initial_val that cause the loop
> to exit on the first iteration. It also forces mixed-width latches
> (!latch->zext32 && latch->jmp32) with non-zero upper 32 bits in initial_val
> onto the constant path where they are rejected.
>
> Should konst be evaluated before bound_val is overwritten?
No. Only the log at level 2 reads min_header_count.
The verifier uses max_header_count. It's an upper bound for every bound
in the range.
The mixed-width latch has no count with a constant bound either.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
2026-11-04 0:12 ` sashiko-bot
@ 2026-11-04 1:06 ` Alexei Starovoitov
0 siblings, 0 replies; 47+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:06 UTC (permalink / raw)
To: sashiko-reviews, bpf
On Tue, Nov 03, 2026 at 04:12 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
> - [Low] bpf: __free(kfree) variable declared as NULL at top of check_func_ptr_read()
Not worth a respin.
The overlap label prints the message and returns. It frees nothing.
targets is the only __free() variable there, so there is no order to break.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range
2026-11-03 23:40 ` [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range Alexei Starovoitov
2026-11-04 0:21 ` sashiko-bot
@ 2026-11-04 7:26 ` Eduard Zingerman
1 sibling, 0 replies; 47+ messages in thread
From: Eduard Zingerman @ 2026-11-04 7:26 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, memxor
On Tue, 2026-11-03 at 23:40 +0000, Alexei Starovoitov wrote:
...
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 8662205a3b59..eac792dc7b10 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
> @@ -2035,11 +2035,20 @@ static int match_linear_latch(struct bpf_verifier_env *env,
> return false;
> }
>
> +/* What is known about a scalar or about the offset of a pointer. */
> struct scev_value {
> - u64 value; /* scalar value or a pointer offset */
> + u64 umin, umax;
> + s64 smin, smax;
> + u32 u32_min, u32_max;
> + s32 s32_min, s32_max;
> int ptr_reg; /* SCEV register identifying the pointer origin, or -1. */
> };
Nice set of improvements.
Quick question: can the above use cnum{32,64} instead of direct bounds?
Reason being, that I thought compute_backedge_taken_count() can be
expressed in terms of cnums as well, in which case there would be no
need to split it as two functions.
...
> +/*
> + * The first value or the bound is not a constant. The count is for the pair that is
> + * the furthest apart.
> + */
> +static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_func_state *st,
> + const struct linear_latch *latch,
> + const struct scev_value *initial,
> + const struct scev_value *step_val,
> + const struct scev_value *bound,
> + struct bpf_loop_iters *iters, u64 *result)
cnums aside, it appears surprising that both compute_backedge_taken_count()
and compute_backedge_taken_max() are necessary.
For example, I'd expect that compute_backedge_taken_max() should be
expressible using compute_backedge_taken_count() as:
high := compute_backedge_taken_count(..., min(initial), step, max(bound), ...);
low := compute_backedge_taken_count(..., max(initial), step, min(bound), ...);
... overflow check for iv with `low` and `high` ...
Or vice-versa, compute_backedge_taken_count() seem to be expressible
as a special case for compute_backedge_taken_max() with ranges
[initial, initial] and [bound, bound].
Is there some precision loss I don't understand?
--
Also, you'd consider it a nit, but compute_backedge_taken_count() has
a nice set of comments explaining where each condition is coming from.
...
^ permalink raw reply [flat|nested] 47+ messages in thread
end of thread, other threads:[~2026-11-04 7:26 UTC | newest]
Thread overview: 47+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-03 23:40 [PATCH bpf-next 00/37] bpf: More precision, more counted loops, larger limits Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 01/37] bpf: Keep the range of a packet pointer when a number >= 0 is added Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 02/37] selftests/bpf: Test range of a packet pointer after a number " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 03/37] bpf: Refuse fexit on an unreliable static subprog that returns in R0:R2 Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 04/37] bpf: Don't trust BTF of a static subprog that reads more registers Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 05/37] selftests/bpf: Test trampolines on subprogs that take more than BTF says Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 06/37] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
2026-11-04 0:14 ` sashiko-bot
2026-11-04 1:05 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 07/37] selftests/bpf: Test static subprogs that don't take what BTF says Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 08/37] bpf: Don't treat a partial spill as widened in scalar evolution Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 09/37] selftests/bpf: Test loops that change a partial spill Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 10/37] bpf: Let scalar evolution see loads from read-only maps Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 11/37] selftests/bpf: Test scalar evolution with a loop bound in .rodata Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 12/37] bpf: Helpers don't write through ARG_ANYTHING Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 13/37] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 14/37] bpf: Fold sums of constants and two shifts by 32 in scalar evolution Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 15/37] selftests/bpf: Test sums of constants and " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 16/37] bpf: Count loops whose first value or bound is a range Alexei Starovoitov
2026-11-04 0:21 ` sashiko-bot
2026-11-04 1:05 ` Alexei Starovoitov
2026-11-04 7:26 ` Eduard Zingerman
2026-11-03 23:40 ` [PATCH bpf-next 17/37] selftests/bpf: Test " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 18/37] bpf: Describe more registers of a counted loop Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 19/37] selftests/bpf: Test " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 20/37] bpf: Compute scev for a loop that has more than 256 exits Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 21/37] selftests/bpf: Test a loop with 301 exits Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 22/37] bpf: Describe a 32-bit counter that lives on the stack Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 23/37] selftests/bpf: Test " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 24/37] bpf: Count a loop with several back edges Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 25/37] selftests/bpf: Test loops " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 26/37] bpf: Compute bounds for a shift, a division and a remainder by a range Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 27/37] selftests/bpf: Test " Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 28/37] bpf: Count a loop whose latch is false in the first iteration Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 29/37] bpf: Count '!=' across 0 in 32 bits and loops that go down Alexei Starovoitov
2026-11-04 0:26 ` sashiko-bot
2026-11-04 1:06 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 30/37] bpf: Count a loop whose latch tests a flag that a jump sets Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 31/37] selftests/bpf: Test more latches of counted loops Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 32/37] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
2026-11-04 0:12 ` sashiko-bot
2026-11-04 1:06 ` Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 33/37] bpf: Keep the frame of analyze_subprog() small Alexei Starovoitov
2026-11-03 23:40 ` [PATCH bpf-next 34/37] bpf: Allow 24 arguments of static subprogs Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 35/37] selftests/bpf: Test a static subprog with 24 arguments Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 36/37] bpf: Bump BPF_MAX_SUBPROGS to 1024 Alexei Starovoitov
2026-11-03 23:41 ` [PATCH bpf-next 37/37] selftests/bpf: Make room in veristat for the stats of 1024 subprogs Alexei Starovoitov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox