[PATCH bpf-next v3 00/45] bpf: A verifier that joins states

[PATCH bpf-next v3 00/45] bpf: A verifier that joins states

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier follows each path of a prog, and each branch can double the
number of paths. The work grows with the paths, not with the prog:
  insns processed                                    today    this series
  rust-bpf  scx_beerland/beerland_init_task        1000001 *        17893
  rust-bpf  scx_qmap/qmap_init_cids                 131088 *          196
  scx       scx_eevdf/eevdf_dispatch                350041          15199
  Cilium    bpf_host/cil_to_netdev, Oct 2026         64658           7472
  * "BPF program is too large", "The sequence of 8193 jumps is too complex"

Authors cut their progs until they fit. Nobody can cut core, alloc and the
crates of Rust: of the 653 progs of https://github.com/4ast/rust-bpf 187
load today, 170 hit one of these limits, 164 get "infinite loop detected"
and 122 "recursive call". Now 648 load, and no corpus of C progs loses one.
For rust-bpf both kernels have 4 kfuncs on top, see Numbers.

The idea
========
Do not follow paths. Where paths meet, join their states: r1=1 and r1=5
give r1=[1,5]. Repeat until no state changes. States stay apart only if
they have no join, or if a later check depends on the difference.
  loops      counted by scalar evolution: one state covers all iterations
             walked: a state per iteration, while constants decide an exit
             all other loops: join at the head, widen, add may_goto
  subprogs   A static subprog is verified for the states at its entry, not
             for each chain of calls. Recursion is the same, with widening.
  run time   may_goto limits the time of loops and of recursion. r10 limits
             the depth: it must be in the upper half of its kernel stack.

What users see
==============
 - The limits of 1M processed insns and 8192 jumps are gone. The limits are
   on states, for a prog of N insns: 1024 at an insn, 16 * N + 4096 at
   a time, 256 * N + 64K in all.
 - "infinite loop detected" is gone: the verifier adds may_goto. A loop
   that has no exit gets "loop at insn N never exits".
 - may_goto is also added to 5 progs of the table below that load today:
   loop1, loop2, loop5, tcp_autolowat_enqueue_rcvq (GCC), chaos_enqueue.
 - Recursion is allowed on x86-64 and arm64. Patch 22 lists the exceptions.
 - All added may_goto of an invocation share its time slice of 250 ms. Then
   the prog returns 0. A subprog that cannot unwind, or a frame that holds
   a lock or a reference, leaves the loop. The stderr stream has the cause.
 - A join loses the relation between scalars. This is rejected now:
     if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
     x = arr[n - i];                                    n - i is [-3, 5]
 - A cgroup_skb prog whose r0 can be 2 or 3 attaches to egress only. That
   holds for a range too: 'return 1' and 'return 2' are joined to [1, 2].
 - The log has no "mark_precise:", "N: safe", "depth=" and no scc column.
   Level 2 shows "from 13 to 9: within", "joined" or "widened" where states
   meet. The statistics line keeps its format: "limit 0", "mark_read 0".
 - BPF_F_TEST_STATE_FREQ is accepted and has no effect.

Numbers
=======
A: bpf-next 8424858a5de0. B: this series, the same objects. The selftests
that the series adds or changes are not counted. None of them is lost.

Corpus                 Progs  Lost  Gained  Insns  States   Mem  Time
---------------------  -----  ----  ------  -----  ------  ----  ----
selftests               3043     0       1   -63%   +120%  -18%  -27%
selftests, no_alu32     2905     0       1   -66%    +86%  -27%  -34%
selftests, cpuv4        2914     0       1   -64%    +96%  -27%  -32%
selftests, bpf_gcc      2733     0       5   -69%    +96%  -16%  -24%
37 builds of Cilium     5189     0       0   -75%    +11%   -6%  -31%
sched-ext/scx            495     0       0   -89%    -34%  -66%  -67%
scx, as BPF CI has it    428     0       0   -85%    -22%  -38%  -39%
tools/sched_ext          240     0       0   -69%    +37%   +3%  -11%
public progs of Meta     511     0       0   -85%    -35%  -55%  -48%
all                    18458     0       8   -74%    +10%  -19%  -36%

Progs load in A and in B. The per cent compare sums over these progs:
75.1M -> 19.2M processed insns, 61 -> 39 s. Mem is mem_peak of veristat.
Time is the best of 3 runs on kernels without debug options. B counts each
state that arrives at a basic block as a state, A only those that it saves.
Gained: loop3 in the 4 builds, 4 progs that A rejects only in GCC's code.
Of the progs that take 1000 insns or more in B, two take over 5% more than
in A. loop5, 82 -> 4182: a walk up to its limit of 4096 insns, then a join.
res_spin_lock_test_held_lock_max, 2356 -> 4088: states that hold different
numbers of locks have no join.

rust-bpf               Progs  load in A  load in B  B: insns   time
16 schedulers            225         52        225      1.6M  1.9 s
crates                   428        135        423      7.3M  9.4 s
For rust-bpf A and B have the kfuncs of patches 30-31 of [1] on top. They
are in no series now. No released compiler emits the code of rust-bpf.

How to read the series
======================
Start with patch 7, the overview. Then 4-5, 13-14, 15, 17 and 19.
  1-2    what is added to a prog: may_goto, the checks of recursion
  3-5    union and widening of a range, join.c: the join of two states
  6-10   data structures of flow.c, passes ahead of the main pass: order of
         the insns, places for may_goto, what later checks depend on
  11-14  the sets of states, iterations, what is not joined, the merge
  15-18  how states move: edges, calls and exits, loops, added edges
  19     the switch. The check of one insn against one state is the same.
  20-22  more counted loops, static subprogs on their own, recursion
  23-26  removal of pruning, precision tracking and SCCs: 3382 lines
  27-28  callx: a pointer to one of a set of functions, live registers
  29-38  code that needs more than that: casts, the context in a struct,
         subprogs that check for their caller, loops that are not counted
  39-45  selftests: 39-40 adjust tests, 41-45 add tests
Each patch builds. Patches 1-18 add the code that patch 19 switches to.
Some selftests fail from patch 19 to 39.

Follow-ups
==========
 - Documentation/bpf/verifier.rst: the algorithm, with examples.
 - Liveness: no spilled pointer is kept for reads that see only its bytes.
 - No second pass over the prog when a loop gets may_goto late: half of the
   insns for rust-bpf.
 - bpf_prog_info tells that an invocation of the prog was cut short.

v2 -> v3
========
 - A scalar argument of a static subprog loses delta together with its id.
   v2 kept delta, took wrong bounds in the callee and let an access out of
   bounds load (sashiko).
 - A loop with calls is walked for two iterations. v2 widened it at once
   and rejected strobemeta.h with two maps or two strings.
 - The frame of a recursive subprog grows before the stack of a chain of
   calls is checked, not after (bpf-ci, sashiko).
 - A loop that ends when a flag is false is counted by the jump that clears
   the flag, also when code stands between that jump and the test. States
   with 0 and 1 in a flag are not joined in front of its test.
 - Patches 29-38 are new. Each loads code that bpf-next loads and v2
   rejects: casts, the context in a struct, r0 at a dead exit (bpf-ci,
   sashiko), a register that only a loop sets (bpf-ci), loops that count
   down in bytes, loop-carried values in long loops, and numbers that only
   a jump of the loop stops (Ihor).
 - The numbers below are those of v2.

v1 -> v2
========
v1 is part 3 of [1]. Its patches for the verifier of today and the time
slice of may_goto are in bpf-next.
 - Recursion: may_goto and r10, no per-cpu counters of bytes and calls.
 - flow.c comes in 9 patches and join.c in 2. In v1 each was one patch.
 - No public prog of Meta is lost (Ihor): widening goes to fixed stops, and
   constants that are added to a pointer to the packet are not joined.
 - A state is compared only with the states of a related shape (Ihor).
 - bpf_iter_*_next() stops a loop only if the iterator was created before
   the loop (Ihor). A loop with two entries is walked.
 - Offloaded progs get neither may_goto nor recursion (Ihor).
 - The failures of v1 in BPF CI (s390x, arm64, gcc) are fixed.

[1] https://lore.kernel.org/bpf/20261026025943.404450-1-alexei.starovoitov@gmail.com/

Signed-off-by: Alexei Starovoitov <ast@kernel.org>

Alexei Starovoitov (45):
  bpf: Add may_goto to the loops that the verifier marks
  bpf: Limit depth and time of recursive subprogs at run time
  bpf: Add union and widening of the range of a register
  bpf: Add join of registers and stack slots
  bpf: Add join of verifier states
  bpf: Tell which arguments of a call a join must leave alone
  bpf: Add the data structures of the main pass over the control flow
    graph
  bpf: Order the insns, find what stops a loop and where may_goto can go
  bpf: Find the registers that hold a number or a pointer to arena
  bpf: Find the registers and stack slots that later checks depend on
  bpf: Add the sets of states at an insn and the entries of functions
  bpf: Compare states of different iterations of a loop
  bpf: Decide which states are not joined
  bpf: Merge the state that arrives into the set at an insn
  bpf: Move states along edges, into loops and subprogs and out of them
  bpf: Make the state after a call from the caller's state and an exit
    state
  bpf: Enter and leave loops in the main pass, build flow.c
  bpf: Tell the passes over the graph where an added may_goto goes
  bpf: Verify programs as a fixpoint over the control flow graph
  bpf: Keep the count of a loop when a register has no closed form
  bpf: Verify static subprogs on their own
  bpf: Allow recursive subprogs
  bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth
  bpf: Remove precision tracking
  bpf: Remove prune points, levels of state comparison and pop_stack()
  bpf: Update comments about paths, pruning and explored states
  bpf: Track a pointer to one of a set of functions
  bpf: callx reads what the subprogs whose address is taken read
  bpf: Don't take what bpf_core_cast() returns for a number
  bpf: Don't widen away a register that only the loop sets
  bpf: A base that nothing is known about can be the context
  bpf: Keep the frames of the caller for a subprog that loads data_end
  bpf: Keep the frames of the caller for ctx of struct_ops with a
    reference
  bpf: R0 is live at the exit that an added may_goto goes to
  bpf: Keep the step of a number in the union of two ranges
  bpf: Keep the first iteration of a counted loop apart
  bpf: What a call returns in each iteration is like what the loop loads
  bpf: Find where the jumps of a loop stop a number
  selftests/bpf: Adjust tests to log lines that are gone
  selftests/bpf: Adjust tests to new verdicts, messages and order
  selftests/bpf: Test states that are not joined
  selftests/bpf: Test static subprogs that are verified on their own
  selftests/bpf: Test loops that are counted, walked or widened
  selftests/bpf: Test progs that run out of budget
  selftests/bpf: Test callx of one of a set of functions

 Documentation/bpf/bpf_design_QA.rst           |    7 +-
 Documentation/bpf/kfuncs.rst                  |    2 +-
 Documentation/bpf/linux-notes.rst             |    4 +-
 include/linux/bpf.h                           |    6 +
 include/linux/bpf_verifier.h                  |  577 +--
 include/linux/filter.h                        |   16 +
 kernel/bpf/Makefile                           |    2 +-
 kernel/bpf/backtrack.c                        | 1052 ------
 kernel/bpf/cfg.c                              |  297 +-
 kernel/bpf/const_fold.c                       |  134 +-
 kernel/bpf/core.c                             |   19 +
 kernel/bpf/diagnostics.c                      |   13 -
 kernel/bpf/diagnostics.h                      |    1 -
 kernel/bpf/fixups.c                           |  333 ++
 kernel/bpf/flow.c                             | 3302 +++++++++++++++++
 kernel/bpf/join.c                             |  664 ++++
 kernel/bpf/liveness.c                         |  114 +-
 kernel/bpf/log.c                              |   14 +-
 kernel/bpf/loops.c                            |  175 +-
 kernel/bpf/scev.c                             |  231 +-
 kernel/bpf/states.c                           | 1181 +-----
 kernel/bpf/verifier.c                         | 2942 +++++++++------
 .../bpf/prog_tests/bpf_verif_scale.c          |    4 +-
 .../bpf/prog_tests/cgroup_skb_egress_ret.c    |   23 +
 .../selftests/bpf/prog_tests/log_buf.c        |    4 +-
 .../selftests/bpf/prog_tests/log_fixup.c      |    6 +-
 .../selftests/bpf/prog_tests/reg_bounds.c     |    5 +-
 .../prog_tests/test_struct_ops_refcounted.c   |    2 +
 .../selftests/bpf/prog_tests/verifier.c       |   19 +
 .../selftests/bpf/progs/aggregate_ret_func.c  |   15 -
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |    7 -
 ...f_qdisc_fail__invalid_dynptr_cross_frame.c |    4 +-
 .../bpf/progs/cgroup_skb_egress_ret.c         |   12 +
 .../bpf/progs/compute_live_registers.c        |    8 +-
 .../selftests/bpf/progs/cpumask_failure.c     |    2 +-
 .../testing/selftests/bpf/progs/dynptr_fail.c |    4 +-
 .../selftests/bpf/progs/exceptions_assert.c   |   24 +-
 tools/testing/selftests/bpf/progs/iters.c     |  114 +-
 .../selftests/bpf/progs/iters_state_safety.c  |    8 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |   12 +-
 .../bpf/progs/mem_rdonly_untrusted.c          |   16 +-
 .../selftests/bpf/progs/stack_arg_precision.c |   48 +-
 .../bpf/progs/struct_ops_refcounted_subprog.c |   30 +
 .../selftests/bpf/progs/test_global_func15.c  |    5 -
 .../selftests/bpf/progs/test_global_func3.c   |  125 +
 .../selftests/bpf/progs/timer_failure.c       |    9 +-
 .../selftests/bpf/progs/verifier_align.c      |   10 +-
 .../selftests/bpf/progs/verifier_and.c        |    4 +-
 .../bpf/progs/verifier_arena_stack.c          |   82 +
 .../bpf/progs/verifier_array_access.c         |   14 +-
 .../selftests/bpf/progs/verifier_bounds.c     |  114 +-
 .../bpf/progs/verifier_bounds_deduction.c     |    2 +-
 .../bpf/progs/verifier_bounds_step.c          |   73 +-
 .../selftests/bpf/progs/verifier_budget.c     | 1898 ++++++++++
 .../bpf/progs/verifier_calls_in_a_row.c       |  223 ++
 .../selftests/bpf/progs/verifier_callx.c      |  128 +-
 .../selftests/bpf/progs/verifier_callx_many.c |  113 +
 .../bpf/progs/verifier_callx_rodata.c         |  167 +-
 .../selftests/bpf/progs/verifier_cfg.c        |    8 +-
 .../bpf/progs/verifier_global_subprogs.c      |   11 +-
 .../bpf/progs/verifier_helper_value_access.c  |    8 +-
 .../selftests/bpf/progs/verifier_idoms.c      |    9 +-
 .../bpf/progs/verifier_iterating_callbacks.c  |   70 +-
 .../bpf/progs/verifier_jumps_ahead.c          |  670 ++++
 .../bpf/progs/verifier_linked_scalars.c       |    7 +-
 .../selftests/bpf/progs/verifier_live_stack.c |   37 +-
 .../bpf/progs/verifier_liveness_exp.c         |    7 +-
 .../selftests/bpf/progs/verifier_loop_as_is.c |  294 ++
 .../bpf/progs/verifier_loop_hierarchy.c       |  125 +-
 .../selftests/bpf/progs/verifier_loop_latch.c |   82 +-
 .../selftests/bpf/progs/verifier_loop_need.c  |  371 ++
 .../selftests/bpf/progs/verifier_loop_walk.c  |  306 ++
 .../selftests/bpf/progs/verifier_loops1.c     |   16 +-
 .../bpf/progs/verifier_map_ret_val.c          |    2 +-
 .../bpf/progs/verifier_mem_size_reg.c         |   52 +-
 .../selftests/bpf/progs/verifier_movsx.c      |    3 +-
 .../selftests/bpf/progs/verifier_precision.c  |  186 +-
 .../selftests/bpf/progs/verifier_ptr_off.c    |  279 ++
 .../selftests/bpf/progs/verifier_scalar_ids.c |  329 +-
 .../selftests/bpf/progs/verifier_scev.c       |  269 +-
 .../bpf/progs/verifier_search_pruning.c       |   40 +-
 .../selftests/bpf/progs/verifier_spill_fill.c |  224 +-
 .../selftests/bpf/progs/verifier_spin_lock.c  |    4 +-
 .../selftests/bpf/progs/verifier_stack_arg.c  |  181 +
 .../selftests/bpf/progs/verifier_stack_ptr.c  |   13 +-
 .../bpf/progs/verifier_subprog_callers.c      |  543 +++
 .../bpf/progs/verifier_subprog_insn_stats.c   |   16 +-
 .../bpf/progs/verifier_subprog_precision.c    |  353 +-
 .../selftests/bpf/progs/verifier_tailcall.c   |   27 +
 .../selftests/bpf/progs/verifier_uninit.c     |    2 +-
 .../selftests/bpf/progs/verifier_unpriv.c     |   23 +-
 .../bpf/progs/verifier_unpriv_perf.c          |    4 +-
 .../bpf/progs/verifier_value_or_null.c        |    2 +-
 .../bpf/progs/verifier_value_ptr_arith.c      |   61 +-
 .../selftests/bpf/progs/verifier_var_off.c    |  206 +-
 tools/testing/selftests/bpf/verifier/calls.c  |   46 +-
 .../testing/selftests/bpf/verifier/precise.c  |   69 +-
 97 files changed, 14016 insertions(+), 5359 deletions(-)
 delete mode 100644 kernel/bpf/backtrack.c
 create mode 100644 kernel/bpf/flow.c
 create mode 100644 kernel/bpf/join.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_budget.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_many.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_need.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_walk.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_ptr_off.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_callers.c

-- 
2.55.0

[PATCH bpf-next v3 01/45] bpf: Add may_goto to the loops that the verifier marks

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

SpinLock::lock() of rust-bpf (scx_common.rs), here in central_enqueue:

  while self.locked.compare_exchange(0, 1, Acquire, Relaxed).is_err() {
      while self.locked.load(Relaxed) != 0 { core::hint::spin_loop(); }
  }

  infinite loop detected at insn 127

111 of the 225 progs of the schedulers of rust-bpf get this message. Memory
decides when the loop ends: no analysis finds a bound. The same holds for
a lookup in a BTreeMap. C has bpf_for(), can_loop, bpf_loop(). Nobody can
put them into core and alloc.

With one state for all iterations the verifier can still show that each of
them is safe. Then the kernel must stop the loop at run time. may_goto does
that: after 250 ms it jumps to its target. That jump is a new edge in the
prog, and must be verified. One target for all loops is not possible:

  lock(), see above                 pick_idle_cpu_in() of scx_p2dq.rs
  Not to the exit of the loop:        let cores = core.then(..::idle_smt);
  the caller goes on as if it         cpus.filter(..).find(..)
  had the lock. Unwind instead:     Not out of the function: the code
                                    after the loop releases the cpumask.
  may_goto L
  insn                              may_goto T
  ...                               if ... goto T
  last insn of the subprog
  L: r0 = 0
  unwind_exit, main prog: exit      T is outside of the loop

Thus the main pass selects the edge for each loop, verifies it, and
records it in insn_aux_data. bpf_insert_loop_guards() adds the insns after
the main pass. No code sets loop_guard in this patch: no prog changes.
 - unwind_exit is for a subprog whose callers have cleanup code for it
   (bail_unwinds): their destructors run, the code after the call does not.
 - The inner loop of lock() goes around without a jump that leaves. Then
   may_goto is at the loop head (guard_at_header): 142 of the 225 progs.
 - may_goto has a 16-bit offset. If L is more than 2048 insns away it gets
   'r0 = 0; exit' of its own, and one more jump in each iteration.

Progs without such a loop get the same code. 174 of the 225 progs have one.
An added may_goto costs what a may_goto of the prog costs. It is 10 insns,
14 if no register is dead: thus it gets the live registers of the insn that
follows it. That, and one L per subprog, not per loop: 6 % less jited code.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  33 ++++++
 kernel/bpf/fixups.c          | 219 +++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |   4 +
 3 files changed, 256 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d6e41444ab14..7e268540dae9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -815,6 +815,10 @@ struct bpf_insn_aux_data {
 	u64 cast_arena:1; /* addr_space_cast of an address in arena */
 	u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
 	u64 bb_end:1; /* last instruction of a basic block */
+	u64 loop_guard:1; /* may_goto is added in front of this instruction */
+	u64 guard_place:1; /* it's the place for may_goto if the loop needs one */
+	u64 guard_at_header:1; /* there is no better place than the head of the loop */
+	u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
 	u64 need_scev:1;
 	u64 alu_state:5; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
@@ -878,6 +882,8 @@ struct bpf_insn_aux_data {
 	 * For a loop header, identifies the parent loop instead.
 	 */
 	s32 loop_header;
+	/* where may_goto in front of this instruction goes */
+	s32 guard_to;
 	/* additional information about the loop if this instruction is a loop header */
 	struct bpf_loop *loop;
 };
@@ -1021,6 +1027,8 @@ struct bpf_subprog_info {
 	 * a hidden pointer to the result, a pair in two registers, no dead arguments.
 	 */
 	bool stack_args_from_code: 1;
+	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
+	bool bail_unwinds: 1;
 	u8 code_stack_args;
 	u16 stack_arg_cnt; /* incoming + max outgoing */
 	u16 max_out_stack_arg_cnt;
@@ -1521,6 +1529,7 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_split_cur_state(struct bpf_verifier_env *env);
@@ -1845,6 +1854,30 @@ struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *en
 const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);
 int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
+
+/*
+ * Where the program goes from may_goto and from the checks that the verifier adds:
+ * - May_goto in front of a jump goes where the jump goes, with the registers and the stack
+ *   as they are in front of the jump. bpf_insn_successors() has that edge.
+ * - A subprog that is out of budget goes to bail_insn, an exit. No such edge.
+ * - May_goto in front of the head of a loop goes to where the loops around it are left: from
+ *   @idx to what is returned here, -1 if there is none. No such edge either. The registers
+ *   and the stack are what they are in front of @idx.
+ *
+ * The main pass looks at the states that go there. What is computed from the graph before
+ * the main pass has to take these edges into account where the main pass believes it:
+ * - Against the direction of the edges, liveness: a register that is live at the target is
+ *   live at @idx. Stack that is dead at @idx stays dead: a read of it at the target is
+ *   rejected.
+ * - In the direction of the edges, bpf_compute_const_regs() and compute_subprog_args(): what
+ *   holds on all edges of the graph to the target doesn't hold on this one.
+ */
+static inline int bpf_head_guard_target(const struct bpf_verifier_env *env, int idx)
+{
+	const struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+
+	return aux->guard_at_header ? aux->guard_to : -1;
+}
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index f95af2ca3591..6fd79c2fbdc1 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -587,6 +587,225 @@ static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
 	return 0;
 }
 
+static bool jumps_to_itself(const struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+	return (class == BPF_JMP || class == BPF_JMP32) && op != BPF_CALL && op != BPF_EXIT &&
+	       insn->code != (BPF_JMP32 | BPF_JA) && insn->off == -1;
+}
+
+/* How a subprog leaves when it is out of budget, see push_budget_exit() */
+static struct bpf_insn budget_exit_insn(struct bpf_verifier_env *env, struct bpf_subprog_info *sub)
+{
+	if (!sub->bail_unwinds)
+		return BPF_EXIT_INSN();
+	sub->can_unwind = true;
+	env->prog->jit_required = 1;
+	return BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0);
+}
+
+/*
+ * may_goto has 16 bits for the offset, and what is between it and its target is going
+ * to grow: may_goto itself becomes 10 insns, 14 when it has to save a register.
+ */
+#define BPF_GUARD_NEAR	2048
+
+static bool guard_leaves_subprog(const struct bpf_insn_aux_data *aux)
+{
+	return aux->loop_guard && !aux->guard_no_unwind;
+}
+
+/*
+ * 'r0 = 0; exit' behind the last insn of the subprog, for all its guards that are near.
+ * Returns the number of insns that are going to be added.
+ */
+static int append_budget_exit(struct bpf_verifier_env *env, int s)
+{
+	struct bpf_subprog_info *sub = &env->subprog_info[s];
+	int last = (sub + 1)->start - 1, n = 0, i;
+	struct bpf_insn buf[4], *insn;
+
+	for (i = max_t(int, sub->start, last + 1 - BPF_GUARD_NEAR); i <= last; i++)
+		if (guard_leaves_subprog(&env->insn_aux_data[i]))
+			break;
+	if (i > last)
+		return 0;
+
+	buf[n++] = env->prog->insnsi[last];
+	buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+	if (bpf_ret_reg_pair(env, s))
+		buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
+	buf[n++] = budget_exit_insn(env, sub);
+	insn = bpf_patch_list_add(env, last, buf, n);
+	return IS_ERR(insn) ? PTR_ERR(insn) : n - 1;
+}
+
+/*
+ * Add may_goto in front of the insns that the main pass marked, see
+ * check_loop_guard(). It verified the edge from the insn to where may_goto goes.
+ */
+int bpf_insert_loop_guards(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int len = env->prog->len, i, r, s, to, n, jmp, err = 0;
+	struct bpf_insn *patch, *insn;
+	int *target, *leave, *moved;
+	bool indirect;
+	u16 *zero;
+	s64 off;
+
+	for (i = 0; i < len; i++)
+		if (aux[i].loop_guard)
+			break;
+	if (i == len)
+		return 0;
+
+	/*
+	 * Where the subprog has 'r0 = 0; exit', 0 if it has none. And how far that moved
+	 * the subprog: guard_to is from the time before.
+	 */
+	leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
+	if (!leave)
+		return -ENOMEM;
+	moved = leave + env->subprog_cnt;
+	for (s = 0; s < env->subprog_cnt && !err; s++) {
+		leave[s] = append_budget_exit(env, s);
+		if (leave[s] < 0)
+			err = leave[s];
+	}
+	/* A guard cannot jump into a patch that is in the list: 'r0 = 0' has no index yet. */
+	err = err ?: bpf_patch_list_commit(env);
+	if (err) {
+		kvfree(leave);
+		return err;
+	}
+	aux = env->insn_aux_data;
+	len = env->prog->len;
+	for (s = 0, n = 0; s < env->subprog_cnt; s++) {
+		moved[s] = n;
+		n += leave[s];
+		if (!leave[s])
+			continue;
+		i = env->subprog_info[s + 1].start;
+		leave[s] = i - leave[s];
+		/*
+		 * The list gives what is known about an insn to the last insn of its patch,
+		 * but for indirect_target. Here the original insn is the first one.
+		 */
+		indirect = aux[leave[s] - 1].indirect_target;
+		swap(aux[leave[s] - 1], aux[i - 1]);
+		aux[leave[s] - 1].indirect_target = indirect;
+		aux[i - 1].indirect_target = false;
+		/* the last insn may be dead code */
+		while (--i >= leave[s])
+			aux[i].seen = 1;
+	}
+
+	/* where the guard of the insn goes */
+	target = kvcalloc(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+	zero = kvcalloc(len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+	patch = kvcalloc(8, sizeof(*patch), GFP_KERNEL_ACCOUNT);
+	if (!target || !zero || !patch) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < len; i++) {
+		s = bpf_find_containing_subprog(env, i) - env->subprog_info;
+		target[i] = aux[i].loop_guard ? aux[i].guard_to + moved[s] : -1;
+		if (guard_leaves_subprog(&aux[i])) {
+			/* see push_budget_exit() */
+			if (leave[s] && leave[s] - i <= BPF_GUARD_NEAR) {
+				target[i] = leave[s];
+			} else {
+				target[i] = -2;
+				zero[i] = BIT(BPF_REG_0);
+				if (bpf_ret_reg_pair(env, s))
+					zero[i] |= BIT(BPF_REG_2);
+			}
+		}
+	}
+
+	for (i = 0; i < len; i++) {
+		to = target[i];
+		if (to == -1)
+			continue;
+		/*
+		 *	may_goto to		or	may_goto +1
+		 *	insn				goto +N
+		 *					r0 = 0, ...
+		 *					exit
+		 *					insn
+		 */
+		n = 0;
+		if (zero[i]) {
+			patch[n++] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0);
+			patch[n++] = BPF_JMP_A(hweight16(zero[i]) + 1);
+			for (r = BPF_REG_0; r < BPF_REG_FP; r++)
+				if (zero[i] & BIT(r))
+					patch[n++] = BPF_MOV64_IMM(r, 0);
+		}
+		jmp = n++;
+		/* What is behind moves with this patch. The other patches are not there yet. */
+		if (to == -2)
+			off = 0;
+		else if (to > i)
+			off = to - i + n - jmp - 1;
+		else
+			off = to - (i + jmp) - 1;
+		if (off > S16_MAX || off < S16_MIN) {
+			verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
+			err = -ERANGE;
+			break;
+		}
+		patch[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
+		if (to == -2)
+			patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env, i));
+		patch[n] = env->prog->insnsi[i];
+		/* A jump to itself is a back edge. It goes through may_goto like the others. */
+		if (jumps_to_itself(&patch[n]))
+			patch[n].off -= n;
+		else
+			err = bpf_adj_moved_insn(&patch[n], n);
+		if (err) {
+			verbose(env, "insn %d cannot be patched due to 16-bit range\n", i);
+			break;
+		}
+		n++;
+		insn = bpf_patch_list_add(env, i, patch, n);
+		if (IS_ERR(insn)) {
+			err = PTR_ERR(insn);
+			break;
+		}
+	}
+	err = err ?: bpf_patch_list_commit(env);
+	if (err)
+		goto out;
+	/*
+	 * bpf_do_misc_fixups() keeps the address of the count of may_goto in a register that is
+	 * dead at the may_goto. A new insn has no live_regs_before: all registers would be dead.
+	 * What is live in front of the insn is live in front of its guard. Nothing else is:
+	 *  - The guard of a jump goes where the jump goes.
+	 *  - What is live where the guard at the head of a loop goes is live at the head,
+	 *    see bpf_head_guard_target().
+	 *  - On the way out of the subprog R0, and R2 of a pair, are set to 0.
+	 */
+	aux = env->insn_aux_data;
+	for (i = 0, n = 0; i < len; i++) {
+		if (target[i] == -1)
+			continue;
+		r = zero[i] ? 3 + hweight16(zero[i]) : 1;
+		aux[i + n].live_regs_before = aux[i + n + r].live_regs_before;
+		n += r;
+	}
+out:
+	kvfree(leave);
+	kvfree(target);
+	kvfree(zero);
+	kvfree(patch);
+	return err;
+}
+
 /* Clean up dynamically allocated fields of aux data for instructions [start, ...] */
 void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
 {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b7329d164771..46ce62879d2a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -24664,9 +24664,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret == 0 && env->callx_edges)
 		ret = sort_subprogs_topo(env);
 
+	/* the driver numbers the insns like the program that it was given */
 	if (ret == 0 && bpf_prog_is_offloaded(env->prog->aux))
 		ret = bpf_prog_offload_finalize(env);
 
+	if (ret == 0)
+		ret = bpf_insert_loop_guards(env);
+
 skip_full_check:
 	kvfree(env->explored_states);
 
-- 
2.55.0

Re: [PATCH bpf-next v3 01/45] bpf: Add may_goto to the loops that the verifier marks

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: bpf_insert_loop_guards() logs shifted insn index instead of orig_idx
--

commit 4a3883a17feaa6708a44e871a9a6eab87f5197bc
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Add may_goto to the loops that the verifier marks

This commit introduces bpf_insert_loop_guards() to insert may_goto
instructions in front of loop instructions marked by the verifier, along
with per-subprogram budget exit sequences when needed.

> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index f95af2ca3591b..6fd79c2fbdc1e 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -587,6 +587,225 @@ static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)

[ ... ]

> +int bpf_insert_loop_guards(struct bpf_verifier_env *env)
> +{
> +	struct bpf_insn_aux_data *aux = env->insn_aux_data;
> +	int len = env->prog->len, i, r, s, to, n, jmp, err = 0;
> +	struct bpf_insn *patch, *insn;
> +	int *target, *leave, *moved;
> +	bool indirect;
> +	u16 *zero;
> +	s64 off;
> +
> +	for (i = 0; i < len; i++)
> +		if (aux[i].loop_guard)
> +			break;
> +	if (i == len)
> +		return 0;
> +
> +	/*
> +	 * Where the subprog has 'r0 = 0; exit', 0 if it has none. And how far that moved
> +	 * the subprog: guard_to is from the time before.
> +	 */
> +	leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
> +	if (!leave)
> +		return -ENOMEM;
> +	moved = leave + env->subprog_cnt;
> +	for (s = 0; s < env->subprog_cnt && !err; s++) {
> +		leave[s] = append_budget_exit(env, s);
> +		if (leave[s] < 0)
> +			err = leave[s];
> +	}
> +	/* A guard cannot jump into a patch that is in the list: 'r0 = 0' has no index yet. */
> +	err = err ?: bpf_patch_list_commit(env);

[ ... ]

> +	for (i = 0; i < len; i++) {
> +		to = target[i];
> +		if (to == -1)
> +			continue;
> +		/*
> +		 *	may_goto to		or	may_goto +1
> +		 *	insn				goto +N
> +		 *					r0 = 0, ...
> +		 *					exit
> +		 *					insn
> +		 */
> +		n = 0;
> +		if (zero[i]) {
> +			patch[n++] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0);
> +			patch[n++] = BPF_JMP_A(hweight16(zero[i]) + 1);
> +			for (r = BPF_REG_0; r < BPF_REG_FP; r++)
> +				if (zero[i] & BIT(r))
> +					patch[n++] = BPF_MOV64_IMM(r, 0);
> +		}
> +		jmp = n++;
> +		/* What is behind moves with this patch. The other patches are not there yet. */
> +		if (to == -2)
> +			off = 0;
> +		else if (to > i)
> +			off = to - i + n - jmp - 1;
> +		else
> +			off = to - (i + jmp) - 1;
> +		if (off > S16_MAX || off < S16_MIN) {
> +			verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
> +			err = -ERANGE;
> +			break;
> +		}
> +		patch[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
> +		if (to == -2)
> +			patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env, i));
> +		patch[n] = env->prog->insnsi[i];
> +		/* A jump to itself is a back edge. It goes through may_goto like the others. */
> +		if (jumps_to_itself(&patch[n]))
> +			patch[n].off -= n;
> +		else
> +			err = bpf_adj_moved_insn(&patch[n], n);
> +		if (err) {
> +			verbose(env, "insn %d cannot be patched due to 16-bit range\n", i);
> +			break;
> +		}

[Severity: Low]
As reported by Sashiko on v2 [1], should these two verbose() calls in
bpf_insert_loop_guards() log aux[i].orig_idx instead of i?

The first pass in bpf_insert_loop_guards() calls append_budget_exit() for
each subprogram with a near loop guard that leaves the subprogram, and
commits those patches via bpf_patch_list_commit(env). Each appended budget
exit adds 2 or 3 instructions to the end of its subprogram, shifting the
instruction index i in all subsequent subprograms by moved[s]. If the
second pass then fails because off overflows 16 bits or
bpf_adj_moved_insn() returns -ERANGE, logging i reports the shifted index
rather than the original instruction index from the user program (unlike
bpf_patch_list_insert(), bpf_patch_list_commit(), and
bpf_convert_ctx_accesses() in kernel/bpf/fixups.c, which log orig_idx).

Similarly, in kernel/bpf/verifier.c:bpf_check():

	if (ret == 0)
		ret = bpf_insert_loop_guards(env);

skip_full_check:
	kvfree(env->explored_states);

	/* might decrease stack depth, keep it before passes that
	 * allocate additional slots.
	 */
	if (ret == 0)
		ret = bpf_remove_fastcall_spills_fills(env);

	if (ret == 0)
		ret = check_max_stack_depth(env);

	/* instruction rewrites happen after this point */
	if (ret == 0)
		ret = bpf_optimize_bpf_loop(env);

bpf_insert_loop_guards() is called before check_max_stack_depth(), ahead of
the "instruction rewrites happen after this point" boundary. Because helpers
called by check_max_stack_depth() (such as find_call_insn() in
check_exception_cb() and check_recursion(), as well as check_callbacks())
log raw instruction indices in env->prog->insnsi rather than orig_idx,
rewriting the program in bpf_insert_loop_guards() before
check_max_stack_depth() will also shift the instruction indices reported by
those diagnostics. Should bpf_insert_loop_guards() be moved after
check_max_stack_depth() with the other instruction rewrite passes?

[1] https://lore.kernel.org/r/20261105204513.2230344-2-alexei.starovoitov@gmail.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=1

Re: [PATCH bpf-next v3 01/45] bpf: Add may_goto to the loops that the verifier marks

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1634 bytes --]

> bpf: Add may_goto to the loops that the verifier marks

TL;DR: Nits only in bpf_insert_loop_guards(): the scratch patch array could
live on the stack instead of kvcalloc, and the to == -2 case writes patch[jmp]
twice. No bugs reported.

> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c

[ ... ]

> +	patch = kvcalloc(8, sizeof(*patch), GFP_KERNEL_ACCOUNT);
> +	if (!target || !zero || !patch) {
> +		err = -ENOMEM;
> +		goto out;
> +	}

this isn't a bug, but could patch be a small on-stack array, as
append_budget_exit() does with its struct bpf_insn buf[4]?  It is a fixed
scratch area of at most 6 insns, and the heap allocation adds an -ENOMEM
path.

[ ... ]

> +		if (to == -2)
> +			off = 0;
> +		else if (to > i)
> +			off = to - i + n - jmp - 1;
> +		else
> +			off = to - (i + jmp) - 1;
> +		if (off > S16_MAX || off < S16_MIN) {
> +			verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
> +			err = -ERANGE;
> +			break;
> +		}
> +		patch[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
> +		if (to == -2)
> +			patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env, i));

this isn't a bug, but in the to == -2 case patch[jmp] is first written as a
may_goto and then overwritten by budget_exit_insn() on the next line.  Could
the exit insn be built directly in that case?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 02/45] bpf: Limit depth and time of recursive subprogs at run time

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Rust recurses where C has a loop. scx_flatcg.rs of rust-bpf has

  struct CgrpCtx { parent: Option<Arc<CgrpCtx>>, ... }

The drop of the last Arc of a cgroup drops the Arc of its parent:
Arc::drop_slow() calls itself, and 8 of the 15 progs are rejected:
"recursive call from drop_slow<..>() to drop_slow<..>()". quicksort() under
sort_unstable_by() in scx_rusty.rs, BTreeMap, serde_json and regex-automata
recurse too: 100 of the 225 progs of the schedulers of rust-bpf, 100 of its
437 tests. The verifier cannot know how deep the calls go, or how many
there are.

Limit both at run time. Each subprog on a cycle of the call graph starts
with two checks:

  entry:  may_goto bail                                 limits the time
          if (r10 is in the upper half of the stack)    limits the depth
                  goto body
          bpf_prog_report_recursion()
  bail:   r0 = 0; exit
  body:   ...

Both are needed. The depth does not limit the time:

  int fib(int n) { return n < 2 ? n : fib(n - 1) + fib(n - 2); }

fib(90) is never deeper than 90 frames and makes 10^19 calls. There is no
loop in it, so no may_goto of a loop bounds it. The cycle is in the call
graph, and it goes through the first insn of fib(). With may_goto there
fib(90) ends after 250 ms, the time slice that the loops and recursions of
an invocation share. The time does not limit the depth:
'int f(int n) { return f(n + 1) + 1; }' is out of stack in less than 1 ms.

r10 is an address in the kernel stack that the prog runs on. Thus the check
sees all that is in this stack, not only the frames of this prog, without
per-cpu state. On arm64 a stack is aligned to its size: one bit of r10
tells the half. On x86-64 r10 is compared with the tops of the task stack
and the irq stack. In any other stack, like in an NMI that interrupts the
kernel, the check fails. Other archs get no recursion. The stack in arena
has no check: the verifier makes the frame in the kernel stack so large
that a half of that stack is full before the stack in arena. It does so in
check_call_depth(): the limit on the stack of a chain of calls has to see
the frames that the JIT gets.

No subprog is marked as recursive yet. A call runs 12 more insns on x86-64
(17 on the irq stack), 8 on arm64: +0.2 to +0.5 nsec on x86-64.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |   3 +
 include/linux/filter.h       |  16 +++++
 kernel/bpf/core.c            |  19 ++++++
 kernel/bpf/fixups.c          | 113 +++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |  49 +++++++++++++++
 6 files changed, 202 insertions(+)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index b011acc22c50..cdf3c9b90987 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1944,6 +1944,8 @@ struct bpf_prog_aux {
 	bool has_prologue;
 	bool might_sleep;
 	bool can_unwind;
+	/* frames of recursive subprogs can be on the stack when it is called */
+	bool in_recursion;
 	bool kprobe_write_ctx;
 	/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
 	bool uses_arena_insns;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7e268540dae9..2f935061b8fd 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -995,6 +995,8 @@ struct bpf_subprog_info {
 	bool has_tail_call: 1;
 	bool might_throw: 1;
 	bool can_unwind: 1; /* has unwind_exit */
+	bool recursive: 1; /* on a cycle of the call graph */
+	bool in_recursion: 1; /* recursive, or a recursive subprog gets to it */
 	bool via_callx: 1; /* callx calls it */
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
@@ -1532,6 +1534,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
+int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
 int bpf_split_cur_state(struct bpf_verifier_env *env);
 int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 
diff --git a/include/linux/filter.h b/include/linux/filter.h
index a58ad2318412..61448be53f8d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1401,6 +1401,21 @@ bool bpf_jit_supports_large_stack(void);
 bool bpf_jit_supports_timed_may_goto(void);
 bool bpf_jit_supports_fsession(void);
 
+/*
+ * Whether a recursion in @prog can be limited at run time:
+ *  - r10 tells how much of the kernel stack is used, on the architectures whose stacks
+ *    emit_stack_check() knows.
+ *  - All may_goto of the prog share one time slice. That is the timed form, which only JITed
+ *    progs get, see bpf_do_misc_fixups(). The other form counts in each frame.
+ *  - A walk of the stack finds the prog when there is no room, to tell user space, see
+ *    bpf_prog_report_recursion().
+ */
+static inline bool bpf_prog_has_invocation_budget(const struct bpf_prog *prog)
+{
+	return (IS_ENABLED(CONFIG_X86_64) || IS_ENABLED(CONFIG_ARM64)) && prog->jit_requested &&
+	       bpf_jit_supports_timed_may_goto() && bpf_jit_supports_exceptions();
+}
+
 struct bpf_jit_arg_abi {
 	/* Argument registers of the kernel convention. */
 	u8 nr_arg_regs;
@@ -1438,6 +1453,7 @@ u64 arch_bpf_timed_may_goto(void);
 u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *);
 void bpf_timed_may_goto_save(const struct bpf_prog *prog, struct bpf_timed_may_goto *saved);
 void bpf_timed_may_goto_restore(const struct bpf_prog *prog, struct bpf_timed_may_goto *saved);
+u64 bpf_prog_report_recursion(void);
 bool bpf_helper_changes_pkt_data(enum bpf_func_id func_id);
 
 static inline bool bpf_dump_raw_ok(const struct cred *cred)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 8e2fb66940c7..f88a1c041a7a 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3956,6 +3956,25 @@ static noinline void bpf_prog_report_may_goto_violation(void)
 #endif
 }
 
+/* Called by a recursive subprog that finds no room, see bpf_insert_recursion_checks() */
+u64 bpf_prog_report_recursion(void)
+{
+#ifdef CONFIG_BPF_SYSCALL
+	struct bpf_stream_stage ss;
+	struct bpf_prog *prog;
+
+	prog = bpf_prog_find_from_stack();
+	/* The calls that follow get here too. One report is enough until user space reads it. */
+	if (!prog || atomic_read(&prog->aux->stream[BPF_STDERR - 1]->capacity))
+		return 0;
+	bpf_stream_stage(ss, prog, BPF_STDERR, ({
+		bpf_stream_printk(ss, "ERROR: No room in the stack for a call of a recursive function\n");
+		bpf_stream_dump_stack(ss);
+	}));
+#endif
+	return 0;
+}
+
 u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *p)
 {
 	u64 time = ktime_get_mono_fast_ns();
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 6fd79c2fbdc1..3413c7addca8 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -806,6 +806,118 @@ int bpf_insert_loop_guards(struct bpf_verifier_env *env)
 	return err;
 }
 
+/*
+ * if (r10 is in the upper half of the kernel stack) goto +@off
+ *
+ * r10 of a subprog without a private stack is an address in the kernel stack that it runs on.
+ * arm64: the stacks of tasks, of interrupts and of SDEI events have THREAD_SIZE bytes and are
+ * aligned to twice that size. One bit of an address tells the half.
+ * x86-64: the stack for interrupts is aligned to a page only. But the cpu has the top of the stack
+ * of the task and the top of its stack for interrupts. In any other stack (NMI, #DB, #MC, #DF,
+ * #VC) the check fails: they have half of the size.
+ * R0 has nothing at the entry of a subprog.
+ */
+static int emit_stack_check(struct bpf_insn *buf, int off)
+{
+	int n = 0;
+
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+	const void __percpu *top[] = { &cpu_current_top_of_stack, &hardirq_stack_ptr };
+	int i;
+
+	static_assert(IRQ_STACK_SIZE == THREAD_SIZE);
+	for (i = 0; i < 2; i++) {
+		buf[n++] = BPF_MOV64_IMM(BPF_REG_0, (u32)(__force unsigned long)top[i]);
+		buf[n++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
+		buf[n++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0);
+		buf[n++] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_10);
+		buf[n++] = BPF_JMP_IMM(BPF_JLT, BPF_REG_0, THREAD_SIZE / 2, i ? off : off + 5);
+	}
+#else
+	buf[n++] = BPF_JMP_IMM(BPF_JSET, BPF_REG_10, THREAD_SIZE / 2, off);
+#endif
+	return n;
+}
+
+/*
+ * How deep a recursion goes is not known, so it is limited at run time. In the same
+ * spirit as may_goto: the program goes on for as long as there is room and takes a way
+ * out that was verified when there is none.
+ *
+ * Every subprog on a cycle of the call graph begins with:
+ *
+ *	may_goto bail;
+ *	if (r10 is in the upper half of the kernel stack) goto body;
+ *	bpf_prog_report_recursion();
+ * bail:
+ *	r0 = 0;
+ *	exit;
+ * body:
+ *
+ * It is the stack as it is: whatever is there is counted, be it the callers of the prog in the
+ * kernel, another prog, a trampoline or a helper that calls a callback. An invocation in
+ * another stack has nothing to do with it.
+ *
+ * The lower half is for what is below r10: the frame of the subprog, then what it calls.
+ * That is the next recursive subprog, which doesn't get past its check, or a chain of calls that
+ * check_max_stack_depth() looked at together with that frame.
+ *
+ * A subprog that calls itself twice doesn't have to go deep to go on forever: hence may_goto.
+ * Once the time slice of the invocation is over every call returns at once, and the frames
+ * that are there come to an end.
+ */
+int bpf_insert_recursion_checks(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	struct bpf_insn buf[24], nop = BPF_JMP_A(0), *insn;
+	bool found = false;
+	int s, n, jmp, err;
+
+	for (s = 1; s < env->subprog_cnt; s++)
+		found |= si[s].recursive;
+	if (!found)
+		return 0;
+	env->prog->jit_required = 1;
+
+	for (s = 1; s < env->subprog_cnt; s++) {
+		if (!si[s].recursive)
+			continue;
+		n = 0;
+		/* The exit drops the frame in the stack in arena. Take it before the check. */
+		if (is_arena_sp_adj(&env->prog->insnsi[si[s].start])) {
+			buf[n++] = env->prog->insnsi[si[s].start];
+			/* one insn for one: done at once, the list copies the nop */
+			insn = bpf_patch_list_add(env, si[s].start, &nop, 1);
+			if (IS_ERR(insn))
+				return PTR_ERR(insn);
+		}
+		jmp = n++;
+		n += emit_stack_check(buf + n, bpf_ret_reg_pair(env, s) ? 4 : 3);
+		/* The arguments are not needed any more */
+		buf[n++] = BPF_EMIT_KERNEL_CALL(bpf_prog_report_recursion);
+		buf[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, n - jmp - 1, 0);
+		buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+		if (bpf_ret_reg_pair(env, s))
+			buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
+		buf[n++] = budget_exit_insn(env, &si[s]);
+		/* calls get to the check, jumps to the first insn do not */
+		err = bpf_patch_list_insert(env, si[s].start, buf, n);
+		if (err)
+			return err;
+	}
+	err = bpf_patch_list_commit(env);
+	if (err)
+		return err;
+	/* may_goto keeps the address of its count in a register that is dead: R0 */
+	for (s = 1; s < env->subprog_cnt; s++) {
+		if (!si[s].recursive)
+			continue;
+		n = si[s].start + is_arena_sp_adj(&env->prog->insnsi[si[s].start]);
+		env->insn_aux_data[n].live_regs_before = (u16)~BIT(BPF_REG_0);
+	}
+	return 0;
+}
+
 /* Clean up dynamically allocated fields of aux data for instructions [start, ...] */
 void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
 {
@@ -1878,6 +1990,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
 		func[i]->aux->can_unwind = env->subprog_info[i].can_unwind;
+		func[i]->aux->in_recursion = env->subprog_info[i].in_recursion;
 		func[i]->aux->token = prog->aux->token;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 46ce62879d2a..8f61613fb5a0 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6103,6 +6103,47 @@ static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_gra
 	} while (changed);
 }
 
+/*
+ * The stack in arena has no check of its own, see bpf_insert_recursion_checks(). A frame
+ * of a recursive subprog takes at least the same part of a half of the kernel stack as of the
+ * room in the stack in arena: there is room in arena for the frames that pass the check. The
+ * last of them and the one that doesn't pass come on top.
+ *
+ * Returns 1 when frames in the kernel stack grew.
+ */
+static int pad_recursive_frames(struct bpf_verifier_env *env, struct bpf_call_graph *cg,
+				struct bpf_subprog_call_depth_info *dinfo)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	u32 arena_frame = 0, depth = 0, room, frame;
+	int s;
+
+	for_each_set_bit(s, cg->live, cg->cnt) {
+		depth = max_t(u32, depth, dinfo[s].arena_depth + si[s].arena_stack_depth);
+		if (si[s].recursive)
+			arena_frame = max(arena_frame, si[s].arena_stack_depth);
+	}
+	if (!arena_frame)
+		return 0;
+	if (depth + 2 * arena_frame >= BPF_ARENA_STACK_MAX)
+		goto no_room;
+	room = BPF_ARENA_STACK_MAX - depth - 2 * arena_frame;
+	env->max_arena_stack_depth = BPF_ARENA_STACK_MAX;
+
+	for_each_set_bit(s, cg->live, cg->cnt) {
+		if (!si[s].recursive)
+			continue;
+		frame = DIV_ROUND_UP(si[s].arena_stack_depth * (THREAD_SIZE / 2), room);
+		if (frame > env->stack_limit)
+			goto no_room;
+		si[s].stack_depth = max(si[s].stack_depth, frame);
+	}
+	return 1;
+no_room:
+	verbose(env, "no room in the stack in arena for recursion\n");
+	return -EACCES;
+}
+
 static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
 {
 	struct bpf_subprog_info *si = env->subprog_info;
@@ -6113,6 +6154,11 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
 	if (!dinfo)
 		return -ENOMEM;
 	compute_call_depth(env, cg, dinfo);
+	err = pad_recursive_frames(env, cg, dinfo);
+	if (err < 0)
+		goto out;
+	if (err)
+		compute_call_depth(env, cg, dinfo);
 
 	for_each_set_bit(s, cg->live, cg->cnt) {
 		/* A caller of it is over a limit. That one is reported, not what it calls. */
@@ -24687,6 +24733,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret == 0)
 		ret = bpf_optimize_bpf_loop(env);
 
+	if (ret == 0)
+		ret = bpf_insert_recursion_checks(env);
+
 	if (ret == 0)
 		bpf_lower_cast_mov(env);
 
-- 
2.55.0

[PATCH bpf-next v3 03/45] bpf: Add union and widening of the range of a register

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

format_cgrp_path() of scx_layered, prog layered_runnable:

  u32 len = 0;
  bpf_for(level, 1, max_level + 1) {
          if (level > 1 && len < MAX_PATH - 1)
                  path[len++] = '/';
          ...
          len += ret - 1;
  }

The verifier follows each path, and its work grows with the number of
paths. To make it grow with the size of the prog, the states that meet at
an insn must become one state. Then the loop head has one state for all
iterations. len in it is the union of what arrives there: 0 from above,
more after each pass through the loop. A union can grow by a small step
per pass, 'n++' by one, and the passes do not end. Widening ends them: the
bound that moved is put far out at once, and the jumps in the loop make
the range small again. How far out decides about this prog, because the
compiler tests len in 32 bits:

  len at the head  after 'if w7 > 0xffe'    path[len++] = '/'
  [0, S64_MAX]     [0, 0x7fffffff00000ffe]  R2 unbounded memory access
  [0, U32_MAX]     [0, 4094]                ok

A 32-bit compare cannot bring back what widening to 64 bits threw away.
Thus bpf_reg_widen_range() moves a bound to the nearest of a few stops:

  lower bound:  S64_MIN <- S32_MIN <- 0
  upper bound:                        0 -> S32_MAX -> U32_MAX -> S64_MAX

The price is one more pass through the loop per stop. rusty_dispatch of
rust-bpf has '(1..doms.len()).map(..).filter(..).any(..)', which is
'r9 += 1; if r9 < r8 goto' with r8 from memory. No jump holds r9 back:
[2, S32_MAX], [2, U32_MAX], [2, S64_MAX], unknown: 4 passes. Scalar
evolution knows that each iteration adds a constant to r9: @linear. Such
a number goes on until it wraps. It becomes an unknown scalar at once, or
[0, U32_MAX] if it is computed in 32 bits: 1 pass, 914 -> 536 insns. The
225 progs of the schedulers of rust-bpf take 13 % more insns without it.

bpf_reg_union_range() is the union. It differs from bpf_reg_union() in
that it has no reg_bounds_sync(): each part of the result must include
that part of the two inputs, and an input is not always in sync.
bpf_reg_forget_range() drops all that is known about a number.
The join of two registers is the caller. Nothing calls them in this patch.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 11 ++++++
 kernel/bpf/verifier.c        | 74 ++++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 2f935061b8fd..79faf084cbcd 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1341,6 +1341,7 @@ struct bpf_verifier_env {
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
 	struct scev *scev;
+	struct bpf_join_state *join;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1531,6 +1532,13 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+
+/* join.c */
+int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
+		    const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
+		    bool widen);
+void bpf_join_free(struct bpf_verifier_env *env);
+
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
@@ -2059,6 +2067,9 @@ void bpf_trim_pkt_range(struct bpf_reg_state *reg, s64 smax);
 int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 		      struct cnum64 r64, struct cnum32 r32, u16 base, u16 step);
 bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
+void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src);
+void bpf_reg_forget_range(struct bpf_reg_state *reg);
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear);
 int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
 		  const struct bpf_reg_state *src);
 
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8f61613fb5a0..754c64d7c9cb 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -18180,6 +18180,80 @@ bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_re
 	       a->id == b->id && a->parent_id == b->parent_id && a->map_uid == b->map_uid;
 }
 
+/* Nothing is known about the number or the variable part of the offset anymore. */
+void bpf_reg_forget_range(struct bpf_reg_state *reg)
+{
+	reg->var_off = tnum_unknown;
+	__mark_reg_unbounded(reg);
+	reg_step_reset(reg);
+}
+
+/*
+ * @reg is what @old became after a trip around a cycle, @old included. The end
+ * of the range that moved is going to move again. Put it where it may come
+ * to rest. There are few such places, so this is done a few times at most.
+ * It's put at the nearest of these places: every value between the range and the place is
+ * made up, and the program is checked with it. Wherever it's put, the range only gets larger.
+ * @linear: every iteration adds a constant to it, in that many bits. Then nothing but a jump
+ * stops it. It goes on until it wraps, and is any number of that many bits.
+ */
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear)
+{
+	bool moved = reg->r64.base != old->r64.base || reg->r64.size != old->r64.size ||
+		     reg->r32.base != old->r32.base || reg->r32.size != old->r32.size;
+	s64 min = reg_smin(reg), max = reg_smax(reg);
+
+	/* Only the step is gone, see bpf_reg_union_range() */
+	if (!moved && reg->var_off.value == old->var_off.value &&
+	    reg->var_off.mask == old->var_off.mask)
+		return;
+
+	if (linear && (max > reg_smax(old) || min < reg_smin(old))) {
+		if (linear == 64 || min < 0 || max > U32_MAX) {
+			bpf_reg_forget_range(reg);
+			return;
+		}
+		min = 0;
+		max = U32_MAX;
+	}
+	if (max > reg_smax(old))
+		max = max <= 0 ? 0 :
+		      max <= S32_MAX ? S32_MAX :
+		      max <= U32_MAX ? U32_MAX : S64_MAX;
+	if (min < reg_smin(old))
+		min = min >= 0 ? 0 :
+		      min >= S32_MIN ? S32_MIN : S64_MIN;
+	if (min == S64_MIN && max == S64_MAX) {
+		bpf_reg_forget_range(reg);
+		return;
+	}
+	/* every part has to include what it did before, hence no reg_bounds_sync() */
+	reg->r64 = cnum64_union(reg->r64, cnum64_from_srange(min, max));
+	if (min >= 0 && max <= U32_MAX) {
+		reg->r32 = cnum32_union(reg->r32, cnum32_from_urange(min, max));
+		reg->var_off = tnum_union(reg->var_off, tnum_range(min, max));
+	} else if (min >= S32_MIN && max <= S32_MAX) {
+		reg->r32 = cnum32_union(reg->r32, cnum32_from_srange(min, max));
+		reg->var_off = tnum_unknown;
+	} else {
+		reg->r32 = CNUM32_UNBOUNDED;
+		reg->var_off = min >= 0 ? tnum_union(reg->var_off, tnum_range(min, max))
+					: tnum_unknown;
+	}
+}
+
+void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src)
+{
+	acc->r64 = cnum64_union(acc->r64, src->r64);
+	acc->r32 = cnum32_union(acc->r32, src->r32);
+	acc->var_off = tnum_union(acc->var_off, src->var_off);
+	reg_step_reset(acc);
+	/*
+	 * No reg_bounds_sync(): every part of the result has to include that
+	 * part of both inputs, and an input is not always in sync.
+	 */
+}
+
 /*
  * Extends acc's scalar range to include src's scalar range.
  * Caller must ensure that other kinds of metadata is compatible.
-- 
2.55.0

[PATCH bpf-next v3 04/45] bpf: Add join of registers and stack slots

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

ipv6_hdrlen_offset() of Cilium skips extension headers, unrolled:

  for (i = 0; i < IPV6_MAX_HEADERS; i++) {
          int hdrlen = ipv6_skip_exthdr(ctx, &newnh, l3_off + len);
          ...  8, (opthdr->hdrlen + 2) << 2 or (opthdr->hdrlen + 1) << 3
          len += hdrlen;
  }

The verifier follows each path. Three lengths for each header: the insn
behind this loop is verified 76 times, and each time the rest of the prog
follows, where more paths fork. tail_nodeport_nat_ingress_ipv6 of bpf_xdp,
as of May 2026, has 4095 insns, the verifier processes 182050.

Add the join of two states: one state that covers the two. Here hdrlen is
[8, 2048]. A caller that replaces two states with their join verifies the
code that follows one time. This prog: 3 times in place of 76, 7932 insns.
This is the first half of join.c: ids, registers, stack slots.

Only numbers are joined: scalars, and the variable offsets of pointers. A
join loses information. Where the prog needs it there is no join, and the
caller keeps the two states:
 - map value + 8 and map value + 16: two fields, not a range
   scx_rusty: create_save_cpumask(&kptr) for three kptrs in .data
   "R1 doesn't have constant offset. kptr has to be at the constant offset"
 - packet pointers that are checked up to different ends
   Cilium: l4_csum_replace(ctx, l4_off + csum->offset, ..)
   "invalid access to packet": 206 progs of bpf_xdp in 37 builds of Cilium
 - a scalar and a pointer, pointers to two objects: of no use to a prog

Each path makes its own ids. An id in the join stands for a pair: the id
in state a and the id in state b. The pairs must be one to one, as in
check_ids(). If not, a scalar loses its id, and a pointer has no join.
 - 'bpf_for(i, 0, 10) sum += i;' of iters_num: each bpf_iter_num_next()
   returns a pointer with a new id. If ids had to be equal, the main pass
   says "more than 1024 states that cannot be joined at insn 16".
 - scx_pandemonium: 'if (.. && (u32)src_cpu < MAX_CPUS) pair_warm_ns[..]'
   tests r5 and indexes with its copy in fp-88. Paths meet between copy
   and test. A join without ids: "R3 unbounded memory access".

With 'widen', at a loop head, a scalar that grew is widened. If each
iteration adds a constant to it, it becomes an unknown scalar of 64 or 32
bits at once: scev_compute_linear_regs() sets linear_regs, linear32_regs of
struct bpf_loop for that. A pointer whose offset grew has no join: no later
comparison gives it a bound.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   3 +
 kernel/bpf/join.c            | 400 +++++++++++++++++++++++++++++++++++
 kernel/bpf/scev.c            |  17 ++
 3 files changed, 420 insertions(+)
 create mode 100644 kernel/bpf/join.c

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 79faf084cbcd..edddb25c4c12 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -760,6 +760,9 @@ struct bpf_loop {
 	 * loops.
 	 */
 	unsigned long store_base_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/* what gets a constant added in every iteration, in 64 and in 32 bits */
+	unsigned long linear_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	unsigned long linear32_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 };
 
 struct bpf_insn_aux_data {
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
new file mode 100644
index 000000000000..fe513b7a8e88
--- /dev/null
+++ b/kernel/bpf/join.c
@@ -0,0 +1,400 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Join of verifier states.
+ *
+ * bpf_states_join(dst, a, b) computes the state that describes every concrete
+ * state described by 'a' or by 'b'. With @widen a number that is larger in the
+ * result than in 'a' gets the bound that moved put further out, so that a sequence
+ * of states at a loop head ends. A pointer whose offset is larger has no join.
+ *
+ * Numbers and offsets of pointers are joined. A pointer and a number, pointers
+ * to different objects, different references and locks are not: there is no
+ * join and the states stay apart.
+ *
+ * The contract is checked on every call: both inputs have to be within the
+ * result, see bpf_state_subsumes(). A violation is a verifier bug.
+ *
+ * Inputs have to be at the same insn and cleaned with
+ * bpf_clean_verifier_state(): dead registers and stack slots are the same on
+ * both sides then, but for dynptrs, iterators and irq flags: they stay.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/cnum.h>
+#include <linux/tnum.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+
+struct join_idpair {
+	u32 a;
+	u32 b;
+	bool bad;
+};
+
+struct bpf_join_state {
+	struct bpf_verifier_env *env;
+	struct join_idpair pairs[BPF_ID_MAP_SIZE];
+	u32 npairs;
+	bool pairs_overflow;
+	bool widen;
+	/* the states came around this loop in the frame that is joined, NULL if not */
+	const struct bpf_loop *loop;
+	/* the register or stack slot that is joined, as scalar evolution numbers it, or -1 */
+	int place;
+};
+
+static struct bpf_join_state *join_state(struct bpf_verifier_env *env)
+{
+	if (!env->join) {
+		env->join = kvzalloc_obj(*env->join, GFP_KERNEL_ACCOUNT);
+		if (env->join)
+			env->join->env = env;
+	}
+	return env->join;
+}
+
+void bpf_join_free(struct bpf_verifier_env *env)
+{
+	kvfree(env->join);
+	env->join = NULL;
+}
+
+/*
+ * An id in the result stands for the pair (id in 'a', id in 'b'). For the
+ * result to be within both inputs the pairs have to form a bijection,
+ * see check_ids(). The first pass over all locations records the pairs. An id
+ * seen with two partners makes all pairs it takes part in bad.
+ * The second pass keeps the id of 'a' for good pairs.
+ */
+static void jid_mark_bad(struct bpf_join_state *js, u32 a, u32 b)
+{
+	u32 i;
+
+	for (i = 0; i < js->npairs; i++)
+		if ((a && js->pairs[i].a == a) || (b && js->pairs[i].b == b))
+			js->pairs[i].bad = true;
+}
+
+static void jid_note(struct bpf_join_state *js, u32 ida, u32 idb, bool consistent)
+{
+	u32 a = ida & ~BPF_ADD_CONST, b = idb & ~BPF_ADD_CONST;
+	struct join_idpair *p;
+	bool other = false;
+	u32 i;
+
+	if (!a || !b)
+		return;
+	for (i = 0; i < js->npairs; i++) {
+		p = &js->pairs[i];
+		if (p->a == a && p->b == b) {
+			if (!consistent)
+				jid_mark_bad(js, a, b);
+			return;
+		}
+		if (p->a == a || p->b == b)
+			other = true;
+	}
+	if (other) {
+		jid_mark_bad(js, a, b);
+		consistent = false;
+	}
+	if (js->npairs == ARRAY_SIZE(js->pairs)) {
+		js->pairs_overflow = true;
+		return;
+	}
+	p = &js->pairs[js->npairs++];
+	p->a = a;
+	p->b = b;
+	p->bad = !consistent;
+}
+
+static bool jid_good(struct bpf_join_state *js, u32 ida, u32 idb)
+{
+	u32 a = ida & ~BPF_ADD_CONST, b = idb & ~BPF_ADD_CONST;
+	u32 i;
+
+	if (!a || !b || js->pairs_overflow)
+		return false;
+	if ((ida ^ idb) & BPF_ADD_CONST)
+		return false;
+	for (i = 0; i < js->npairs; i++)
+		if (js->pairs[i].a == a && js->pairs[i].b == b)
+			return !js->pairs[i].bad;
+	return false;
+}
+
+/* Ids that have to match: both zero, or a good pair */
+static bool jid_strict_ok(struct bpf_join_state *js, u32 ida, u32 idb)
+{
+	if (!ida && !idb)
+		return true;
+	return jid_good(js, ida, idb);
+}
+
+static bool range_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	return ra->r64.base == rb->r64.base && ra->r64.size == rb->r64.size &&
+	       ra->r32.base == rb->r32.base && ra->r32.size == rb->r32.size &&
+	       ra->var_off.value == rb->var_off.value && ra->var_off.mask == rb->var_off.mask &&
+	       ra->base == rb->base && ra->step == rb->step;
+}
+
+/* Does every iteration add a constant to what is in the place? In 64 or in 32 bits, 0 if not */
+static int join_place_is_linear(const struct bpf_join_state *js)
+{
+	if (!js->loop || js->place < 0 || js->place >= BPF_SCEV_REGS_NUM)
+		return 0;
+	return test_bit(js->place, js->loop->linear_regs) ? 64 :
+	       test_bit(js->place, js->loop->linear32_regs) ? 32 : 0;
+}
+
+/*
+ * Join of the range of two registers of the same type, numbers or pointers
+ * with variable offset. @dst is a copy of @ra. Returns false if there is no join.
+ */
+static bool join_range(struct bpf_join_state *js, struct bpf_reg_state *dst,
+		       const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	if (range_equal(ra, rb))
+		return true;
+	bpf_reg_union_range(dst, rb);
+	if (!js->widen || range_equal(dst, ra))
+		return true;
+	/*
+	 * The bound of a number that grows around a cycle is put where it may come to rest. If
+	 * that is far out the number gets a bound again from the next comparison. The offset of
+	 * a pointer doesn't: there is no access through a pointer at an offset that is not
+	 * known. So there is nothing to gain from that join. If the offset stops to grow
+	 * the next state is within the last one.
+	 */
+	if (dst->type != SCALAR_VALUE)
+		return false;
+	bpf_reg_widen_range(dst, ra, join_place_is_linear(js));
+	return true;
+}
+
+static bool reg_fixed_part_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	return !memcmp(ra, rb, offsetof(struct bpf_reg_state, var_off));
+}
+
+static bool reg_value_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	return !memcmp(ra, rb, offsetof(struct bpf_reg_state, id));
+}
+
+static bool scalar_link_consistent(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	if ((ra->id ^ rb->id) & BPF_ADD_CONST)
+		return false;
+	if ((ra->id & BPF_ADD_CONST) && ra->delta != rb->delta)
+		return false;
+	return true;
+}
+
+/*
+ * Join two registers, regsafe() documents what each type needs.
+ * Pass 1 records id pairs, pass 2 computes @dst, which is a copy of @ra on entry.
+ * Returns false if there is no join.
+ */
+static bool join_reg(struct bpf_join_state *js, struct bpf_reg_state *dst,
+		     const struct bpf_reg_state *ra, const struct bpf_reg_state *rb, int pass)
+{
+	if (ra->type == NOT_INIT || rb->type == NOT_INIT) {
+		if (pass == 2 && ra->type != NOT_INIT)
+			*dst = *rb;
+		return true;
+	}
+
+	if (ra->type != rb->type)
+		return false;
+
+	switch (base_type(ra->type)) {
+	case SCALAR_VALUE:
+		if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
+			return false;
+		if (pass == 1) {
+			jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
+			return true;
+		}
+		join_range(js, dst, ra, rb);
+		/* a link that exists on one side only is dropped */
+		if (ra->id && !jid_good(js, ra->id, rb->id)) {
+			dst->id = 0;
+			dst->delta = 0;
+		}
+		return true;
+	case PTR_TO_MAP_KEY:
+	case PTR_TO_MAP_VALUE:
+	case PTR_TO_MEM:
+	case PTR_TO_BUF:
+	case PTR_TO_TP_BUFFER:
+		if (!reg_fixed_part_equal(ra, rb))
+			return false;
+		/*
+		 * Two places in an object are two pointers: what is at a place
+		 * and how much is left behind it is lost in a range of them.
+		 */
+		if (!js->widen && tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+		    ra->var_off.value != rb->var_off.value)
+			return false;
+		break;
+	case PTR_TO_PACKET_META:
+	case PTR_TO_PACKET:
+		if ((ra->range < 0 || rb->range < 0) && ra->range != rb->range)
+			return false;
+		/* how much of the packet was checked goes with numbers elsewhere in the state */
+		if (!js->widen && ra->range != rb->range)
+			return false;
+		/*
+		 * Displacements between pointers that share an id are kept. No id is an id too:
+		 * the start of the packet plus a constant, see find_good_pkt_pointers().
+		 */
+		if (ra->r64.base != rb->r64.base)
+			return false;
+		if (pass == 1) {
+			jid_note(js, ra->id, rb->id, true);
+			return true;
+		}
+		if (!jid_strict_ok(js, ra->id, rb->id))
+			return false;
+		if (rb->range != ra->range)
+			dst->range = min(ra->range, rb->range);
+		return join_range(js, dst, ra, rb);
+	case PTR_TO_ARENA:
+		/* The offset is not tracked. */
+		return true;
+	case PTR_TO_INSN:
+		if (!reg_fixed_part_equal(ra, rb))
+			return false;
+		return pass == 1 || join_range(js, dst, ra, rb);
+	default:
+		if (!reg_value_equal(ra, rb))
+			return false;
+		break;
+	}
+
+	if (pass == 1) {
+		jid_note(js, ra->id, rb->id, true);
+		jid_note(js, ra->parent_id, rb->parent_id, true);
+		jid_note(js, ra->map_uid, rb->map_uid, true);
+		return true;
+	}
+	if (!jid_strict_ok(js, ra->id, rb->id) ||
+	    !jid_strict_ok(js, ra->parent_id, rb->parent_id) ||
+	    !jid_strict_ok(js, ra->map_uid, rb->map_uid))
+		return false;
+	return join_range(js, dst, ra, rb);
+}
+
+static bool slot_is_special(const struct bpf_stack_state *s)
+{
+	u8 t = s->slot_type[BPF_REG_SIZE - 1];
+
+	return t == STACK_DYNPTR || t == STACK_ITER || t == STACK_IRQ_FLAG;
+}
+
+static bool stype_is_invalid(u8 t)
+{
+	return t == STACK_INVALID || t == STACK_POISON;
+}
+
+static bool slot_has_spill(const struct bpf_stack_state *s)
+{
+	int i;
+
+	for (i = 0; i < BPF_REG_SIZE; i++)
+		if (s->slot_type[i] == STACK_SPILL)
+			return true;
+	return false;
+}
+
+static bool join_special_slot(struct bpf_join_state *js, const struct bpf_stack_state *sa,
+			      const struct bpf_stack_state *sb, int pass)
+{
+	const struct bpf_reg_state *ra = &sa->spilled_ptr, *rb = &sb->spilled_ptr;
+
+	if (memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type)))
+		return false;
+	switch (sa->slot_type[BPF_REG_SIZE - 1]) {
+	case STACK_DYNPTR:
+		if (ra->dynptr.type != rb->dynptr.type ||
+		    ra->dynptr.first_slot != rb->dynptr.first_slot)
+			return false;
+		break;
+	case STACK_ITER:
+		if (ra->type != rb->type || ra->iter.btf != rb->iter.btf ||
+		    ra->iter.btf_id != rb->iter.btf_id || ra->iter.state != rb->iter.state ||
+		    ra->iter.loops != rb->iter.loops)
+			return false;
+		break;
+	case STACK_IRQ_FLAG:
+		if (ra->irq.kfunc_class != rb->irq.kfunc_class)
+			return false;
+		break;
+	}
+	if (pass == 1) {
+		jid_note(js, ra->id, rb->id, true);
+		jid_note(js, ra->parent_id, rb->parent_id, true);
+		return true;
+	}
+	return jid_strict_ok(js, ra->id, rb->id) &&
+	       jid_strict_ok(js, ra->parent_id, rb->parent_id);
+}
+
+/*
+ * Join of one 8 byte stack slot, @ds is a copy of @sa on entry.
+ * stacksafe() documents which slot is within which.
+ */
+static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
+		      const struct bpf_stack_state *sa, const struct bpf_stack_state *sb, int pass)
+{
+	static const struct bpf_stack_state invalid_slot;
+	bool a_ptr, b_ptr;
+	u8 res[BPF_REG_SIZE];
+	int i;
+
+	if (!sb)
+		sb = &invalid_slot;
+
+	if (slot_is_special(sa) || slot_is_special(sb))
+		return join_special_slot(js, sa, sb, pass);
+
+	if (!memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type)) &&
+	    sa->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL)
+		return join_reg(js, &ds->spilled_ptr, &sa->spilled_ptr, &sb->spilled_ptr, pass);
+
+	a_ptr = slot_has_spill(sa) && sa->spilled_ptr.type != SCALAR_VALUE;
+	b_ptr = slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE;
+	/* a pointer and something else */
+	if (a_ptr || b_ptr)
+		return false;
+
+	/* the prog can't read uninitialized stack: slots of different kinds have no join */
+	if (!js->env->allow_uninit_stack)
+		return !memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type));
+
+	/* scalar spills, MISC, ZERO, INVALID, POISON: no ids are kept */
+	if (pass == 1)
+		return true;
+
+	for (i = 0; i < BPF_REG_SIZE; i++) {
+		u8 x = sa->slot_type[i], y = sb->slot_type[i];
+
+		if (x == STACK_SPILL)
+			x = STACK_MISC;
+		if (y == STACK_SPILL)
+			y = STACK_MISC;
+		if (stype_is_invalid(x) && stype_is_invalid(y))
+			res[i] = sa->slot_type[i];
+		else if (stype_is_invalid(x) || stype_is_invalid(y))
+			res[i] = STACK_INVALID;
+		else if (x == y)
+			res[i] = x;
+		else
+			res[i] = STACK_MISC;
+	}
+	memcpy(ds->slot_type, res, sizeof(res));
+	bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
+	return true;
+}
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index bb1c67176d78..819b563a5b54 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1676,6 +1676,20 @@ static int compute_header_scevs(struct bpf_verifier_env *env, struct env *header
 	}
 	return 0;
 }
+
+/* Find the registers and stack slots that get a constant added in every iteration of the loop */
+static void scev_compute_linear_regs(struct scev *scev, struct bpf_loop *loop,
+				     struct env *header_env)
+{
+	u32 r, same, base, slope;
+	bool zext32;
+
+	for (r = 0; r < REGS_NUM; r++)
+		if (is_linear_or_linear32(scev, header_env->reg2scev[r], &base, &slope, &zext32) &&
+		    is_reg(scev, base, &same) && same == r)
+			__set_bit(r, zext32 ? loop->linear32_regs : loop->linear_regs);
+}
+
 static int instantiate_header_scevs(struct scev *scev, u32 id, void *priv)
 {
 	u32 reg, *reg2scev = priv;
@@ -1784,6 +1798,9 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
 					   find_loop_env(scev, header, idx));
 		if (err)
 			return err;
+		if (aux[idx].loop)
+			scev_compute_linear_regs(scev, aux[idx].loop,
+						 find_header_env(scev, header));
 	}
 
 	for (i = 0; i < env->prog->len; i++)
-- 
2.55.0

Re: [PATCH bpf-next v3 04/45] bpf: Add join of registers and stack slots

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: join_reg() zeroes dst->delta under explore_alu_limits, failing regsafe()
--

commit 909a8651c00556d259164c0f04d677d2fd83cfc0
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Add join of registers and stack slots

This commit adds the first half of verifier state joining in
kernel/bpf/join.c to merge scalar registers, pointer variable offsets, IDs,
and stack slots across execution paths, along with linear register tracking
in kernel/bpf/scev.c for loop widening.

> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> new file mode 100644
> index 0000000000000..fe513b7a8e884
> --- /dev/null
> +++ b/kernel/bpf/join.c

[ ... ]

> +static bool join_reg(struct bpf_join_state *js, struct bpf_reg_state *dst,
> +		     const struct bpf_reg_state *ra, const struct bpf_reg_state *rb, int pass)
> +{
> +	if (ra->type == NOT_INIT || rb->type == NOT_INIT) {
> +		if (pass == 2 && ra->type != NOT_INIT)
> +			*dst = *rb;
> +		return true;
> +	}
> +
> +	if (ra->type != rb->type)
> +		return false;
> +
> +	switch (base_type(ra->type)) {
> +	case SCALAR_VALUE:
> +		if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
> +			return false;
> +		if (pass == 1) {
> +			jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
> +			return true;
> +		}
> +		join_range(js, dst, ra, rb);
> +		/* a link that exists on one side only is dropped */
> +		if (ra->id && !jid_good(js, ra->id, rb->id)) {
> +			dst->id = 0;
> +			dst->delta = 0;
> +		}
> +		return true;

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], in join_reg() in
kernel/bpf/join.c, can dropping an unmatched or inconsistent scalar ID link
under js->env->explore_alu_limits cause the resulting state to fail the
subsumption check in regsafe()?

When js->env->explore_alu_limits is true and two SCALAR_VALUE registers ra
and rb have identical values prior to id (so reg_value_equal(ra, rb) is true)
with a non-zero constant delta (ra->delta == rb->delta != 0), pass 2 of
join_reg() zeroes dst->id and dst->delta if jid_good(js, ra->id, rb->id)
returns false (for example, when one side has BPF_ADD_CONST64 and the other
has BPF_ADD_CONST32, or when the base ID is paired with a different partner
in another register).

However, in struct bpf_reg_state in include/linux/bpf_verifier.h, the delta
field is located before id:

struct bpf_reg_state {
	enum bpf_reg_type type;
	s32 delta;
	...
	u32 id;
};

When bpf_state_subsumes() checks the joined state via states_equal() ->
regsafe() in kernel/bpf/states.c, the explore_alu_limits path compares all
fields prior to id with memcmp():

	case SCALAR_VALUE:
		if (env->explore_alu_limits) {
			return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
			       check_scalar_ids(rold->id, rcur->id, idmap);
		}

Because dst->delta was cleared to 0 while ra->delta and rb->delta are
non-zero, memcmp() returns non-zero and regsafe() returns false, triggering a
verifier bug warning and failing verification with -EFAULT.

When js->env->explore_alu_limits is set and ra->delta != 0, should join_reg()
refuse to join the scalars if jid_good(js, ra->id, rb->id) is false?

[1] https://lore.kernel.org/r/ee77c73bff0b56a40098080a5335d5fd5e4d8592ac15afe769c1c4661f026576@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=4

[PATCH bpf-next v3 05/45] bpf: Add join of verifier states

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

central_dispatch() of tools/sched_ext/scx_central.bpf.c:

  bpf_for(cpu, 0, nr_cpu_ids) {
          ...
          if (dispatch_to_cpu(cpu))
                  *gimme = false;
  }
  ...
  dispatch_to_cpu(central_cpu);

To verify dispatch_to_cpu() one time, not for each call, the states of the
two calls must meet at its first insn. Numbers, like cpu, can be joined.
But at one call the iterator of bpf_for() is held, at the other nothing.
One state cannot stand for the two: the code after the return destroys the
iterator for one of them only. A state is more than registers and slots.

Add the second half of join.c and build it. bpf_states_join() joins two
states at one insn, frame by frame. There is no join, and the caller keeps
the two states, when one of these differs:

  frames                number, subprog, callsite, kind of callback
  locks, RCU, preempt   the counts, the lock
  references            number and types. The ids must pair one to one
  held since the entry  entry_refs, entry_rcu_locks, entry_preempt_locks
  loop stack            the loops, the state that entered them, the count,
                        the iteration if iterations are kept apart
  flags                 speculative, sleepable, unwound, ...

 - A register, its spill and the reference have one id. Thus the pairs of
   ids are collected over all frames and references first, and the join
   is made in a second pass.
 - Only live registers count, and the caller removes dead stack slots with
   bpf_clean_verifier_state(). What is dead does not prevent a join.
 - A subprog that is verified without its callers has what they hold in
   its state, as above. entry_* tell how much of it is still held.
 - Such a subprog has its incoming stack arguments in the slots for
   outgoing ones, until it has loaded them. The two kinds are not joined.
   quicksort(right, scratch, limit, pivot_ref, is_less) of Rust's core is
   recursive and takes 7 words: bpfland_init of rust-bpf fails without it.

bpf_states_join() examines each result with the comparison that pruning
uses: the join must cover state a and state b. If it does not, that is a
verifier bug and the prog is rejected, not loaded. The fields that join.c
compares come with this patch. Nothing sets them, nothing calls the join.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  65 ++++++++-
 kernel/bpf/Makefile          |   2 +-
 kernel/bpf/join.c            | 247 +++++++++++++++++++++++++++++++++++
 kernel/bpf/states.c          |  12 ++
 4 files changed, 324 insertions(+), 2 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index edddb25c4c12..1b08ccd5d766 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -120,6 +120,11 @@ struct bpf_reg_state {
 			/* packing following two fields to fit iter state into 16 bytes */
 			enum bpf_iter_state state:2;
 			int depth:30;
+			/*
+			 * The number of loops, in all frames, that the program was in
+			 * when it made the iterator, see iter_stops_loops().
+			 */
+			u32 loops:30;
 		} iter;
 
 		/* For irq stack slots */
@@ -363,12 +368,35 @@ struct loop_stack_entry {
 	struct bpf_verifier_state *entry_state;
 	u32 min_header_count;
 	u32 max_header_count;
-	u32 loop_id:31;
+	u32 loop_id:23;
 	u32 terminates:1;
+	/* the loop had may_goto when the state entered it, see check_loop_guard() */
+	u32 guarded:1;
+	/*
+	 * The iterations are kept apart: states with different @trip are not
+	 * joined for as long as they are in the loop.
+	 */
+	u32 apart:1;
+	/* max_header_count follows from the state at the entry, not from patience */
+	u32 counted:1;
+	/*
+	 * How many times the state came back to the head of the loop. With @apart that is
+	 * the iteration that it is in. A state that stands for many iterations is a join, and has
+	 * the largest number of the states that it is made of.
+	 */
+	u32 trip;
+	/* the number of references that the state had when it entered the loop */
+	u32 refs;
 	int iv_reg;
 	struct cnum64 iv_range;
 };
 
+/* The state is in a walk on patience, which may not get to the end of the loop */
+static inline bool bpf_loop_walk_may_give_up(const struct loop_stack_entry *entry)
+{
+	return entry->apart && entry->terminates && !entry->counted;
+}
+
 #define LOOP_STACK_SIZE 16
 
 /* state of the program:
@@ -426,6 +454,14 @@ struct bpf_func_state {
 	 * subprogram (loops never cross subprogram boundaries).
 	 */
 	u32 loop_stack_cnt;
+	/*
+	 * How much of what was held when the frame was entered is held ever since. For
+	 * references that is the first entry_refs of refs[]. What is released and taken again
+	 * doesn't count. More than that is for this frame to release.
+	 */
+	u32 entry_refs;
+	u32 entry_rcu_locks;
+	u32 entry_preempt_locks;
 
 	/* The following fields should be last. See copy_func_state() */
 	/* The state of the stack. Each element of the array describes BPF_REG_SIZE
@@ -558,8 +594,15 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	/* a global function or an extension, and what it calls: exit of frame 0 is not the end */
+	bool callers_unknown;
 	/* The callee left with unwind_exit. Next insn is unwind_goto. */
 	bool unwound;
+	/*
+	 * A function that is verified on its own changed packet data. Its callers
+	 * and their packet pointers are not in the state, see bpf_flow_return().
+	 */
+	bool changed_pkt_data;
 
 	/*
 	 * First and last insn idx of this verifier state.
@@ -620,6 +663,16 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
 	return NULL;
 }
 
+/*
+ * The function that is verified on its own has no caller to look at. Its incoming arguments
+ * are in stack_arg_regs of frame 0 until it is done with loading them. Until then it has
+ * no outgoing arguments.
+ */
+static inline bool bpf_stack_args_are_incoming(const struct bpf_func_state *frame)
+{
+	return !frame->frameno && !frame->no_stack_arg_load;
+}
+
 /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
 #define bpf_for_each_spilled_reg(iter, frame, reg, mask)			\
 	for (iter = 0, reg = bpf_get_spilled_reg(iter, frame, mask);		\
@@ -754,6 +807,8 @@ struct bpf_loop {
 	bool exits_overflow;
 	/* @exits of a loop in this one is cut. Then @exits of this loop can lack exits too. */
 	bool nested_exits_overflow;
+	/* and it is going to be there, see bpf_flow_loop_gets_guards() */
+	bool guarded;
 	/*
 	 * Loop-entry registers used to compute stack addresses for loads,
 	 * stores and calls requiring fixed stack offsets, in this and nested
@@ -885,6 +940,11 @@ struct bpf_insn_aux_data {
 	 * For a loop header, identifies the parent loop instead.
 	 */
 	s32 loop_header;
+	/*
+	 * bpf_iter_*_next(): the innermost loop around it that counted on it and that it doesn't
+	 * stop, -1 if none was found. It doesn't stop the loops around that one either.
+	 */
+	s32 not_stopped;
 	/* where may_goto in front of this instruction goes */
 	s32 guard_to;
 	/* additional information about the loop if this instruction is a loop header */
@@ -1535,6 +1595,9 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+			struct bpf_verifier_state *cur);
 
 /* join.c */
 int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 269810b128c4..3e0b096cfc9f 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index fe513b7a8e88..4da051767e1d 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -398,3 +398,250 @@ static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
 	bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
 	return true;
 }
+
+static bool join_loop_stack(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
+{
+	u32 i;
+
+	if (fa->loop_stack_cnt != fb->loop_stack_cnt)
+		return false;
+	/* what is known about a counted loop comes from the state it was entered with */
+	for (i = 0; i < fa->loop_stack_cnt; i++)
+		if (fa->loop_stack[i].loop_id != fb->loop_stack[i].loop_id ||
+		    fa->loop_stack[i].terminates != fb->loop_stack[i].terminates ||
+		    fa->loop_stack[i].guarded != fb->loop_stack[i].guarded ||
+		    fa->loop_stack[i].counted != fb->loop_stack[i].counted ||
+		    fa->loop_stack[i].apart != fb->loop_stack[i].apart ||
+		    (fa->loop_stack[i].apart && fa->loop_stack[i].trip != fb->loop_stack[i].trip) ||
+		    fa->loop_stack[i].refs != fb->loop_stack[i].refs ||
+		    fa->loop_stack[i].max_header_count != fb->loop_stack[i].max_header_count ||
+		    fa->loop_stack[i].entry_state != fb->loop_stack[i].entry_state)
+			return false;
+	return true;
+}
+
+static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
+		      const struct bpf_func_state *fa, const struct bpf_func_state *fb,
+		      u16 live_regs, int pass)
+{
+	static const struct bpf_reg_state not_init = { .type = NOT_INIT };
+	struct bpf_verifier_env *env = js->env;
+	u32 i, nslots_b;
+	bool in_a, in_b;
+
+	if (pass == 1) {
+		if (fa->callsite != fb->callsite || fa->frameno != fb->frameno ||
+		    fa->subprogno != fb->subprogno)
+			return false;
+		if (fa->in_callback_fn != fb->in_callback_fn ||
+		    fa->in_async_callback_fn != fb->in_async_callback_fn ||
+		    fa->in_exception_callback_fn != fb->in_exception_callback_fn ||
+		    fa->in_rcu_callback_fn != fb->in_rcu_callback_fn ||
+		    fa->async_entry_cnt != fb->async_entry_cnt ||
+		    memcmp(&fa->callback_ret_range, &fb->callback_ret_range,
+			   sizeof(fa->callback_ret_range)))
+			return false;
+		if (fa->entry_refs != fb->entry_refs ||
+		    fa->entry_rcu_locks != fb->entry_rcu_locks ||
+		    fa->entry_preempt_locks != fb->entry_preempt_locks)
+			return false;
+		if (!join_loop_stack(fa, fb))
+			return false;
+	}
+
+	for (i = 0; i < MAX_BPF_REG; i++) {
+		if (!(live_regs & BIT(i))) {
+			if (pass == 2 && i < BPF_REG_FP)
+				bpf_mark_reg_not_init(env, &fd->regs[i]);
+			continue;
+		}
+		js->place = i;
+		if (!join_reg(js, &fd->regs[i], &fa->regs[i], &fb->regs[i], pass))
+			return false;
+	}
+
+	nslots_b = bpf_stack_nr_slots(fb);
+	for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
+		const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
+
+		js->place = MAX_BPF_REG + i;
+		if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
+			return false;
+	}
+	js->place = -1;
+	/* Slots that exist only in 'b' are STACK_INVALID in the result. */
+	for (i = bpf_stack_nr_slots(fa); pass == 1 && i < nslots_b; i++) {
+		const struct bpf_stack_state *sb = bpf_stack_slot(fb, i);
+
+		if (slot_is_special(sb) ||
+		    (slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE))
+			return false;
+	}
+
+	if (fa->out_stack_arg_cnt < fb->out_stack_arg_cnt)
+		return false;
+	/* incoming arguments are not joined with outgoing ones: that side has no outgoing ones */
+	in_a = bpf_stack_args_are_incoming(fa) && !bpf_stack_args_are_incoming(fb);
+	in_b = bpf_stack_args_are_incoming(fb) && !bpf_stack_args_are_incoming(fa);
+	for (i = 0; i < fa->out_stack_arg_cnt; i++) {
+		const struct bpf_reg_state *rb;
+
+		if (in_a) {
+			if (pass == 2)
+				bpf_mark_reg_not_init(env, &fd->stack_arg_regs[i]);
+			continue;
+		}
+		rb = i < fb->out_stack_arg_cnt && !in_b ? &fb->stack_arg_regs[i] : &not_init;
+		if (!join_reg(js, &fd->stack_arg_regs[i], &fa->stack_arg_regs[i], rb, pass))
+			return false;
+	}
+
+	if (pass == 2) {
+		fd->no_stack_arg_load = fa->no_stack_arg_load || fb->no_stack_arg_load;
+		fd->callback_depth = min(fa->callback_depth, fb->callback_depth);
+		for (i = 0; i < fd->loop_stack_cnt; i++)
+			fd->loop_stack[i].trip = max(fd->loop_stack[i].trip, fb->loop_stack[i].trip);
+	}
+	return true;
+}
+
+/* Things that can't be different in two states that are joined */
+static bool join_check_state(const struct bpf_verifier_state *a,
+			     const struct bpf_verifier_state *b)
+{
+	int i;
+
+	if (a->curframe != b->curframe || a->insn_idx != b->insn_idx)
+		return false;
+	if (a->speculative != b->speculative || a->in_sleepable != b->in_sleepable ||
+	    a->callers_unknown != b->callers_unknown ||
+	    a->unwound != b->unwound || a->changed_pkt_data != b->changed_pkt_data)
+		return false;
+	if (a->active_locks != b->active_locks ||
+	    a->active_preempt_locks != b->active_preempt_locks ||
+	    a->active_rcu_locks != b->active_rcu_locks ||
+	    a->active_lock_ptr != b->active_lock_ptr ||
+	    !a->active_irq_id != !b->active_irq_id ||
+	    !a->active_lock_id != !b->active_lock_id)
+		return false;
+	if (a->acquired_refs != b->acquired_refs)
+		return false;
+	for (i = 0; i < a->acquired_refs; i++) {
+		if (a->refs[i].type != b->refs[i].type)
+			return false;
+		switch (a->refs[i].type) {
+		case REF_TYPE_PTR:
+		case REF_TYPE_IRQ:
+			break;
+		default:
+			if (a->refs[i].ptr != b->refs[i].ptr)
+				return false;
+		}
+	}
+	return true;
+}
+
+static void join_refs_note(struct bpf_join_state *js, const struct bpf_verifier_state *a,
+			   const struct bpf_verifier_state *b)
+{
+	int i;
+
+	jid_note(js, a->active_irq_id, b->active_irq_id, true);
+	jid_note(js, a->active_lock_id, b->active_lock_id, true);
+	for (i = 0; i < a->acquired_refs; i++) {
+		jid_note(js, a->refs[i].id, b->refs[i].id, true);
+		if (a->refs[i].type == REF_TYPE_PTR)
+			jid_note(js, a->refs[i].parent_id, b->refs[i].parent_id, true);
+	}
+}
+
+static bool join_refs_check(struct bpf_join_state *js, const struct bpf_verifier_state *a,
+			    const struct bpf_verifier_state *b)
+{
+	int i;
+
+	if (!jid_strict_ok(js, a->active_irq_id, b->active_irq_id) ||
+	    !jid_strict_ok(js, a->active_lock_id, b->active_lock_id))
+		return false;
+	for (i = 0; i < a->acquired_refs; i++) {
+		if (!jid_strict_ok(js, a->refs[i].id, b->refs[i].id))
+			return false;
+		if (a->refs[i].type == REF_TYPE_PTR &&
+		    !jid_strict_ok(js, a->refs[i].parent_id, b->refs[i].parent_id))
+			return false;
+	}
+	return true;
+}
+
+/* The result has to describe everything that an input describes */
+static int join_check(struct bpf_join_state *js, struct bpf_verifier_state *dst,
+		      const struct bpf_verifier_state *in, const char *which)
+{
+	struct bpf_verifier_env *env = js->env;
+
+	if (bpf_state_subsumes(env, dst, (struct bpf_verifier_state *)in))
+		return 0;
+	/* check_ids() has no room left: that says nothing about the join */
+	if (env->idmap_scratch.cnt == BPF_ID_MAP_SIZE)
+		return 1;
+	bpf_log(&env->log, "join:");
+	print_verifier_state(env, dst, dst->curframe, true);
+	bpf_log(&env->log, "input %s:", which);
+	print_verifier_state(env, in, in->curframe, true);
+	verifier_bug(env, "join of states at insn %u does not subsume input '%s'",
+		     dst->insn_idx, which);
+	return -EFAULT;
+}
+
+/*
+ * Compute in @dst the state that describes everything that @a or @b describe.
+ * @dst has to be a valid state (zeroed is fine) different from @a and @b, its
+ * previous content is released. Bookkeeping fields are copied from @a.
+ *
+ * Returns 0 on success, 1 if there is no join, negative error otherwise.
+ * When 1 is returned the content of @dst is undefined, but it's a valid state.
+ */
+int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
+		    const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
+		    bool widen)
+{
+	struct bpf_join_state *js = join_state(env);
+	const struct bpf_func_state *frame;
+	int pass, fr, err;
+
+	if (!js)
+		return -ENOMEM;
+	js->widen = widen;
+	js->npairs = 0;
+	js->pairs_overflow = false;
+
+	if (!join_check_state(a, b))
+		return 1;
+	err = bpf_copy_verifier_state(dst, a);
+	if (err)
+		return err;
+
+	for (pass = 1; pass <= 2; pass++) {
+		if (pass == 1)
+			join_refs_note(js, a, b);
+		else if (!join_refs_check(js, a, b))
+			return 1;
+		for (fr = 0; fr <= a->curframe; fr++) {
+			u32 insn_idx = bpf_frame_insn_idx((struct bpf_verifier_state *)a, fr);
+			u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
+			u32 cnt;
+
+			/* not the states that get to the head through a call of the function */
+			frame = a->frame[fr];
+			cnt = frame->loop_stack_cnt;
+			js->loop = widen && fr == a->curframe && cnt &&
+				   frame->loop_stack[cnt - 1].loop_id == insn_idx ?
+				   env->insn_aux_data[insn_idx].loop : NULL;
+			if (!join_func(js, dst->frame[fr], a->frame[fr], b->frame[fr],
+				       live_regs, pass))
+				return 1;
+		}
+	}
+
+	return join_check(js, dst, a, "a") ?: join_check(js, dst, b, "b");
+}
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 50480c42bcad..4136a708a78e 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -1765,3 +1765,15 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 	 */
 	return bpf_split_cur_state(env);
 }
+
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+	return clean_verifier_state(env, st);
+}
+
+/* Everything that @cur describes is described by @old. */
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+			struct bpf_verifier_state *cur)
+{
+	return states_equal(env, old, cur, RANGE_WITHIN);
+}
-- 
2.55.0

Re: [PATCH bpf-next v3 05/45] bpf: Add join of verifier states

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] bpf: asymmetric unallocated and poisoned stack handling in join_func()
- [Medium] bpf: join_reg() zeroing dst->delta breaks explore_alu_limits in join_check()
- [Medium] bpf: asymmetric out_stack_arg_cnt check in join_func() blocks valid joins
--

commit 1d1d176d0db968688ddcdf9cb26ad2c24d9339f4
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Add join of verifier states

This commit adds the second half of kernel/bpf/join.c and enables building it.
It implements bpf_states_join() to merge two verifier states frame by frame at
a single instruction and verifies that the joined state subsumes both inputs.

> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> index fe513b7a8e884..4da051767e1d9 100644
> --- a/kernel/bpf/join.c
> +++ b/kernel/bpf/join.c
> @@ -398,3 +398,250 @@ static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
>  	bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
>  	return true;
>  }
> +

[ ... ]

> +static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
> +		      const struct bpf_func_state *fa, const struct bpf_func_state *fb,
> +		      u16 live_regs, int pass)
> +{

[ ... ]

> +	nslots_b = bpf_stack_nr_slots(fb);
> +	for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
> +		const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
> +
> +		js->place = MAX_BPF_REG + i;
> +		if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
> +			return false;
> +	}
> +	js->place = -1;
> +	/* Slots that exist only in 'b' are STACK_INVALID in the result. */
> +	for (i = bpf_stack_nr_slots(fa); pass == 1 && i < nslots_b; i++) {
> +		const struct bpf_stack_state *sb = bpf_stack_slot(fb, i);
> +
> +		if (slot_is_special(sb) ||
> +		    (slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE))
> +			return false;
> +	}

[Severity: Medium]
As reported by Sashiko on v2 [1], how does bpf_states_join() -> join_func() ->
join_slot() handle stack slots when !env->allow_uninit_stack?

First, in the loop over slots that exist only in fb (for i >=
bpf_stack_nr_slots(fa)), join_func() only rejects special slots and non-scalar
spills, without checking !js->env->allow_uninit_stack. If fb has a live
initialized slot at i >= bpf_stack_nr_slots(fa) (such as a scalar STACK_SPILL,
STACK_MISC, or STACK_ZERO) while fa has allocated_stack <= i * 8, the join
succeeds and leaves slot i unallocated in fd. If the program later reads slot
i under a condition that only holds along path b, won't
check_stack_slot_within_bounds() falsely reject a valid program with -EACCES
when !env->allow_uninit_stack?

Second, when bpf_clean_verifier_state() marks dead stack slots in fa as
STACK_POISON via __clean_func_state(), and slot i is unallocated in fb (so
join_slot() substitutes invalid_slot with STACK_INVALID), join_slot() runs:

	if (!js->env->allow_uninit_stack)
		return !memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type));

Won't memcmp() treat dead STACK_POISON bytes in sa and unallocated
STACK_INVALID bytes in sb as unequal, causing dead stack slots to prevent a
join when !env->allow_uninit_stack?

> +
> +	if (fa->out_stack_arg_cnt < fb->out_stack_arg_cnt)
> +		return false;

[Severity: Medium]
As reported by Sashiko on v2 [2], should join_func() reject the join whenever
fa->out_stack_arg_cnt < fb->out_stack_arg_cnt?

Since out_stack_arg_cnt is an allocation high-water mark grown by
grow_stack_arg_slots() in check_stack_arg_write(), it is not reset to 0 when
invalidate_outgoing_stack_args() or bpf_flow_return() marks
stack_arg_regs[0..out_stack_arg_cnt - 1] as NOT_INIT after a call returns.

If state a reaches a loop header or branch target before any call with stack
arguments (fa->out_stack_arg_cnt == 0) and state b arrives after a call with
stack arguments whose slots are now NOT_INIT (fb->out_stack_arg_cnt > 0),
won't this check prevent joining or widening the states even though slots at
i >= fa->out_stack_arg_cnt are implicitly NOT_INIT in fa and fd?

[ ... ]

> +/* The result has to describe everything that an input describes */
> +static int join_check(struct bpf_join_state *js, struct bpf_verifier_state *dst,
> +		      const struct bpf_verifier_state *in, const char *which)
> +{
> +	struct bpf_verifier_env *env = js->env;
> +
> +	if (bpf_state_subsumes(env, dst, (struct bpf_verifier_state *)in))
> +		return 0;
> +	/* check_ids() has no room left: that says nothing about the join */
> +	if (env->idmap_scratch.cnt == BPF_ID_MAP_SIZE)
> +		return 1;
> +	bpf_log(&env->log, "join:");
> +	print_verifier_state(env, dst, dst->curframe, true);
> +	bpf_log(&env->log, "input %s:", which);
> +	print_verifier_state(env, in, in->curframe, true);
> +	verifier_bug(env, "join of states at insn %u does not subsume input '%s'",
> +		     dst->insn_idx, which);
> +	return -EFAULT;
> +}

[Severity: Medium]
As reported by Sashiko on v2 [3], can bpf_states_join() trigger verifier_bug()
in join_check() when env->explore_alu_limits is true?

In join_reg() for SCALAR_VALUE:

	case SCALAR_VALUE:
		if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
			return false;
		if (pass == 1) {
			jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
			return true;
		}
		join_range(js, dst, ra, rb);
		/* a link that exists on one side only is dropped */
		if (ra->id && !jid_good(js, ra->id, rb->id)) {
			dst->id = 0;
			dst->delta = 0;
		}
		return true;

Two SCALAR_VALUE registers ra and rb with equal ranges and the same non-zero
delta pass reg_value_equal(ra, rb). If their IDs do not form a valid bijection
across the state (so jid_good(js, ra->id, rb->id) is false in pass 2),
join_reg() clears dst->id = 0 and dst->delta = 0.

When join_check() subsequently calls bpf_state_subsumes() -> states_equal() ->
func_states_equal() -> regsafe(), regsafe() checks:

	if (env->explore_alu_limits) {
		return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
		       check_scalar_ids(rold->id, rcur->id, idmap);
	}

Because delta lies before id in struct bpf_reg_state, won't memcmp() fail when
comparing dst->delta (0) against ra->delta (non-zero), causing join_check() to
hit verifier_bug() and return -EFAULT?

[1] https://lore.kernel.org/r/20261105204513.2230344-6-alexei.starovoitov@gmail.com
[2] https://lore.kernel.org/r/20261105204513.2230344-6-alexei.starovoitov@gmail.com
[3] https://lore.kernel.org/r/20261105204513.2230344-6-alexei.starovoitov@gmail.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=5

Re: [PATCH bpf-next v3 05/45] bpf: Add join of verifier states

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1975 bytes --]

> bpf: Add join of verifier states

TL;DR: Nit only: join_loop_stack() in join.c repeats fa->loop_stack[i] and
fb->loop_stack[i] many times; local entry pointers would read better, as in
loop_stack_safe().

> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> --- /dev/null
> +++ b/kernel/bpf/join.c

[ ... ]

> +static bool join_loop_stack(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
> +{
> +	u32 i;
> +
> +	if (fa->loop_stack_cnt != fb->loop_stack_cnt)
> +		return false;
> +	/* what is known about a counted loop comes from the state it was entered with */
> +	for (i = 0; i < fa->loop_stack_cnt; i++)
> +		if (fa->loop_stack[i].loop_id != fb->loop_stack[i].loop_id ||
> +		    fa->loop_stack[i].terminates != fb->loop_stack[i].terminates ||
> +		    fa->loop_stack[i].guarded != fb->loop_stack[i].guarded ||
> +		    fa->loop_stack[i].counted != fb->loop_stack[i].counted ||
> +		    fa->loop_stack[i].apart != fb->loop_stack[i].apart ||
> +		    (fa->loop_stack[i].apart && fa->loop_stack[i].trip != fb->loop_stack[i].trip) ||
> +		    fa->loop_stack[i].refs != fb->loop_stack[i].refs ||
> +		    fa->loop_stack[i].max_header_count != fb->loop_stack[i].max_header_count ||
> +		    fa->loop_stack[i].entry_state != fb->loop_stack[i].entry_state)
> +			return false;
> +	return true;
> +}

This isn't a bug, but could join_loop_stack() take local pointers for the
two entries, something like:

	const struct loop_stack_entry *ea = &fa->loop_stack[i];
	const struct loop_stack_entry *eb = &fb->loop_stack[i];

fa->loop_stack[i] and fb->loop_stack[i] are spelled out many times in the
comparison.  loop_stack_safe() in states.c uses old_stack and cur_stack
locals, so the comparison would read the same way.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 06/45] bpf: Tell which arguments of a call a join must leave alone

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

get_some_addr() of the selftest mem_rdonly_untrusted:

  if (bpf_get_prandom_u32())
          return bpf_rdonly_cast(0, bpf_core_type_id_kernel(struct sock));
  else
          return bpf_rdonly_cast(0, 0);

The compiler makes one call of the two, with r2 = 0 or the type id. The
paths meet in front of the call, and a join makes a range of two constants:

  R2 must be a known constant

cil_sock4_connect() and cil_sock4_sendmsg() of Cilium's bpf_sock:

  if (err == -EHOSTUNREACH || err == -ENOMEM) {
          try_set_retval(err);

The compiler decides the 'if' where err is set, and keeps a flag. All
paths meet, then the flag is tested. In the join err is [-113, 1]:

  At bpf_set_retval the register R1 has .. should have been in [-4095, 0]

The progs are valid. A pass that joins states must know, ahead of a call,
which registers it must not join.

antistall_set() of scx_layered:

  bpf_for_each(scx_dsq, __p, dsq_id, 0) {

bpf_iter_scx_dsq_next() ends this loop: it needs no may_goto. The pass that
finds the places for may_goto must know that, before the main pass sets
is_iter_next in insn_aux_data. With may_goto in each such loop layered_init
and mitosis_init are rejected on the added edge, and the schedulers in
tools/sched_ext take 48 % more insns.

  bpf_call_const_arg_regs()    a helper or a kfunc takes only a known
                               constant in these registers
  bpf_call_checked_arg_regs()  it compares them with a range: sizes of
                               memory, the argument of bpf_set_retval()
  bpf_insn_is_iter_next()      the call is bpf_iter_*_next()

The first two go by the types of the arguments in struct bpf_func_proto,
like check_func_arg(), for r1 - r5. bpf_set_retval() is checked elsewhere
and is added by hand. Nothing calls the three in this patch.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  3 ++
 kernel/bpf/verifier.c        | 62 ++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 1b08ccd5d766..dafbf2851b22 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1606,6 +1606,9 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
 void bpf_join_free(struct bpf_verifier_env *env);
 
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
+u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
+u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
+bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 754c64d7c9cb..07a9097e3d4e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -9206,6 +9206,13 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type)
 	return type == ARG_MEM_SIZE || type == ARG_MEM_SIZE_OR_ZERO;
 }
 
+/* A number that check_func_arg() takes only when it is a known constant */
+static bool arg_type_is_const(enum bpf_arg_type type)
+{
+	return type == ARG_CONST_SCALAR || type == ARG_CONST_MEM_SIZE ||
+	       type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
+}
+
 static bool arg_type_is_raw_mem(enum bpf_arg_type type)
 {
 	/*
@@ -15370,6 +15377,61 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return info;
 }
 
+/* insn_aux_data->is_iter_next is not there before the main pass */
+bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	struct bpf_call_arg_meta meta;
+
+	return bpf_pseudo_kfunc_call(insn) &&
+	       !bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) &&
+	       bpf_is_iter_next_kfunc(&meta);
+}
+
+/*
+ * check_func_arg() goes by the types of the arguments in the proto of a helper or a kfunc.
+ * So does what has to be known ahead of the call: the registers with arguments of a kind.
+ */
+static u16 call_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn,
+			 bool (*is)(enum bpf_arg_type type))
+{
+	const struct btf_func_model *fm = NULL;
+	const struct bpf_func_proto *fn;
+	struct bpf_kfunc_desc *desc;
+	u32 i, slot = 0;
+	u16 mask = 0;
+
+	if (bpf_pseudo_kfunc_call(insn)) {
+		desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+		if (!desc)
+			return 0;
+		fn = &desc->proto;
+		fm = &desc->func_model;
+	} else if (bpf_get_helper_proto(env, insn->imm, &fn) < 0) {
+		return 0;
+	}
+	for (i = 0; i < ARRAY_SIZE(fn->arg_type) && fn->arg_type[i] != ARG_UNUSED &&
+		    slot < MAX_BPF_FUNC_REG_ARGS; i++) {
+		if (is(fn->arg_type[i]))
+			mask |= BIT(BPF_REG_1 + slot);
+		slot += fm ? btf_func_model_arg_slots(fm, i) : 1;
+	}
+	return mask;
+}
+
+/* The registers in which the helper or the kfunc takes nothing but a known constant */
+u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	return call_arg_regs(env, insn, arg_type_is_const);
+}
+
+/* The registers in which it takes a number that has to be in a range */
+u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	if (bpf_helper_call(insn) && insn->imm == BPF_FUNC_set_retval)
+		return BIT(BPF_REG_1);
+	return call_arg_regs(env, insn, arg_type_is_mem_size);
+}
+
 /*
  * Describe a kfunc's stack access through argument slot @arg (0-based).
  * As for helpers, must_write describes destruction of prior verifier state,
-- 
2.55.0

[PATCH bpf-next v3 07/45] bpf: Add the data structures of the main pass over the control flow graph

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier follows each path of a prog from the entry to exit. It ends a
path early only where an earlier path left a state that covers the current
one. A live value that differs from path to path defeats that. Then the
work grows with the number of paths, not with the size of the prog:

  for b in bytes { h = (h ^ *b as u64).wrapping_mul(0x100000001b3); }

In this FNV hash h is a new number in each iteration, so no state covers
the next one. Branches that set different values do the same to C code.

  insns processed                                     today   this series
  rust-bpf   hash_fnv_bytes                         1000001 *         160
  rust-bpf   scx_qmap/qmap_init_cids                 131088 *         196
  selftests  test_verif_scale3/balancer_ingress      821107         57155
  scx        scx_eevdf/eevdf_dispatch                350041         15199
  Cilium     bpf_host/cil_to_netdev, Oct 2026         64658          7472
  * "BPF program is too large", "The sequence of 8193 jumps is too complex"

flow.c is a new main pass: a data-flow analysis over the control flow
graph that runs to a fixpoint. Where two paths meet, it joins their states
into one state that covers both: r1=1 and r1=5 give r1=[1,5]. At the head
of a loop it widens what grows: h becomes an unknown scalar, and one more
pass over the body ends the loop. A join can lose a fact that a later check
depends on. Thus a basic block has a set of states, not one state:

  a state arrives at a basic block        the verifier
  a state of the set covers it            drops it
  it has a join with a state of the set   replaces that state with the join
  all other cases                         adds it to the set

The verifier verifies a block with each new state of its set and sends the
results to the successors of the block. The pass ends when no set has a new
state. The check of one insn does not change.

Add the head comment, the limits and the data structures:

  struct flow_state   a state in a set, or the state of a call that waits
  struct flow_set     the states at one insn
  struct flow_entry   the calls that wait for a subprog, its exit states
  struct flow_need    what the checks after an insn depend on

The limits are on states and grow with the prog. For a prog of N insns:
1024 at an insn, 16 * N + 4096 at a time, 256 * N + 64K in all.
Nothing builds flow.c until its last function is there.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 181 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 181 insertions(+)
 create mode 100644 kernel/bpf/flow.c

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
new file mode 100644
index 000000000000..01681957855d
--- /dev/null
+++ b/kernel/bpf/flow.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * The main pass of the verifier as a fixpoint over the control flow graph.
+ *
+ * Every insn that starts a basic block has a set of states. The state that
+ * arrives there is
+ * - dropped, if it is within a state of the set,
+ * - joined with a state of the set, if there is a join, see bpf_states_join(),
+ *   that doesn't make up what neither of them has, see flow_join_makes_up(),
+ * - added to the set otherwise.
+ * What is new in the set is pending. Pending states are processed in the order
+ * of the program in which a loop is done with before what follows it, see
+ * flow_compute_order(). The pass ends when nothing is pending.
+ *
+ * Loops. The state that enters a loop is made to stand for all iterations with
+ * what scalar evolution knows about the loop, see bpf_flow_loop_enter().
+ * The state that comes back to the loop head is expected to be within it.
+ * If it is not, the two are joined, what grew is forgotten, and the loop is
+ * done again.
+ *
+ * Functions. A static function that gets nothing but numbers and pointers to
+ * what is not the stack is verified on its own, for the states at its entry
+ * and not for every chain of calls. The call waits. What comes out of the
+ * function goes to all calls that wait for the states it comes from.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/filter.h>
+#include <linux/jhash.h>
+#include <linux/rbtree.h>
+#include <linux/slab.h>
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+/*
+ * States at an insn that have no join: kinds of values, references, chains of calls.
+ * A state that arrives is compared with those that it may be related to, see flow_may_relate().
+ * At worst that is all of them, and the work is the square of it.
+ */
+#define BPF_FLOW_MAX_STATES	1024
+/*
+ * All states in the sets at a time: that many for every insn of the prog, and some. Not counted:
+ * calls that wait, what came out of a function, the states that loops were entered with.
+ */
+#define BPF_FLOW_LIVE_PER_INSN	16
+#define BPF_FLOW_LIVE_BASE	4096
+/* All states that were ever there. It bounds the work by the size of the prog. */
+#define BPF_FLOW_TOTAL_PER_INSN	256
+#define BPF_FLOW_TOTAL_BASE	65536
+/*
+ * What comes out of a function is joined that many times. Then it is widened, like what
+ * comes back to a loop head: the exit states have to stop changing.
+ */
+#define BPF_FLOW_MAX_JOINS	16
+/* With fewer states at an insn it's faster to compare them than to look at their shapes. */
+#define BPF_FLOW_MIN_SHAPES	8
+/* bpf_loop() with that many iterations at most is walked. */
+#define BPF_FLOW_MAX_CALLBACKS	8
+/* What a callback returns to the helper with is joined that many times before it is widened. */
+#define BPF_FLOW_CALLBACK_JOINS	4
+
+enum flow_cycle {
+	FLOW_NO_CYCLE,
+	/* what grew is forgotten */
+	FLOW_CYCLE,
+	/* after a while. A flag that a callback sets doesn't grow any further. */
+	FLOW_CALLBACK_CYCLE,
+};
+
+/* See flow_may_relate() */
+enum flow_word {
+	FLOW_CHAIN,
+	FLOW_CTX,
+	FLOW_LAYOUT,
+	FLOW_PTRS,
+	FLOW_OFFS,
+	FLOW_SHAPE_WORDS,
+};
+
+struct flow_shape {
+	u32 word[FLOW_SHAPE_WORDS];
+	/* how much of the packet is checked, the sum over the pointers to the packet */
+	u32 pkt_range;
+	/* what comes after FLOW_CTX is computed */
+	bool places;
+};
+
+struct flow_state {
+	struct bpf_verifier_state st;
+	/* states at the insn, see struct flow_set, calls of struct flow_entry */
+	struct list_head node;
+	struct rb_node rb;		/* pending states */
+	int prev_insn_idx;
+	u32 seq;
+	struct flow_shape shape;
+	u16 joins;
+	bool pending;
+	/* the order of pending states, see flow_set_pending() */
+	u8 key_len;
+	u32 key[MAX_CALL_FRAMES + 1];
+};
+
+/* what came out of a function that is verified on its own */
+struct flow_exit {
+	struct list_head node;
+	struct bpf_verifier_state st;
+	u16 joins;
+};
+
+/*
+ * Every call that waits makes an entry of the function: the state at its first insn and what
+ * becomes of it. Entries whose states met are one set: what comes out of the function for one of
+ * them is for the calls of all of them. One entry stands for the set and has its lists.
+ */
+struct flow_entry {
+	struct flow_entry *up;		/* towards the one that stands for the set: up == itself */
+	struct list_head calls;		/* struct flow_state: the calls that wait */
+	struct list_head exits;		/* struct flow_exit: what came out */
+};
+
+struct flow_func {
+	int exit_idx;			/* all exits are compared as if they were here */
+};
+
+/* States at an insn, in the order in which they came. */
+struct flow_set {
+	struct list_head list;
+};
+
+struct flow_united {
+	u32 a, b;
+};
+
+struct flow_kept {
+	struct list_head node;
+	struct bpf_verifier_state st;
+};
+
+/* What the checks ahead of an insn look at, see enum flow_need_kind */
+struct flow_need {
+	u64 slots;			/* of the stack, fp-8 is bit 0 */
+	u16 regs;
+};
+
+/* see flow_compute_need_const() */
+enum flow_need_kind {
+	FLOW_NEED_ARG,
+	FLOW_NEED_STACK_OFF,
+	FLOW_NEED_PTR_OFF,
+	FLOW_NEED_JUMP,
+	FLOW_NEED_VALUE_OFF,
+	FLOW_NEED_BASE,
+	FLOW_NEED_JUMP_ALU,
+	FLOW_NEED_KINDS,
+};
+
+struct bpf_flow {
+	/* computed once */
+	u32 len;			/* of the prog during the pass */
+	u32 nr_funcs;
+	u32 *pos;			/* place of the insn in the order */
+	u32 *rank;			/* callers go first */
+	struct flow_need *need[FLOW_NEED_KINDS];
+	unsigned long *point;		/* insn starts a basic block */
+	unsigned long *keep;		/* states that were processed are compared with */
+
+	/* of the pass */
+	struct flow_set *states;	/* per insn */
+	struct flow_set *enter;		/* per insn: the states that enter the loop at its head */
+	u32 *nr_states;			/* per insn: the two sets are counted together */
+	struct rb_root_cached pending;
+	struct flow_func *funcs;
+	struct flow_entry **entries;	/* bpf_verifier_state.flow_entry is the index, 0 is none */
+	u32 entry_cnt, entry_cap;
+	struct flow_united *united;
+	u32 united_cnt, united_cap;
+	struct list_head kept;
+	struct bpf_verifier_state tmp;
+	u32 seq;
+	u32 live_states;
+};
-- 
2.55.0

Re: [PATCH bpf-next v3 07/45] bpf: Add the data structures of the main pass over the control flow graph

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]

> bpf: Add the data structures of the main pass over the control flow graph

TL;DR: Comment-only nit: the head comment's rule for verifying static
functions on their own is incomplete, since later patches refuse more cases
(e.g. maybe-NULL pointer args). Suggest pointing to func_is_on_its_own().

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> new file mode 100644
> index 000000000000..01681957855d
> --- /dev/null
> +++ b/kernel/bpf/flow.c

[ ... ]

> + * Functions. A static function that gets nothing but numbers and pointers to
> + * what is not the stack is verified on its own, for the states at its entry
> + * and not for every chain of calls. The call waits. What comes out of the
> + * function goes to all calls that wait for the states it comes from.

This isn't a bug, but is this rule complete as written?

The func_is_on_its_own() function added later in the series (in
"bpf: Verify static subprogs on their own") also refuses functions for
other reasons:

    if (sub->changes_pkt_data || sub->has_tail_call || sub->has_ld_abs ||
        sub->is_cb || sub->is_async_cb || sub->is_exception_cb)
            return false;
    if (state->speculative)
            return false;
    ...
    if (state->acquired_refs > BPF_ID_MAP_SIZE / 2)
            return false;

and arg_needs_caller() refuses some pointer arguments:

    if (reg_is_pkt_pointer(reg) && reg->id)
            return true;
    if ((reg->type == PTR_TO_PACKET_END ||
         (reg->type == PTR_TO_PACKET && sub->loads_pkt_end)) && !sub->recursive)
            return true;
    return type_may_be_null(reg->type) && !sub->recursive;

So a static non-recursive subprog that gets a map value pointer that may be
NULL, or that calls a helper that changes packet data, is still verified for
every chain of calls.  Later commits narrow the rule further, and the
paragraph is not updated.

The other paragraphs of this comment point to the function that holds the
details, such as bpf_states_join() and flow_compute_order().  Could this one
say that the stated condition is necessary but not sufficient, and refer to
func_is_on_its_own()?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 08/45] bpf: Order the insns, find what stops a loop and where may_goto can go

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

A fixpoint shows that each iteration of a loop is safe, not that it ends:

  do { i++; sum += PT_REGS_RC(ctx); } while (i < 0x100000000ULL);

Today selftest loop3 hits the limit of 1M insns. The main pass widens i and
is done, so the verifier has to add a may_goto. That is an edge that the
prog does not have. Its place decides which progs load:

1. None, if each iteration executes a may_goto that leaves the loop, or
bpf_iter_*_next() of an iterator that was created before the loop.

  bpf_repeat(1000000) {
          map_val = bpf_map_lookup_elem(&hash_map, &key);
          if (map_val) { found = true; break; }
  }
  if (found) *map_val = 123;

An added may_goto goes where 'break' goes, without the test. Selftest
iters/iter_tricky_but_fine: "R6 invalid mem access 'map_value_or_null'".
layered_init of scx_layered and mitosis_init of scx_mitosis fail alike.

2. In front of a conditional jump that leaves the loop, to its target: the
prog gets there with the same registers and stack. One jump of the loop
itself, bpf_insn_leaves_loop(), that each iteration goes through is enough:
the 'while' of loop3. Else each back edge gets one: a may_goto at the head
skips the body, and the first exit can read what the body sets.

3. At the head, if a back edge has none. The compiler makes one loop of

  while self.locked.compare_exchange(0, 1, Acquire, Relaxed).is_err() {
      while self.locked.load(Relaxed) != 0 { core::hint::spin_loop(); }
  }

and the inner back edge has none. Without this place the main pass says
"loop at insn 127 never exits" for 142 of the 225 progs of the schedulers
of rust-bpf. This edge can skip what stops an outer loop: it leaves all.

The added insn executes in each state. Thus a loop has it for all states or
for none. The main pass can find out late: it first goes through loop3
iteration by iteration, as today, and gives up. Then the pass is repeated.

The main pass takes the blocks that have a new state in the order of
flow_compute_order(): reverse post order, but a loop stays together. Else
the code behind a 'break' is verified before the loop is done, then again.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   1 +
 kernel/bpf/flow.c            | 335 +++++++++++++++++++++++++++++++++++
 kernel/bpf/loops.c           |   8 +
 3 files changed, 344 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index dafbf2851b22..53507aed9f20 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1609,6 +1609,7 @@ int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
+bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 01681957855d..0f5acdf1cb94 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -179,3 +179,338 @@ struct bpf_flow {
 	u32 seq;
 	u32 live_states;
 };
+
+/*
+ * The order. Reverse post order is a topological order of the graph without
+ * back edges, but it may put what follows a loop between the insns of
+ * the loop. Keep the insns of a loop together, the head first: in the list of
+ * its parent the loop takes the place of the insn of it that comes first.
+ *
+ * Every cycle has a back edge of the depth first search in it and
+ * bpf_compute_loops() makes the target of such edge a loop head. So every edge
+ * that goes backwards in the order is in a loop whose head is at or before its
+ * target, loops that are not reducible included.
+ */
+static int flow_compute_order(struct bpf_verifier_env *env, struct bpf_flow *flow)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int len = env->prog->len, i, s, n, h, top, depth;
+	int *next, *head, *tail, *stack;
+	unsigned long *linked;
+	u32 pos = 0;
+	int err = -ENOMEM;
+
+	/* lists of members are indexed by loop head, len is the function */
+	next = kvmalloc_array(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+	head = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+	tail = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+	stack = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+	linked = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+	if (!next || !head || !tail || !stack || !linked)
+		goto out;
+
+	for (i = 0; i <= len; i++)
+		head[i] = tail[i] = -1;
+
+	for (s = 0; s < env->subprog_cnt; s++) {
+		int first = env->subprog_info[s].postorder_start;
+		int last = env->subprog_info[s + 1].postorder_start - 1;
+
+		head[len] = tail[len] = -1;
+		for (i = last; i >= first; i--) {
+			n = env->cfg.insn_postorder[i];
+			/* n, then the heads of the loops around it that are not in a list yet */
+			depth = 0;
+			for (h = n; h >= 0 && !test_bit(h, linked); h = aux[h].loop_header)
+				stack[depth++] = h;
+			while (depth) {
+				h = stack[--depth];
+				top = aux[h].loop_header >= 0 ? aux[h].loop_header : len;
+				__set_bit(h, linked);
+				next[h] = -1;
+				if (tail[top] < 0)
+					head[top] = h;
+				else
+					next[tail[top]] = h;
+				tail[top] = h;
+			}
+		}
+
+		depth = 0;
+		n = head[len];
+		while (n >= 0 || depth) {
+			if (n < 0) {
+				n = next[stack[--depth]];
+				continue;
+			}
+			flow->pos[n] = pos++;
+			if (aux[n].loop && head[n] >= 0) {
+				stack[depth++] = n;
+				n = head[n];
+			} else {
+				n = next[n];
+			}
+		}
+	}
+	err = 0;
+out:
+	kvfree(next);
+	kvfree(head);
+	kvfree(tail);
+	kvfree(stack);
+	kvfree(linked);
+	return err;
+}
+
+/* Does the insn at @idx stop the loop @h? */
+static bool flow_is_bound(struct bpf_verifier_env *env, int h, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	int t;
+
+	/* Once it is out of budget it's a jump. It has to leave the loop. */
+	if (bpf_is_may_goto_insn(insn)) {
+		t = bpf_loop_at_index(env, idx + insn->off + 1);
+		return t != h && !bpf_is_nested_loop(env, t, h);
+	}
+	if (!bpf_insn_is_iter_next(env, insn))
+		return false;
+	/* unless the main pass finds that it doesn't, see iter_stops_loops() */
+	t = env->insn_aux_data[idx].not_stopped;
+	return t < 0 || bpf_is_nested_loop(env, h, t);
+}
+
+/*
+ * What stops a loop at run time is may_goto that leaves it, or bpf_iter_*_next() of
+ * an iterator that is older than the loop, that every trip goes through. A loop without
+ * it gets may_goto unless the number of its trips turns out to be known.
+ *
+ * The place for it is in front of a conditional jump that leaves the loop, with
+ * the same target. The registers and the stack are what they are when
+ * the program leaves the loop there. One such jump that every trip goes
+ * through is enough. Otherwise every back edge needs one.
+ */
+static void flow_set_guard(struct bpf_verifier_env *env, int h, int n)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int t = n + env->prog->insnsi[n].off + 1;
+	int i = bpf_loop_at_index(env, t);
+
+	if (i == h || bpf_is_nested_loop(env, i, h))
+		t = n + 1;
+	aux[n].guard_place = true;
+	aux[n].guard_to = t;
+	aux[h].loop->has_guards = true;
+}
+
+static void flow_place_guards(struct bpf_verifier_env *env, int h, u32 *seen)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *loop = aux[h].loop;
+	int i, n, o, t, found = -1;
+
+	if (loop->backedges_overflow)
+		goto header;
+
+	/*
+	 * The head of a loop with more than one entry doesn't dominate what is in the loop.
+	 * Then an insn that dominates a back edge can be passed once, on the way in, and
+	 * never again. Every trip goes through the jump that is the back edge, and
+	 * through the head.
+	 */
+	if (loop->irreducible) {
+		for (i = 0; i < loop->backedges_cnt; i++)
+			if (!loop->backedges[i].latches_cnt ||
+			    loop->backedges[i].latches[0] != loop->backedges[i].from)
+				goto header;
+		for (i = 0; i < loop->backedges_cnt; i++)
+			flow_set_guard(env, h, loop->backedges[i].from);
+		return;
+	}
+
+	/* seen[n]: the number of back edges that n dominates */
+	for (i = 0; i < loop->backedges_cnt; i++)
+		for (n = loop->backedges[i].from; n >= 0; n = n == h ? -1 : env->idoms[n])
+			seen[n]++;
+	for (n = loop->backedges[0].from; n >= 0 && found < 0; n = n == h ? -1 : env->idoms[n])
+		if (seen[n] == loop->backedges_cnt && bpf_insn_leaves_loop(env, h, n))
+			found = n;
+	for (i = 0; i < loop->backedges_cnt; i++)
+		for (n = loop->backedges[i].from; n >= 0; n = n == h ? -1 : env->idoms[n])
+			seen[n] = 0;
+	if (found >= 0) {
+		flow_set_guard(env, h, found);
+		return;
+	}
+
+	for (i = 0; i < loop->backedges_cnt; i++)
+		if (!loop->backedges[i].latches_cnt)
+			goto header;
+	for (i = 0; i < loop->backedges_cnt; i++)
+		flow_set_guard(env, h, loop->backedges[i].latches[0]);
+	return;
+
+header:
+	/*
+	 * In front of a jump may_goto goes where the jump goes. From the head to where the loop
+	 * is left is not an edge of the program: it skips what is in the loop. If that is what
+	 * stops a loop around this one, bpf_iter_*_next() or i++, and the exit leads back to
+	 * the head of that loop, the program goes round and round from may_goto to may_goto.
+	 * So leave all loops.
+	 *
+	 * What is derived from the graph in the direction of the edges has to know about this
+	 * one, see bpf_head_guard_target().
+	 */
+	for (o = h; aux[o].loop_header >= 0; o = aux[o].loop_header)
+		;
+	/* a loop that has no exit can only be stopped together with the program */
+	if (!aux[o].loop->exits_cnt)
+		return;
+	t = aux[o].loop->exits[0].to;
+	aux[h].guard_place = true;
+	aux[h].guard_at_header = true;
+	aux[h].guard_to = t;
+	loop->has_guards = true;
+}
+
+/* Does every trip of the loop @h go through an insn that stops it? */
+static bool flow_loop_is_bounded(struct bpf_verifier_env *env, int h)
+{
+	struct bpf_loop *loop = env->insn_aux_data[h].loop;
+	int i, n;
+
+	/* see flow_place_guards() about loops with more than one entry */
+	if (loop->backedges_overflow || loop->irreducible)
+		return false;
+	for (i = 0; i < loop->backedges_cnt; i++) {
+		for (n = loop->backedges[i].from; n >= 0 && n != h; n = env->idoms[n])
+			if (flow_is_bound(env, h, n))
+				break;
+		if (n < 0 || !flow_is_bound(env, h, n))
+			return false;
+	}
+	return true;
+}
+
+/*
+ * The main pass got to bpf_iter_*_next() at @insn_idx in a state in which it doesn't stop
+ * the loop @h around it. What is assumed about the loops stays as it is until the pass
+ * is over, so that all states are treated alike.
+ */
+void bpf_flow_next_is_no_bound(struct bpf_verifier_env *env, int insn_idx, int h)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
+
+	if (aux->not_stopped >= 0 && !bpf_is_nested_loop(env, h, aux->not_stopped))
+		return;
+	aux->not_stopped = h;
+	env->restart = true;
+}
+
+/*
+ * True if a loop was taken for bounded in the pass and is not. The pass is done again then.
+ * A loop that has something else that stops it, like the outer one of two nested
+ * bpf_for(), is not affected. Every time at least one loop is found out for good.
+ */
+bool bpf_flow_loops_lost_bound(struct bpf_verifier_env *env)
+{
+	struct bpf_loop *loop;
+	bool lost = false;
+	int h;
+
+	for (h = 0; h < env->prog->len; h++) {
+		loop = env->insn_aux_data[h].loop;
+		if (!loop || !loop->rt_bounded || flow_loop_is_bounded(env, h))
+			continue;
+		loop->rt_bounded = false;
+		lost = true;
+	}
+	return lost;
+}
+
+/*
+ * A state is in the loop @h without a count, so the loop gets may_goto. At run time may_goto is
+ * there for everybody, also for those who know the number of trips or got to the end of the loop
+ * trip by trip. It can be out of budget in the first trip, because of a loop before this one.
+ *
+ * So a loop has may_goto or it has not, and a state takes the edges of may_goto if the loop had
+ * it when the state entered. If a state entered without it, with a count or on a walk that may
+ * give up, all is done again, and the loop has may_goto from the start.
+ */
+void bpf_flow_loop_gets_guards(struct bpf_verifier_env *env, int h)
+{
+	struct bpf_loop *loop = env->insn_aux_data[h].loop;
+
+	if (loop->guarded)
+		return;
+	loop->guarded = true;
+	if (loop->no_guard_edges)
+		env->guards_late = true;
+}
+
+/* The main prog and the global functions are about to be verified, maybe not for the first time */
+void bpf_flow_new_round(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int i;
+
+	env->restart = false;
+	env->guards_late = false;
+	for (i = 0; i < env->prog->len; i++) {
+		aux[i].guard_edge_dropped = false;
+		if (aux[i].loop)
+			aux[i].loop->no_guard_edges = false;
+	}
+}
+
+/*
+ * Which loops have something that stops them at run time, and where may_goto goes in the others.
+ * Returns 1 if a place is at the head of a loop.
+ */
+int bpf_flow_place_guards(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int len = env->prog->len, h, i, depth, ret = 0;
+	struct bpf_loop *loop;
+	u32 *seen;
+
+	seen = kvcalloc(len, sizeof(*seen), GFP_KERNEL_ACCOUNT);
+	if (!seen)
+		return -ENOMEM;
+
+	for (i = 0; i < len; i++) {
+		aux[i].not_stopped = -1;
+		depth = 0;
+		for (h = bpf_loop_at_index(env, i); h >= 0; h = aux[h].loop_header) {
+			/* the main pass doesn't get that deep */
+			if (++depth > LOOP_STACK_SIZE) {
+				verbose(env, "Too many nested loops at %d\n", i);
+				kvfree(seen);
+				return -E2BIG;
+			}
+			aux[h].loop->nr_insns++;
+			if (bpf_pseudo_call(&env->prog->insnsi[i]) ||
+			    bpf_is_callx(&env->prog->insnsi[i]))
+				aux[h].loop->has_calls = true;
+		}
+	}
+
+	for (h = 0; h < len; h++) {
+		loop = aux[h].loop;
+		if (!loop)
+			continue;
+		/* a deep nest of loops takes long */
+		if (signal_pending(current)) {
+			ret = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		loop->rt_bounded = flow_loop_is_bounded(env, h);
+		/* also when it's bounded: the main pass may find that it's not */
+		flow_place_guards(env, h, seen);
+		if (aux[h].guard_at_header)
+			ret = 1;
+	}
+	kvfree(seen);
+	return ret;
+}
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index cc15712f70d4..729e0055f86f 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -540,6 +540,14 @@ bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n)
 	       (f != h && !bpf_is_nested_loop(env, f, h));
 }
 
+/* The same for an insn that is in the loop @h itself */
+bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n)
+{
+	if (bpf_loop_at_index(env, n) != h)
+		return false;
+	return bpf_jump_leaves_loop(env, h, n);
+}
+
 int bpf_compute_loops(struct bpf_verifier_env *env)
 {
 	int i, j, k, s, t, iloop, sloop, err = 0, len = env->prog->len;
-- 
2.55.0

[PATCH bpf-next v3 09/45] bpf: Find the registers that hold a number or a pointer to arena

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The allocator of tools/sched_ext/scx_sdt.bpf.c looks for a free slot:

  if ((word & 0xffffffff) == 0) { num += 32; word >>= 32; }
  if ((word & 0xffff) == 0) { num += 16; word >>= 16; }
  ...                                         six times, this is ffs()
  return (i * 64) + ffs(freeslots);

The main pass joins the two states behind each 'if': num is [0, 32], then
[0, 48]. It must not join two constants that the prog adds to a pointer to
the packet, the context or a kernel object: these need the exact offset.
The states meet before the add, and there the add is only 'r0 += r3'. If r0
can be such a pointer, num stays a constant and each 'if' doubles the
states: up to 256 at an insn, not 2.

  insns processed with this pass, and without it:
  scx_sdt    sdt_init_task                               666       4897
  scx_rusty  rusty_init_task, the same allocator        1404      22884
  all progs of: selftests +23 %, scx +72 %, schedulers of rust-bpf +129 %
  rust-bpf   the main pass rejects rand_ranges_shuffle:
             "more than 1024 states that cannot be joined at insn 393"

bpf_flow_compute_kinds() goes forward through the control flow graph,
before the main pass. In front of each insn it finds the registers, the
stack slots and the stack arguments that hold, on each path to the insn:
 - a scalar or a pointer to arena: no check depends on where it points,
 - that, or a pointer to a map value that has only scalars and that is not
   read-only: only the lowest and the highest offset count,
 - the context: a load through it can give a pointer to the packet.
It starts with 'all is known' and removes what is false on some path, until
nothing changes. The result is plain_regs and value_regs of an insn.

  ALU op other than 64-bit add, sub, mov     scalar, addr_space_cast too
  64-bit add, sub                            what both operands are
  load through a scalar or a map value       scalar
  load through the context                   is_valid_access() tells
  other load                                 scalar if below 8 bytes
  spill and fill through r10, r11            what the prog stored
  helper, kfunc                              from the prototype
  static subprog                             true at all calls, all exits
  global subprog, callback, target of callx  only the context

The result only selects the states that the main pass does not join. If it
takes a pointer for a scalar, a join can reject a valid prog. If it takes
a scalar for a pointer, there are more states. No unsafe prog loads.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 307 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 307 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 0f5acdf1cb94..837d5054c2a3 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -514,3 +514,310 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)
 	kvfree(seen);
 	return ret;
 }
+
+/*
+ * The places of a function that hold a value: the registers, the arguments on the stack that it
+ * got and those that it hands over, the slots of the stack that struct flow_need has a bit for.
+ */
+enum {
+	FLOW_ARG_IN	= 16,
+	FLOW_ARG_OUT	= FLOW_ARG_IN + MAX_STACK_ARG_SLOTS,
+	FLOW_SLOT	= 64,
+	FLOW_PLACES	= 128,
+};
+static_assert(FLOW_ARG_OUT + MAX_STACK_ARG_SLOTS <= FLOW_SLOT);
+
+enum flow_kind {
+	FLOW_IS_PLAIN,	/* a number or a pointer to arena: nothing checks where it points to */
+	FLOW_IS_VALUE,	/* that, or a pointer to a map value that holds nothing but numbers */
+	FLOW_IS_CTX,	/* the context, as the prog or a global function got it */
+	FLOW_KINDS,
+	FLOW_IS_OTHER = FLOW_KINDS,
+};
+
+/* What the places hold for sure in front of an insn */
+struct flow_kinds {
+	unsigned long is[FLOW_KINDS][BITS_TO_LONGS(FLOW_PLACES)];
+};
+
+static void flow_kind_set(struct flow_kinds *k, u32 place, enum flow_kind kind)
+{
+	int i;
+
+	for (i = 0; i < FLOW_KINDS; i++)
+		__assign_bit(place, k->is[i], i == kind);
+	if (kind == FLOW_IS_PLAIN)
+		__set_bit(place, k->is[FLOW_IS_VALUE]);
+}
+
+static void flow_kind_copy(struct flow_kinds *to, u32 place, const struct flow_kinds *from,
+			   u32 from_place)
+{
+	int i;
+
+	for (i = 0; i < FLOW_KINDS; i++)
+		__assign_bit(place, to->is[i], test_bit(from_place, from->is[i]));
+}
+
+/* What holds on both ways to an insn. Returns true if that is less than before. */
+static bool flow_kinds_meet(struct flow_kinds *to, const struct flow_kinds *from)
+{
+	unsigned long lost = 0;
+	int i, j;
+
+	for (i = 0; i < FLOW_KINDS; i++) {
+		for (j = 0; j < ARRAY_SIZE(to->is[i]); j++) {
+			lost |= to->is[i][j] & ~from->is[i][j];
+			to->is[i][j] &= from->is[i][j];
+		}
+	}
+	return lost;
+}
+
+/* The first insn of a function that is called from @k */
+static bool flow_kinds_enter(struct flow_kinds *to, const struct flow_kinds *k)
+{
+	struct flow_kinds e = {};
+	int i;
+
+	/* r12 points to arena. The rest is not set. */
+	bitmap_fill(e.is[FLOW_IS_PLAIN], FLOW_PLACES);
+	bitmap_fill(e.is[FLOW_IS_VALUE], FLOW_PLACES);
+	flow_kind_set(&e, BPF_REG_FP, FLOW_IS_OTHER);
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+		flow_kind_copy(&e, i, k, i);
+	for (i = 0; i < MAX_STACK_ARG_SLOTS; i++)
+		flow_kind_copy(&e, FLOW_ARG_IN + i, k, FLOW_ARG_OUT + i);
+	return flow_kinds_meet(to, &e);
+}
+
+/* The place that @insn loads from or stores to through r10 or r11, -1 if it's something else */
+static int flow_mem_place(const struct bpf_insn *insn, u8 base)
+{
+	int n = (abs(insn->off) - 1) / BPF_REG_SIZE;
+
+	if (!insn->off)
+		return -1;
+	if (base == BPF_REG_FP && insn->off < 0 && n < 64)
+		return FLOW_SLOT + n;
+	if (base == BPF_REG_PARAMS && n < MAX_STACK_ARG_SLOTS)
+		return (insn->off < 0 ? FLOW_ARG_OUT : FLOW_ARG_IN) + n;
+	return -1;
+}
+
+static enum flow_kind flow_map_value_kind(const struct bpf_map *map)
+{
+	if (map->map_type == BPF_MAP_TYPE_ARENA)
+		return FLOW_IS_PLAIN;
+	/* what is at a known offset of a read-only map is known: a number, a string, a function */
+	if (map->map_flags & BPF_F_RDONLY_PROG)
+		return FLOW_IS_OTHER;
+	return IS_ERR_OR_NULL(map->record) && !map->inner_map_meta ? FLOW_IS_VALUE : FLOW_IS_OTHER;
+}
+
+/* What @insn loads through the register @base */
+static enum flow_kind flow_loaded_kind(struct bpf_verifier_env *env, const struct flow_kinds *k,
+				       const struct bpf_insn *insn, u8 base)
+{
+	struct bpf_insn_access_aux info = { .reg_type = SCALAR_VALUE };
+	int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
+	bool tstamp = env->prog->tstamp_type_access, valid;
+
+	/* a number that is good for a load is an address in arena */
+	if (test_bit(base, k->is[FLOW_IS_VALUE]))
+		return FLOW_IS_PLAIN;
+	/* only the context has pointers that are shorter than a register */
+	if (!test_bit(base, k->is[FLOW_IS_CTX]))
+		return size < BPF_REG_SIZE ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
+	valid = env->ops->is_valid_access &&
+		env->ops->is_valid_access(insn->off, size, BPF_READ, env->prog, &info);
+	/* it notes what the prog reads, and this is a question */
+	env->prog->tstamp_type_access = tstamp;
+	return valid && info.reg_type == SCALAR_VALUE ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
+}
+
+/* What a helper or a kfunc returns */
+static enum flow_kind flow_returned_kind(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	const struct bpf_func_proto *fn;
+	struct bpf_call_arg_meta meta;
+	const struct btf_type *t;
+
+	if (bpf_pseudo_kfunc_call(insn)) {
+		if (bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) < 0)
+			return FLOW_IS_OTHER;
+		t = btf_type_skip_modifiers(meta.btf, meta.func_proto->type, NULL);
+		if (!btf_type_is_ptr(t))
+			return FLOW_IS_PLAIN;
+		/* void * is a number, unless the verifier knows the kfunc. Such kfunc may fail. */
+		t = btf_type_skip_modifiers(meta.btf, t->type, NULL);
+		return btf_type_is_void(t) && !(meta.kfunc_flags & KF_RET_NULL) ? FLOW_IS_PLAIN :
+										  FLOW_IS_OTHER;
+	}
+	if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
+		return FLOW_IS_OTHER;
+	if (fn->ret_type == RET_INTEGER || fn->ret_type == RET_VOID)
+		return FLOW_IS_PLAIN;
+	if (base_type(fn->ret_type) == RET_PTR_TO_MAP_VALUE &&
+	    (aux->const_reg_map_mask & BIT(BPF_REG_1)))
+		return flow_map_value_kind(env->used_maps[aux->const_reg_vals[BPF_REG_1]]);
+	return FLOW_IS_OTHER;
+}
+
+/*
+ * @k is what holds in front of the insn at @idx. Make it what holds behind it.
+ * @ret: the registers that are FLOW_IS_PLAIN at all exits of a subprog.
+ * Returns true if less is known than before at the entry or about the exits of a subprog.
+ */
+static bool flow_kinds_step(struct bpf_verifier_env *env, struct flow_kinds *kinds, u16 *ret,
+			    struct flow_kinds *k, int idx)
+{
+	struct bpf_subprog_info *sub = env->subprog_info;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+	u8 dst = insn->dst_reg, src = insn->src_reg;
+	bool wide = BPF_SIZE(insn->code) == BPF_DW && BPF_MODE(insn->code) == BPF_MEM &&
+		    !(insn->off % BPF_REG_SIZE);
+	bool lost = false;
+	u16 regs = U16_MAX;
+	int i, s;
+
+	if (class == BPF_ALU64 && (op == BPF_ADD || op == BPF_SUB)) {
+		__clear_bit(dst, k->is[FLOW_IS_CTX]);
+		for (i = 0; BPF_SRC(insn->code) == BPF_X && i < FLOW_IS_CTX; i++)
+			if (!test_bit(src, k->is[i]))
+				__clear_bit(dst, k->is[i]);
+	} else if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && !insn->off) {
+		flow_kind_copy(k, dst, k, src);
+	} else if (class == BPF_ALU || class == BPF_ALU64) {
+		/*
+		 * addr_space_cast too. It leaves a pointer that is not in arena as it is, and
+		 * a compiler casts what it cannot know. Nearly all of that is in arena.
+		 */
+		flow_kind_set(k, dst, FLOW_IS_PLAIN);
+	} else if (class == BPF_LDX) {
+		s = flow_mem_place(insn, src);
+		if (wide && s >= 0)
+			flow_kind_copy(k, dst, k, s);
+		else
+			flow_kind_set(k, dst, flow_loaded_kind(env, k, insn, src));
+	} else if (class == BPF_STX || class == BPF_ST) {
+		i = bpf_atomic_load_reg(insn);
+		if (i >= 0)
+			flow_kind_set(k, i, flow_loaded_kind(env, k, insn,
+						insn->imm == BPF_LOAD_ACQ ? src : dst));
+		s = flow_mem_place(insn, dst);
+		if (class == BPF_STX && wide && s >= 0)
+			flow_kind_copy(k, s, k, src);
+		else if (s >= 0)
+			flow_kind_set(k, s, FLOW_IS_PLAIN);
+	} else if (bpf_is_ldimm64(insn)) {
+		if (src == BPF_PSEUDO_MAP_VALUE || src == BPF_PSEUDO_MAP_IDX_VALUE)
+			flow_kind_set(k, dst, flow_map_value_kind(
+					env->used_maps[env->insn_aux_data[idx].map_index]));
+		else
+			flow_kind_set(k, dst, src ? FLOW_IS_OTHER : FLOW_IS_PLAIN);
+	} else if (insn->code == (BPF_JMP | BPF_EXIT) && !bpf_is_unwind_exit(insn)) {
+		s = bpf_find_containing_subprog(env, idx) - sub;
+		lost = ret[s] & ~k->is[FLOW_IS_PLAIN][0];
+		ret[s] &= k->is[FLOW_IS_PLAIN][0];
+	} else if (class == BPF_LD || (class == BPF_JMP && op == BPF_CALL)) {
+		s = bpf_pseudo_call(insn) ? bpf_find_subprog(env, idx + insn->imm + 1) : -1;
+		if (s >= 0 && !bpf_subprog_is_global(env, s)) {
+			lost = flow_kinds_enter(&kinds[sub[s].start], k);
+			regs = ret[s];
+		} else if (bpf_is_callx(insn)) {
+			regs = 0;
+		}
+		/* a subprog may return a pair in R0 and R2, the rest is not set */
+		regs |= ~(BIT(BPF_REG_0) | BIT(BPF_REG_2));
+		for (i = BPF_REG_0; i <= BPF_REG_5; i++)
+			flow_kind_set(k, i, regs & BIT(i) ? FLOW_IS_PLAIN : FLOW_IS_OTHER);
+		for (i = 0; i < MAX_STACK_ARG_SLOTS; i++)
+			flow_kind_set(k, FLOW_ARG_OUT + i, FLOW_IS_PLAIN);
+		if (bpf_helper_call(insn) || bpf_pseudo_kfunc_call(insn))
+			flow_kind_set(k, BPF_REG_0, flow_returned_kind(env, idx));
+	}
+	return lost;
+}
+
+/*
+ * Find the registers that hold a number or a pointer to arena, and those that point to the value
+ * of a map that holds nothing but numbers at worst, whichever way the prog gets to an insn:
+ * plain_regs and value_regs of struct bpf_insn_aux_data. The others may hold any pointer.
+ *
+ * Forwards, from "all is known" down, until nothing changes. Arguments go from the callers to
+ * the callees and results the other way, so the subprogs are looked at in both orders in turn.
+ *
+ * Nothing depends on it but which states are kept apart. A pointer that is taken for
+ * a number costs a rule. A number that is taken for a pointer costs states. So it's fine to
+ * miss a store through a pointer to the stack, and to take what callx calls for unknown.
+ *
+ * The edges that the verifier adds, see bpf_head_guard_target(): may_goto at the head of a loop
+ * takes what holds there to its target. A subprog that is out of budget goes to an exit with 0
+ * in place of the result, a number.
+ */
+int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *sub = env->subprog_info;
+	int len = env->prog->len, cnt = env->subprog_cnt, i, j, n, s, t, idx, err = 0;
+	struct flow_kinds *kinds, k = {};
+	struct bpf_iarray *succ;
+	bool lost = true;
+	u16 *ret;
+
+	kinds = kvmalloc_array(len, sizeof(*kinds), GFP_KERNEL_ACCOUNT);
+	ret = kvmalloc_array(env->subprog_cnt, sizeof(*ret), GFP_KERNEL_ACCOUNT);
+	if (!kinds || !ret) {
+		kvfree(kinds);
+		kvfree(ret);
+		return -ENOMEM;
+	}
+	memset(kinds, 0xff, len * sizeof(*kinds));
+	memset(ret, 0xff, env->subprog_cnt * sizeof(*ret));
+
+	/* what the kernel, a helper, callx or another prog calls: only the context is known */
+	for (s = 0; s < env->subprog_cnt; s++) {
+		if (s && !bpf_subprog_is_global(env, s) && !sub[s].addr_taken)
+			continue;
+		flow_kinds_enter(&kinds[sub[s].start], &k);
+		for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++)
+			if (sub[s].args[i].arg_type == ARG_PTR_TO_CTX ||
+			    (!s && !i && env->prog->type != BPF_PROG_TYPE_EXT))
+				flow_kind_set(&kinds[sub[s].start], BPF_REG_1 + i, FLOW_IS_CTX);
+	}
+
+	for (n = 0; lost; n++) {
+		lost = false;
+		/* a chain of calls takes a round for each link */
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		for (j = 0; j < cnt; j++) {
+			/* it has callees first */
+			s = env->subprog_topo_order[n & 1 ? j : cnt - 1 - j];
+			for (i = sub[s + 1].postorder_start; i-- > sub[s].postorder_start;) {
+				idx = env->cfg.insn_postorder[i];
+				k = kinds[idx];
+				lost |= flow_kinds_step(env, kinds, ret, &k, idx);
+				succ = bpf_insn_successors(env, idx);
+				iarray_for_each(t, succ)
+					lost |= flow_kinds_meet(&kinds[t], &k);
+				t = bpf_head_guard_target(env, idx);
+				if (t >= 0)
+					lost |= flow_kinds_meet(&kinds[t], &kinds[idx]);
+			}
+		}
+	}
+	for (i = 0; i < len; i++) {
+		env->insn_aux_data[i].plain_regs = kinds[i].is[FLOW_IS_PLAIN][0];
+		env->insn_aux_data[i].value_regs = kinds[i].is[FLOW_IS_VALUE][0];
+	}
+	kvfree(kinds);
+	kvfree(ret);
+	return err;
+}
-- 
2.55.0

[PATCH bpf-next v3 10/45] bpf: Find the registers and stack slots that later checks depend on

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The join of two constants is not a constant, and some checks need one. The
two states meet before the insn that checks. Thus the verifier must know,
where they meet, that a later check depends on a register or a stack slot.
A table per kind of check, FLOW_NEED_*, tells the rules that refuse a join:

ARG        an argument that must be a constant
    if (x) return bpf_rdonly_cast(0, ID); return bpf_rdonly_cast(0, 0);
  The compiler makes one call of it: "R2 must be a known constant" in
  selftest mem_rdonly_untrusted/mixed_mem_type.
STACK_OFF  an add to a pointer to the stack
    if (... || depth >= BT_MAXLVL_PRINT) return 0; ind = stack[depth];
  The compiler tests one copy of depth and adds another. bt_print() of scx:
  "invalid variable-offset read from stack R3"
PTR_OFF    an add to a pointer to the packet, the context, a kernel object,
           a map value that is read-only or has a timer, a lock, a pointer
    if (data + hdr_len + inner_ip_offset > data_end) return TC_ACT_UNSPEC;
    inner_protocol = inner_iph->protocol;
  With [20, 40] as the offset the test tells nothing about data + hdr_len:
  "invalid access to packet, off=48 size=1" in katran/tcdecapinfo.
JUMP       a jump that compares with a constant, an argument or a return
           value that the verifier compares with a range
    if (ret < 0) ..; to_sport = state->to_sport;
  ret is a negative constant where state can be NULL. As a part of a range
  it passes the test: "R3 invalid mem access 'map_value_or_null'" in
  tail_handle_snat_fwd_ipv4 of Cilium.

Three more tables are only for bpf_flow_is_needed(): does a check depend
on a value that a counted loop changes and that has no closed form? If not,
the value is an unknown scalar in the loop, and the verifier has no reason
to walk the iterations one by one:

    for (int i = 0; i < STACK_MAX_LEN; ++i) {
      if (!frame_ptr || !get_frame_data(frame_ptr, pidData, &frame, &sym))

Each iteration fills the 24 stack slots of sym anew, and no check looks at
them. pyperf600_nounroll takes 1241 insns, 154K with a walk of all 600.

VALUE_OFF  PTR_OFF, STACK_OFF, and an offset in each other map value
BASE       the address of a load or a store
JUMP_ALU   JUMP, and the values that the tested value is calculated from

flow_compute_need_const() goes backward through the control flow graph
until nothing changes, like the calculation of live registers. A use that
it misses can reject a valid prog. It cannot accept an unsafe prog.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 324 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 324 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 837d5054c2a3..1eecd9ee5123 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -515,6 +515,74 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)
 	return ret;
 }
 
+/* The slot that @insn spills to or fills from, if it's one that struct flow_need has a bit for */
+static int flow_fp_slot(const struct bpf_insn *insn, u8 base)
+{
+	u8 mode = BPF_MODE(insn->code);
+
+	if (base != BPF_REG_FP || (mode != BPF_MEM && mode != BPF_MEMSX) ||
+	    insn->off >= 0 || insn->off < -64 * BPF_REG_SIZE || insn->off % BPF_REG_SIZE)
+		return -1;
+	return -insn->off / BPF_REG_SIZE - 1;
+}
+
+/*
+ * A number is added to a pointer to the stack. A store at an offset that is not known
+ * forgets what the slots it may hit hold, pointers above all.
+ */
+static u16 flow_stack_off_regs(struct bpf_verifier_env *env, int idx)
+{
+	u16 ptrs = env->insn_aux_data[idx].stack_ptrs;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 op = BPF_OP(insn->code);
+
+	if (BPF_CLASS(insn->code) != BPF_ALU64 || BPF_SRC(insn->code) != BPF_X ||
+	    (op != BPF_ADD && op != BPF_SUB))
+		return 0;
+	if ((ptrs & BIT(insn->dst_reg)) && !(ptrs & BIT(insn->src_reg)))
+		return BIT(insn->src_reg);
+	if (op == BPF_ADD && (ptrs & BIT(insn->src_reg)) && !(ptrs & BIT(insn->dst_reg)))
+		return BIT(insn->dst_reg);
+	return 0;
+}
+
+/*
+ * A jump that compares a register with a constant, which may be in the other register.
+ * A call does the same: the verifier compares a size, or what bpf_set_retval() takes,
+ * with what is allowed. So does the exit of the prog, see flow_compute_need_const().
+ */
+static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	u16 consts = aux->const_reg_mask | aux->const_reg_wide_mask;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+	if (bpf_helper_call(insn) || bpf_pseudo_kfunc_call(insn))
+		return bpf_call_checked_arg_regs(env, insn);
+	if ((class != BPF_JMP && class != BPF_JMP32) ||
+	    op == BPF_JA || op == BPF_CALL || op == BPF_EXIT || op == BPF_JCOND)
+		return 0;
+	if (BPF_SRC(insn->code) == BPF_K || (consts & BIT(insn->src_reg)))
+		return BIT(insn->dst_reg);
+	return consts & BIT(insn->dst_reg) ? BIT(insn->src_reg) : 0;
+}
+
+/* With or without sign extension */
+static bool flow_is_copy(const struct bpf_insn *insn)
+{
+	return BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_X && !insn->imm &&
+	       (!insn->off || insn->off == 8 || insn->off == 16 || insn->off == 32);
+}
+
+static bool flow_is_shift(const struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+	return (class == BPF_ALU || class == BPF_ALU64) &&
+	       (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH);
+}
+
 /*
  * The places of a function that hold a value: the registers, the arguments on the stack that it
  * got and those that it hands over, the slots of the stack that struct flow_need has a bit for.
@@ -821,3 +889,259 @@ int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
 	kvfree(ret);
 	return err;
 }
+
+/*
+ * A number is added to what may be a pointer whose offset the verifier looks at:
+ * - An object of the kernel, the context, a socket take a known offset only.
+ * - The packet: the offset was compared with the end of the packet before the states met.
+ * - The value of a map that has a timer, a lock, a pointer in the middle.
+ * - The value of a read-only map: what is loaded from a known offset is known.
+ * Not a value that holds nothing but numbers. Only the lowest and the highest offset
+ * count there, and the join of two constants has both. @any_value: that one too.
+ */
+static u16 flow_ptr_off_regs(struct bpf_verifier_env *env, int idx, bool any_value)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u16 both = BIT(insn->dst_reg) | BIT(insn->src_reg), ptrs;
+	u8 op = BPF_OP(insn->code);
+
+	if (BPF_CLASS(insn->code) != BPF_ALU64 || BPF_SRC(insn->code) != BPF_X ||
+	    (op != BPF_ADD && op != BPF_SUB) || (aux->stack_ptrs & both))
+		return 0;
+	ptrs = both & ~(any_value ? aux->plain_regs : aux->value_regs);
+	if (op == BPF_SUB)
+		ptrs &= BIT(insn->dst_reg);
+	/* the one that may be the pointer makes the other one the number */
+	return ptrs && ptrs != both ? both ^ ptrs : ptrs;
+}
+
+/* The base of a load or a store, unless it's a number or a pointer to arena for sure */
+static u16 flow_base_regs(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code);
+	u16 regs = 0;
+
+	if (class == BPF_LDX || bpf_atomic_is_load_acq(insn))
+		regs = BIT(insn->src_reg);
+	else if (class == BPF_ST || class == BPF_STX)
+		regs = BIT(insn->dst_reg);
+	return regs & ~env->insn_aux_data[idx].plain_regs & GENMASK(BPF_REG_9, BPF_REG_0);
+}
+
+/*
+ * Some arguments of kfuncs and helpers have to be known constants: the id of a type,
+ * the size of what is reserved in a ring buffer, the size of a slice of a dynptr.
+ * A compiler makes one call of
+ *	if (x) p = bpf_rdonly_cast(q, ID); else p = bpf_rdonly_cast(q, 0);
+ * and one store of
+ *	if (x) a[n++] = p; else b[m++] = p;
+ * where a and b are on the stack.
+ * and one load of
+ *	x = ipv6 ? *(u16 *)(pkt + 54) : *(u16 *)(pkt + 34);	y = z ? task->pid : task->tgid;
+ * The join of two constants is not a constant. Find the registers and the slots of the stack
+ * that end up in such arguments, or in offsets in the stack or from a pointer,
+ * like live registers are found: backwards, until nothing changes.
+ * States in which they differ are not joined.
+ * Nothing depends on it but what is known: it's fine to miss a store through a pointer.
+ *
+ * FLOW_NEED_JUMP: those that a jump ahead compares with a constant, see flow_join_makes_up().
+ * As they are, or extended: without 32-bit insns a compiler tests
+ *	r7 = r8; r7 <<= 32; r7 >>= 32;
+ * and goes on with r8.
+ * A slot that is filled through another pointer to the stack than r10 is not one of them, and
+ * flow_null_differs() counts on that.
+ *
+ * The last three kinds keep no states apart. They are for bpf_flow_is_needed(), which asks
+ * whether anything ahead looks at a number that a loop changes from iteration to iteration.
+ * FLOW_NEED_VALUE_OFF: FLOW_NEED_PTR_OFF, and the offset in a map value that holds nothing but
+ * numbers. It has to be in the value. It can be an argument, or what a function returns:
+ *	return v->hist[count_bits(x)];
+ * So can an offset in the stack. FLOW_NEED_STACK_OFF doesn't follow it through a return, so this
+ * table has it too.
+ * FLOW_NEED_BASE: the base of a load or a store, as it is, behind a cast, moved by a constant:
+ *	p = NULL; for (i = 0; i < 3; i++) { if (p) n += p->len; p = next(); }
+ * FLOW_NEED_JUMP_ALU: FLOW_NEED_JUMP, and what the tested number is made of step by step:
+ *	for (i = 0; i < 3; i++) x /= 10; if (x < 8) ...
+ * In x op= y that is x. y is not followed.
+ * What is stored through a pointer to the stack other than r10 counts as tested. It's a result
+ * for the caller, and no table follows it from the store to the fill:
+ *	count_bits(x, &n); return v->hist[n];
+ * These two go through callx, to every function whose address is taken.
+ */
+static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_need **pneed,
+				   enum flow_need_kind kind)
+{
+	struct bpf_insn *insns = env->prog->insnsi, *insn;
+	struct flow_need in, out, *need;
+	struct bpf_iarray *succ;
+	bool jump = kind == FLOW_NEED_JUMP || kind == FLOW_NEED_JUMP_ALU;
+	bool callx = kind == FLOW_NEED_BASE || kind == FLOW_NEED_JUMP_ALU;
+	bool changed = true, any = false, keep, from_src, reg;
+	int i, s, idx, slot, err = 0;
+	u16 *gen, callx_regs = 0, pair = BIT(BPF_REG_0) | BIT(BPF_REG_2);
+	u8 *ret, class, op;
+
+	need = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
+	gen = kvcalloc(env->prog->len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+	/* the callers need what the subprog returns: r0, and r2 of a pair */
+	ret = kvcalloc(env->subprog_cnt, sizeof(u8), GFP_KERNEL_ACCOUNT);
+	*pneed = need;
+	if (!need || !gen || !ret) {
+		kvfree(gen);
+		kvfree(ret);
+		return -ENOMEM;
+	}
+	for (i = 0; i < env->prog->len; i++) {
+		if (jump)
+			gen[i] = flow_tested_regs(env, i);
+		else if (kind == FLOW_NEED_STACK_OFF)
+			gen[i] = flow_stack_off_regs(env, i);
+		else if (kind == FLOW_NEED_PTR_OFF || kind == FLOW_NEED_VALUE_OFF)
+			gen[i] = flow_ptr_off_regs(env, i, kind == FLOW_NEED_VALUE_OFF);
+		else if (kind == FLOW_NEED_BASE)
+			gen[i] = flow_base_regs(env, i);
+		else if (bpf_helper_call(&insns[i]) || bpf_pseudo_kfunc_call(&insns[i]))
+			gen[i] = bpf_call_const_arg_regs(env, &insns[i]);
+		if (kind == FLOW_NEED_JUMP_ALU && BPF_CLASS(insns[i].code) == BPF_STX &&
+		    insns[i].dst_reg != BPF_REG_FP &&
+		    (env->insn_aux_data[i].stack_ptrs & BIT(insns[i].dst_reg)))
+			gen[i] = BIT(insns[i].src_reg);
+		if (kind == FLOW_NEED_VALUE_OFF)
+			gen[i] |= flow_stack_off_regs(env, i);
+		any |= gen[i];
+	}
+	if (jump) {
+		/* the verifier compares what the prog and its callbacks return with a range */
+		ret[0] = BIT(BPF_REG_0);
+		for (i = 0; i < env->prog->len; i++) {
+			if (!bpf_pseudo_func(&insns[i]))
+				continue;
+			s = bpf_find_subprog(env, i + insns[i].imm + 1);
+			if (s >= 0)
+				ret[s] = BIT(BPF_REG_0);
+		}
+		any = true;
+	}
+	while (any && changed) {
+		changed = false;
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		for (s = 0; callx && s < env->subprog_cnt; s++)
+			if (env->subprog_info[s].addr_taken)
+				callx_regs |= need[env->subprog_info[s].start].regs;
+		for (i = 0; i < env->cfg.cur_postorder; i++) {
+			idx = env->cfg.insn_postorder[i];
+			insn = &insns[idx];
+			class = BPF_CLASS(insn->code);
+			succ = bpf_insn_successors(env, idx);
+			out.regs = 0;
+			out.slots = 0;
+			for (s = 0; s < succ->cnt; s++) {
+				out.regs |= need[succ->items[s]].regs;
+				out.slots |= need[succ->items[s]].slots;
+			}
+			in = out;
+			if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) {
+				in.regs &= ~GENMASK(BPF_REG_5, BPF_REG_0);
+				if (bpf_is_callx(insn))
+					in.regs |= callx_regs & GENMASK(BPF_REG_5, BPF_REG_1);
+				s = idx + insn->imm + 1;
+				if (bpf_pseudo_call(insn) &&
+				    !bpf_subprog_is_global(env, bpf_find_subprog(env, s))) {
+					if (kind != FLOW_NEED_PTR_OFF)
+						in.regs |= need[s].regs &
+							   GENMASK(BPF_REG_5, BPF_REG_1);
+					s = bpf_find_subprog(env, s);
+					/*
+					 * Not for an index where states are kept apart: that is
+					 * one of many numbers.
+					 */
+					if ((kind == FLOW_NEED_ARG || kind == FLOW_NEED_VALUE_OFF ||
+					     jump) &&
+					    (out.regs & pair & ~ret[s])) {
+						ret[s] |= out.regs & pair;
+						changed = true;
+					}
+				}
+			} else if (insn->code == (BPF_JMP | BPF_EXIT)) {
+				s = bpf_find_containing_subprog(env, idx) - env->subprog_info;
+				in.regs |= ret[s];
+			} else if (class == BPF_LDX || class == BPF_LD) {
+				slot = class == BPF_LDX ? flow_fp_slot(insn, insn->src_reg) : -1;
+				if (slot >= 0 && (out.regs & BIT(insn->dst_reg)))
+					in.slots |= BIT_ULL(slot);
+				in.regs &= ~BIT(insn->dst_reg);
+			} else if (class == BPF_STX || class == BPF_ST) {
+				slot = flow_fp_slot(insn, insn->dst_reg);
+				if (slot >= 0 && (out.slots & BIT_ULL(slot))) {
+					in.slots &= ~BIT_ULL(slot);
+					if (class == BPF_STX)
+						in.regs |= BIT(insn->src_reg);
+				}
+			} else if ((class == BPF_ALU || class == BPF_ALU64) &&
+				   (out.regs & BIT(insn->dst_reg))) {
+				op = BPF_OP(insn->code);
+				reg = BPF_SRC(insn->code) == BPF_X;
+				/* the result is known when what it is made of is */
+				keep = op != BPF_MOV;
+				from_src = reg;
+				if (jump)
+					from_src = flow_is_copy(insn);
+				if (kind == FLOW_NEED_JUMP)
+					keep = flow_is_shift(insn) && !reg;
+				if (kind == FLOW_NEED_BASE) {
+					keep = !reg && (op == BPF_ADD || op == BPF_SUB);
+					from_src = reg && op == BPF_MOV;
+				}
+				if (!keep)
+					in.regs &= ~BIT(insn->dst_reg);
+				if (from_src)
+					in.regs |= BIT(insn->src_reg);
+			}
+			in.regs |= gen[idx];
+			if (in.regs != need[idx].regs || in.slots != need[idx].slots) {
+				need[idx] = in;
+				changed = true;
+			}
+		}
+	}
+	/*
+	 * An offset that is an argument: not through the call, see above, and not where the calls
+	 * meet. A function would be looked at once for each offset that it's called with.
+	 */
+	for (s = 0; kind == FLOW_NEED_PTR_OFF && s < env->subprog_cnt; s++)
+		need[env->subprog_info[s].start].regs = 0;
+	kvfree(gen);
+	kvfree(ret);
+	if (!any) {
+		kvfree(need);
+		*pneed = NULL;
+	}
+	return err;
+}
+
+/*
+ * Does anything ahead of the insn at @idx look at what @r holds there? @r is a register or a stack
+ * slot of the function as scev.c counts them. See flow_compute_need_const(): it's a guess.
+ */
+bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r)
+{
+	const struct flow_need *need;
+	int i;
+
+	/* no table has a bit for it */
+	if (r >= MAX_BPF_REG + 64)
+		return true;
+	for (i = 0; i < FLOW_NEED_KINDS; i++) {
+		need = env->flow->need[i];
+		if (need && (r < MAX_BPF_REG ? need[idx].regs & BIT(r) :
+					       need[idx].slots & BIT_ULL(r - MAX_BPF_REG)))
+			return true;
+	}
+	return false;
+}
-- 
2.55.0

Re: [PATCH bpf-next v3 10/45] bpf: Find the registers and stack slots that later checks depend on

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: false R0 dependency for BPF_END | BPF_TO_BE in flow_compute_need_const()
--

commit 6774fce732def6414deae7e4151e138be57cfe46
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Find the registers and stack slots that later checks depend on

This commit adds backward dataflow analysis in kernel/bpf/flow.c to identify
registers and stack slots that subsequent verifier checks depend on, used both
to prevent joining states with differing constants and to determine whether
loop-modified values are needed after a counted loop.

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> index 837d5054c2a3d..1eecd9ee5123b 100644
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c

[ ... ]

> @@ -821,3 +889,259 @@ int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
>  	kvfree(ret);
>  	return err;
>  }
> +

[ ... ]

> +static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_need **pneed,
> +				   enum flow_need_kind kind)
> +{

[ ... ]

> +			} else if ((class == BPF_ALU || class == BPF_ALU64) &&
> +				   (out.regs & BIT(insn->dst_reg))) {
> +				op = BPF_OP(insn->code);
> +				reg = BPF_SRC(insn->code) == BPF_X;
> +				/* the result is known when what it is made of is */
> +				keep = op != BPF_MOV;
> +				from_src = reg;
> +				if (jump)
> +					from_src = flow_is_copy(insn);
> +				if (kind == FLOW_NEED_JUMP)
> +					keep = flow_is_shift(insn) && !reg;
> +				if (kind == FLOW_NEED_BASE) {
> +					keep = !reg && (op == BPF_ADD || op == BPF_SUB);
> +					from_src = reg && op == BPF_MOV;
> +				}
> +				if (!keep)
> +					in.regs &= ~BIT(insn->dst_reg);
> +				if (from_src)
> +					in.regs |= BIT(insn->src_reg);

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], in flow_compute_need_const()
in kernel/bpf/flow.c, does a BPF_ALU | BPF_END | BPF_TO_BE instruction falsely
mark BPF_REG_0 as needed in in.regs?

Because BPF_TO_BE (0x08) shares the same bit value as BPF_X (0x08) in
BPF_SRC(insn->code), reg evaluates to true on a big-endian byte-swap
instruction even though BPF_END is a unary in-place operation on dst_reg with
insn->src_reg == BPF_REG_0.

For FLOW_NEED_ARG, FLOW_NEED_STACK_OFF, FLOW_NEED_PTR_OFF, and
FLOW_NEED_VALUE_OFF, from_src remains equal to reg, so:

	if (from_src)
		in.regs |= BIT(insn->src_reg);

sets BIT(BPF_REG_0) whenever the byte-swapped dst_reg is needed downstream.
Wouldn't this prevent joining states where R0 holds different constants prior
to a byte-swap and cause bpf_flow_is_needed() to report R0 as needed at loop
headers?

Should reg exclude op == BPF_END (for example, reg = BPF_SRC(insn->code) ==
BPF_X && op != BPF_END)?

[1] https://lore.kernel.org/r/506d7ef52474ac7df140bd5f84ac7b7bcf359769a1a46b05aba0751fcdf065af@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=10

[PATCH bpf-next v3 11/45] bpf: Add the sets of states at an insn and the entries of functions

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest test_verif_scale1 has these lines 90 times. jhash() is static:

  ptr = data + i;
  if (ptr + nh_off > data_end) break;
  ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++);

Today the verifier verifies jhash() again for each state at each call:
530362 insns. The main pass verifies a static subprog for the states at its
first insn, and the call waits for the states at the exits. Each call that
waits makes an entry. Its number is in the entry state and in all states
that come from it. 4095 states arrive at the 90 calls. At the first insn of
jhash() a state of a different call covers 4005 of them. Then each exit
state is for the calls of the two:

  call A  -->  entry 1  --+
                          +-->  one set: calls { A, B }, exit states
  call B  -->  entry 2  --+

Thus the entries are disjoint sets, with union and find: 50765 insns. If
they stay apart, the main pass hits its limit of states at an insn:
"more than 1024 states that cannot be joined at insn 903". The same for
pandemonium_enqueue of scx_pandemonium and 100 of the 225 progs of the
schedulers of rust-bpf.

All new states are in one tree. flow_set_pending() gives them an order:
 1. Callers are before the subprogs that they call: all calls get to
    a subprog before it is verified.
 2. Then the position of the call in each frame, then of the insn. Thus
    a callee that keeps the frames of its callers, like a callback, is
    complete before the code after the call.
 3. A walk verifies a loop iteration by iteration, as today. An iteration
    is complete before the next one:

  for (int i = 0; i < MAX_AFFINITY_CANDIDATES; i++) {
      if (*val >= nr_cpu_ids) continue;
      if (++checked >= affinity_search_online) break;

'continue' is a back edge in the middle of the loop. By position alone it
runs ahead through all iterations, and each is done again when the rest of
the one before it arrives: pandemonium_select_cpu, 130073 insns, not 25589.

bpf_flow_prepare() marks the insns where a state goes into a set, and those
where a second state can arrive. Only there the set keeps a state that the
verifier used: to keep them all doubles the peak of states of Cilium, scx.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 324 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 324 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 1eecd9ee5123..8daee1f22461 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1145,3 +1145,327 @@ bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r)
 	}
 	return false;
 }
+
+int bpf_flow_prepare(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_insn *insns = env->prog->insnsi;
+	/* the rest is for loops */
+	int len = env->prog->len, kinds = FLOW_NEED_VALUE_OFF, i, s, err;
+	struct bpf_iarray *succ;
+	struct bpf_flow *flow;
+	u8 *preds;
+
+	flow = kzalloc_obj(*flow, GFP_KERNEL_ACCOUNT);
+	if (!flow)
+		return -ENOMEM;
+	env->flow = flow;
+	INIT_LIST_HEAD(&flow->kept);
+	flow->pending = RB_ROOT_CACHED;
+
+	flow->pos = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	flow->rank = kvcalloc(env->subprog_cnt + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	flow->point = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+	flow->keep = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+	flow->states = kvmalloc_array(len, sizeof(*flow->states), GFP_KERNEL_ACCOUNT);
+	flow->enter = kvmalloc_array(len, sizeof(*flow->enter), GFP_KERNEL_ACCOUNT);
+	flow->nr_states = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	flow->funcs = kvcalloc(env->subprog_cnt + 1, sizeof(*flow->funcs), GFP_KERNEL_ACCOUNT);
+	preds = kvcalloc(len, 1, GFP_KERNEL_ACCOUNT);
+	if (!flow->pos || !flow->rank || !flow->point || !flow->keep || !flow->states ||
+	    !flow->enter ||
+	    !flow->nr_states || !flow->funcs || !preds) {
+		kvfree(preds);
+		return -ENOMEM;
+	}
+	for (i = 0; i < len; i++) {
+		INIT_LIST_HEAD(&flow->states[i].list);
+		INIT_LIST_HEAD(&flow->enter[i].list);
+	}
+	for (i = 0; i <= env->subprog_cnt; i++)
+		flow->funcs[i].exit_idx = -1;
+	/* from here on there is something for bpf_flow_reset() to look at */
+	flow->len = len;
+	flow->nr_funcs = env->subprog_cnt + 1;
+
+	/* subprog_topo_order[] has callees first */
+	for (i = 0; i < env->subprog_cnt; i++)
+		flow->rank[env->subprog_topo_order[i]] = env->subprog_cnt - i;
+
+	for (i = 0; i < len; i++) {
+		succ = bpf_insn_successors(env, i);
+		iarray_for_each(s, succ) {
+			if (preds[s] < 2)
+				preds[s]++;
+			if (aux[i].bb_end)
+				__set_bit(s, flow->point);
+		}
+		/* where the callees come back to */
+		if (bpf_pseudo_call(&insns[i]) || bpf_is_callx(&insns[i])) {
+			if (i + 1 < len) {
+				__set_bit(i + 1, flow->point);
+				__set_bit(i + 1, flow->keep);
+			}
+		}
+		if (aux[i].calls_callback) {
+			__set_bit(i, flow->point);
+			__set_bit(i, flow->keep);
+		}
+		if (aux[i].loop) {
+			__set_bit(i, flow->point);
+			__set_bit(i, flow->keep);
+			kinds = FLOW_NEED_KINDS;
+		}
+		if (bpf_is_ldimm64(&insns[i]))
+			i++;
+	}
+	for (i = 0; i < len; i++)
+		if (preds[i] > 1)
+			__set_bit(i, flow->keep);
+	for (i = 0; i < env->subprog_cnt; i++) {
+		__set_bit(env->subprog_info[i].start, flow->point);
+		__set_bit(env->subprog_info[i].start, flow->keep);
+	}
+	kvfree(preds);
+
+	err = flow_compute_order(env, flow);
+	if (err)
+		return err;
+	for (i = 0; !err && i < kinds; i++)
+		err = flow_compute_need_const(env, &flow->need[i], i);
+	return err;
+}
+
+static void flow_free_state(struct bpf_verifier_env *env, struct flow_state *s)
+{
+	bpf_free_verifier_state(&s->st, false);
+	kfree(s);
+	env->flow->live_states--;
+}
+
+/* What a pass leaves behind. */
+void bpf_flow_reset(struct bpf_verifier_env *env)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *s, *sn;
+	struct flow_kept *k, *kn;
+	struct flow_exit *x, *xn;
+	struct flow_entry *e;
+	int i;
+
+	if (!flow || !flow->states || !flow->enter)
+		return;
+	for (i = 0; i < flow->len; i++) {
+		list_for_each_entry_safe(s, sn, &flow->states[i].list, node) {
+			list_del(&s->node);
+			flow_free_state(env, s);
+		}
+		list_for_each_entry_safe(s, sn, &flow->enter[i].list, node) {
+			list_del(&s->node);
+			flow_free_state(env, s);
+		}
+		flow->nr_states[i] = 0;
+	}
+	flow->pending = RB_ROOT_CACHED;
+	for (i = 0; i < flow->nr_funcs && flow->funcs; i++)
+		flow->funcs[i].exit_idx = -1;
+	for (i = 1; i <= flow->entry_cnt; i++) {
+		e = flow->entries[i];
+		list_for_each_entry_safe(s, sn, &e->calls, node) {
+			bpf_free_verifier_state(&s->st, false);
+			kfree(s);
+		}
+		list_for_each_entry_safe(x, xn, &e->exits, node) {
+			bpf_free_verifier_state(&x->st, false);
+			kfree(x);
+		}
+		kfree(e);
+	}
+	list_for_each_entry_safe(k, kn, &flow->kept, node) {
+		bpf_free_verifier_state(&k->st, false);
+		kfree(k);
+	}
+	INIT_LIST_HEAD(&flow->kept);
+	bpf_free_verifier_state(&flow->tmp, false);
+	memset(&flow->tmp, 0, sizeof(flow->tmp));
+	flow->entry_cnt = 0;
+	flow->united_cnt = 0;
+}
+
+void bpf_flow_free(struct bpf_verifier_env *env)
+{
+	struct bpf_flow *flow = env->flow;
+	int i;
+
+	if (!flow)
+		return;
+	bpf_flow_reset(env);
+	kvfree(flow->pos);
+	kvfree(flow->rank);
+	for (i = 0; i < FLOW_NEED_KINDS; i++)
+		kvfree(flow->need[i]);
+	kvfree(flow->point);
+	kvfree(flow->keep);
+	kvfree(flow->states);
+	kvfree(flow->enter);
+	kvfree(flow->nr_states);
+	kvfree(flow->funcs);
+	kvfree(flow->entries);
+	kvfree(flow->united);
+	kfree(flow);
+	env->flow = NULL;
+}
+
+bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx)
+{
+	return test_bit(insn_idx, env->flow->point);
+}
+
+/* A copy of @st that lives as long as the pass. */
+struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
+					       const struct bpf_verifier_state *st)
+{
+	struct flow_kept *k = kzalloc_obj(*k, GFP_KERNEL_ACCOUNT);
+
+	if (!k)
+		return NULL;
+	list_add(&k->node, &env->flow->kept);
+	if (bpf_copy_verifier_state(&k->st, st))
+		return NULL;
+	return &k->st;
+}
+
+/*
+ * Both are in a loop whose iterations are kept apart: an iteration is done with before
+ * the next one begins. A back edge in the middle of the loop would run ahead otherwise and
+ * leave the rest of every iteration behind, see flow_earlier_trip(). The state that stands
+ * for all iterations comes last.
+ */
+static int flow_cmp_trips(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
+{
+	const struct loop_stack_entry *ea, *eb;
+	u32 i, ta, tb;
+
+	for (i = 0; i < fa->loop_stack_cnt && i < fb->loop_stack_cnt; i++) {
+		ea = &fa->loop_stack[i];
+		eb = &fb->loop_stack[i];
+		if (ea->loop_id != eb->loop_id)
+			break;
+		ta = ea->apart ? ea->trip : U32_MAX;
+		tb = eb->apart ? eb->trip : U32_MAX;
+		if (ta != tb)
+			return ta < tb ? -1 : 1;
+	}
+	return 0;
+}
+
+static bool flow_less(struct rb_node *na, const struct rb_node *nb)
+{
+	const struct flow_state *a = rb_entry(na, struct flow_state, rb);
+	const struct flow_state *b = rb_entry(nb, struct flow_state, rb);
+	int cmp;
+	u32 i;
+
+	for (i = 0; i < a->key_len && i < b->key_len; i++) {
+		/* key[i] is the place in frame[i - 1] */
+		if (i) {
+			cmp = flow_cmp_trips(a->st.frame[i - 1], b->st.frame[i - 1]);
+			if (cmp)
+				return cmp < 0;
+		}
+		if (a->key[i] != b->key[i])
+			return a->key[i] < b->key[i];
+	}
+	if (a->key_len != b->key_len)
+		return a->key_len < b->key_len;
+	return a->seq < b->seq;
+}
+
+/*
+ * Callers first, so that the calls of a function get to it before it is looked
+ * at. Then by the place of the call in every frame: what is called in the frames of
+ * its callers is done with before what follows the call. What follows the call of
+ * a function that is verified on its own goes on with every state that comes out.
+ */
+static void flow_set_pending(struct bpf_flow *flow, struct flow_state *s)
+{
+	const struct bpf_verifier_state *st = &s->st;
+	u32 i;
+
+	s->key[0] = flow->rank[st->frame[0]->subprogno];
+	/* the place in the function makes the rank of the function redundant below */
+	for (i = 0; i < st->curframe && i + 2 < ARRAY_SIZE(s->key); i++)
+		s->key[i + 1] = flow->pos[st->frame[i + 1]->callsite];
+	s->key[i + 1] = flow->pos[st->insn_idx];
+	s->key_len = i + 2;
+	s->pending = true;
+	rb_add_cached(&s->rb, &flow->pending, flow_less);
+}
+
+static void flow_unlink(struct bpf_verifier_env *env, struct flow_state *s)
+{
+	struct bpf_flow *flow = env->flow;
+
+	if (s->pending)
+		rb_erase_cached(&s->rb, &flow->pending);
+	s->pending = false;
+	list_del(&s->node);
+	flow->nr_states[s->st.insn_idx]--;
+}
+
+/* The entry that stands for the set that entry @id is in */
+static struct flow_entry *flow_entry_find(struct bpf_flow *flow, u32 id)
+{
+	struct flow_entry *e = flow->entries[id];
+
+	while (e->up != e) {
+		e->up = e->up->up;
+		e = e->up;
+	}
+	return e;
+}
+
+static int flow_entry_new(struct bpf_flow *flow)
+{
+	struct flow_entry **entries, *e;
+	u32 cap;
+
+	if (flow->entry_cnt + 1 >= flow->entry_cap) {
+		cap = max(64U, flow->entry_cap * 2);
+		entries = kvrealloc(flow->entries, cap * sizeof(*entries), GFP_KERNEL_ACCOUNT);
+		if (!entries)
+			return -ENOMEM;
+		flow->entries = entries;
+		flow->entry_cap = cap;
+	}
+	e = kmalloc_obj(*e, GFP_KERNEL_ACCOUNT);
+	if (!e)
+		return -ENOMEM;
+	e->up = e;
+	INIT_LIST_HEAD(&e->calls);
+	INIT_LIST_HEAD(&e->exits);
+	flow->entries[++flow->entry_cnt] = e;
+	return flow->entry_cnt;
+}
+
+/* It delivers states, so it is done when nothing is in the middle of that. */
+static int flow_unite(struct bpf_flow *flow, u32 a, u32 b)
+{
+	struct flow_united *u;
+	u32 cap;
+
+	if (!a || !b || flow_entry_find(flow, a) == flow_entry_find(flow, b))
+		return 0;
+	if (flow->united_cnt == flow->united_cap) {
+		cap = max(16U, flow->united_cap * 2);
+		u = kvrealloc(flow->united, cap * sizeof(*u), GFP_KERNEL_ACCOUNT);
+		if (!u)
+			return -ENOMEM;
+		flow->united = u;
+		flow->united_cap = cap;
+	}
+	u = &flow->united[flow->united_cnt++];
+	u->a = a;
+	u->b = b;
+	return 0;
+}
-- 
2.55.0

[PATCH bpf-next v3 12/45] bpf: Compare states of different iterations of a loop

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest loop1, nested_loops():

  for (j = 0; j < 300; j++)
      for (i = 0; i < j; i++) {
          ...
          sum += i * m;
      }

Scalar evolution counts both loops, but has no closed form for sum, and the
verifier checks sum: it is the return value. It walks such a loop: it
verifies the iterations one by one, each with its own values, as it does
today. A join of j=5 and j=6 ends that. Thus a state has the number of its
iteration in each loop that is walked, and flow_trips_behind() tells the
merge, the code that puts a state into a set, which states not to join.
But then the states of all iterations stay in the sets, up to the limit:

  more than 1024 states that cannot be joined at insn 10

A walk only goes forward: what follows a state of iteration N is of
iteration N or later. flow_earlier_trip() finds the states that are two or
more iterations behind, and the merge frees them. nested_loops() then has
21 states at a time and takes 9954 insns (today: 5505 and 316200).
lavd_init and qmap_init of rust-bpf fail in the same way.

Crate spin, Mutex::lock():

  while self.is_locked() { R::relax(); }

No iteration changes what the verifier sees: each ends in the state of the
one before, with a larger number. Thus in a walk of a loop without a count
a state of an earlier iteration can cover a later one. flow_trips_near():
the merge drops a state that a state of its own iteration or of the two
before it covers. If only states of one iteration are compared, 164 of the
225 progs of the schedulers of rust-bpf take more insns, 9 % in sum.

The drop closes a cycle: the earlier state leads to the later one and now
stands for it. "No state returns to the head, so the loop ends at run time"
does not hold any more. Thus flow_cycle_closed() gives the loop a may_goto.
It does nothing if the loop is counted, or if may_goto or bpf_iter_*_next()
stops each iteration. It rejects a loop that has no place for may_goto.

flow_within() is 'covers' for the merge: the comparison of pruning, also of
the states that entered each counted loop: its ranges come from them.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 173 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 8daee1f22461..f17eb3467708 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1469,3 +1469,176 @@ static int flow_unite(struct bpf_flow *flow, u32 a, u32 b)
 	u->b = b;
 	return 0;
 }
+
+/*
+ * What is known about a counted loop comes from the state it was entered
+ * with. @t stands for @s only if that holds for these states too.
+ */
+static bool flow_loops_within(struct bpf_verifier_env *env, const struct bpf_verifier_state *t,
+			      const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *lt, *ls;
+	u32 fr, i;
+
+	for (fr = 0; fr <= t->curframe; fr++) {
+		if (t->frame[fr]->loop_stack_cnt != s->frame[fr]->loop_stack_cnt)
+			return false;
+		for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+			lt = &t->frame[fr]->loop_stack[i];
+			ls = &s->frame[fr]->loop_stack[i];
+			if (!lt->terminates || lt->entry_state == ls->entry_state)
+				continue;
+			if (!ls->terminates ||
+			    !bpf_state_subsumes(env, lt->entry_state, ls->entry_state))
+				return false;
+		}
+	}
+	return true;
+}
+
+static bool flow_within(struct bpf_verifier_env *env, struct bpf_verifier_state *t,
+			struct bpf_verifier_state *s)
+{
+	return t->curframe == s->curframe &&
+	       t->frame[0]->subprogno == s->frame[0]->subprogno &&
+	       /* an async callback returns to the kernel, a call of the function to its caller */
+	       !t->flow_entry == !s->flow_entry &&
+	       bpf_state_subsumes(env, t, s) && flow_loops_within(env, t, s);
+}
+
+/*
+ * How many iterations is @t behind @s, in the loop in which that is the most? Only the loops
+ * whose iterations are kept apart are looked at.
+ * 0 if both are in the same iteration of every loop, U32_MAX if @t is ahead of @s in a loop.
+ */
+static u32 flow_trips_behind(const struct bpf_verifier_state *t, const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *lt, *ls;
+	u32 fr, i, behind = 0;
+
+	if (t->curframe != s->curframe)
+		return U32_MAX;
+	for (fr = 0; fr <= t->curframe; fr++) {
+		if (t->frame[fr]->loop_stack_cnt != s->frame[fr]->loop_stack_cnt)
+			return U32_MAX;
+		for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+			lt = &t->frame[fr]->loop_stack[i];
+			ls = &s->frame[fr]->loop_stack[i];
+			if (!lt->apart && !ls->apart)
+				continue;
+			if (lt->trip > ls->trip)
+				return U32_MAX;
+			behind = max(behind, ls->trip - lt->trip);
+		}
+	}
+	return behind;
+}
+
+/*
+ * A state is compared with the states of its own iteration and of the two before it. The older ones
+ * are gone, see flow_earlier_trip(), but for what is pending or waits: every iteration of a loop
+ * calls a recursive function, and what comes out of it goes to all the calls at once.
+ */
+static bool flow_trips_near(const struct bpf_verifier_state *t, const struct bpf_verifier_state *s)
+{
+	return flow_trips_behind(t, s) <= 2;
+}
+
+/*
+ * @s is within @t, and from now on @t stands for it.
+ *
+ * A walk on patience gets to the end of a loop when no state comes back to the head any more.
+ * Then the loop ends at run time too: the number of the iteration grows at the head, nothing makes
+ * it smaller, and there is a largest one.
+ * If @t is of an earlier iteration than @s that is not true any more. What becomes of @t got to @s
+ * and is @t again: as far as the verifier can see the program goes around for ever. A spin lock
+ * is such a loop. So the loop gets may_goto.
+ *
+ * It's asked for every loop that @t is behind in, not only for the outermost one. This is the only
+ * way for the number of a loop to get smaller. In a walk it doesn't change in another way but at
+ * the head: a state never stands for a state of an earlier iteration, see loop_stack_safe(), and it
+ * takes the place of, or is joined with, states of its own iterations only, see flow_merge().
+ * Where the iterations are not kept apart a join has the larger number, and a state may stand
+ * for one with a smaller number. Such loop has a count or may_goto, or something stops it, since
+ * the state entered it.
+ */
+static int flow_cycle_closed(struct bpf_verifier_env *env, const struct bpf_verifier_state *t,
+			     const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *lt, *ls;
+	struct bpf_loop *loop;
+	u32 fr, i;
+
+	for (fr = 0; fr <= t->curframe; fr++) {
+		for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+			lt = &t->frame[fr]->loop_stack[i];
+			ls = &s->frame[fr]->loop_stack[i];
+			if (lt->trip >= ls->trip)
+				continue;
+			/* counted: what comes back is expected to be within what is at the head */
+			if (lt->terminates && !bpf_loop_walk_may_give_up(lt))
+				continue;
+			loop = env->insn_aux_data[lt->loop_id].loop;
+			/* it goes around for as long as the iterator has elements or may_goto budget */
+			if (loop->rt_bounded)
+				continue;
+			if (!loop->has_guards) {
+				verbose(env, "loop at insn %d never exits\n", lt->loop_id);
+				return -EINVAL;
+			}
+			bpf_flow_loop_gets_guards(env, lt->loop_id);
+		}
+	}
+	return 0;
+}
+
+/*
+ * @t is of an earlier iteration of a loop whose iterations are kept apart than @s.
+ * What comes after @s is of later iterations too, so @t is not going to be
+ * of use. If something of its iteration does come it is looked at again.
+ *
+ * Not so where a state stands for many iterations. States of different kinds are at the head of
+ * such loop. One of them comes to rest early, and what comes around is within it for as long as
+ * the others take.
+ */
+static bool flow_earlier_trip(const struct bpf_verifier_state *t,
+			      const struct bpf_verifier_state *s)
+{
+	const struct bpf_func_state *ft, *fs;
+	u32 fr, i;
+
+	if (t->curframe != s->curframe)
+		return false;
+	for (fr = 0; fr <= t->curframe; fr++) {
+		ft = t->frame[fr];
+		fs = s->frame[fr];
+		if (ft->subprogno != fs->subprogno || ft->callsite != fs->callsite ||
+		    ft->loop_stack_cnt != fs->loop_stack_cnt)
+			return false;
+		for (i = 0; i < ft->loop_stack_cnt; i++) {
+			if (ft->loop_stack[i].loop_id != fs->loop_stack[i].loop_id ||
+			    ft->loop_stack[i].apart != fs->loop_stack[i].apart)
+				return false;
+			if (ft->loop_stack[i].apart &&
+			    ft->loop_stack[i].trip != fs->loop_stack[i].trip)
+				return ft->loop_stack[i].trip + 1 < fs->loop_stack[i].trip;
+		}
+	}
+	return false;
+}
+
+/* Can flow_earlier_trip() be true for a state and @s? */
+static bool flow_has_earlier_trips(const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *entry;
+	u32 fr, i;
+
+	for (fr = 0; fr <= s->curframe; fr++) {
+		for (i = 0; i < s->frame[fr]->loop_stack_cnt; i++) {
+			entry = &s->frame[fr]->loop_stack[i];
+			if (entry->apart && entry->trip > 1)
+				return true;
+		}
+	}
+	return false;
+}
-- 
2.55.0

[PATCH bpf-next v3 13/45] bpf: Decide which states are not joined

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Cilium, nat.h, in tail_handle_snat_fwd_ipv4:

  if (ret < 0) ..; to_sport = state->to_sport;

(ret=-155, state=map_value_or_null) and (ret=0, state=map_value) meet in
front of that. Numbers always have a join. But ret=[-155, 0] with
map_value_or_null has ret=0 without state, as no path has. It passes the
jump: "R3 invalid mem access 'map_value_or_null'".

A join must not make up what a later check can see. A number is tested if
a later jump compares it with a constant, or the verifier checks its range
at a call or at exit. States that have a join stay apart if they differ in:

 - A tested number, and the ranges leave a gap: -155 and 0, not 0 and 1.
   113 progs of Cilium, profiler3 and test_seg6_loop without alu32.
 - Two tested numbers, each without a common value: (err >= 0, ret=0) and
   (err < 0, ret=DROP). The join has err < 0 with ret=0.
 - A flag, 0 and 1, that a jump in this basic block or the next one tests:
   'ok = i < 64 && v->in[i] == 2; if (v->x == 7) v->n++;
    if (ok) v->w[i] = 1;'
   clang keeps ok in a register. (ok=0, i>63), (ok=1, i<=63), (ok=0, i<=63)
   meet. The join has ok=1 with i>63: "R0 unbounded memory access".
 - A register that only one of them sets. scx_layered/layered_runnable:
   'scoped_guard(rcu) { if (...) return result; nsid = ...; }' meets at the
   unlock with a flag of the compiler, then reads nsid: "R7 !read_ok".
 - Two tested numbers, (0, x) and (y, 0). xdp_synproxy_kern and
   test_tcp_custom_syncookie: 'if (hdr->ipv4) ... else if (hdr->ipv6) ...'
   after the packet changed: "R1 type=scalar expected=fp, pkt, ...".
 - A constant that becomes a stack offset. bt_print(), 5 selftests of scx:
   'if (... || depth >= BT_MAXLVL_PRINT) return 0; ind = stack[depth];'
   A copy of depth is tested: "invalid variable-offset read from stack".
 - A constant that is added to a pointer. katran/tcdecapinfo, 20 or 40 in
   'if (data + hdr_len + inner_ip_offset > data_end)'. With [20, 40] the
   jump says nothing about data + hdr_len: "invalid access to packet".
 - A constant that a kfunc or helper must know. One call is made of
   'c ? bpf_rdonly_cast(0, id) : bpf_rdonly_cast(0, 0)' in
   mem_rdonly_untrusted/mixed_mem_type: "R2 must be a known constant".
 - The number of calls of the callback of bpf_loop(N, ...), N <= 8:
   'bpf_loop(1, cb, &ctx, 0); return choice_arr[ctx.i];' in
   verifier_iterating_callbacks: "invalid access to map value".

The rules only refuse joins: no unsafe prog loads because of them. They
cost states: the first one 10 % of Cilium's insns, K flags 2^K states. Thus
most hold only while each limit is less than half used: flow_has_room().

The shape of a state is a filter in front of all that: hashes of its calls
and loops, of what it holds, of the places, objects and constant offsets of
its pointers. The load of test_cls_redirect_subprogs, 301 states at an
insn, takes three times as long without the shape.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 566 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 559 insertions(+), 7 deletions(-)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index f17eb3467708..9eff1d60cc4e 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -148,6 +148,7 @@ enum flow_need_kind {
 	FLOW_NEED_STACK_OFF,
 	FLOW_NEED_PTR_OFF,
 	FLOW_NEED_JUMP,
+	FLOW_NEED_FLAG,
 	FLOW_NEED_VALUE_OFF,
 	FLOW_NEED_BASE,
 	FLOW_NEED_JUMP_ALU,
@@ -568,6 +569,19 @@ static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
 	return consts & BIT(insn->dst_reg) ? BIT(insn->src_reg) : 0;
 }
 
+/* A jump that compares a register with 0 or 1 */
+static u16 flow_flag_regs(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+
+	if (BPF_SRC(insn->code) != BPF_K || !bpf_insn_is_cond_jump(insn->code) ||
+	    BPF_OP(insn->code) == BPF_JCOND)
+		return 0;
+	if (BPF_OP(insn->code) == BPF_JSET ? !(insn->imm & 1) : (u32)insn->imm > 1)
+		return 0;
+	return BIT(insn->dst_reg);
+}
+
 /* With or without sign extension */
 static bool flow_is_copy(const struct bpf_insn *insn)
 {
@@ -953,6 +967,15 @@ static u16 flow_base_regs(struct bpf_verifier_env *env, int idx)
  * A slot that is filled through another pointer to the stack than r10 is not one of them, and
  * flow_null_differs() counts on that.
  *
+ * FLOW_NEED_FLAG: those of which a jump tells 0 from 1, see flow_join_makes_up(). As they are,
+ * extended, or masked: a bool is tested as "w1 &= 1; if w1 != 0". Not through a call.
+ * The jump ends the basic block of the insn, or a block that this block leads to. Not further:
+ * every state that is kept apart goes through all that is on the way to the jump once more.
+ * So there are two rounds. In the first one nothing comes from the blocks ahead. What it finds
+ * at the first insn of a block is what the second one takes at the end of the blocks that lead
+ * there. In both a block depends on nothing that the round itself computes for another block,
+ * so the order in which the insns are looked at doesn't matter.
+ *
  * The last three kinds keep no states apart. They are for bpf_flow_is_needed(), which asks
  * whether anything ahead looks at a number that a loop changes from iteration to iteration.
  * FLOW_NEED_VALUE_OFF: FLOW_NEED_PTR_OFF, and the offset in a map value that holds nothing but
@@ -974,10 +997,11 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 				   enum flow_need_kind kind)
 {
 	struct bpf_insn *insns = env->prog->insnsi, *insn;
-	struct flow_need in, out, *need;
+	struct flow_need in, out, *need, *from, *ahead = NULL;
 	struct bpf_iarray *succ;
 	bool jump = kind == FLOW_NEED_JUMP || kind == FLOW_NEED_JUMP_ALU;
 	bool callx = kind == FLOW_NEED_BASE || kind == FLOW_NEED_JUMP_ALU;
+	bool flag = kind == FLOW_NEED_FLAG, second = false;
 	bool changed = true, any = false, keep, from_src, reg;
 	int i, s, idx, slot, err = 0;
 	u16 *gen, callx_regs = 0, pair = BIT(BPF_REG_0) | BIT(BPF_REG_2);
@@ -987,8 +1011,11 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 	gen = kvcalloc(env->prog->len, sizeof(u16), GFP_KERNEL_ACCOUNT);
 	/* the callers need what the subprog returns: r0, and r2 of a pair */
 	ret = kvcalloc(env->subprog_cnt, sizeof(u8), GFP_KERNEL_ACCOUNT);
+	if (flag)
+		ahead = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
 	*pneed = need;
-	if (!need || !gen || !ret) {
+	if (!need || !gen || !ret || (flag && !ahead)) {
+		kvfree(ahead);
 		kvfree(gen);
 		kvfree(ret);
 		return -ENOMEM;
@@ -996,6 +1023,8 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 	for (i = 0; i < env->prog->len; i++) {
 		if (jump)
 			gen[i] = flow_tested_regs(env, i);
+		else if (flag)
+			gen[i] = flow_flag_regs(env, i);
 		else if (kind == FLOW_NEED_STACK_OFF)
 			gen[i] = flow_stack_off_regs(env, i);
 		else if (kind == FLOW_NEED_PTR_OFF || kind == FLOW_NEED_VALUE_OFF)
@@ -1024,6 +1053,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 		}
 		any = true;
 	}
+again:
 	while (any && changed) {
 		changed = false;
 		if (signal_pending(current)) {
@@ -1041,9 +1071,10 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 			succ = bpf_insn_successors(env, idx);
 			out.regs = 0;
 			out.slots = 0;
+			from = flag && env->insn_aux_data[idx].bb_end ? ahead : need;
 			for (s = 0; s < succ->cnt; s++) {
-				out.regs |= need[succ->items[s]].regs;
-				out.slots |= need[succ->items[s]].slots;
+				out.regs |= from[succ->items[s]].regs;
+				out.slots |= from[succ->items[s]].slots;
 			}
 			in = out;
 			if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) {
@@ -1051,7 +1082,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 				if (bpf_is_callx(insn))
 					in.regs |= callx_regs & GENMASK(BPF_REG_5, BPF_REG_1);
 				s = idx + insn->imm + 1;
-				if (bpf_pseudo_call(insn) &&
+				if (bpf_pseudo_call(insn) && !flag &&
 				    !bpf_subprog_is_global(env, bpf_find_subprog(env, s))) {
 					if (kind != FLOW_NEED_PTR_OFF)
 						in.regs |= need[s].regs &
@@ -1090,10 +1121,12 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 				/* the result is known when what it is made of is */
 				keep = op != BPF_MOV;
 				from_src = reg;
-				if (jump)
+				if (jump || flag)
 					from_src = flow_is_copy(insn);
-				if (kind == FLOW_NEED_JUMP)
+				if (kind == FLOW_NEED_JUMP || flag)
 					keep = flow_is_shift(insn) && !reg;
+				if (flag && op == BPF_AND && !reg)
+					keep = insn->imm & 1;
 				if (kind == FLOW_NEED_BASE) {
 					keep = !reg && (op == BPF_ADD || op == BPF_SUB);
 					from_src = reg && op == BPF_MOV;
@@ -1110,12 +1143,18 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 			}
 		}
 	}
+	if (flag && !err && !second) {
+		memcpy(ahead, need, env->prog->len * sizeof(*need));
+		second = changed = true;
+		goto again;
+	}
 	/*
 	 * An offset that is an argument: not through the call, see above, and not where the calls
 	 * meet. A function would be looked at once for each offset that it's called with.
 	 */
 	for (s = 0; kind == FLOW_NEED_PTR_OFF && s < env->subprog_cnt; s++)
 		need[env->subprog_info[s].start].regs = 0;
+	kvfree(ahead);
 	kvfree(gen);
 	kvfree(ret);
 	if (!any) {
@@ -1642,3 +1681,516 @@ static bool flow_has_earlier_trips(const struct bpf_verifier_state *s)
 	}
 	return false;
 }
+
+/* @flag: 0 and 1 */
+static bool flow_two_consts(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb,
+			    bool flag)
+{
+	return ra->type == SCALAR_VALUE && rb->type == SCALAR_VALUE &&
+	       tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+	       ra->var_off.value != rb->var_off.value &&
+	       (!flag || (ra->var_off.value | rb->var_off.value) == 1);
+}
+
+/* Constants that have to stay constants, see flow_compute_need_const() */
+static bool flow_needed_consts_differ(const struct bpf_flow *flow, enum flow_need_kind kind,
+				      const struct bpf_verifier_state *a,
+				      const struct bpf_verifier_state *b)
+{
+	const struct flow_need *needs = flow->need[kind], *need;
+	const struct bpf_stack_state *sa, *sb;
+	const struct bpf_func_state *fa, *fb;
+	bool flag = kind == FLOW_NEED_FLAG;
+	u32 fr, i, nr;
+
+	if (!needs || a->curframe != b->curframe)
+		return false;
+	for (fr = 0; fr <= a->curframe; fr++) {
+		/* the callers need what they need when the call returns */
+		need = &needs[fr == a->curframe ? a->insn_idx : a->frame[fr + 1]->callsite + 1];
+		fa = a->frame[fr];
+		fb = b->frame[fr];
+		for (i = BPF_REG_0; need->regs && i < BPF_REG_FP; i++)
+			if ((need->regs & BIT(i)) &&
+			    flow_two_consts(&fa->regs[i], &fb->regs[i], flag))
+				return true;
+		nr = min(bpf_stack_nr_slots(fa), bpf_stack_nr_slots(fb));
+		for (i = 0; i < nr && i < 64 && need->slots >> i; i++) {
+			sa = bpf_stack_slot(fa, i);
+			sb = bpf_stack_slot(fb, i);
+			if ((need->slots & BIT_ULL(i)) &&
+			    bpf_is_spilled_scalar_reg(sa) && bpf_is_spilled_scalar_reg(sb) &&
+			    flow_two_consts(&sa->spilled_ptr, &sb->spilled_ptr, flag))
+				return true;
+		}
+	}
+	return false;
+}
+
+/* regsafe() and join_reg() say what each type needs */
+static void flow_shape_ptr(struct flow_shape *sh, u32 place, const struct bpf_reg_state *reg)
+{
+	u32 ids = (reg->id != 0) | (reg->parent_id != 0) << 1 | (reg->map_uid != 0) << 2;
+	u32 len = offsetof(struct bpf_reg_state, var_off);
+	bool has_off = false, var_off = false;
+	u64 off = reg->var_off.value;
+
+	switch (base_type(reg->type)) {
+	case NOT_INIT:
+	case SCALAR_VALUE:
+		return;
+	case PTR_TO_ARENA:
+		/* the offset is not tracked */
+		len = sizeof(reg->type);
+		ids = 0;
+		break;
+	case PTR_TO_PACKET_META:
+	case PTR_TO_PACKET:
+		len = sizeof(reg->type);
+		ids &= 1;
+		/* behind the end of the packet, or at the end: the same in both states */
+		if (reg->range < 0)
+			ids |= -reg->range << 1;
+		else
+			sh->pkt_range += reg->range;
+		if (reg->id) {
+			/* displacements between pointers that share an id are kept */
+			has_off = true;
+			off = reg->r64.base;
+		} else if (reg->range >= 0) {
+			/* join_reg() takes two such constants for two places in the packet */
+			has_off = true;
+			var_off = !tnum_is_const(reg->var_off) || off > MAX_PACKET_OFF;
+		}
+		break;
+	case PTR_TO_INSN:
+		/* a range of offsets is a set of targets */
+		ids = 0;
+		break;
+	case PTR_TO_MAP_KEY:
+	case PTR_TO_MAP_VALUE:
+	case PTR_TO_MEM:
+	case PTR_TO_BUF:
+	case PTR_TO_TP_BUFFER:
+	case PTR_TO_STACK:
+	case PTR_TO_BTF_ID:
+		has_off = true;
+		var_off = !tnum_is_const(reg->var_off);
+		break;
+	default:
+		len = offsetof(struct bpf_reg_state, id);
+	}
+	sh->word[FLOW_LAYOUT] = jhash_1word(place << 1 | var_off, sh->word[FLOW_LAYOUT]);
+	sh->word[FLOW_PTRS] = jhash2((const u32 *)reg, len / sizeof(u32),
+				     sh->word[FLOW_PTRS] + ids);
+	if (has_off && !var_off)
+		sh->word[FLOW_OFFS] = jhash_2words(lower_32_bits(off), upper_32_bits(off),
+						   sh->word[FLOW_OFFS]);
+}
+
+/* stacksafe() and join_special_slot() say what each kind needs */
+static void flow_shape_special(struct flow_shape *sh, u32 place, const struct bpf_stack_state *slot)
+{
+	const struct bpf_reg_state *reg = &slot->spilled_ptr;
+	u32 kind = slot->slot_type[BPF_REG_SIZE - 1] << 2 | (reg->id != 0);
+	u32 *ptrs = &sh->word[FLOW_PTRS];
+
+	switch (slot->slot_type[BPF_REG_SIZE - 1]) {
+	case STACK_DYNPTR:
+		kind |= (reg->parent_id != 0) << 1;
+		*ptrs = jhash_3words(kind, reg->dynptr.type, reg->dynptr.first_slot, *ptrs);
+		break;
+	case STACK_ITER:
+		*ptrs = jhash_3words(lower_32_bits((unsigned long)reg->iter.btf),
+				     upper_32_bits((unsigned long)reg->iter.btf),
+				     reg->iter.btf_id, *ptrs + kind);
+		*ptrs = jhash_3words(reg->type, reg->iter.state, reg->iter.loops, *ptrs);
+		break;
+	case STACK_IRQ_FLAG:
+		*ptrs = jhash_2words(kind, reg->irq.kfunc_class, *ptrs);
+		break;
+	}
+	sh->word[FLOW_LAYOUT] = jhash_1word(place << 1, sh->word[FLOW_LAYOUT]);
+}
+
+/* The part of the shape that is made of the registers and the stack, when it is asked for */
+static void flow_shape_places(struct bpf_verifier_env *env, struct flow_state *s)
+{
+	struct bpf_verifier_state *st = &s->st;
+	const struct bpf_stack_state *slot;
+	const struct bpf_func_state *frame;
+	struct flow_shape *sh = &s->shape;
+	u32 fr, i, args;
+	u16 live;
+
+	sh->places = true;
+	for (fr = 0; fr <= st->curframe; fr++) {
+		frame = st->frame[fr];
+		/* a state that does a call again is not cleaned, dead registers are not compared */
+		live = env->insn_aux_data[bpf_frame_insn_idx(st, fr)].live_regs_before;
+		for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+			if (live & BIT(i))
+				flow_shape_ptr(sh, fr << 10 | i, &frame->regs[i]);
+		for (i = 0; i < bpf_stack_nr_slots(frame); i++) {
+			slot = bpf_stack_slot(frame, i);
+			switch (slot->slot_type[BPF_REG_SIZE - 1]) {
+			case STACK_SPILL:
+				flow_shape_ptr(sh, fr << 10 | (i + 16), &slot->spilled_ptr);
+				break;
+			case STACK_DYNPTR:
+			case STACK_ITER:
+			case STACK_IRQ_FLAG:
+				flow_shape_special(sh, fr << 10 | (i + 16), slot);
+				break;
+			}
+		}
+		/* incoming arguments are not compared with outgoing ones: other places */
+		args = 16 + MAX_BPF_STACK_SLOTS;
+		if (bpf_stack_args_are_incoming(frame))
+			args += MAX_STACK_ARG_SLOTS;
+		for (i = 0; i < frame->out_stack_arg_cnt; i++)
+			flow_shape_ptr(sh, fr << 10 | (i + args), &frame->stack_arg_regs[i]);
+	}
+}
+
+/* The part of the shape that is not */
+static void flow_shape_ctx(const struct bpf_verifier_state *st, struct flow_shape *sh)
+{
+	const struct bpf_reference_state *ref;
+	const struct loop_stack_entry *entry;
+	const struct bpf_func_state *frame;
+	u32 chain = st->curframe, ctx, fr, i;
+
+	ctx = jhash_3words(st->in_sleepable | st->callers_unknown << 1 |
+			   st->unwound << 2 | st->changed_pkt_data << 3 |
+			   (st->active_irq_id != 0) << 4 | (st->active_lock_id != 0) << 5,
+			   st->acquired_refs, st->active_locks, 0);
+	ctx = jhash_2words(st->active_preempt_locks, st->active_rcu_locks, ctx);
+	ctx = jhash_2words(lower_32_bits((unsigned long)st->active_lock_ptr),
+			   upper_32_bits((unsigned long)st->active_lock_ptr), ctx);
+	for (i = 0; i < st->acquired_refs; i++) {
+		ref = &st->refs[i];
+		ctx = jhash_1word(ref->type, ctx);
+		if (ref->type != REF_TYPE_PTR && ref->type != REF_TYPE_IRQ)
+			ctx = jhash_2words(lower_32_bits((unsigned long)ref->ptr),
+					   upper_32_bits((unsigned long)ref->ptr), ctx);
+	}
+	for (fr = 0; fr <= st->curframe; fr++) {
+		frame = st->frame[fr];
+		chain = jhash_3words(frame->subprogno, frame->callsite, frame->loop_stack_cnt,
+				     chain);
+		ctx = jhash_2words(frame->entry_refs, frame->in_rcu_callback_fn, ctx);
+		ctx = jhash_2words(frame->entry_rcu_locks, frame->entry_preempt_locks, ctx);
+		for (i = 0; i < frame->loop_stack_cnt; i++) {
+			entry = &frame->loop_stack[i];
+			chain = jhash_1word(entry->loop_id << 1 | entry->apart, chain);
+			ctx = jhash_3words(entry->counted << 2 | entry->guarded << 1 |
+					   entry->terminates,
+					   entry->max_header_count,
+					   entry->counted ? entry->trip : 0, ctx);
+		}
+	}
+	memset(sh, 0, sizeof(*sh));
+	sh->word[FLOW_CHAIN] = chain;
+	sh->word[FLOW_CTX] = ctx;
+}
+
+/* Are the words up to @last the same? */
+static __always_inline bool flow_same_shape(const struct flow_state *a, const struct flow_state *b,
+					    enum flow_word last)
+{
+	u32 i;
+
+	for (i = 0; i <= last; i++)
+		if (a->shape.word[i] != b->shape.word[i])
+			return false;
+	return true;
+}
+
+/*
+ * A state that arrives is compared with the states at the insn: is it within one of them, see
+ * bpf_state_subsumes(), does it have a join with one of them, see bpf_states_join()? Most of
+ * them have nothing to do with it. The shape of a state tells that before the registers are
+ * compared or, for a join, copied. It has what must be the same in two states that are related:
+ *
+ * FLOW_CHAIN	the chain of calls and the loops that every frame is in
+ * FLOW_CTX	what is known about every loop, the iteration of a counted one,
+ *		the flags of the state and of its frames, locks and references
+ * FLOW_LAYOUT	which registers, stack slots and stack arguments hold a pointer, a dynptr,
+ *		an iterator or an irq flag, which of the pointers are at a constant offset
+ * FLOW_PTRS	what the pointers point to and which of their ids are set, the kind of the others
+ * FLOW_OFFS	the constant offsets
+ *
+ * Nothing about numbers is in it.
+ *
+ * States that differ in one of the first two words are not related. Neither are states of
+ * the same layout that differ in one of the last two: a pointer is related to a pointer to the same
+ * thing only, and two constant offsets are two places in it.
+ * States of different layout may be related, whatever the last two words are: nothing in
+ * a register or in a slot stands for a pointer there, and a range of offsets stands for a constant,
+ * or is joined with it.
+ *
+ * A word is a hash. It is a filter and proves nothing: the states that get through it go
+ * through all the comparisons. So it's fine to look at the first two words only: @places.
+ */
+static bool flow_may_relate(struct bpf_verifier_env *env, struct flow_state *t,
+			    struct flow_state *s, bool places)
+{
+	if (!flow_same_shape(t, s, FLOW_CTX))
+		return false;
+	if (!places)
+		return true;
+	if (!t->shape.places)
+		flow_shape_places(env, t);
+	if (!s->shape.places)
+		flow_shape_places(env, s);
+	return !flow_same_shape(t, s, FLOW_LAYOUT) || flow_same_shape(t, s, FLOW_OFFS);
+}
+
+/*
+ * A pointer to the packet stands for one with at least as many checked bytes, and without
+ * widening it's joined with one that has as many. In states of the same layout that holds for
+ * the sums too. Returns 0 if the sums are the same, or say nothing.
+ */
+static int flow_cmp_pkt_range(const struct flow_state *t, const struct flow_state *s)
+{
+	if (!t->shape.places || !s->shape.places || !flow_same_shape(t, s, FLOW_LAYOUT) ||
+	    t->shape.pkt_range == s->shape.pkt_range)
+		return 0;
+	return t->shape.pkt_range < s->shape.pkt_range ? -1 : 1;
+}
+
+/*
+ * bpf_loop(N, ...) with a small N: the calls of the callback are looked at one by one, like
+ * the iterations of a loop that is walked. The call stops to enter the callback when
+ * callback_depth gets to N.
+ */
+static bool flow_callback_depth_differs(struct bpf_verifier_env *env,
+					const struct bpf_verifier_state *a,
+					const struct bpf_verifier_state *b)
+{
+	u32 max = BPF_FLOW_MAX_CALLBACKS, fr;
+	const struct bpf_func_state *fa, *fb;
+	struct bpf_insn *insn;
+
+	if (a->curframe != b->curframe)
+		return false;
+	/* at the call, and in the callback and what it calls */
+	for (fr = 0; fr <= a->curframe; fr++) {
+		insn = &env->prog->insnsi[fr == a->curframe ? a->insn_idx :
+					  a->frame[fr + 1]->callsite];
+		if (!bpf_helper_call(insn) || insn->imm != BPF_FUNC_loop)
+			continue;
+		fa = a->frame[fr];
+		fb = b->frame[fr];
+		if (fa->callback_depth != fb->callback_depth &&
+		    fa->regs[BPF_REG_1].type == SCALAR_VALUE &&
+		    reg_umax(&fa->regs[BPF_REG_1]) <= max &&
+		    fb->regs[BPF_REG_1].type == SCALAR_VALUE &&
+		    reg_umax(&fb->regs[BPF_REG_1]) <= max)
+			return true;
+	}
+	return false;
+}
+
+static const struct bpf_reg_state *flow_slot_number(const struct bpf_func_state *frame, u32 i)
+{
+	const struct bpf_stack_state *slot = bpf_stack_slot(frame, i);
+
+	if (slot->slot_type[0] != STACK_SPILL || slot->slot_type[BPF_REG_SIZE - 1] != STACK_SPILL ||
+	    slot->spilled_ptr.type != SCALAR_VALUE)
+		return NULL;
+	return &slot->spilled_ptr;
+}
+
+/* Returns true when the answer is known */
+static bool flow_null_pair(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb,
+			   bool *a_null, bool *b_null)
+{
+	if (tnum_equals_const(ra->var_off, 0) && !tnum_is_const(rb->var_off)) {
+		*a_null = true;
+		if (reg_umin(rb) == 1 && reg_umax(rb) == U64_MAX)
+			return true;
+	}
+	if (tnum_equals_const(rb->var_off, 0) && !tnum_is_const(ra->var_off)) {
+		*b_null = true;
+		if (reg_umin(ra) == 1 && reg_umax(ra) == U64_MAX)
+			return true;
+	}
+	return *a_null && *b_null;
+}
+
+/*
+ * Two things of which one is there: "if (hdr->ipv4) ... else if (hdr->ipv6)". The one that
+ * is not there is 0. What the other one is may be unknown, e.g. a pointer to the packet
+ * after the packet was changed. (0, x) is not within (y, 0) and the join (?, ?) forgets that
+ * it's one or the other.
+ * The same for a number that was compared with 0: 0 on one path, not 0 on the other.
+ *
+ * Only where a jump ahead can see it: FLOW_NEED_JUMP, see flow_join_makes_up(). What no jump
+ * looks at again sends no state another way, and every such number doubles the states.
+ *
+ *	for (i = 0; i < n; i++) { p = a[i]; if (!p) continue; put(p); }
+ * a[] is on the stack. The jump finds out about the slot that p was filled from too. Nothing
+ * looks at a[i] again, but the loop reads through a pointer that moves, so all of a[] is live
+ * until the loop is done: 2^n states if they stay apart.
+ * They don't, because FLOW_NEED_JUMP has only what is filled through r10, see flow_fp_slot().
+ * It cannot tell a[i] from a[i + 1]. With what is filled through other pointers to the stack in
+ * it, all of a[] is tested in all of the loop.
+ */
+static bool flow_null_differs(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
+			      const struct bpf_verifier_state *b)
+{
+	const struct bpf_func_state *fa, *fb;
+	const struct bpf_reg_state *ra, *rb;
+	const struct flow_need *need = flow->need[FLOW_NEED_JUMP];
+	bool a_null = false, b_null = false;
+	u32 i;
+
+	if (!need || a->curframe != b->curframe)
+		return false;
+	need += a->insn_idx;
+	fa = a->frame[a->curframe];
+	fb = b->frame[b->curframe];
+	for (i = BPF_REG_0; i < BPF_REG_FP; i++) {
+		ra = &fa->regs[i];
+		rb = &fb->regs[i];
+		if ((need->regs & BIT(i)) &&
+		    ra->type == SCALAR_VALUE && rb->type == SCALAR_VALUE &&
+		    flow_null_pair(ra, rb, &a_null, &b_null))
+			return true;
+	}
+	for (i = 0; i < 64 && i < bpf_stack_nr_slots(fa) && i < bpf_stack_nr_slots(fb); i++) {
+		if (!(need->slots & BIT_ULL(i)))
+			continue;
+		ra = flow_slot_number(fa, i);
+		rb = flow_slot_number(fb, i);
+		if (ra && rb && flow_null_pair(ra, rb, &a_null, &b_null))
+			return true;
+	}
+	return false;
+}
+
+static const struct bpf_reg_state *flow_number_at(const struct bpf_func_state *frame, u8 loc)
+{
+	const struct bpf_reg_state *reg;
+
+	if (loc <= BPF_REG_FP)
+		reg = &frame->regs[loc];
+	else if (loc - BPF_REG_FP - 1 < bpf_stack_nr_slots(frame) &&
+		 bpf_is_spilled_scalar_reg(bpf_stack_slot(frame, loc - BPF_REG_FP - 1)))
+		reg = &bpf_stack_slot(frame, loc - BPF_REG_FP - 1)->spilled_ptr;
+	else
+		return NULL;
+	return reg->type == SCALAR_VALUE ? reg : NULL;
+}
+
+/*
+ * A register is set in one of the states and not in the other. Only live registers are in
+ * a state, so something ahead may read it. The state in which it's not set doesn't get there:
+ * a jump sends it another way. The join would not know which way to go, and would get
+ * there without the register. There is nothing to gain from such a join but a state less.
+ */
+static bool flow_unset_differs(const struct bpf_verifier_state *a,
+			       const struct bpf_verifier_state *b)
+{
+	u32 fr, i;
+
+	for (fr = 0; fr <= a->curframe; fr++)
+		for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+			if ((a->frame[fr]->regs[i].type == NOT_INIT) !=
+			    (b->frame[fr]->regs[i].type == NOT_INIT))
+				return true;
+	return false;
+}
+
+static bool flow_disjoint(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+	return cnum64_is_empty(cnum64_intersect(a->r64, b->r64)) ||
+	       cnum32_is_empty(cnum32_intersect(a->r32, b->r32)) ||
+	       ((a->var_off.value ^ b->var_off.value) & ~(a->var_off.mask | b->var_off.mask));
+}
+
+/* The smallest range that has both has numbers that are in neither */
+static bool flow_gap(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+	if (cnum64_is_empty(cnum64_intersect(a->r64, b->r64)) &&
+	    a->r64.size + b->r64.size + 1 > a->r64.size &&
+	    cnum64_union(a->r64, b->r64).size > a->r64.size + b->r64.size + 1)
+		return true;
+	return cnum32_is_empty(cnum32_intersect(a->r32, b->r32)) &&
+	       cnum32_union(a->r32, b->r32).size > (u64)a->r32.size + b->r32.size + 1;
+}
+
+/*
+ * Numbers always have a join. It must not make up what neither state has where a jump ahead
+ * can see it. FLOW_NEED_JUMP has the registers and the stack slots that a jump ahead compares
+ * with a constant, or the verifier does at a call or at the exit.
+ *
+ *	if (ret < 0) goto err; if (ret > 0) goto skip; use(p);
+ * ret = [-2, -1] and ret = [1, 2], both without p, meet in front of that. [-2, 2] has 0, gets
+ * through both jumps, and there is no p. So two ranges that leave a gap are not joined. Ranges
+ * that touch, 0 and 1, -1 and 0, or have something in common make up no number.
+ *
+ *	err = update(); ret = err < 0 ? DROP : 0; ... if (err < 0) goto out; ... if (ret < 0) ...
+ * (err >= 0, ret = 0) and (err < 0, ret = DROP) meet. The join has err < 0 with ret = 0. So
+ * two numbers, each with nothing in common between the states, are not joined either.
+ *
+ *	fatal = err == -ENOMEM; ... if (fatal) bpf_set_retval(err);
+ * (fatal = 0, err = 1) and (fatal = 1, err = -ENOMEM) meet. The same: the join has
+ * fatal = 1 with err = 1, which bpf_set_retval() doesn't take.
+ *
+ *	ok = i < 64 && in[i]; if (x) n++; if (ok) a[i] = 1;
+ * (ok = 0, i >= 64), (ok = 0, i < 64) and (ok = 1, i < 64) meet. 0 and 1 touch, and i < 64 comes
+ * with either, so none of the above keeps them apart. The join has ok = 1 with i >= 64. A flag is
+ * made to send states different ways. So 0 and 1 are not joined where a jump that is not far
+ * tells one from the other: FLOW_NEED_FLAG.
+ *
+ * The states go on to the jumps on their own. Behind the last one the numbers are dead, and what
+ * is left is joined.
+ */
+static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
+			       const struct bpf_verifier_state *b)
+{
+	const struct bpf_reg_state *ra, *rb;
+	const struct flow_need *tested = flow->need[FLOW_NEED_JUMP], *need;
+	u32 fr, i, id = 0;
+	bool one = false;
+
+	if (!tested)
+		return false;
+	for (fr = 0; fr <= a->curframe; fr++) {
+		/* the callers go on when the call returns */
+		need = &tested[fr == a->curframe ? a->insn_idx : a->frame[fr + 1]->callsite + 1];
+		for (i = BPF_REG_0; i < BPF_REG_FP + 1 + 64; i++) {
+			if (i <= BPF_REG_FP ? !(need->regs & BIT(i)) :
+					      !(need->slots & BIT_ULL(i - BPF_REG_FP - 1)))
+				continue;
+			ra = flow_number_at(a->frame[fr], i);
+			rb = flow_number_at(b->frame[fr], i);
+			if (!ra || !rb || !flow_disjoint(ra, rb))
+				continue;
+			if (flow_gap(ra, rb))
+				return true;
+			/* copies of a number are one number */
+			if (one && (!id || id != (ra->id & ~BPF_ADD_CONST)))
+				return true;
+			one = true;
+			id = ra->id & ~BPF_ADD_CONST;
+		}
+	}
+	return flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b);
+}
+
+/*
+ * States that have a join are kept apart while there is room: half of every limit.
+ * After that they are joined, which may lose what the prog needs, and does not run into the limit.
+ */
+static bool flow_has_room(const struct bpf_verifier_env *env, u32 nr_states)
+{
+	u64 len = env->prog->len;
+
+	return nr_states < BPF_FLOW_MAX_STATES / 2 &&
+	       env->flow->live_states < (BPF_FLOW_LIVE_PER_INSN * len + BPF_FLOW_LIVE_BASE) / 2 &&
+	       env->total_states < (BPF_FLOW_TOTAL_PER_INSN * len + BPF_FLOW_TOTAL_BASE) / 2;
+}
-- 
2.55.0

Re: [PATCH bpf-next v3 13/45] bpf: Decide which states are not joined

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

> bpf: Decide which states are not joined

TL;DR: flow_flag_regs() in flow.c marks registers tested against 0 or 1 with
any operator (e.g. "if r < 0"), not only those telling 0 from 1; this
needlessly keeps states apart and costs states, though not unsound. Fix
comment or code?

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c

[ ... ]

> @@ -568,6 +569,19 @@ static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
>  	return consts & BIT(insn->dst_reg) ? BIT(insn->src_reg) : 0;
>  }
>  
> +/* A jump that compares a register with 0 or 1 */
> +static u16 flow_flag_regs(struct bpf_verifier_env *env, int idx)
> +{
> +	struct bpf_insn *insn = &env->prog->insnsi[idx];
> +
> +	if (BPF_SRC(insn->code) != BPF_K || !bpf_insn_is_cond_jump(insn->code) ||
> +	    BPF_OP(insn->code) == BPF_JCOND)
> +		return 0;
> +	if (BPF_OP(insn->code) == BPF_JSET ? !(insn->imm & 1) : (u32)insn->imm > 1)
> +		return 0;
> +	return BIT(insn->dst_reg);
> +}

[ ... ]

> @@ -953,6 +967,15 @@ static u16 flow_base_regs(struct bpf_verifier_env *env, int idx)
> + * FLOW_NEED_FLAG: those of which a jump tells 0 from 1, see flow_join_makes_up(). As they are,
> + * extended, or masked: a bool is tested as "w1 &= 1; if w1 != 0". Not through a call.

The comment says FLOW_NEED_FLAG holds the registers of which a jump tells 0
from 1, but flow_flag_regs() selects any conditional jump against an
immediate of 0 or 1, whatever the operator.

Eight operator/immediate pairs send 0 and 1 down the same branch: JSLT 0,
JSGE 0, JLT 0, JGE 0, JGT 1, JLE 1, JSGT 1 and JSLE 1.  JSLT/JSGE 0 is the
usual 'if (err < 0)' test.

Take two states that meet where one has the register equal to 0 and the
other equal to 1, with an 'if rX s< 0' in this block or the next.
flow_needed_consts_differ(flow, FLOW_NEED_FLAG, ...) sees
flow_two_consts(..., flag=true) return true, so flow_join_makes_up() keeps
the states apart:

    flow_join_makes_up()
      flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b)

The FLOW_NEED_JUMP part would have let them join, and the join [0, 1] makes
up nothing at that jump, because both values take the same branch.

This does not make verification unsound, since the rule only refuses joins,
but it costs states (the commit message prices this rule at 2^K states for K
flags).

Should the comment say 'compares with 0 or 1', or should flow_flag_regs()
leave out the operators for which 0 and 1 go the same way?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 14/45] bpf: Merge the state that arrives into the set at an insn

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

scx_flow, flow_dispatch() is a row of blocks like this one:

  if (likely(left) && likely(visits < (u32)FLOW_DISPATCH_MAX_VISIT)) {
      if (nq0 > 0) {
          node_moved = flow_move_one_hint(node_dsq, cpu, &visits, nq0);
          if (node_moved > left)
              node_moved = left;
          left -= node_moved;
      }
  }

Each way through a block leaves other ranges in left, visits, node_moved.
No check behind the block depends on the way. Today's verifier follows all
paths: 70716 insns for a prog of 1117. With one state where the ways meet
it takes 3115. flow_merge() puts a state S into the set at an insn:

  a state of the set covers S ?  ------ yes -->  drop S
     | no
  S covers a state T of the set ?  ---- yes -->  drop T
     |
  S has a join with a state T,  ------- yes -->  drop T, S = join(S, T),
  and no rule refuses it ?                       start again
     | no
  add S to the set: it is new, the verifier goes on from it

A state is dropped only if a state of the set covers it. Thus the verified
states cover each execution of the prog.

S can come around a cycle: to the head of a loop, to the first insn of
a recursive subprog, from a callback to its helper. There a bound can grow
by one step each time: i=[0,1], [0,2], ... Thus the join is a widening: it
gives up the bound that grew. S and T differ in what the cycle changes, so
the rules that refuse a join are off. These stay on:
 - A constant that becomes a stack offset: bt_print() of scx has
   'stack[depth++] = ind + 1' and 'ind = stack[depth]' in bpf_for().
 - A constant that is added to a pointer, if a state of the set has it too.
 - The calls of the callback of bpf_loop(N <= 8). What other callbacks come
   back with is joined 4 times, then widened: selftest
   verifier_iterating_callbacks/bpf_loop_iter_limit_nested needs both.

States that cannot be joined are what is left of the paths. Limits on them
replace 1M insns and 8192 jumps: 1024 at an insn, 16 * N + 4096 at a time,
256 * N + 64K in all, for a prog of N insns. Then the load fails: -E2BIG.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 261 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 261 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 9eff1d60cc4e..41e01bb33fb0 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -2194,3 +2194,264 @@ static bool flow_has_room(const struct bpf_verifier_env *env, u32 nr_states)
 	       env->flow->live_states < (BPF_FLOW_LIVE_PER_INSN * len + BPF_FLOW_LIVE_BASE) / 2 &&
 	       env->total_states < (BPF_FLOW_TOTAL_PER_INSN * len + BPF_FLOW_TOTAL_BASE) / 2;
 }
+
+/* The state of @set behind @t, the first one if @t is NULL, that @s may be related to */
+static struct flow_state *flow_next_related(struct bpf_verifier_env *env,
+					    const struct flow_set *set, struct flow_state *s,
+					    struct flow_state *t, bool places)
+{
+	t = t ? list_next_entry(t, node) : list_first_entry(&set->list, struct flow_state, node);
+	list_for_each_entry_from(t, &set->list, node)
+		if (flow_may_relate(env, t, s, places))
+			return t;
+	return NULL;
+}
+
+static void flow_link(struct bpf_verifier_env *env, struct flow_set *set, struct flow_state *s)
+{
+	list_add_tail(&s->node, &set->list);
+	env->flow->nr_states[s->st.insn_idx]++;
+}
+
+static void flow_sweep_trip(struct bpf_verifier_env *env, struct flow_state *t,
+			    const struct flow_state *s)
+{
+	if (t->pending || !flow_earlier_trip(&t->st, &s->st))
+		return;
+	flow_unlink(env, t);
+	flow_free_state(env, t);
+}
+
+/* States of earlier iterations than @s go away, see flow_earlier_trip() */
+static void flow_sweep_trips(struct bpf_verifier_env *env, struct flow_set *set,
+			     const struct flow_state *s)
+{
+	struct flow_state *t, *tn;
+
+	if (!flow_has_earlier_trips(&s->st))
+		return;
+	list_for_each_entry_safe(t, tn, &set->list, node)
+		if (flow_same_shape(t, s, FLOW_CHAIN))
+			flow_sweep_trip(env, t, s);
+}
+
+/*
+ * Put *@ps into @set, one of the sets of states at its insn. Returns
+ * 1 if it is within a state of the set. It is gone then.
+ * 0 if it is in the set now, as it was or joined with states of the set.
+ * @cycle: it came around a cycle, what grew is forgotten.
+ */
+static int flow_merge(struct bpf_verifier_env *env, struct flow_set *set,
+		      struct flow_state **ps, enum flow_cycle cycle)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *s = *ps, *t, *tn;
+	int insn_idx = s->st.insn_idx, err;
+	u32 *nr_states = &flow->nr_states[insn_idx];
+	bool places, widen_all, widen, offs_apart = false, offs_change = false;
+	/* the set of the states that enter a loop is a stop on the way, see bpf_flow_deliver() */
+	bool log = (env->log.level & BPF_LOG_LEVEL2) && set == &flow->states[insn_idx];
+	const char *what = NULL;
+	int prev = s->prev_insn_idx;
+
+again:
+	if (signal_pending(current)) {
+		err = -EAGAIN;
+		goto err_free;
+	}
+	if (need_resched())
+		cond_resched();
+
+	/* again after a join: an offset may not be a constant any more, a pointer may be gone */
+	flow_shape_ctx(&s->st, &s->shape);
+	places = *nr_states >= BPF_FLOW_MIN_SHAPES;
+	for (t = flow_next_related(env, set, s, NULL, places); t;
+	     t = flow_next_related(env, set, s, t, places)) {
+		if (flow_cmp_pkt_range(t, s) > 0 || !flow_trips_near(&t->st, &s->st) ||
+		    !flow_within(env, &t->st, &s->st))
+			continue;
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "from %d to %d: within\n", prev, insn_idx);
+		err = flow_cycle_closed(env, &t->st, &s->st) ?:
+		      flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+		flow_free_state(env, s);
+		return err ?: 1;
+	}
+
+	flow_sweep_trips(env, set, s);
+
+	widen_all = cycle == FLOW_CYCLE;
+	for (t = flow_next_related(env, set, s, NULL, places); t; t = tn) {
+		tn = flow_next_related(env, set, s, t, places);
+		/*
+		 * Iterations that are kept apart have no join. Neither does a state take the place
+		 * of a state of another iteration that it covers: that one is done with, or has its
+		 * own way to go, see flow_cycle_closed().
+		 * The shape is a hash, and what follows goes through the frames of both.
+		 */
+		if (flow_trips_behind(&t->st, &s->st))
+			continue;
+		widen = widen_all || (cycle == FLOW_CALLBACK_CYCLE &&
+				      max(s->joins, t->joins) >= BPF_FLOW_CALLBACK_JOINS);
+		if (!widen && flow_has_room(env, *nr_states) &&
+		    flow_null_differs(flow, &t->st, &s->st))
+			continue;
+		if (!widen && flow_needed_consts_differ(flow, FLOW_NEED_ARG, &t->st, &s->st))
+			continue;
+		if (flow_callback_depth_differs(env, &t->st, &s->st))
+			continue;
+		/* what grows in a loop is not replaced with what is larger, it is forgotten */
+		if (!widen && flow_cmp_pkt_range(s, t) <= 0 && flow_within(env, &s->st, &t->st)) {
+			err = flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+			if (err)
+				goto err_free;
+			/*
+			 * What a callback comes back with and covers what is there grew, like
+			 * what is joined with it. It counts, or nothing ends "*ctx += 1".
+			 */
+			s->joins = max(s->joins, t->joins) + (cycle == FLOW_CALLBACK_CYCLE);
+			flow_unlink(env, t);
+			flow_free_state(env, t);
+			what = what ?: "joined";
+			continue;
+		}
+		if (!widen && flow_cmp_pkt_range(s, t))
+			continue;
+		if (!widen && flow_has_room(env, *nr_states) &&
+		    (flow_unset_differs(&t->st, &s->st) ||
+		     flow_join_makes_up(flow, &t->st, &s->st)))
+			continue;
+		/*
+		 * Around a loop too:
+		 *	u8 stack[10], depth = 0;
+		 *	loop {
+		 *		if (...) { if (depth >= 10) break; stack[depth++] = x; }
+		 *		else { depth--; if (depth >= 10) break; x = stack[depth]; }
+		 *	}
+		 * depth is in a 32-bit register. A compiler tests 'depth & 255' and computes it
+		 * anew for the index. What is found out about a copy says nothing about
+		 * a range, and everything about a constant.
+		 */
+		if (flow_has_room(env, *nr_states) &&
+		    flow_needed_consts_differ(flow, FLOW_NEED_STACK_OFF, &t->st, &s->st))
+			continue;
+		/*
+		 * Two states enter a loop, with the header at pkt + 34 and at pkt + 54. What comes
+		 * around the loop with 34 is for the state with 34. Next to 54 it looks like
+		 * a number that grew, and would be forgotten.
+		 */
+		if ((!widen || !offs_change) && flow_has_room(env, *nr_states) &&
+		    flow_needed_consts_differ(flow, FLOW_NEED_PTR_OFF, &t->st, &s->st)) {
+			offs_apart |= widen;
+			continue;
+		}
+		err = bpf_states_join(env, &flow->tmp, &t->st, &s->st, widen);
+		if (err < 0)
+			goto err_free;
+		if (err)
+			continue;
+		what = widen ? "widened" : "joined";
+		err = flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+		if (err)
+			goto err_free;
+		flow->tmp.flow_entry = t->st.flow_entry;
+		flow->tmp.last_insn_idx = s->prev_insn_idx;
+		s->joins = max(s->joins, t->joins) + 1;
+		bpf_free_verifier_state(&s->st, false);
+		s->st = flow->tmp;
+		memset(&flow->tmp, 0, sizeof(flow->tmp));
+		flow_unlink(env, t);
+		flow_free_state(env, t);
+		/* the join may be within, or have a join with, what is left in the set */
+		goto again;
+	}
+
+	/* No state has the offsets of what came around the loop: they are what changes in it */
+	if (offs_apart && !offs_change && !what) {
+		offs_change = true;
+		goto again;
+	}
+
+	if (*nr_states >= BPF_FLOW_MAX_STATES) {
+		verbose(env, "more than %d states that cannot be joined at insn %d\n",
+			BPF_FLOW_MAX_STATES, insn_idx);
+		err = -E2BIG;
+		goto err_free;
+	}
+	if (flow->live_states > BPF_FLOW_LIVE_PER_INSN * env->prog->len + BPF_FLOW_LIVE_BASE ||
+	    env->total_states >
+	    BPF_FLOW_TOTAL_PER_INSN * (u64)env->prog->len + BPF_FLOW_TOTAL_BASE) {
+		verbose(env, "too many states for %d insns: %d, %d at a time\n",
+			env->prog->len, env->total_states, flow->live_states);
+		err = -E2BIG;
+		goto err_free;
+	}
+	/*
+	 * The state is printed when its turn comes. That says enough about a state that
+	 * is new, and about a state that goes on to the next insn.
+	 */
+	if (prev >= 0 && insn_idx == prev + 1 + bpf_is_ldimm64(&env->prog->insnsi[prev]) &&
+	    what && !strcmp(what, "joined"))
+		what = NULL;
+	if (log && what)
+		verbose(env, "from %d to %d: %s\n", prev, insn_idx, what);
+	flow_link(env, set, s);
+	env->max_states_per_insn = max(env->max_states_per_insn, *nr_states);
+	return 0;
+
+err_free:
+	flow_free_state(env, s);
+	return err;
+}
+
+static struct flow_state *flow_new_state(struct bpf_verifier_env *env, int prev_insn_idx)
+{
+	struct flow_state *s = kzalloc_obj(*s, GFP_KERNEL_ACCOUNT);
+	struct bpf_flow *flow = env->flow;
+
+	if (!s)
+		return NULL;
+	s->prev_insn_idx = prev_insn_idx;
+	s->seq = ++flow->seq;
+	env->total_states++;
+	flow->live_states++;
+	env->peak_states = max(env->peak_states, flow->live_states);
+	return s;
+}
+
+/* What @st holds is taken and @st is left empty. What doesn't matter at @insn_idx goes. */
+static struct flow_state *flow_arrive(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+				      int insn_idx, int prev_insn_idx, bool *enters)
+{
+	struct bpf_verifier_state *cur = env->cur_state;
+	int cur_idx = env->insn_idx, err;
+	struct flow_state *s;
+
+	if (insn_idx < 0 || insn_idx >= env->prog->len) {
+		verbose(env, "invalid insn idx %d insn_cnt %d\n", insn_idx, env->prog->len);
+		return ERR_PTR(-EFAULT);
+	}
+
+	s = flow_new_state(env, prev_insn_idx);
+	if (!s)
+		return ERR_PTR(-ENOMEM);
+	s->st = *st;
+	memset(st, 0, sizeof(*st));
+	s->st.insn_idx = insn_idx;
+	s->st.last_insn_idx = prev_insn_idx;
+
+	env->cur_state = &s->st;
+	env->insn_idx = insn_idx;
+	err = bpf_flow_loop_leave(env, enters);
+	if (!err)
+		err = bpf_clean_verifier_state(env, &s->st);
+	env->cur_state = cur;
+	env->insn_idx = cur_idx;
+	if (err) {
+		flow_free_state(env, s);
+		return ERR_PTR(err);
+	}
+	/* when the registers that it was shared with are dead, or are those of the caller */
+	bpf_clear_singular_ids(env, &s->st);
+	return s;
+}
-- 
2.55.0

Re: [PATCH bpf-next v3 14/45] bpf: Merge the state that arrives into the set at an insn

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3795 bytes --]

> bpf: Merge the state that arrives into the set at an insn

TL;DR:
- flow_merge() may widen away a register set only in the loop body, so the
  verifier could reject valid programs; the fix is later in the series
- commit message omits that states are also dropped by trip sweeping

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c

[ ... ]

> +		if (!widen && flow_cmp_pkt_range(s, t))
> +			continue;
> +		if (!widen && flow_has_room(env, *nr_states) &&
> +		    (flow_unset_differs(&t->st, &s->st) ||
> +		     flow_join_makes_up(flow, &t->st, &s->st)))
> +			continue;

When a state comes back to a loop head (cycle == FLOW_CYCLE), widen_all is
true, so this check is skipped and flow_unset_differs() never runs.  Can a
state where a live register is NOT_INIT then be widened with one where the
register is set?

bpf_states_join() -> join_reg() makes the result NOT_INIT whenever either
side is NOT_INIT:

  if (ra->type == NOT_INIT || rb->type == NOT_INIT) {
          if (pass == 2 && ra->type != NOT_INIT)
                  *dst = *rb;
          return true;
  }

So the joined state at the loop head loses a register that only the loop
body sets, and a later read of it fails.  For example:

  bpf_for(i, 0, 64) {
          if (v[i] != want)
                  continue;
          last = bpf_get_prandom_u32();
          found = true;
  }
  if (found)
          bpf_printk("%llu", last);

The state that enters has found = 0 and no 'last'.  The state that comes
around has found = 1 and 'last'.  They are widened into found = [0,1] with
'last' NOT_INIT, and the load fails with "R7 !read_ok".  The old verifier
accepted this program.  Outside a loop, flow_unset_differs() keeps such
states apart while flow_has_room() is true, but at a loop head it is not
consulted.

The path is bpf_flow_deliver() -> flow_add() -> flow_merge(env,
&flow->states[insn_idx], &s, backedge ? FLOW_CYCLE : cycle).  flow.c is not
built at this commit, and the path becomes reachable in 58e4d8b43844 ("bpf:
Verify programs as a fixpoint over the control flow graph").

This is fixed later in the series by 37b41439b09a ("bpf: Don't widen away a
register that only the loop sets"), which changes the condition to:

  if (flow_has_room(env, *nr_states) &&
      (flow_unset_differs(&t->st, &s->st) ||
       (!widen && flow_join_makes_up(flow, &t->st, &s->st))))
          continue;

Should that change be folded into this commit, so that no commit in the
series rejects such programs?

Regarding the commit message:

>     A state is dropped only if a state of the set covers it. Thus the verified
>     states cover each execution of the prog.

The commit message says a state is dropped only if a state of the set covers
it.  flow_merge() also drops states through flow_sweep_trips() ->
flow_sweep_trip(), and no state covers those:

  static void flow_sweep_trip(struct bpf_verifier_env *env, struct flow_state *t,
                              const struct flow_state *s)
  {
          if (t->pending || !flow_earlier_trip(&t->st, &s->st))
                  return;
          flow_unlink(env, t);
          flow_free_state(env, t);
  }

This removes states of earlier iterations of a loop whose iterations are kept
apart.  The coverage argument still holds, because only states that are no
longer pending are swept, so their successors were already explored.  But the
changelog does not mention this second way to drop a state, or the
not-pending condition that makes it safe.  Could the changelog mention it?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 15/45] bpf: Move states along edges, into loops and subprogs and out of them

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest test_verif_scale1 has these lines 90 times. jhash() is static:

  ptr = data + i;
  if (ptr + nh_off > data_end) break;
  ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++);

The 90th call is reached with i = 0 .. 89, in 90 states. All calls: 4095.
jhash() sees one of 90 pointers. If a state in jhash() has the frames of
its callers, as today, states of different calls have no join:
"more than 1024 states that cannot be joined at insn 903". Thus a state in
a static subprog has one frame, and the caller's state waits at the call:

  bpf_flow_call()   saves the caller. The entry state goes to the set at
                    the first insn of the subprog, like any state.
  bpf_flow_exit()   a state at exit goes to all calls that wait for its
                    entry state. The exit states are a set too.

When the merge drops or joins an entry state, its calls wait for the state
that took its place, and flow_do_united() gives them its exit states. 4005
of the 4095 calls end so: 50765 insns, today 530362. Else the limit stops
this test, 100 progs of the schedulers of rust-bpf and 124 of its tests.

The exit states of an entry are joined like the states at an insn, and the
joins must end. Thus an exit state that was joined more than 16 times is
widened. The entry and the exits of a recursive subprog are widened at
once.

Loops. Scalar evolution derives the count and the ranges of a loop from the
state that enters it. Thus the states that enter are a set of their own.
A copy of each new one, with these ranges, goes to the set at the head.
When a walk ends before the loop does, one state stands for the rest:
flow_end_walk() joins, with widening, the state that came back to the head
and the state one iteration before. They differ in what the loop changes.
Without it the schedulers of rust-bpf take 9 % more insns.

bpf_flow_deliver() sends a state along an edge, bpf_flow_next() gives the
next new state. An added may_goto is an edge that the prog does not have:
bpf_flow_guard_edge() sends a state along it only if the loop has may_goto
by now. Else it drops the state and notes that. If the loop gets may_goto
later, the prog is verified again.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 391 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 391 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 41e01bb33fb0..c4a6384d34bc 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -2455,3 +2455,394 @@ static struct flow_state *flow_arrive(struct bpf_verifier_env *env, struct bpf_v
 	bpf_clear_singular_ids(env, &s->st);
 	return s;
 }
+
+/* Is that where may_goto in front of @insn_idx goes, as far as it's known? */
+static bool flow_guard_edge_is_real(struct bpf_verifier_env *env, int insn_idx, u8 edge)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+
+	return aux[bpf_loop_at_index(env, insn_idx)].loop->guarded &&
+	       aux[insn_idx].guard_no_unwind == (edge == BPF_GUARD_LEAVES_LOOP);
+}
+
+/*
+ * @st gets to @insn_idx if the loop has may_goto in front of @prev_insn_idx and that is where it
+ * goes, see check_loop_guard(). If that is not so, or not known yet, @st is dropped. If it turns
+ * out to be so later all is done again.
+ */
+int bpf_flow_guard_edge(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			int insn_idx, int prev_insn_idx, enum bpf_guard_edge edge)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+
+	if (flow_guard_edge_is_real(env, prev_insn_idx, edge)) {
+		aux[prev_insn_idx].loop_guard = true;
+		return bpf_flow_deliver(env, st, insn_idx, prev_insn_idx);
+	}
+	/* it's not going to leave the subprog any more */
+	if (edge == BPF_GUARD_LEAVES_SUBPROG && aux[prev_insn_idx].guard_no_unwind)
+		return 0;
+	aux[prev_insn_idx].guard_edge_dropped = true;
+	aux[bpf_loop_at_index(env, prev_insn_idx)].loop->no_guard_edges = true;
+	return 0;
+}
+
+/*
+ * @s came back to the head of a loop, and the walk through the iterations ends, see
+ * bpf_flow_loop_enter(). From now on a state stands for many iterations. The first such state is
+ * the join of @s with the state that it comes from, one iteration ago. Both were seen, and what
+ * differs between them is what the loop changes. If they have no join @s goes on as it is.
+ * A state of another walk, which began with another state, has nothing to do with @s.
+ */
+static int flow_end_walk(struct bpf_verifier_env *env, struct flow_set *set, struct flow_state *s)
+{
+	struct bpf_func_state *frame = s->st.frame[s->st.curframe];
+	struct loop_stack_entry *entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	u32 top = frame->loop_stack_cnt - 1, walk = entry->walk_id;
+	struct bpf_flow *flow = env->flow;
+	int insn_idx = s->st.insn_idx, err;
+	struct flow_state *t;
+
+	list_for_each_entry_reverse(t, &set->list, node) {
+		/*
+		 * The rules that hold for what comes around a loop, see flow_merge(). And one
+		 * more: the iteration has set a register, see flow_unset_differs(). The join
+		 * would be without it.
+		 */
+		if (flow_trips_behind(&t->st, &s->st) ||
+		    t->st.frame[t->st.curframe]->loop_stack[top].walk_id != walk ||
+		    (flow_has_room(env, flow->nr_states[insn_idx]) &&
+		     (flow_unset_differs(&t->st, &s->st) ||
+		      flow_needed_consts_differ(flow, FLOW_NEED_STACK_OFF, &t->st, &s->st) ||
+		      flow_needed_consts_differ(flow, FLOW_NEED_PTR_OFF, &t->st, &s->st))))
+			continue;
+		err = bpf_states_join(env, &flow->tmp, &t->st, &s->st, true);
+		if (err < 0)
+			return err;
+		if (err)
+			continue;
+		err = flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+		if (err)
+			return err;
+		flow->tmp.flow_entry = t->st.flow_entry;
+		flow->tmp.last_insn_idx = s->prev_insn_idx;
+		bpf_free_verifier_state(&s->st, false);
+		s->st = flow->tmp;
+		memset(&flow->tmp, 0, sizeof(flow->tmp));
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "from %d to %d: widened\n", s->prev_insn_idx, insn_idx);
+		break;
+	}
+
+	frame = s->st.frame[s->st.curframe];
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	entry->guarded = env->insn_aux_data[insn_idx].loop->guarded;
+	entry->apart = false;
+	entry->probe = false;
+	entry->decided = false;
+	entry->terminates = false;
+	entry->trip++;
+	entry->max_header_count = 0;
+	return 0;
+}
+
+/*
+ * @s goes into the set of the states at its insn. It is taken.
+ * @enters: it enters a loop.
+ */
+static int flow_add(struct bpf_verifier_env *env, struct flow_state *s, enum flow_cycle cycle,
+		    bool enters)
+{
+	struct bpf_verifier_state *cur = env->cur_state;
+	int cur_idx = env->insn_idx, insn_idx = s->st.insn_idx, err;
+	const struct loop_stack_entry *entry;
+	const struct bpf_func_state *frame;
+	bool backedge = false, drop = false, ends_walk = false;
+	struct bpf_flow *flow = env->flow;
+
+	env->cur_state = &s->st;
+	env->insn_idx = insn_idx;
+	err = bpf_flow_loop_enter(env, &backedge, &drop, &ends_walk);
+	env->cur_state = cur;
+	env->insn_idx = cur_idx;
+	if (!err && ends_walk)
+		err = flow_end_walk(env, &flow->states[insn_idx], s);
+	if (!err && enters) {
+		frame = s->st.frame[s->st.curframe];
+		entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+		/* it has a count, see bpf_flow_loop_gets_guards() */
+		if (entry->terminates && !entry->guarded && !bpf_loop_walk_may_give_up(entry))
+			env->insn_aux_data[insn_idx].loop->no_guard_edges = true;
+	}
+	if (err || drop) {
+		flow_free_state(env, s);
+		return err;
+	}
+
+	err = flow_merge(env, &flow->states[insn_idx], &s, backedge ? FLOW_CYCLE : cycle);
+	if (err)
+		return err < 0 ? err : 0;
+	flow_set_pending(flow, s);
+	return 0;
+}
+
+/*
+ * @st gets to @insn_idx. What it holds is taken and @st is left empty.
+ */
+int bpf_flow_deliver(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+		     int insn_idx, int prev_insn_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct bpf_subprog_info *sub;
+	struct flow_state *s, *e;
+	enum flow_cycle cycle;
+	bool enters = false;
+	int err;
+
+	s = flow_arrive(env, st, insn_idx, prev_insn_idx, &enters);
+	if (IS_ERR(s))
+		return PTR_ERR(s);
+
+	/*
+	 * Cycles that have no loop head: the callback that came back to the helper
+	 * that calls it, and calls that go through the entry of a function.
+	 */
+	sub = bpf_find_containing_subprog(env, insn_idx);
+	cycle = FLOW_NO_CYCLE;
+	if (sub->recursive && sub->start == insn_idx)
+		cycle = FLOW_CYCLE;
+	else if (env->insn_aux_data[insn_idx].calls_callback && prev_insn_idx >= 0 &&
+		 env->prog->insnsi[prev_insn_idx].code == (BPF_JMP | BPF_EXIT))
+		cycle = FLOW_CALLBACK_CYCLE;
+
+	/*
+	 * What is known about a loop is derived from the state that enters it. So
+	 * the states that enter are a set of their own, and what gets to
+	 * the set at the head of the loop is derived from what is new in it.
+	 */
+	if (enters) {
+		err = flow_merge(env, &flow->enter[insn_idx], &s, cycle);
+		if (err)
+			return err < 0 ? err : 0;
+		cycle = FLOW_NO_CYCLE;
+		e = s;
+		s = flow_new_state(env, prev_insn_idx);
+		if (!s)
+			return -ENOMEM;
+		err = bpf_copy_verifier_state(&s->st, &e->st);
+		if (err) {
+			flow_free_state(env, s);
+			return err;
+		}
+	}
+	return flow_add(env, s, cycle, enters);
+}
+
+static void flow_free_call(struct flow_state *w)
+{
+	list_del(&w->node);
+	bpf_free_verifier_state(&w->st, false);
+	kfree(w);
+}
+
+/* @x came out of the function that the call @w waits for. @w may be gone afterwards. */
+static int flow_apply_exit(struct bpf_verifier_env *env, struct flow_state *w,
+			   struct flow_exit *x)
+{
+	int call_idx = w->st.insn_idx, err;
+	struct bpf_verifier_state st = {};
+	struct flow_state *s;
+
+	err = bpf_copy_verifier_state(&st, &w->st);
+	if (!err)
+		err = bpf_flow_return(env, &st, &x->st, call_idx);
+	if (!err)
+		err = bpf_flow_deliver(env, &st, call_idx + 1, call_idx);
+	bpf_free_verifier_state(&st, false);
+	if (err != 1)
+		return err;
+
+	/*
+	 * The call is verified in the context of the caller: it doesn't wait any more, it is
+	 * done again. Not with bpf_flow_deliver(): the state was at the insn before, it may
+	 * be there still.
+	 */
+	s = flow_new_state(env, w->st.last_insn_idx);
+	if (!s)
+		return -ENOMEM;
+	s->st = w->st;
+	memset(&w->st, 0, sizeof(w->st));
+	flow_free_call(w);
+	flow_shape_ctx(&s->st, &s->shape);
+	flow_link(env, &env->flow->states[call_idx], s);
+	flow_set_pending(env->flow, s);
+	return 0;
+}
+
+/* What came out for the set of @from goes to the calls that wait for the set of @to */
+static int flow_apply_exits(struct bpf_verifier_env *env, struct flow_entry *from,
+			    struct flow_entry *to)
+{
+	struct flow_state *w, *wn;
+	struct flow_exit *x;
+	int err;
+
+	list_for_each_entry(x, &from->exits, node) {
+		list_for_each_entry_safe(w, wn, &to->calls, node) {
+			err = flow_apply_exit(env, w, x);
+			if (err)
+				return err;
+		}
+	}
+	return 0;
+}
+
+static int flow_do_united(struct bpf_verifier_env *env)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_entry *a, *b;
+	struct flow_united u;
+	int err;
+
+	while (flow->united_cnt) {
+		u = flow->united[--flow->united_cnt];
+		a = flow_entry_find(flow, u.a);
+		b = flow_entry_find(flow, u.b);
+		if (a == b)
+			continue;
+		err = flow_apply_exits(env, a, b) ?: flow_apply_exits(env, b, a);
+		if (err)
+			return err;
+		list_splice_tail_init(&a->calls, &b->calls);
+		list_splice_tail_init(&a->exits, &b->exits);
+		a->up = b;
+	}
+	return 0;
+}
+
+/*
+ * The call at @call_idx in @caller of a function that is verified on its own,
+ * @entry is the state at its first insn. Both are taken.
+ */
+int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		  struct bpf_verifier_state *entry, int subprog, int call_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *w;
+	int id;
+
+	w = kzalloc_obj(*w, GFP_KERNEL_ACCOUNT);
+	if (!w)
+		return -ENOMEM;
+	w->st = *caller;
+	memset(caller, 0, sizeof(*caller));
+	w->st.insn_idx = call_idx;
+
+	id = flow_entry_new(flow);
+	if (id < 0) {
+		bpf_free_verifier_state(&w->st, false);
+		kfree(w);
+		return id;
+	}
+	list_add_tail(&w->node, &flow->entries[id]->calls);
+
+	entry->flow_entry = id;
+	if (env->log.level & BPF_LOG_LEVEL2)
+		verbose(env, "call at %d waits for func#%d\n", call_idx, subprog);
+	return bpf_flow_deliver(env, entry, env->subprog_info[subprog].start, call_idx);
+}
+
+/* @st came to exit of the function that is verified on its own. It is taken. */
+int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_func *f = &flow->funcs[st->frame[0]->subprogno];
+	u32 id = st->flow_entry, cnt = 0;
+	struct flow_exit *x, *o, *on;
+	struct flow_state *w, *wn;
+	struct flow_entry *entry;
+	u16 joins = 0;
+	int err;
+
+	st->insn_idx = exit_idx;
+	err = bpf_clean_verifier_state(env, st);
+	if (err)
+		return err;
+	if (f->exit_idx < 0)
+		f->exit_idx = exit_idx;
+	st->insn_idx = f->exit_idx;
+	entry = flow_entry_find(flow, id);
+
+	list_for_each_entry_safe(o, on, &entry->exits, node) {
+		if (bpf_state_subsumes(env, &o->st, st))
+			return 0;
+		if (flow_has_room(env, ++cnt) && flow_join_makes_up(flow, &o->st, st))
+			continue;
+		/* what comes out of a recursive function gets into it again */
+		err = bpf_states_join(env, &flow->tmp, &o->st, st,
+				      env->subprog_info[st->frame[0]->subprogno].recursive ||
+				      o->joins > BPF_FLOW_MAX_JOINS);
+		if (err < 0)
+			return err;
+		if (err)
+			continue;
+		joins = max(joins, o->joins) + 1;
+		bpf_free_verifier_state(st, false);
+		*st = flow->tmp;
+		memset(&flow->tmp, 0, sizeof(flow->tmp));
+		st->flow_entry = id;
+		list_del(&o->node);
+		bpf_free_verifier_state(&o->st, false);
+		kfree(o);
+	}
+
+	x = kzalloc_obj(*x, GFP_KERNEL_ACCOUNT);
+	if (!x)
+		return -ENOMEM;
+	x->st = *st;
+	memset(st, 0, sizeof(*st));
+	x->joins = joins;
+	list_add_tail(&x->node, &entry->exits);
+
+	list_for_each_entry_safe(w, wn, &entry->calls, node) {
+		err = flow_apply_exit(env, w, x);
+		if (err)
+			return err;
+	}
+	return 0;
+}
+
+/*
+ * Make the next pending state the current one. env->cur_state has to be empty.
+ * Returns -ENOENT when nothing is pending.
+ */
+int bpf_flow_next(struct bpf_verifier_env *env, int *prev_insn_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *s;
+	struct rb_node *n;
+	int err;
+
+	do {
+		err = flow_do_united(env);
+		if (err)
+			return err;
+	} while (flow->united_cnt);
+	n = rb_first_cached(&flow->pending);
+	if (!n)
+		return -ENOENT;
+	s = rb_entry(n, struct flow_state, rb);
+	rb_erase_cached(&s->rb, &flow->pending);
+	s->pending = false;
+
+	env->insn_idx = s->st.insn_idx;
+	*prev_insn_idx = s->prev_insn_idx;
+	if (test_bit(s->st.insn_idx, flow->keep))
+		return bpf_copy_verifier_state(env->cur_state, &s->st);
+	*env->cur_state = s->st;
+	memset(&s->st, 0, sizeof(s->st));
+	s->st.insn_idx = env->insn_idx;
+	flow_unlink(env, s);
+	flow_free_state(env, s);
+	return 0;
+}
-- 
2.55.0

[PATCH bpf-next v3 16/45] bpf: Make the state after a call from the caller's state and an exit state

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

match_one() of scx_layered runs in three nested bpf_for() loops and calls
a static subprog, get_pid_nr_ns():

  scoped_guard(rcu) { ... nspid = get_pid_nr_ns(p_pid, pid_ns);

It is verified for the states at its first insn, not for each chain of
calls. One exit state is the result for all calls that wait:

  2: call f  --- { r1=scalar } --->  +---------------------+
  7: call f  --- { r1=scalar } --->  | f: verified 1 time  |
  3: ...     <-- { r0=[0,7] } -----  |                     |
  8: ...     <-- { r0=[0,7] } -----  +---------------------+

That works only if calls that differ in ids give the same entry state. Here
the callers hold three iterators, and each bpf_iter_num_new() has a new id.
What is held is a part of the entry state: it decides what the subprog may
do. In the subprog the id of refs[i] of the caller becomes a name,
BPF_INHERITED_ID + i. A name stands for one object of the caller. Thus
check_ids() and the join do not rename it: it is equal only to itself.

If a call with something held keeps the frames of the callers instead,
layered_runnable takes 5219 insns, not 4459, and recursion has no end.
core::fmt of Rust is recursive, and progs format a message while they hold
a reference. 27 progs of the schedulers of rust-bpf fail:
"the call stack of 33 frames is too deep".

bpf_flow_return() makes the state after the call:
 - r0, and r2 of a pair, come from the exit state. r1-r5 and the outgoing
   stack arguments are not set.
 - What is held is what the caller held and what the subprog took. The
   names become the ids of the caller again.
 - All other ids get new values for each call. If not, two calls in
   sequence return the same object.

Selftest test_spin_lock, lock_static_subprog_unlock():

  bpf_spin_lock(&lockA);
  ret = static_subprog_unlock(ctx);

An exit state does not tell what a release does to the caller. Thus the
frame counts what it holds without interruption since its entry. If that is
less than the caller held, the function returns 1, and the call is verified
again with the frames of the caller. Also: rcu_read_lock, preempt_lock. The
same if it changed the packet through a function that is its argument.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  18 +++++
 kernel/bpf/join.c            |   3 +
 kernel/bpf/states.c          |   4 +
 kernel/bpf/verifier.c        | 138 +++++++++++++++++++++++++++++++++++
 4 files changed, 163 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 53507aed9f20..1963a079589d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1092,6 +1092,8 @@ struct bpf_subprog_info {
 	 * a hidden pointer to the result, a pair in two registers, no dead arguments.
 	 */
 	bool stack_args_from_code: 1;
+	/* it releases what its caller holds, so it's verified in the context of such caller */
+	bool releases_inherited: 1;
 	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
 	bool bail_unwinds: 1;
 	u8 code_stack_args;
@@ -1124,6 +1126,20 @@ struct bpf_id_pair {
 	u32 cur;
 };
 
+/*
+ * A function that is verified on its own, see func_is_on_its_own(), has no ids of its caller.
+ * It has names for the ids of what is held: BPF_INHERITED_ID + i is the id of refs[i] at the call.
+ * - A name depends only on the place. Each id in the state at the entry is replaced with its name.
+ *   The ids of the caller can be names that it got from its own caller: they are replaced too.
+ * - A name stands for the same thing in all states of the function. It is equal only to itself:
+ *   check_ids() and jid_note() do not rename it. So two calls share a state at the entry, and what
+ *   comes out for it, only if the same places have the same names.
+ * - bpf_flow_return() puts the ids of the call that waits back. All other ids get new values.
+ * - env->id_gen stays below BPF_MAX_ID. So no id is a name or has a bit of BPF_ADD_CONST.
+ */
+#define BPF_INHERITED_ID	0x20000000
+#define BPF_MAX_ID		(BPF_INHERITED_ID / 2)
+
 struct bpf_idmap {
 	u32 tmp_id_gen;
 	u32 cnt;
@@ -1605,6 +1621,8 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
 		    bool widen);
 void bpf_join_free(struct bpf_verifier_env *env);
 
+int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		    const struct bpf_verifier_state *exit, int call_idx);
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 4da051767e1d..0110e5f85003 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -84,6 +84,9 @@ static void jid_note(struct bpf_join_state *js, u32 ida, u32 idb, bool consisten
 
 	if (!a || !b)
 		return;
+	/* a name of what the caller has, see BPF_INHERITED_ID */
+	if (a != b && (a | b) >= BPF_INHERITED_ID)
+		consistent = false;
 	for (i = 0; i < js->npairs; i++) {
 		p = &js->pairs[i];
 		if (p->a == a && p->b == b) {
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 4136a708a78e..9d3a86b5820e 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -350,6 +350,10 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
 	if (old_id == 0) /* cur_id == 0 as well */
 		return true;
 
+	/* a name of what the caller has, see BPF_INHERITED_ID */
+	if (((old_id | cur_id) & ~BPF_ADD_CONST) >= BPF_INHERITED_ID)
+		return old_id == cur_id;
+
 	for (i = 0; i < idmap->cnt; i++) {
 		if (map[i].old == old_id)
 			return map[i].cur == cur_id;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 07a9097e3d4e..d8543eb8bb19 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11748,8 +11748,146 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return 0;
 }
 
+
 static int process_bpf_exit_full(struct bpf_verifier_env *env,
 				 bool *do_print_state, bool exception_exit);
+static void account_current_path(struct bpf_verifier_env *env);
+
+/*
+ * The function can be verified on its own, for the state at its entry, when
+ * what it does depends on its arguments only and it cannot get to what its
+ * callers have: nothing points to their stack.
+ *
+ * What the callers hold, references, locks and so on, is a part of that state:
+ * it matters for what the function may do. Calls that differ in ids only
+ * have to get to the same state, so the ids of what is held are replaced with
+ * BPF_INHERITED_ID + the position for the function, and put back at the return.
+ */
+
+static u32 fresh_id(struct bpf_verifier_env *env, struct bpf_idmap *idmap, u32 id)
+{
+	u32 flags = id & BPF_ADD_CONST, i;
+
+	id &= ~BPF_ADD_CONST;
+	if (!id)
+		return 0;
+	for (i = 0; i < idmap->cnt; i++)
+		if (idmap->map[i].old == id)
+			return idmap->map[i].cur | flags;
+	if (idmap->cnt == BPF_ID_MAP_SIZE)
+		return 0;
+	idmap->map[idmap->cnt].old = id;
+	idmap->map[idmap->cnt].cur = ++env->id_gen;
+	return idmap->map[idmap->cnt++].cur | flags;
+}
+
+/*
+ * @caller waits at the call at @call_idx and @exit is what came out of the
+ * function. Make @caller what it is after the call.
+ * Returns 1 if the call has to be verified in the context of the caller.
+ */
+int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		    const struct bpf_verifier_state *exit, int call_idx)
+{
+	struct bpf_func_state *frame = caller->frame[caller->curframe];
+	const struct bpf_func_state *callee = exit->frame[0];
+	struct bpf_idmap *idmap = &env->idmap_scratch;
+	u32 caller_rcu = caller->active_rcu_locks, caller_preempt = caller->active_preempt_locks;
+	struct bpf_reference_state *held;
+	struct bpf_reg_state *reg;
+	u32 i, j, nregs, nheld, kept = 0, id;
+	int err;
+
+	if (exit->unwound && !bpf_is_unwind_goto(&env->prog->insnsi[call_idx + 1])) {
+		verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
+			call_idx);
+		return -EINVAL;
+	}
+	caller->unwound = exit->unwound;
+
+	/*
+	 * bpf_check_cfg() takes the subprog that loads the address of a function for the
+	 * one that calls it. This one got the address from its caller. What it means for
+	 * the packet pointers of the caller is found out in the context of the caller.
+	 */
+	if (exit->changed_pkt_data) {
+		env->subprog_info[callee->subprogno].changes_pkt_data = true;
+		return 1;
+	}
+
+	for (i = 0; i < CALLER_SAVED_REGS; i++)
+		bpf_mark_reg_not_init(env, &frame->regs[caller_saved[i]]);
+	for (i = 0; i < frame->out_stack_arg_cnt; i++)
+		bpf_mark_reg_not_init(env, &frame->stack_arg_regs[i]);
+
+	/* what is held now is what was held at the call and what the function took */
+	held = caller->refs;
+	nheld = caller->acquired_refs;
+	caller->refs = NULL;
+	caller->acquired_refs = 0;
+	err = copy_reference_state(caller, exit);
+	if (err) {
+		kfree(held);
+		return err;
+	}
+
+	/*
+	 * The same state comes out for every call. Ids in it have to be new
+	 * every time, or two calls in a row return the same object.
+	 */
+	idmap->cnt = 0;
+	for (i = 0; i < nheld; i++) {
+		idmap->map[idmap->cnt].old = BPF_INHERITED_ID + i;
+		idmap->map[idmap->cnt++].cur = held[i].id;
+	}
+	/*
+	 * What is taken is added at the end. What is released leaves no hole. Only what the
+	 * function held all the time is kept: the state at its exit looks the same when it
+	 * released a lock and took it again, or took another lock that has no id either.
+	 */
+	for (i = 0; i < nheld && i < callee->entry_refs; i++) {
+		/* see inherited_id() */
+		for (j = 0; held[j].id != held[i].id; j++)
+			;
+		id = held[i].id ? BPF_INHERITED_ID + j : 0;
+		if (caller->refs[i].id != id || caller->refs[i].type != held[i].type)
+			break;
+		caller->refs[i] = held[i];
+		kept++;
+	}
+	kfree(held);
+	/* What it means for the caller is found out in the context of the caller. */
+	if (kept != nheld || callee->entry_rcu_locks < caller_rcu ||
+	    callee->entry_preempt_locks < caller_preempt) {
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "func#%d releases what its caller holds at insn %d\n",
+				callee->subprogno, call_idx);
+		env->subprog_info[callee->subprogno].releases_inherited = true;
+		return 1;
+	}
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		reg = &frame->regs[ret_regs[i]];
+		*reg = callee->regs[ret_regs[i]];
+		reg->id = fresh_id(env, idmap, reg->id);
+		reg->parent_id = fresh_id(env, idmap, reg->parent_id);
+		reg->map_uid = fresh_id(env, idmap, reg->map_uid);
+	}
+	/* those that were held are the first */
+	for (i = nheld; i < caller->acquired_refs; i++) {
+		caller->refs[i].id = fresh_id(env, idmap, caller->refs[i].id);
+		if (caller->refs[i].type == REF_TYPE_PTR)
+			caller->refs[i].parent_id = fresh_id(env, idmap, caller->refs[i].parent_id);
+	}
+	caller->active_irq_id = fresh_id(env, idmap, caller->active_irq_id);
+	caller->active_lock_id = fresh_id(env, idmap, caller->active_lock_id);
+	/* fresh_id() had no room for an id, or there are no ids left */
+	if (idmap->cnt == BPF_ID_MAP_SIZE || env->id_gen >= BPF_MAX_ID) {
+		verbose(env, "too many ids at the call at insn %d\n", call_idx);
+		return -E2BIG;
+	}
+	return 0;
+}
 
 /*
  * Call of a static subprog. The callee is verified in the context of
-- 
2.55.0

Re: [PATCH bpf-next v3 16/45] bpf: Make the state after a call from the caller's state and an exit state

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing exit->speculative propagation in bpf_flow_return()
--

commit 7ddf7363f0596e710519469bc5585f001bf30b01
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Make the state after a call from the caller's state and an exit state

This commit introduces bpf_flow_return() and BPF_INHERITED_ID to construct
the caller's post-call verifier state from the waiting caller state and the
callee's exit state when verifying a static subprogram on its own. Held
reference IDs are mapped to positional names at subprog entry and restored
or refreshed upon return, falling back to caller-context verification if
inherited references or locks were released.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 07a9097e3d4e1..d8543eb8bb195 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -11748,8 +11748,146 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins

[ ... ]

> +int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
> +		    const struct bpf_verifier_state *exit, int call_idx)
> +{
> +	struct bpf_func_state *frame = caller->frame[caller->curframe];
> +	const struct bpf_func_state *callee = exit->frame[0];
> +	struct bpf_idmap *idmap = &env->idmap_scratch;
> +	u32 caller_rcu = caller->active_rcu_locks, caller_preempt = caller->active_preempt_locks;
> +	struct bpf_reference_state *held;
> +	struct bpf_reg_state *reg;
> +	u32 i, j, nregs, nheld, kept = 0, id;
> +	int err;
> +
> +	if (exit->unwound && !bpf_is_unwind_goto(&env->prog->insnsi[call_idx + 1])) {
> +		verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
> +			call_idx);
> +		return -EINVAL;
> +	}
> +	caller->unwound = exit->unwound;

[Severity: High]
Should bpf_flow_return() also propagate exit->speculative to
caller->speculative here?

When !env->bypass_spec_v1, a non-speculative caller entering a static
subprogram verified on its own has caller->speculative == false. Inside the
subprogram, a statically predicted conditional branch or pointer ALU
operation can spawn a speculative path via sanitize_speculative_path() ->
push_stack(..., true) with speculative = true.

When that speculative path reaches BPF_EXIT, process_bpf_exit_full() ->
exit_func_on_its_own() -> bpf_flow_exit() -> flow_apply_exit() calls
bpf_flow_return() with exit->speculative == true and caller->speculative ==
false, and then delivers the resulting state to call_idx + 1 with
caller->speculative still false.

Because the post-call state loses its speculative flag:

1. sanitize_mark_insn_seen() marks instructions reached only along the
   callee's speculative exit path as seen, preventing sanitize_dead_code()
   from patching architecturally dead instructions after the call.

2. If the speculative return value in R0/R2 triggers an unsafe operation in
   the caller that would normally be recovered with a speculation barrier
   via error_recoverable_with_nospec(err) && state->speculative in
   do_check_block(), state->speculative is false, so no nospec barrier is
   inserted and the program is rejected.

3. flow_merge() and bpf_states_join() can join this speculative return state
   with the architectural return state, polluting the caller's architectural
   return register bounds with speculative values.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=16

[PATCH bpf-next v3 17/45] bpf: Enter and leave loops in the main pass, build flow.c

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The main pass joins the states that meet at an insn. At the head of a loop
with a join alone real progs are rejected. bpf_flow_loop_enter() has three
methods for a state that enters a loop, and tries them in this order:
1. Count. scx_lavd/lavd_dispatch:
  for (int j = 0; j < LAVD_CPDOM_MAX_NR; j++) {
      ...
      cpdom_id = get_neighbor_id(cpdomc, i, j);
Scalar evolution finds at most 128 iterations. The state gets j = [0, 127]
and covers all of them: one pass through the body, no may_goto, as today.

2. Walk: each iteration has its own state. Selftest netif_receive_skb:
  for (i = 0; i < len; i++) {
      delta = s1[i] - s2[i];
      if (delta || s1[i] == 0 || s2[i] == 0)
          break;
The compiler tests 'i - 9 < 247', unsigned: no count. After a join i is
unknown: "math between map_value pointer and register with unbounded .."
In a walk i is a constant and decides its test.
flow_try_walk() starts a walk. It goes on while each iteration but the
first decides a jump that leaves the loop, for at most 4096 insns, or two
iterations of a longer loop. A loop with calls gets two iterations: its
size says nothing about the subprogs that it calls. strobemeta.h with
NO_UNROLL, SUBPROGS and two maps:
  for (int i = 0; i < STROBE_MAX_MAPS; ++i)
      payload_off = read_map_var(cfg, i, ..., data, payload_off);
The compiler makes a flag of i: no count. Only a walk bounds payload_off.
A loop with two entries is walked whatever its jumps do. bpfjailer:
  for (int i = BPFJ_MAX_POD_PER_PID - 1; i >= 0; --i) {
      if (i >= num_pods)
          continue;
The compiler copies the test in front of the loop and jumps into its
middle. Scalar evolution says "unsupported loop: irreducible". With a join
14 progs get a may_goto in a loop of 8 iterations.

3. Join, widen, add may_goto. Test hash_fnv_bytes of rust-bpf:
  for b in bytes {
      h = (h ^ *b as u64).wrapping_mul(0x100000001b3);
The length is unknown and h differs in each iteration. Today that is
"BPF program is too large. Processed 1000001 insn". Now the walk ends in
the second iteration, flow.c widens h to an unknown scalar, and the state
that returns next is covered: 160 insns. That shows that each iteration is
safe, not that the loop ends. Thus the loop gets a may_goto. If there is
no place for one: "loop at insn N never exits".

bpf_flow_loop_leave() removes the loops that an edge leaves. flow.c and
the two functions call each other, so flow.o is built from here on, and
do_check() does not use it yet. A loop that leaks a reference in each
iteration has no fixpoint: check_refs_taken_in_loop() rejects a state that
returns to the head with a reference that no register and no slot holds.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  72 ++++++++++
 kernel/bpf/Makefile          |   2 +-
 kernel/bpf/verifier.c        | 263 +++++++++++++++++++++++++++++++++++
 3 files changed, 336 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 1963a079589d..243715bad1e8 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -364,6 +364,13 @@ struct bpf_loop_iters {
 	struct cnum64 iv_range;
 };
 
+/* Where may_goto that the verifier adds goes, see check_loop_guard() */
+enum bpf_guard_edge {
+	BPF_NO_GUARD_EDGE,
+	BPF_GUARD_LEAVES_SUBPROG,
+	BPF_GUARD_LEAVES_LOOP,
+};
+
 struct loop_stack_entry {
 	struct bpf_verifier_state *entry_state;
 	u32 min_header_count;
@@ -379,6 +386,10 @@ struct loop_stack_entry {
 	u32 apart:1;
 	/* max_header_count follows from the state at the entry, not from patience */
 	u32 counted:1;
+	/* the walk goes on for as long as each iteration decides a jump that leaves the loop */
+	u32 probe:1;
+	/* this one did */
+	u32 decided:1;
 	/*
 	 * How many times the state came back to the head of the loop. With @apart that is
 	 * the iteration that it is in. A state that stands for many iterations is a join, and has
@@ -387,6 +398,12 @@ struct loop_stack_entry {
 	u32 trip;
 	/* the number of references that the state had when it entered the loop */
 	u32 refs;
+	/*
+	 * A walk on patience: which one it is, and how many insns may be looked at in it, see
+	 * env->walk_insns. The size of the loop says nothing about the functions that it calls.
+	 */
+	u32 walk_id;
+	u32 walk_patience;
 	int iv_reg;
 	struct cnum64 iv_range;
 };
@@ -603,6 +620,11 @@ struct bpf_verifier_state {
 	 * and their packet pointers are not in the state, see bpf_flow_return().
 	 */
 	bool changed_pkt_data;
+	/*
+	 * Not zero in a function that is verified on its own: the call that the
+	 * state at its entry came from, see bpf_flow_call().
+	 */
+	u32 flow_entry;
 
 	/*
 	 * First and last insn idx of this verifier state.
@@ -807,8 +829,23 @@ struct bpf_loop {
 	bool exits_overflow;
 	/* @exits of a loop in this one is cut. Then @exits of this loop can lack exits too. */
 	bool nested_exits_overflow;
+	/* every trip goes through an insn that stops the loop, see flow_is_bound() */
+	bool rt_bounded;
+	/* insns in the loop, nested loops included */
+	u32 nr_insns;
+	/*
+	 * a walk on patience whose jumps were all decided, or the two trips of a loop with calls,
+	 * didn't get to the end of the loop: there is no other try
+	 */
+	bool walk_failed;
+	/* functions of the program are called in the loop */
+	bool has_calls;
+	/* not rt_bounded: there are places for may_goto, see guard_to in bpf_insn_aux_data */
+	bool has_guards;
 	/* and it is going to be there, see bpf_flow_loop_gets_guards() */
 	bool guarded;
+	/* a state went through it without the edges of may_goto, see bpf_flow_loop_gets_guards() */
+	bool no_guard_edges;
 	/*
 	 * Loop-entry registers used to compute stack addresses for loads,
 	 * stores and calls requiring fixed stack offsets, in this and nested
@@ -860,6 +897,9 @@ struct bpf_insn_aux_data {
 	DECLARE_BITMAP(may_write_mask, MAX_BPF_STACK_SLOTS);
 	DECLARE_BITMAP(live_stack_before, MAX_BPF_STACK_SLOTS);
 	u16 stack_ptrs; /* bitmask of regs that may hold a frame pointer here (arg_track) */
+	/* regs that hold a number or a pointer to arena; that, or a pointer to a plain map value */
+	u16 plain_regs;
+	u16 value_regs;
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
 	u64 nospec:1; /* do not execute this instruction speculatively */
@@ -877,6 +917,7 @@ struct bpf_insn_aux_data {
 	u64 guard_place:1; /* it's the place for may_goto if the loop needs one */
 	u64 guard_at_header:1; /* there is no better place than the head of the loop */
 	u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
+	u64 guard_edge_dropped:1; /* an edge of may_goto from here was dropped */
 	u64 need_scev:1;
 	u64 alu_state:5; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
@@ -1060,6 +1101,7 @@ struct bpf_subprog_info {
 	bool can_unwind: 1; /* has unwind_exit */
 	bool recursive: 1; /* on a cycle of the call graph */
 	bool in_recursion: 1; /* recursive, or a recursive subprog gets to it */
+	bool addr_taken: 1; /* callx calls nothing else */
 	bool via_callx: 1; /* callx calls it */
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
@@ -1273,6 +1315,10 @@ struct bpf_verifier_env {
 	bool test_state_freq;		/* test verifier with different pruning frequency */
 	bool test_reg_invariants;	/* fail verification on register invariants violations */
 	bool uses_arena_sp;		/* the program has insns with BPF_REG_ARENA_SP */
+	bool restart;			/* see bpf_flow_next_is_no_bound() */
+	bool guards_late;		/* see bpf_flow_loop_gets_guards() */
+	u32 walk_cnt;			/* see loop_stack_entry.walk_id */
+	u32 *walk_insns;		/* the insns that were looked at in each of them */
 	u32 max_arena_stack_depth;	/* of all call chains */
 	struct bpf_verifier_state *cur_state; /* current verifier state */
 	/* Search pruning optimization, array of list_heads for
@@ -1420,6 +1466,7 @@ struct bpf_verifier_env {
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
 	struct scev *scev;
+	struct bpf_flow *flow;
 	struct bpf_join_state *join;
 };
 
@@ -1621,6 +1668,31 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
 		    bool widen);
 void bpf_join_free(struct bpf_verifier_env *env);
 
+/* flow.c */
+int bpf_flow_prepare(struct bpf_verifier_env *env);
+void bpf_flow_reset(struct bpf_verifier_env *env);
+void bpf_flow_free(struct bpf_verifier_env *env);
+bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx);
+struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
+					       const struct bpf_verifier_state *st);
+int bpf_flow_deliver(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+		     int insn_idx, int prev_insn_idx);
+int bpf_flow_guard_edge(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			int insn_idx, int prev_insn_idx, enum bpf_guard_edge edge);
+int bpf_flow_next(struct bpf_verifier_env *env, int *prev_insn_idx);
+int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		  struct bpf_verifier_state *entry, int subprog, int call_idx);
+int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx);
+void bpf_flow_next_is_no_bound(struct bpf_verifier_env *env, int insn_idx, int h);
+bool bpf_flow_loops_lost_bound(struct bpf_verifier_env *env);
+void bpf_flow_loop_gets_guards(struct bpf_verifier_env *env, int h);
+void bpf_flow_new_round(struct bpf_verifier_env *env);
+int bpf_flow_place_guards(struct bpf_verifier_env *env);
+int bpf_flow_compute_kinds(struct bpf_verifier_env *env);
+bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r);
+/* verifier.c, for flow.c */
+int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters);
+int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop, bool *ends_walk);
 int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
 		    const struct bpf_verifier_state *exit, int call_idx);
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 3e0b096cfc9f..ab3ba42596ec 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o flow.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d8543eb8bb19..8c89bf67a842 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21318,6 +21318,269 @@ static int handle_loop_entry_exit(struct bpf_verifier_env *env)
 	return 0;
 }
 
+/*
+ * The current state came back to the head of a loop. If it has a reference more than the states
+ * that are there it is neither within one of them nor joined with one. It goes around again:
+ *	bpf_for(i, 0, n) { if (!p) p = bpf_ringbuf_reserve(...); ... }
+ * That ends when the loop stops taking references, or when there is no place left to keep them.
+ * A reference that no live register and no live stack slot holds cannot be released any more.
+ * There is nothing to wait for.
+ *
+ * Only what was taken in the loop is looked at. New references are added at the end. What callers
+ * hold is not in the state of a function that is verified on its own.
+ */
+static int check_refs_taken_in_loop(struct bpf_verifier_env *env,
+				    const struct loop_stack_entry *entry)
+{
+	u32 mask = (1 << STACK_SPILL) | (1 << STACK_DYNPTR) | (1 << STACK_ITER);
+	struct bpf_verifier_state *st = env->cur_state;
+	struct bpf_stack_state *stack;
+	struct bpf_func_state *frame;
+	struct bpf_reg_state *reg;
+	bool held;
+	int i, j, id;
+
+	for (i = entry->refs; i < st->acquired_refs; i++) {
+		if (st->refs[i].type != REF_TYPE_PTR)
+			continue;
+		id = st->refs[i].id;
+		held = false;
+		for (j = 0; j < st->acquired_refs; j++)
+			if (st->refs[j].type == REF_TYPE_PTR && st->refs[j].parent_id == id)
+				held = true;
+		bpf_for_each_reg_in_vstate_mask(st, frame, reg, stack, mask, ({
+			if (reg->id == id || reg->parent_id == id)
+				held = true;
+		}));
+		if (held)
+			continue;
+		verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", id, st->refs[i].insn_idx);
+		verbose(env, "nothing holds it in the next iteration of the loop at insn %d\n",
+			entry->loop_id);
+		return -EINVAL;
+	}
+	return 0;
+}
+
+/* How many insns it may take to go through the iterations of a loop one by one. */
+#define BPF_FLOW_MAX_WALK	100000
+/* The same when it is not known that the walk gets to the end of the loop. */
+#define BPF_FLOW_MAX_TRY	4096
+
+/* How many times the current insn is looked at because of the loops around it that are walked. */
+static u64 flow_walk_factor(const struct bpf_verifier_state *st)
+{
+	const struct loop_stack_entry *e;
+	u64 factor = 1;
+	u32 fr, i;
+
+	for (fr = 0; fr <= st->curframe; fr++) {
+		for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++) {
+			e = &st->frame[fr]->loop_stack[i];
+			/* the walk of a loop with two entries pays as it goes, see env->walk_insns */
+			if (e->counted || e->probe)
+				factor = min_t(u64, factor * max(e->max_header_count, 1U), U32_MAX);
+		}
+	}
+	return factor;
+}
+
+/*
+ * The current state enters the loop of @entry, for which there is no count. It's found out on the
+ * way whether the loop has an end: the walk goes on for as long as the jumps that leave the loop
+ * are decided. Returns 1 if the iterations are kept apart from now on, 0 if that takes too long.
+ */
+static int flow_try_walk(struct bpf_verifier_env *env, struct loop_stack_entry *entry)
+{
+	struct bpf_loop *loop = env->insn_aux_data[entry->loop_id].loop;
+	u64 factor = flow_walk_factor(env->cur_state);
+	u32 patience, *walk_insns;
+
+	patience = env->allow_uninit_stack && !loop->rt_bounded &&
+		   (!loop->has_calls || loop->has_guards) ? BPF_FLOW_MAX_TRY : 0;
+	if (loop->walk_failed)
+		patience = 0;
+	/* Two trips for a loop of any length. Many a loop is left in the second one. */
+	if (patience && factor == 1)
+		patience = max_t(u64, patience, 2ULL * loop->nr_insns);
+	if (factor * loop->nr_insns > patience)
+		return 0;
+	entry->max_header_count = max(patience / (u32)(factor * loop->nr_insns), 1U);
+	/*
+	 * A compiler makes a flag of the counter of
+	 *	for (i = 0; i < 2; i++) off += f(i);
+	 * and there is nothing to count. In the second trip the flag decides the jump.
+	 * A loop with calls gets these two trips, whatever they take, and no more: its size says
+	 * nothing about the functions that it calls. It gets may_goto at once, so that nothing is
+	 * done again when the second trip comes back to the head.
+	 */
+	if (loop->has_calls) {
+		entry->max_header_count = min(entry->max_header_count, 2U);
+		patience = U32_MAX;
+		bpf_flow_loop_gets_guards(env, entry->loop_id);
+		entry->guarded = true;
+	}
+	entry->terminates = true;
+	/* scalar evolution has nothing to say about a loop with two entries */
+	entry->probe = !loop->irreducible;
+	entry->walk_id = env->walk_cnt + 1;
+	if (is_power_of_2(entry->walk_id)) {
+		walk_insns = kvrealloc(env->walk_insns, 2 * entry->walk_id * sizeof(u32),
+				       GFP_KERNEL_ACCOUNT);
+		if (!walk_insns)
+			return -ENOMEM;
+		env->walk_insns = walk_insns;
+	}
+	env->walk_cnt++;
+	env->walk_insns[entry->walk_id] = 0;
+	entry->walk_patience = patience;
+	entry->apart = true;
+	return 1;
+}
+
+/*
+ * The current state gets to env->insn_idx. Leave the loops that the insn is not in.
+ * @enters: the insn is the head of a loop that the state is not in yet.
+ */
+int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters)
+{
+	struct bpf_func_state *frame = cur_func(env);
+	int err;
+
+	err = loop_stack_pop(env);
+	if (err)
+		return err;
+	*enters = env->insn_aux_data[env->insn_idx].loop &&
+		  (!frame->loop_stack_cnt ||
+		   frame->loop_stack[frame->loop_stack_cnt - 1].loop_id != env->insn_idx);
+	return 0;
+}
+
+/*
+ * Enter the loops that env->insn_idx is in.
+ *
+ * When the number of iterations follows from the state that enters a loop at its head
+ * the state is made to stand for all of them, with the ranges that induction variables
+ * have in that many iterations. The state that comes back to the head is then expected
+ * to be within it.
+ * Otherwise the state is left as it is. What comes back to the head is joined with it.
+ * @ends_walk: the state came back to the head and is the last one that is kept apart from
+ * the states of other iterations.
+ */
+int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop, bool *ends_walk)
+{
+	struct bpf_verifier_state *cur = env->cur_state, *entry_state;
+	const struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *entry;
+	struct bpf_loop_iters iters;
+	struct bpf_loop *loop;
+	bool pushed, decided;
+	u32 trip;
+	u64 per_trip;
+	int err;
+
+	err = loop_stack_push(env, &pushed);
+	if (err)
+		return err;
+
+	if (frame->loop_stack_cnt == 0)
+		return 0;
+
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	if (env->insn_idx != entry->loop_id)
+		return 0;
+
+	/*
+	 * No edge goes from a call to itself: the call goes on with one function of a set, see
+	 * split_func_set(), in the same iteration.
+	 */
+	if (!pushed && cur->last_insn_idx == env->insn_idx &&
+	    BPF_CLASS(insn->code) == BPF_JMP && BPF_OP(insn->code) == BPF_CALL)
+		return 0;
+
+	/* The callback came back to the helper that calls it, in the same iteration. */
+	if (frame->callback_depth)
+		return 0;
+
+	/* If nothing was pushed and env->insn_idx is a loop header, we've taken a backedge. */
+	if (!pushed) {
+		if (!entry->apart) {
+			entry->trip++;
+			*backedge = true;
+			return check_refs_taken_in_loop(env, entry) ?: maybe_clamp_scev_regs(env);
+		}
+		loop = env->insn_aux_data[env->insn_idx].loop;
+		/* the next iteration is another kind of state, there is no cycle */
+		trip = entry->trip;
+		decided = !entry->probe || entry->decided || !trip;
+		entry->decided = false;
+		if (decided && ++entry->trip < entry->max_header_count &&
+		    (entry->counted ||
+		     env->walk_insns[entry->walk_id] <= entry->walk_patience))
+			return 0;
+		/* there is no such iteration at run time */
+		if (entry->counted) {
+			*drop = true;
+			return 0;
+		}
+		/*
+		 * The walk doesn't get to the end of the loop. From now on what comes back to
+		 * the head is joined with what is there. The loop gets may_goto.
+		 */
+		if (!loop->rt_bounded && !loop->has_guards) {
+			verbose(env, "loop at insn %d never exits\n", env->insn_idx);
+			return -EINVAL;
+		}
+		if (!loop->rt_bounded)
+			bpf_flow_loop_gets_guards(env, env->insn_idx);
+		/* the jumps were decided, it's the loop that is too long. Or it had its two trips */
+		if (decided || loop->has_calls)
+			loop->walk_failed = true;
+		/* the iteration that it went through: its state is at the head */
+		entry->trip = trip;
+		*ends_walk = true;
+		return 0;
+	}
+
+	loop = env->insn_aux_data[env->insn_idx].loop;
+	err = bpf_compute_loop_iters(env, cur, &iters);
+	if (err < 0)
+		return err;
+
+	/* @per_trip: the insns of one iteration, times the walks around the loop. */
+	per_trip = flow_walk_factor(cur) * loop->nr_insns;
+	/* The walk has to get to the end of the loop, so it is known to have one. Or not: */
+	if (!err) {
+		err = flow_try_walk(env, entry);
+		if (err)
+			return err < 0 ? err : 0;
+		if (!loop->rt_bounded && !loop->has_guards) {
+			verbose(env, "loop at insn %d never exits\n", env->insn_idx);
+			return -EINVAL;
+		}
+		if (!loop->rt_bounded) {
+			bpf_flow_loop_gets_guards(env, env->insn_idx);
+			entry->guarded = true;
+		}
+		return 0;
+	}
+
+	/* What the ranges are derived from for as long as the loop is looked at. */
+	entry_state = bpf_flow_keep_state(env, cur);
+	if (!entry_state)
+		return -ENOMEM;
+
+	entry->min_header_count = iters.min_header_count;
+	entry->max_header_count = iters.max_header_count;
+	entry->iv_reg = iters.iv_reg;
+	entry->iv_range = iters.iv_range;
+	entry->entry_state = entry_state;
+	entry->terminates = true;
+	err = bpf_widen_scev_regs(env, cur, entry_state, &iters);
+	return err < 0 ? err : 0;
+}
+
 static int do_check(struct bpf_verifier_env *env)
 {
 	bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
-- 
2.55.0

Re: [PATCH bpf-next v3 17/45] bpf: Enter and leave loops in the main pass, build flow.c

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2837 bytes --]

> bpf: Enter and leave loops in the main pass, build flow.c

TL;DR: The allow_uninit_stack (CAP_PERFMON) gate in flow_try_walk() may skip
loop walking for CAP_BPF-only loaders, so their loops always get may_goto
(possible early stop at run time) or are rejected; commit message doesn't say.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +static int flow_try_walk(struct bpf_verifier_env *env, struct loop_stack_entry *entry)
> +{
> +	struct bpf_loop *loop = env->insn_aux_data[entry->loop_id].loop;
> +	u64 factor = flow_walk_factor(env->cur_state);
> +	u32 patience, *walk_insns;
> +
> +	patience = env->allow_uninit_stack && !loop->rt_bounded &&
> +		   (!loop->has_calls || loop->has_guards) ? BPF_FLOW_MAX_TRY : 0;

Is the env->allow_uninit_stack term intended here?  That flag is
bpf_allow_uninit_stack(token), i.e. CAP_PERFMON, and has nothing to do
with loops.  bpf_compute_loop_iters() already returns 0 early when
!env->allow_uninit_stack, so a loader with CAP_BPF but without CAP_PERFMON
gets neither the count nor the walk from bpf_flow_loop_enter().

For every loop that is not rt_bounded, that loader ends up in:

	err = flow_try_walk(env, entry);      /* patience == 0 -> returns 0 */
	...
	if (!loop->rt_bounded) {
		bpf_flow_loop_gets_guards(env, env->insn_idx);
		entry->guarded = true;
	}

and the state is then widened in flow_merge() (FLOW_CYCLE).

Before this series such loaders had bounded loops checked by visiting each
iteration in is_state_visited(), which 6228cec8147e removes.  Once
58e4d8b43844 wires bpf_flow_loop_enter() into do_check(), a CAP_BPF-only
program (a socket filter, or tc/XDP with CAP_BPF+CAP_NET_ADMIN, or a token
that does not delegate CAP_PERFMON) would see two changes:

- Every such loop gets a may_goto, even one that a walk would show to end.
  The comment of bpf_flow_loop_gets_guards() says the may_goto can be out
  of budget in the first trip, so the loop can stop early at run time
  where it did not before.

- A loop that needs the per-iteration constant is rejected.  The
  netif_receive_skb example in the commit message is one: "After a join i
  is unknown: math between map_value pointer and register with unbounded
  ..".  A loop with no place for a may_goto is rejected with "loop at insn
  N never exits".

None of the later commits up to b3ce1aebb13f remove the gate.

Could the commit message say that loops of loaders without CAP_PERFMON now
always get may_goto, if this is intended?

> +	if (loop->walk_failed)
> +		patience = 0;

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 18/45] bpf: Tell the passes over the graph where an added may_goto goes

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

A spin lock in Rust, test atomic_spin_lock of rust-bpf:

  for _ in 0..n {
      while lock.compare_exchange_weak(...).is_err() {
          core::hint::spin_loop();
      }
      data += 1;
      lock.store(false, Release);
  }

The compiler makes one loop with two back edges. The spin does not pass the
test of the counter: no jump that leaves the loop is in each iteration.
Today: "infinite loop detected at insn 41". The only place for may_goto
that each iteration passes is the head. Its target is an exit of all loops
around it. Without that place: "loop at insn 41 never exits". 142 of the
225 progs of the schedulers of rust-bpf have such a loop.

The prog has no edge from the head to the exit. The main pass verifies the
state that goes there. But four passes before it see only the control flow
graph, and the main pass and scalar evolution rely on their results:

  pass                      if it does not know the edge
  ------------------------  -----------------------------------------------
  bpf_compute_const_regs()  r7 == &ro on all edges to the exit, r7 == &rw
                            at the head. 'for (i = 0; i < *r7; i++)' after
                            the exit gets its count from ro. The count is
                            wrong for the state that comes from the head.
  compute_subprog_args()    The same with fp-8 and fp-16: a store through
                            r7 changes a slot that the pass does not know.
  live registers            A register that the loop sets on each path to
                            the exit is dead at the head. The state at the
                            head drops it. A read after the exit is
                            rejected.
  bpf_edge_leaves_loop()    'if i > 255' sets a flag, the way from there
                            out of the loop leads through a jump of a
                            nested loop. may_goto in front of that jump
                            takes its other edge: the count is wrong.

The place and the target depend only on the loops. Thus call
bpf_flow_place_guards() directly after the loops are known. If a loop has
this place bpf_check() calculates the constants again, with nothing known
at the target. compute_subprog_args() joins what it knows at the head into
the target. A register that is live at the target is live at the head.

  do { ..; more = ++i < 256 && v->in[i & 63]; 10 insns; } while (more);
clang keeps the flag: 'if w6 > 255' sets it and stays in the loop. The old
bpf_edge_leaves_loop() lost it at 'w1 &= 1': no count, 15611 insns, and "R7
unbounded memory access" with one state for all iterations. Now it
propagates constants from the edge with the functions of const_fold.c: 112
insns. It has to know the added edges too. It runs after the constants are
calculated again. It does not enter a nested loop: may_goto there goes
where a jump of that loop goes, whatever the constants say.

The edge is there for each loop that can get a may_goto at its head, if it
gets one or not. Thus the passes can know less than before. The loop of
compute_live_registers/may_goto is such a loop: r1 is live at its head.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h                  |   7 +-
 kernel/bpf/const_fold.c                       | 128 +++++++++++++-
 kernel/bpf/liveness.c                         |  41 +++--
 kernel/bpf/loops.c                            | 167 ++++++++----------
 kernel/bpf/scev.c                             |  12 +-
 kernel/bpf/verifier.c                         |  15 ++
 .../bpf/progs/compute_live_registers.c        |   8 +-
 .../selftests/bpf/progs/verifier_loop_latch.c |   6 +-
 8 files changed, 264 insertions(+), 120 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 243715bad1e8..e5bee65807fa 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -810,6 +810,8 @@ struct bpf_backedge {
 	 */
 	int latches[MAX_LATCHES];
 	u8 latches_cnt;
+	/* bit k: the loop is left on the edge of latches[k] that is taken, not on the other one */
+	u8 taken_leaves;
 };
 
 struct bpf_loop_exit {
@@ -976,6 +978,8 @@ struct bpf_insn_aux_data {
 	u16 const_reg_map_mask;
 	u16 const_reg_subprog_mask;
 	u32 const_reg_vals[10];
+	/* bpf_edge_leaves_loop() + 1 for the edge that is not taken and for the one that is */
+	u8 edge_leaves[2];
 	/*
 	 * Header index of the innermost loop containing this instruction, -1 if none.
 	 * For a loop header, identifies the parent loop instead.
@@ -1700,7 +1704,8 @@ u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
-bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
+int bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t, int *budget);
+int bpf_find_latches(struct bpf_verifier_env *env, bool flags);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
 int bpf_split_cur_state(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index eac2bf80274c..bcaec1488fc9 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -250,7 +250,7 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
 	struct const_arg_info ci_out[MAX_BPF_REG];
 	struct bpf_iarray *succ;
 	bool changed;
-	int i, r;
+	int i, r, t;
 
 	/* kvzalloc zeroes memory, so all entries start as CONST_ARG_UNVISITED (0) */
 	ci_in = kvzalloc_objs(*ci_in, insn_cnt, GFP_KERNEL_ACCOUNT);
@@ -264,6 +264,12 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
 		for (r = 0; r < MAX_BPF_REG; r++)
 			ci_in[start][r] = unknown;
 	}
+	/* The same from the head of a loop to where the loop is left, once the loops are known. */
+	for (i = 0; i < insn_cnt; i++) {
+		t = bpf_head_guard_target(env, i);
+		for (r = 0; r < MAX_BPF_REG && t >= 0; r++)
+			ci_in[t][r] = unknown;
+	}
 
 redo:
 	changed = false;
@@ -344,6 +350,126 @@ static int eval_const_branch(u8 opcode, u64 dst_val, u64 src_val)
 	}
 }
 
+/* 1: the registers @ci take the conditional jump @insn, 0: they don't, -1: not known */
+static int const_branch_taken(const struct bpf_insn *insn, const struct const_arg_info *ci)
+{
+	u64 dst = ci[insn->dst_reg].val, src = insn->imm;
+
+	if (!ci_is_const(&ci[insn->dst_reg]))
+		return -1;
+	if (BPF_SRC(insn->code) == BPF_X) {
+		if (!ci_is_const(&ci[insn->src_reg]))
+			return -1;
+		src = ci[insn->src_reg].val;
+	}
+	/* see bpf_prune_dead_branches() */
+	if (BPF_CLASS(insn->code) == BPF_JMP32) {
+		dst = (s32)dst;
+		src = (s32)src;
+	}
+	return eval_const_branch(insn->code, dst, src);
+}
+
+/* The registers in front of the insn @idx on the paths that got there so far */
+struct edge_state {
+	int idx;
+	struct const_arg_info ci[MAX_BPF_REG];
+};
+
+#define EDGE_OPEN	8	/* insns that wait for their turn */
+
+/*
+ * Does the program leave the loop at @h when the conditional jump at @n goes on at @t?
+ * It does when @t is not in the loop. A compiler also makes
+ *	w1 = 0; if w7 > 255 goto l; ...; w1 = 1; l: ...; w1 &= 1; if w1 != 0 goto head;
+ * of 'do { ...; more = ++i < 256 && ...; ...; } while (more);'. The jump that leaves the loop
+ * looks at 1 or 0. The jump that counts stays in the loop on both edges.
+ *
+ * So propagate constants once more, over what can be reached from this edge only. Start with
+ * the constants at @n. A jump that the constants decide has one successor. Two paths that
+ * meet are joined: both started at the edge. If nothing leads to @h the edge leaves the loop.
+ *
+ * The insn that is first in reverse post order goes first. A cycle without @h is a nested
+ * loop, so all that leads to the insn was looked at and every insn has one turn.
+ *
+ * Every edge that the program has at run time must be followed, the edges of may_goto that the
+ * verifier adds too. May_goto of @h and of the loops around it leaves @h. A nested loop can get
+ * one in front of a jump that the constants decide the other way. So don't go into a nested loop.
+ *
+ * @budget: the insns to look at. 0: look at @t only. An edge is followed once: a jump can
+ * dominate more than one back edge. Returns 1, 0, or an error.
+ */
+int bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t, int *budget)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[n];
+	struct bpf_insn *insns = env->prog->insnsi, *insn;
+	int *nums = env->cfg.postorder_nums;
+	int i, l, s, first, taken, cnt = 1;
+	u8 *seen = &aux->edge_leaves[t != n + 1];
+	struct edge_state *open, *cur;
+	struct bpf_iarray *succ;
+	int ret = 0;
+
+	l = bpf_loop_at_index(env, t);
+	if (l != h)
+		return !bpf_is_nested_loop(env, l, h);
+	if (*seen)
+		return *seen - 1;
+	if (t == h || !*budget)
+		return 0;
+
+	open = kcalloc(EDGE_OPEN + 1, sizeof(*open), GFP_KERNEL_ACCOUNT);
+	if (!open)
+		return -ENOMEM;
+	cur = &open[EDGE_OPEN];
+	open[0].idx = t;
+	for (i = 0; i < MAX_BPF_REG; i++) {
+		open[0].ci[i].state = CONST_ARG_UNKNOWN;
+		if (i < ARRAY_SIZE(aux->const_reg_vals) && (aux->const_reg_mask & BIT(i))) {
+			open[0].ci[i].state = CONST_ARG_CONST;
+			open[0].ci[i].val = aux->const_reg_vals[i];
+		}
+	}
+	while (cnt) {
+		for (i = 1, first = 0; i < cnt; i++)
+			if (nums[open[i].idx] > nums[open[first].idx])
+				first = i;
+		*cur = open[first];
+		open[first] = open[--cnt];
+		if (!*budget)
+			goto out;
+		--*budget;
+		insn = &insns[cur->idx];
+		const_reg_xfer(env, cur->ci, insn, insns, cur->idx);
+		taken = bpf_insn_is_cond_jump(insn->code) ? const_branch_taken(insn, cur->ci) : -1;
+		succ = bpf_insn_successors(env, cur->idx);
+		iarray_for_each(s, succ) {
+			if (taken >= 0 && s != cur->idx + 1 + (taken ? insn->off : 0))
+				continue;
+			l = bpf_loop_at_index(env, s);
+			if (s == h || bpf_is_nested_loop(env, l, h))
+				goto out;
+			if (l != h)
+				continue;
+			for (i = 0; i < cnt && open[i].idx != s; i++)
+				;
+			if (i == EDGE_OPEN)
+				goto out;
+			if (i < cnt) {
+				const_reg_join(open[i].ci, cur->ci);
+				continue;
+			}
+			open[cnt] = *cur;
+			open[cnt++].idx = s;
+		}
+	}
+	ret = 1;
+out:
+	*seen = ret + 1;
+	kfree(open);
+	return ret;
+}
+
 /*
  * Rewrite conditional branches with constant outcomes into unconditional
  * jumps using register values resolved by bpf_compute_const_regs() pass.
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 525c0faff1a6..6a2ee407412b 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1054,6 +1054,22 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in
 	return true;
 }
 
+/* What leaves @idx gets to @target */
+static bool arg_track_join_all(struct bpf_verifier_env *env, int idx, int target,
+			       struct arg_track *in, struct arg_track *stack_in,
+			       const struct arg_track *out, const struct arg_track *stack_out,
+			       u32 nslots)
+{
+	bool changed = false;
+	int r;
+
+	for (r = 0; r < MAX_AT_TRACK_REGS; r++)
+		changed |= arg_track_join(env, idx, target, r, &in[r], out[r]);
+	for (r = 0; r < nslots; r++)
+		changed |= arg_track_join(env, idx, target, -r - 1, &stack_in[r], stack_out[r]);
+	return changed;
+}
+
 /*
  * Compute the result when an ALU op destroys offset precision.
  * If a single arg is identifiable, preserve it with OFF_IMPRECISE.
@@ -2069,6 +2085,7 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
 		int i = idx - start;
 		struct bpf_insn *insn = &insns[idx];
 		struct bpf_iarray *succ;
+		int ti;
 
 		if (!arg_is_visited(&at_in[i][0]) && !arg_is_visited(&at_in[i][1]))
 			continue;
@@ -2076,6 +2093,12 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
 		memcpy(at_out, at_in[i], sizeof(at_out));
 		memcpy(at_stack_out, &at_stack_in[(size_t)i * nslots], nslots * sizeof(*at_stack_out));
 
+		ti = bpf_head_guard_target(env, idx) - start;
+		if (ti >= 0)
+			changed |= arg_track_join_all(env, idx, start + ti, at_in[ti],
+						      &at_stack_in[(size_t)ti * nslots],
+						      at_out, at_stack_out, nslots);
+
 		arg_track_xfer(env, insn, idx, at_out, at_stack_out, nslots,
 			       at_stack_arg_entry, instance, callsites);
 		arg_track_log(env, insn, idx, at_in[i], &at_stack_in[(size_t)i * nslots], at_out,
@@ -2085,21 +2108,14 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
 		succ = bpf_insn_successors(env, idx);
 		for (int s = 0; s < succ->cnt; s++) {
 			int target = succ->items[s];
-			int ti;
 
 			/* Filter: stay within the subprogram's range */
 			if (target < start || target >= end)
 				continue;
 			ti = target - start;
-
-			for (r = 0; r < MAX_AT_TRACK_REGS; r++)
-				changed |= arg_track_join(env, idx, target, r,
-							  &at_in[ti][r], at_out[r]);
-
-			for (r = 0; r < nslots; r++)
-				changed |= arg_track_join(env, idx, target, -r - 1,
-							  &at_stack_in[(size_t)ti * nslots + r],
-							  at_stack_out[r]);
+			changed |= arg_track_join_all(env, idx, target, at_in[ti],
+						      &at_stack_in[(size_t)ti * nslots],
+						      at_out, at_stack_out, nslots);
 		}
 	}
 	if (changed)
@@ -2780,6 +2796,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			u32 new_out = 0;
 			u32 new_in = 0;
 			u32 use;
+			int t;
 
 			succ = bpf_insn_successors(env, insn_idx);
 			for (int s = 0; s < succ->cnt; ++s)
@@ -2794,6 +2811,10 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
 				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
 			new_in = (new_out & ~live->def) | use;
+			/* may_goto that is added at the head of a loop goes there */
+			t = bpf_head_guard_target(env, insn_idx);
+			if (t >= 0)
+				new_in |= state[t].in;
 			if (new_out != live->out || new_in != live->in) {
 				live->in = new_in;
 				live->out = new_out;
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 729e0055f86f..c77378b3fbde 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -304,97 +304,35 @@ int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx)
 	return aux[idx].loop ? idx : aux[idx].loop_header;
 }
 
-/* @known: the registers that hold the constant @val behind @insn. False: a jump or the like. */
-static bool track_consts(const struct bpf_insn *insn, u16 *known, s32 *val)
-{
-	u8 class = BPF_CLASS(insn->code);
-
-	if (class == BPF_ST || (class == BPF_STX && BPF_MODE(insn->code) == BPF_MEM))
-		return true;
-	/* r6 - r9 are what they were when the call returns */
-	if (insn->code == (BPF_JMP | BPF_CALL)) {
-		*known &= ~GENMASK(BPF_REG_5, BPF_REG_0);
-		return true;
-	}
-	if (class != BPF_ALU && class != BPF_ALU64 && class != BPF_LDX)
-		return false;
-	*known &= ~BIT(insn->dst_reg);
-	if (class != BPF_LDX && BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_K) {
-		*known |= BIT(insn->dst_reg);
-		val[insn->dst_reg] = insn->imm;
-	}
-	return true;
-}
-
-/*
- * Does the program leave the loop at @h when the conditional jump at @n goes on at @t?
- * It does when @t is not in the loop. A compiler also makes
- *	w2 = 1; if w1 == 0 goto l; w2 = 0; l: ...; if w2 != 1 goto head;
- * of 'while (n--)'. So follow the constants that are set next to the jump for as long as there
- * is one way to go.
- */
-bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t)
-{
-	struct bpf_insn *insns = env->prog->insnsi, *insn;
-	s32 val[MAX_BPF_REG];
-	u16 known = 0;
-	int i, l;
-	u8 op;
-
-	/* the insns in front of the jump, for as long as nothing else leads to the next one */
-	for (i = n; i && i > n - 4 && !env->insn_aux_data[i].loop && env->idoms[i] == i - 1; i--)
-		;
-	for (; i < n; i++)
-		if (!track_consts(&insns[i], &known, val))
-			known = 0;
-	for (i = 0; i < 16; i++, t++) {
-		l = bpf_loop_at_index(env, t);
-		if (l != h && !bpf_is_nested_loop(env, l, h))
-			return true;
-		insn = &insns[t];
-		op = BPF_OP(insn->code);
-		if (t == h) {
-			return false;
-		} else if (insn->code == (BPF_JMP | BPF_JA)) {
-			t += insn->off;
-		} else if (is_cond_jmp_insn(insn)) {
-			/* a 32-bit and a 64-bit jump agree on a number that is not negative */
-			if (!(known & BIT(insn->dst_reg)) || val[insn->dst_reg] < 0 ||
-			    BPF_SRC(insn->code) != BPF_K || (op != BPF_JEQ && op != BPF_JNE))
-				return false;
-			if ((val[insn->dst_reg] == insn->imm) == (op == BPF_JEQ))
-				t += insn->off;
-		} else if (!track_consts(insn, &known, val)) {
-			return false;
-		}
-	}
-	return false;
-}
-
 /*
  * Collect conditional jumps within n's loop that dominate 'n' and leave
  * the loop on one edge, see bpf_edge_leaves_loop(), nearest first.
  */
-static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
-				       struct bpf_backedge *backedge)
+static int find_dominating_conditions(struct bpf_verifier_env *env, int n,
+				      struct bpf_backedge *backedge, int budget)
 {
 	struct bpf_insn *insns = env->prog->insnsi;
-	bool t_exits_loop, f_exits_loop;
+	int t_exits_loop, f_exits_loop;
 	int *idoms = env->idoms;
 	int n_loop;
 
 	n_loop = bpf_loop_at_index(env, n);
 	if (idoms_intersect(env, n, n_loop) != n_loop)
-		return;
+		return 0;
 	while (n >= 0) {
 		if (is_cond_jmp_insn(&insns[n]) && bpf_loop_at_index(env, n) == n_loop) {
-			t_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + insns[n].off + 1);
-			f_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + 1);
+			t_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + insns[n].off + 1,
+							    &budget);
+			f_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + 1, &budget);
+			if (t_exits_loop < 0 || f_exits_loop < 0)
+				return -ENOMEM;
 			/* latches[0] is the latch: it has to leave the loop itself */
 			if ((t_exits_loop || f_exits_loop) &&
 			    (backedge->latches_cnt || bpf_jump_leaves_loop(env, n_loop, n))) {
 				if (backedge->latches_cnt == MAX_LATCHES)
-					return;
+					return 0;
+				if (t_exits_loop)
+					backedge->taken_leaves |= BIT(backedge->latches_cnt);
 				backedge->latches[backedge->latches_cnt++] = n;
 			}
 		}
@@ -402,6 +340,60 @@ static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
 			break;
 		n = idoms[n];
 	}
+	return 0;
+}
+
+/* The insns to look at, per back edge, to find the jumps that leave a loop by way of a flag */
+#define FLAG_BUDGET	4096
+
+/*
+ * Without @flags: the jumps that leave a loop themselves. That tells where the verifier may add
+ * may_goto. It takes the constants that hold with these edges to follow a flag.
+ */
+int bpf_find_latches(struct bpf_verifier_env *env, bool flags)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_backedge *backedge;
+	struct bpf_loop *loop;
+	int i, j, k, err;
+
+	for (i = 0; i < env->prog->len; i++) {
+		loop = aux[i].loop;
+		if (!loop)
+			continue;
+		if (signal_pending(current))
+			return -EAGAIN;
+		cond_resched();
+		for (j = 0; j < loop->backedges_cnt; j++) {
+			backedge = &loop->backedges[j];
+			backedge->latches_cnt = 0;
+			backedge->taken_leaves = 0;
+			/*
+			 * In theory, the backedge->from and it's latch can reside in
+			 * an inner loop of `i`, e.g.:
+			 *
+			 *  1:  r7 += 1;
+			 *  2:  r6 += 1;
+			 *      if r7 == 2 goto 1b;
+			 *      if r6 < 2 goto 2b;
+			 *
+			 * For now, let's assume that latches are unknown for such cases.
+			 * (GCC/LLVM handle this by inserting artificial cfg nodes).
+			 */
+			if (bpf_loop_at_index(env, backedge->from) != i)
+				continue;
+			err = find_dominating_conditions(env, backedge->from, backedge,
+							 flags ? FLAG_BUDGET : 0);
+			if (err)
+				return err;
+			for (k = 1; k < backedge->latches_cnt; k++)
+				if ((env->log.level & BPF_LOG_LEVEL2) &&
+				    !bpf_jump_leaves_loop(env, i, backedge->latches[k]))
+					bpf_log(&env->log, "loop at %d, exit by way of a flag at %d\n",
+						i, backedge->latches[k]);
+		}
+	}
+	return 0;
 }
 
 /*
@@ -572,30 +564,9 @@ int bpf_compute_loops(struct bpf_verifier_env *env)
 		if (err)
 			goto out;
 	}
-	/* find latches */
-	for (i = 0; i < len; i++) {
-		loop = aux[i].loop;
-		if (!loop)
-			continue;
-		for (j = 0; j < loop->backedges_cnt; j++) {
-			backedge = &loop->backedges[j];
-			/*
-			 * In theory, the backedge->from and it's latch can reside in
-			 * an inner loop of `i`, e.g.:
-			 *
-			 *  1:  r7 += 1;
-			 *  2:  r6 += 1;
-			 *      if r7 == 2 goto 1b;
-			 *      if r6 < 2 goto 2b;
-			 *
-			 * For now, let's assume that latches are unknown for such cases.
-			 * (GCC/LLVM handle this by inserting artificial cfg nodes).
-			 */
-			if (bpf_loop_at_index(env, backedge->from) != i)
-				continue;
-			find_dominating_conditions(env, backedge->from, backedge);
-		}
-	}
+	err = bpf_find_latches(env, false);
+	if (err)
+		goto out;
 	/* find exits */
 	for (i = 0; i < len; i++) {
 		iloop = aux[i].loop ? i : aux[i].loop_header;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 819b563a5b54..8ce188a715e8 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2043,13 +2043,12 @@ static bool loop_invariant(struct scev *scev, u32 id)
 
 static int match_linear_latch(struct bpf_verifier_env *env,
 			      u32 header,
-			      u32 latch_idx,
+			      u32 latch_idx, bool taken_leaves,
 			      struct linear_latch *latch)
 {
 	struct bpf_insn *insn = &env->prog->insnsi[latch_idx];
 	struct scev *scev = env->scev;
 	struct env *latch_env;
-	u32 true_branch_tgt;
 	u32 src_reg_scev;
 	u32 dst_reg_scev;
 	u32 t, l, r, op;
@@ -2058,8 +2057,7 @@ static int match_linear_latch(struct bpf_verifier_env *env,
 	op = BPF_OP(insn->code);
 	latch->jmp32 = BPF_CLASS(insn->code) == BPF_JMP32;
 	/* Flip the condition if true branch jumps out of the loop. */
-	true_branch_tgt = latch_idx + bpf_jmp_offset(insn) + 1;
-	if (bpf_edge_leaves_loop(env, header, latch_idx, true_branch_tgt))
+	if (taken_leaves)
 		op = bpf_rev_opcode(op);
 	switch (op) {
 	case BPF_JSLT:
@@ -3289,7 +3287,8 @@ static int count_backedge(struct bpf_verifier_env *env, struct bpf_func_state *s
 
 	for (j = 0; j < backedge->latches_cnt; j++) {
 		latch_idx = backedge->latches[j];
-		linear_latch = match_linear_latch(env, insn_idx, latch_idx, &latch);
+		linear_latch = match_linear_latch(env, insn_idx, latch_idx,
+						  backedge->taken_leaves & BIT(j), &latch);
 		if (linear_latch < 0)
 			return linear_latch;
 		if (!linear_latch) {
@@ -3305,6 +3304,9 @@ static int count_backedge(struct bpf_verifier_env *env, struct bpf_func_state *s
 					insn_idx, latch_idx);
 			continue;
 		}
+		/* a flag can have other reasons to end the loop */
+		if (!bpf_jump_leaves_loop(env, insn_idx, latch_idx))
+			tmp.min_header_count = 0;
 		if (!found || tmp.max_header_count < iters->max_header_count) {
 			iters->max_header_count = tmp.max_header_count;
 			iters->iv_reg = tmp.iv_reg;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8c89bf67a842..5af42d2dcab9 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -25219,6 +25219,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/*
+	 * It took the constants to remove dead branches and find the loops. Now it's known
+	 * which edges the verifier may add, see bpf_head_guard_target(). What follows
+	 * believes the constants.
+	 */
+	ret = bpf_flow_place_guards(env);
+	if (ret > 0)
+		ret = bpf_compute_const_regs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
+	ret = bpf_find_latches(env, true);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index ba43304836ca..f5c505dfd464 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -316,11 +316,15 @@ __naked void gotol(void)
 }
 #endif
 
+/*
+ * If the verifier had to add may_goto to this loop it would be at the head, insn 0, and go to
+ * insn 3. What is live at 3 is live at 0.
+ */
 SEC("socket")
 __log_level(2)
-__msg("0: .......... (b7) r1 = 1")
+__msg("0: .1........ (b7) r1 = 1")
 __msg("1: .1........ (e5) may_goto pc+1")
-__msg("2: .......... (05) goto pc-3")
+__msg("2: .1........ (05) goto pc-3")
 __msg("3: .1........ (bf) r0 = r1")
 __msg("4: 0......... (95) exit")
 __naked void may_goto(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
index 1fcefce9ab01..6dd46edd8ed0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -130,17 +130,17 @@ LOOP(down_two_tests, "w8 = 4; r6 += 2043;",
 /* 'while (n--)': the jump that counts sets a flag, the jump that leaves the loop tests it */
 #define FLAG(mid)	"w8 += 1; w2 = 1; if w8 == 0 goto 2f; w2 = 0; 2: " mid " if w2 != 1 goto 1b;"
 
-OK("a flag", "2048")
+OK("a flag", "[0..2048]")
 LOOP(flag, "w8 = -2048;", FLAG("r3 = r6;"))
 
 BAD("a flag, one too many")
 LOOP(flag_bad, "w8 = -2049;", FLAG("r3 = r6;"))
 
-OK("a flag, a store behind it", "2048")
+OK("a flag, a store behind it", "[0..2048]")
 LOOP(flag_store, "w8 = -2048;",
      "w8 += 1; w2 = 1; *(u32 *)(r10 - 4) = r8; if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
 
-OK("a flag, a call behind it", "2048")
+OK("a flag, a call behind it", "[0..2048]")
 LOOP(flag_call, "w8 = -2048;",
      "w8 += 1; w9 = 1; if w8 == 0 goto 2f; w9 = 0; 2: call %[bpf_get_prandom_u32]; if w9 != 1 goto 1b;")
 
-- 
2.55.0

Re: [PATCH bpf-next v3 18/45] bpf: Tell the passes over the graph where an added may_goto goes

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] bpf: missing head guard target edge in update_insn() stack liveness
- [High] bpf: bpf_prune_dead_branches() folds jumps before head guards are placed
--

commit dc67a790eecafbb6bb6ef2d65c1ca1f5bfcd6434
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Tell the passes over the graph where an added may_goto goes

This commit places loop guards right after loop discovery so pre-passes over
the control flow graph account for may_goto edges added at loop headers, and
extends bpf_edge_leaves_loop() to propagate constants when identifying loop
latches controlled by flags.

> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index 525c0faff1a66..6a2ee407412b8 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c

[ ... ]

> @@ -2780,6 +2796,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>  			u32 new_out = 0;
>  			u32 new_in = 0;
>  			u32 use;
> +			int t;
>  
>  			succ = bpf_insn_successors(env, insn_idx);
>  			for (int s = 0; s < succ->cnt; ++s)
> @@ -2794,6 +2811,10 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>  				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
>  				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
>  			new_in = (new_out & ~live->def) | use;
> +			/* may_goto that is added at the head of a loop goes there */
> +			t = bpf_head_guard_target(env, insn_idx);
> +			if (t >= 0)
> +				new_in |= state[t].in;
>  			if (new_out != live->out || new_in != live->in) {
>  				live->in = new_in;
>  				live->out = new_out;

[Severity: High]
As reported by Sashiko on v2 [1], bpf_compute_live_registers() unions
state[t].in into new_in for t = bpf_head_guard_target(env, insn_idx), and
compute_subprog_args() joins the forward pointer and spill state from the
loop header into t, but does update_insn() in kernel/bpf/liveness.c also need
to union FM_LIVE_BEFORE of t into new_before for stack slot liveness?

In bpf_compute_subprog_arg_access() -> analyze_subprog() ->
update_instance() -> update_insn(), backward stack liveness only inspects
normal CFG successors:

kernel/bpf/liveness.c:update_insn() {
	...
	for (s = 0; s < succ->cnt; ++s) {
		unsigned long *succ_live;

		succ_live = rel_mask(fm, relative_idx(instance, succ->items[s]), FM_LIVE_BEFORE);
		for (w = 0; w < fm->words; w++)
			new_after[w] |= succ_live[w];
	}
	...
	for (w = 0; w < fm->words; w++) {
		unsigned long new_before = (new_after[w] & ~must_write[w]) | may_read[w];
	...
}

If a stack slot in the current or caller frame is initialized before the
loop, overwritten (FM_MUST_WRITE) on every normal CFG path through the loop
to the exit t (or by the loop header instruction insn_idx itself), and then
read at or after t, update_insn() clears its liveness bit via ~must_write[w]
and leaves FM_LIVE_BEFORE as 0 at the loop header insn_idx.

When a verifier state reaches insn_idx during the main pass,
bpf_clean_verifier_state() -> clean_verifier_state() -> __clean_func_state()
sees bpf_stack_slot_alive() as false, resets spilled_ptr to NOT_INIT, and
marks the slot as STACK_POISON before the state is sent along the header
may_goto edge to t. Can that cause valid stack reads at or after t to be
rejected, or cause __clean_func_state() and join_slot() at t to mishandle
spilled pointers or uninitialized stack slots that were prematurely poisoned
at the loop header?

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 8c89bf67a8424..5af42d2dcab94 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -25219,6 +25219,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
>  	if (ret < 0)
>  		goto skip_full_check;
>  
> +	/*
> +	 * It took the constants to remove dead branches and find the loops. Now it's known
> +	 * which edges the verifier may add, see bpf_head_guard_target(). What follows
> +	 * believes the constants.
> +	 */
> +	ret = bpf_flow_place_guards(env);
> +	if (ret > 0)
> +		ret = bpf_compute_const_regs(env);
> +	if (ret < 0)
> +		goto skip_full_check;

[Severity: High]
When bpf_flow_place_guards() returns a positive value and bpf_check()
re-runs bpf_compute_const_regs(), what happens to conditional jumps at or
after a head-guard target that were already rewritten in-place by
bpf_prune_dead_branches() earlier in bpf_check()?

Earlier in bpf_check(), the first bpf_compute_const_regs() pass and
bpf_prune_dead_branches() run before bpf_compute_loops() and
bpf_flow_place_guards():

kernel/bpf/verifier.c:bpf_check() {
	...
	ret = bpf_compute_const_regs(env);
	if (ret < 0)
		goto skip_full_check;

	ret = bpf_prune_dead_branches(env);
	if (ret < 0)
		goto skip_full_check;
	...
}

At that point aux[h].guard_at_header is not yet set. If a register (or a
read-only map load derived from it, such as r7 == &ro on normal exits versus
r7 == &rw at the loop header) resolves to a constant at the exit target t in
the first bpf_compute_const_regs() pass, bpf_prune_dead_branches()
permanently rewrites any conditional jump testing that constant at or after t
into BPF_JMP_A(taken ? insn->off : 0) and recomputes the CFG postorder.

Re-running bpf_compute_const_regs() after bpf_flow_place_guards() clears
aux->const_reg_mask at t, but won't the in-place BPF_JMP_A instruction
rewrites in env->prog->insnsi remain in effect when a state jumps from the
loop header to t via may_goto?

[1] https://lore.kernel.org/r/20261105204513.2230344-19-alexei.starovoitov@gmail.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=18

[PATCH bpf-next v3 19/45] bpf: Verify programs as a fixpoint over the control flow graph

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  for i in 0..n { v.push(i * i); }
Test vec_push_pop of rust-bpf: "The sequence of 8193 jumps is too complex."
The verifier follows each path from the first insn to exit, and each branch
can double the number of paths. 123 tests of rust-bpf hit this limit or
1M insns. C progs are cut until they fit:
Cilium's bpf_xdp/tail_nodeport_nat_ingress_ipv6 took 182050 insns in May
2026, scx_eevdf/eevdf_dispatch takes 350041.

Do not follow paths. Where two paths meet, make one state that covers both,
a join: r1=1 and r1=5 give r1=[1,5]. What is safe for the join is safe for
both. Repeat until no state changes. The work depends on the size of the
prog, not on the number of paths: 1761, 7932 and 21209 insns with this
patch, and 120 of the 123 load. do_check() takes the next new state from
flow.c and checks one basic block. The check of an insn is the same.

What user space sees:
 - The limits of 1M processed insns and 8192 jumps are gone, from
   bpf_design_QA.rst too. flow.c limits states, for a prog of N insns:
     1024 at an insn      "more than 1024 states that cannot be joined"
     16 * N + 4096 at a time, 256 * N + 64K in all    "too many states"
 - "infinite loop detected" is gone: 111 of the 225 progs of the schedulers
   of rust-bpf. A fixpoint shows that each iteration is safe, not that the
   loop ends:
     counted, or walked to its end                      nothing is added
     each iteration has may_goto or bpf_iter_*_next()   nothing is added
     no exit                                            rejected
     all other loops                                    may_goto
 - An added may_goto is a new edge. check_loop_guard() verifies it. It goes
   to 'r0 = 0; exit': the probe loop of hashbrown exits only with a bucket,
   and the code after the loop uses it: "R6 invalid mem access 'scalar'".
   If the frame holds more than at its entry, or is a subprog that cannot
   unwind, may_goto goes to the exit of the loop: the prog releases there
   what it holds. While a walk can get to the end of its loop the edge is
   dropped. If it gives up, do_check_all() verifies the prog again.
 - A walk goes on while each iteration decides a jump that leaves the loop.
   A jump that is decided only because states are kept apart by a flag does
   not count: it repeats the jump that set the flag, which went both ways.
   lexical_numbers of rust-bpf: 42136 insns, 99506 if it counts.
 - Log level 2 shows basic blocks, not paths: "from 25 to 19: joined",
   "widened", or "within" for "safe". iter(19)=3 is a walk.
 - A join loses the relation between scalars. This is rejected now:
     if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
     arr[n - i]                   i=[3,7] n=[4,8], thus n - i is [-3, 5]
 - 1 and 2 are joined at exit, and a test for 'r0 is 2 or 3' misses [1, 2].
   Thus a cgroup_skb prog whose r0 can be 2 or 3 attaches to egress only.
 - Static subprogs and recursion are as before. Pruning, jmp_history and
   precision marks are only disabled here.
   kfuncs.rst says 'compares and joins states' for 'state pruning checks'.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 Documentation/bpf/bpf_design_QA.rst |   7 +-
 Documentation/bpf/kfuncs.rst        |   2 +-
 include/linux/bpf_verifier.h        |  11 +
 kernel/bpf/backtrack.c              |   7 +
 kernel/bpf/const_fold.c             |   6 +
 kernel/bpf/flow.c                   |  21 +-
 kernel/bpf/log.c                    |   5 +
 kernel/bpf/states.c                 |  50 ++-
 kernel/bpf/verifier.c               | 625 ++++++++++++++++++++--------
 9 files changed, 533 insertions(+), 201 deletions(-)

diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst
index 98d70b8a2533..f86b272920ae 100644
--- a/Documentation/bpf/bpf_design_QA.rst
+++ b/Documentation/bpf/bpf_design_QA.rst
@@ -92,11 +92,8 @@ Q: What are the verifier limits?
 A: The only limit known to the user space is BPF_MAXINSNS (4096).
 It's the maximum number of instructions that the unprivileged bpf
 program can have. The verifier has various internal limits.
-Like the maximum number of instructions that can be explored during
-program analysis. Currently, that limit is set to 1 million.
-Which essentially means that the largest program can consist
-of 1 million NOP instructions. There is a limit to the maximum number
-of subsequent branches, a limit to the number of nested bpf-to-bpf
+The largest program can consist of 1 million instructions.
+There is a limit to the number of nested bpf-to-bpf
 calls, a limit to the number of the verifier states per instruction,
 a limit to the number of maps used by the program.
 All these limits can be hit with a sufficiently complex program.
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index ac9e3a175378..c7da603cab77 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -159,7 +159,7 @@ An example is given below::
 Here, bpf_obj_new uses local_type_id argument to find out the size of that type
 ID in program's BTF and return a sized pointer to it. Each type ID will have a
 distinct size, hence it is crucial to treat each such call as distinct when
-values don't match during verifier state pruning checks.
+values don't match when the verifier compares and joins states.
 
 Hence, whenever a constant scalar argument is accepted by a kfunc which is not a
 size parameter, and the value of the constant matters for program safety, __k
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index e5bee65807fa..91ef4e0a1204 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -390,6 +390,11 @@ struct loop_stack_entry {
 	u32 probe:1;
 	/* this one did */
 	u32 decided:1;
+	/*
+	 * The state is kept apart from one with the other value of a flag, see
+	 * flow_join_makes_up(). The test of the flag is decided in both, and decides nothing.
+	 */
+	u32 flag_apart:1;
 	/*
 	 * How many times the state came back to the head of the loop. With @apart that is
 	 * the iteration that it is in. A state that stands for many iterations is a join, and has
@@ -1140,9 +1145,14 @@ struct bpf_subprog_info {
 	bool stack_args_from_code: 1;
 	/* it releases what its caller holds, so it's verified in the context of such caller */
 	bool releases_inherited: 1;
+	/* a call of it was verified on its own: its insns are not in insns_total of the caller */
+	bool on_its_own: 1;
 	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
 	bool bail_unwinds: 1;
+	/* some call of it is not followed by unwind_goto */
+	bool plain_call: 1;
 	u8 code_stack_args;
+	int bail_insn; /* an exit of it, for the main pass. -1 if there is none */
 	u16 stack_arg_cnt; /* incoming + max outgoing */
 	u16 max_out_stack_arg_cnt;
 };
@@ -1693,6 +1703,7 @@ void bpf_flow_loop_gets_guards(struct bpf_verifier_env *env, int h);
 void bpf_flow_new_round(struct bpf_verifier_env *env);
 int bpf_flow_place_guards(struct bpf_verifier_env *env);
 int bpf_flow_compute_kinds(struct bpf_verifier_env *env);
+u16 bpf_flow_flag_regs(struct bpf_verifier_env *env, int idx);
 bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r);
 /* verifier.c, for flow.c */
 int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index a85ee6ee3776..a6e69c1f5ff6 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -16,6 +16,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
 	struct bpf_jmp_history_entry *p;
 	size_t alloc_size;
 
+	/* FLOW: nothing looks at the history */
+	return 0;
+
 	/* combine instruction flags if we already recorded this instruction */
 	if (env->cur_hist_ent) {
 		/* atomic instructions push insn_flags twice, for READ and
@@ -861,6 +864,10 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
 	struct bpf_func_state *func;
 	bool tmp, skip_first = true;
 	struct bpf_stack_state *ss;
+
+	/* FLOW: there are no precision marks */
+	bt_reset(bt);
+	return 0;
 	struct bpf_reg_state *reg;
 	int i, fr, err;
 
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index bcaec1488fc9..7e088378c302 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -263,6 +263,12 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
 
 		for (r = 0; r < MAX_BPF_REG; r++)
 			ci_in[start][r] = unknown;
+		/*
+		 * The subprog gets there from where it is out of budget too, see
+		 * push_budget_exit(). That is not an edge of the graph.
+		 */
+		for (r = 0; r < MAX_BPF_REG && env->subprog_info[i].bail_insn >= 0; r++)
+			ci_in[env->subprog_info[i].bail_insn][r] = unknown;
 	}
 	/* The same from the head of a loop to where the loop is left, once the loops are known. */
 	for (i = 0; i < insn_cnt; i++) {
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index c4a6384d34bc..62115e396635 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -570,7 +570,7 @@ static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
 }
 
 /* A jump that compares a register with 0 or 1 */
-static u16 flow_flag_regs(struct bpf_verifier_env *env, int idx)
+u16 bpf_flow_flag_regs(struct bpf_verifier_env *env, int idx)
 {
 	struct bpf_insn *insn = &env->prog->insnsi[idx];
 
@@ -1024,7 +1024,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 		if (jump)
 			gen[i] = flow_tested_regs(env, i);
 		else if (flag)
-			gen[i] = flow_flag_regs(env, i);
+			gen[i] = bpf_flow_flag_regs(env, i);
 		else if (kind == FLOW_NEED_STACK_OFF)
 			gen[i] = flow_stack_off_regs(env, i);
 		else if (kind == FLOW_NEED_PTR_OFF || kind == FLOW_NEED_VALUE_OFF)
@@ -2145,13 +2145,18 @@ static bool flow_gap(const struct bpf_reg_state *a, const struct bpf_reg_state *
  * with either, so none of the above keeps them apart. The join has ok = 1 with i >= 64. A flag is
  * made to send states different ways. So 0 and 1 are not joined where a jump that is not far
  * tells one from the other: FLOW_NEED_FLAG.
+ *	while (more) { ...; more = next(&x); }
+ * That jump can be the one that leaves a loop. It is decided in both states, one stays and one
+ * leaves. It's the jump that set the flag, which went both ways, once more: no sign that the
+ * loop gets anywhere. A walk on patience has to know, see mark_loop_exit_decided().
  *
  * The states go on to the jumps on their own. Behind the last one the numbers are dead, and what
  * is left is joined.
  */
-static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
-			       const struct bpf_verifier_state *b)
+static bool flow_join_makes_up(const struct bpf_flow *flow, struct bpf_verifier_state *a,
+			       struct bpf_verifier_state *b)
 {
+	struct bpf_func_state *fa = a->frame[a->curframe], *fb = b->frame[b->curframe];
 	const struct bpf_reg_state *ra, *rb;
 	const struct flow_need *tested = flow->need[FLOW_NEED_JUMP], *need;
 	u32 fr, i, id = 0;
@@ -2179,7 +2184,13 @@ static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_ver
 			id = ra->id & ~BPF_ADD_CONST;
 		}
 	}
-	return flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b);
+	if (!flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b))
+		return false;
+	for (i = 0; i < fa->loop_stack_cnt; i++)
+		fa->loop_stack[i].flag_apart = fa->loop_stack[i].probe;
+	for (i = 0; i < fb->loop_stack_cnt; i++)
+		fb->loop_stack[i].flag_apart = fb->loop_stack[i].probe;
+	return true;
 }
 
 /*
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 916408c63355..e1c9b56cd13f 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -802,6 +802,11 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 		verbose(env, " cb");
 	if (state->in_async_callback_fn)
 		verbose(env, " async_cb");
+	/* the iterations of the loops in which they are kept apart */
+	for (i = 0; i < state->loop_stack_cnt; i++)
+		if (state->loop_stack[i].apart)
+			verbose(env, " iter(%d)=%u", state->loop_stack[i].loop_id,
+				state->loop_stack[i].trip);
 	if (state->loop_stack_cnt) {
 		verbose(env, " loop_stack=");
 		for (i = 0; i < state->loop_stack_cnt; i++)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 9d3a86b5820e..4b2506f3e3e2 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -840,6 +840,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			    old_reg->iter.btf != cur_reg->iter.btf ||
 			    old_reg->iter.btf_id != cur_reg->iter.btf_id ||
 			    old_reg->iter.state != cur_reg->iter.state ||
+			    old_reg->iter.loops != cur_reg->iter.loops ||
 			    /* ignore {old_reg,cur_reg}->iter.depth, see above */
 			    !check_ids(old_reg->id, cur_reg->id, idmap))
 				return false;
@@ -872,16 +873,19 @@ static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *
 			   struct bpf_func_state *cur, struct bpf_idmap *idmap,
 			   enum exact_level exact)
 {
-	int i, nslots;
+	int i, nslots, cur_cnt = cur->out_stack_arg_cnt;
 
-	nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt);
+	/* incoming arguments are not compared with outgoing ones, the other way is ruled out */
+	if (bpf_stack_args_are_incoming(cur) && !bpf_stack_args_are_incoming(old))
+		cur_cnt = 0;
+	nslots = max(old->out_stack_arg_cnt, cur_cnt);
 	for (i = 0; i < nslots; i++) {
 		struct bpf_reg_state *old_arg, *cur_arg;
 		struct bpf_reg_state not_init = { .type = NOT_INIT };
 
 		old_arg = i < old->out_stack_arg_cnt ?
 			  &old->stack_arg_regs[i] : &not_init;
-		cur_arg = i < cur->out_stack_arg_cnt ?
+		cur_arg = i < cur_cnt ?
 			  &cur->stack_arg_regs[i] : &not_init;
 		if (!regsafe(env, old_arg, cur_arg, idmap, exact))
 			return false;
@@ -940,6 +944,23 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
 	return true;
 }
 
+/*
+ * In a loop that is walked without a count a state of an earlier iteration stands for a state of
+ * a later one, see flow_cycle_closed(). Never the other way round.
+ *
+ * In a loop that is counted and walked the state that gets to iteration max_header_count is
+ * dropped: there is no such iteration at run time. The number in a state is the number of
+ * iterations that the program has done, and only states with the same number are compared. They
+ * differ in what is counted anyway.
+ *
+ * The walk of a loop with two entries is meant to get to the end of the loop, like today's walk of
+ * all paths. A state that is within the state of the iteration before it is on its way there:
+ *	for (i = 7; i >= 0; i--) { if (i >= n) continue; ... }
+ * i is in 0..6, then in 0..5. So here too only states with the same number are compared.
+ *
+ * Where the iterations are not kept apart a state stands for many of them, and the number says
+ * nothing about what it describes.
+ */
 static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			    struct bpf_func_state *cur)
 {
@@ -953,7 +974,13 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
 	for (i = 0; i < old->loop_stack_cnt; i++) {
 		if (old_stack[i].loop_id == cur_stack[i].loop_id &&
 		    old_stack[i].max_header_count == cur_stack[i].max_header_count &&
-		    old_stack[i].terminates == cur_stack[i].terminates)
+		    old_stack[i].terminates == cur_stack[i].terminates &&
+		    old_stack[i].guarded == cur_stack[i].guarded &&
+		    old_stack[i].counted == cur_stack[i].counted &&
+		    old_stack[i].apart == cur_stack[i].apart &&
+		    (!old_stack[i].apart ||
+		     (old_stack[i].probe ? old_stack[i].trip <= cur_stack[i].trip :
+					   old_stack[i].trip == cur_stack[i].trip)))
 			continue;
 		return false;
 	}
@@ -999,9 +1026,19 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (!old->no_stack_arg_load && cur->no_stack_arg_load)
 		return false;
 
+	if (memcmp(&old->callback_ret_range, &cur->callback_ret_range,
+		   sizeof(old->callback_ret_range)))
+		return false;
+
 	if (old->in_rcu_callback_fn != cur->in_rcu_callback_fn)
 		return false;
 
+	/* what the frame did to what its callers hold is a part of what it tells them */
+	if (old->entry_refs != cur->entry_refs ||
+	    old->entry_rcu_locks != cur->entry_rcu_locks ||
+	    old->entry_preempt_locks != cur->entry_preempt_locks)
+		return false;
+
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (((1 << i) & live_regs) &&
 		    !regsafe(env, &old->regs[i], &cur->regs[i],
@@ -1047,10 +1084,13 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->speculative && !cur->speculative)
 		return false;
 
+	if (old->callers_unknown != cur->callers_unknown)
+		return false;
+
 	if (old->in_sleepable != cur->in_sleepable)
 		return false;
 
-	if (old->unwound != cur->unwound)
+	if (old->unwound != cur->unwound || old->changed_pkt_data != cur->changed_pkt_data)
 		return false;
 
 	if (!refsafe(old, cur, &env->idmap_scratch))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5af42d2dcab9..4801e9ac0089 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -194,13 +194,14 @@ struct bpf_verifier_stack_elem {
 	struct bpf_verifier_state st;
 	int insn_idx;
 	int prev_insn_idx;
+	/* it goes there only if may_goto in front of prev_insn_idx does, see check_loop_guard() */
+	enum bpf_guard_edge guard_edge;
 	struct bpf_verifier_stack_elem *next;
 	/* length of verifier log at the time this state was pushed on stack */
 	u32 log_pos;
 	u64 diag_log_pos;
 };
 
-#define BPF_COMPLEXITY_LIMIT_JMP_SEQ	8192
 #define BPF_COMPLEXITY_LIMIT_STATES	64
 
 #define BPF_GLOBAL_PERCPU_MA_MAX_SIZE  512
@@ -1038,6 +1039,16 @@ static bool in_rcu_cs(struct bpf_verifier_env *env);
 
 static bool is_kfunc_rcu_protected(struct bpf_call_arg_meta *meta);
 
+/* The number of loops that the frames of @st below @frameno are in */
+static u32 nr_loops_below(const struct bpf_verifier_state *st, u32 frameno)
+{
+	u32 fr, nr = 0;
+
+	for (fr = 0; fr < frameno; fr++)
+		nr += st->frame[fr]->loop_stack_cnt;
+	return nr;
+}
+
 static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 				 struct bpf_call_arg_meta *meta,
 				 struct bpf_reg_state *reg, int insn_idx,
@@ -1070,6 +1081,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 		st->iter.btf = btf;
 		st->iter.btf_id = btf_id;
 		st->iter.state = BPF_ITER_STATE_ACTIVE;
+		st->iter.loops = nr_loops_below(env->cur_state, env->cur_state->curframe + 1);
 		st->iter.depth = 0;
 
 		for (j = 0; j < BPF_REG_SIZE; j++)
@@ -1605,9 +1617,14 @@ static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx)
 
 static void release_reference_state(struct bpf_verifier_state *state, int idx)
 {
-	int last_idx;
+	int last_idx, fr;
 	size_t rem;
 
+	/* the frames that were entered with it don't hold it ever since */
+	for (fr = 0; fr <= state->curframe; fr++)
+		if (idx < state->frame[fr]->entry_refs)
+			state->frame[fr]->entry_refs--;
+
 	/* IRQ state requires the relative ordering of elements remaining the
 	 * same, since it relies on the refs array to behave as a stack, so that
 	 * it can detect out-of-order IRQ restore. Hence use memmove to shift
@@ -1622,6 +1639,20 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx)
 	return;
 }
 
+/* bpf_rcu_read_unlock() or bpf_preempt_enable() is done, see release_reference_state() */
+static void release_context_state(struct bpf_verifier_state *state)
+{
+	struct bpf_func_state *frame;
+	int fr;
+
+	for (fr = 0; fr <= state->curframe; fr++) {
+		frame = state->frame[fr];
+		frame->entry_rcu_locks = min(frame->entry_rcu_locks, state->active_rcu_locks);
+		frame->entry_preempt_locks = min(frame->entry_preempt_locks,
+						 state->active_preempt_locks);
+	}
+}
+
 static bool find_reference_state(struct bpf_verifier_state *state, int id)
 {
 	int i;
@@ -1789,7 +1820,10 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 		return err;
 	dst_state->speculative = src->speculative;
 	dst_state->in_sleepable = src->in_sleepable;
+	dst_state->callers_unknown = src->callers_unknown;
 	dst_state->unwound = src->unwound;
+	dst_state->flow_entry = src->flow_entry;
+	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
 	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
@@ -1918,11 +1952,6 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	if (err)
 		return ERR_PTR(-ENOMEM);
 	elem->st.speculative |= speculative;
-	if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
-		verbose(env, "The sequence of %d jumps is too complex.\n",
-			env->stack_size);
-		return ERR_PTR(-E2BIG);
-	}
 	if (elem->st.parent) {
 		++elem->st.parent->branches;
 		/* WARN_ON(branches > 2) technically makes sense here,
@@ -2601,12 +2630,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	elem->diag_log_pos = bpf_diag_event_log_save(env);
 	env->head = elem;
 	env->stack_size++;
-	if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
-		verbose(env,
-			"The sequence of %d jumps is too complex for async cb.\n",
-			env->stack_size);
-		return ERR_PTR(-E2BIG);
-	}
 	/* Unlike push_stack() do not bpf_copy_verifier_state().
 	 * The caller state doesn't matter.
 	 * This is async callback. It starts in a fresh stack.
@@ -3459,7 +3482,17 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
 	bool seen = false;
 	int i;
 
+	for (i = 0; i < env->subprog_cnt; i++)
+		env->subprog_info[i].bail_insn = -1;
+
 	for (i = 0; i < env->prog->len; i++, insn++) {
+		if (bpf_is_ldimm64(insn)) {
+			i++;
+			insn++;
+			continue;
+		}
+		if (insn->code == (BPF_JMP | BPF_EXIT))
+			bpf_find_containing_subprog(env, i)->bail_insn = i;
 		if (bpf_is_unwind_exit(insn)) {
 			bpf_find_containing_subprog(env, i)->can_unwind = true;
 			seen = true;
@@ -4626,6 +4659,23 @@ static int check_stack_write(struct bpf_verifier_env *env,
 	return err;
 }
 
+/*
+ * The function that is verified on its own has no caller to look at. Its incoming
+ * arguments are in its stack_arg_regs until it is done with loading them.
+ */
+static void end_stack_arg_loads(struct bpf_verifier_env *env, struct bpf_func_state *state)
+{
+	int i;
+
+	if (state->no_stack_arg_load)
+		return;
+	state->no_stack_arg_load = true;
+	if (state->frameno)
+		return;
+	for (i = 0; i < state->out_stack_arg_cnt; i++)
+		bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]);
+}
+
 /*
  * Write a value to the outgoing stack arg area.
  * off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7).
@@ -4651,6 +4701,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		return -EINVAL;
 	}
 
+	end_stack_arg_loads(env, state);
 	err = grow_stack_arg_slots(env, state, spi + 1);
 	if (err)
 		return err;
@@ -4670,7 +4721,6 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
 	}
 	bpf_diag_mod_end(env);
-	state->no_stack_arg_load = true;
 	return bpf_push_jmp_history(env, env->cur_state,
 				    INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
 }
@@ -4699,7 +4749,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
 		return -EACCES;
 	}
 
-	caller = vstate->frame[vstate->curframe - 1];
+	caller = vstate->frame[vstate->curframe ? vstate->curframe - 1 : 0];
 	arg = bpf_get_spilled_stack_arg(spi, caller);
 	if (!arg) {
 		verbose(env, "invalid read from uninitialized stack arg off %d\n", off);
@@ -9068,6 +9118,38 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
 	return &bpf_stack_slot(cur_st->frame[iter_frameno], iter_spi)->spilled_ptr;
 }
 
+/*
+ * bpf_iter_*_next() that every trip of a loop goes through stops the loop if the iterator is
+ * older than the loop: every trip takes an element, and there are only so many. It doesn't stop
+ * - the loop that makes the iterator, which starts from the beginning in every trip then:
+ *	while (v->x) { bpf_for(i, 0, 10) ...; }
+ *   Of the loops that the program is in, the ones that it entered before it made the iterator
+ *   are among the first iter.loops.
+ * - any loop if nothing was left already: the program was told so and did not leave.
+ * A loop with a known number of trips doesn't count on the iterator. Nor does a walk on patience:
+ * there is none in a loop that is taken for bounded, see flow_try_walk().
+ */
+static void iter_stops_loops(struct bpf_verifier_env *env, int insn_idx,
+			     const struct bpf_reg_state *iter)
+{
+	const struct bpf_verifier_state *st = env->cur_state;
+	struct bpf_func_state *frame = st->frame[st->curframe];
+	u32 below = nr_loops_below(st, st->curframe);
+	struct loop_stack_entry *entry;
+	bool stops;
+	int i;
+
+	/* the loops of this subprog: flow_is_bound() doesn't look into the functions it calls */
+	for (i = frame->loop_stack_cnt - 1; i >= 0; i--) {
+		entry = &frame->loop_stack[i];
+		stops = iter->iter.state != BPF_ITER_STATE_DRAINED && below + i >= iter->iter.loops;
+		if (entry->terminates || stops)
+			continue;
+		bpf_flow_next_is_no_bound(env, insn_idx, entry->loop_id);
+		return;
+	}
+}
+
 /* process_iter_next_call() is called when verifier gets to iterator's next
  * "method" (e.g., bpf_iter_num_next() for numbers iterator) call. We'll refer
  * to it as just "iter_next()" in comments below.
@@ -9149,7 +9231,7 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
 static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 				  struct bpf_call_arg_meta *meta)
 {
-	struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
+	struct bpf_verifier_state *cur_st = env->cur_state, *queued_st;
 	struct bpf_func_state *cur_fr = cur_st->frame[cur_st->curframe], *queued_fr;
 	struct bpf_reg_state *cur_iter, *queued_iter;
 
@@ -9164,20 +9246,9 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 		return -EFAULT;
 	}
 
+	iter_stops_loops(env, insn_idx, cur_iter);
+
 	if (cur_iter->iter.state == BPF_ITER_STATE_ACTIVE) {
-		/* Because iter_next() call is a checkpoint is_state_visitied()
-		 * should guarantee parent state with same call sites and insn_idx.
-		 */
-		if (!cur_st->parent || cur_st->parent->insn_idx != insn_idx ||
-		    !same_callsites(cur_st->parent, cur_st)) {
-			verifier_bug(env, "bad parent state for iter next call");
-			return -EFAULT;
-		}
-		/* Note cur_st->parent in the call below, it is necessary to skip
-		 * checkpoint created for cur_st by is_state_visited()
-		 * right at this instruction.
-		 */
-		prev_st = find_prev_entry(env, cur_st->parent, insn_idx);
 		/* branch out active iter state */
 		queued_st = push_stack(env, insn_idx + 1, insn_idx, false);
 		if (IS_ERR(queued_st))
@@ -9186,8 +9257,6 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 		queued_iter = get_iter_from_state(queued_st, meta);
 		queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
 		queued_iter->iter.depth++;
-		if (prev_st)
-			widen_imprecise_scalars(env, prev_st, queued_st);
 
 		queued_fr = queued_st->frame[queued_st->curframe];
 		mark_ptr_not_null_reg(&queued_fr->regs[BPF_REG_0]);
@@ -11514,6 +11583,9 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
 			callsite,
 			state->curframe + 1 /* frameno within this callchain */,
 			subprog /* subprog number within this prog */);
+	callee->entry_refs = state->acquired_refs;
+	callee->entry_rcu_locks = state->active_rcu_locks;
+	callee->entry_preempt_locks = state->active_preempt_locks;
 	err = set_callee_state_cb(env, caller, callee, callsite);
 	if (err)
 		goto err_out;
@@ -11748,6 +11820,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return 0;
 }
 
+enum {
+	PROCESS_BPF_EXIT = 1,
+	INSN_IDX_UPDATED = 2,
+};
 
 static int process_bpf_exit_full(struct bpf_verifier_env *env,
 				 bool *do_print_state, bool exception_exit);
@@ -11890,9 +11966,9 @@ int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *cal
 }
 
 /*
- * Call of a static subprog. The callee is verified in the context of
- * the caller, hence set up a new frame and continue from the first
- * instruction of the callee.
+ * Call of a static subprog. Unless the callee is verified on its own it is
+ * verified in the context of the caller, hence set up a new frame and
+ * continue from the first instruction of the callee.
  */
 static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
 				  int *insn_idx)
@@ -11921,7 +11997,7 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
 				    callee_info->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS));
 	if (stack_arg_cnt > caller_info->stack_arg_cnt)
 		caller_info->stack_arg_cnt = stack_arg_cnt;
-
+	/* btf_check_subprog_call() doesn't get to it when BTF is not reliable */
 	err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS + callee_incoming,
 					bpf_subprog_name(env, subprog), NULL, NULL);
 	if (err)
@@ -12439,7 +12515,15 @@ static bool is_rbtree_lock_required_kfunc(u32 btf_id);
 
 static void account_processed_insn(struct bpf_verifier_env *env)
 {
+	struct bpf_verifier_state *st = env->cur_state;
 	struct bpf_func_state *frame = cur_func(env);
+	u32 fr, i;
+
+	/* every walk on patience that the state is in pays for the insn */
+	for (fr = 0; fr <= st->curframe; fr++)
+		for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++)
+			if (bpf_loop_walk_may_give_up(&st->frame[fr]->loop_stack[i]))
+				env->walk_insns[st->frame[fr]->loop_stack[i].walk_id]++;
 
 	env->insn_processed++;
 	frame->insns_subtotal++;
@@ -12533,7 +12617,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	}
 
 	caller = state->frame[state->curframe - 1];
-	if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx])) {
+	if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx]) || state->unwound) {
 		/* the helper or the kfunc that called it doesn't expect that */
 		if (callee->in_callback_fn) {
 			verbose(env, "callback cannot unwind\n");
@@ -12552,6 +12636,16 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EACCES;
 		}
 
+		/* It's called any number of times: what it takes it has to release. */
+		for (i = callee->entry_refs; i < state->acquired_refs; i++) {
+			if (state->refs[i].type != REF_TYPE_PTR)
+				continue;
+			verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
+				state->refs[i].id, state->refs[i].insn_idx);
+			bpf_diag_leak(env, state->refs[i].id, state->refs[i].insn_idx, *insn_idx);
+			return -EINVAL;
+		}
+
 		/* we are going to rely on register's precise value */
 		err = mark_chain_precision(env, BPF_REG_0);
 		if (err)
@@ -16129,6 +16223,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			return -EINVAL;
 		}
 		env->cur_state->active_rcu_locks--;
+		release_context_state(env->cur_state);
 		bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_RCU, false,
 					env->cur_state->active_rcu_locks);
 		if (!in_rcu_cs(env))
@@ -16146,6 +16241,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			return -EINVAL;
 		}
 		env->cur_state->active_preempt_locks--;
+		release_context_state(env->cur_state);
 		bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_PREEMPT, false,
 					env->cur_state->active_preempt_locks);
 		if (!in_rcu_cs(env))
@@ -19563,6 +19659,25 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
 	}
 }
 
+/* It's known which way the jump at env->insn_idx goes */
+static void mark_loop_exit_decided(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+	struct bpf_func_state *frame = st->frame[st->curframe];
+	bool flag = bpf_flow_flag_regs(env, env->insn_idx);
+	struct loop_stack_entry *entry;
+	u32 i;
+
+	for (i = 0; i < frame->loop_stack_cnt; i++) {
+		entry = &frame->loop_stack[i];
+		/* only because the states are kept apart. That holds for the first such jump. */
+		if (entry->flag_apart && flag)
+			entry->flag_apart = false;
+		else if (entry->apart && entry->probe &&
+			 bpf_jump_leaves_loop(env, entry->loop_id, env->insn_idx))
+			entry->decided = true;
+	}
+}
+
 static int check_cond_jmp_op(struct bpf_verifier_env *env,
 			     struct bpf_insn *insn, int *insn_idx)
 {
@@ -19675,6 +19790,8 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	env->true_reg1 = *dst_reg;
 	env->true_reg2 = *src_reg;
 	pred = is_branch_taken(env, dst_reg, src_reg, opcode, is_jmp32);
+	if (pred >= 0)
+		mark_loop_exit_decided(env, this_branch);
 	if (pred >= 0) {
 		/* If we get here with a dst_reg pointer type it is because
 		 * above is_branch_taken() special cased the 0 comparison.
@@ -20287,7 +20404,7 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	}
 
 	if (!tnum_is_unknown(enforce_attach_type_range) &&
-	    tnum_in(enforce_attach_type_range, reg->var_off))
+	    reg_smax(reg) > 1)
 		env->prog->enforce_expected_attach_type = 1;
 	return 0;
 }
@@ -20783,10 +20900,6 @@ static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type typ
 	return 0;
 }
 
-enum {
-	PROCESS_BPF_EXIT = 1,
-	INSN_IDX_UPDATED = 2,
-};
 
 static int process_bpf_exit_full(struct bpf_verifier_env *env,
 				 bool *do_print_state,
@@ -21102,7 +21215,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 			}
 			mark_reg_scratched(env, BPF_REG_0);
 			if (bpf_in_stack_arg_cnt(&env->subprog_info[cur_func(env)->subprogno]))
-				cur_func(env)->no_stack_arg_load = true;
+				end_stack_arg_loads(env, cur_func(env));
 			if (bpf_is_callx(insn))
 				return check_func_callx(env, insn, &env->insn_idx);
 			if (insn->src_reg == BPF_PSEUDO_CALL)
@@ -21145,6 +21258,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 	return -EFAULT;
 }
 
+static int flow_try_walk(struct bpf_verifier_env *env, struct loop_stack_entry *entry);
+
 /*
  * Push the loop entered at insn_idx onto the stack. Usually this adds a single
  * entry on top of its enclosing loop. However, an edge may enter an inner loop
@@ -21159,7 +21274,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
  * Such a bypass makes the enclosing loop irreducible, so its header is missing
  * from the stack and both headers (1) and (2) need to be pushed onto stack.
  * Only the innermost loop (the one actually entered at insn_idx) carries SCEV bounds;
- * the bypassed ancestors are irreducible and pushed as non-terminating.
+ * the bypassed ancestors are irreducible: walked, see flow_try_walk(), or pushed as
+ * non-terminating.
  *
  * Assumes loop_stack_pop() has already truncated the stack to the common ancestor
  * of the bpf_loop_at_index(env->insn_idx) and whatever was at the top of the loop stack.
@@ -21172,7 +21288,7 @@ static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
 	u32 missing_headers[LOOP_STACK_SIZE];
 	u32 cnt = frame->loop_stack_cnt;
 	u32 num_missing = 0;
-	int h;
+	int h, err;
 
 	*pushed = false;
 
@@ -21196,7 +21312,25 @@ static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
 
 	for (; num_missing; num_missing--, cnt++) {
 		h = missing_headers[num_missing - 1];
-		loop_stack[cnt] = (struct loop_stack_entry){ .loop_id = h };
+		/* wherever it is entered: a loop can have more than one entry */
+		if (!aux[h].loop->exits_cnt) {
+			verbose(env, "loop at insn %d never exits\n", h);
+			return -EINVAL;
+		}
+		loop_stack[cnt] = (struct loop_stack_entry){
+			.loop_id = h,
+			.refs = env->cur_state->acquired_refs,
+		};
+		/* there is no count for a loop that is not entered at its head */
+		if (h != env->insn_idx && !aux[h].loop->rt_bounded) {
+			frame->loop_stack_cnt = cnt;
+			err = flow_try_walk(env, &loop_stack[cnt]);
+			if (err < 0)
+				return err;
+			if (!err)
+				bpf_flow_loop_gets_guards(env, h);
+		}
+		loop_stack[cnt].guarded = aux[h].loop->guarded;
 		if (env->log.level & BPF_LOG_LEVEL2)
 			verbose(env, "entering loop %d\n", h);
 		*pushed = true;
@@ -21265,59 +21399,6 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
 	return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry);
 }
 
-static int handle_loop_entry_exit(struct bpf_verifier_env *env)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_func_state *frame = cur_func(env);
-	struct loop_stack_entry *entry;
-	struct bpf_loop_iters iters;
-	bool pushed, terminates;
-	int err;
-
-	err = loop_stack_pop(env);
-	if (err)
-		return err;
-
-	err = loop_stack_push(env, &pushed);
-	if (err)
-		return err;
-
-	if (frame->loop_stack_cnt == 0)
-		return 0;
-
-	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
-	if (env->insn_idx != entry->loop_id)
-		return 0;
-
-	/* If nothing was pushed and env->insn_idx is a loop header, we've taken a backedge. */
-	if (!pushed)
-		return maybe_clamp_scev_regs(env);
-
-	err = bpf_compute_loop_iters(env, env->cur_state, &iters);
-	if (err < 0)
-		return err;
-
-	terminates = err == 1;
-	if (!terminates)
-		return 0;
-
-	/* Create a loop_entry checkpoint before widening */
-	err = bpf_split_cur_state(env);
-	if (err)
-		return err;
-
-	entry->min_header_count = iters.min_header_count;
-	entry->max_header_count = iters.max_header_count;
-	entry->iv_reg = iters.iv_reg;
-	entry->iv_range = iters.iv_range;
-	entry->entry_state = cur->parent;
-	entry->terminates = true;
-	err = bpf_widen_scev_regs(env, cur, cur->parent, &iters);
-	if (err < 0)
-		return err;
-	return 0;
-}
-
 /*
  * The current state came back to the head of a loop. If it has a reference more than the states
  * that are there it is neither within one of them nor joined with one. It goes around again:
@@ -21515,6 +21596,7 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 		trip = entry->trip;
 		decided = !entry->probe || entry->decided || !trip;
 		entry->decided = false;
+		entry->flag_apart = false;
 		if (decided && ++entry->trip < entry->max_header_count &&
 		    (entry->counted ||
 		     env->walk_insns[entry->walk_id] <= entry->walk_patience))
@@ -21581,14 +21663,174 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	return err < 0 ? err : 0;
 }
 
-static int do_check(struct bpf_verifier_env *env)
+/* Can the main program return 0? */
+static bool zero_is_valid_retval(struct bpf_verifier_env *env)
+{
+	struct bpf_retval_range range;
+
+	if (program_returns_void(env) || !return_retval_range(env, &range))
+		return true;
+	return range.minval <= 0 && range.maxval >= 0;
+}
+
+/*
+ * A loop or a recursion is out of budget in the current frame of @st. What the
+ * program computed so far is not what it was meant to compute, so it's better
+ * not to go on. The way to stop is to unwind: 'r0 = 0; unwind_exit'. The landing
+ * pads of the callers are code like any other. They release what the callers
+ * hold, or the program is rejected.
+ *
+ * False when it cannot:
+ * - There is no landing pad for what this frame took itself.
+ * - There is a call of the subprog that unwind_goto doesn't follow. That is every
+ *   program that is compiled without unwinding.
+ * - A helper that calls a callback doesn't expect it to unwind.
+ * Then the program goes on along a path that it has, like after may_goto of
+ * its own. It releases what it holds there.
+ */
+static bool budget_can_unwind(struct bpf_verifier_env *env, const struct bpf_verifier_state *st)
+{
+	const struct bpf_func_state *frame = st->frame[st->curframe];
+	const struct bpf_subprog_info *sub = &env->subprog_info[frame->subprogno];
+	bool top;
+
+	if (sub->bail_insn < 0)
+		return false;
+	if (st->acquired_refs != frame->entry_refs ||
+	    st->active_rcu_locks != frame->entry_rcu_locks ||
+	    st->active_preempt_locks != frame->entry_preempt_locks)
+		return false;
+	if (frame->in_callback_fn || frame->in_exception_callback_fn)
+		return false;
+	/* the kernel called it: either exit ends it */
+	top = !st->curframe && !st->callers_unknown &&
+	      (!frame->subprogno || frame->in_async_callback_fn);
+	if (top && !frame->subprogno && !zero_is_valid_retval(env))
+		return false;
+	return top || sub->bail_unwinds;
+}
+
+/*
+ * The current frame leaves with 0 in place of the result: with unwind_exit if the subprog
+ * can do that, with exit otherwise. The insns are added when the main pass is over. Until
+ * then an exit that the subprog has stands for them.
+ */
+static int push_budget_exit(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
+	struct bpf_verifier_state *branch;
+	struct bpf_reg_state *regs;
+
+	branch = push_stack(env, sub->bail_insn, env->insn_idx, false);
+	if (IS_ERR(branch))
+		return PTR_ERR(branch);
+	branch->unwound = sub->bail_unwinds;
+	regs = branch->frame[branch->curframe]->regs;
+	__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	if (bpf_ret_reg_pair(env, cur_func(env)->subprogno))
+		__mark_reg_const_zero(env, &regs[BPF_REG_2]);
+	return 0;
+}
+
+/*
+ * The loop that is not counted and has nothing that stops it at run time gets
+ * may_goto in front of this insn. When it is out of budget the program goes
+ * where may_goto goes with the registers and the stack as they are here.
+ * Every state that goes through the loop does, see bpf_flow_loop_gets_guards().
+ */
+static int check_loop_guard(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[env->insn_idx];
+	struct bpf_verifier_state *branch;
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *entry;
+	struct bpf_loop *loop;
+	int h, err;
+
+	h = bpf_loop_at_index(env, env->insn_idx);
+	/* The callback came back to the helper that calls it: the program is not in front of it. */
+	if (h < 0 || !frame->loop_stack_cnt || frame->callback_depth)
+		return 0;
+	loop = env->insn_aux_data[h].loop;
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	if (loop->rt_bounded || !env->insn_aux_data[env->insn_idx].guard_place ||
+	    entry->loop_id != h)
+		return 0;
+	if (verifier_bug_if(!entry->terminates && !entry->guarded, env,
+			    "nothing stops the loop at %d", h))
+		return -EFAULT;
+	if (!entry->guarded && !bpf_loop_walk_may_give_up(entry))
+		return 0;
+
+	/* the device runs the insns that the driver was given, without may_goto */
+	if (bpf_prog_is_offloaded(env->prog->aux)) {
+		verbose(env, "loop at insn %d has no known bound in an offloaded program\n", h);
+		return -EINVAL;
+	}
+
+	/*
+	 * A guard has one way out, and only that one is looked at.
+	 *
+	 * Leave the subprog, if that is possible every time the program gets here:
+	 *	do { p = lookup(); } while (!p); *p
+	 * The code that follows the loop counts on what the jump that leaves it has found out.
+	 *
+	 * Otherwise leave the loop:
+	 *	lock(); do { ... } while (...); unlock();
+	 * The code that follows the loop releases what is held.
+	 *
+	 * It's the same way for all states that get here. As long as they leave the subprog
+	 * what would leave the loop is dropped. Both are dropped as long as the loop has
+	 * no may_goto: the state is in a walk that may get to the end of the loop.
+	 */
+	if (!aux->guard_no_unwind && !budget_can_unwind(env, env->cur_state)) {
+		aux->guard_no_unwind = true;
+		/* What would leave the loop from the states that were here is gone. All again. */
+		if (aux->guard_edge_dropped && loop->guarded)
+			env->guards_late = true;
+	}
+	if (!aux->guard_no_unwind) {
+		err = push_budget_exit(env);
+		if (err)
+			return err;
+		env->head->guard_edge = BPF_GUARD_LEAVES_SUBPROG;
+	}
+
+	branch = push_stack(env, env->insn_aux_data[env->insn_idx].guard_to, env->insn_idx, false);
+	if (IS_ERR(branch))
+		return PTR_ERR(branch);
+	env->head->guard_edge = BPF_GUARD_LEAVES_LOOP;
+	return 0;
+}
+
+/* The states that the insn pushed go where they were pushed to. */
+static int flow_drain(struct bpf_verifier_env *env)
+{
+	struct bpf_verifier_stack_elem *elem;
+	int err = 0;
+
+	while ((elem = env->head)) {
+		env->head = elem->next;
+		env->stack_size--;
+		if (!err && elem->guard_edge)
+			err = bpf_flow_guard_edge(env, &elem->st, elem->insn_idx,
+						  elem->prev_insn_idx, elem->guard_edge);
+		else if (!err)
+			err = bpf_flow_deliver(env, &elem->st, elem->insn_idx,
+					       elem->prev_insn_idx);
+		bpf_free_verifier_state(&elem->st, false);
+		kfree(elem);
+	}
+	return err;
+}
+
+/* Process the current state until it gets to the start of a basic block. */
+static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 {
-	bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
 	struct bpf_verifier_state *state = env->cur_state;
 	struct bpf_insn *insns = env->prog->insnsi;
 	int insn_cnt = env->prog->len;
-	bool do_print_state = false;
-	int prev_insn_idx = -1;
+	bool do_print_state = true;
 
 	for (;;) {
 		struct bpf_insn *insn;
@@ -21601,9 +21843,6 @@ static int do_check(struct bpf_verifier_env *env)
 		if (need_resched())
 			cond_resched();
 
-		/* reset current history entry on each new instruction */
-		env->cur_hist_ent = NULL;
-
 		env->prev_insn_idx = prev_insn_idx;
 		if (env->insn_idx >= insn_cnt) {
 			verbose(env, "invalid insn idx %d insn_cnt %d\n",
@@ -21616,18 +21855,10 @@ static int do_check(struct bpf_verifier_env *env)
 
 		account_processed_insn(env);
 
-		if (env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) {
-			verbose(env,
-				"BPF program is too large. Processed %d insn\n",
-				env->insn_processed);
-			return -E2BIG;
-		}
-
 		state->last_insn_idx = env->prev_insn_idx;
 		state->insn_idx = env->insn_idx;
 		/*
-		 * Record the incoming edge so active and queued paths use the same
-		 * branch-recording path. A zero-offset conditional has identical
+		 * Record the incoming edge. A zero-offset conditional has identical
 		 * successors, so its outcome cannot be reconstructed from the edge.
 		 */
 		if (!state->speculative && prev_insn_idx >= 0 && prev_insn_idx < insn_cnt) {
@@ -21647,40 +21878,6 @@ static int do_check(struct bpf_verifier_env *env)
 			}
 		}
 
-		/*
-		 * Possibly widen the registers before creating a checkpoint
-		 * in bpf_is_state_visited(). The next loop iteration will
-		 * have a chance to hit this checkpoint and converge.
-		 */
-		err = handle_loop_entry_exit(env);
-		if (err)
-			return err;
-
-		if (bpf_is_prune_point(env, env->insn_idx)) {
-			err = bpf_is_state_visited(env, env->insn_idx);
-			if (err < 0)
-				return err;
-			if (err == 1) {
-				/* found equivalent state, can prune the search */
-				if (env->log.level & BPF_LOG_LEVEL) {
-					if (do_print_state)
-						verbose(env, "\nfrom %d to %d%s: safe\n",
-							env->prev_insn_idx, env->insn_idx,
-							env->cur_state->speculative ?
-							" (speculative execution)" : "");
-					else
-						verbose(env, "%d: safe\n", env->insn_idx);
-				}
-				goto process_bpf_exit;
-			}
-		}
-
-		if (bpf_is_jmp_point(env, env->insn_idx)) {
-			err = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);
-			if (err)
-				return err;
-		}
-
 		if (env->log.level & BPF_LOG_LEVEL2 && do_print_state) {
 			verbose(env, "\nfrom %d to %d%s:",
 				env->prev_insn_idx, env->insn_idx,
@@ -21690,13 +21887,6 @@ static int do_check(struct bpf_verifier_env *env)
 			do_print_state = false;
 		}
 
-		if (bpf_loop_at_index(env, env->insn_idx) >= 0 &&
-		    cur_func(env)->loop_stack_cnt == 0) {
-			verifier_bug(env, "loop stack empty at %d, while inside the loop %d\n",
-				     env->insn_idx, bpf_loop_at_index(env, env->insn_idx));
-			return -EFAULT;
-		}
-
 		if (env->log.level & BPF_LOG_LEVEL) {
 			if (verifier_state_scratched(env))
 				print_insn_state(env, state, state->curframe);
@@ -21747,6 +21937,10 @@ static int do_check(struct bpf_verifier_env *env)
 		if (state->speculative && insn_aux->nospec)
 			goto process_bpf_exit;
 
+		err = check_loop_guard(env);
+		if (err)
+			return err;
+
 		err = do_check_insn(env, &do_print_state);
 		if (error_recoverable_with_nospec(err) && state->speculative) {
 			/* Prevent this speculative path from ever reaching the
@@ -21768,19 +21962,9 @@ static int do_check(struct bpf_verifier_env *env)
 		}
 
 		if (state->speculative && insn_aux->nospec_result) {
-			/* If we are on a path that performed a jump-op, this
-			 * may skip a nospec patched-in after the jump. This can
-			 * currently never happen because nospec_result is only
-			 * used for the write-ops
-			 * `*(size*)(dst_reg+off)=src_reg|imm32` and helper
-			 * calls. These must never skip the following insn
-			 * (i.e., bpf_insn_successors()'s opcode_info.can_jump
-			 * is false). Still, add a warning to document this in
-			 * case nospec_result is used elsewhere in the future.
-			 *
-			 * All non-branch instructions have a single
-			 * fall-through edge. For these, nospec_result should
-			 * already work.
+			/*
+			 * All non-branch instructions have a single fall-through
+			 * edge. For these, nospec_result should already work.
 			 */
 			if (verifier_bug_if((BPF_CLASS(insn->code) == BPF_JMP ||
 					     BPF_CLASS(insn->code) == BPF_JMP32) &&
@@ -21790,23 +21974,40 @@ static int do_check(struct bpf_verifier_env *env)
 process_bpf_exit:
 			account_current_path(env);
 			mark_verifier_state_scratched(env);
-			err = bpf_update_branch_counts(env, env->cur_state);
-			if (err)
-				return err;
-			err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
-					pop_log);
-			if (err < 0) {
-				if (err != -ENOENT)
-					return err;
-				break;
-			} else {
-				do_print_state = true;
-				continue;
-			}
+			bpf_free_verifier_state(state, false);
+			memset(state, 0, sizeof(*state));
+			return flow_drain(env);
+		}
+
+		err = flow_drain(env);
+		if (err)
+			return err;
+
+		if (env->insn_idx < insn_cnt && bpf_flow_is_point(env, env->insn_idx)) {
+			account_current_path(env);
+			mark_verifier_state_scratched(env);
+			return bpf_flow_deliver(env, state, env->insn_idx, prev_insn_idx);
 		}
 	}
+}
 
-	return 0;
+static int do_check(struct bpf_verifier_env *env)
+{
+	int prev_insn_idx = -1;
+	int err;
+
+	err = bpf_flow_deliver(env, env->cur_state, env->insn_idx, -1);
+	while (!err) {
+		err = bpf_flow_next(env, &prev_insn_idx);
+		if (err == -ENOENT)
+			return 0;
+		if (err)
+			break;
+		/* What is known about how the state came to be is not kept with it. */
+		bpf_diag_event_log_restore(env, 0);
+		err = do_check_block(env, prev_insn_idx);
+	}
+	return err;
 }
 
 static int find_btf_percpu_datasec(struct btf *btf)
@@ -22902,6 +23103,7 @@ static void free_states(struct bpf_verifier_env *env)
 	bpf_free_verifier_state(env->cur_state, true);
 	env->cur_state = NULL;
 	while (!pop_stack(env, NULL, NULL, false));
+	bpf_flow_reset(env);
 
 	list_for_each_safe(pos, tmp, &env->free_list) {
 		sl = container_of(pos, struct bpf_verifier_state_list, node);
@@ -22957,6 +23159,8 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 	state->speculative = false;
 	state->branches = 1;
 	state->in_sleepable = context == BPF_FUNC_CTX_SLEEPABLE;
+	/* nobody knows what the callers of a global function or of an extension hold */
+	state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
 	env->no_rcu_read_lock = context == BPF_FUNC_CTX_NO_RCU_LOCK;
 	state->frame[0] = kzalloc_obj(struct bpf_func_state, GFP_KERNEL_ACCOUNT);
 	if (!state->frame[0]) {
@@ -23063,7 +23267,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 		 * it's matching expected signature, and otherwise mark BTF
 		 * info for main program as unreliable
 		 */
-		if (env->prog->aux->func_info_aux) {
+		if (env->prog->aux->func_info_aux && !env->prog->aux->func_info_aux[0].unreliable) {
 			ret = btf_prepare_func_args(env, 0);
 			if (ret || sub->arg_slot_cnt != 1 ||
 			    sub->args[0].arg_type != ARG_PTR_TO_CTX) {
@@ -23196,6 +23400,44 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
+/*
+ * What stops a loop is assumed before the main pass and checked in it, see flow_is_bound(). The
+ * main prog and global functions may get to the same loop. It has may_goto for all of them or
+ * for none, so all are done again when the assumption doesn't hold, whatever came out.
+ * The same when a loop gets may_goto, or its may_goto gets another way out, and what went through
+ * the loop before is not there any more, see bpf_flow_loop_gets_guards().
+ *
+ * A round that is dropped counts as paths that never run. What it found out about the insns, the
+ * subprogs and the prog stays. Each of these is made for more than one state that gets to an insn:
+ * it grows to the safe side, or another value is refused, or the last one that gets there sets it.
+ * Where the round got to is dropped with its verdict: the passes that follow take an insn that was
+ * seen for one that passed its checks, and a function that is called has to be safe.
+ */
+static int do_check_all(struct bpf_verifier_env *env)
+{
+	struct bpf_func_info_aux *sub_aux;
+	u64 log_pos = env->log.end_pos;
+	int i, ret;
+
+	for (;;) {
+		bpf_flow_new_round(env);
+		ret = do_check_main(env);
+		ret = ret ?: do_check_subprogs(env);
+		if (!(env->restart && bpf_flow_loops_lost_bound(env)) && !env->guards_late)
+			return ret;
+		/* the log is dropped when a function is found to be safe */
+		bpf_vlog_reset(&env->log, min(log_pos, env->log.end_pos));
+		for (i = 0; i < env->prog->len; i++)
+			env->insn_aux_data[i].seen = 0;
+		for (i = 1; env->prog->aux->func_info && i < env->subprog_cnt; i++) {
+			sub_aux = subprog_aux(env, i);
+			memset(sub_aux->called, 0, sizeof(sub_aux->called));
+			memset(sub_aux->verified, 0, sizeof(sub_aux->verified));
+			sub_aux->called_with_pkt_ptrs = false;
+		}
+	}
+}
+
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
@@ -25196,6 +25438,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = sort_subprogs_topo(env);
 	if (ret < 0)
 		goto skip_full_check;
@@ -25238,6 +25483,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_flow_compute_kinds(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	if (env->log.level & BPF_LOG_LEVEL2)
 		log_program(env);
 
@@ -25249,14 +25498,17 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_flow_prepare(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	bpf_free_subprog_jts(env);
 
 	ret = mark_fastcall_patterns(env);
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = do_check_main(env);
-	ret = ret ?: do_check_subprogs(env);
+	ret = do_check_all(env);
 
 	/* reject recursion through the callx edges found by the main pass */
 	if (ret == 0 && env->callx_edges)
@@ -25418,6 +25670,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	bpf_patch_list_free(env);
 	kvfree(env->fd_array);
 	bpf_free_scev(env);
+	kvfree(env->walk_insns);
+	bpf_flow_free(env);
+	bpf_join_free(env);
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.postorder_nums);
 	kvfree(env->cfg.insn_postorder);
-- 
2.55.0

Re: [PATCH bpf-next v3 19/45] bpf: Verify programs as a fixpoint over the control flow graph

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 10929 bytes --]

> bpf: Verify programs as a fixpoint over the control flow graph

TL;DR:
- Dead last-exit bail_insn may make valid loop programs fail with "R0
  !read_ok"; fixed only by a later patch
- Offloaded programs may reject bounded loops that previously loaded
- Log level 1 no longer rewinds per path, so -ENOSPC may break loads

> diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
> --- a/kernel/bpf/const_fold.c
> +++ b/kernel/bpf/const_fold.c
> @@ -263,6 +263,12 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
>  
>  		for (r = 0; r < MAX_BPF_REG; r++)
>  			ci_in[start][r] = unknown;
> +		/*
> +		 * The subprog gets there from where it is out of budget too, see
> +		 * push_budget_exit(). That is not an edge of the graph.
> +		 */
> +		for (r = 0; r < MAX_BPF_REG && env->subprog_info[i].bail_insn >= 0; r++)
> +			ci_in[env->subprog_info[i].bail_insn][r] = unknown;
>  	}

Only the const pass is told about the extra edge to bail_insn.  Liveness is
not, even though the comment above bpf_head_guard_target() in
bpf_verifier.h says liveness must account for these edges too.

bail_insn is chosen in check_unwind_insns() before bpf_compute_const_regs()
and bpf_prune_dead_branches() run, so pruning can turn the last exit into
dead code.  For example, with a const volatile bool debug that is 0 at load
time, GCC may place the exit of "if (debug) return ..." last.

bpf_compute_live_registers() only visits insns in env->cfg.insn_postorder,
so live_regs_before stays 0 at a dead bail_insn.  When
check_loop_guard() -> push_budget_exit() -> push_stack(env,
sub->bail_insn, ...) delivers the state, clean_verifier_state() uses that
live mask and __clean_func_state() marks R0 NOT_INIT.  The exit check then
rejects the program with "R0 !read_ok".

Can this reject a valid program?  It needs a main prog with a loop that gets
may_goto and a last exit that is dead after constant pruning.

A later commit in the series fixes this: a989e69c3268 ("bpf: R0 is live at
the exit that an added may_goto goes to").  It adds R0, and R2 for a pair,
to insn_aux[bail_insn].live_regs_before in bpf_compute_live_registers().
Should that fix be part of this commit, next to the const_fold handling of
the same edge?

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -3459,7 +3482,17 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
>  	bool seen = false;
>  	int i;
>  
> +	for (i = 0; i < env->subprog_cnt; i++)
> +		env->subprog_info[i].bail_insn = -1;
> +
>  	for (i = 0; i < env->prog->len; i++, insn++) {
> +		if (bpf_is_ldimm64(insn)) {
> +			i++;
> +			insn++;
> +			continue;
> +		}
> +		if (insn->code == (BPF_JMP | BPF_EXIT))
> +			bpf_find_containing_subprog(env, i)->bail_insn = i;
>  		if (bpf_is_unwind_exit(insn)) {

check_unwind_insns() sets bail_insn to the last BPF_EXIT of each subprog,
and push_budget_exit() later sends the out-of-budget state there with
R0 = 0.  Is anything making sure this exit is reachable?

check_unwind_insns() runs before bpf_compute_const_regs() and
bpf_prune_dead_branches().  Consider:

  const volatile bool debug;
  if (debug)
          return skb->len + 7;
  while (*p)
          ;
  return 0;

The 'if (debug)' becomes a BPF_JMP_A, and if the exit of the dead branch is
last, bail_insn is dead code with live_regs_before == 0.  The unbounded loop
gets may_goto, and budget_can_unwind() is true in the main prog.  Then:

check_loop_guard()
  push_budget_exit()
flow_drain()
  bpf_flow_guard_edge()
    bpf_flow_deliver()
      flow_arrive()
        bpf_clean_verifier_state()
          __clean_func_state()   /* R0 -> NOT_INIT, liveness says dead */

and at the exit process_bpf_exit_full() -> check_return_code() fails with
"R0 !read_ok", so a valid program fails to load.  This is the "potentially
unsafe usage" case in bpf.md, where the verifier reads a register that
liveness reports dead.

a989e69c3268 ("bpf: R0 is live at the exit that an added may_goto goes to")
later in the series marks R0, and R2 for a register pair, live at bail_insn
in bpf_compute_live_registers().

[ ... ]

> +/*
> + * The current frame leaves with 0 in place of the result: with unwind_exit if the subprog
> + * can do that, with exit otherwise. The insns are added when the main pass is over. Until
> + * then an exit that the subprog has stands for them.
> + */
> +static int push_budget_exit(struct bpf_verifier_env *env)
> +{
> +	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
> +	struct bpf_verifier_state *branch;
> +	struct bpf_reg_state *regs;
> +
> +	branch = push_stack(env, sub->bail_insn, env->insn_idx, false);
> +	if (IS_ERR(branch))
> +		return PTR_ERR(branch);
> +	branch->unwound = sub->bail_unwinds;
> +	regs = branch->frame[branch->curframe]->regs;
> +	__mark_reg_const_zero(env, &regs[BPF_REG_0]);

push_budget_exit() sends a copy of the state to the last BPF_EXIT of the
subprog in program order.  Can that exit be dead code after
bpf_prune_dead_branches()?

bpf_compute_live_registers() only walks env->cfg.insn_postorder, so a dead
insn keeps live_regs_before == 0, and the budget edge is not a CFG edge, so
liveness never learns that R0 (or R2 for a ret_reg_pair subprog) is read
there.  In the chain flow_drain() -> bpf_flow_guard_edge() ->
bpf_flow_deliver(), bpf_clean_verifier_state() then sets the R0 that
push_budget_exit() just made 0 to NOT_INIT.

process_bpf_exit_full() -> check_return_code() -> check_reg_arg(env,
BPF_REG_0, SRC_OP) then fails with "R0 !read_ok".  This looks like it
affects every non-void main program (or async callback) with a loop that
gets a guard while it holds nothing, which is the case the commit message
says now loads ("It goes to 'r0 = 0; exit'").  It also affects loops the old
path walk accepted but whose walk now runs out of patience (BPF_FLOW_MAX_TRY
= 4096 insns).

This is fixed later in the series by a989e69c3268, which marks R0, and R2
for a register pair, live at bail_insn.

[ ... ]

> +	/* the device runs the insns that the driver was given, without may_goto */
> +	if (bpf_prog_is_offloaded(env->prog->aux)) {
> +		verbose(env, "loop at insn %d has no known bound in an offloaded program\n", h);
> +		return -EINVAL;
> +	}

The check above this lets two kinds of state through: guarded entries, and
entries still in a walk on patience (bpf_loop_walk_may_give_up()).  A walk
is not a loop that needs may_goto, and the comment further down says "Both
are dropped as long as the loop has no may_goto: the state is in a walk that
may get to the end of the loop".  The commit message also says "counted, or
walked to its end -> nothing is added".

For an offloaded program, does the first state that reaches the guard_place
insn of any uncounted loop get -EINVAL, even when the walk would have
reached the end of the loop without a may_goto being added?  The path is
flow_try_walk() returning 1 and then check_loop_guard() at the guard place
in the first trip.

Before this commit such bounded loops were checked by path exploration, and
the NFP verifier hooks do not reject backward jumps, so these programs
loaded.  Would rejecting only when entry->guarded is set still catch every
loop that really gets may_goto?  A walk that gives up calls
bpf_flow_loop_gets_guards(), and the next entry or round is guarded.  No
later commit in the series changes this check.

[ ... ]

> @@ -21790,23 +21974,40 @@ static int do_check(struct bpf_verifier_env *env)
>  process_bpf_exit:
>  			account_current_path(env);
>  			mark_verifier_state_scratched(env);
> -			err = bpf_update_branch_counts(env, env->cur_state);
> -			if (err)
> -				return err;
> -			err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
> -					pop_log);
[ ... ]
> +			bpf_free_verifier_state(state, false);
> +			memset(state, 0, sizeof(*state));
> +			return flow_drain(env);

At log_level 1 the verifier log is no longer rewound when a path ends.
Before this patch do_check() used pop_log = !(env->log.level &
BPF_LOG_LEVEL2) and pop_stack() did:

    if (pop_log) bpf_vlog_reset(&env->log, head->log_pos);

so at level 1 the log only held the current path from the entry.  Now
do_check_block() prints every insn at BPF_LOG_LEVEL, and the only resets
left are the one at the end of a successful pass in do_check_common() and
the one in do_check_all() between rounds.

Can this make a valid program fail to load?  bpf_vlog_update_len_max()
records the largest end_pos reached, and bpf_vlog_finalize() returns
-ENOSPC when log->ubuf && log->len_max > log->len_total, which bpf_check()
takes as the result.  Take a switch of 64 cases of 100 insns each, loaded
with log_level=1 and a 64KB buffer.  Before, the peak was about one path.
Now it is all 6400 insns of the pass.  On a failing program, -ENOSPC may
also replace the real -EACCES or -EINVAL.

On failure, level 1 also no longer shows the path that leads to the error.
It shows the tail of the whole flow-order trace, which mixes blocks of
unrelated paths.

The commit message only says "Log level 2 shows basic blocks, not paths".
It does not mention that level 1 loses its per-path rewinding.  Later
commits in the series (8a4d520dc379 removes pop_stack() entirely) do not
restore it.

[ ... ]

> +static int do_check(struct bpf_verifier_env *env)
> +{
> +	int prev_insn_idx = -1;
> +	int err;
> +
> +	err = bpf_flow_deliver(env, env->cur_state, env->insn_idx, -1);
> +	while (!err) {
> +		err = bpf_flow_next(env, &prev_insn_idx);
> +		if (err == -ENOENT)
> +			return 0;
> +		if (err)
> +			break;
> +		/* What is known about how the state came to be is not kept with it. */
> +		bpf_diag_event_log_restore(env, 0);
> +		err = do_check_block(env, prev_insn_idx);
> +	}
> +	return err;
>  }

The same log growth at level 1 applies in the new do_check().  Nothing
rewinds the log in do_check(), do_check_block(), flow_drain() or flow.c
between states, and the only reset in do_check_common() runs on success
only.  With BPF_LOG_FIXED the earlier blocks fill the buffer first, so the
error message at the end is cut off.  At level 1 the blocks have no "from X
to Y" header, so the trace is hard to follow.

I searched the rest of the series.  8a4d520dc379 ("bpf: Remove prune
points, levels of state comparison and pop_stack()") removes pop_stack() and
elem->log_pos without a replacement, and do_check() at the end of the series
still never rewinds.

Would it work to save env->log.end_pos at the start of do_check() and, when
!(env->log.level & BPF_LOG_LEVEL2), call bpf_vlog_reset() back to that
position before each do_check_block()?  At level 1 the log would then hold
only the block being checked plus its error.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

Re: [PATCH bpf-next v3 19/45] bpf: Verify programs as a fixpoint over the control flow graph

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [High] bpf: missing bpf_head_guard_target() edge in update_insn() and need_const
- [High] bpf: missing loop_stack refs update in release_reference_state()
- [Medium] bpf: join_func() and func_states_equal() reject post-call stack arg states
- [Medium] bpf: unreset loop_guard flag and ignored fatal errors in do_check_all()
- [Medium] bpf: do_check_block() mishandles nospec ordering with check_loop_guard()
- [Medium] bpf: check_loop_guard() rejects offloaded progs during patience walk
--

commit 2c82419db13e7e33be1c8b05f84811e97169e868
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Verify programs as a fixpoint over the control flow graph

This commit replaces path-by-path exploration in the BPF verifier with a
fixpoint iteration over the control flow graph that joins states at merge
points. It also adds loop guard injection and bounded patience walks for
loops without statically known trip counts.

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> index c4a6384d34bce..62115e396635a 100644
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c
> @@ -1024,7 +1024,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
>  		if (jump)
>  			gen[i] = flow_tested_regs(env, i);
>  		else if (flag)
> -			gen[i] = flow_flag_regs(env, i);
> +			gen[i] = bpf_flow_flag_regs(env, i);

[Severity: High]
As reported by Sashiko on v2 [1], when a loop has no dominating exit
condition and flow_place_guards() places a guard at the loop header h with
aux[h].guard_at_header = true and aux[h].guard_to = t, check_loop_guard()
pushes the state at h along a synthetic BPF_GUARD_LEAVES_LOOP edge to t when
aux->guard_no_unwind is true (for example, when a lock, reference, or RCU
read lock is held across the loop).

While bpf_compute_const_regs(), analyze_subprog(), bpf_flow_compute_kinds(),
and bpf_compute_live_registers() account for t = bpf_head_guard_target(env,
idx), update_insn() in kernel/bpf/liveness.c and flow_compute_need_const() in
kernel/bpf/flow.c only propagate across bpf_insn_successors(env, idx):

kernel/bpf/liveness.c:update_insn() {
	succ = bpf_insn_successors(env, insn_idx);
	if (succ->cnt == 0)
		return false;

	/* All instructions of one frame array share the same mask width */
	for (s = 0; s < succ->cnt; ++s) {
		unsigned long *succ_live;

		succ_live = rel_mask(fm, relative_idx(instance, succ->items[s]), FM_LIVE_BEFORE);
		for (w = 0; w < fm->words; w++)
			new_after[w] |= succ_live[w];
	}
}

If a stack slot holds a value or spilled kernel pointer before entering loop
header h, is overwritten on all normal in-loop CFG paths to t (setting
FM_MUST_WRITE), and is read at or after t, update_insn() does not union
FM_LIVE_BEFORE from t = bpf_head_guard_target(env, h) into new_before at h,
leaving the slot marked dead at h.

When a state arrives at h in bpf_flow_deliver() -> flow_arrive(),
bpf_clean_verifier_state() clears the dead slot's spilled_ptr to NOT_INIT and
marks slot_type as STACK_POISON before check_loop_guard() pushes that state
from h to t. At t, reading the slot fails with a false-positive poisoned slot
error when !env->allow_uninit_stack, or if env->allow_uninit_stack is true
and the header guard state joins at t with a normal loop-exit state where the
slot was overwritten with a scalar, join_slot() in kernel/bpf/join.c turns
STACK_MISC/STACK_ZERO + STACK_POISON into STACK_INVALID. Can this allow
check_stack_read_fixed_off() at t to read back the unoverwritten kernel
pointer spilled before h as an unbounded SCALAR_VALUE when the header
may_goto guard triggers at runtime, and should update_insn() and
flow_compute_need_const() also propagate from bpf_head_guard_target()?

[ ... ]

> diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
> index 9d3a86b5820e9..4b2506f3e3e27 100644
> --- a/kernel/bpf/states.c
> +++ b/kernel/bpf/states.c

[ ... ]

> @@ -999,9 +1026,19 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
>  	if (!old->no_stack_arg_load && cur->no_stack_arg_load)
>  		return false;

[Severity: Medium]
As reported by Sashiko on v2 [2], every function frame starts with
no_stack_arg_load = false and out_stack_arg_cnt = 0, and do_check_insn() only
calls end_stack_arg_loads() on BPF_CALL when bpf_in_stack_arg_cnt() is
non-zero.

When a path writes an outgoing stack argument in check_stack_arg_write(),
end_stack_arg_loads() sets state->no_stack_arg_load = true and
grow_stack_arg_slots() increases state->out_stack_arg_cnt. After the call
finishes, invalidate_outgoing_stack_args() resets all state->stack_arg_regs[i]
to NOT_INIT, but state->out_stack_arg_cnt remains at its grown value.

When this post-call state arrives at a loop header or merge point where an
earlier pre-call state (with fa->out_stack_arg_cnt == 0 and
fa->no_stack_arg_load == false) is stored:

1. func_states_equal() rejects !old->no_stack_arg_load &&
   cur->no_stack_arg_load even when the subprogram has no incoming stack
   arguments.
2. join_func() in kernel/bpf/join.c rejects fa->out_stack_arg_cnt <
   fb->out_stack_arg_cnt:

kernel/bpf/join.c:join_func() {
	if (fa->out_stack_arg_cnt < fb->out_stack_arg_cnt)
		return false;
}

Does this prevent joining, widening, and state subsumption whenever a call
with stack arguments occurs inside a loop or on one side of a branch?

[ ... ]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 5af42d2dcab94..4801e9ac0089d 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -1605,9 +1617,14 @@ static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx)
>  
>  static void release_reference_state(struct bpf_verifier_state *state, int idx)
>  {
> -	int last_idx;
> +	int last_idx, fr;
>  	size_t rem;
>  
> +	/* the frames that were entered with it don't hold it ever since */
> +	for (fr = 0; fr <= state->curframe; fr++)
> +		if (idx < state->frame[fr]->entry_refs)
> +			state->frame[fr]->entry_refs--;

[Severity: High]
As reported by Sashiko on v2 [3], loop_stack_push() records
env->cur_state->acquired_refs in loop_stack[cnt].refs when entering a loop,
and check_refs_taken_in_loop() scans from i = entry->refs to
st->acquired_refs at loop backedges to ensure references acquired inside the
loop are still held in a register or stack slot.

When release_reference_state() removes state->refs[idx] and shifts the
remaining entries down with memmove(), it decrements
state->frame[fr]->entry_refs if idx < entry_refs, but it does not decrement
state->frame[fr]->loop_stack[l].refs when idx < loop_stack[l].refs.

If a program acquires a reference before entering a loop, releases that
pre-loop reference inside the loop, and then acquires a new reference inside
the loop and leaks it before taking the backedge, memmove() shifts the
in-loop reference down to index entry->refs - 1 while entry->refs stays
unchanged. Can this cause check_refs_taken_in_loop() to miss the leaked
in-loop reference at the backedge, and should loop_stack_safe() in
kernel/bpf/states.c also check old_stack[i].refs == cur_stack[i].refs to
match join_loop_stack()?

[ ... ]

> @@ -21581,14 +21663,174 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop

[ ... ]

> +static int check_loop_guard(struct bpf_verifier_env *env)
> +{
> +	struct bpf_insn_aux_data *aux = &env->insn_aux_data[env->insn_idx];
> +	struct bpf_verifier_state *branch;
> +	struct bpf_func_state *frame = cur_func(env);
> +	struct loop_stack_entry *entry;
> +	struct bpf_loop *loop;
> +	int h, err;
> +
> +	h = bpf_loop_at_index(env, env->insn_idx);
> +	/* The callback came back to the helper that calls it: the program is not in front of it. */
> +	if (h < 0 || !frame->loop_stack_cnt || frame->callback_depth)
> +		return 0;
> +	loop = env->insn_aux_data[h].loop;
> +	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
> +	if (loop->rt_bounded || !env->insn_aux_data[env->insn_idx].guard_place ||
> +	    entry->loop_id != h)
> +		return 0;
> +	if (verifier_bug_if(!entry->terminates && !entry->guarded, env,
> +			    "nothing stops the loop at %d", h))
> +		return -EFAULT;
> +	if (!entry->guarded && !bpf_loop_walk_may_give_up(entry))
> +		return 0;
> +
> +	/* the device runs the insns that the driver was given, without may_goto */
> +	if (bpf_prog_is_offloaded(env->prog->aux)) {
> +		verbose(env, "loop at insn %d has no known bound in an offloaded program\n", h);
> +		return -EINVAL;
> +	}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [4], when bpf_compute_loop_iters()
returns 0 and bpf_flow_loop_enter() starts a patience walk via
flow_try_walk(), entry->apart = true, entry->terminates = true,
entry->counted = false, and entry->guarded remains false unless the walk later
gives up.

Because bpf_loop_walk_may_give_up(entry) is true during the walk, doesn't
check_loop_guard() reject offloaded programs with -EINVAL on the very first
iteration of the walk, before the walk has a chance to reach the end of the
loop without needing may_goto?

[ ... ]

> @@ -21747,6 +21937,10 @@ static int do_check(struct bpf_verifier_env *env)
>  		if (state->speculative && insn_aux->nospec)
>  			goto process_bpf_exit;
>  
> +		err = check_loop_guard(env);
> +		if (err)
> +			return err;
> +
>  		err = do_check_insn(env, &do_print_state);
>  		if (error_recoverable_with_nospec(err) && state->speculative) {
>  			/* Prevent this speculative path from ever reaching the
> @@ -21768,19 +21962,9 @@ static int do_check(struct bpf_verifier_env *env)
>  		}
>  
>  		if (state->speculative && insn_aux->nospec_result) {
> -			/* If we are on a path that performed a jump-op, this
> -			 * may skip a nospec patched-in after the jump. This can
> -			 * currently never happen because nospec_result is only
> -			 * used for the write-ops
> -			 * `*(size*)(dst_reg+off)=src_reg|imm32` and helper
> -			 * calls. These must never skip the following insn
> -			 * (i.e., bpf_insn_successors()'s opcode_info.can_jump
> -			 * is false). Still, add a warning to document this in
> -			 * case nospec_result is used elsewhere in the future.
> -			 *
> -			 * All non-branch instructions have a single
> -			 * fall-through edge. For these, nospec_result should
> -			 * already work.
> +			/*
> +			 * All non-branch instructions have a single fall-through
> +			 * edge. For these, nospec_result should already work.
>  			 */
>  			if (verifier_bug_if((BPF_CLASS(insn->code) == BPF_JMP ||
>  					     BPF_CLASS(insn->code) == BPF_JMP32) &&
> @@ -21790,23 +21974,40 @@ static int do_check(struct bpf_verifier_env *env)
>  process_bpf_exit:
>  			account_current_path(env);
>  			mark_verifier_state_scratched(env);
> -			err = bpf_update_branch_counts(env, env->cur_state);
> -			if (err)
> -				return err;
> -			err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
> -					pop_log);
> -			if (err < 0) {
> -				if (err != -ENOENT)
> -					return err;
> -				break;
> -			} else {
> -				do_print_state = true;
> -				continue;
> -			}
> +			bpf_free_verifier_state(state, false);
> +			memset(state, 0, sizeof(*state));
> +			return flow_drain(env);
> +		}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [5], when bpf_insert_loop_guards()
prepends a may_goto sequence in front of a guarded instruction,
patch_list_adj_aux() in kernel/bpf/fixups.c copies old->nospec onto the last
instruction of the patch (the original instruction after may_goto), so at
runtime the injected may_goto executes before the BPF_ST_NOSPEC() barrier.

In do_check_block(), however, the early exit on state->speculative &&
insn_aux->nospec runs before check_loop_guard(env), and do_check_all() does
not clear insn_aux->nospec across rounds. Once insn_aux->nospec is set on the
guarded instruction, won't subsequent speculative states reaching it jump to
process_bpf_exit before check_loop_guard() and skip verifying the speculative
may_goto branch?

Conversely, when insn_aux->nospec is not yet set and do_check_insn() fails
with a recoverable error on a speculative path, jumping to process_bpf_exit
calls flow_drain(env), which delivers any states pushed onto env->head by the
failed do_check_insn() call (such as other_branch pushed in
check_cond_jmp_op() before regs_bounds_sanity_check_branches() fails, or
callback states pushed in push_callback_call() before a later check fails).
Shouldcheck_loop_guard() run before the insn_aux->nospec check, and should
states pushed by a failed speculative do_check_insn() be discarded instead of
drained?

[ ... ]

> @@ -23196,6 +23400,44 @@ static int do_check_main(struct bpf_verifier_env *env)
>  	return ret;
>  }
>  
> +/*
> + * What stops a loop is assumed before the main pass and checked in it, see flow_is_bound(). The
> + * main prog and global functions may get to the same loop. It has may_goto for all of them or
> + * for none, so all are done again when the assumption doesn't hold, whatever came out.
> + * The same when a loop gets may_goto, or its may_goto gets another way out, and what went through
> + * the loop before is not there any more, see bpf_flow_loop_gets_guards().
> + *
> + * A round that is dropped counts as paths that never run. What it found out about the insns, the
> + * subprogs and the prog stays. Each of these is made for more than one state that gets to an insn:
> + * it grows to the safe side, or another value is refused, or the last one that gets there sets it.
> + * Where the round got to is dropped with its verdict: the passes that follow take an insn that was
> + * seen for one that passed its checks, and a function that is called has to be safe.
> + */
> +static int do_check_all(struct bpf_verifier_env *env)
> +{
> +	struct bpf_func_info_aux *sub_aux;
> +	u64 log_pos = env->log.end_pos;
> +	int i, ret;
> +
> +	for (;;) {
> +		bpf_flow_new_round(env);
> +		ret = do_check_main(env);
> +		ret = ret ?: do_check_subprogs(env);
> +		if (!(env->restart && bpf_flow_loops_lost_bound(env)) && !env->guards_late)
> +			return ret;
> +		/* the log is dropped when a function is found to be safe */
> +		bpf_vlog_reset(&env->log, min(log_pos, env->log.end_pos));
> +		for (i = 0; i < env->prog->len; i++)
> +			env->insn_aux_data[i].seen = 0;
> +		for (i = 1; env->prog->aux->func_info && i < env->subprog_cnt; i++) {
> +			sub_aux = subprog_aux(env, i);
> +			memset(sub_aux->called, 0, sizeof(sub_aux->called));
> +			memset(sub_aux->verified, 0, sizeof(sub_aux->verified));
> +			sub_aux->called_with_pkt_ptrs = false;
> +		}
> +	}
> +}

[Severity: Medium]
As reported by Sashiko on v2 [6], when do_check_all() restarts verification
for a new round, it clears env->insn_aux_data[i].seen = 0 and calls
bpf_flow_new_round(env), which resets guard_edge_dropped and no_guard_edges,
but neither resets env->insn_aux_data[i].loop_guard = false (or
env->total_states).

If bpf_flow_guard_edge() sets aux[i].loop_guard = true during round 1 on a
path or static subprogram that becomes unreachable in round 2,
bpf_insert_loop_guards() and append_budget_exit() in kernel/bpf/fixups.c will
still see aux[i].loop_guard == true even though aux[i].seen == 0, patching
may_goto at a dead instruction and setting seen = 1 on the appended budget
exit instructions of an otherwise dead subprogram.

In addition, if a fatal error such as -EAGAIN, -ENOMEM, -EFAULT, or -E2BIG
occurs after env->guards_late or env->restart was set earlier in the round,
won't do_check_all() ignore ret, reset the verifier log, and start another
round instead of returning the error immediately?

[1] https://lore.kernel.org/r/20261105204513.2230344-20-alexei.starovoitov@gmail.com
[2] https://lore.kernel.org/r/20261105204513.2230344-20-alexei.starovoitov@gmail.com
[3] https://lore.kernel.org/r/20261105204513.2230344-20-alexei.starovoitov@gmail.com
[4] https://lore.kernel.org/r/30511ed9caaa7fab02343ad6adbd72f0c7d689b7ae89d4b0c37bdf2beb45a414@mail.kernel.org
[5] https://lore.kernel.org/r/30511ed9caaa7fab02343ad6adbd72f0c7d689b7ae89d4b0c37bdf2beb45a414@mail.kernel.org
[6] https://lore.kernel.org/r/20261105204513.2230344-20-alexei.starovoitov@gmail.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=19

[PATCH bpf-next v3 20/45] bpf: Keep the count of a loop when a register has no closed form

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest pyperf600_nounroll, a profiler that reads the stack of Python:
  for (int i = 0; i < STACK_MAX_LEN; ++i) {                  /* 600 */
      if (!frame_ptr || !get_frame_data(..., &frame, &sym))
          break;
      ...
      event->stack[i] = *symbol_id;
      frame_ptr = frame.f_back;
  }
i is an induction variable: 600 iterations, i = [0, 599]. get_frame_data()
fills frame and sym, which are on the stack, in each iteration. These slots
have no closed form, "can't widen *fp-64, expr is ?", and scalar evolution
gives no count for the full loop. Then i is not known either. The main pass
goes through the 600 iterations, 153987 insns, and adds a may_goto without
need. pyperf600, the same loop unrolled 150 times, and strobemeta are
rejected: "math between map_value pointer and register with unbounded .."
res_spin_lock_test_held_lock_max, scx_lavd/lavd_dispatch and
arena_topology_node_init of scx_lavd, scx_chaos and scx_p2dq are rejected
too.

Keep the count. bpf_compute_loop_iters() reports the registers and slots
without a closed form in forget_regs, and bpf_flow_loop_enter() selects
a method:

1. No check depends on the value, bpf_flow_is_needed(): the bytes of sym,
   a hash, a checksum. It is an unknown scalar in the state that covers all
   iterations. pyperf600_nounroll: 1241 insns, no may_goto.

2. A check depends on it. Selftest strobemeta_nounroll1:
     for (int i = 0; i < STROBE_MAX_STRS; ++i)               /* 25 */
         payload_off = read_str_var(cfg, i, .., data, payload_off);
   read_str_var() reads a string to &data->payload[off] and returns off or
   off + len. payload_off is loop-carried, but the step is not a constant:
   "(any r8 (+ r8 ?))". As an unknown scalar: the same "math between ..".
   Walk the loop: each iteration has its own state, with [0, i] in it. The
   count tells where the walk ends. The limit is 100000 insns. 17 progs of
   the selftests (strobemeta, profiler2, linked_list, xdp_synproxy) and
   scx_lavd/lavd_dispatch need it. A pointer, and a slot that does not hold
   one full register, always count as needed.

3. The walk is too long: an unknown scalar, as in 1.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   4 ++
 kernel/bpf/scev.c            | 103 +++++++++++++++++++++++++++++------
 kernel/bpf/verifier.c        |  16 +++++-
 3 files changed, 106 insertions(+), 17 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 91ef4e0a1204..bd62bbb90b39 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -353,6 +353,8 @@ struct bpf_loop_iters {
 	u32 min_header_count;   /* min number of times header is executed */
 	u32 max_header_count;   /* max number of times header is executed */
 	unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/* live, changed by the loop, and there is no summary of how */
+	unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* [0, max] at the entry, and the loop only clears bits */
 	unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/*
@@ -362,6 +364,8 @@ struct bpf_loop_iters {
 	 */
 	int iv_reg;
 	struct cnum64 iv_range;
+	/* a pointer to the stack that stores go through moves in the loop */
+	bool spill_base;
 };
 
 /* Where may_goto that the verifier adds goes, see check_loop_guard() */
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 8ce188a715e8..c3ce13a52f6f 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1967,8 +1967,10 @@ scev_reg_for_read(struct bpf_verifier_env *env, const struct bpf_func_state *st,
 
 /*
  * Materialize a writable value in the current state, growing its stack if needed.
- * This does not go through check_stack_write(). The callers skip is_unbound_scev_reg() slots,
- * thus the buffer of a live dynptr slice, see bpf_reg_state::buf, does not become a spill here.
+ * This does not go through check_stack_write(). The buffer of a live dynptr slice, see
+ * bpf_reg_state::buf, does not become a spill here. The callers skip the slots that are
+ * is_unbound_scev_reg() when the loop is entered. A slot that a call in the loop may write
+ * has no description: it is in forget_regs, and they skip these too.
  */
 static int scev_reg_for_write(struct bpf_verifier_env *env, struct bpf_func_state *st,
 			      u32 r, struct bpf_reg_state **out)
@@ -3345,7 +3347,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	struct bpf_loop *loop;
 	int insn_idx = st->insn_idx;
 	u64 until, false_until;
-	bool same = true;
+	bool same = true, walk = false;
 	int err, i;
 	u32 r;
 
@@ -3438,10 +3440,12 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	if (err)
 		return err;
 
+	bitmap_zero(iters->forget_regs, REGS_NUM);
 	bitmap_zero(iters->mask_regs, REGS_NUM);
+	iters->spill_base = false;
 	header_env = find_header_env(scev, insn_idx);
 	for (r = 0; r < REGS_NUM; r++) {
-		struct bpf_reg_state *reg;
+		struct bpf_reg_state *reg = NULL;
 		u32 ra, r_scev, r_expr;
 		bool spill_base = false;
 
@@ -3488,13 +3492,14 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 
 		reg = scev_reg_for_linear(cur_func, r);
 		/*
-		 * The offset of a pointer to arena is not tracked: what enters the loop stands
-		 * for all iterations. If the loop makes something else of it the state that
-		 * returns to the head is not covered by the state at the head, and the body is
-		 * verified again from that state.
+		 * The offset of a pointer to arena is not tracked, so there is nothing to lose.
+		 * If the loop makes something else of it the state that returns to the head
+		 * is not covered by the state at the head.
 		 */
-		if (reg && reg->type == PTR_TO_ARENA)
+		if (reg && reg->type == PTR_TO_ARENA) {
+			__set_bit(r, iters->forget_regs);
 			continue;
+		}
 		/* If SCEV for r is (linear <reg> <slope>) */
 		if (reg && is_simple_linear(scev, r_scev, &coefs) && is_widenable_reg_type(reg)) {
 			/* 32-bit ALU operations on pointers convert them to scalars */
@@ -3543,10 +3548,69 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 				bpf_log(log, ", requires exact stack-offset tracking");
 			bpf_log(log, "\n");
 		}
-		return 0;
+		/* the count stands, it is this one that is not known in the loop */
+		if (spill_base) {
+			iters->spill_base = true;
+			walk = true;
+			continue;
+		}
+		__set_bit(r, iters->forget_regs);
+		/*
+		 * Only a walk through the iterations tells what it is in each of them. That is not
+		 * worth it for a number that nothing looks at: a sum, a hash, what a call leaves in
+		 * r1 - r5. It's not known in the loop then. A pointer, or a slot that holds
+		 * something else than a register, would be of no use any more.
+		 */
+		if (!reg || (reg->type != SCALAR_VALUE && reg->type != NOT_INIT) ||
+		    bpf_flow_is_needed(env, insn_idx, r)) {
+			walk = true;
+			continue;
+		}
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, nothing looks at it\n", insn_idx);
 	}
 
-	return 1;
+	/* 2: the count is known, what some of the live registers are is not and it matters */
+	return walk ? 2 : 1;
+}
+
+/* The loop changes @r in a way that has no summary. If it is a number it is not known. */
+static void scev_forget_reg(struct bpf_verifier_env *env, struct bpf_func_state *func, u32 r)
+{
+	struct bpf_stack_state *stack;
+	int i;
+
+	if (r < MAX_BPF_REG) {
+		if (func->regs[r].type == SCALAR_VALUE) {
+			bpf_mark_reg_unknown_imprecise(&func->regs[r]);
+			mark_scev_reg_scratched(env, r);
+		} else if (func->regs[r].type == PTR_TO_ARENA) {
+			/* NULL or not is all that is known about it */
+			bpf_reg_forget_range(&func->regs[r]);
+			mark_scev_reg_scratched(env, r);
+		}
+		return;
+	}
+	stack = scev_regno_to_stack(func, r);
+	if (!stack)
+		return;
+	if (bpf_is_spilled_reg(stack) && stack->spilled_ptr.type == PTR_TO_ARENA) {
+		bpf_reg_forget_range(&stack->spilled_ptr);
+		mark_scev_reg_scratched(env, r);
+		return;
+	}
+	if (bpf_is_spilled_reg(stack) && stack->spilled_ptr.type != SCALAR_VALUE)
+		return;
+	for (i = 0; i < BPF_REG_SIZE; i++)
+		if (stack->slot_type[i] == STACK_SPILL || stack->slot_type[i] == STACK_ZERO)
+			break;
+	if (i == BPF_REG_SIZE)
+		return;
+	for (i = 0; i < BPF_REG_SIZE; i++)
+		if (stack->slot_type[i] == STACK_SPILL || stack->slot_type[i] == STACK_ZERO)
+			stack->slot_type[i] = STACK_MISC;
+	bpf_mark_reg_not_init(env, &stack->spilled_ptr);
+	mark_scev_reg_scratched(env, r);
 }
 
 /* bpf_compute_loop_iters() checked that every leaf can be unioned into acc. */
@@ -3591,6 +3655,11 @@ static int union_any_reg(struct bpf_verifier_env *env, struct bpf_func_state *lo
 	return 0;
 }
 
+/*
+ * Make @st, which is at the head of a counted loop, stand for all iterations. @loop_entry is the
+ * state that entered the loop and @iters is what bpf_compute_loop_iters() says about it.
+ * @st is a copy of @loop_entry.
+ */
 int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
 			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters)
 {
@@ -3621,16 +3690,18 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 		if (!scev_reg_alive(env, st, r))
 			continue;
 
+		if (test_bit(r, iters->forget_regs)) {
+			scev_forget_reg(env, cur_func, r);
+			continue;
+		}
+
 		r_expr = header_env->reg2scev[r];
 		spill32 = spill32_counter(scev, cur_func, r_expr, r, &coefs);
-		if (!spill32 && (is_unbound_scev_reg(env, cur_func, r) ||
+		/* @iters says nothing about a register that is not known in @loop_entry */
+		if (!spill32 && (is_unbound_scev_reg(env, entry_func, r) ||
 				 is_const_scev_reg(env, insn_idx, r)))
 			continue;
 
-		reg = scev_reg_for_linear(cur_func, r);
-		if (reg && reg->type == PTR_TO_ARENA)
-			continue;
-
 		if (test_bit(r, iters->mask_regs)) {
 			/* The range stays. It's no longer equal to the registers with its id. */
 			err = scev_reg_for_write(env, cur_func, r, &reg);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4801e9ac0089..103f36803563 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21630,8 +21630,22 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	if (err < 0)
 		return err;
 
-	/* @per_trip: the insns of one iteration, times the walks around the loop. */
+	/*
+	 * There is no summary of what the loop does to something that is live.
+	 * Go through the iterations one by one if that is not too long.
+	 * @per_trip: the insns of one iteration, times the walks around the loop.
+	 */
 	per_trip = flow_walk_factor(cur) * loop->nr_insns;
+	if (err == 2 && per_trip <= BPF_FLOW_MAX_WALK &&
+	    iters.max_header_count <= BPF_FLOW_MAX_WALK / (u32)per_trip) {
+		entry->max_header_count = iters.max_header_count;
+		entry->terminates = true;
+		entry->counted = true;
+		entry->apart = true;
+		return 0;
+	}
+	if (err == 2 && iters.spill_base)
+		err = 0;
 	/* The walk has to get to the end of the loop, so it is known to have one. Or not: */
 	if (!err) {
 		err = flow_try_walk(env, entry);
-- 
2.55.0

Re: [PATCH bpf-next v3 20/45] bpf: Keep the count of a loop when a register has no closed form

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3813 bytes --]

> bpf: Keep the count of a loop when a register has no closed form

TL;DR: In bpf_flow_loop_enter(), a long counted loop whose register has no
closed form may no longer fall back to flow_try_walk(), so programs that
loaded before could be rejected (unbounded min value), until later patches.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -21630,8 +21630,22 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
>  	if (err < 0)
>  		return err;
>  
> -	/* @per_trip: the insns of one iteration, times the walks around the loop. */
> +	/*
> +	 * There is no summary of what the loop does to something that is live.
> +	 * Go through the iterations one by one if that is not too long.
> +	 * @per_trip: the insns of one iteration, times the walks around the loop.
> +	 */
>  	per_trip = flow_walk_factor(cur) * loop->nr_insns;
> +	if (err == 2 && per_trip <= BPF_FLOW_MAX_WALK &&
> +	    iters.max_header_count <= BPF_FLOW_MAX_WALK / (u32)per_trip) {
> +		entry->max_header_count = iters.max_header_count;
> +		entry->terminates = true;
> +		entry->counted = true;
> +		entry->apart = true;
> +		return 0;
> +	}
> +	if (err == 2 && iters.spill_base)
> +		err = 0;
>  	/* The walk has to get to the end of the loop, so it is known to have one. Or not: */
>  	if (!err) {
>  		err = flow_try_walk(env, entry);

When bpf_compute_loop_iters() returns 2 because a register that a check
depends on has no closed form, and the counted walk is longer than
BPF_FLOW_MAX_WALK, this code no longer falls back to flow_try_walk().
Unless spill_base is set, err stays 2 and the code goes on to
bpf_widen_scev_regs(), which calls scev_forget_reg() and turns the needed
scalar into an unknown scalar for all iterations.

Before this commit that register hit cant_widen, bpf_compute_loop_iters()
returned 0, and the loop was walked by flow_try_walk().  If that walk gave
up, the loop got a may_goto and the states at the head were joined, which
keeps any bound that each iteration gives the value.

Can this reject programs that loaded before?  For example:

  idx = 0;
  for (i = 0; i < 100000; i++)
          idx = table[idx];       /* u8 table[256] */

Here idx has no closed form and bpf_flow_is_needed() reports it as needed,
so the return value is 2.  per_trip is about 12 and max_header_count of
100000 is larger than BPF_FLOW_MAX_WALK / 12, so the counted walk is
skipped.  spill_base is not set, so idx goes through scev_forget_reg() and
bpf_mark_reg_unknown_imprecise(), and then table + idx fails with "math
between map_value pointer and register with unbounded min value".  Before,
the join kept idx in [0, 255].  The same seems to apply to a value that a
call returns in each iteration, such as bpf_get_prandom_u32() & 15 used as
an index over 20000 iterations.

The commit message lists this as method 3 ("The walk is too long: an
unknown scalar, as in 1.") but does not mention that such programs are now
rejected.

Later commits in this series address this: 602c3e06835f ("bpf: Keep the
first iteration of a counted loop apart") describes that "a prog that
loaded before is rejected: math between map_value pointer and register with
unbounded min value" and handles values loaded anew in each iteration, and
01f19b3a79be ("bpf: What a call returns in each iteration is like what the
loop loads") extends that to call results.  The fallback in
bpf_flow_loop_enter() is unchanged at the end of the series, so is this
commit bisect-clean for such programs?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 21/45] bpf: Verify static subprogs on their own

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest test_verif_scale1, 90 times in a row, jhash() is static noinline:
  ptr = data + i;
  if (ptr + nh_off > data_end)
      break;
  ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++);
The verifier verifies a static subprog in the frames of its callers. States
with different frames have no join: each insn of jhash() has a state for
each way to it: "more than 1024 states that cannot be joined at insn 903".
In Rust each function of core and alloc is a static subprog: 30 tests of
rust-bpf are rejected too, its schedulers take 6 times the insns, those of
sched-ext/scx +9 .. +15 %.

Verify a static subprog for the states at its first insn. The entry state
has one frame, with the arguments and nothing of the callers. It arrives at
the first insn like each state at an insn: dropped, joined or added. The
call waits. Each exit state goes to all calls that wait for this entry.

  2: call f  --- { r1=scalar } --->  +---------------------+
  7: call f  --- { r1=scalar } --->  | f: verified 1 time  |
  3: ...     <-- { r0=[0,7] } -----  |                     |
  8: ...     <-- { r0=[0,7] } -----  +---------------------+

test_verif_scale1: 50765 insns, today 530362.

The entry state has what a check asks the frames of the callers for: the
references and locks that they hold, sleepable, rcu callback.
inherited_id() gives what is held a name, an id that depends only on its
position, from 2^29: calls that differ only in ids share an entry. 27 progs
of the schedulers of rust-bpf call core::fmt while they hold a reference.
All other ids stay below 2^28: "too many ids". A scalar argument loses its
id and its delta. The frames of the callers stay, func_is_on_its_own(), if
 - an argument is a pointer to the stack or a dynptr: it is their memory,
 - an argument is a pointer with the id of something that is not held: the
   id of its NULL check, or of a variable offset into the packet,
 - an argument can be NULL or is the end of the packet: the subprog does
   a check for the caller ('direct packet access: test29'),
 - the subprog is a callback, changes packet data, has a tail call or
   LD_ABS, or releases what the caller holds.
BPF_LOG_STATS adds its insns to each caller: verifier_subprog_insn_stats.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   3 +
 kernel/bpf/verifier.c        | 224 ++++++++++++++++++++++++++++++++++-
 2 files changed, 226 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index bd62bbb90b39..2a47184a5ff2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1195,6 +1195,9 @@ struct bpf_id_pair {
  *   check_ids() and jid_note() do not rename it. So two calls share a state at the entry, and what
  *   comes out for it, only if the same places have the same names.
  * - bpf_flow_return() puts the ids of the call that waits back. All other ids get new values.
+ * - What comes out has the ranges of all calls that share the state at the entry. So a name is good
+ *   for what holds whatever the ranges are: the same object. Other ids have no name: a number
+ *   loses its id, any other argument keeps the frames of the caller, see arg_needs_caller().
  * - env->id_gen stays below BPF_MAX_ID. So no id is a name or has a bit of BPF_ADD_CONST.
  */
 #define BPF_INHERITED_ID	0x20000000
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 103f36803563..b61166f67094 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1822,6 +1822,7 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	dst_state->in_sleepable = src->in_sleepable;
 	dst_state->callers_unknown = src->callers_unknown;
 	dst_state->unwound = src->unwound;
+	dst_state->changed_pkt_data = src->changed_pkt_data;
 	dst_state->flow_entry = src->flow_entry;
 	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
@@ -11292,6 +11293,8 @@ static void clear_all_pkt_pointers(struct bpf_verifier_env *env)
 			mark_reg_invalid(env, reg);
 		}
 	}));
+	if (env->cur_state->flow_entry)
+		env->cur_state->changed_pkt_data = true;
 }
 
 static bool holds_pkt_pointers(struct bpf_verifier_env *env)
@@ -11840,6 +11843,180 @@ static void account_current_path(struct bpf_verifier_env *env);
  * BPF_INHERITED_ID + the position for the function, and put back at the return.
  */
 
+/* 0 if it is not held */
+static u32 inherited_id(const struct bpf_verifier_state *state, u32 id)
+{
+	int i;
+
+	/* the first one: a lock in an object that the caller owns has the id of the reference */
+	for (i = 0; id && i < state->acquired_refs; i++)
+		if (state->refs[i].id == id)
+			return BPF_INHERITED_ID + i;
+	return 0;
+}
+
+/* What func_is_on_its_own() asks of an argument, be it in a register or on the stack */
+static bool arg_needs_caller(const struct bpf_verifier_state *state,
+			     const struct bpf_subprog_info *sub, const struct bpf_reg_state *reg)
+{
+	/* only the ids of what is held have names */
+	if (reg->type != SCALAR_VALUE &&
+	    ((reg->id && !inherited_id(state, reg->id)) ||
+	     (reg->parent_id && !inherited_id(state, reg->parent_id)) ||
+	     (reg->map_uid && !inherited_id(state, reg->map_uid))))
+		return true;
+	/*
+	 * Pointers to the packet that share an id have ranges of the same size, see
+	 * find_good_pkt_pointers(). What comes out of the function has the range of all calls.
+	 */
+	if (reg_is_pkt_pointer(reg) && reg->id)
+		return true;
+	/* it checks how much of the packet is there, for the caller */
+	if (reg->type == PTR_TO_PACKET_END && !sub->recursive)
+		return true;
+	/* or whether there is something to point to */
+	return type_may_be_null(reg->type) && !sub->recursive;
+}
+
+static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier_state *state,
+			       int subprog)
+{
+	struct bpf_func_state *caller = state->frame[state->curframe];
+	struct bpf_subprog_info *sub = &env->subprog_info[subprog];
+	u16 live = env->insn_aux_data[sub->start].live_regs_before;
+	u32 i;
+
+	if (sub->changes_pkt_data || sub->has_tail_call || sub->has_ld_abs ||
+	    sub->is_cb || sub->is_async_cb || sub->is_exception_cb)
+		return false;
+	if (state->speculative)
+		return false;
+	if (sub->releases_inherited &&
+	    (state->acquired_refs || state->active_rcu_locks || state->active_preempt_locks))
+		return false;
+	/* what must not be done in that callback is refused by looking at the frames */
+	if (in_rbtree_lock_required_cb(env))
+		return false;
+	/* there are that many names for what is held, and bpf_flow_return() has room for them */
+	if (state->acquired_refs > BPF_ID_MAP_SIZE / 2)
+		return false;
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+		if (!(live & BIT(i)))
+			continue;
+		if (caller->regs[i].type == PTR_TO_STACK ||
+		    base_type(caller->regs[i].type) == CONST_PTR_TO_DYNPTR)
+			return false;
+		if (arg_needs_caller(state, sub, &caller->regs[i]))
+			return false;
+	}
+	for (i = 0; i < bpf_in_stack_arg_cnt(sub); i++) {
+		if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
+		    base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
+			return false;
+		if (arg_needs_caller(state, sub, &caller->stack_arg_regs[i]))
+			return false;
+	}
+	return true;
+}
+
+static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int insn_idx)
+{
+	u16 live = env->insn_aux_data[env->subprog_info[subprog].start].live_regs_before;
+	struct bpf_verifier_state *state = env->cur_state;
+	struct bpf_func_state *caller = state->frame[state->curframe], *callee;
+	struct bpf_verifier_state entry = {};
+	int i, err;
+
+	callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
+	if (!callee)
+		return -ENOMEM;
+	entry.frame[0] = callee;
+	entry.in_sleepable = state->in_sleepable;
+	entry.callers_unknown = state->callers_unknown;
+	init_func_state(env, callee, BPF_MAIN_FUNC, 0, subprog);
+	/* the frame that says so is not there when the function is verified */
+	callee->in_rcu_callback_fn = in_rcu_callback(env);
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+		if (live & BIT(i))
+			callee->regs[i] = caller->regs[i];
+	err = grow_stack_arg_slots(env, callee, bpf_in_stack_arg_cnt(&env->subprog_info[subprog]));
+	if (err) {
+		bpf_free_verifier_state(&entry, false);
+		return err;
+	}
+	for (i = 0; i < callee->out_stack_arg_cnt; i++)
+		callee->stack_arg_regs[i] = caller->stack_arg_regs[i];
+
+	err = copy_reference_state(&entry, state);
+	if (err) {
+		bpf_free_verifier_state(&entry, false);
+		return err;
+	}
+	for (i = 0; i < entry.acquired_refs; i++) {
+		/* the id of a lock is the id of the pointer to it, which may have none */
+		entry.refs[i].id = inherited_id(state, entry.refs[i].id);
+		if (entry.refs[i].type == REF_TYPE_PTR)
+			entry.refs[i].parent_id = inherited_id(state, entry.refs[i].parent_id);
+	}
+	entry.active_irq_id = inherited_id(state, entry.active_irq_id);
+	entry.active_lock_id = inherited_id(state, entry.active_lock_id);
+	callee->entry_refs = entry.acquired_refs;
+	callee->entry_rcu_locks = entry.active_rcu_locks;
+	callee->entry_preempt_locks = entry.active_preempt_locks;
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+		/* what a scalar is linked to stays with the caller, so does the delta to it */
+		if (callee->regs[i].type == SCALAR_VALUE)
+			clear_scalar_id(&callee->regs[i]);
+		callee->regs[i].id = inherited_id(state, callee->regs[i].id);
+		callee->regs[i].parent_id = inherited_id(state, callee->regs[i].parent_id);
+		callee->regs[i].map_uid = inherited_id(state, callee->regs[i].map_uid);
+	}
+	for (i = 0; i < callee->out_stack_arg_cnt; i++) {
+		if (callee->stack_arg_regs[i].type == SCALAR_VALUE)
+			clear_scalar_id(&callee->stack_arg_regs[i]);
+		callee->stack_arg_regs[i].id = inherited_id(state, callee->stack_arg_regs[i].id);
+		callee->stack_arg_regs[i].parent_id =
+			inherited_id(state, callee->stack_arg_regs[i].parent_id);
+		callee->stack_arg_regs[i].map_uid =
+			inherited_id(state, callee->stack_arg_regs[i].map_uid);
+	}
+
+	account_current_path(env);
+	env->subprog_info[subprog].on_its_own = true;
+	err = bpf_flow_call(env, state, &entry, subprog, insn_idx);
+	bpf_free_verifier_state(&entry, false);
+	/* the call waits, there is nothing to go on with */
+	return err ?: PROCESS_BPF_EXIT;
+}
+
+/* The current state is at exit of a function that is verified on its own. */
+static int exit_func_on_its_own(struct bpf_verifier_env *env)
+{
+	struct bpf_verifier_state *state = env->cur_state;
+	struct bpf_func_state *callee = state->frame[0];
+	u32 i, nregs;
+	int err;
+
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+			continue;
+		verbose(env, "cannot return stack pointer to the caller\n");
+		return -EINVAL;
+	}
+	state->unwound |= bpf_is_unwind_exit(&env->prog->insnsi[env->insn_idx]);
+
+	/* Invalidate callee-local dynptrs and their slices before the frame goes away. */
+	err = destroy_dynptrs_in_stack_slots(env, callee, 0,
+					     callee->allocated_stack / BPF_REG_SIZE - 1);
+	if (err)
+		return err;
+
+	account_current_path(env);
+	err = bpf_flow_exit(env, state, env->insn_idx);
+	return err ?: PROCESS_BPF_EXIT;
+}
+
 static u32 fresh_id(struct bpf_verifier_env *env, struct bpf_idmap *idmap, u32 id)
 {
 	u32 flags = id & BPF_ADD_CONST, i;
@@ -12003,6 +12180,9 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
 	if (err)
 		return err;
 
+	if (func_is_on_its_own(env, state, subprog))
+		return call_func_on_its_own(env, subprog, *insn_idx);
+
 	/*
 	 * For regular function entry setup new frame and continue
 	 * from that frame.
@@ -20907,6 +21087,9 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 {
 	struct bpf_func_state *cur_frame = cur_func(env);
 
+	if (!exception_exit && !env->cur_state->curframe && env->cur_state->flow_entry)
+		return exit_func_on_its_own(env);
+
 	/* We must do check_reference_leak here before
 	 * prepare_func_exit to handle the case when
 	 * state->curframe > 0, it may be a callback function,
@@ -21857,6 +22040,12 @@ static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 		if (need_resched())
 			cond_resched();
 
+		/* no insn makes as many ids as there are from here to BPF_INHERITED_ID */
+		if (env->id_gen >= BPF_MAX_ID) {
+			verbose(env, "too many ids\n");
+			return -E2BIG;
+		}
+
 		env->prev_insn_idx = prev_insn_idx;
 		if (env->insn_idx >= insn_cnt) {
 			verbose(env, "invalid insn idx %d insn_cnt %d\n",
@@ -23452,6 +23641,37 @@ static int do_check_all(struct bpf_verifier_env *env)
 	}
 }
 
+/*
+ * What it took to verify a static subprog on its own is not in insns_total of its callers.
+ * Add it, once per caller. The main prog and global subprogs have all of it already.
+ */
+static void add_insns_of_callees(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	struct bpf_insn *insn = env->prog->insnsi;
+	int *added, i, j, s, c;
+
+	added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
+	if (!added)
+		return;
+	/* callees first */
+	for (i = 0; i < env->subprog_cnt; i++) {
+		s = env->subprog_topo_order[i];
+		if (!s || bpf_subprog_is_global(env, s))
+			continue;
+		for (j = si[s].start; j < si[s + 1].start; j++) {
+			if (!bpf_pseudo_call(&insn[j]))
+				continue;
+			c = bpf_find_subprog(env, j + insn[j].imm + 1);
+			if (c <= 0 || c == s || !si[c].on_its_own || added[c] == s)
+				continue;
+			added[c] = s;
+			si[s].insns_total += si[c].insns_total;
+		}
+	}
+	kvfree(added);
+}
+
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
@@ -25523,10 +25743,12 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		goto skip_full_check;
 
 	ret = do_check_all(env);
-
 	/* reject recursion through the callx edges found by the main pass */
 	if (ret == 0 && env->callx_edges)
 		ret = sort_subprogs_topo(env);
+	/* while the calls are what they are in the program */
+	if (env->log.level & BPF_LOG_STATS)
+		add_insns_of_callees(env);
 
 	/* the driver numbers the insns like the program that it was given */
 	if (ret == 0 && bpf_prog_is_offloaded(env->prog->aux))
-- 
2.55.0

Re: [PATCH bpf-next v3 21/45] bpf: Verify static subprogs on their own

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5363 bytes --]

> bpf: Verify static subprogs on their own

TL;DR: Verifying static subprogs on their own may reject previously loading
programs (struct_ops ref context, subprog loading data_end); later series
patches fix both. PTR_TO_STACK type-flag compare looks latent.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +/* What func_is_on_its_own() asks of an argument, be it in a register or on the stack */
> +static bool arg_needs_caller(const struct bpf_verifier_state *state,
> +			     const struct bpf_subprog_info *sub, const struct bpf_reg_state *reg)
> +{
> +	/* only the ids of what is held have names */
> +	if (reg->type != SCALAR_VALUE &&
> +	    ((reg->id && !inherited_id(state, reg->id)) ||
> +	     (reg->parent_id && !inherited_id(state, reg->parent_id)) ||
> +	     (reg->map_uid && !inherited_id(state, reg->map_uid))))
> +		return true;

Can this reject a struct_ops program that holds a reference it got at entry
through a __ref context argument?

The context register is PTR_TO_CTX with id 0, so arg_needs_caller() returns
false and func_is_on_its_own() lets the callee be verified on its own.
call_func_on_its_own() then renames every held reference:

    entry.refs[i].id = inherited_id(state, entry.refs[i].id);

When the callee loads the referenced pointer from the context,
__check_ctx_access() looks up the original id kept in ctx_arg_info[]:

    if (info->ref_id &&
        !find_reference_state(env->cur_state, info->ref_id)) {
            verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n", off);
            return -EACCES;

The entry state only has the BPF_INHERITED_ID + i names, so the lookup
fails.  For example, the skb of Qdisc_ops.enqueue, or the task in the
selftest 'struct_ops/test_refcounted', passed on to:

    static __noinline int pid_of(unsigned long long *ctx)
    { struct task_struct *task = (void *)ctx[1]; return task->pid; }

would now be rejected, while it loaded before this patch.  The commit
message lists the cases that keep the frames of the caller, and a context
whose ids live outside the register state is not among them.

A later commit in the series, d67af9e9ebf4 ("bpf: Keep the frames of the
caller for ctx of struct_ops with a reference"), adds the missing case to
arg_needs_caller().

> +	/*
> +	 * Pointers to the packet that share an id have ranges of the same size, see
> +	 * find_good_pkt_pointers(). What comes out of the function has the range of all calls.
> +	 */
> +	if (reg_is_pkt_pointer(reg) && reg->id)
> +		return true;
> +	/* it checks how much of the packet is there, for the caller */
> +	if (reg->type == PTR_TO_PACKET_END && !sub->recursive)
> +		return true;

Can a static subprog that checks a packet pointer for its caller now be
rejected when it loads data_end from the context itself, instead of taking
it as an argument?

    static __noinline bool has(struct xdp_md *ctx, void *p, u32 n)
    { return p + n <= (void *)(long)ctx->data_end; }
    ...
    if (!has(ctx, p, 14)) return XDP_DROP;
    return *(char *)(p + 13) ? XDP_PASS : XDP_DROP;

    invalid access to packet, off=13 size=1, R6(id=0,off=13,r=0)

The arguments are PTR_TO_CTX, PTR_TO_PACKET with id 0, and a scalar, so
arg_needs_caller() returns false for each of them.  In the callee,
find_good_pkt_pointers() walks bpf_for_each_reg_in_vstate() over the
callee's own one-frame state only, and the caller's r6 keeps r=0.  With the
frames of the caller, the same walk updated r6 in the caller's frame.

The commit message names "the subprog does a check for the caller" as a
reason to keep the frames, but only covers a PTR_TO_PACKET_END argument.

A later commit in the series, 8686ffec6023 ("bpf: Keep the frames of the
caller for a subprog that loads data_end"), adds
bpf_subprog_info.loads_pkt_end and keeps the frames for a PTR_TO_PACKET
argument of such a subprog.

[ ... ]

> +	for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
> +		if (!(live & BIT(i)))
> +			continue;
> +		if (caller->regs[i].type == PTR_TO_STACK ||
> +		    base_type(caller->regs[i].type) == CONST_PTR_TO_DYNPTR)
> +			return false;

A subsystem pattern flags this as potentially concerning: the exact compare
of caller->regs[i].type against PTR_TO_STACK imposes a restriction on a
match, and a register with any type flag set would skip it, while the
CONST_PTR_TO_DYNPTR test on the same line uses base_type().

Should this be base_type(caller->regs[i].type) == PTR_TO_STACK?  The same
applies to the stack_arg_regs loop below, and to the
reg->type == PTR_TO_PACKET_END test in arg_needs_caller().

If a pointer to the caller's stack ever carried a type flag, the callee
would be verified with no caller frame, and the pointer's frameno would
refer to a frame of the callee's own state.

In this tree I did not find a path that puts a flagged PTR_TO_STACK into a
register.  The only flagged values are the spilled_ptr of STACK_ITER slots,
which are never filled into a register, and callback setup copies
PTR_TO_STACK unchanged.  So this looks latent rather than reachable today.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

Re: [PATCH bpf-next v3 21/45] bpf: Verify static subprogs on their own

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] bpf: bpf_flow_return() drops speculative state flag on callee exit
- [Medium] bpf: call_func_on_its_own() loses async callback context in frame[0]
- [Medium] bpf: add_insns_of_callees() double-counts insns and misses callx edges
- [Medium] bpf: bpf_flow_return() lazy fallback leaves stale post-call states
--

commit 34bdfd07d4f78d7029c50c8aec27497f8117e123
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Verify static subprogs on their own

This commit verifies eligible static subprograms in a single standalone frame
rather than in the call frames of their callers, joining states at the
subprogram entry and applying exit states to all waiting call sites.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 103f368035638..b61166f670945 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -11840,6 +11843,180 @@ static void account_current_path(struct bpf_verifier_env *env);

[ ... ]

> +static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier_state *state,
> +			       int subprog)
> +{
> +	struct bpf_func_state *caller = state->frame[state->curframe];
> +	struct bpf_subprog_info *sub = &env->subprog_info[subprog];
> +	u16 live = env->insn_aux_data[sub->start].live_regs_before;
> +	u32 i;
> +
> +	if (sub->changes_pkt_data || sub->has_tail_call || sub->has_ld_abs ||
> +	    sub->is_cb || sub->is_async_cb || sub->is_exception_cb)
> +		return false;
> +	if (state->speculative)
> +		return false;
> +	if (sub->releases_inherited &&
> +	    (state->acquired_refs || state->active_rcu_locks || state->active_preempt_locks))
> +		return false;

[Severity: Medium]
What happens if a static subprogram verified on its own has multiple exit
paths, or is called from two sites whose entries are merged, where one exit
leaves caller-held references, locks, or packet data intact while a later
exit releases an inherited resource or changes packet data?

In kernel/bpf/flow.c:flow_apply_exit(), each exit state is applied to waiting
callers as it arrives:

	err = bpf_copy_verifier_state(&st, &w->st);
	if (!err)
		err = bpf_flow_return(env, &st, &x->st, call_idx);
	if (!err)
		err = bpf_flow_deliver(env, &st, call_idx + 1, call_idx);
	bpf_free_verifier_state(&st, false);
	if (err != 1)
		return err;

When an earlier non-releasing exit reaches flow_apply_exit() (or is applied
via flow_do_united() when two entry states are joined by flow_merge()), a
post-call state is delivered to call_idx + 1 while w remains on entry->calls.

When a subsequent exit reaches bpf_flow_return() and sets
sub->releases_inherited = true (or sub->changes_pkt_data = true), returning 1
re-queues w at call_idx for in-frame verification, but any post-call states
already delivered to call_idx + 1 from earlier exits are not retracted. Can
those stale post-call states continue executing in the caller and trigger
false check_resource_leak() failures at program exit?

[ ... ]

> +static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int insn_idx)
> +{
> +	u16 live = env->insn_aux_data[env->subprog_info[subprog].start].live_regs_before;
> +	struct bpf_verifier_state *state = env->cur_state;
> +	struct bpf_func_state *caller = state->frame[state->curframe], *callee;
> +	struct bpf_verifier_state entry = {};
> +	int i, err;
> +
> +	callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
> +	if (!callee)
> +		return -ENOMEM;
> +	entry.frame[0] = callee;
> +	entry.in_sleepable = state->in_sleepable;
> +	entry.callers_unknown = state->callers_unknown;
> +	init_func_state(env, callee, BPF_MAIN_FUNC, 0, subprog);
> +	/* the frame that says so is not there when the function is verified */
> +	callee->in_rcu_callback_fn = in_rcu_callback(env);

[Severity: Medium]
As reported by Sashiko on v2 [1], when a non-sleepable async callback
scheduled from a global subprogram verified with env->no_rcu_read_lock == true
calls a static subprogram verified on its own, call_func_on_its_own() replaces
entry.frame[0] with callee and only sets callee->in_rcu_callback_fn.

Because push_async_cb() sets async_cb->frame[0]->in_async_callback_fn = true
and async_cb->frame[0]->callback_ret_range rather than in_rcu_callback_fn,
has_rcu_read_lock() relies on state->frame[0]->in_async_callback_fn:

	if (state->active_rcu_locks || in_rcu_callback(env))
		return true;
	if (in_sleepable(env))
		return false;
	return !env->no_rcu_read_lock || state->frame[0]->in_async_callback_fn;

And check_return_code() (which check_kfunc_call() invokes for bpf_throw()
during do_check_all()) also checks frame->in_async_callback_fn and
frame->callback_ret_range on env->cur_state->frame[0].

Should call_func_on_its_own() also preserve state->frame[0]->in_async_callback_fn
and state->frame[0]->callback_ret_range on callee so RCU read-lock tracking
and async callback return-range checks are not lost inside the standalone
callee frame?

[ ... ]

> +/* The current state is at exit of a function that is verified on its own. */
> +static int exit_func_on_its_own(struct bpf_verifier_env *env)
> +{
> +	struct bpf_verifier_state *state = env->cur_state;
> +	struct bpf_func_state *callee = state->frame[0];
> +	u32 i, nregs;
> +	int err;
> +
> +	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
> +	for (i = 0; i < nregs; i++) {
> +		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
> +			continue;
> +		verbose(env, "cannot return stack pointer to the caller\n");
> +		return -EINVAL;
> +	}
> +	state->unwound |= bpf_is_unwind_exit(&env->prog->insnsi[env->insn_idx]);
> +
> +	/* Invalidate callee-local dynptrs and their slices before the frame goes away. */
> +	err = destroy_dynptrs_in_stack_slots(env, callee, 0,
> +					     callee->allocated_stack / BPF_REG_SIZE - 1);
> +	if (err)
> +		return err;
> +
> +	account_current_path(env);
> +	err = bpf_flow_exit(env, state, env->insn_idx);
> +	return err ?: PROCESS_BPF_EXIT;
> +}

[Severity: High]
As reported by bot+bpf-ci@kernel.org on v2 [2], when speculation mitigation is
active (!env->bypass_spec_v1 or !env->bypass_spec_v4), a static subprogram
entered with state->speculative == false can spawn speculative paths inside
the callee via check_cond_jmp_op(), sanitize_ptr_alu(), or
check_unwind_goto().

When such a speculative path reaches BPF_EXIT, process_bpf_exit_full() routes
it through exit_func_on_its_own() -> bpf_flow_exit() -> flow_apply_exit() ->
bpf_flow_return(). In bpf_flow_return(), caller->unwound is copied from
exit->unwound, but exit->speculative is not propagated to caller->speculative:

	if (exit->unwound && !bpf_is_unwind_goto(&env->prog->insnsi[call_idx + 1])) {
		verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
			call_idx);
		return -EINVAL;
	}
	caller->unwound = exit->unwound;

In addition, in kernel/bpf/flow.c:flow_merge(), a non-speculative state t->st
can subsume a speculative state s->st via flow_within() ->
bpf_state_subsumes() and unite their flow_entry sets via flow_unite().

Could this cause speculative return states to resume in the caller as
non-speculative architectural paths (marking dead instructions as seen in
sanitize_mark_insn_seen() and failing the state->speculative check on
recoverable errors in do_check_block()) or cross-unite distinct call entries?

[ ... ]

> @@ -23452,6 +23641,37 @@ static int do_check_all(struct bpf_verifier_env *env)
>  	}
>  }
>  
> +/*
> + * What it took to verify a static subprog on its own is not in insns_total of its callers.
> + * Add it, once per caller. The main prog and global subprogs have all of it already.
> + */
> +static void add_insns_of_callees(struct bpf_verifier_env *env)
> +{
> +	struct bpf_subprog_info *si = env->subprog_info;
> +	struct bpf_insn *insn = env->prog->insnsi;
> +	int *added, i, j, s, c;
> +
> +	added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
> +	if (!added)
> +		return;
> +	/* callees first */
> +	for (i = 0; i < env->subprog_cnt; i++) {
> +		s = env->subprog_topo_order[i];
> +		if (!s || bpf_subprog_is_global(env, s))
> +			continue;
> +		for (j = si[s].start; j < si[s + 1].start; j++) {
> +			if (!bpf_pseudo_call(&insn[j]))
> +				continue;
> +			c = bpf_find_subprog(env, j + insn[j].imm + 1);
> +			if (c <= 0 || c == s || !si[c].on_its_own || added[c] == s)
> +				continue;
> +			added[c] = s;
> +			si[s].insns_total += si[c].insns_total;
> +		}
> +	}
> +	kvfree(added);
> +}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [3], if a static subprogram c is
verified in-frame for one call from s (or falls back to in-frame verification
when bpf_flow_return() returns 1, or across a do_check_all() restart where
on_its_own is not cleared) and is also verified on its own for another call,
prepare_func_exit() -> account_processed_insns() has already added the
in-frame instructions of c into both si[c].insns_total and si[s].insns_total.
When add_insns_of_callees() then adds the full si[c].insns_total to
si[s].insns_total, won't the in-frame portion be counted twice in
si[s].insns_total?

Also, since check_func_callx() calls check_static_func_call() and can verify
indirect targets on their own via call_func_on_its_own(), does
add_insns_of_callees() need to account for callx edges in env->callx_edges in
addition to bpf_pseudo_call() instructions?

[1] https://lore.kernel.org/r/20261105204513.2230344-22-alexei.starovoitov@gmail.com
[2] https://lore.kernel.org/r/fbc6a76bd8cb0b115a602fe1f8064612f09afac8924a23ae5d64cf4e7fe68b43@mail.kernel.org
[3] https://lore.kernel.org/r/fbc6a76bd8cb0b115a602fe1f8064612f09afac8924a23ae5d64cf4e7fe68b43@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=21

[PATCH bpf-next v3 22/45] bpf: Allow recursive subprogs

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  candidates.sort_unstable_by(|a, b| b.0.cmp(&a.0));    scx_rusty.rs

quicksort() of the core library calls itself, and rusty_tick is rejected:
"recursive call from quicksort<..>() to quicksort<..>()". The same for
progs that drop a list, a tree or an Arc of a parent, that use BTreeMap,
serde_json or regex-automata: 100 of the 225 progs of the schedulers of
rust-bpf and 100 of its 437 tests have a subprog that gets to itself.
A verifier that follows each chain of calls has no choice: there is no
last chain.

  depth(n) { return n ? (depth(n - 1) + 1) & 255 : 0; }

The main pass verifies a static subprog for its entry states, and a call
waits for the exit states. A call of a subprog that is in progress waits
like each other call. The first insn of a recursive subprog is like a loop
head: the entry states are joined with widening. So are the exit states.

    main prog: call depth          entry state { r1=5 }
                  v
  +-->  first insn of depth()      join with widening: { r1=[0,5] }
  |        ...
  +---  call depth(n - 1)          entry state { r1=[0,4] }, the call waits
           ^
        exit of depth()            goes to all calls that wait

How deep the calls go and how many there are is not known. Thus each
subprog on a cycle of the call graph gets the checks of
"bpf: Limit depth and time of recursive subprogs at run time". When a check
fails the subprog returns 0. This is a new edge, and
check_recursion_guard() verifies it.
linux-notes.rst says 'recursive calls are limited at run time' now.

mark_recursive_subprogs() finds the cycles before the main pass: callx can
call each subprog whose address is taken, a helper calls its callback. The
main pass finds what callx calls. A mark that it does not confirm is
dropped, and the subprog gets no check.

Still rejected: a call of the main prog; recursion in an extension, in an
offloaded prog, with a tail call, LD_ABS or bpf_clone_redirect(), or in
a prog that gets a private stack without a request, like kprobe, raw_tp and
fentry progs; an extension of a subprog that is called in a recursion;
a pointer to the stack as an argument of a call between recursive subprogs.
In the static limit on the stack of a chain of calls a recursion counts as
one frame, the last one.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 Documentation/bpf/linux-notes.rst |   4 +-
 kernel/bpf/cfg.c                  |  43 ++++-
 kernel/bpf/liveness.c             |  40 ++++-
 kernel/bpf/verifier.c             | 281 ++++++++++++++++++++++++++----
 4 files changed, 318 insertions(+), 50 deletions(-)

diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
index 6c036b54a29f..12f0dc43e154 100644
--- a/Documentation/bpf/linux-notes.rst
+++ b/Documentation/bpf/linux-notes.rst
@@ -36,8 +36,8 @@ In both cases only static functions can be referenced. Therefore all functions
 that can be called indirectly are known to the verifier before it starts to
 analyze the program, and ``callx`` is verified as a direct call of every
 function that ``dst`` may point to at that instruction. The same rules apply:
-the calls can not be recursive, and the depth of the call chain and its
-combined stack size are limited.
+the depth of the call chain and its combined stack size are limited, and
+recursive calls are limited at run time.
 
 Calling helper or kernel functions through a register, indirect calls of global
 functions, and tail calls in functions that are called via ``callx`` are not
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 2ee950e9ac82..04c1c98ba248 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -75,21 +75,28 @@ static void mark_subprog_might_throw(struct bpf_verifier_env *env, int off)
 	subprog->might_throw = true;
 }
 
-/* 't' is an index of a call-site.
+/*
+ * 't' is an index of a call-site.
  * 'w' is a callee entry point.
  * Eventually this function would be called when env->cfg.insn_state[w] == EXPLORED.
- * Rely on DFS traversal order and absence of recursive calls to guarantee that
- * callee's effect marks would be correct at that moment.
+ * DFS traversal order guarantees that callee's effect marks are correct at that
+ * moment, unless the callee calls one of its callers: see close_callee_effects().
+ * Returns true if the marks of the caller changed.
  */
-static void merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
+static bool merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
 {
 	struct bpf_subprog_info *caller, *callee;
 
 	caller = bpf_find_containing_subprog(env, t);
 	callee = bpf_find_containing_subprog(env, w);
+	if (caller->changes_pkt_data >= callee->changes_pkt_data &&
+	    caller->might_sleep >= callee->might_sleep &&
+	    caller->might_throw >= callee->might_throw)
+		return false;
 	caller->changes_pkt_data |= callee->changes_pkt_data;
 	caller->might_sleep |= callee->might_sleep;
 	caller->might_throw |= callee->might_throw;
+	return true;
 }
 
 enum {
@@ -534,6 +541,33 @@ static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
 	return DONE_EXPLORING;
 }
 
+/*
+ * A subprog that calls one of its callers is EXPLORED before that caller is: it
+ * took the marks that the caller had so far, and so did everybody who called it
+ * since. Merge again at every call until nothing changes.
+ */
+static void close_callee_effects(struct bpf_verifier_env *env)
+{
+	struct bpf_insn *insn = env->prog->insnsi;
+	struct bpf_func_ptr *ptrs;
+	bool changed;
+	u32 cnt, j;
+	int t;
+
+	do {
+		changed = false;
+		for (t = 0; t < env->prog->len; t++) {
+			if (bpf_pseudo_call(&insn[t]) || bpf_pseudo_func(&insn[t]))
+				changed |= merge_callee_effects(env, t, t + insn[t].imm + 1);
+			ptrs = insn_func_ptrs(env, t, &cnt);
+			for (j = 0; j < cnt; j++)
+				changed |= merge_callee_effects(env, t, ptrs[j].xlated_off);
+			if (bpf_is_ldimm64(&insn[t]))
+				t++;
+		}
+	} while (changed);
+}
+
 /* Visits the instruction at index t and returns one of the following:
  *  < 0 - an error occurred
  *  DONE_EXPLORING - the instruction was fully explored
@@ -759,6 +793,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 		}
 	}
 	ret = 0; /* cfg looks good */
+	close_callee_effects(env);
 	env->prog->aux->changes_pkt_data = env->subprog_info[0].changes_pkt_data;
 	env->prog->aux->might_sleep = env->subprog_info[0].might_sleep;
 
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 6a2ee407412b..290d6e8147de 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2354,6 +2354,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 		struct func_instance *callee_instance;
 		int callee, target;
 		int caller_reg, cb_callee_reg;
+		u16 live;
 
 		j = idx - start; /* relative index within this subprog */
 		memset(callee_args, 0, MAX_AT_TRACK_REGS * sizeof(*callee_args));
@@ -2369,11 +2370,17 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 			if (callee < 0 || bpf_subprog_is_global(env, callee))
 				continue;
 
-			/* Build entry args: R1-R5 and stack args from at_in at call site */
+			/*
+			 * Build entry args: R1-R5 and stack args from at_in at call site.
+			 * What is left over in a register that the callee doesn't read is no arg.
+			 */
+			live = env->insn_aux_data[target].live_regs_before;
 			for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
-				callee_args[r] = info[subprog].at_in[j][r];
+				callee_args[r] = live & BIT(r) ? info[subprog].at_in[j][r] : none;
 			for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
-				callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r];
+				callee_args[MAX_BPF_REG + r] =
+					r < bpf_in_stack_arg_cnt(&env->subprog_info[callee]) ?
+					info[subprog].at_in[j][MAX_BPF_REG + r] : none;
 		} else if (bpf_calls_callback(env, idx)) {
 			callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg);
 			if (callee == -2) {
@@ -2408,6 +2415,17 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 		if (!has_fp_args(callee_args))
 			continue;
 
+		/*
+		 * at_in[] is kept per subprog and the snapshot of the stack per call insn:
+		 * the walk must not enter a subprog that it has not left.
+		 */
+		if (env->subprog_info[subprog].recursive && env->subprog_info[callee].recursive) {
+			verbose(env, "recursive %s() passes a pointer to the stack to %s()\n",
+				bpf_subprog_name(env, subprog), bpf_subprog_name(env, callee));
+			err = -EINVAL;
+			goto out_free;
+		}
+
 		if (depth == MAX_CALL_FRAMES - 1) {
 			verbose(env, "%s() passes a pointer to the stack to %s()\n",
 				bpf_subprog_name(env, subprog), bpf_subprog_name(env, callee));
@@ -2781,14 +2799,15 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
 	}
 
-	/* Forward pass: resolve stack access through FP-derived pointers */
-	err = bpf_compute_subprog_arg_access(env);
-	if (err)
-		goto out;
-
 	changed = true;
 	while (changed) {
 		changed = false;
+		/* a cycle of calls takes a round for each link */
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			goto out;
+		}
+		cond_resched();
 		for (i = 0; i < env->cfg.cur_postorder; ++i) {
 			int insn_idx = env->cfg.insn_postorder[i];
 			struct insn_live_regs *live = &state[insn_idx];
@@ -2838,6 +2857,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
 	}
 
+	/*
+	 * Forward pass: resolve stack access through FP-derived pointers.
+	 * It looks at live_regs_before of the callees.
+	 */
+	err = bpf_compute_subprog_arg_access(env);
 out:
 	kvfree(state);
 	return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b61166f67094..e29fc3ee363a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3441,7 +3441,7 @@ static int check_subprogs(struct bpf_verifier_env *env)
 /*
  * The callee of callx is known to the main verification pass only, which
  * records the 'caller' -> 'callee' edge of the call graph for the checks
- * that follow it: absence of recursion and the maximum stack depth.
+ * that follow it: recursion that was not seen coming and the maximum stack depth.
  */
 static int record_callx_edge(struct bpf_verifier_env *env, int caller, int callee)
 {
@@ -3529,9 +3529,9 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
  * the current subprog to discover callees (direct subprogs and sync
  * callbacks).
  *
- * The callees of callx are not known before the main verification pass.
- * When callx is used the sort is repeated after it with the recorded callx
- * edges added to the call graph to reject recursion through indirect calls.
+ * The callees of callx are not known before the main verification pass: callx
+ * is no edge here. The order within a cycle is arbitrary. For recursion see
+ * mark_recursive_subprogs() and check_recursion().
  */
 static int sort_subprogs_topo(struct bpf_verifier_env *env)
 {
@@ -3591,20 +3591,19 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env)
 				}
 				if (color[callee] == 2)
 					continue;
-				if (color[callee] == 1) {
-					if (bpf_pseudo_func(&insn[idx]))
-						continue;
-					verbose(env, "recursive call from %s() to %s()\n",
-						bpf_subprog_name(env, cur),
-						bpf_subprog_name(env, callee));
-					bpf_diag_program_structure(
-						env, idx, "recursive subprogram call",
-						"Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
-						"This bpf2bpf call would make the subprogram call graph recursive. "
-						"The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
+				/*
+				 * The call that is its own target never returns. It is
+				 * also what libbpf leaves in place of a call that it
+				 * could not resolve.
+				 */
+				if (bpf_pseudo_call(&insn[idx]) && insn[idx].imm == -1) {
+					verbose(env, "call at insn %d calls itself\n", idx);
 					ret = -EINVAL;
 					goto out;
 				}
+				/* recursion: the order within the cycle is arbitrary */
+				if (color[callee] == 1)
+					continue;
 				color[callee] = 1;
 				dfs_stack[top++] = callee;
 				pushed = true;
@@ -5851,7 +5850,8 @@ static void close_call_graph(unsigned long *calls, int cnt)
  *
  * What follows for a subprog from its callers, or from what it calls, is computed
  * over this graph and not by a walk of one chain of calls after another: the
- * number of chains grows exponentially.
+ * number of chains grows exponentially, and there is no end to them when subprogs
+ * call themselves.
  */
 struct bpf_call_graph {
 	int cnt; /* subprogs */
@@ -5861,6 +5861,7 @@ struct bpf_call_graph {
 	unsigned long *live; /* what the kernel calls and what that gets to */
 	unsigned long *in_cb; /* callbacks and what they get to */
 	unsigned long *in_async_cb; /* async callbacks and what they get to */
+	unsigned long *in_recursion; /* recursive subprogs and what they get to */
 	unsigned long *tail_calls; /* subprogs that have a tail call */
 	unsigned long *arena_sp; /* subprogs that count on r12 of the caller */
 };
@@ -5891,14 +5892,15 @@ static int build_call_graph(struct bpf_verifier_env *env, struct bpf_call_graph
 
 	cg->cnt = cnt;
 	cg->words = words;
-	cg->calls = kvcalloc((2 * cnt + 5) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+	cg->calls = kvcalloc((2 * cnt + 6) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
 	if (!cg->calls)
 		return -ENOMEM;
 	cg->reach = cg->calls + cnt * words;
 	cg->live = cg->reach + cnt * words;
 	cg->in_cb = cg->live + words;
 	cg->in_async_cb = cg->in_cb + words;
-	cg->tail_calls = cg->in_async_cb + words;
+	cg->in_recursion = cg->in_async_cb + words;
+	cg->tail_calls = cg->in_recursion + words;
 	cg->arena_sp = cg->tail_calls + words;
 
 	for (s = 0; s < cnt; s++) {
@@ -5933,6 +5935,8 @@ static int build_call_graph(struct bpf_verifier_env *env, struct bpf_call_graph
 			add_reach(cg, cg->in_cb, s);
 		if (si[s].is_async_cb)
 			add_reach(cg, cg->in_async_cb, s);
+		if (test_bit(s, subprog_reach(cg, s)))
+			add_reach(cg, cg->in_recursion, s);
 		if (si[s].has_tail_call)
 			__set_bit(s, cg->tail_calls);
 	}
@@ -5975,20 +5979,45 @@ static int check_exception_cb(struct bpf_verifier_env *env, struct bpf_call_grap
 	return 0;
 }
 
+static bool bpf_prog_reenters_datapath(const struct bpf_prog *prog);
+
 /*
- * The DFS of sort_subprogs_topo() follows the address of a function like a call, and
- * tolerates a cycle only when such address closes it, since it doesn't know what it is
- * taken for. Such cycle counts as a recursion, unless the function is an async callback:
- * its address is not an edge of the graph. The main verification pass limits the
- * depth of the recursion, but it doesn't follow calls of global functions.
+ * Every subprog on a cycle has to check the stack at run time. mark_recursive_subprogs()
+ * didn't see the cycle coming when a helper got the callback from a table of
+ * functions, or from another subprog. A subprog that it marked for callx, which turned out
+ * to call something else, needs no check.
  */
 static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
 {
 	int j, s;
 
 	for_each_set_bit(s, cg->live, cg->cnt) {
-		if (!test_bit(s, subprog_reach(cg, s)))
+		if (!test_bit(s, subprog_reach(cg, s))) {
+			env->subprog_info[s].recursive = false;
 			continue;
+		}
+		if (env->subprog_info[s].recursive) {
+			/*
+			 * These progs run on top of other progs. Their frames belong in the
+			 * private stack, which has room for one frame of a subprog.
+			 */
+			if (bpf_enable_priv_stack(env->prog) != NO_PRIV_STACK &&
+			    !env->prog->aux->priv_stack_requested) {
+				verbose(env, "recursive %s() in a tracing program\n",
+					bpf_subprog_name(env, s));
+				return -EINVAL;
+			}
+			/* It runs on top of its own frames. The half of the stack is not enough. */
+			if (bpf_prog_reenters_datapath(env->prog)) {
+				verbose(env, "recursive %s() in a program that calls bpf_clone_redirect()\n",
+					bpf_subprog_name(env, s));
+				return -EINVAL;
+			}
+			if (bpf_prog_has_invocation_budget(env->prog))
+				continue;
+			verbose(env, "JIT does not support recursive calls\n");
+			return -EOPNOTSUPP;
+		}
 		/* the last call of the cycle */
 		for_each_set_bit(j, subprog_reach(cg, s), cg->cnt)
 			if (test_bit(s, subprog_calls(cg, j)))
@@ -5997,9 +6026,9 @@ static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *
 			bpf_subprog_name(env, j), bpf_subprog_name(env, s));
 		bpf_diag_program_structure(
 			env, find_call_insn(env, j, s), "recursive subprogram call",
-			"Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
-			"This call or callback would make the subprogram call graph recursive. "
-			"The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
+			"Load the address of the callback in the subprogram that passes it to the helper, or rewrite the recursion as a loop.",
+			"This call or callback closes a cycle of the call graph that was not seen before the main pass: "
+			"the callback comes from a table of functions or from another subprogram. Nothing checks the stack.");
 		return -EINVAL;
 	}
 	return 0;
@@ -6114,7 +6143,8 @@ static int subprog_stack_size(struct bpf_verifier_env *env, int s)
 
 /*
  * A subprog is entered with no more than what one of its callers is entered with plus
- * the frame of that caller. The graph has no cycles, so this comes to an end.
+ * the frame of that caller. Not when both are recursive: the callee checks the stack at run
+ * time. A cycle adds nothing, so this comes to an end.
  */
 static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph *cg,
 			       struct bpf_subprog_call_depth_info *dinfo)
@@ -6134,11 +6164,17 @@ static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_gra
 		for_each_set_bit(s, cg->live, cg->cnt) {
 			if (dinfo[s].depth < 0)
 				continue;
-			depth = dinfo[s].depth + subprog_stack_size(env, s);
-			arena_depth = dinfo[s].arena_depth + si[s].arena_stack_depth;
 			for_each_set_bit(j, subprog_calls(cg, s), cg->cnt) {
 				callee = &dinfo[j];
-				frame = bpf_subprog_is_global(env, j) ? 0 : dinfo[s].frame + 1;
+				depth = dinfo[s].depth;
+				arena_depth = dinfo[s].arena_depth;
+				if (!si[s].recursive || !si[j].recursive) {
+					depth += subprog_stack_size(env, s);
+					arena_depth += si[s].arena_stack_depth;
+				}
+				frame = dinfo[s].frame + !si[j].recursive;
+				if (bpf_subprog_is_global(env, j))
+					frame = 0;
 				if (depth <= callee->depth && arena_depth <= callee->arena_depth &&
 				    frame <= callee->frame)
 					continue;
@@ -6265,6 +6301,10 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
 		 * of caller's stack as shown on the example above.
 		 */
 		depth = dinfo[s].depth;
+		/* nor of the frames of recursive subprogs, see bpf_insert_recursion_checks() */
+		si[s].in_recursion = test_bit(s, cg->in_recursion);
+		if (si[s].in_recursion)
+			depth += THREAD_SIZE / 2;
 		if (s && si[s].has_tail_call && depth >= 256) {
 			verbose(env,
 				"tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\n",
@@ -6302,6 +6342,151 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
 	return err;
 }
 
+/*
+ * The address of the callback at @idx goes nowhere but to a helper that keeps it for
+ * later: a timer, a work queue, call_rcu.
+ */
+static bool func_is_kept_for_later(struct bpf_verifier_env *env, int idx, int end)
+{
+	struct bpf_insn *insn = env->prog->insnsi;
+	u8 class, reg = insn[idx].dst_reg;
+	int i;
+
+	if (reg < BPF_REG_1 || reg > BPF_REG_5)
+		return false;
+	for (i = idx + 2; i < end; i++) {
+		class = BPF_CLASS(insn[i].code);
+		if (class == BPF_JMP || class == BPF_JMP32)
+			return bpf_is_async_callback_calling_insn(&insn[i]);
+		if (insn[i].dst_reg == reg)
+			return false;
+		if (bpf_is_ldimm64(&insn[i]))
+			i++;
+		else if (insn[i].src_reg == reg)
+			return false;
+	}
+	return false;
+}
+
+/*
+ * Find the subprogs on cycles of the call graph. It is not known yet what callx
+ * calls: it may be any subprog whose address is taken. A subprog that is
+ * marked for no reason is treated like a recursive one all the same.
+ */
+static int mark_recursive_subprogs(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	struct bpf_insn *insn = env->prog->insnsi;
+	int cnt = env->subprog_cnt, words = BITS_TO_LONGS(cnt);
+	unsigned long *calls, *taken, *callx;
+	bool plain_callx = false;
+	int i, j, s, err = 0;
+
+	/* calls + s * words: the subprogs that s calls */
+	calls = kvcalloc((cnt + 2) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+	if (!calls)
+		return -ENOMEM;
+	taken = calls + cnt * words;
+	callx = taken + words;
+
+	for (s = 0; s < cnt; s++) {
+		for (i = si[s].start; i < si[s + 1].start; i++) {
+			if (bpf_is_callx(&insn[i])) {
+				__set_bit(s, callx);
+				if (!bpf_is_unwind_goto(&insn[i + 1]))
+					plain_callx = true;
+			}
+			if (bpf_pseudo_call(&insn[i]) || bpf_pseudo_func(&insn[i])) {
+				j = bpf_find_subprog(env, i + insn[i].imm + 1);
+				if (j < 0)
+					continue;
+				/*
+				 * The first insns of the main prog are for the kernel. A call from
+				 * an async callback is on no cycle.
+				 */
+				if (!j && bpf_pseudo_call(&insn[i])) {
+					verbose(env, "recursive call of the main program\n");
+					kvfree(calls);
+					return -EINVAL;
+				}
+				/* the helper that calls a callback doesn't expect it to unwind */
+				if (bpf_pseudo_func(&insn[i]) || !bpf_is_unwind_goto(&insn[i + 1]))
+					si[j].plain_call = true;
+				/*
+				 * The helper that takes a callback calls it, or keeps it: the
+				 * callback that arms itself again doesn't call itself.
+				 */
+				if (bpf_pseudo_call(&insn[i]) ||
+				    !func_is_kept_for_later(env, i, si[s + 1].start))
+					__set_bit(j, calls + s * words);
+				if (bpf_pseudo_func(&insn[i]))
+					__set_bit(j, taken);
+			}
+			if (bpf_is_ldimm64(&insn[i]))
+				i++;
+		}
+	}
+	for (i = 0; i < env->func_ptr_cnt; i++) {
+		j = bpf_find_subprog(env, env->func_ptrs[i].xlated_off);
+		if (j >= 0)
+			__set_bit(j, taken);
+	}
+	for_each_set_bit(s, callx, cnt)
+		bitmap_or(calls + s * words, calls + s * words, taken, cnt);
+
+	/*
+	 * Callers of a global subprog look at can_unwind before the subprog is verified.
+	 * The main prog returns to the kernel.
+	 */
+	for (s = 0; s < cnt; s++) {
+		if (plain_callx && test_bit(s, taken))
+			si[s].plain_call = true;
+		si[s].bail_unwinds = si[s].can_unwind ||
+				     (s && !si[s].plain_call && !bpf_subprog_is_global(env, s) &&
+				      bpf_jit_supports_unwind() &&
+				      !bpf_prog_is_offloaded(env->prog->aux));
+	}
+
+	close_call_graph(calls, cnt);
+
+	for (s = 0; s < cnt; s++) {
+		if (!test_bit(s, calls + s * words))
+			continue;
+		err = -EINVAL;
+		if (!s) {
+			verbose(env, "recursive call of the main program\n");
+			break;
+		}
+		/*
+		 * Each call of the function that it replaces is an invocation of its own: a
+		 * loop in the prog that it extends gets a new time slice each time.
+		 */
+		if (env->prog->type == BPF_PROG_TYPE_EXT) {
+			verbose(env, "recursive %s() in an extension\n", bpf_subprog_name(env, s));
+			break;
+		}
+		/* the device runs the insns that the driver was given, without the checks */
+		if (bpf_prog_is_offloaded(env->prog->aux)) {
+			verbose(env, "recursive %s() in an offloaded program\n",
+				bpf_subprog_name(env, s));
+			break;
+		}
+		if (si[s].has_tail_call || si[s].has_ld_abs) {
+			verbose(env, "recursive %s() cannot have %s\n", bpf_subprog_name(env, s),
+				si[s].has_tail_call ? "tail calls" : "LD_ABS");
+			break;
+		}
+		if (si[s].bail_insn < 0) {
+			verbose(env, "recursive %s() has no exit\n", bpf_subprog_name(env, s));
+			break;
+		}
+		si[s].recursive = true;
+		err = 0;
+	}
+	kvfree(calls);
+	return err;
+}
+
 static int check_max_stack_depth(struct bpf_verifier_env *env)
 {
 	enum priv_stack_mode priv_stack_mode = PRIV_STACK_UNKNOWN;
@@ -6342,10 +6527,11 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
 		 * All async_cb subprogs use normal kernel stack. If a particular
 		 * subprog appears in both main prog and async_cb subtree, that
 		 * subprog will use normal kernel stack to avoid potential nesting.
+		 * The private stack has room for one frame of a subprog.
 		 * x86-64 uses R9 for both private stack frame pointer and arg6.
 		 */
 		if (priv_stack_mode != PRIV_STACK_ADAPTIVE || test_bit(s, cg.in_async_cb) ||
-		    (IS_ENABLED(CONFIG_X86_64) && si[s].stack_arg_cnt)) {
+		    si[s].recursive || (IS_ENABLED(CONFIG_X86_64) && si[s].stack_arg_cnt)) {
 			si[s].priv_stack_mode = NO_PRIV_STACK;
 			continue;
 		}
@@ -22021,6 +22207,21 @@ static int flow_drain(struct bpf_verifier_env *env)
 	return err;
 }
 
+/*
+ * A subprog on a cycle of the call graph checks the stack at run time, see
+ * bpf_insert_recursion_checks(). When there is no room, or the time slice is over, it leaves
+ * at once with 0 in place of the result: it unwinds if it can, see bail_unwinds, otherwise
+ * it returns and callers have to be fine with that.
+ */
+static int check_recursion_guard(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
+
+	if (!sub->recursive || env->insn_idx != sub->start)
+		return 0;
+	return push_budget_exit(env);
+}
+
 /* Process the current state until it gets to the start of a basic block. */
 static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 {
@@ -22144,6 +22345,10 @@ static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 		if (err)
 			return err;
 
+		err = check_recursion_guard(env);
+		if (err)
+			return err;
+
 		err = do_check_insn(env, &do_print_state);
 		if (error_recoverable_with_nospec(err) && state->speculative) {
 			/* Prevent this speculative path from ever reaching the
@@ -24240,6 +24445,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			bpf_log(log, "Only fentry can attach to %s, which can unwind\n", tname);
 			return -EINVAL;
 		}
+		/* It can have tail calls, which are not allowed there, see check_call_depth() */
+		if (aux->func && aux->func[subprog]->aux->in_recursion && prog_extension) {
+			bpf_log(log, "Cannot replace %s, which is called in a recursion\n", tname);
+			return -EINVAL;
+		}
 		conservative = aux->func_info_aux[subprog].unreliable;
 		if (prog_extension) {
 			if (aux->func_info_aux[subprog].linkage != BTF_FUNC_GLOBAL ||
@@ -25672,10 +25882,12 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = sort_subprogs_topo(env);
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = sort_subprogs_topo(env);
+	/* what no JIT can run is reported first */
+	ret = mark_recursive_subprogs(env);
 	if (ret < 0)
 		goto skip_full_check;
 
@@ -25743,9 +25955,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		goto skip_full_check;
 
 	ret = do_check_all(env);
-	/* reject recursion through the callx edges found by the main pass */
-	if (ret == 0 && env->callx_edges)
-		ret = sort_subprogs_topo(env);
 	/* while the calls are what they are in the program */
 	if (env->log.level & BPF_LOG_STATS)
 		add_insns_of_callees(env);
-- 
2.55.0

Re: [PATCH bpf-next v3 22/45] bpf: Allow recursive subprogs

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4234 bytes --]

> bpf: Allow recursive subprogs

TL;DR:
- Possible kernel stack overflow: a tail-calling fentry/fexit prog attached to
  a recursive subprog isn't rejected at attach time
- Nit: sort_subprogs_topo() callx walk looks like dead code

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -3529,9 +3529,9 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
>   * the current subprog to discover callees (direct subprogs and sync
>   * callbacks).
>   *
> - * The callees of callx are not known before the main verification pass.
> - * When callx is used the sort is repeated after it with the recorded callx
> - * edges added to the call graph to reject recursion through indirect calls.
> + * The callees of callx are not known before the main verification pass: callx
> + * is no edge here. The order within a cycle is arbitrary. For recursion see
> + * mark_recursive_subprogs() and check_recursion().
>   */
>  static int sort_subprogs_topo(struct bpf_verifier_env *env)

This isn't a bug, but the updated comment says "callx is no edge here", while
the body of sort_subprogs_topo() still has a callx walk:

	if (bpf_is_callx(&insn[idx])) {
		/* find a callee that is not explored yet */
		callee = -1;
		do {
			callee = next_callx_callee(env, cur, callee + 1);
		} while (callee >= 0 && color[callee] == 2);
		if (callee < 0)
			continue;

bpf_check() no longer calls sort_subprogs_topo() after do_check_all(), so
the only call left runs before the main pass.  env->callx_edges is only
allocated by record_callx_edge(), which is reached from check_func_callx()
and push_callback_call() during the main pass.

Doesn't that mean next_callx_callee() always returns -1 here, and this
branch always ends in continue?  Could it shrink to a plain continue, or be
dropped, since bpf_pseudo_call() never matches a callx and the final else
already does continue?

[ ... ]

> @@ -24240,6 +24445,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
>  			bpf_log(log, "Only fentry can attach to %s, which can unwind\n", tname);
>  			return -EINVAL;
>  		}
> +		/* It can have tail calls, which are not allowed there, see check_call_depth() */
> +		if (aux->func && aux->func[subprog]->aux->in_recursion && prog_extension) {
> +			bpf_log(log, "Cannot replace %s, which is called in a recursion\n", tname);
> +			return -EINVAL;
> +		}

This check rejects only extensions of a subprog that is called in a
recursion.  Can a tracing program that does tail calls still attach there?

An fentry program, or an fexit program when the target cannot unwind, can
attach to a subprog with in_recursion set.  bpf_tail_call is in
bpf_base_func_proto(), so such a program can use it.

The trampoline runs the tracing program between one recursive frame and the
next.  Any subprog with has_tail_call makes check_max_stack_depth() set
NO_PRIV_STACK, so that chain runs on the kernel stack and by the 256-byte
rule in check_call_depth() can take about 8K.

The run-time check in bpf_insert_recursion_checks() only keeps r10 of a
recursive subprog in the upper half of the stack.  Its comment says the
lower half is for "the frame of the subprog, then what it calls", which is
a chain that check_max_stack_depth() looked at together with that frame.  A
trampoline followed by a tracing tail-call chain is not such a chain.

With CAP_BPF and CAP_PERFMON on x86-64 with 16K stacks, a prog could
recurse until r10 of R_k is just above THREAD_SIZE / 2, with each call of R
going through the trampoline of an fentry program F that does tail calls.
The stack would then hold about 8K of recursion, R_k's frame (up to
stack_limit), the trampoline, and F's tail-call chain of about 8K or more,
which exceeds THREAD_SIZE.

Should the attach check also refuse tracing programs whose
aux->tail_call_reachable is set (or that have no private stack) when the
target subprog is in_recursion, the same way it refuses extensions?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

Re: [PATCH bpf-next v3 22/45] bpf: Allow recursive subprogs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] bpf: off-by-one frame count for recursion in compute_call_depth()
- [Medium] bpf: uncleared recursive marks on non-live subprogs in check_recursion()
- [Medium] bpf: async callback edge mismatch in mark_recursive_subprogs()
- [Medium] bpf: spurious budget exit on loop back-edge in check_recursion_guard()
--

commit 3e4e0bb05c4c0b3219c8960f2496445434c33bfc
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Allow recursive subprogs

This commit allows recursive static subprograms in BPF programs by identifying
cycles in the call graph prior to main verification, verifying runtime
recursion guard edges at subprogram entry, and adjusting liveness and stack
depth checks for recursive call cycles.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index b61166f670945..e29fc3ee363a9 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -5975,20 +5979,45 @@ static int check_exception_cb(struct bpf_verifier_env *env, struct bpf_call_grap
>  	return 0;
>  }
>  
> +static bool bpf_prog_reenters_datapath(const struct bpf_prog *prog);
> +
>  /*
> - * The DFS of sort_subprogs_topo() follows the address of a function like a call, and
> - * tolerates a cycle only when such address closes it, since it doesn't know what it is
> - * taken for. Such cycle counts as a recursion, unless the function is an async callback:
> - * its address is not an edge of the graph. The main verification pass limits the
> - * depth of the recursion, but it doesn't follow calls of global functions.
> + * Every subprog on a cycle has to check the stack at run time. mark_recursive_subprogs()
> + * didn't see the cycle coming when a helper got the callback from a table of
> + * functions, or from another subprog. A subprog that it marked for callx, which turned out
> + * to call something else, needs no check.
>   */
>  static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
>  {
>  	int j, s;
>  
>  	for_each_set_bit(s, cg->live, cg->cnt) {
> -		if (!test_bit(s, subprog_reach(cg, s)))
> +		if (!test_bit(s, subprog_reach(cg, s))) {
> +			env->subprog_info[s].recursive = false;
>  			continue;
> +		}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], check_recursion() only
iterates over cg->live when clearing env->subprog_info[s].recursive.

Before the main pass, mark_recursive_subprogs() scans all subprograms from 0
to env->subprog_cnt - 1, adds every subprogram in env->func_ptrs and
BPF_PSEUDO_FUNC instructions to taken, and connects every subprogram
containing callx to all subprograms in taken. If an uncalled subprogram s in a
function pointer table has callx or is part of a cycle in that
over-approximated graph, mark_recursive_subprogs() sets si[s].recursive =
true.

When the main pass never calls s via callx, record_callx_edge() does not
record an edge to s and build_call_graph() leaves s out of cg->live. Because
check_recursion() only walks cg->live, si[s].recursive remains true and skips
the bpf_enable_priv_stack(), bpf_prog_reenters_datapath(), and
bpf_prog_has_invocation_budget() checks.

Later in bpf_check(), bpf_insert_recursion_checks() runs before dead code
elimination and scans all subprograms:

kernel/bpf/fixups.c:bpf_insert_recursion_checks() {
	for (s = 1; s < env->subprog_cnt; s++)
		found |= si[s].recursive;
	if (!found)
		return 0;
	env->prog->jit_required = 1;

	for (s = 1; s < env->subprog_cnt; s++) {
		if (!si[s].recursive)
			continue;
		...
}

Doesn't this leave si[s].recursive set on unreachable subprograms, forcing
env->prog->jit_required = 1 and patching recursion checks into dead
subprograms even on architectures or program types that do not support
recursion?

[ ... ]
> @@ -6134,11 +6164,17 @@ static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_gra
>  		for_each_set_bit(s, cg->live, cg->cnt) {
>  			if (dinfo[s].depth < 0)
>  				continue;
> -			depth = dinfo[s].depth + subprog_stack_size(env, s);
> -			arena_depth = dinfo[s].arena_depth + si[s].arena_stack_depth;
>  			for_each_set_bit(j, subprog_calls(cg, s), cg->cnt) {
>  				callee = &dinfo[j];
> -				frame = bpf_subprog_is_global(env, j) ? 0 : dinfo[s].frame + 1;
> +				depth = dinfo[s].depth;
> +				arena_depth = dinfo[s].arena_depth;
> +				if (!si[s].recursive || !si[j].recursive) {
> +					depth += subprog_stack_size(env, s);
> +					arena_depth += si[s].arena_stack_depth;
> +				}
> +				frame = dinfo[s].frame + !si[j].recursive;
> +				if (bpf_subprog_is_global(env, j))
> +					frame = 0;

[Severity: Medium]
As reported by Sashiko on v2 [2], in compute_call_depth(), dinfo[0].frame
starts at 0 for the main program, and check_call_depth() checks
dinfo[s].frame >= MAX_CALL_FRAMES and dinfo[s].frame >= MAX_CALL_FRAMES_INTERP
directly without adding 1 for subprogram s itself.

When a non-recursive caller s (!si[s].recursive) calls a recursive callee j
(si[j].recursive), frame = dinfo[s].frame + !si[j].recursive adds 0 on entry
into the recursion, 0 on edges within the recursion, and 1 only for a
subsequent non-recursive callee k when exiting the recursion. This causes the
entire recursive component to count as 0 frames instead of 1 frame in
dinfo[].frame, allowing a static call chain containing recursive subprograms
to exceed MAX_CALL_FRAMES (32) or MAX_CALL_FRAMES_INTERP (16) without being
rejected by check_call_depth().

Should frame be incremented by (!si[s].recursive || !si[j].recursive) to match
the condition used for depth and arena_depth right above it, so that a
recursion counts as one frame in the static call chain?

[ ... ]
> @@ -6302,6 +6342,151 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
>  	return err;
>  }
>  
> +/*
> + * The address of the callback at @idx goes nowhere but to a helper that keeps it for
> + * later: a timer, a work queue, call_rcu.
> + */
> +static bool func_is_kept_for_later(struct bpf_verifier_env *env, int idx, int end)
> +{
> +	struct bpf_insn *insn = env->prog->insnsi;
> +	u8 class, reg = insn[idx].dst_reg;
> +	int i;
> +
> +	if (reg < BPF_REG_1 || reg > BPF_REG_5)
> +		return false;
> +	for (i = idx + 2; i < end; i++) {
> +		class = BPF_CLASS(insn[i].code);
> +		if (class == BPF_JMP || class == BPF_JMP32)
> +			return bpf_is_async_callback_calling_insn(&insn[i]);
> +		if (insn[i].dst_reg == reg)
> +			return false;
> +		if (bpf_is_ldimm64(&insn[i]))
> +			i++;
> +		else if (insn[i].src_reg == reg)
> +			return false;
> +	}
> +	return false;
> +}
> +
> +/*
> + * Find the subprogs on cycles of the call graph. It is not known yet what callx
> + * calls: it may be any subprog whose address is taken. A subprog that is
> + * marked for no reason is treated like a recursive one all the same.
> + */
> +static int mark_recursive_subprogs(struct bpf_verifier_env *env)
> +{
[ ... ]
> +				/*
> +				 * The helper that takes a callback calls it, or keeps it: the
> +				 * callback that arms itself again doesn't call itself.
> +				 */
> +				if (bpf_pseudo_call(&insn[i]) ||
> +				    !func_is_kept_for_later(env, i, si[s + 1].start))
> +					__set_bit(j, calls + s * words);

[Severity: Medium]
As reported by Sashiko on v2 [3], mark_recursive_subprogs() and
build_call_graph() use different criteria to decide whether a bpf_pseudo_func
instruction for an async callback forms a call edge:

1. In mark_recursive_subprogs(), func_is_kept_for_later() syntactically checks
whether bpf_pseudo_func loads into R1-R5 and reaches an async callback helper
or kfunc with no intervening jumps, calls, or register copies. If an async
callback registration in a live subprogram is on an untaken branch during
do_check_all(), func_is_kept_for_later() returns true (so si[j].recursive is
false), while push_callback_call() is never reached (so si[j].is_async_cb
remains false). When build_call_graph() later scans all instructions of
subprograms, it only skips bpf_pseudo_func when si[j].is_async_cb is set:

kernel/bpf/verifier.c:build_call_graph() {
	...
	if (si[j].is_async_cb && !bpf_pseudo_call(insn + i))
		continue;
	__set_bit(j, subprog_calls(cg, s));
}

If the callback re-arms itself, build_call_graph() adds the cycle to cg->live
and check_recursion() rejects the program because si[j].recursive was not set.

2. Conversely, if a live async callback that re-arms itself (directly or via a
helper subprog) loads its bpf_pseudo_func address into a callee-saved
register, copies the register, or has an intervening branch or call before the
async callback helper, func_is_kept_for_later() returns false and
mark_recursive_subprogs() marks the subprogs on the cycle as recursive = true
before the main pass. Even though si[j].is_async_cb becomes true during the
main pass, the premature recursive mark causes false rejections in
mark_recursive_subprogs() (for BPF_PROG_TYPE_EXT), in analyze_subprog() (when
passing stack pointers between the subprogs), and in check_recursion_guard()
(which injects a synthetic R0 = 0 early exit at subprog entry).

Can mark_recursive_subprogs() and build_call_graph() use a consistent analysis
for async callbacks so valid programs with timers, workqueues, or RCU
callbacks are not rejected?

[ ... ]
> @@ -22021,6 +22207,21 @@ static int flow_drain(struct bpf_verifier_env *env)
>  	return err;
>  }
>  
> +/*
> + * A subprog on a cycle of the call graph checks the stack at run time, see
> + * bpf_insert_recursion_checks(). When there is no room, or the time slice is over, it leaves
> + * at once with 0 in place of the result: it unwinds if it can, see bail_unwinds, otherwise
> + * it returns and callers have to be fine with that.
> + */
> +static int check_recursion_guard(struct bpf_verifier_env *env)
> +{
> +	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
> +
> +	if (!sub->recursive || env->insn_idx != sub->start)
> +		return 0;
> +	return push_budget_exit(env);
> +}

[Severity: Medium]
As reported by Sashiko on v2 [4], in bpf_insert_recursion_checks(), the
runtime recursion check is inserted before si[s].start via
bpf_patch_list_insert(), so only calls enter the check while intra-subprogram
jumps to si[s].start land after the check.

However, in do_check_block(), check_recursion_guard() calls push_budget_exit()
every time env->insn_idx == sub->start, without checking whether the state
just entered the subprogram or arrived via an intra-subprogram loop back-edge
to sub->start.

If a recursive subprogram has a loop header at sub->start and holds a
reference, spinlock, RCU lock, or IRQ state across the back-edge to sub->start
that is released later in the loop or after the loop exits, won't
check_recursion_guard() push a spurious exit branch with those resources still
held and cause verification to fail?

[1] https://lore.kernel.org/r/b77dd21709ea9d7691f0ffeb58635a660aef73738c24b1c059a01a1da42d5af4@mail.kernel.org
[2] https://lore.kernel.org/r/20261105204513.2230344-23-alexei.starovoitov@gmail.com
[3] https://lore.kernel.org/r/20261105204513.2230344-23-alexei.starovoitov@gmail.com
[4] https://lore.kernel.org/r/20261105204513.2230344-23-alexei.starovoitov@gmail.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=22

[PATCH bpf-next v3 23/45] bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

is_state_visited() is the pruning of a verifier that follows each path of
a prog. It saves states at selected insns and stops a path when a saved
state covers the current state. The main pass is in flow.c now. Each insn
that starts a basic block has a set of states, and a state that arrives is
dropped, joined or added. Nothing calls is_state_visited().

Remove it, and all that exists only for it: 1550 lines.

  removed                     task in the old pass     what does it now
  --------------------------  -----------------------  --------------------
  explored_states, free_list  Hold the saved states.   The sets of states
                                                       in flow.c.
  branches, dfs_depth,        Tell when all paths      No paths. The pass
  equal_state                 from a state are         ends when no set has
                              explored.                a new state.
  bpf_compute_scc(), SCC      Bring precision marks    The marks are not
  visits, backedges           over the back edges of   used. loop_header
                              a loop.                  tells the loops.
  states_maybe_looping(),     Reject a path that       The verifier adds
  iter.depth                  comes back in the same   may_goto to a loop
                              state.                   that it cannot
                                                       bound.
  widen_imprecise_scalars(),  Make a loop with an      A loop that is not
  callback_unroll_depth,      iterator, a callback or  counted is walked,
  may_goto_depth              may_goto converge.       or widened at its
                                                       head.
  bpf_split_cur_state()       Save the state that      flow.c keeps the
                              enters a counted loop.   states that enter.

What user space sees:
 - BPF_F_TEST_STATE_FREQ told is_state_visited() to save a state each time.
   The flag is UAPI and selftests set it. The kernel accepts it as before,
   and it has no effect.
 - The "Program dump" of log level 2 has no scc column.
 - The log does not print "depth=" for an iterator.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 123 +----
 kernel/bpf/cfg.c             | 198 -------
 kernel/bpf/log.c             |   5 +-
 kernel/bpf/states.c          | 998 -----------------------------------
 kernel/bpf/verifier.c        | 240 +--------
 5 files changed, 8 insertions(+), 1556 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 2a47184a5ff2..8392b056c7f3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -119,7 +119,6 @@ struct bpf_reg_state {
 			u32 btf_id;
 			/* packing following two fields to fit iter state into 16 bytes */
 			enum bpf_iter_state state:2;
-			int depth:30;
 			/*
 			 * The number of loops, in all frames, that the program was in
 			 * when it made the iterator, see iter_stops_loops().
@@ -561,52 +560,6 @@ struct bpf_verifier_state {
 	struct bpf_verifier_state *parent;
 	/* Acquired reference states */
 	struct bpf_reference_state *refs;
-	/*
-	 * 'branches' field is the number of branches left to explore:
-	 * 0 - all possible paths from this state reached bpf_exit or
-	 * were safely pruned
-	 * 1 - at least one path is being explored.
-	 * This state hasn't reached bpf_exit
-	 * 2 - at least two paths are being explored.
-	 * This state is an immediate parent of two children.
-	 * One is fallthrough branch with branches==1 and another
-	 * state is pushed into stack (to be explored later) also with
-	 * branches==1. The parent of this state has branches==1.
-	 * The verifier state tree connected via 'parent' pointer looks like:
-	 * 1
-	 * 1
-	 * 2 -> 1 (first 'if' pushed into stack)
-	 * 1
-	 * 2 -> 1 (second 'if' pushed into stack)
-	 * 1
-	 * 1
-	 * 1 bpf_exit.
-	 *
-	 * Once do_check() reaches bpf_exit, it calls update_branch_counts()
-	 * and the verifier state tree will look:
-	 * 1
-	 * 1
-	 * 2 -> 1 (first 'if' pushed into stack)
-	 * 1
-	 * 1 -> 1 (second 'if' pushed into stack)
-	 * 0
-	 * 0
-	 * 0 bpf_exit.
-	 * After pop_stack() the do_check() will resume at second 'if'.
-	 *
-	 * If is_state_visited() sees a state with branches > 0 it means
-	 * there is a loop. If such state is exactly equal to the current state
-	 * it's an infinite loop. Note states_equal() checks for states
-	 * equivalency, so two states being 'states_equal' does not mean
-	 * infinite loop. The exact comparison is provided by
-	 * states_maybe_looping() function. It's a stronger pre-check and
-	 * much faster than states_equal().
-	 *
-	 * This algorithm may not find all possible infinite loops or
-	 * loop iteration count may be too high.
-	 * In such cases BPF_COMPLEXITY_LIMIT_INSNS limit kicks in.
-	 */
-	u32 branches;
 	u32 insn_idx;
 	u32 curframe;
 
@@ -641,10 +594,6 @@ struct bpf_verifier_state {
 	 */
 	u32 first_insn_idx;
 	u32 last_insn_idx;
-	/* if this state is a backedge state then equal_state
-	 * records cached state to which this state is equal.
-	 */
-	struct bpf_verifier_state *equal_state;
 	/* jmp history recorded from first to last.
 	 * backtracking is using it to go from last to first.
 	 * For most states jmp_history_cnt is [0-3].
@@ -652,9 +601,6 @@ struct bpf_verifier_state {
 	 */
 	struct bpf_jmp_history_entry *jmp_history;
 	u32 jmp_history_cnt;
-	u32 dfs_depth;
-	u32 callback_unroll_depth;
-	u32 may_goto_depth;
 };
 
 /* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
@@ -754,15 +700,6 @@ static inline bool bpf_stack_args_are_incoming(const struct bpf_func_state *fram
 						1 << STACK_SPILL, __expr);	\
 	})
 
-/* linked list of verifier states used to prune search */
-struct bpf_verifier_state_list {
-	struct bpf_verifier_state state;
-	struct list_head node;
-	u32 miss_cnt;
-	u32 hit_cnt:31;
-	u32 in_free_list:1;
-};
-
 struct bpf_loop_inline_state {
 	unsigned int initialized:1; /* set to true upon first entry */
 	unsigned int fit_for_inline:1; /* true if callback function is the same
@@ -966,11 +903,6 @@ struct bpf_insn_aux_data {
 	u64 jump_target:1;
 
 	unsigned int orig_idx; /* original instruction index, initialized once */
-	/*
-	 * CFG strongly connected component this instruction belongs to,
-	 * zero if it is a singleton SCC.
-	 */
-	u32 scc;
 	/* registers alive before this instruction. */
 	u16 live_regs_before;
 	/*
@@ -1217,38 +1149,6 @@ struct bpf_idset {
 	} entries[BPF_ID_MAP_SIZE];
 };
 
-/* see verifier.c:compute_scc_callchain() */
-struct bpf_scc_callchain {
-	/* call sites from bpf_verifier_state->frame[*]->callsite leading to this SCC */
-	u32 callsites[MAX_CALL_FRAMES - 1];
-	/* last frame in a chain is identified by SCC id */
-	u32 scc;
-};
-
-/* verifier state waiting for propagate_backedges() */
-struct bpf_scc_backedge {
-	struct bpf_scc_backedge *next;
-	struct bpf_verifier_state state;
-};
-
-struct bpf_scc_visit {
-	struct bpf_scc_callchain callchain;
-	/* first state in current verification path that entered SCC
-	 * identified by the callchain
-	 */
-	struct bpf_verifier_state *entry_state;
-	struct bpf_scc_backedge *backedges; /* list of backedges */
-	u32 num_backedges;
-};
-
-/* An array of bpf_scc_visit structs sharing tht same bpf_scc_callchain->scc
- * but having different bpf_scc_callchain->callsites.
- */
-struct bpf_scc_info {
-	u32 num_visits;
-	struct bpf_scc_visit visits[];
-};
-
 struct bpf_liveness;
 struct scev;
 
@@ -1333,7 +1233,6 @@ struct bpf_verifier_env {
 	struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
 	int stack_size;			/* number of states to be processed */
 	bool strict_alignment;		/* perform strict pointer alignment checks */
-	bool test_state_freq;		/* test verifier with different pruning frequency */
 	bool test_reg_invariants;	/* fail verification on register invariants violations */
 	bool uses_arena_sp;		/* the program has insns with BPF_REG_ARENA_SP */
 	bool restart;			/* see bpf_flow_next_is_no_bound() */
@@ -1342,11 +1241,6 @@ struct bpf_verifier_env {
 	u32 *walk_insns;		/* the insns that were looked at in each of them */
 	u32 max_arena_stack_depth;	/* of all call chains */
 	struct bpf_verifier_state *cur_state; /* current verifier state */
-	/* Search pruning optimization, array of list_heads for
-	 * lists of struct bpf_verifier_state_list.
-	 */
-	struct list_head *explored_states;
-	struct list_head free_list;	/* list of struct bpf_verifier_state_list */
 	struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
 	struct btf_mod_pair used_btfs[MAX_USED_BTFS]; /* array of BTF's used by BPF program */
 	struct bpf_map *insn_array_maps[MAX_USED_MAPS]; /* array of INSN_ARRAY map's to be relocated */
@@ -1423,9 +1317,7 @@ struct bpf_verifier_env {
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
 	/* number of instructions analyzed by the verifier */
-	u32 prev_insn_processed, insn_processed;
-	/* number of jmps, calls, exits analyzed so far */
-	u32 prev_jmps_processed, jmps_processed;
+	u32 insn_processed;
 	/* maximum combined stack depth */
 	u32 max_stack_depth;
 	/* stack budget of the program, see bpf_prog_stack_limit() */
@@ -1443,9 +1335,6 @@ struct bpf_verifier_env {
 	u32 peak_states;
 	/* longest register parentage chain walked for liveness marking */
 	u32 longest_mark_read_walk;
-	u32 free_list_size;
-	u32 explored_states_size;
-	u32 num_backedges;
 	/*
 	 * The program's fd_array comes in two shapes, told apart by whether
 	 * the caller passed fd_array_cnt. They are mutually exclusive:
@@ -1478,11 +1367,7 @@ struct bpf_verifier_env {
 	char tmp_arg_name[32];
 	struct bpf_insn insn_buf[INSN_BUF_SIZE];
 	struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
-	struct bpf_scc_callchain callchain_buf;
 	struct bpf_liveness *liveness;
-	/* array of pointers to bpf_scc_info indexed by SCC id */
-	struct bpf_scc_info **scc_info;
-	u32 scc_cnt;
 	struct bpf_iarray *succ;
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
@@ -1678,7 +1563,6 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
 int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
 			struct bpf_verifier_state *cur);
@@ -1726,15 +1610,11 @@ int bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t, int
 int bpf_find_latches(struct bpf_verifier_env *env, bool flags);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
-int bpf_split_cur_state(struct bpf_verifier_env *env);
-int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 
 void bpf_clear_jmp_history(struct bpf_verifier_state *state);
 int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 			    const struct bpf_verifier_state *src);
-struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
 void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
-void bpf_free_backedges(struct bpf_scc_visit *visit);
 int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
 			 int insn_flags, int spi, int frame, const u16 *linked_regs,
 			 u8 linked_regs_cnt);
@@ -2092,7 +1972,6 @@ int bpf_prune_dead_branches(struct bpf_verifier_env *env);
 int bpf_check_cfg(struct bpf_verifier_env *env);
 void bpf_free_subprog_jts(struct bpf_verifier_env *env);
 int bpf_compute_postorder(struct bpf_verifier_env *env);
-int bpf_compute_scc(struct bpf_verifier_env *env);
 
 struct bpf_map_desc {
 	struct bpf_map *ptr;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 04c1c98ba248..94140d1fe1c3 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -868,201 +868,3 @@ int bpf_compute_postorder(struct bpf_verifier_env *env)
 	kvfree(state);
 	return 0;
 }
-
-/*
- * Compute strongly connected components (SCCs) on the CFG.
- * Assign an SCC number to each instruction, recorded in env->insn_aux[*].scc.
- * If instruction is a sole member of its SCC and there are no self edges,
- * assign it SCC number of zero.
- * Uses a non-recursive adaptation of Tarjan's algorithm for SCC computation.
- */
-int bpf_compute_scc(struct bpf_verifier_env *env)
-{
-	const u32 NOT_ON_STACK = U32_MAX;
-
-	struct bpf_insn_aux_data *aux = env->insn_aux_data;
-	const u32 insn_cnt = env->prog->len;
-	int stack_sz, dfs_sz, err = 0;
-	u32 *stack, *pre, *low, *dfs, *dfs_pos;
-	u32 i, j, t, w;
-	u32 next_preorder_num;
-	u32 next_scc_id;
-	bool assign_scc;
-	struct bpf_iarray *succ;
-
-	next_preorder_num = 1;
-	next_scc_id = 1;
-	/*
-	 * - 'stack' accumulates vertices in DFS order, see invariant comment below;
-	 * - 'pre[t] == p' => preorder number of vertex 't' is 'p';
-	 * - 'low[t] == n' => smallest preorder number of the vertex reachable from 't' is 'n';
-	 * - 'dfs' DFS traversal stack, used to emulate explicit recursion;
-	 * - 'dfs_pos[k] == j' => the frame 'dfs[k]' resumes visiting its
-	 *   successors at index 'j'.
-	 */
-	stack = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
-	pre = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
-	low = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
-	dfs = kvcalloc(insn_cnt, sizeof(*dfs), GFP_KERNEL_ACCOUNT);
-	dfs_pos = kvcalloc(insn_cnt, sizeof(*dfs_pos), GFP_KERNEL_ACCOUNT);
-	if (!stack || !pre || !low || !dfs || !dfs_pos) {
-		err = -ENOMEM;
-		goto exit;
-	}
-	/*
-	 * References:
-	 * [1] R. Tarjan "Depth-First Search and Linear Graph Algorithms"
-	 * [2] D. J. Pearce "A Space-Efficient Algorithm for Finding Strongly Connected Components"
-	 *
-	 * The algorithm maintains the following invariant:
-	 * - suppose there is a path 'u' ~> 'v', such that 'pre[v] < pre[u]';
-	 * - then, vertex 'u' remains on stack while vertex 'v' is on stack.
-	 *
-	 * Consequently:
-	 * - If 'low[v] < pre[v]', there is a path from 'v' to some vertex 'u',
-	 *   such that 'pre[u] == low[v]'; vertex 'u' is currently on the stack,
-	 *   and thus there is an SCC (loop) containing both 'u' and 'v'.
-	 * - If 'low[v] == pre[v]', loops containing 'v' have been explored,
-	 *   and 'v' can be considered the root of some SCC.
-	 *
-	 * Here is a pseudo-code for an explicitly recursive version of the algorithm:
-	 *
-	 *    NOT_ON_STACK = insn_cnt + 1
-	 *    pre = [0] * insn_cnt
-	 *    low = [0] * insn_cnt
-	 *    scc = [0] * insn_cnt
-	 *    stack = []
-	 *
-	 *    next_preorder_num = 1
-	 *    next_scc_id = 1
-	 *
-	 *    def recur(w):
-	 *        nonlocal next_preorder_num
-	 *        nonlocal next_scc_id
-	 *
-	 *        pre[w] = next_preorder_num
-	 *        low[w] = next_preorder_num
-	 *        next_preorder_num += 1
-	 *        stack.append(w)
-	 *        for s in successors(w):
-	 *            # Note: for classic algorithm the block below should look as:
-	 *            #
-	 *            # if pre[s] == 0:
-	 *            #     recur(s)
-	 *            #     low[w] = min(low[w], low[s])
-	 *            # elif low[s] != NOT_ON_STACK:
-	 *            #     low[w] = min(low[w], pre[s])
-	 *            #
-	 *            # But replacing both 'min' instructions with 'low[w] = min(low[w], low[s])'
-	 *            # does not break the invariant and makes iterative version of the algorithm
-	 *            # simpler. See 'Algorithm #3' from [2].
-	 *
-	 *            # 's' not yet visited
-	 *            if pre[s] == 0:
-	 *                recur(s)
-	 *            # if 's' is on stack, pick lowest reachable preorder number from it;
-	 *            # if 's' is not on stack 'low[s] == NOT_ON_STACK > low[w]',
-	 *            # so 'min' would be a noop.
-	 *            low[w] = min(low[w], low[s])
-	 *
-	 *        if low[w] == pre[w]:
-	 *            # 'w' is the root of an SCC, pop all vertices
-	 *            # below 'w' on stack and assign same SCC to them.
-	 *            while True:
-	 *                t = stack.pop()
-	 *                low[t] = NOT_ON_STACK
-	 *                scc[t] = next_scc_id
-	 *                if t == w:
-	 *                    break
-	 *            next_scc_id += 1
-	 *
-	 *    for i in range(0, insn_cnt):
-	 *        if pre[i] == 0:
-	 *            recur(i)
-	 *
-	 * Below implementation replaces explicit recursion with array 'dfs'.
-	 */
-	for (i = 0; i < insn_cnt; i++) {
-		if (pre[i])
-			continue;
-		stack_sz = 0;
-		dfs_sz = 1;
-		dfs[0] = i;
-		dfs_pos[0] = 0;
-dfs_continue:
-		while (dfs_sz) {
-			w = dfs[dfs_sz - 1];
-			if (pre[w] == 0) {
-				low[w] = next_preorder_num;
-				pre[w] = next_preorder_num;
-				next_preorder_num++;
-				stack[stack_sz++] = w;
-			}
-			/* Visit remaining 'w' successors */
-			succ = bpf_insn_successors(env, w);
-			for (j = dfs_pos[dfs_sz - 1]; j < succ->cnt; ++j) {
-				if (pre[succ->items[j]]) {
-					low[w] = min(low[w], low[succ->items[j]]);
-				} else {
-					/*
-					 * Once DFS for succ->items[j] is complete,
-					 * pre[succ->items[j]] is non-zero, hence
-					 * resuming at 'j' allows to follow the
-					 * low[w] = min(...) update branch above.
-					 */
-					dfs_pos[dfs_sz - 1] = j;
-					dfs_pos[dfs_sz] = 0;
-					dfs[dfs_sz] = succ->items[j];
-					dfs_sz++;
-					goto dfs_continue;
-				}
-			}
-			/*
-			 * Preserve the invariant: if some vertex above in the stack
-			 * is reachable from 'w', keep 'w' on the stack.
-			 */
-			if (low[w] < pre[w]) {
-				dfs_sz--;
-				goto dfs_continue;
-			}
-			/*
-			 * Assign SCC number only if component has two or more elements,
-			 * or if component has a self reference, or if instruction is a
-			 * callback calling function (implicit loop).
-			 */
-			assign_scc = stack[stack_sz - 1] != w;	/* two or more elements? */
-			for (j = 0; j < succ->cnt; ++j) {	/* self reference? */
-				if (succ->items[j] == w) {
-					assign_scc = true;
-					break;
-				}
-			}
-			if (bpf_calls_callback(env, w)) /* implicit loop? */
-				assign_scc = true;
-			/* Pop component elements from stack */
-			do {
-				t = stack[--stack_sz];
-				low[t] = NOT_ON_STACK;
-				if (assign_scc)
-					aux[t].scc = next_scc_id;
-			} while (t != w);
-			if (assign_scc)
-				next_scc_id++;
-			dfs_sz--;
-		}
-	}
-	env->scc_info = kvzalloc_objs(*env->scc_info, next_scc_id,
-				      GFP_KERNEL_ACCOUNT);
-	if (!env->scc_info) {
-		err = -ENOMEM;
-		goto exit;
-	}
-	env->scc_cnt = next_scc_id;
-exit:
-	kvfree(stack);
-	kvfree(pre);
-	kvfree(low);
-	kvfree(dfs);
-	kvfree(dfs_pos);
-	return err;
-}
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index e1c9b56cd13f..00cb219e24cd 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -779,11 +779,10 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 			if (!reg->id)
 				continue;
 
-			verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)",
+			verbose(env, " fp%d=iter_%s(id=%d,state=%s)",
 				(-i - 1) * BPF_REG_SIZE,
 				iter_type_str(reg->iter.btf, reg->iter.btf_id),
-				reg->id, iter_state_str(reg->iter.state),
-				reg->iter.depth);
+				reg->id, iter_state_str(reg->iter.state));
 			break;
 		case STACK_MISC:
 		case STACK_ZERO:
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 4b2506f3e3e2..343f29b52d79 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -7,297 +7,6 @@
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
-#define BPF_COMPLEXITY_LIMIT_STATES	64
-
-static bool is_may_goto_insn_at(struct bpf_verifier_env *env, int insn_idx)
-{
-	return bpf_is_may_goto_insn(&env->prog->insnsi[insn_idx]);
-}
-
-static bool is_iter_next_insn(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].is_iter_next;
-}
-
-static void update_peak_states(struct bpf_verifier_env *env)
-{
-	u32 cur_states;
-
-	cur_states = env->explored_states_size + env->free_list_size + env->num_backedges;
-	env->peak_states = max(env->peak_states, cur_states);
-}
-
-/* struct bpf_verifier_state->parent refers to states
- * that are in either of env->{expored_states,free_list}.
- * In both cases the state is contained in struct bpf_verifier_state_list.
- */
-static struct bpf_verifier_state_list *state_parent_as_list(struct bpf_verifier_state *st)
-{
-	if (st->parent)
-		return container_of(st->parent, struct bpf_verifier_state_list, state);
-	return NULL;
-}
-
-static bool incomplete_read_marks(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st);
-
-/* A state can be freed if it is no longer referenced:
- * - is in the env->free_list;
- * - has no children states;
- */
-static void maybe_free_verifier_state(struct bpf_verifier_env *env,
-				      struct bpf_verifier_state_list *sl)
-{
-	if (!sl->in_free_list
-	    || sl->state.branches != 0
-	    || incomplete_read_marks(env, &sl->state))
-		return;
-	list_del(&sl->node);
-	bpf_free_verifier_state(&sl->state, false);
-	kfree(sl);
-	env->free_list_size--;
-}
-
-/* For state @st look for a topmost frame with frame_insn_idx() in some SCC,
- * if such frame exists form a corresponding @callchain as an array of
- * call sites leading to this frame and SCC id.
- * E.g.:
- *
- *    void foo()  { A: loop {... SCC#1 ...}; }
- *    void bar()  { B: loop { C: foo(); ... SCC#2 ... }
- *                  D: loop { E: foo(); ... SCC#3 ... } }
- *    void main() { F: bar(); }
- *
- * @callchain at (A) would be either (F,SCC#2) or (F,SCC#3) depending
- * on @st frame call sites being (F,C,A) or (F,E,A).
- */
-static bool compute_scc_callchain(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st,
-				  struct bpf_scc_callchain *callchain)
-{
-	u32 i, scc, insn_idx;
-
-	memset(callchain, 0, sizeof(*callchain));
-	for (i = 0; i <= st->curframe; i++) {
-		insn_idx = bpf_frame_insn_idx(st, i);
-		scc = env->insn_aux_data[insn_idx].scc;
-		if (scc) {
-			callchain->scc = scc;
-			break;
-		} else if (i < st->curframe) {
-			callchain->callsites[i] = insn_idx;
-		} else {
-			return false;
-		}
-	}
-	return true;
-}
-
-/* Check if bpf_scc_visit instance for @callchain exists. */
-static struct bpf_scc_visit *scc_visit_lookup(struct bpf_verifier_env *env,
-					      struct bpf_scc_callchain *callchain)
-{
-	struct bpf_scc_info *info = env->scc_info[callchain->scc];
-	struct bpf_scc_visit *visits = info->visits;
-	u32 i;
-
-	if (!info)
-		return NULL;
-	for (i = 0; i < info->num_visits; i++)
-		if (memcmp(callchain, &visits[i].callchain, sizeof(*callchain)) == 0)
-			return &visits[i];
-	return NULL;
-}
-
-/* Allocate a new bpf_scc_visit instance corresponding to @callchain.
- * Allocated instances are alive for a duration of the do_check_common()
- * call and are freed by free_states().
- */
-static struct bpf_scc_visit *scc_visit_alloc(struct bpf_verifier_env *env,
-					     struct bpf_scc_callchain *callchain)
-{
-	struct bpf_scc_visit *visit;
-	struct bpf_scc_info *info;
-	u32 scc, num_visits;
-	u64 new_sz;
-
-	scc = callchain->scc;
-	info = env->scc_info[scc];
-	num_visits = info ? info->num_visits : 0;
-	new_sz = sizeof(*info) + sizeof(struct bpf_scc_visit) * (num_visits + 1);
-	info = kvrealloc(env->scc_info[scc], new_sz, GFP_KERNEL_ACCOUNT);
-	if (!info)
-		return NULL;
-	env->scc_info[scc] = info;
-	info->num_visits = num_visits + 1;
-	visit = &info->visits[num_visits];
-	memset(visit, 0, sizeof(*visit));
-	memcpy(&visit->callchain, callchain, sizeof(*callchain));
-	return visit;
-}
-
-/* Form a string '(callsite#1,callsite#2,...,scc)' in env->tmp_str_buf */
-static char *format_callchain(struct bpf_verifier_env *env, struct bpf_scc_callchain *callchain)
-{
-	char *buf = env->tmp_str_buf;
-	int i, delta = 0;
-
-	delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "(");
-	for (i = 0; i < ARRAY_SIZE(callchain->callsites); i++) {
-		if (!callchain->callsites[i])
-			break;
-		delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u,",
-				  callchain->callsites[i]);
-	}
-	delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u)", callchain->scc);
-	return env->tmp_str_buf;
-}
-
-/* If callchain for @st exists (@st is in some SCC), ensure that
- * bpf_scc_visit instance for this callchain exists.
- * If instance does not exist or is empty, assign visit->entry_state to @st.
- */
-static int maybe_enter_scc(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain))
-		return 0;
-	visit = scc_visit_lookup(env, callchain);
-	visit = visit ?: scc_visit_alloc(env, callchain);
-	if (!visit)
-		return -ENOMEM;
-	if (!visit->entry_state) {
-		visit->entry_state = st;
-		if (env->log.level & BPF_LOG_LEVEL2)
-			verbose(env, "SCC enter %s\n", format_callchain(env, callchain));
-	}
-	return 0;
-}
-
-static int propagate_backedges(struct bpf_verifier_env *env, struct bpf_scc_visit *visit);
-
-/* If callchain for @st exists (@st is in some SCC), make it empty:
- * - set visit->entry_state to NULL;
- * - flush accumulated backedges.
- */
-static int maybe_exit_scc(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain))
-		return 0;
-	visit = scc_visit_lookup(env, callchain);
-	if (!visit) {
-		/*
-		 * If path traversal stops inside an SCC, corresponding bpf_scc_visit
-		 * must exist for non-speculative paths. For non-speculative paths
-		 * traversal stops when:
-		 * a. Verification error is found, maybe_exit_scc() is not called.
-		 * b. Top level BPF_EXIT is reached. Top level BPF_EXIT is not a member
-		 *    of any SCC.
-		 * c. A checkpoint is reached and matched. Checkpoints are created by
-		 *    is_state_visited(), which calls maybe_enter_scc(), which allocates
-		 *    bpf_scc_visit instances for checkpoints within SCCs.
-		 * (c) is the only case that can reach this point.
-		 */
-		if (!st->speculative) {
-			verifier_bug(env, "scc exit: no visit info for call chain %s",
-				     format_callchain(env, callchain));
-			return -EFAULT;
-		}
-		return 0;
-	}
-	if (visit->entry_state != st)
-		return 0;
-	if (env->log.level & BPF_LOG_LEVEL2)
-		verbose(env, "SCC exit %s\n", format_callchain(env, callchain));
-	visit->entry_state = NULL;
-	env->num_backedges -= visit->num_backedges;
-	visit->num_backedges = 0;
-	update_peak_states(env);
-	return propagate_backedges(env, visit);
-}
-
-/* Lookup an bpf_scc_visit instance corresponding to @st callchain
- * and add @backedge to visit->backedges. @st callchain must exist.
- */
-static int add_scc_backedge(struct bpf_verifier_env *env,
-			    struct bpf_verifier_state *st,
-			    struct bpf_scc_backedge *backedge)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain)) {
-		verifier_bug(env, "add backedge: no SCC in verification path, insn_idx %d",
-			     st->insn_idx);
-		return -EFAULT;
-	}
-	visit = scc_visit_lookup(env, callchain);
-	if (!visit) {
-		verifier_bug(env, "add backedge: no visit info for call chain %s",
-			     format_callchain(env, callchain));
-		return -EFAULT;
-	}
-	if (env->log.level & BPF_LOG_LEVEL2)
-		verbose(env, "SCC backedge %s\n", format_callchain(env, callchain));
-	backedge->next = visit->backedges;
-	visit->backedges = backedge;
-	visit->num_backedges++;
-	env->num_backedges++;
-	update_peak_states(env);
-	return 0;
-}
-
-/* bpf_reg_state->live marks for registers in a state @st are incomplete,
- * if state @st is in some SCC and not all execution paths starting at this
- * SCC are fully explored.
- */
-static bool incomplete_read_marks(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain))
-		return false;
-	visit = scc_visit_lookup(env, callchain);
-	if (!visit)
-		return false;
-	return !!visit->backedges;
-}
-
-int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_verifier_state_list *sl = NULL, *parent_sl;
-	struct bpf_verifier_state *parent;
-	int err;
-
-	while (st) {
-		u32 br = --st->branches;
-
-		/* verifier_bug_if(br > 1, ...) technically makes sense here,
-		 * but see comment in push_stack(), hence:
-		 */
-		verifier_bug_if((int)br < 0, env, "%s:branches_to_explore=%d", __func__, br);
-		if (br)
-			break;
-		err = maybe_exit_scc(env, st);
-		if (err)
-			return err;
-		parent = st->parent;
-		parent_sl = state_parent_as_list(st);
-		if (sl)
-			maybe_free_verifier_state(env, sl);
-		st = parent;
-		sl = parent_sl;
-	}
-	return 0;
-}
-
 /* check %cur's range satisfies %old's */
 static bool range_within(const struct bpf_reg_state *old,
 			 const struct bpf_reg_state *cur)
@@ -830,18 +539,11 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		case STACK_ITER:
 			old_reg = &old_slot->spilled_ptr;
 			cur_reg = &cur_slot->spilled_ptr;
-			/* iter.depth is not compared between states as it
-			 * doesn't matter for correctness and would otherwise
-			 * prevent convergence; we maintain it only to prevent
-			 * infinite loop check triggering, see
-			 * iter_active_depths_differ()
-			 */
 			if (old_reg->type != cur_reg->type ||
 			    old_reg->iter.btf != cur_reg->iter.btf ||
 			    old_reg->iter.btf_id != cur_reg->iter.btf_id ||
 			    old_reg->iter.state != cur_reg->iter.state ||
 			    old_reg->iter.loops != cur_reg->iter.loops ||
-			    /* ignore {old_reg,cur_reg}->iter.depth, see above */
 			    !check_ids(old_reg->id, cur_reg->id, idmap))
 				return false;
 			break;
@@ -1110,706 +812,6 @@ static bool states_equal(struct bpf_verifier_env *env,
 	return true;
 }
 
-/* find precise scalars in the previous equivalent state and
- * propagate them into the current state
- */
-static int propagate_precision(struct bpf_verifier_env *env,
-			       const struct bpf_verifier_state *old,
-			       struct bpf_verifier_state *cur,
-			       bool *changed)
-{
-	struct bpf_reg_state *state_reg;
-	struct bpf_func_state *state;
-	int i, err = 0, fr;
-	bool first;
-
-	for (fr = old->curframe; fr >= 0; fr--) {
-		state = old->frame[fr];
-		state_reg = state->regs;
-		first = true;
-		for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
-			if (state_reg->type != SCALAR_VALUE ||
-			    !state_reg->precise)
-				continue;
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				if (first)
-					verbose(env, "frame %d: propagating r%d", fr, i);
-				else
-					verbose(env, ",r%d", i);
-			}
-			bpf_bt_set_frame_reg(&env->bt, fr, i);
-			first = false;
-		}
-
-		for (i = 0; i < bpf_stack_nr_slots(state); i++) {
-			struct bpf_stack_state *ss = bpf_stack_slot(state, i);
-
-			if (!bpf_is_spilled_reg(ss))
-				continue;
-			state_reg = &ss->spilled_ptr;
-			if (state_reg->type != SCALAR_VALUE ||
-			    !state_reg->precise)
-				continue;
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				if (first)
-					verbose(env, "frame %d: propagating fp%d",
-						fr, (-i - 1) * BPF_REG_SIZE);
-				else
-					verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
-			}
-			bpf_bt_set_frame_slot(&env->bt, fr, i);
-			first = false;
-		}
-		for (i = 0; i < state->out_stack_arg_cnt; i++) {
-			state_reg = &state->stack_arg_regs[i];
-			if (state_reg->type != SCALAR_VALUE || !state_reg->precise)
-				continue;
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				if (first)
-					verbose(env, "frame %d: propagating r11%d",
-						fr, -(i + 1) * BPF_REG_SIZE);
-				else
-					verbose(env, ",r11%d", -(i + 1) * BPF_REG_SIZE);
-			}
-			bt_set_frame_stack_arg_slot(&env->bt, fr, i);
-			first = false;
-		}
-		if (!first && (env->log.level & BPF_LOG_LEVEL2))
-			verbose(env, "\n");
-	}
-
-	err = bpf_mark_chain_precision(env, cur, -1, changed);
-	if (err < 0)
-		return err;
-
-	return 0;
-}
-
-#define MAX_BACKEDGE_ITERS 64
-
-/* Propagate read and precision marks from visit->backedges[*].state->equal_state
- * to corresponding parent states of visit->backedges[*].state until fixed point is reached,
- * then free visit->backedges.
- * After execution of this function incomplete_read_marks() will return false
- * for all states corresponding to @visit->callchain.
- */
-static int propagate_backedges(struct bpf_verifier_env *env, struct bpf_scc_visit *visit)
-{
-	struct bpf_scc_backedge *backedge;
-	struct bpf_verifier_state *st;
-	bool changed;
-	int i, err;
-
-	i = 0;
-	do {
-		if (i++ > MAX_BACKEDGE_ITERS) {
-			if (env->log.level & BPF_LOG_LEVEL2)
-				verbose(env, "%s: too many iterations\n", __func__);
-			for (backedge = visit->backedges; backedge; backedge = backedge->next)
-				bpf_mark_all_scalars_precise(env, &backedge->state);
-			break;
-		}
-		changed = false;
-		for (backedge = visit->backedges; backedge; backedge = backedge->next) {
-			st = &backedge->state;
-			err = propagate_precision(env, st->equal_state, st, &changed);
-			if (err)
-				return err;
-		}
-	} while (changed);
-
-	bpf_free_backedges(visit);
-	return 0;
-}
-
-static bool states_maybe_looping(struct bpf_verifier_state *old,
-				 struct bpf_verifier_state *cur)
-{
-	struct bpf_func_state *fold, *fcur;
-	int i, fr = cur->curframe;
-
-	if (old->curframe != fr)
-		return false;
-
-	fold = old->frame[fr];
-	fcur = cur->frame[fr];
-	for (i = 0; i < MAX_BPF_REG; i++)
-		if (memcmp(&fold->regs[i], &fcur->regs[i],
-			   offsetofend(struct bpf_reg_state, step)))
-			return false;
-	return true;
-}
-
-/* is_state_visited() handles iter_next() (see process_iter_next_call() for
- * terminology) calls specially: as opposed to bounded BPF loops, it *expects*
- * states to match, which otherwise would look like an infinite loop. So while
- * iter_next() calls are taken care of, we still need to be careful and
- * prevent erroneous and too eager declaration of "infinite loop", when
- * iterators are involved.
- *
- * Here's a situation in pseudo-BPF assembly form:
- *
- *   0: again:                          ; set up iter_next() call args
- *   1:   r1 = &it                      ; <CHECKPOINT HERE>
- *   2:   call bpf_iter_num_next        ; this is iter_next() call
- *   3:   if r0 == 0 goto done
- *   4:   ... something useful here ...
- *   5:   goto again                    ; another iteration
- *   6: done:
- *   7:   r1 = &it
- *   8:   call bpf_iter_num_destroy     ; clean up iter state
- *   9:   exit
- *
- * This is a typical loop. Let's assume that we have a prune point at 1:,
- * before we get to `call bpf_iter_num_next` (e.g., because of that `goto
- * again`, assuming other heuristics don't get in a way).
- *
- * When we first time come to 1:, let's say we have some state X. We proceed
- * to 2:, fork states, enqueue ACTIVE, validate NULL case successfully, exit.
- * Now we come back to validate that forked ACTIVE state. We proceed through
- * 3-5, come to goto, jump to 1:. Let's assume our state didn't change, so we
- * are converging. But the problem is that we don't know that yet, as this
- * convergence has to happen at iter_next() call site only. So if nothing is
- * done, at 1: verifier will use bounded loop logic and declare infinite
- * looping (and would be *technically* correct, if not for iterator's
- * "eventual sticky NULL" contract, see process_iter_next_call()). But we
- * don't want that. So what we do in process_iter_next_call() when we go on
- * another ACTIVE iteration, we bump slot->iter.depth, to mark that it's
- * a different iteration. So when we suspect an infinite loop, we additionally
- * check if any of the *ACTIVE* iterator states depths differ. If yes, we
- * pretend we are not looping and wait for next iter_next() call.
- *
- * This only applies to ACTIVE state. In DRAINED state we don't expect to
- * loop, because that would actually mean infinite loop, as DRAINED state is
- * "sticky", and so we'll keep returning into the same instruction with the
- * same state (at least in one of possible code paths).
- *
- * This approach allows to keep infinite loop heuristic even in the face of
- * active iterator. E.g., C snippet below is and will be detected as
- * infinitely looping:
- *
- *   struct bpf_iter_num it;
- *   int *p, x;
- *
- *   bpf_iter_num_new(&it, 0, 10);
- *   while ((p = bpf_iter_num_next(&t))) {
- *       x = p;
- *       while (x--) {} // <<-- infinite loop here
- *   }
- *
- */
-static bool iter_active_depths_differ(struct bpf_verifier_state *old, struct bpf_verifier_state *cur)
-{
-	struct bpf_reg_state *slot, *cur_slot;
-	struct bpf_func_state *state;
-	int i, fr;
-
-	for (fr = old->curframe; fr >= 0; fr--) {
-		state = old->frame[fr];
-		for (i = 0; i < bpf_stack_nr_slots(state); i++) {
-			struct bpf_stack_state *ss = bpf_stack_slot(state, i);
-
-			if (ss->slot_type[0] != STACK_ITER)
-				continue;
-
-			slot = &ss->spilled_ptr;
-			if (slot->iter.state != BPF_ITER_STATE_ACTIVE)
-				continue;
-
-			cur_slot = &bpf_stack_slot(cur->frame[fr], i)->spilled_ptr;
-			if (cur_slot->iter.depth != slot->iter.depth)
-				return true;
-		}
-	}
-	return false;
-}
-
-static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *func;
-	struct bpf_reg_state *reg;
-	int i, j;
-
-	for (i = 0; i <= st->curframe; i++) {
-		func = st->frame[i];
-		for (j = 0; j < BPF_REG_FP; j++) {
-			reg = &func->regs[j];
-			if (reg->type != SCALAR_VALUE)
-				continue;
-			reg->precise = false;
-		}
-		for (j = 0; j < bpf_stack_nr_slots(func); j++) {
-			struct bpf_stack_state *ss = bpf_stack_slot(func, j);
-
-			if (!bpf_is_spilled_reg(ss))
-				continue;
-			reg = &ss->spilled_ptr;
-			if (reg->type != SCALAR_VALUE)
-				continue;
-			reg->precise = false;
-		}
-	}
-}
-
-int bpf_split_cur_state(struct bpf_verifier_env *env)
-{
-	struct bpf_verifier_state *cur = env->cur_state, *new;
-	struct bpf_verifier_state_list *new_sl;
-	struct list_head *head;
-	int insn_idx = cur->insn_idx;
-	int err;
-
-	head = bpf_explored_state(env, insn_idx);
-	new_sl = kzalloc_obj(struct bpf_verifier_state_list, GFP_KERNEL_ACCOUNT);
-	if (!new_sl)
-		return -ENOMEM;
-	env->total_states++;
-	env->explored_states_size++;
-	update_peak_states(env);
-	env->prev_jmps_processed = env->jmps_processed;
-	env->prev_insn_processed = env->insn_processed;
-
-	/* forget precise markings we inherited, see __mark_chain_precision */
-	if (env->bpf_capable)
-		mark_all_scalars_imprecise(env, cur);
-
-	bpf_clear_singular_ids(env, cur);
-
-	/* add new state to the head of linked list */
-	new = &new_sl->state;
-	err = bpf_copy_verifier_state(new, cur);
-	if (err) {
-		bpf_free_verifier_state(new, false);
-		kfree(new_sl);
-		return err;
-	}
-	new->insn_idx = insn_idx;
-	verifier_bug_if(new->branches != 1, env,
-			"%s:branches_to_explore=%d insn %d",
-			__func__, new->branches, insn_idx);
-	err = maybe_enter_scc(env, new);
-	if (err) {
-		bpf_free_verifier_state(new, false);
-		kfree(new_sl);
-		return err;
-	}
-
-	cur->parent = new;
-	cur->last_insn_idx = -1;
-	cur->first_insn_idx = insn_idx;
-	cur->dfs_depth = new->dfs_depth + 1;
-	bpf_clear_jmp_history(cur);
-	list_add(&new_sl->node, head);
-	return 0;
-}
-
-/* Force a checkpoint upon reaching a loop header for a terminating loop. */
-static bool need_loop_checkpoint(struct bpf_verifier_env *env, int insn_idx)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_func_state *frame = cur->frame[cur->curframe];
-	struct loop_stack_entry *top;
-
-	if (bpf_loop_at_index(env, insn_idx) != insn_idx || !frame->loop_stack_cnt)
-		return false;
-	top = &frame->loop_stack[frame->loop_stack_cnt - 1];
-	return top->loop_id == insn_idx && top->terminates;
-}
-
-/* Searches for an innermost loop entry starting from an innermost frame. */
-static struct loop_stack_entry *current_loop(struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *frame;
-	int fr;
-
-	for (fr = st->curframe; fr >= 0; fr--) {
-		frame = st->frame[fr];
-		if (frame->loop_stack_cnt)
-			return &frame->loop_stack[frame->loop_stack_cnt - 1];
-	}
-	return NULL;
-}
-
-/* True if any of the loops call chain is currently in terminates. */
-static bool in_a_terminating_loop(struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *frame;
-	int l, fr;
-
-	for (fr = st->curframe; fr >= 0; fr--) {
-		frame = st->frame[fr];
-		for (l = frame->loop_stack_cnt - 1; l >= 0; l--)
-			if (frame->loop_stack[l].terminates)
-				return true;
-	}
-	return false;
-}
-
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
-{
-	struct loop_stack_entry *old_loop, *cur_loop;
-	struct bpf_verifier_state_list *sl;
-	struct bpf_verifier_state *cur = env->cur_state;
-	bool force_new_state, add_new_state, loop, loop_checkpoint;
-	int n, err, states_cnt = 0;
-	struct list_head *pos, *tmp, *head;
-
-	loop_checkpoint = need_loop_checkpoint(env, insn_idx);
-	force_new_state = env->test_state_freq || bpf_is_force_checkpoint(env, insn_idx) ||
-			  loop_checkpoint ||
-			  /* Avoid accumulating infinitely long jmp history */
-			  cur->jmp_history_cnt > 40;
-
-	/* bpf progs typically have pruning point every 4 instructions
-	 * http://vger.kernel.org/bpfconf2019.html#session-1
-	 * Do not add new state for future pruning if the verifier hasn't seen
-	 * at least 2 jumps and at least 8 instructions.
-	 * This heuristics helps decrease 'total_states' and 'peak_states' metric.
-	 * In tests that amounts to up to 50% reduction into total verifier
-	 * memory consumption and 20% verifier time speedup.
-	 */
-	add_new_state = force_new_state;
-	if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
-	    env->insn_processed - env->prev_insn_processed >= 8)
-		add_new_state = true;
-
-	/* keep cleaning the current state as registers/stack become dead */
-	err = clean_verifier_state(env, cur);
-	if (err)
-		return err;
-
-	loop = false;
-	head = bpf_explored_state(env, insn_idx);
-	list_for_each_safe(pos, tmp, head) {
-		sl = container_of(pos, struct bpf_verifier_state_list, node);
-		states_cnt++;
-		if (sl->state.insn_idx != insn_idx)
-			continue;
-
-		if (sl->state.branches) {
-			struct bpf_func_state *old_top_frame = sl->state.frame[0];
-
-			if (old_top_frame->in_async_callback_fn &&
-			    old_top_frame->async_entry_cnt != cur->frame[0]->async_entry_cnt) {
-				/* Different async_entry_cnt means that the verifier is
-				 * processing another entry into async callback.
-				 * Seeing the same state is not an indication of infinite
-				 * loop or infinite recursion.
-				 * But finding the same state doesn't mean that it's safe
-				 * to stop processing the current state. The previous state
-				 * hasn't yet reached bpf_exit, since state.branches > 0.
-				 * Checking in_async_callback_fn alone is not enough either.
-				 * Since the verifier still needs to catch infinite loops
-				 * inside async callbacks.
-				 */
-				goto skip_inf_loop_check;
-			}
-			/* BPF open-coded iterators loop detection is special.
-			 * states_maybe_looping() logic is too simplistic in detecting
-			 * states that *might* be equivalent, because it doesn't know
-			 * about ID remapping, so don't even perform it.
-			 * See process_iter_next_call() and iter_active_depths_differ()
-			 * for overview of the logic. When current and one of parent
-			 * states are detected as equivalent, it's a good thing: we prove
-			 * convergence and can stop simulating further iterations.
-			 * It's safe to assume that iterator loop will finish, taking into
-			 * account iter_next() contract of eventually returning
-			 * sticky NULL result.
-			 *
-			 * Note, that states have to be compared exactly in this case because
-			 * read and precision marks might not be finalized inside the loop.
-			 * E.g. as in the program below:
-			 *
-			 *     1. r7 = -16
-			 *     2. r6 = bpf_get_prandom_u32()
-			 *     3. while (bpf_iter_num_next(&fp[-8])) {
-			 *     4.   if (r6 != 42) {
-			 *     5.     r7 = -32
-			 *     6.     r6 = bpf_get_prandom_u32()
-			 *     7.     continue
-			 *     8.   }
-			 *     9.   r0 = r10
-			 *    10.   r0 += r7
-			 *    11.   r8 = *(u64 *)(r0 + 0)
-			 *    12.   r6 = bpf_get_prandom_u32()
-			 *    13. }
-			 *
-			 * Here verifier would first visit path 1-3, create a checkpoint at 3
-			 * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
-			 * not have read or precision mark for r7 yet, thus inexact states
-			 * comparison would discard current state with r7=-32
-			 * => unsafe memory access at 11 would not be caught.
-			 */
-			if (is_iter_next_insn(env, insn_idx)) {
-				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					struct bpf_func_state *cur_frame, *iter_frame, *old_frame;
-					struct bpf_reg_state *iter_state, *iter_reg;
-					struct bpf_reg_state *old_iter;
-					int spi;
-
-					cur_frame = cur->frame[cur->curframe];
-					/* btf_check_iter_kfuncs() enforces that
-					 * iter state pointer is always the first arg
-					 */
-					iter_reg = &cur_frame->regs[BPF_REG_1];
-					/* current state is valid due to states_equal(),
-					 * so we can assume valid iter and reg state,
-					 * no need for extra (re-)validations
-					 */
-					spi = bpf_get_spi(iter_reg->var_off.value);
-					iter_frame = bpf_func(env, iter_reg);
-					iter_state = &bpf_stack_slot(iter_frame, spi)->spilled_ptr;
-					old_frame = sl->state.frame[iter_reg->frameno];
-					old_iter = &bpf_stack_slot(old_frame, spi)->spilled_ptr;
-					/*
-					 * states_equal() matches ids through idmap.
-					 * The loop converged only if it's the same
-					 * iterator. The one that was destroyed and
-					 * created again starts from the beginning.
-					 */
-					if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE &&
-					    iter_state->id == old_iter->id) {
-						loop = true;
-						goto hit;
-					}
-				}
-				goto skip_inf_loop_check;
-			}
-			if (is_may_goto_insn_at(env, insn_idx)) {
-				if (sl->state.may_goto_depth != cur->may_goto_depth &&
-				    states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					loop = true;
-					goto hit;
-				}
-			}
-			if (bpf_calls_callback(env, insn_idx)) {
-				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					loop = true;
-					goto hit;
-				}
-				goto skip_inf_loop_check;
-			}
-			/*
-			 * If old state belongs to a control flow loop that we know terminates,
-			 * it should be safe to prune current state. However, account for the
-			 * following situation:
-			 *
-			 *   for (;;) {
-			 *     for (i = 0; i < 10; i++)
-			 *       ...
-			 *   }
-			 *
-			 * Here a backedge to a non-terminating outer loop is dominated by an exit
-			 * from the inner loop. Pruning at a re-entry to the inner loop therefore
-			 * would hide the fact that outer loop is non-terminating.
-			 * Hence, only allow pruning states within the same entry state.
-			 */
-			old_loop = current_loop(&sl->state);
-			cur_loop = current_loop(cur);
-			if (old_loop && cur_loop && in_a_terminating_loop(cur)) {
-				if (old_loop->terminates && cur_loop->terminates &&
-				    old_loop->entry_state == cur_loop->entry_state &&
-				    states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					loop = true;
-					goto hit;
-				}
-				goto skip_inf_loop_check;
-			}
-
-			/* attempt to detect infinite loop to avoid unnecessary doomed work */
-			if (states_maybe_looping(&sl->state, cur) &&
-			    states_equal(env, &sl->state, cur, EXACT) &&
-			    !iter_active_depths_differ(&sl->state, cur) &&
-			    sl->state.may_goto_depth == cur->may_goto_depth &&
-			    sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
-				verbose_linfo(env, insn_idx, "; ");
-				verbose(env, "infinite loop detected at insn %d\n", insn_idx);
-				verbose(env, "cur state:");
-				print_verifier_state(env, cur, cur->curframe, true);
-				verbose(env, "old state:");
-				print_verifier_state(env, &sl->state, cur->curframe, true);
-				return -EINVAL;
-			}
-			/* if the verifier is processing a loop, avoid adding new state
-			 * too often, since different loop iterations have distinct
-			 * states and may not help future pruning.
-			 * This threshold shouldn't be too low to make sure that
-			 * a loop with large bound will be rejected quickly.
-			 * The most abusive loop will be:
-			 * r1 += 1
-			 * if r1 < 1000000 goto pc-2
-			 * 1M insn_procssed limit / 100 == 10k peak states.
-			 * This threshold shouldn't be too high either, since states
-			 * at the end of the loop are likely to be useful in pruning.
-			 */
-skip_inf_loop_check:
-			if (!force_new_state &&
-			    env->jmps_processed - env->prev_jmps_processed < 20 &&
-			    env->insn_processed - env->prev_insn_processed < 100)
-				add_new_state = false;
-			goto miss;
-		}
-		/* See comments for mark_all_regs_read_and_precise() */
-		loop = incomplete_read_marks(env, &sl->state);
-		if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
-hit:
-			sl->hit_cnt++;
-
-			/* if previous state reached the exit with precision and
-			 * current state is equivalent to it (except precision marks)
-			 * the precision needs to be propagated back in
-			 * the current state.
-			 */
-			err = 0;
-			if (bpf_is_jmp_point(env, env->insn_idx))
-				err = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);
-			err = err ? : propagate_precision(env, &sl->state, cur, NULL);
-			if (err)
-				return err;
-			/* When processing iterator based loops above propagate_liveness and
-			 * propagate_precision calls are not sufficient to transfer all relevant
-			 * read and precision marks. E.g. consider the following case:
-			 *
-			 *  .-> A --.  Assume the states are visited in the order A, B, C.
-			 *  |   |   |  Assume that state B reaches a state equivalent to state A.
-			 *  |   v   v  At this point, state C is not processed yet, so state A
-			 *  '-- B   C  has not received any read or precision marks from C.
-			 *             Thus, marks propagated from A to B are incomplete.
-			 *
-			 * The verifier mitigates this by performing the following steps:
-			 *
-			 * - Prior to the main verification pass, strongly connected components
-			 *   (SCCs) are computed over the program's control flow graph,
-			 *   intraprocedurally.
-			 *
-			 * - During the main verification pass, `maybe_enter_scc()` checks
-			 *   whether the current verifier state is entering an SCC. If so, an
-			 *   instance of a `bpf_scc_visit` object is created, and the state
-			 *   entering the SCC is recorded as the entry state.
-			 *
-			 * - This instance is associated not with the SCC itself, but with a
-			 *   `bpf_scc_callchain`: a tuple consisting of the call sites leading to
-			 *   the SCC and the SCC id. See `compute_scc_callchain()`.
-			 *
-			 * - When a verification path encounters a `states_equal(...,
-			 *   RANGE_WITHIN)` condition, there exists a call chain describing the
-			 *   current state and a corresponding `bpf_scc_visit` instance. A copy
-			 *   of the current state is created and added to
-			 *   `bpf_scc_visit->backedges`.
-			 *
-			 * - When a verification path terminates, `maybe_exit_scc()` is called
-			 *   from `bpf_update_branch_counts()`. For states with `branches == 0`, it
-			 *   checks whether the state is the entry state of any `bpf_scc_visit`
-			 *   instance. If it is, this indicates that all paths originating from
-			 *   this SCC visit have been explored. `propagate_backedges()` is then
-			 *   called, which propagates read and precision marks through the
-			 *   backedges until a fixed point is reached.
-			 *   (In the earlier example, this would propagate marks from A to B,
-			 *    from C to A, and then again from A to B.)
-			 *
-			 * A note on callchains
-			 * --------------------
-			 *
-			 * Consider the following example:
-			 *
-			 *     void foo() { loop { ... SCC#1 ... } }
-			 *     void main() {
-			 *       A: foo();
-			 *       B: ...
-			 *       C: foo();
-			 *     }
-			 *
-			 * Here, there are two distinct callchains leading to SCC#1:
-			 * - (A, SCC#1)
-			 * - (C, SCC#1)
-			 *
-			 * Each callchain identifies a separate `bpf_scc_visit` instance that
-			 * accumulates backedge states. The `propagate_{liveness,precision}()`
-			 * functions traverse the parent state of each backedge state, which
-			 * means these parent states must remain valid (i.e., not freed) while
-			 * the corresponding `bpf_scc_visit` instance exists.
-			 *
-			 * Associating `bpf_scc_visit` instances directly with SCCs instead of
-			 * callchains would break this invariant:
-			 * - States explored during `C: foo()` would contribute backedges to
-			 *   SCC#1, but SCC#1 would only be exited once the exploration of
-			 *   `A: foo()` completes.
-			 * - By that time, the states explored between `A: foo()` and `C: foo()`
-			 *   (i.e., `B: ...`) may have already been freed, causing the parent
-			 *   links for states from `C: foo()` to become invalid.
-			 */
-			if (loop) {
-				struct bpf_scc_backedge *backedge;
-
-				backedge = kzalloc_obj(*backedge,
-						       GFP_KERNEL_ACCOUNT);
-				if (!backedge)
-					return -ENOMEM;
-				err = bpf_copy_verifier_state(&backedge->state, cur);
-				backedge->state.equal_state = &sl->state;
-				backedge->state.insn_idx = insn_idx;
-				err = err ?: add_scc_backedge(env, &sl->state, backedge);
-				if (err) {
-					bpf_free_verifier_state(&backedge->state, false);
-					kfree(backedge);
-					return err;
-				}
-			}
-			return 1;
-		}
-miss:
-		/* when new state is not going to be added do not increase miss count.
-		 * Otherwise several loop iterations will remove the state
-		 * recorded earlier. The goal of these heuristics is to have
-		 * states from some iterations of the loop (some in the beginning
-		 * and some at the end) to help pruning.
-		 */
-		if (add_new_state)
-			sl->miss_cnt++;
-		/* heuristic to determine whether this state is beneficial
-		 * to keep checking from state equivalence point of view.
-		 * Higher numbers increase max_states_per_insn and verification time,
-		 * but do not meaningfully decrease insn_processed.
-		 * 'n' controls how many times state could miss before eviction.
-		 * Use bigger 'n' for checkpoints because evicting checkpoint states
-		 * too early would hinder iterator convergence.
-		 */
-		n = (bpf_is_force_checkpoint(env, insn_idx) || loop_checkpoint) &&
-		    sl->state.branches > 0 ? 64 : 3;
-		if (sl->miss_cnt > sl->hit_cnt * n + n) {
-			/* the state is unlikely to be useful. Remove it to
-			 * speed up verification
-			 */
-			sl->in_free_list = true;
-			list_del(&sl->node);
-			list_add(&sl->node, &env->free_list);
-			env->free_list_size++;
-			env->explored_states_size--;
-			maybe_free_verifier_state(env, sl);
-		}
-	}
-
-	if (env->max_states_per_insn < states_cnt)
-		env->max_states_per_insn = states_cnt;
-
-	if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
-		return 0;
-
-	if (!add_new_state)
-		return 0;
-
-	/* There were no equivalent states, remember the current one.
-	 * Technically the current state is not proven to be safe yet,
-	 * but it will either reach outer most bpf_exit (which means it's safe)
-	 * or it will be rejected. When there are no loops the verifier won't be
-	 * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
-	 * again on the way to bpf_exit.
-	 * When looping the sl->state.branches will be > 0 and this state
-	 * will not be considered for equivalence until branches == 0.
-	 */
-	return bpf_split_cur_state(env);
-}
-
 int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
 {
 	return clean_verifier_state(env, st);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e29fc3ee363a..e3be9f5e6787 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -202,8 +202,6 @@ struct bpf_verifier_stack_elem {
 	u64 diag_log_pos;
 };
 
-#define BPF_COMPLEXITY_LIMIT_STATES	64
-
 #define BPF_GLOBAL_PERCPU_MA_MAX_SIZE  512
 
 #define BPF_PRIV_STACK_MIN_SIZE		64
@@ -1082,7 +1080,6 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 		st->iter.btf_id = btf_id;
 		st->iter.state = BPF_ITER_STATE_ACTIVE;
 		st->iter.loops = nr_loops_below(env->cur_state, env->cur_state->curframe + 1);
-		st->iter.depth = 0;
 
 		for (j = 0; j < BPF_REG_SIZE; j++)
 			slot->slot_type[j] = STACK_ITER;
@@ -1826,14 +1823,9 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	dst_state->flow_entry = src->flow_entry;
 	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
-	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
 	dst_state->first_insn_idx = src->first_insn_idx;
 	dst_state->last_insn_idx = src->last_insn_idx;
-	dst_state->dfs_depth = src->dfs_depth;
-	dst_state->callback_unroll_depth = src->callback_unroll_depth;
-	dst_state->may_goto_depth = src->may_goto_depth;
-	dst_state->equal_state = src->equal_state;
 	for (i = 0; i <= src->curframe; i++) {
 		dst = dst_state->frame[i];
 		if (!dst) {
@@ -1849,45 +1841,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	return 0;
 }
 
-static u32 state_htab_size(struct bpf_verifier_env *env)
-{
-	return env->prog->len;
-}
-
-struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_func_state *state = cur->frame[cur->curframe];
-
-	return &env->explored_states[(idx ^ state->callsite) % state_htab_size(env)];
-}
-
-static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_state *b)
-{
-	int fr;
-
-	if (a->curframe != b->curframe)
-		return false;
-
-	for (fr = a->curframe; fr >= 0; fr--)
-		if (a->frame[fr]->callsite != b->frame[fr]->callsite)
-			return false;
-
-	return true;
-}
-
-void bpf_free_backedges(struct bpf_scc_visit *visit)
-{
-	struct bpf_scc_backedge *backedge, *next;
-
-	for (backedge = visit->backedges; backedge; backedge = next) {
-		bpf_free_verifier_state(&backedge->state, false);
-		next = backedge->next;
-		kfree(backedge);
-	}
-	visit->backedges = NULL;
-}
-
 static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
 		     int *insn_idx, bool pop_log)
 {
@@ -1953,18 +1906,6 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	if (err)
 		return ERR_PTR(-ENOMEM);
 	elem->st.speculative |= speculative;
-	if (elem->st.parent) {
-		++elem->st.parent->branches;
-		/* WARN_ON(branches > 2) technically makes sense here,
-		 * but
-		 * 1. speculative states will bump 'branches' for non-branch
-		 * instructions
-		 * 2. is_state_visited() heuristics may decide not to create
-		 * a new state for a sequence of branches and all such current
-		 * and cloned states will be pointing to a single parent state
-		 * which might have large 'branches' count.
-		 */
-	}
 	return &elem->st;
 }
 
@@ -2636,7 +2577,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	 * This is async callback. It starts in a fresh stack.
 	 * Initialize it similar to do_check_common().
 	 */
-	elem->st.branches = 1;
 	elem->st.in_sleepable = is_sleepable;
 	frame = kzalloc_obj(*frame, GFP_KERNEL_ACCOUNT);
 	if (!frame)
@@ -9217,85 +9157,6 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-/* Look for a previous loop entry at insn_idx: nearest parent state
- * stopped at insn_idx with callsites matching those in cur->frame.
- */
-static struct bpf_verifier_state *find_prev_entry(struct bpf_verifier_env *env,
-						  struct bpf_verifier_state *cur,
-						  int insn_idx)
-{
-	struct bpf_verifier_state_list *sl;
-	struct bpf_verifier_state *st;
-	struct list_head *pos, *head;
-
-	/* Explored states are pushed in stack order, most recent states come first */
-	head = bpf_explored_state(env, insn_idx);
-	list_for_each(pos, head) {
-		sl = container_of(pos, struct bpf_verifier_state_list, node);
-		/* If st->branches != 0 state is a part of current DFS verification path,
-		 * hence cur & st for a loop.
-		 */
-		st = &sl->state;
-		if (st->insn_idx == insn_idx && st->branches && same_callsites(st, cur) &&
-		    st->dfs_depth < cur->dfs_depth)
-			return st;
-	}
-
-	return NULL;
-}
-
-/*
- * Check if scalar registers are exact for the purpose of not widening.
- * More lenient than regs_exact()
- */
-static bool scalars_exact_for_widen(const struct bpf_reg_state *rold,
-				    const struct bpf_reg_state *rcur)
-{
-	return !memcmp(rold, rcur, offsetof(struct bpf_reg_state, id));
-}
-
-static void maybe_widen_reg(struct bpf_verifier_env *env,
-			    struct bpf_reg_state *rold, struct bpf_reg_state *rcur)
-{
-	if (rold->type != SCALAR_VALUE)
-		return;
-	if (rold->type != rcur->type)
-		return;
-	if (rold->precise || rcur->precise || scalars_exact_for_widen(rold, rcur))
-		return;
-	__mark_reg_unknown(env, rcur);
-}
-
-static int widen_imprecise_scalars(struct bpf_verifier_env *env,
-				   struct bpf_verifier_state *old,
-				   struct bpf_verifier_state *cur)
-{
-	struct bpf_func_state *fold, *fcur;
-	int i, fr, num_slots;
-
-	for (fr = old->curframe; fr >= 0; fr--) {
-		fold = old->frame[fr];
-		fcur = cur->frame[fr];
-
-		for (i = 0; i < MAX_BPF_REG; i++)
-			maybe_widen_reg(env,
-					&fold->regs[i],
-					&fcur->regs[i]);
-
-		num_slots = min(bpf_stack_nr_slots(fold), bpf_stack_nr_slots(fcur));
-		for (i = 0; i < num_slots; i++) {
-			struct bpf_stack_state *old_ss = bpf_stack_slot(fold, i);
-			struct bpf_stack_state *cur_ss = bpf_stack_slot(fcur, i);
-
-			if (!bpf_is_spilled_reg(old_ss) || !bpf_is_spilled_reg(cur_ss))
-				continue;
-
-			maybe_widen_reg(env, &old_ss->spilled_ptr, &cur_ss->spilled_ptr);
-		}
-	}
-	return 0;
-}
-
 static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_st,
 						 struct bpf_call_arg_meta *meta)
 {
@@ -9443,13 +9304,11 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 
 		queued_iter = get_iter_from_state(queued_st, meta);
 		queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
-		queued_iter->iter.depth++;
 
 		queued_fr = queued_st->frame[queued_st->curframe];
 		mark_ptr_not_null_reg(&queued_fr->regs[BPF_REG_0]);
 	}
 
-	/* switch to DRAINED state, but keep the depth unchanged */
 	/* mark current iter state as drained and assume returned NULL */
 	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
 	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
@@ -12003,7 +11862,6 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	if (err)
 		return err;
 
-	callback_state->callback_unroll_depth++;
 	callback_state->frame[callback_state->curframe - 1]->callback_depth++;
 	caller->callback_depth = 0;
 	return 0;
@@ -12959,10 +12817,9 @@ static bool retval_range_within(struct bpf_retval_range range, const struct bpf_
 
 static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 {
-	struct bpf_verifier_state *state = env->cur_state, *prev_st;
+	struct bpf_verifier_state *state = env->cur_state;
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
-	bool in_callback_fn;
 	u32 i, nregs;
 	int err;
 
@@ -13051,11 +12908,9 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	invalidate_stack_slices(env, callee->frameno, INT_MIN, 0);
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
-	 * there function call logic would reschedule callback visit. If iteration
-	 * converges is_state_visited() would prune that visit eventually.
+	 * there function call logic would reschedule callback visit.
 	 */
-	in_callback_fn = callee->in_callback_fn;
-	if (in_callback_fn)
+	if (callee->in_callback_fn)
 		*insn_idx = callee->callsite;
 	else
 		*insn_idx = callee->callsite + 1;
@@ -13072,24 +12927,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	free_func_state(callee);
 	state->frame[state->curframe--] = NULL;
 	invalidate_outgoing_stack_args(env, caller);
-
-	/* for callbacks widen imprecise scalars to make programs like below verify:
-	 *
-	 *   struct ctx { int i; }
-	 *   void cb(int idx, struct ctx *ctx) { ctx->i++; ... }
-	 *   ...
-	 *   struct ctx = { .i = 0; }
-	 *   bpf_loop(100, cb, &ctx, 0);
-	 *
-	 * This is similar to what is done in process_iter_next_call() for open
-	 * coded iterators.
-	 */
-	prev_st = in_callback_fn ? find_prev_entry(env, state, *insn_idx) : NULL;
-	if (prev_st) {
-		err = widen_imprecise_scalars(env, prev_st, state);
-		if (err)
-			return err;
-	}
 	return 0;
 }
 
@@ -20083,19 +19920,14 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	}
 
 	if (opcode == BPF_JCOND) {
-		struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
+		struct bpf_verifier_state *queued_st;
 		int idx = *insn_idx;
 
-		prev_st = find_prev_entry(env, cur_st->parent, idx);
-
 		/* branch out 'fallthrough' insn as a new state to explore */
 		queued_st = push_stack(env, idx + 1, idx, false);
 		if (IS_ERR(queued_st))
 			return PTR_ERR(queued_st);
 
-		queued_st->may_goto_depth++;
-		if (prev_st)
-			widen_imprecise_scalars(env, prev_st, queued_st);
 		*insn_idx += insn->off;
 		return 0;
 	}
@@ -21563,7 +21395,6 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 	case BPF_JMP32: {
 		u8 opcode = BPF_OP(insn->code);
 
-		env->jmps_processed++;
 		if (opcode == BPF_CALL) {
 			if (env->cur_state->active_locks) {
 				/* similar to static subprog calls callx is allowed under a lock */
@@ -23503,46 +23334,10 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 
 static void free_states(struct bpf_verifier_env *env)
 {
-	struct bpf_verifier_state_list *sl;
-	struct list_head *head, *pos, *tmp;
-	struct bpf_scc_info *info;
-	int i, j;
-
 	bpf_free_verifier_state(env->cur_state, true);
 	env->cur_state = NULL;
 	while (!pop_stack(env, NULL, NULL, false));
 	bpf_flow_reset(env);
-
-	list_for_each_safe(pos, tmp, &env->free_list) {
-		sl = container_of(pos, struct bpf_verifier_state_list, node);
-		bpf_free_verifier_state(&sl->state, false);
-		kfree(sl);
-	}
-	INIT_LIST_HEAD(&env->free_list);
-
-	for (i = 0; i < env->scc_cnt; ++i) {
-		info = env->scc_info[i];
-		if (!info)
-			continue;
-		for (j = 0; j < info->num_visits; j++)
-			bpf_free_backedges(&info->visits[j]);
-		kvfree(info);
-		env->scc_info[i] = NULL;
-	}
-
-	if (!env->explored_states)
-		return;
-
-	for (i = 0; i < state_htab_size(env); i++) {
-		head = &env->explored_states[i];
-
-		list_for_each_safe(pos, tmp, head) {
-			sl = container_of(pos, struct bpf_verifier_state_list, node);
-			bpf_free_verifier_state(&sl->state, false);
-			kfree(sl);
-		}
-		INIT_LIST_HEAD(&env->explored_states[i]);
-	}
 }
 
 static int do_check_common(struct bpf_verifier_env *env, int subprog,
@@ -23565,7 +23360,6 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 		return -ENOMEM;
 	state->curframe = 0;
 	state->speculative = false;
-	state->branches = 1;
 	state->in_sleepable = context == BPF_FUNC_CTX_SLEEPABLE;
 	/* nobody knows what the callers of a global function or of an extension hold */
 	state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
@@ -25659,13 +25453,9 @@ static void log_program(struct bpf_verifier_env *env)
 	u64 pos, insn_pos;
 	u32 i, j;
 
-	verbose(env, "Program dump (scc? loop_header? idom insn#: live_regs_before):\n");
+	verbose(env, "Program dump (loop_header? idom insn#: live_regs_before):\n");
 	for (i = 0; i < insn_cnt; ++i) {
 		verbose_linfo(env, i, "    ; ");
-		if (env->insn_aux_data[i].scc)
-			verbose(env, "%3d ", env->insn_aux_data[i].scc);
-		else
-			verbose(env, "    ");
 		if (env->insn_aux_data[i].loop_header >= 0)
 			verbose(env, "%3d ", env->insn_aux_data[i].loop_header);
 		else
@@ -25788,21 +25578,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
 		env->strict_alignment = false;
 
-	if (is_priv)
-		env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
 	env->test_reg_invariants = attr->prog_flags & BPF_F_TEST_REG_INVARIANTS;
 
-	env->explored_states = kvzalloc_objs(struct list_head,
-					     state_htab_size(env),
-					     GFP_KERNEL_ACCOUNT);
-	ret = -ENOMEM;
-	if (!env->explored_states)
-		goto skip_full_check;
-
-	for (i = 0; i < state_htab_size(env); i++)
-		INIT_LIST_HEAD(&env->explored_states[i]);
-	INIT_LIST_HEAD(&env->free_list);
-
 	/* Prepare BTF and func_info needed to discover all subprograms. */
 	ret = bpf_prepare_btf_info(env, attr, uattr);
 	if (ret < 0)
@@ -25891,10 +25668,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = bpf_compute_scc(env);
-	if (ret < 0)
-		goto skip_full_check;
-
 	/* must precede the first bpf_ret_reg_pair() user below */
 	ret = bpf_compute_subprog_ret_regs(env);
 	if (ret < 0)
@@ -25967,8 +25740,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		ret = bpf_insert_loop_guards(env);
 
 skip_full_check:
-	kvfree(env->explored_states);
-
 	/* might decrease stack depth, keep it before passes that
 	 * allocate additional slots.
 	 */
@@ -26121,7 +25892,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.postorder_nums);
 	kvfree(env->cfg.insn_postorder);
-	kvfree(env->scc_info);
 	kvfree(env->succ);
 	kvfree(env->gotox_tmp_buf);
 	kvfree(env->callx_edges);
-- 
2.55.0

[PATCH bpf-next v3 24/45] bpf: Remove precision tracking

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Precision tracking is a part of pruning. The verifier followed each path
and stopped a path when a state that it saw before at the same insn covered
the current state. To stop more paths, the comparison ignored a scalar
unless a later check depended on its value. To find these scalars, the
verifier went back from each check through the history of the path and
marked them as precise.

The verifier does not prune paths any more. It joins the states that meet
at the start of a basic block, and the join covers both:

  r1=0 and r1=1 meet      pruning                      join
  ----------------------  ---------------------------  ---------------
  a check depends on r1   r1 is precise,               verify r1=[0,1]
                          verify the two states
  no check depends on r1  drop the second state        verify r1=[0,1]

Each later check sees the full range of r1. Thus the verifier does not have
to find out afterwards which scalars a check depends on, and nothing reads
the marks.

One task of the marks is left. Some checks need a constant, for example the
size for bpf_ringbuf_reserve(), and the join of two constants is a range.
flow_compute_need_const() finds the registers and stack slots that go into
such checks, one time, before the main pass. States that differ in them are
not joined.

Remove the 'precise' bit, backtrack.c, jmp_history, the parent link of
a state and the marks that scalar evolution asks for: 1650 lines. With the
bit gone, __mark_reg_unknown() and bpf_mark_reg_unknown_imprecise() are one
function, bpf_mark_reg_unknown(). __mark_reg_const_zero() does not need
env, and BPF_REGMASK_ARGS has no user.

What user space sees:
 - The log has no "mark_precise:" lines and no 'P' in "R0=Pscalar()".
 - Nothing sets longest_mark_read_walk. veristat and other tools parse the
   statistics line, thus it continues to show "mark_read 0".

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  120 +---
 kernel/bpf/Makefile          |    2 +-
 kernel/bpf/backtrack.c       | 1059 ----------------------------------
 kernel/bpf/log.c             |    2 -
 kernel/bpf/scev.c            |   44 +-
 kernel/bpf/states.c          |   11 +-
 kernel/bpf/verifier.c        |  475 ++-------------
 7 files changed, 64 insertions(+), 1649 deletions(-)
 delete mode 100644 kernel/bpf/backtrack.c

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 8392b056c7f3..e932c480feb2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -35,8 +35,8 @@ static inline u16 imod(s64 v, u16 step)
 #define BPF_MAX_VAR_SIZ	(1 << 29)
 /*
  * size of tmp_str_buf in bpf_verifier.
- * we need at least 1399 bytes to fit full stack mask representation
- * (in the "-8,-16,...,-2048" form)
+ * The longest text is the list of stack slots of fmt_spis_mask(),
+ * which cuts what does not fit.
  */
 #define TMP_STR_BUF_LEN 1408
 /* Patch buffer size */
@@ -201,9 +201,7 @@ struct bpf_reg_state {
 	 * other registers. Kept outside the metadata union for ID remapping
 	 * during state comparisons.
 	 */
-	u32 map_uid:31;
-	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
-	u32 precise:1;
+	u32 map_uid;
 };
 
 static inline s64 reg_smin(const struct bpf_reg_state *reg)
@@ -295,10 +293,6 @@ enum bpf_stack_slot_type {
 #define BPF_HALF_REG_SIZE	4
 #define STACK_SLOT_SZ		4
 
-#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
-			  (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
-			  (1 << BPF_REG_5))
-
 #define BPF_MAIN_FUNC (-1)
 
 #define BPF_DYNPTR_SIZE		sizeof(struct bpf_dynptr_kern)
@@ -351,7 +345,6 @@ struct bpf_retval_range {
 struct bpf_loop_iters {
 	u32 min_header_count;   /* min number of times header is executed */
 	u32 max_header_count;   /* max number of times header is executed */
-	unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* live, changed by the loop, and there is no summary of how */
 	unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* [0, max] at the entry, and the loop only clears bits */
@@ -511,44 +504,6 @@ struct bpf_func_state {
 /* What the interpreter has stack for. A longer chain of calls requires the JIT. */
 #define MAX_CALL_FRAMES_INTERP 16
 
-/* instruction history flags, used in bpf_jmp_history_entry.flags field.
- * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
- */
-enum {
-	INSN_F_STACK_ACCESS = BIT(0),
-
-	INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
-	INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
-
-	INSN_F_STACK_ARG_ACCESS = BIT(3),
-};
-
-/* Registers linked to one jump condition that a history entry can record */
-#define BPF_LINKED_REGS_MAX	5
-
-struct bpf_jmp_history_entry {
-	/* insn idx can't be bigger than 1 million */
-	u32 idx : 20;
-	u32 frame : 5;	/* stack access frame number */
-	/* special INSN_F_xxx flags */
-	u32 flags : 4;
-	u32 : 3;
-	u32 prev_idx : 20;
-	u32 spi : 12;	/* stack slot index */
-	/*
-	 * Scalar registers and spilled scalars linked to the condition of
-	 * this jump, which need precision tracking together when the jump is
-	 * backtracked. Each is packed as 5 bits of frame number, one bit
-	 * telling a register from a stack slot and 10 bits of register or
-	 * slot index, see linked_regs_pack().
-	 */
-	u16 linked_regs[BPF_LINKED_REGS_MAX];
-	u8 linked_regs_cnt;
-};
-
-static_assert(MAX_CALL_FRAMES <= (1 << 5));
-static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
-
 /* Maximum number of bpf_reg_state objects that can exist at once */
 #define MAX_STACK_ARG_SLOTS (MAX_BPF_STATIC_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
 #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
@@ -557,7 +512,6 @@ static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
 struct bpf_verifier_state {
 	/* call stack tracking */
 	struct bpf_func_state *frame[MAX_CALL_FRAMES];
-	struct bpf_verifier_state *parent;
 	/* Acquired reference states */
 	struct bpf_reference_state *refs;
 	u32 insn_idx;
@@ -588,19 +542,8 @@ struct bpf_verifier_state {
 	 */
 	u32 flow_entry;
 
-	/*
-	 * First and last insn idx of this verifier state.
-	 * last_insn_idx is -1 if no instructions have been executed yet.
-	 */
-	u32 first_insn_idx;
+	/* The insn that the state comes from, -1 if no instructions have been executed yet. */
 	u32 last_insn_idx;
-	/* jmp history recorded from first to last.
-	 * backtracking is using it to go from last to first.
-	 * For most states jmp_history_cnt is [0-3].
-	 * For loops can go up to ~40.
-	 */
-	struct bpf_jmp_history_entry *jmp_history;
-	u32 jmp_history_cnt;
 };
 
 /* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
@@ -941,8 +884,8 @@ struct bpf_insn_aux_data {
 #define MAX_USED_BTFS 64 /* max number of BTFs accessed by one BPF program */
 
 /*
- * Longest line the verifier log can carry: a full stack mask of
- * MAX_BPF_STACK_SLOTS slots, see TMP_STR_BUF_LEN, plus its prefix.
+ * Longest line the verifier log can carry: tmp_str_buf, see
+ * TMP_STR_BUF_LEN, plus its prefix.
  */
 #define BPF_VERIFIER_TMP_LOG_SIZE	2048
 
@@ -1105,14 +1048,6 @@ static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
 struct bpf_diag;
 struct bpf_verifier_env;
 
-struct backtrack_state {
-	struct bpf_verifier_env *env;
-	u32 frame;
-	u32 reg_masks[MAX_CALL_FRAMES];
-	unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
-	u32 stack_arg_masks[MAX_CALL_FRAMES];
-};
-
 struct bpf_id_pair {
 	u32 old;
 	u32 cur;
@@ -1310,8 +1245,6 @@ struct bpf_verifier_env {
 		u32 gotox_edges;
 		bool subprog_jts_ready;
 	} cfg;
-	struct backtrack_state bt;
-	struct bpf_jmp_history_entry *cur_hist_ent;
 	/* Per-callsite snapshot of the parent's tracked spill slots for cross-frame fills. */
 	struct spill_snapshot **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
@@ -1333,8 +1266,6 @@ struct bpf_verifier_env {
 	 * memory consumption during verification
 	 */
 	u32 peak_states;
-	/* longest register parentage chain walked for liveness marking */
-	u32 longest_mark_read_walk;
 	/*
 	 * The program's fd_array comes in two shapes, told apart by whether
 	 * the caller passed fd_array_cnt. They are mutually exclusive:
@@ -1561,7 +1492,6 @@ int bpf_check_ext_prologue(struct bpf_verifier_log *log,
 			   const struct bpf_prog *tgt_prog, u32 btf_id);
 void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 
-int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
@@ -1611,25 +1541,15 @@ int bpf_find_latches(struct bpf_verifier_env *env, bool flags);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
 
-void bpf_clear_jmp_history(struct bpf_verifier_state *state);
 int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 			    const struct bpf_verifier_state *src);
 void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
-			 int insn_flags, int spi, int frame, const u16 *linked_regs,
-			 u8 linked_regs_cnt);
-void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg);
 void bpf_mark_reg_known_scalar(struct bpf_reg_state *reg, u64 imm);
-void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg);
+void bpf_mark_reg_unknown(struct bpf_reg_state *reg);
 int bpf_grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size);
-void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st);
 void bpf_clear_singular_ids(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
-int bpf_mark_chain_precision(struct bpf_verifier_env *env,
-			     struct bpf_verifier_state *starting_state,
-			     int regno, bool *changed);
 
 static inline int bpf_get_spi(s32 off)
 {
@@ -1706,31 +1626,6 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
 	return reg->type == SCALAR_VALUE && tnum_equals_const(reg->var_off, 0);
 }
 
-static inline void bpf_bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
-{
-	bt->reg_masks[frame] |= 1 << reg;
-}
-
-static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	__set_bit(slot, bt->stack_masks[frame]);
-}
-
-static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	bt->stack_arg_masks[frame] |= 1 << slot;
-}
-
-static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
-{
-	return bt->reg_masks[frame] & (1 << reg);
-}
-
-static inline bool bt_is_frame_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	return test_bit(slot, bt->stack_masks[frame]);
-}
-
 bool bpf_map_is_rdonly(const struct bpf_map *map);
 struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
 				       const struct bpf_map *map, u32 *cnt);
@@ -1952,7 +1847,6 @@ static inline int bpf_head_guard_target(const struct bpf_verifier_env *env, int
 
 	return aux->guard_at_header ? aux->guard_to : -1;
 }
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 
 /* Kinds of member a by-value struct or union may be composed of. */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index ab3ba42596ec..d26c62326e58 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o flow.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
deleted file mode 100644
index a6e69c1f5ff6..000000000000
--- a/kernel/bpf/backtrack.c
+++ /dev/null
@@ -1,1059 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
-#include <linux/bpf.h>
-#include <linux/bpf_verifier.h>
-#include <linux/filter.h>
-#include <linux/bitmap.h>
-
-#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-
-/* for any branch, call, exit record the history of jmps in the given state */
-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
-			 int insn_flags, int spi, int frame, const u16 *linked_regs,
-			 u8 linked_regs_cnt)
-{
-	u32 cnt = cur->jmp_history_cnt;
-	struct bpf_jmp_history_entry *p;
-	size_t alloc_size;
-
-	/* FLOW: nothing looks at the history */
-	return 0;
-
-	/* combine instruction flags if we already recorded this instruction */
-	if (env->cur_hist_ent) {
-		/* atomic instructions push insn_flags twice, for READ and
-		 * WRITE sides, but they should agree on stack slot
-		 */
-		verifier_bug_if((env->cur_hist_ent->flags & insn_flags) &&
-				(env->cur_hist_ent->flags & insn_flags) != insn_flags,
-				env, "insn history: insn_idx %d cur flags %x new flags %x",
-				env->insn_idx, env->cur_hist_ent->flags, insn_flags);
-		env->cur_hist_ent->flags |= insn_flags;
-		env->cur_hist_ent->spi = spi;
-		env->cur_hist_ent->frame = frame;
-		verifier_bug_if(env->cur_hist_ent->linked_regs_cnt != 0, env,
-				"insn history: insn_idx %d has %u linked regs",
-				env->insn_idx, env->cur_hist_ent->linked_regs_cnt);
-		if (linked_regs_cnt)
-			memcpy(env->cur_hist_ent->linked_regs, linked_regs,
-			       linked_regs_cnt * sizeof(*linked_regs));
-		env->cur_hist_ent->linked_regs_cnt = linked_regs_cnt;
-		return 0;
-	}
-
-	cnt++;
-	alloc_size = kmalloc_size_roundup(size_mul(cnt, sizeof(*p)));
-	p = krealloc(cur->jmp_history, alloc_size, GFP_KERNEL_ACCOUNT);
-	if (!p)
-		return -ENOMEM;
-	cur->jmp_history = p;
-
-	p = &cur->jmp_history[cnt - 1];
-	p->idx = env->insn_idx;
-	p->prev_idx = env->prev_insn_idx;
-	p->flags = insn_flags;
-	p->spi = spi;
-	p->frame = frame;
-	if (linked_regs_cnt)
-		memcpy(p->linked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));
-	p->linked_regs_cnt = linked_regs_cnt;
-	cur->jmp_history_cnt = cnt;
-	env->cur_hist_ent = p;
-
-	return 0;
-}
-
-static bool is_atomic_load_insn(const struct bpf_insn *insn)
-{
-	return BPF_CLASS(insn->code) == BPF_STX &&
-	       BPF_MODE(insn->code) == BPF_ATOMIC &&
-	       insn->imm == BPF_LOAD_ACQ;
-}
-
-static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
-{
-	return BPF_CLASS(insn->code) == BPF_STX &&
-	       BPF_MODE(insn->code) == BPF_ATOMIC &&
-	       (insn->imm & BPF_FETCH);
-}
-
-/* Backtrack one insn at a time. If idx is not at the top of recorded
- * history then previous instruction came from straight line execution.
- * Return -ENOENT if we exhausted all instructions within given state.
- *
- * It's legal to have a bit of a looping with the same starting and ending
- * insn index within the same state, e.g.: 3->4->5->3, so just because current
- * instruction index is the same as state's first_idx doesn't mean we are
- * done. If there is still some jump history left, we should keep going. We
- * need to take into account that we might have a jump history between given
- * state's parent and itself, due to checkpointing. In this case, we'll have
- * history entry recording a jump from last instruction of parent state and
- * first instruction of given state.
- */
-static int get_prev_insn_idx(struct bpf_verifier_state *st, int i,
-			     u32 *history)
-{
-	u32 cnt = *history;
-
-	if (i == st->first_insn_idx) {
-		if (cnt == 0)
-			return -ENOENT;
-		if (cnt == 1 && st->jmp_history[0].idx == i)
-			return -ENOENT;
-	}
-
-	if (cnt && st->jmp_history[cnt - 1].idx == i) {
-		i = st->jmp_history[cnt - 1].prev_idx;
-		(*history)--;
-	} else {
-		i--;
-	}
-	return i;
-}
-
-static struct bpf_jmp_history_entry *get_jmp_hist_entry(struct bpf_verifier_state *st,
-						        u32 hist_end, int insn_idx)
-{
-	if (hist_end > 0 && st->jmp_history[hist_end - 1].idx == insn_idx)
-		return &st->jmp_history[hist_end - 1];
-	return NULL;
-}
-
-static inline void bt_init(struct backtrack_state *bt, u32 frame)
-{
-	bt->frame = frame;
-}
-
-static inline void bt_reset(struct backtrack_state *bt)
-{
-	struct bpf_verifier_env *env = bt->env;
-
-	memset(bt, 0, sizeof(*bt));
-	bt->env = env;
-}
-
-static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)
-{
-	return bitmap_empty(bt->stack_masks[frame], MAX_BPF_STACK_SLOTS);
-}
-
-static inline bool bt_stack_empty(struct backtrack_state *bt)
-{
-	return bt_frame_stack_empty(bt, bt->frame);
-}
-
-static inline bool bt_empty(struct backtrack_state *bt)
-{
-	u32 mask = 0;
-	int i;
-
-	for (i = 0; i <= bt->frame; i++) {
-		mask |= bt->reg_masks[i] | bt->stack_arg_masks[i];
-		if (!bt_frame_stack_empty(bt, i))
-			return false;
-	}
-
-	return mask == 0;
-}
-
-static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	bt->stack_arg_masks[frame] &= ~(1 << slot);
-}
-
-static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	return bt->stack_arg_masks[frame] & (1 << slot);
-}
-
-static inline int bt_subprog_enter(struct backtrack_state *bt)
-{
-	if (bt->frame == MAX_CALL_FRAMES - 1) {
-		verifier_bug(bt->env, "subprog enter from frame %d", bt->frame);
-		return -EFAULT;
-	}
-	bt->frame++;
-	return 0;
-}
-
-static inline int bt_subprog_exit(struct backtrack_state *bt)
-{
-	if (bt->frame == 0) {
-		verifier_bug(bt->env, "subprog exit from frame 0");
-		return -EFAULT;
-	}
-	bt->frame--;
-	return 0;
-}
-
-static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
-{
-	bt->reg_masks[frame] &= ~(1 << reg);
-}
-
-static inline void bt_set_reg(struct backtrack_state *bt, u32 reg)
-{
-	bpf_bt_set_frame_reg(bt, bt->frame, reg);
-}
-
-static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)
-{
-	bt_clear_frame_reg(bt, bt->frame, reg);
-}
-
-static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	__clear_bit(slot, bt->stack_masks[frame]);
-}
-
-static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)
-{
-	return bt->reg_masks[frame];
-}
-
-static inline u32 bt_reg_mask(struct backtrack_state *bt)
-{
-	return bt->reg_masks[bt->frame];
-}
-
-static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
-{
-	return bt->stack_masks[frame];
-}
-
-static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
-{
-	return bt_frame_stack_mask(bt, bt->frame);
-}
-
-static inline u32 bt_stack_arg_mask(struct backtrack_state *bt)
-{
-	return bt->stack_arg_masks[bt->frame];
-}
-
-static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
-{
-	return bt->reg_masks[bt->frame] & (1 << reg);
-}
-
-/* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
-static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
-{
-	DECLARE_BITMAP(mask, 64);
-	bool first = true;
-	int i, n;
-
-	buf[0] = '\0';
-
-	bitmap_from_u64(mask, reg_mask);
-	for_each_set_bit(i, mask, 32) {
-		n = snprintf(buf, buf_sz, "%sr%d", first ? "" : ",", i);
-		first = false;
-		buf += n;
-		buf_sz -= n;
-		if (buf_sz < 0)
-			break;
-	}
-}
-
-/* format stack slots bitmask, e.g., "-8,-24,-40" for slots 0, 2 and 4 */
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)
-{
-	bool first = true;
-	int i, n;
-
-	buf[0] = '\0';
-
-	for_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {
-		n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
-		first = false;
-		buf += n;
-		buf_sz -= n;
-		if (buf_sz < 0)
-			break;
-	}
-}
-
-/* For given verifier state backtrack_insn() is called from the last insn to
- * the first insn. Its purpose is to compute a bitmask of registers and
- * stack slots that needs precision in the parent verifier state.
- *
- * @idx is an index of the instruction we are currently processing;
- * @subseq_idx is an index of the subsequent instruction that:
- *   - *would be* executed next, if jump history is viewed in forward order;
- *   - *was* processed previously during backtracking.
- */
-static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
-			  struct bpf_jmp_history_entry *hist, struct backtrack_state *bt)
-{
-	struct bpf_insn *insn = env->prog->insnsi + idx;
-	u8 class = BPF_CLASS(insn->code);
-	u8 opcode = BPF_OP(insn->code);
-	u8 mode = BPF_MODE(insn->code);
-	u32 dreg = insn->dst_reg;
-	u32 sreg = insn->src_reg;
-	u32 spi, i, fr;
-
-	if (insn->code == 0)
-		return 0;
-	if (env->log.level & BPF_LOG_LEVEL2) {
-		fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_reg_mask(bt));
-		verbose(env, "mark_precise: frame%d: regs=%s ",
-			bt->frame, env->tmp_str_buf);
-		bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
-		verbose(env, "stack=%s before ", env->tmp_str_buf);
-		verbose(env, "%d: ", idx);
-		bpf_verbose_insn(env, insn);
-		verbose(env, "\n");
-	}
-
-	/* If there is a history record that some registers gained range at this insn,
-	 * propagate precision marks to those registers, so that bt_is_reg_set()
-	 * accounts for these registers.
-	 */
-	bpf_bt_sync_linked_regs(bt, hist);
-
-	if (class == BPF_ALU || class == BPF_ALU64) {
-		if (!bt_is_reg_set(bt, dreg))
-			return 0;
-		if (opcode == BPF_END || opcode == BPF_NEG) {
-			/* sreg is reserved and unused
-			 * dreg still need precision before this insn
-			 */
-			return 0;
-		} else if (opcode == BPF_MOV) {
-			if (BPF_SRC(insn->code) == BPF_X) {
-				/* dreg = sreg or dreg = (s8, s16, s32)sreg
-				 * dreg needs precision after this insn
-				 * sreg needs precision before this insn
-				 */
-				bt_clear_reg(bt, dreg);
-				if (sreg != BPF_REG_FP && sreg != BPF_REG_ARENA_SP)
-					bt_set_reg(bt, sreg);
-			} else {
-				/* dreg = K
-				 * dreg needs precision after this insn.
-				 * Corresponding register is already marked
-				 * as precise=true in this verifier state.
-				 * No further markings in parent are necessary
-				 */
-				bt_clear_reg(bt, dreg);
-			}
-		} else {
-			if (BPF_SRC(insn->code) == BPF_X) {
-				/* dreg += sreg
-				 * both dreg and sreg need precision
-				 * before this insn
-				 */
-				if (sreg != BPF_REG_FP)
-					bt_set_reg(bt, sreg);
-			} /* else dreg += K
-			   * dreg still needs precision before this insn
-			   */
-		}
-	} else if (class == BPF_LDX ||
-		   is_atomic_load_insn(insn) ||
-		   is_atomic_fetch_insn(insn)) {
-		u32 load_reg = dreg;
-
-		/*
-		 * Atomic fetch operation writes the old value into
-		 * a register (sreg or r0) and if it was tracked for
-		 * precision, propagate to the stack slot like we do
-		 * in regular ldx.
-		 */
-		if (is_atomic_fetch_insn(insn))
-			load_reg = insn->imm == BPF_CMPXCHG ?
-				   BPF_REG_0 : sreg;
-
-		if (!bt_is_reg_set(bt, load_reg))
-			return 0;
-		bt_clear_reg(bt, load_reg);
-
-		if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
-			spi = hist->spi;
-			/*
-			 * Stack arg read: callee reads from r11+off, but
-			 * the data lives in the caller's stack_arg_regs.
-			 * Set the mask in the caller frame so precision
-			 * is marked in the caller's slot at the callee
-			 * entry checkpoint.
-			 */
-			bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi);
-			return 0;
-		}
-
-		/* scalars can only be spilled into stack w/o losing precision.
-		 * Load from any other memory can be zero extended.
-		 * The desire to keep that precision is already indicated
-		 * by 'precise' mark in corresponding register of this state.
-		 * No further tracking necessary.
-		 */
-		if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
-			return 0;
-		/* dreg = *(u64 *)[fp - off] was a fill from the stack.
-		 * that [fp - off] slot contains scalar that needs to be
-		 * tracked with precision
-		 */
-		spi = hist->spi;
-		fr = hist->frame;
-		bpf_bt_set_frame_slot(bt, fr, spi);
-	} else if (class == BPF_STX || class == BPF_ST) {
-		if (bt_is_reg_set(bt, dreg))
-			/* stx & st shouldn't be using _scalar_ dst_reg
-			 * to access memory. It means backtracking
-			 * encountered a case of pointer subtraction.
-			 */
-			return -ENOTSUPP;
-
-		if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
-			spi = hist->spi;
-			if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi))
-				return 0;
-			bt_clear_frame_stack_arg_slot(bt, bt->frame, spi);
-			if (class == BPF_STX)
-				bt_set_reg(bt, sreg);
-			return 0;
-		}
-
-		/* scalars can only be spilled into stack */
-		if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
-			return 0;
-		spi = hist->spi;
-		fr = hist->frame;
-		if (!bt_is_frame_slot_set(bt, fr, spi))
-			return 0;
-		bt_clear_frame_slot(bt, fr, spi);
-		if (class == BPF_STX)
-			bt_set_reg(bt, sreg);
-	} else if (class == BPF_JMP || class == BPF_JMP32) {
-		if (bpf_pseudo_call(insn) || bpf_is_callx(insn)) {
-			int subprog_insn_idx, subprog = -1;
-
-			if (bpf_pseudo_call(insn)) {
-				subprog_insn_idx = idx + insn->imm + 1;
-				subprog = bpf_find_subprog(env, subprog_insn_idx);
-				if (subprog < 0)
-					return -EFAULT;
-			}
-
-			/* callx calls static subprogs only */
-			if (subprog >= 0 && bpf_subprog_is_global(env, subprog)) {
-				/* check that jump history doesn't have any
-				 * extra instructions from subprog; the next
-				 * instruction after call to global subprog
-				 * should be literally next instruction in
-				 * caller program
-				 */
-				verifier_bug_if(idx + 1 != subseq_idx, env,
-						"extra insn from subprog");
-				/* global subprog always sets R0 */
-				bt_clear_reg(bt, BPF_REG_0);
-				/* and if it does not set R2, main pass would catch it */
-				bt_clear_reg(bt, BPF_REG_2);
-				/* r1-r5 are invalidated after subprog call,
-				 * so for global func call it shouldn't be set
-				 * anymore
-				 */
-				if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
-					verifier_bug(env, "global subprog unexpected regs %x",
-						     bt_reg_mask(bt));
-					return -EFAULT;
-				}
-				return 0;
-			} else {
-				/* static subprog call instruction, which
-				 * means that we are exiting current subprog,
-				 * so only r1-r5 could be still requested as
-				 * precise, r0 and r6-r10 or any stack slot in
-				 * the current frame should be zero by now
-				 */
-				if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) {
-					verifier_bug(env, "static subprog unexpected regs %x",
-						     bt_reg_mask(bt));
-					return -EFAULT;
-				}
-				/* we are now tracking register spills correctly,
-				 * so any instance of leftover slots is a bug
-				 */
-				if (!bt_stack_empty(bt)) {
-					bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-							   bt_stack_mask(bt));
-					verifier_bug(env, "static subprog leftover stack slots %s",
-						     env->tmp_str_buf);
-					return -EFAULT;
-				}
-				/* propagate r1-r5 to the caller */
-				for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
-					if (bt_is_reg_set(bt, i)) {
-						bt_clear_reg(bt, i);
-						bpf_bt_set_frame_reg(bt, bt->frame - 1, i);
-					}
-				}
-				if (bt_stack_arg_mask(bt)) {
-					verifier_bug(env,
-						     "static subprog leftover stack arg slots %x",
-						     bt_stack_arg_mask(bt));
-					return -EFAULT;
-				}
-				if (bt_subprog_exit(bt))
-					return -EFAULT;
-				return 0;
-			}
-		} else if (bpf_is_sync_callback_calling_insn(insn) && idx != subseq_idx - 1) {
-			/* exit from callback subprog to callback-calling helper or
-			 * kfunc call. Use idx/subseq_idx check to discern it from
-			 * straight line code backtracking.
-			 * Unlike the subprog call handling above, we shouldn't
-			 * propagate precision of r1-r5 (if any requested), as they are
-			 * not actually arguments passed directly to callback subprogs
-			 */
-			if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) {
-				verifier_bug(env, "callback unexpected regs %x",
-					     bt_reg_mask(bt));
-				return -EFAULT;
-			}
-			if (!bt_stack_empty(bt)) {
-				bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-						   bt_stack_mask(bt));
-				verifier_bug(env, "callback leftover stack slots %s",
-					     env->tmp_str_buf);
-				return -EFAULT;
-			}
-			/* clear r1-r5 in callback subprog's mask */
-			for (i = BPF_REG_1; i <= BPF_REG_5; i++)
-				bt_clear_reg(bt, i);
-			if (bt_subprog_exit(bt))
-				return -EFAULT;
-			return 0;
-		} else if (opcode == BPF_CALL) {
-			/* kfunc with imm==0 is invalid and fixup_kfunc_call will
-			 * catch this error later. Make backtracking conservative
-			 * with ENOTSUPP.
-			 */
-			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && insn->imm == 0)
-				return -ENOTSUPP;
-			/* regular helper call sets R0 */
-			bt_clear_reg(bt, BPF_REG_0);
-			/* kfunc might also set R2 */
-			bt_clear_reg(bt, BPF_REG_2);
-			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
-				/* if backtracking was looking for registers R1-R5
-				 * they should have been found already.
-				 */
-				verifier_bug(env, "backtracking call unexpected regs %x",
-					     bt_reg_mask(bt));
-				return -EFAULT;
-			}
-			if (insn->src_reg == BPF_REG_0 && insn->imm == BPF_FUNC_tail_call
-			    && subseq_idx - idx != 1) {
-				if (bt_subprog_enter(bt))
-					return -EFAULT;
-			}
-		} else if (opcode == BPF_EXIT) {
-			bool from_subprog_call, r0_precise, r2_precise;
-
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate the precision of the
-			 * return registers, if necessary. In the latter we
-			 * never do that.
-			 */
-			from_subprog_call = subseq_idx - 1 >= 0 &&
-					    (bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) ||
-					     bpf_is_callx(&env->prog->insnsi[subseq_idx - 1]));
-
-			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
-			r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
-
-			/* Backtracking to a nested function call, 'idx' is a part of
-			 * the inner frame 'subseq_idx' is a part of the outer frame.
-			 * In case of a regular function call, the callee defines the
-			 * return registers R0 and R2, so clear them before checking
-			 * that instructions giving precision to registers R1-R5 have
-			 * been found already. R2 is also an argument register, hence
-			 * it has to be cleared before that check.
-			 * In case of a callback from bpf_loop(), R{1,4} in the calling
-			 * frame would be set as precise and that is correct, and R2
-			 * might be a precise helper argument as well, so leave it set.
-			 */
-			bt_clear_reg(bt, BPF_REG_0);
-			if (from_subprog_call)
-				bt_clear_reg(bt, BPF_REG_2);
-			if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
-				verifier_bug(env, "backtracking exit unexpected regs %x",
-					     bt_reg_mask(bt));
-				return -EFAULT;
-			}
-
-			if (bt_subprog_enter(bt))
-				return -EFAULT;
-
-			if (r0_precise)
-				bt_set_reg(bt, BPF_REG_0);
-			if (r2_precise)
-				bt_set_reg(bt, BPF_REG_2);
-			/* r6-r9 and stack slots will stay set in caller frame
-			 * bitmasks until we return back from callee(s)
-			 */
-			return 0;
-		} else if (BPF_SRC(insn->code) == BPF_X) {
-			if (!bt_is_reg_set(bt, dreg) && !bt_is_reg_set(bt, sreg))
-				return 0;
-			/* dreg <cond> sreg
-			 * Both dreg and sreg need precision before
-			 * this insn. If only sreg was marked precise
-			 * before it would be equally necessary to
-			 * propagate it to dreg.
-			 */
-			if (!hist || !(hist->flags & INSN_F_SRC_REG_STACK))
-				bt_set_reg(bt, sreg);
-			if (!hist || !(hist->flags & INSN_F_DST_REG_STACK))
-				bt_set_reg(bt, dreg);
-		} else if (BPF_SRC(insn->code) == BPF_K) {
-			 /* dreg <cond> K
-			  * Only dreg still needs precision before
-			  * this insn, so for the K-based conditional
-			  * there is nothing new to be marked.
-			  */
-		}
-	} else if (class == BPF_LD) {
-		/* It's ld_imm64 or ld_abs or ld_ind.
-		 * For ld_imm64 no further tracking of precision
-		 * into parent is necessary
-		 */
-		if (mode == BPF_IMM) {
-			bt_clear_reg(bt, dreg);
-			return 0;
-		}
-		/*
-		 * BPF_{IND,ABS} are modelled as two branches:
-		 * - fallthrough;
-		 * - implicit subprogram exit.
-		 * It is necessary to switch current frame if
-		 * implicit subprogram exit branch is backtracked.
-		 */
-		if (mode == BPF_IND || mode == BPF_ABS) {
-			if (bt_is_reg_set(bt, dreg))
-				return -ENOTSUPP;
-			if (subseq_idx != idx + 1)
-				if (bt_subprog_enter(bt))
-					return -EFAULT;
-			return 0;
-		}
-	}
-	/* Propagate precision marks to linked registers, to account for
-	 * registers marked as precise in this function.
-	 */
-	bpf_bt_sync_linked_regs(bt, hist);
-	return 0;
-}
-
-/* the scalar precision tracking algorithm:
- * . at the start all registers have precise=false.
- * . scalar ranges are tracked as normal through alu and jmp insns.
- * . once precise value of the scalar register is used in:
- *   .  ptr + scalar alu
- *   . if (scalar cond K|scalar)
- *   .  helper_call(.., scalar, ...) where ARG_CONST is expected
- *   backtrack through the verifier states and mark all registers and
- *   stack slots with spilled constants that these scalar registers
- *   should be precise.
- * . during state pruning two registers (or spilled stack slots)
- *   are equivalent if both are not precise.
- *
- * Note the verifier cannot simply walk register parentage chain,
- * since many different registers and stack slots could have been
- * used to compute single precise scalar.
- *
- * The approach of starting with precise=true for all registers and then
- * backtrack to mark a register as not precise when the verifier detects
- * that program doesn't care about specific value (e.g., when helper
- * takes register as ARG_ANYTHING parameter) is not safe.
- *
- * It's ok to walk single parentage chain of the verifier states.
- * It's possible that this backtracking will go all the way till 1st insn.
- * All other branches will be explored for needing precision later.
- *
- * The backtracking needs to deal with cases like:
- *   R8=map_value(id=0,off=0,ks=4,vs=1952,imm=0) R9_w=map_value(id=0,off=40,ks=4,vs=1952,imm=0)
- * r9 -= r8
- * r5 = r9
- * if r5 > 0x79f goto pc+7
- *    R5_w=inv(id=0,umax_value=1951,var_off=(0x0; 0x7ff))
- * r5 += 1
- * ...
- * call bpf_perf_event_output#25
- *   where .arg5_type = ARG_MEM_SIZE_OR_ZERO
- *
- * and this case:
- * r6 = 1
- * call foo // uses callee's r6 inside to compute r0
- * r0 += r6
- * if r0 == 0 goto
- *
- * to track above reg_mask/stack_mask needs to be independent for each frame.
- *
- * Also if parent's curframe > frame where backtracking started,
- * the verifier need to mark registers in both frames, otherwise callees
- * may incorrectly prune callers. This is similar to
- * commit 7640ead93924 ("bpf: verifier: make sure callees don't prune with caller differences")
- *
- * For now backtracking falls back into conservative marking.
- */
-void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
-				 struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *func;
-	struct bpf_reg_state *reg;
-	int i, j;
-
-	if (env->log.level & BPF_LOG_LEVEL2) {
-		verbose(env, "mark_precise: frame%d: falling back to forcing all scalars precise\n",
-			st->curframe);
-	}
-
-	/* big hammer: mark all scalars precise in this path.
-	 * pop_stack may still get !precise scalars.
-	 * We also skip current state and go straight to first parent state,
-	 * because precision markings in current non-checkpointed state are
-	 * not needed. See why in the comment in __mark_chain_precision below.
-	 */
-	for (st = st->parent; st; st = st->parent) {
-		for (i = 0; i <= st->curframe; i++) {
-			func = st->frame[i];
-			for (j = 0; j < BPF_REG_FP; j++) {
-				reg = &func->regs[j];
-				if (reg->type != SCALAR_VALUE || reg->precise)
-					continue;
-				reg->precise = true;
-				if (env->log.level & BPF_LOG_LEVEL2) {
-					verbose(env, "force_precise: frame%d: forcing r%d to be precise\n",
-						i, j);
-				}
-			}
-			for (j = 0; j < bpf_stack_nr_slots(func); j++) {
-				struct bpf_stack_state *ss = bpf_stack_slot(func, j);
-
-				if (!bpf_is_spilled_reg(ss))
-					continue;
-				reg = &ss->spilled_ptr;
-				if (reg->type != SCALAR_VALUE || reg->precise)
-					continue;
-				reg->precise = true;
-				if (env->log.level & BPF_LOG_LEVEL2) {
-					verbose(env, "force_precise: frame%d: forcing fp%d to be precise\n",
-						i, -(j + 1) * 8);
-				}
-			}
-			for (j = 0; j < func->out_stack_arg_cnt; j++) {
-				reg = &func->stack_arg_regs[j];
-				if (reg->type != SCALAR_VALUE || reg->precise)
-					continue;
-				reg->precise = true;
-				if (env->log.level & BPF_LOG_LEVEL2) {
-					verbose(env,
-						"force_precise: frame%d: forcing r11%d to be precise\n",
-						i, -(j + 1) * BPF_REG_SIZE);
-				}
-			}
-		}
-	}
-}
-
-/*
- * bpf_mark_chain_precision() backtracks BPF program instruction sequence and
- * chain of verifier states making sure that register *regno* (if regno >= 0)
- * and/or stack slot *spi* (if spi >= 0) are marked as precisely tracked
- * SCALARS, as well as any other registers and slots that contribute to
- * a tracked state of given registers/stack slots, depending on specific BPF
- * assembly instructions (see backtrack_insns() for exact instruction handling
- * logic). This backtracking relies on recorded jmp_history and is able to
- * traverse entire chain of parent states. This process ends only when all the
- * necessary registers/slots and their transitive dependencies are marked as
- * precise.
- *
- * One important and subtle aspect is that precise marks *do not matter* in
- * the currently verified state (current state). It is important to understand
- * why this is the case.
- *
- * First, note that current state is the state that is not yet "checkpointed",
- * i.e., it is not yet put into env->explored_states, and it has no children
- * states as well. It's ephemeral, and can end up either a) being discarded if
- * compatible explored state is found at some point or BPF_EXIT instruction is
- * reached or b) checkpointed and put into env->explored_states, branching out
- * into one or more children states.
- *
- * In the former case, precise markings in current state are completely
- * ignored by state comparison code (see regsafe() for details). Only
- * checkpointed ("old") state precise markings are important, and if old
- * state's register/slot is precise, regsafe() assumes current state's
- * register/slot as precise and checks value ranges exactly and precisely. If
- * states turn out to be compatible, current state's necessary precise
- * markings and any required parent states' precise markings are enforced
- * after the fact with propagate_precision() logic, after the fact. But it's
- * important to realize that in this case, even after marking current state
- * registers/slots as precise, we immediately discard current state. So what
- * actually matters is any of the precise markings propagated into current
- * state's parent states, which are always checkpointed (due to b) case above).
- * As such, for scenario a) it doesn't matter if current state has precise
- * markings set or not.
- *
- * Now, for the scenario b), checkpointing and forking into child(ren)
- * state(s). Note that before current state gets to checkpointing step, any
- * processed instruction always assumes precise SCALAR register/slot
- * knowledge: if precise value or range is useful to prune jump branch, BPF
- * verifier takes this opportunity enthusiastically. Similarly, when
- * register's value is used to calculate offset or memory address, exact
- * knowledge of SCALAR range is assumed, checked, and enforced. So, similar to
- * what we mentioned above about state comparison ignoring precise markings
- * during state comparison, BPF verifier ignores and also assumes precise
- * markings *at will* during instruction verification process. But as verifier
- * assumes precision, it also propagates any precision dependencies across
- * parent states, which are not yet finalized, so can be further restricted
- * based on new knowledge gained from restrictions enforced by their children
- * states. This is so that once those parent states are finalized, i.e., when
- * they have no more active children state, state comparison logic in
- * is_state_visited() would enforce strict and precise SCALAR ranges, if
- * required for correctness.
- *
- * To build a bit more intuition, note also that once a state is checkpointed,
- * the path we took to get to that state is not important. This is crucial
- * property for state pruning. When state is checkpointed and finalized at
- * some instruction index, it can be correctly and safely used to "short
- * circuit" any *compatible* state that reaches exactly the same instruction
- * index. I.e., if we jumped to that instruction from a completely different
- * code path than original finalized state was derived from, it doesn't
- * matter, current state can be discarded because from that instruction
- * forward having a compatible state will ensure we will safely reach the
- * exit. States describe preconditions for further exploration, but completely
- * forget the history of how we got here.
- *
- * This also means that even if we needed precise SCALAR range to get to
- * finalized state, but from that point forward *that same* SCALAR register is
- * never used in a precise context (i.e., it's precise value is not needed for
- * correctness), it's correct and safe to mark such register as "imprecise"
- * (i.e., precise marking set to false). This is what we rely on when we do
- * not set precise marking in current state. If no child state requires
- * precision for any given SCALAR register, it's safe to dictate that it can
- * be imprecise. If any child state does require this register to be precise,
- * we'll mark it precise later retroactively during precise markings
- * propagation from child state to parent states.
- *
- * Skipping precise marking setting in current state is a mild version of
- * relying on the above observation. But we can utilize this property even
- * more aggressively by proactively forgetting any precise marking in the
- * current state (which we inherited from the parent state), right before we
- * checkpoint it and branch off into new child state. This is done by
- * mark_all_scalars_imprecise() to hopefully get more permissive and generic
- * finalized states which help in short circuiting more future states.
- */
-int bpf_mark_chain_precision(struct bpf_verifier_env *env,
-			    struct bpf_verifier_state *starting_state,
-			    int regno,
-			    bool *changed)
-{
-	struct bpf_verifier_state *st = starting_state;
-	struct backtrack_state *bt = &env->bt;
-	int first_idx = st->first_insn_idx;
-	int last_idx = starting_state->insn_idx;
-	int subseq_idx = -1;
-	struct bpf_func_state *func;
-	bool tmp, skip_first = true;
-	struct bpf_stack_state *ss;
-
-	/* FLOW: there are no precision marks */
-	bt_reset(bt);
-	return 0;
-	struct bpf_reg_state *reg;
-	int i, fr, err;
-
-	if (!env->bpf_capable)
-		return 0;
-
-	changed = changed ?: &tmp;
-	/* set frame number from which we are starting to backtrack */
-	bt_init(bt, starting_state->curframe);
-
-	/* Do sanity checks against current state of register and/or stack
-	 * slot, but don't set precise flag in current state, as precision
-	 * tracking in the current state is unnecessary.
-	 */
-	func = st->frame[bt->frame];
-	if (regno >= 0) {
-		reg = &func->regs[regno];
-		if (reg->type != SCALAR_VALUE) {
-			verifier_bug(env, "backtracking misuse");
-			return -EFAULT;
-		}
-		bt_set_reg(bt, regno);
-	}
-
-	if (bt_empty(bt))
-		return 0;
-
-	for (;;) {
-		DECLARE_BITMAP(mask, 64);
-		u32 history = st->jmp_history_cnt;
-		struct bpf_jmp_history_entry *hist;
-
-		if (env->log.level & BPF_LOG_LEVEL2) {
-			verbose(env, "mark_precise: frame%d: last_idx %d first_idx %d subseq_idx %d \n",
-				bt->frame, last_idx, first_idx, subseq_idx);
-		}
-
-		if (last_idx < 0) {
-			/* Consecutive checkpoints can have no instructions between them. */
-			if (st->parent)
-				goto parent;
-
-			/* we are at the entry into subprog, which
-			 * is expected for global funcs, but only if
-			 * requested precise registers are R1-R5
-			 * (which are global func's input arguments)
-			 */
-			if (st->curframe == 0 &&
-			    st->frame[0]->subprogno > 0 &&
-			    st->frame[0]->callsite == BPF_MAIN_FUNC &&
-			    bt_stack_empty(bt) &&
-			    (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) == 0) {
-				bitmap_from_u64(mask, bt_reg_mask(bt));
-				for_each_set_bit(i, mask, 32) {
-					reg = &st->frame[0]->regs[i];
-					bt_clear_reg(bt, i);
-					if (reg->type == SCALAR_VALUE) {
-						reg->precise = true;
-						*changed = true;
-					}
-				}
-				return 0;
-			}
-
-			bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
-			verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %s",
-				     st->frame[0]->subprogno, bt_reg_mask(bt), env->tmp_str_buf);
-			return -EFAULT;
-		}
-
-		for (i = last_idx;;) {
-			if (skip_first) {
-				err = 0;
-				skip_first = false;
-			} else {
-				hist = get_jmp_hist_entry(st, history, i);
-				err = backtrack_insn(env, i, subseq_idx, hist, bt);
-			}
-			if (err == -ENOTSUPP) {
-				bpf_mark_all_scalars_precise(env, starting_state);
-				bt_reset(bt);
-				return 0;
-			} else if (err) {
-				return err;
-			}
-			if (bt_empty(bt))
-				/* Found assignment(s) into tracked register in this state.
-				 * Since this state is already marked, just return.
-				 * Nothing to be tracked further in the parent state.
-				 */
-				return 0;
-			subseq_idx = i;
-			i = get_prev_insn_idx(st, i, &history);
-			if (i == -ENOENT)
-				break;
-			if (i >= env->prog->len) {
-				/* This can happen if backtracking reached insn 0
-				 * and there are still reg_mask or stack_mask
-				 * to backtrack.
-				 * It means the backtracking missed the spot where
-				 * particular register was initialized with a constant.
-				 */
-				verifier_bug(env, "backtracking idx %d", i);
-				return -EFAULT;
-			}
-		}
-parent:
-		st = st->parent;
-		if (!st)
-			break;
-
-		for (fr = bt->frame; fr >= 0; fr--) {
-			func = st->frame[fr];
-			bitmap_from_u64(mask, bt_frame_reg_mask(bt, fr));
-			for_each_set_bit(i, mask, 32) {
-				reg = &func->regs[i];
-				if (reg->type != SCALAR_VALUE) {
-					bt_clear_frame_reg(bt, fr, i);
-					continue;
-				}
-				if (reg->precise) {
-					bt_clear_frame_reg(bt, fr, i);
-				} else {
-					reg->precise = true;
-					*changed = true;
-				}
-			}
-
-			for_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {
-				if (verifier_bug_if(i >= bpf_stack_nr_slots(func),
-						    env, "stack slot %d, total slots %d",
-						    i, bpf_stack_nr_slots(func)))
-					return -EFAULT;
-
-				ss = bpf_stack_slot(func, i);
-				if (!bpf_is_spilled_scalar_reg(ss)) {
-					bt_clear_frame_slot(bt, fr, i);
-					continue;
-				}
-				reg = &ss->spilled_ptr;
-				if (reg->precise) {
-					bt_clear_frame_slot(bt, fr, i);
-				} else {
-					reg->precise = true;
-					*changed = true;
-				}
-			}
-			for (i = 0; i < func->out_stack_arg_cnt; i++) {
-				if (!bt_is_frame_stack_arg_slot_set(bt, fr, i))
-					continue;
-				reg = &func->stack_arg_regs[i];
-				if (reg->type != SCALAR_VALUE || reg->precise) {
-					bt_clear_frame_stack_arg_slot(bt, fr, i);
-				} else {
-					reg->precise = true;
-					*changed = true;
-				}
-			}
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-					     bt_frame_reg_mask(bt, fr));
-				verbose(env, "mark_precise: frame%d: parent state regs=%s ",
-					fr, env->tmp_str_buf);
-				bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-					       bt_frame_stack_mask(bt, fr));
-				verbose(env, "stack=%s: ", env->tmp_str_buf);
-				print_verifier_state(env, st, fr, true);
-			}
-		}
-
-		if (bt_empty(bt))
-			return 0;
-
-		subseq_idx = first_idx;
-		last_idx = st->last_insn_idx;
-		first_idx = st->first_insn_idx;
-	}
-
-	/* if we still have requested precise regs or slots, we missed
-	 * something (e.g., stack access through non-r10 register), so
-	 * fallback to marking all precise
-	 */
-	if (!bt_empty(bt)) {
-		bpf_mark_all_scalars_precise(env, starting_state);
-		bt_reset(bt);
-	}
-
-	return 0;
-}
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 00cb219e24cd..db496253bf62 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -637,8 +637,6 @@ static void print_reg_state(struct bpf_verifier_env *env,
 	const char *sep = "";
 
 	t = reg->type;
-	if (t == SCALAR_VALUE && reg->precise)
-		verbose(env, "P");
 	if (t == SCALAR_VALUE && tnum_is_const(reg->var_off)) {
 		bpf_verbose_snum(env, reg->var_off.value);
 		return;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index c3ce13a52f6f..ebdf7d148c85 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1854,7 +1854,7 @@ int bpf_init_scev(struct bpf_verifier_env *env)
 		return -ENOMEM;
 	env->scev = scev;
 	bpf_mark_reg_known_scalar(&scev->zero_reg, 0);
-	bpf_mark_reg_unknown_imprecise(&scev->unknown_reg);
+	bpf_mark_reg_unknown(&scev->unknown_reg);
 	/* A power of two for hash & (exprs_ht_cnt - 1)` indexing. */
 	scev->exprs_ht_cnt = roundup_pow_of_two(max(256U, DIV_ROUND_UP(env->prog->len, 4)));
 	scev->exprs_ht = kvcalloc(scev->exprs_ht_cnt, sizeof(*scev->exprs_ht), GFP_KERNEL_ACCOUNT);
@@ -3198,30 +3198,6 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
 		bounds->r64 = r64;
 }
 
-static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 id,
-			     unsigned long *precise_regs)
-{
-	struct scev *scev = env->scev;
-	struct bpf_reg_state *reg;
-	u32 r, order;
-
-	scev->stack_sz = 0;
-	expr_stack_push(scev, id);
-	while (expr_next(scev, &id, &order)) {
-		if (order & DEPTH_LIMIT) {
-			verifier_bug(env, "scev precision marking exceeds expression depth limit");
-			return -EFAULT;
-		}
-		if (!(order & PRE) || !is_reg(scev, id, &r))
-			continue;
-		/* Synthetic values have no spilled register to mark precise. */
-		reg = scev_reg_for_linear(st, r) ?: scev_reg_for_spill32(st, r);
-		if (reg && reg->type == SCALAR_VALUE)
-			__set_bit(r, precise_regs);
-	}
-	return 0;
-}
-
 /*
  * Does @id have no bit that register @r doesn't have at the head of the loop? 'x & e' and the
  * low bits of x lose bits only. 'x >> e', 'x / e' and 'x % e' are not greater than x as well, but
@@ -3376,17 +3352,11 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	 * which iterations it can be taken. The loop ends, at the latest, in an iteration
 	 * in which no back edge can be taken.
 	 */
-	bitmap_zero(iters->precise_regs, REGS_NUM);
 	for (i = 0; i < loop->backedges_cnt; i++) {
 		err = count_backedge(env, cur_func, insn_idx, &loop->backedges[i],
 				     i ? &tmp : iters, &latch, &until);
 		if (err <= 0)
 			return err;
-		err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs) ?:
-		      mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs) ?:
-		      mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
-		if (err)
-			return err;
 		if (!i) {
 			false_until = until;
 			first = latch;
@@ -3582,7 +3552,7 @@ static void scev_forget_reg(struct bpf_verifier_env *env, struct bpf_func_state
 
 	if (r < MAX_BPF_REG) {
 		if (func->regs[r].type == SCALAR_VALUE) {
-			bpf_mark_reg_unknown_imprecise(&func->regs[r]);
+			bpf_mark_reg_unknown(&func->regs[r]);
 			mark_scev_reg_scratched(env, r);
 		} else if (func->regs[r].type == PTR_TO_ARENA) {
 			/* NULL or not is all that is known about it */
@@ -3675,16 +3645,6 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 	int insn_idx = st->insn_idx;
 	int err;
 
-	for_each_set_bit(r, iters->precise_regs, REGS_NUM) {
-		if (r < MAX_BPF_REG)
-			bpf_bt_set_frame_reg(&env->bt, st->curframe, r);
-		else
-			bpf_bt_set_frame_slot(&env->bt, st->curframe, r - MAX_BPF_REG);
-	}
-	err = bpf_mark_chain_precision(env, st, -1, NULL);
-	if (err)
-		return err;
-
 	header_env = find_header_env(scev, insn_idx);
 	for (r = 0; r < REGS_NUM; r++) {
 		if (!scev_reg_alive(env, st, r))
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 343f29b52d79..5935e16730fd 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -5,8 +5,6 @@
 #include <linux/cnum.h>
 #include <linux/filter.h>
 
-#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-
 /* check %cur's range satisfies %old's */
 static bool range_within(const struct bpf_reg_state *old,
 			 const struct bpf_reg_state *cur)
@@ -182,10 +180,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
 				if (lo_live && stype == STACK_SPILL) {
 					if (spill->type != SCALAR_VALUE)
 						continue;
-					/*
-					 * Can't replace with STACK_ZERO, because
-					 * that requires bpf_mark_chain_precision().
-					 */
+					/* keep what is known about the low half */
 					if (bpf_register_is_null(spill))
 						continue;
 					for (j = 0; j < 4; j++) {
@@ -281,8 +276,6 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 			return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
 			       check_scalar_ids(rold->id, rcur->id, idmap);
 		}
-		if (!rold->precise && exact == NOT_EXACT)
-			return true;
 		/*
 		 * Linked register tracking uses rold->id to detect relationships.
 		 * When rold->id == 0, the register is independent and any linking
@@ -394,7 +387,7 @@ static struct bpf_reg_state unbound_reg;
 
 static __init int unbound_reg_init(void)
 {
-	bpf_mark_reg_unknown_imprecise(&unbound_reg);
+	bpf_mark_reg_unknown(&unbound_reg);
 	return 0;
 }
 late_initcall(unbound_reg_init);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e3be9f5e6787..fb195d20977c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -302,11 +302,6 @@ static int arg_from_argno(argno_t a)
 	return -1;
 }
 
-static int arg_idx_from_argno(argno_t a)
-{
-	return arg_from_argno(a) - 1;
-}
-
 /* Normalize helper register numbers and kfunc argument numbers to ABI slots. */
 static u32 arg_slot_from_argno(argno_t a)
 {
@@ -856,15 +851,12 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
 				      : reg->id);
 }
 
-static void __mark_reg_unknown(const struct bpf_verifier_env *env,
-			       struct bpf_reg_state *reg);
-
 static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	if (!env->allow_ptr_leaks)
 		bpf_mark_reg_not_init(env, reg);
 	else
-		__mark_reg_unknown(env, reg);
+		bpf_mark_reg_unknown(reg);
 }
 
 static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
@@ -1754,13 +1746,6 @@ static void free_func_state(struct bpf_func_state *state)
 	kfree(state);
 }
 
-void bpf_clear_jmp_history(struct bpf_verifier_state *state)
-{
-	kfree(state->jmp_history);
-	state->jmp_history = NULL;
-	state->jmp_history_cnt = 0;
-}
-
 void bpf_free_verifier_state(struct bpf_verifier_state *state,
 			    bool free_self)
 {
@@ -1771,7 +1756,6 @@ void bpf_free_verifier_state(struct bpf_verifier_state *state,
 		state->frame[i] = NULL;
 	}
 	kfree(state->refs);
-	bpf_clear_jmp_history(state);
 	if (free_self)
 		kfree(state);
 }
@@ -1798,13 +1782,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	struct bpf_func_state *dst;
 	int i, err;
 
-	dst_state->jmp_history = copy_array(dst_state->jmp_history, src->jmp_history,
-					  src->jmp_history_cnt, sizeof(*dst_state->jmp_history),
-					  GFP_KERNEL_ACCOUNT);
-	if (!dst_state->jmp_history)
-		return -ENOMEM;
-	dst_state->jmp_history_cnt = src->jmp_history_cnt;
-
 	/* if dst has more stack frames then src frame, free them, this is also
 	 * necessary in case of exceptional exits using bpf_throw.
 	 */
@@ -1823,8 +1800,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	dst_state->flow_entry = src->flow_entry;
 	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
-	dst_state->parent = src->parent;
-	dst_state->first_insn_idx = src->first_insn_idx;
 	dst_state->last_insn_idx = src->last_insn_idx;
 	for (i = 0; i <= src->curframe; i++) {
 		dst = dst_state->frame[i];
@@ -2094,14 +2069,10 @@ static void __mark_reg_known_zero(struct bpf_reg_state *reg)
 	__mark_reg_known(reg, 0);
 }
 
-static void __mark_reg_const_zero(const struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static void __mark_reg_const_zero(struct bpf_reg_state *reg)
 {
 	__mark_reg_known(reg, 0);
 	reg->type = SCALAR_VALUE;
-	/* all scalars are assumed imprecise initially (unless unprivileged,
-	 * in which case everything is forced to be precise)
-	 */
-	reg->precise = !env->bpf_capable;
 }
 
 static void mark_reg_known_zero(struct bpf_verifier_env *env,
@@ -2434,7 +2405,7 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env,
 }
 
 /* Mark a register as having a completely unknown (scalar) value. */
-void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
+void bpf_mark_reg_unknown(struct bpf_reg_state *reg)
 {
 	memset(reg, 0, sizeof(*reg));
 	reg->type = SCALAR_VALUE;
@@ -2443,20 +2414,10 @@ void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
 	reg_step_reset(reg);
 }
 
-/* Mark a register as having a completely unknown (scalar) value,
- * initialize .precise as true when not bpf capable.
- */
-static void __mark_reg_unknown(const struct bpf_verifier_env *env,
-			       struct bpf_reg_state *reg)
-{
-	bpf_mark_reg_unknown_imprecise(reg);
-	reg->precise = !env->bpf_capable;
-}
-
 static void mark_reg_unknown(struct bpf_verifier_env *env,
 			     struct bpf_reg_state *regs, u32 regno)
 {
-	__mark_reg_unknown(env, regs + regno);
+	bpf_mark_reg_unknown(regs + regno);
 }
 
 static int __mark_reg_s32_range(struct bpf_verifier_env *env,
@@ -2482,7 +2443,7 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env,
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg)
 {
-	__mark_reg_unknown(env, reg);
+	bpf_mark_reg_unknown(reg);
 	reg->type = NOT_INIT;
 }
 
@@ -3628,19 +3589,7 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
 	env->insn_aux_data[idx].non_stack_access = true;
 }
 
-/* Layout of one packed linked register in the jump history, see linked_regs_pack() */
-#define LR_FRAMENO_BITS	5
-#define LR_INDEX_BITS	10
-#define LR_FRAMENO_MASK	((1u << LR_FRAMENO_BITS) - 1)
-#define LR_IS_REG	BIT(LR_FRAMENO_BITS)
-#define LR_INDEX_OFF	(LR_FRAMENO_BITS + 1)
-#define LR_INDEX_MASK	((1u << LR_INDEX_BITS) - 1)
-#define LINKED_REGS_MAX	BPF_LINKED_REGS_MAX
-
-static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
-static_assert(MAX_BPF_REG <= (1 << LR_INDEX_BITS));
-static_assert(MAX_BPF_STACK_SLOTS <= (1 << LR_INDEX_BITS));
-static_assert(LR_INDEX_OFF + LR_INDEX_BITS <= 16);
+#define LINKED_REGS_MAX	5
 
 struct linked_reg {
 	u8 frameno;
@@ -3664,38 +3613,6 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s)
 	return NULL;
 }
 
-/*
- * Pack linked registers for a jump history entry, one u16 each:
- * - 5 bits frameno
- * - 1 bit  is_reg
- * - 10 bits register or stack slot index
- */
-static void linked_regs_pack(const struct linked_regs *s, u16 *packed)
-{
-	int i;
-
-	for (i = 0; i < s->cnt; ++i) {
-		const struct linked_reg *e = &s->entries[i];
-
-		packed[i] = e->frameno | (e->is_reg ? LR_IS_REG : 0) | (e->spi << LR_INDEX_OFF);
-	}
-}
-
-static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)
-{
-	int i;
-
-	s->cnt = hist->linked_regs_cnt;
-	for (i = 0; i < s->cnt; ++i) {
-		struct linked_reg *e = &s->entries[i];
-		u16 packed = hist->linked_regs[i];
-
-		e->frameno = packed & LR_FRAMENO_MASK;
-		e->is_reg  = packed & LR_IS_REG;
-		e->spi     = (packed >> LR_INDEX_OFF) & LR_INDEX_MASK;
-	}
-}
-
 const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn)
 {
 	const struct btf_type *func;
@@ -3725,56 +3642,6 @@ void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	print_bpf_insn(&cbs, insn, env->allow_ptr_leaks);
 }
 
-/* If any register R in hist->linked_regs is marked as precise in bt,
- * do bt_set_frame_{reg,slot}(bt, R) for all registers in hist->linked_regs.
- */
-void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist)
-{
-	struct linked_regs linked_regs;
-	bool some_precise = false;
-	int i;
-
-	if (!hist || !hist->linked_regs_cnt)
-		return;
-
-	linked_regs_unpack(hist, &linked_regs);
-	for (i = 0; i < linked_regs.cnt; ++i) {
-		struct linked_reg *e = &linked_regs.entries[i];
-
-		if ((e->is_reg && bt_is_frame_reg_set(bt, e->frameno, e->regno)) ||
-		    (!e->is_reg && bt_is_frame_slot_set(bt, e->frameno, e->spi))) {
-			some_precise = true;
-			break;
-		}
-	}
-
-	if (!some_precise)
-		return;
-
-	for (i = 0; i < linked_regs.cnt; ++i) {
-		struct linked_reg *e = &linked_regs.entries[i];
-
-		if (e->is_reg)
-			bpf_bt_set_frame_reg(bt, e->frameno, e->regno);
-		else
-			bpf_bt_set_frame_slot(bt, e->frameno, e->spi);
-	}
-}
-
-int mark_chain_precision(struct bpf_verifier_env *env, int regno)
-{
-	return bpf_mark_chain_precision(env, env->cur_state, regno, NULL);
-}
-
-/* mark_chain_precision_batch() assumes that env->bt is set in the caller to
- * desired reg and stack masks across all relevant frames
- */
-static int mark_chain_precision_batch(struct bpf_verifier_env *env,
-				      struct bpf_verifier_state *starting_state)
-{
-	return bpf_mark_chain_precision(env, starting_state, -1, NULL);
-}
-
 /* check if register is a constant scalar value */
 static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32)
 {
@@ -3905,8 +3772,6 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 	int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	struct bpf_reg_state *reg = NULL;
-	int insn_flags = INSN_F_STACK_ACCESS;
-	int hist_spi = spi, hist_frame = state->frameno;
 	struct bpf_stack_state *ss = bpf_stack_slot(state, spi);
 
 	/*
@@ -3996,28 +3861,14 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 
 		/* when we zero initialize stack slots mark them as such */
 		if ((reg && bpf_register_is_null(reg)) ||
-		    (!reg && is_bpf_st_mem(insn) && insn->imm == 0)) {
-			/* STACK_ZERO case happened because register spill
-			 * wasn't properly aligned at the stack slot boundary,
-			 * so it's not a register spill anymore; force
-			 * originating register to be precise to make
-			 * STACK_ZERO correct for subsequent states
-			 */
-			err = mark_chain_precision(env, value_regno);
-			if (err)
-				return err;
+		    (!reg && is_bpf_st_mem(insn) && insn->imm == 0))
 			type = STACK_ZERO;
-		}
 
 		/* Mark slots affected by this stack write. */
 		for (i = 0; i < size; i++)
 			ss->slot_type[(slot - i) % BPF_REG_SIZE] = type;
-		insn_flags = 0; /* not a register spill */
 	}
 
-	if (insn_flags)
-		return bpf_push_jmp_history(env, env->cur_state, insn_flags,
-					    hist_spi, hist_frame, NULL, 0);
 	return 0;
 }
 
@@ -4051,10 +3902,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 	struct bpf_reg_state *value_reg = NULL;
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	bool writing_zero = false;
-	/* set if the fact that we're writing a zero is used to let any
-	 * stack slots remain STACK_ZERO
-	 */
-	bool zero_used = false;
 
 	cur = env->cur_state->frame[env->cur_state->curframe];
 	min_off = reg_smin(ptr_reg) + off;
@@ -4105,10 +3952,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 		if (writing_zero && *stype == STACK_SPILL && bpf_is_spilled_scalar_reg(ss)) {
 			struct bpf_reg_state *spill_reg = &ss->spilled_ptr;
 
-			if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0) {
-				zero_used = true;
+			if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0)
 				continue;
-			}
 		}
 
 		/*
@@ -4121,10 +3966,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 
 		/* Update the slot type. */
 		new_type = STACK_MISC;
-		if (writing_zero && *stype == STACK_ZERO) {
+		if (writing_zero && *stype == STACK_ZERO)
 			new_type = STACK_ZERO;
-			zero_used = true;
-		}
 		/* If the slot is STACK_INVALID, we check whether it's OK to
 		 * pretend that it will be initialized by this write. The slot
 		 * might not actually be written to, and so if we mark it as
@@ -4142,12 +3985,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 		}
 		*stype = new_type;
 	}
-	if (zero_used) {
-		/* backtracking doesn't work for STACK_ZERO yet. */
-		err = mark_chain_precision(env, value_regno);
-		if (err)
-			return err;
-	}
 	bpf_diag_record_scrub_stack(env, state, min_off, max_off,
 				    BPF_DIAG_MOD_VAR_WRITE);
 	return 0;
@@ -4162,21 +3999,18 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
  * read.
  *
  * STACK_SPILL bytes backed by spilled scalar const zeroes are also considered
- * zero bytes. In that case, mark the contributing stack slots precise so
- * pruning cannot reuse a zero-spill state for a later non-zero spill state.
- *
- * Returns an error if precision backtracking fails.
+ * zero bytes.
  */
-static int mark_reg_stack_read(struct bpf_verifier_env *env,
-			       /* func where src register points to */
-			       struct bpf_func_state *ptr_state,
-			       int min_off, int max_off, int dst_regno)
+static void mark_reg_stack_read(struct bpf_verifier_env *env,
+				/* func where src register points to */
+				struct bpf_func_state *ptr_state,
+				int min_off, int max_off, int dst_regno)
 {
 	struct bpf_verifier_state *vstate = env->cur_state;
 	struct bpf_func_state *state = vstate->frame[vstate->curframe];
 	int i, slot, spi;
 	u8 *stype;
-	int zeros = 0, zero_spills = 0;
+	int zeros = 0;
 
 	for (i = min_off; i < max_off; i++) {
 		slot = -i - 1;
@@ -4189,7 +4023,6 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
 		}
 		if (stype[slot % BPF_REG_SIZE] == STACK_SPILL &&
 		    bpf_register_is_null(&bpf_stack_slot(ptr_state, spi)->spilled_ptr)) {
-			zero_spills++;
 			zeros++;
 			continue;
 		}
@@ -4199,23 +4032,11 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
 		/* Any access_size read into register is zero extended,
 		 * so the whole register == const_zero.
 		 */
-		__mark_reg_const_zero(env, &state->regs[dst_regno]);
-		if (zero_spills) {
-			for (i = min_off; i < max_off; i++) {
-				slot = -i - 1;
-				spi = slot / BPF_REG_SIZE;
-				stype = bpf_stack_slot(ptr_state, spi)->slot_type;
-				if (stype[slot % BPF_REG_SIZE] == STACK_SPILL)
-					bpf_bt_set_frame_slot(&env->bt, ptr_state->frameno, spi);
-			}
-			return mark_chain_precision_batch(env, env->cur_state);
-		}
+		__mark_reg_const_zero(&state->regs[dst_regno]);
 	} else {
 		/* have read misc data from the stack */
 		mark_reg_unknown(env, state->regs, dst_regno);
 	}
-
-	return 0;
 }
 
 static void bpf_diag_stack_read_invalid(struct bpf_verifier_env *env, int off, int i, int size,
@@ -4279,9 +4100,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 	int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;
 	struct bpf_reg_state *reg;
 	u8 *stype, type;
-	int err;
-	int insn_flags = INSN_F_STACK_ACCESS;
-	int hist_spi = spi, hist_frame = reg_state->frameno;
 	struct bpf_stack_state *ss = bpf_stack_slot(reg_state, spi);
 
 	stype = ss->slot_type;
@@ -4356,18 +4174,13 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 
 				if (spill_cnt == size &&
 				    tnum_is_const(reg->var_off) && reg->var_off.value == 0) {
-					__mark_reg_const_zero(env, &state->regs[dst_regno]);
-					/* this IS register fill, so keep insn_flags */
+					__mark_reg_const_zero(&state->regs[dst_regno]);
 				} else if (zero_cnt == size) {
 					/* similarly to mark_reg_stack_read(), preserve zeroes */
-					__mark_reg_const_zero(env, &state->regs[dst_regno]);
-					insn_flags = 0; /* not restoring original register state */
+					__mark_reg_const_zero(&state->regs[dst_regno]);
 				} else {
-					err = mark_reg_stack_read(env, reg_state, off, off + size,
-								  dst_regno);
-					if (err)
-						return err;
-					insn_flags = 0; /* not restoring original register state */
+					mark_reg_stack_read(env, reg_state, off, off + size,
+							    dst_regno);
 				}
 			}
 		} else if (dst_regno >= 0) {
@@ -4378,10 +4191,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 				 */
 				assign_scalar_id_before_mov(env, reg);
 			state->regs[dst_regno] = *reg;
-			/* mark reg as written since spilled pointer state likely
-			 * has its liveness marks cleared by is_state_visited()
-			 * which resets stack/reg liveness for state transitions
-			 */
 		} else if (__is_pointer_value(env->allow_ptr_leaks, reg)) {
 			/* If dst_regno==-1, the caller is asking us whether
 			 * it is acceptable to use this value as a SCALAR_VALUE
@@ -4412,16 +4221,9 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 			}
 			return -EACCES;
 		}
-		if (dst_regno >= 0) {
-			err = mark_reg_stack_read(env, reg_state, off, off + size, dst_regno);
-			if (err)
-				return err;
-		}
-		insn_flags = 0; /* we are not restoring spilled register */
+		if (dst_regno >= 0)
+			mark_reg_stack_read(env, reg_state, off, off + size, dst_regno);
 	}
-	if (insn_flags)
-		return bpf_push_jmp_history(env, env->cur_state, insn_flags,
-					    hist_spi, hist_frame, NULL, 0);
 	return 0;
 }
 
@@ -4470,10 +4272,7 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg
 
 	min_off = reg_smin(reg) + off;
 	max_off = reg_smax(reg) + off;
-	err = mark_reg_stack_read(env, ptr_state, min_off, max_off + size,
-				  dst_regno);
-	if (err)
-		return err;
+	mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno);
 	return 0;
 }
 
@@ -4661,8 +4460,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
 	}
 	bpf_diag_mod_end(env);
-	return bpf_push_jmp_history(env, env->cur_state,
-				    INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
+	return 0;
 }
 
 /*
@@ -4700,26 +4498,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
 	bpf_diag_mod_begin(env, &cur->regs[dst_regno], arg, BPF_DIAG_MOD_WRITE);
 	cur->regs[dst_regno] = *arg;
 	bpf_diag_mod_end(env);
-	return bpf_push_jmp_history(env, env->cur_state,
-				    INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
-}
-
-static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
-{
-	struct bpf_func_state *caller = cur_func(env);
-	int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS;
-
-	bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi);
-	return mark_chain_precision_batch(env, env->cur_state);
-}
-
-static int mark_arg_precision(struct bpf_verifier_env *env, argno_t argno)
-{
-	int regno = reg_from_argno(argno);
-
-	if (regno >= 0)
-		return mark_chain_precision(env, regno);
-	return mark_stack_arg_precision(env, arg_idx_from_argno(argno));
+	return 0;
 }
 
 static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller,
@@ -5184,14 +4963,8 @@ static int check_map_kptr_access(struct bpf_verifier_env *env,
 			return ret;
 	} else if (class == BPF_STX) {
 		val_reg = reg_state(env, value_regno);
-		if (bpf_register_is_null(val_reg)) {
-			/*
-			 * This store is valid only because the scalar is known to be
-			 * zero. Mark it precise so another scalar cannot be pruned
-			 * against this state.
-			 */
-			return mark_chain_precision(env, value_regno);
-		}
+		if (bpf_register_is_null(val_reg))
+			return 0;
 		if (map_kptr_match_type(env, kptr_field, val_reg, value_regno))
 			return -EACCES;
 	} else if (class == BPF_ST) {
@@ -8264,7 +8037,7 @@ static int check_stack_range_initialized(
 		    (ss->spilled_ptr.type == SCALAR_VALUE ||
 		     env->allow_ptr_leaks)) {
 			if (clobber) {
-				__mark_reg_unknown(env, &ss->spilled_ptr);
+				bpf_mark_reg_unknown(&ss->spilled_ptr);
 				for (j = 0; j < BPF_REG_SIZE; j++)
 					scrub_spilled_slot(&ss->slot_type[j]);
 			}
@@ -8462,9 +8235,6 @@ static int check_mem_size_reg(struct bpf_verifier_env *env,
 	if (err && failure)
 		*failure = BPF_MEM_SIZE_FAIL_MEMORY;
 
-	if (!err)
-		err = mark_arg_precision(env, size_argno);
-
 	return err;
 
 size_error:
@@ -8485,7 +8255,7 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
 	int size, err = 0;
 
 	if (bpf_register_is_null(reg))
-		return mark_arg_precision(env, argno);
+		return 0;
 	if (known_memory)
 		*known_memory = true;
 
@@ -8515,9 +8285,6 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
 static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 					argno_t argno, struct ret_mem_desc *ret_mem)
 {
-	int regno = reg_from_argno(argno);
-	int err;
-
 	if (ret_mem->found) {
 		verifier_bug(env, "only one allocation size argument permitted");
 		return -EFAULT;
@@ -8533,13 +8300,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
 		return -EINVAL;
 	}
 
-	if (regno >= 0)
-		err = mark_chain_precision(env, regno);
-	else
-		err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
-	if (err)
-		return err;
-
 	ret_mem->size = reg->var_off.value;
 	ret_mem->found = true;
 
@@ -8549,9 +8309,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
 static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 			     argno_t argno, struct bpf_call_arg_meta *meta)
 {
-	int regno = reg_from_argno(argno);
-	int err;
-
 	if (meta->arg_constant.found) {
 		verifier_bug(env, "only one constant argument permitted");
 		return -EFAULT;
@@ -8562,13 +8319,6 @@ static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state
 		return -EINVAL;
 	}
 
-	if (regno >= 0)
-		err = mark_chain_precision(env, regno);
-	else
-		err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
-	if (err < 0)
-		return err;
-
 	meta->arg_constant.found = true;
 	meta->arg_constant.value = reg->var_off.value;
 
@@ -9311,7 +9061,7 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 
 	/* mark current iter state as drained and assume returned NULL */
 	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
-	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
+	__mark_reg_const_zero(&cur_fr->regs[BPF_REG_0]);
 
 	return 0;
 }
@@ -9998,7 +9748,7 @@ static int get_max_map_key(struct bpf_verifier_env *env,
 	int spill_size = 0;
 	int zero_size = 0;
 	int stack_off;
-	int i, err;
+	int i;
 	u8 *stype;
 
 	if (!env->bpf_capable)
@@ -10040,15 +9790,6 @@ static int get_max_map_key(struct bpf_verifier_env *env,
 		/* The lookup can return NULL: nothing depends on the key */
 		return -EOPNOTSUPP;
 
-	/*
-	 * We are relying on the range of the key. So mark as precise
-	 * to prevent pruning on it.
-	 */
-	bpf_bt_set_frame_slot(&env->bt, key->frameno, spi);
-	err = mark_chain_precision_batch(env, env->cur_state);
-	if (err < 0)
-		return err;
-
 	*value = reg_umax(reg);
 	return 0;
 }
@@ -10166,12 +9907,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, u32 slot, u32 p
 	if (arg_type_is_raw_mem(arg_type))
 		meta->arg_raw_mem.mask |= BIT(slot);
 
-	if (bpf_register_is_null(reg) && type_may_be_null(arg_type)) {
-		err = mark_arg_precision(env, argno);
-		if (err)
-			return err;
+	if (bpf_register_is_null(reg) && type_may_be_null(arg_type))
 		return 0;
-	}
 
 	err = check_func_arg_nullability(env, reg, arg, argno, arg_type, meta, insn_idx);
 	if (err)
@@ -12869,11 +12606,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EINVAL;
 		}
 
-		/* we are going to rely on register's precise value */
-		err = mark_chain_precision(env, BPF_REG_0);
-		if (err)
-			return err;
-
 		/* enforce R0 return value range, and bpf_callback_t returns 64bit */
 		if (!retval_range_within(callee->callback_ret_range, r0)) {
 			verbose_invalid_scalar(env, r0, callee->callback_ret_range,
@@ -13039,7 +12771,6 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 	struct bpf_reg_state *reg;
 	struct bpf_map *map = meta->map.ptr;
 	u64 val, max;
-	int err;
 
 	if (func_id != BPF_FUNC_tail_call)
 		return 0;
@@ -13057,9 +12788,6 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		return 0;
 	}
 
-	err = mark_chain_precision(env, BPF_REG_3);
-	if (err)
-		return err;
 	if (bpf_map_key_unseen(aux))
 		bpf_map_key_store(aux, val);
 	else if (!bpf_map_key_poisoned(aux) &&
@@ -13202,45 +12930,27 @@ static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *env)
 	return &env->insn_aux_data[env->insn_idx];
 }
 
-/* Returns 1 if R4 is a known zero, 0 if it is not, a negative errno on error. */
-static int loop_flag_is_zero(struct bpf_verifier_env *env)
+static bool loop_flag_is_zero(struct bpf_verifier_env *env)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_4);
-	int err;
-
-	if (!bpf_register_is_null(reg))
-		return 0;
-
-	err = mark_chain_precision(env, BPF_REG_4);
-	if (err)
-		return err;
-	return 1;
+	return bpf_register_is_null(reg_state(env, BPF_REG_4));
 }
 
-static int update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno)
+static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno)
 {
 	struct bpf_loop_inline_state *state = &cur_aux(env)->loop_inline_state;
-	int flag_is_zero;
 
 	if (!state->initialized) {
-		flag_is_zero = loop_flag_is_zero(env);
-		if (flag_is_zero < 0)
-			return flag_is_zero;
 		state->initialized = 1;
-		state->fit_for_inline = flag_is_zero;
+		state->fit_for_inline = loop_flag_is_zero(env);
 		state->callback_subprogno = subprogno;
-		return 0;
+		return;
 	}
 
 	if (!state->fit_for_inline)
-		return 0;
+		return;
 
-	flag_is_zero = loop_flag_is_zero(env);
-	if (flag_is_zero < 0)
-		return flag_is_zero;
-	state->fit_for_inline = (flag_is_zero &&
+	state->fit_for_inline = (loop_flag_is_zero(env) &&
 				 state->callback_subprogno == subprogno);
-	return 0;
 }
 
 /* Returns whether or not the given map can potentially elide
@@ -13459,9 +13169,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 			verbose(env, "get_local_storage() doesn't support non-zero flags\n");
 			return -EINVAL;
 		}
-		err = mark_chain_precision(env, BPF_REG_2);
-		if (err)
-			return err;
 		break;
 	case BPF_FUNC_for_each_map_elem:
 		err = push_callback_call(env, insn, insn_idx, meta.subprogno,
@@ -13479,15 +13186,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		err = check_bpf_snprintf_call(env, regs);
 		break;
 	case BPF_FUNC_loop:
-		err = update_loop_inline_state(env, meta.subprogno);
-		if (err)
-			return err;
-		/* Verifier relies on R1 value to determine if bpf_loop() iteration
-		 * is finished, thus mark it precise.
-		 */
-		err = mark_chain_precision(env, BPF_REG_1);
-		if (err)
-			return err;
+		update_loop_inline_state(env, meta.subprogno);
 		if (cur_func(env)->callback_depth < reg_umax(&regs[BPF_REG_1])) {
 			err = push_callback_call(env, insn, insn_idx, meta.subprogno,
 						 set_loop_callback_state);
@@ -13536,10 +13235,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 			bpf_lsm_get_retval_range(env->prog, &range);
 		}
 
-		err = mark_chain_precision(env, BPF_REG_1);
-		if (err)
-			return err;
-
 		if (!retval_range_within(range, r1)) {
 			verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1");
 			return -EINVAL;
@@ -16540,7 +16235,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		ret_nregs = mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
-			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+			__mark_reg_const_zero(&regs[BPF_REG_0]);
 	} else if (btf_type_is_struct(t)) {
 		struct btf_member_path path = {};
 		const char *member_note = "";
@@ -17171,14 +16866,14 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 		/* Taint dst register if offset had invalid bounds derived from
 		 * e.g. dead branches.
 		 */
-		__mark_reg_unknown(env, dst_reg);
+		bpf_mark_reg_unknown(dst_reg);
 		return 0;
 	}
 
 	if (BPF_CLASS(insn->code) != BPF_ALU64 && !to_scalar) {
 		/* 32-bit ALU ops on pointers produce (meaningless) scalars */
 		if (opcode == BPF_SUB && env->allow_ptr_leaks) {
-			__mark_reg_unknown(env, dst_reg);
+			bpf_mark_reg_unknown(dst_reg);
 			return 0;
 		}
 
@@ -17242,7 +16937,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 	}
 
 	if (to_scalar) {
-		__mark_reg_unknown(env, dst_reg);
+		bpf_mark_reg_unknown(dst_reg);
 		return 0;
 	}
 
@@ -18214,7 +17909,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 	int ret;
 
 	if (!is_safe_to_compute_dst_reg_range(insn, &src_reg)) {
-		__mark_reg_unknown(env, dst_reg);
+		bpf_mark_reg_unknown(dst_reg);
 		return 0;
 	}
 
@@ -18273,7 +17968,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 		if (bpf_insn_is_hmul(insn) &&
 		    (!is_reg_const(dst_reg, false) ||
 		     !is_reg_const(&src_reg, false))) {
-			__mark_reg_unknown(env, dst_reg);
+			bpf_mark_reg_unknown(dst_reg);
 			break;
 		}
 		if (!alu32 && insn->off == BPF_UHMUL) {
@@ -18485,24 +18180,13 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
 				 * This is legal, but we have to reverse our
 				 * src/dest handling in computing the range
 				 */
-				err = mark_chain_precision(env, insn->dst_reg);
-				if (err)
-					return err;
 				off_reg = *dst_reg;
 				return adjust_ptr_min_max_vals(env, insn, insn->src_reg, src_reg,
 							       &off_reg);
 			}
 		} else if (ptr_reg) {
 			/* pointer += scalar */
-			err = mark_chain_precision(env, insn->src_reg);
-			if (err)
-				return err;
 			return adjust_ptr_min_max_vals(env, insn, insn->dst_reg, dst_reg, src_reg);
-		} else if (dst_reg->precise) {
-			/* if dst_reg is precise, src_reg should be precise as well */
-			err = mark_chain_precision(env, insn->src_reg);
-			if (err)
-				return err;
 		}
 	} else {
 		/* Pretend the src is a reg with a known value, since we only
@@ -19891,7 +19575,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	struct bpf_reg_state *eq_branch_regs;
 	struct linked_regs linked_regs = {};
 	u8 opcode = BPF_OP(insn->code);
-	int insn_flags = 0;
 	bool is_jmp32;
 	int pred = -1;
 	int err;
@@ -19952,24 +19635,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 			return -EACCES;
 		}
 
-		if (src_reg->type == PTR_TO_STACK)
-			insn_flags |= INSN_F_SRC_REG_STACK;
-		if (dst_reg->type == PTR_TO_STACK)
-			insn_flags |= INSN_F_DST_REG_STACK;
 	} else {
 		src_reg = &env->fake_reg[0];
 		memset(src_reg, 0, sizeof(*src_reg));
 		src_reg->type = SCALAR_VALUE;
 		__mark_reg_known(src_reg, insn->imm);
-
-		if (dst_reg->type == PTR_TO_STACK)
-			insn_flags |= INSN_F_DST_REG_STACK;
-	}
-
-	if (insn_flags) {
-		err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);
-		if (err)
-			return err;
 	}
 
 	/*
@@ -19990,19 +19660,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	pred = is_branch_taken(env, dst_reg, src_reg, opcode, is_jmp32);
 	if (pred >= 0)
 		mark_loop_exit_decided(env, this_branch);
-	if (pred >= 0) {
-		/* If we get here with a dst_reg pointer type it is because
-		 * above is_branch_taken() special cased the 0 comparison.
-		 */
-		if (!__is_pointer_value(false, dst_reg))
-			err = mark_chain_precision(env, insn->dst_reg);
-		if (BPF_SRC(insn->code) == BPF_X && !err &&
-		    !__is_pointer_value(false, src_reg))
-			err = mark_chain_precision(env, insn->src_reg);
-		if (err)
-			return err;
-	}
-
 	if (pred == 1) {
 		/* Only follow the goto, ignore fall-through. If needed, push
 		 * the fall-through branch for simulation under speculative
@@ -20033,20 +19690,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 		return 0;
 	}
 
-	/* Push scalar registers sharing same ID to jump history,
-	 * do this before creating 'other_branch', so that both
-	 * 'this_branch' and 'other_branch' share this history
-	 * if parent state is created.
-	 */
-	if (linked_regs.cnt > 1) {
-		u16 packed[LINKED_REGS_MAX];
-
-		linked_regs_pack(&linked_regs, packed);
-		err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);
-		if (err)
-			return err;
-	}
-
 	other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx, false);
 	if (IS_ERR(other_branch))
 		return PTR_ERR(other_branch);
@@ -20124,15 +19767,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	    type_may_be_null(dst_reg->type) &&
 	    ((BPF_SRC(insn->code) == BPF_K && insn->imm == 0) ||
 	     (BPF_SRC(insn->code) == BPF_X && bpf_register_is_null(src_reg)))) {
-		/*
-		 * For BPF_X the zero is a property of this execution path,
-		 * hence src_reg has to be precise.
-		 */
-		if (BPF_SRC(insn->code) == BPF_X) {
-			err = mark_chain_precision(env, insn->src_reg);
-			if (err)
-				return err;
-		}
 		/* Mark all identical registers in each branch as either
 		 * safe or unknown depending R == 0 or R != 0 conditional.
 		 */
@@ -20239,7 +19873,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
 	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
 		if (map->map_type == BPF_MAP_TYPE_ARENA) {
-			__mark_reg_unknown(env, dst_reg);
+			bpf_mark_reg_unknown(dst_reg);
 			dst_reg->map_ptr = map;
 			bpf_diag_mod_end(env);
 			return 0;
@@ -20588,10 +20222,6 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 		return -EINVAL;
 	}
 
-	err = mark_chain_precision(env, regno);
-	if (err)
-		return err;
-
 	if (!retval_range_within(range, reg)) {
 		verbose_invalid_scalar(env, reg, range, exit_ctx, reg_name);
 		if (prog->expected_attach_type == BPF_LSM_CGROUP &&
@@ -21940,9 +21570,9 @@ static int push_budget_exit(struct bpf_verifier_env *env)
 		return PTR_ERR(branch);
 	branch->unwound = sub->bail_unwinds;
 	regs = branch->frame[branch->curframe]->regs;
-	__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	__mark_reg_const_zero(&regs[BPF_REG_0]);
 	if (bpf_ret_reg_pair(env, cur_func(env)->subprogno))
-		__mark_reg_const_zero(env, &regs[BPF_REG_2]);
+		__mark_reg_const_zero(&regs[BPF_REG_2]);
 	return 0;
 }
 
@@ -23374,7 +23004,6 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 			BPF_MAIN_FUNC /* callsite */,
 			0 /* frameno */,
 			subprog);
-	state->first_insn_idx = env->subprog_info[subprog].start;
 	state->last_insn_idx = -1;
 
 	regs = state->frame[state->curframe]->regs;
@@ -23694,11 +23323,12 @@ static void print_verification_stats(struct bpf_verifier_env *env)
 				env->subprog_info[i].stack_depth);
 		}
 	}
+	/* user space parses this line. There are no read marks. */
 	verbose(env, "processed %d insns (limit %d) max_states_per_insn %d "
-		"total_states %d peak_states %d mark_read %d\n",
+		"total_states %d peak_states %d mark_read 0\n",
 		env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS,
 		env->max_states_per_insn, env->total_states,
-		env->peak_states, env->longest_mark_read_walk);
+		env->peak_states);
 }
 
 int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
@@ -25500,7 +25130,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (!env)
 		return -ENOMEM;
 
-	env->bt.env = env;
 	env->prog = *prog;
 	env->ops = bpf_verifier_ops[env->prog->type];
 	env->stack_limit = bpf_prog_stack_limit(env->prog);
-- 
2.55.0

Re: [PATCH bpf-next v3 24/45] bpf: Remove precision tracking

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

> bpf: Remove precision tracking

TL;DR: Nit only: in check_stack_read_fixed_off(), the first two branches are
now identical after insn_flags removal, so the if/else chain could possibly
collapse into one mark_reg_stack_read() call.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -4356,18 +4174,13 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
>
>  				if (spill_cnt == size &&
>  				    tnum_is_const(reg->var_off) && reg->var_off.value == 0) {
> -					__mark_reg_const_zero(env, &state->regs[dst_regno]);
> -					/* this IS register fill, so keep insn_flags */
> +					__mark_reg_const_zero(&state->regs[dst_regno]);
>  				} else if (zero_cnt == size) {
>  					/* similarly to mark_reg_stack_read(), preserve zeroes */
> -					__mark_reg_const_zero(env, &state->regs[dst_regno]);
> -					insn_flags = 0; /* not restoring original register state */
> +					__mark_reg_const_zero(&state->regs[dst_regno]);
>  				} else {
> -					err = mark_reg_stack_read(env, reg_state, off, off + size,
> -								  dst_regno);
> -					if (err)
> -						return err;
> -					insn_flags = 0; /* not restoring original register state */
> +					mark_reg_stack_read(env, reg_state, off, off + size,
> +							    dst_regno);
>  				}

This isn't a bug, but now that insn_flags is gone, the first two branches
have the same body, and mark_reg_stack_read() already produces a const
zero in both of those cases.

Could the if/else chain be replaced by a single call to
mark_reg_stack_read(env, reg_state, off, off + size, dst_regno)?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 25/45] bpf: Remove prune points, levels of state comparison and pop_stack()

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The old main pass followed each path of a prog depth first and compared
states at selected insns to prune paths. flow.c goes from basic block to
basic block. Three more parts of the old pass have no user.

  removed              task in the old pass       what does it now
  -------------------  -------------------------  -------------------------
  prune_point,         Select the insns at which  States meet at the start
  jmp_point,           the pass saves and         of each basic block.
  force_checkpoint,    compares states.
  is_iter_next
  enum exact_level     Select one of three ways   bpf_state_subsumes() has
                       to compare states.         one way, the way of
                                                  RANGE_WITHIN.
  pop_stack()          Get the next state when    flow_drain() empties the
                       a path ends.               stack after each insn.

NOT_EXACT ignored scalars without a precision mark, and the marks are gone.
EXACT accepted only identical states, for "infinite loop detected", which
is gone too. What RANGE_WITHIN did stays: the range of each live scalar of
the new state is within its range in the old state.

push_stack() stays. The code that verifies an insn uses it to make a second
state, for example for the other branch of a jump. flow_drain() gives these
states to flow.c.

What user space sees:
 - The statistics line shows "processed %d insns (limit 1000000)". The
   verifier limits the number of states, not of processed insns, and
   BPF_COMPLEXITY_LIMIT_INSNS is only the maximum length of a prog.
   veristat and other tools parse the line. Keep the format, print 0.
 - The report of a register that is not readable said
   "R1 has never been initialized on this path". It needs the history of
   the whole path, and the verifier has that of the current basic block.
   The report says that the register may never have been initialized, or
   that an earlier operation may have invalidated it.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf.h          |  4 +++
 include/linux/bpf_verifier.h | 38 --------------------
 kernel/bpf/cfg.c             | 56 ++---------------------------
 kernel/bpf/diagnostics.c     | 13 -------
 kernel/bpf/diagnostics.h     |  1 -
 kernel/bpf/scev.c            |  4 ---
 kernel/bpf/states.c          | 70 ++++++++----------------------------
 kernel/bpf/verifier.c        | 61 ++++++++-----------------------
 8 files changed, 36 insertions(+), 211 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index cdf3c9b90987..520fd72e21fd 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2633,6 +2633,10 @@ struct bpf_array {
  */
 int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
 
+/*
+ * the longest program. The verifier limits the number of states, not of
+ * processed insns, see BPF_FLOW_MAX_STATES.
+ */
 #define BPF_COMPLEXITY_LIMIT_INSNS      1000000 /* yes. 1M insns */
 #define MAX_TAIL_CALL_CNT 33
 
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index e932c480feb2..9439c8f90abd 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -799,7 +799,6 @@ struct bpf_insn_aux_data {
 	u64 needs_zext:1; /* alu op needs to clear upper bits */
 	u64 prevent_zext:1; /* alu op cannot be zext (already used with 64-bit scalars) */
 	u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
-	u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
 	u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	u64 cast_arena:1; /* addr_space_cast of an address in arena */
 	u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
@@ -830,12 +829,6 @@ struct bpf_insn_aux_data {
 	u64 non_arena_load:1;
 
 	/* below flags are initialized once */
-	u64 jmp_point:1;
-	u64 prune_point:1;
-	/* ensure we check state equivalence and save state checkpoint and
-	 * this instruction, regardless of any heuristics
-	 */
-	u64 force_checkpoint:1;
 	/* true if instruction is a call to a helper function that
 	 * accepts callback function as a parameter.
 	 */
@@ -1166,7 +1159,6 @@ struct bpf_verifier_env {
 	const struct bpf_verifier_ops *ops;
 	struct module *attach_btf_mod;	/* The owner module of prog->aux->attach_btf */
 	struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
-	int stack_size;			/* number of states to be processed */
 	bool strict_alignment;		/* perform strict pointer alignment checks */
 	bool test_reg_invariants;	/* fail verification on register invariants violations */
 	bool uses_arena_sp;		/* the program has insns with BPF_REG_ARENA_SP */
@@ -1384,26 +1376,6 @@ void bpf_verbose_snum32(struct bpf_verifier_env *env, s32 num);
 		bpf_log(&env->log, "verifier bug: " fmt "\n", ##args);				\
 	})
 
-static inline void mark_prune_point(struct bpf_verifier_env *env, int idx)
-{
-	env->insn_aux_data[idx].prune_point = true;
-}
-
-static inline bool bpf_is_prune_point(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].prune_point;
-}
-
-static inline void mark_force_checkpoint(struct bpf_verifier_env *env, int idx)
-{
-	env->insn_aux_data[idx].force_checkpoint = true;
-}
-
-static inline bool bpf_is_force_checkpoint(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].force_checkpoint;
-}
-
 static inline void clear_scalar_id(struct bpf_reg_state *reg)
 {
 	reg->id = 0;
@@ -1420,11 +1392,6 @@ static inline bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx
 	return env->insn_aux_data[insn_idx].calls_callback;
 }
 
-static inline void mark_jmp_point(struct bpf_verifier_env *env, int idx)
-{
-	env->insn_aux_data[idx].jmp_point = true;
-}
-
 static inline void mark_jump_target(struct bpf_verifier_env *env, int idx)
 {
 	env->insn_aux_data[idx].jump_target = true;
@@ -1597,11 +1564,6 @@ static inline u32 bpf_frame_insn_idx(struct bpf_verifier_state *st, u32 frame)
 	       : st->frame[frame + 1]->callsite;
 }
 
-static inline bool bpf_is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].jmp_point;
-}
-
 static inline bool bpf_is_spilled_reg(const struct bpf_stack_state *stack)
 {
 	return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 94140d1fe1c3..bbcf42eae9d9 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -130,12 +130,8 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 		return -EINVAL;
 	}
 
-	if (e == BRANCH) {
-		/* mark branch target for state pruning */
-		mark_prune_point(env, w);
-		mark_jmp_point(env, w);
+	if (e == BRANCH)
 		mark_jump_target(env, w);
-	}
 
 	if (insn_state[w] == 0) {
 		/* tree-edge */
@@ -179,13 +175,8 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
 	if (ret)
 		return ret;
 
-	mark_prune_point(env, t + insn_sz);
-	/* when we exit from subprog, we need to record non-linear history */
-	mark_jmp_point(env, t + insn_sz);
-
 	if (visit_callee) {
 		w = t + insns[t].imm + 1;
-		mark_prune_point(env, t);
 		merge_callee_effects(env, t, w);
 		ret = push_insn(t, w, BRANCH, env);
 	}
@@ -446,14 +437,12 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
 		return -E2BIG;
 	}
 
-	mark_prune_point(env, t);
 	for (i = 0; i < jt->cnt; i++) {
 		w = jt->items[i];
 		if (verifier_bug_if(w < 0 || w >= env->prog->len, env,
 				    "indirect jump from insn %d to %d out of range", t, w))
 			return -EFAULT;
 
-		mark_jmp_point(env, w);
 		mark_jump_target(env, w);
 
 		/* EXPLORED || DISCOVERED */
@@ -528,14 +517,11 @@ static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
 	}
 
 	/* all functions are explored, their effects are known */
-	mark_prune_point(env, t);
 	for (i = 0; i < cnt; i++) {
 		w = ptrs[i].xlated_off;
 		merge_callee_effects(env, t, w);
 
-		/* the same marks as push_insn() leaves on a branch target */
-		mark_prune_point(env, w);
-		mark_jmp_point(env, w);
+		/* the same mark as push_insn() leaves on a branch target */
 		mark_jump_target(env, w);
 	}
 	return DONE_EXPLORING;
@@ -599,28 +585,13 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 		return DONE_EXPLORING;
 
 	case BPF_CALL:
-		if (bpf_is_async_callback_calling_insn(insn))
-			/* Mark this call insn as a prune point to trigger
-			 * is_state_visited() check before call itself is
-			 * processed by __check_func_call(). Otherwise new
-			 * async state will be pushed for further exploration.
-			 */
-			mark_prune_point(env, t);
 		/* For functions that invoke callbacks it is not known how many times
 		 * callback would be called. Verifier models callback calling functions
 		 * by repeatedly visiting callback bodies and returning to origin call
 		 * instruction.
-		 * In order to stop such iteration verifier needs to identify when a
-		 * state identical some state from a previous iteration is reached.
-		 * Check below forces creation of checkpoint before callback calling
-		 * instruction to allow search for such identical states.
 		 */
-		if (bpf_is_sync_callback_calling_insn(insn)) {
+		if (bpf_is_sync_callback_calling_insn(insn))
 			mark_calls_callback(env, t);
-			mark_force_checkpoint(env, t);
-			mark_prune_point(env, t);
-			mark_jmp_point(env, t);
-		}
 		if (bpf_helper_call(insn)) {
 			const struct bpf_func_proto *fp;
 
@@ -640,21 +611,6 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 			struct bpf_call_arg_meta meta;
 
 			ret = bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta);
-			if (ret == 0 && bpf_is_iter_next_kfunc(&meta)) {
-				mark_prune_point(env, t);
-				/* Checking and saving state checkpoints at iter_next() call
-				 * is crucial for fast convergence of open-coded iterator loop
-				 * logic, so we need to force it. If we don't do that,
-				 * is_state_visited() might skip saving a checkpoint, causing
-				 * unnecessarily long sequence of not checkpointed
-				 * instructions and jumps, leading to exhaustion of jump
-				 * history buffer, and potentially other undesired outcomes.
-				 * It is expected that with correct open-coded iterators
-				 * convergence will happen quickly, so we don't run a risk of
-				 * exhausting memory.
-				 */
-				mark_force_checkpoint(env, t);
-			}
 			/* Same as helpers, if called in a non-sleepable context
 			 * program will be rejected anyway, so we should end up
 			 * with precise sleepable marks on subprogs, except for
@@ -683,18 +639,12 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 		if (ret)
 			return ret;
 
-		mark_prune_point(env, t + off + 1);
-		mark_jmp_point(env, t + off + 1);
 		mark_jump_target(env, t + off + 1);
 
 		return ret;
 
 	default:
 		/* conditional jump with two edges */
-		mark_prune_point(env, t);
-		if (bpf_is_may_goto_insn(insn))
-			mark_force_checkpoint(env, t);
-
 		ret = push_insn(t, t + 1, FALLTHROUGH, env);
 		if (ret)
 			return ret;
diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
index 08e12714bb95..3de3e1486bbd 100644
--- a/kernel/bpf/diagnostics.c
+++ b/kernel/bpf/diagnostics.c
@@ -392,13 +392,6 @@ static u32 log_pos(const struct bpf_diag_log *log, u32 idx)
 	return pos < log->cap ? pos : pos - log->cap;
 }
 
-u64 bpf_diag_event_log_save(struct bpf_verifier_env *env)
-{
-	struct bpf_diag *diag = env->diag;
-
-	return diag ? log_end(&diag->log) : 0;
-}
-
 void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos)
 {
 	struct bpf_diag *diag = env->diag;
@@ -1357,10 +1350,6 @@ void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int reg
 		diag_reason(
 			env, "R%d is not readable here. A previous operation invalidated this register, so the verifier cannot use it as an input.",
 			regno);
-	else if (log && !log->first_seq)
-		diag_reason(env,
-			    "R%d has never been initialized on this path, so the verifier cannot use it as an input.",
-			    regno);
 	else
 		diag_reason(
 			env, "R%d is not readable here. It may never have been initialized, or an earlier operation may have invalidated it.",
@@ -1375,8 +1364,6 @@ void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int reg
 	if (invalidated)
 		diag_suggestion(
 			env, "Avoid using the register after it is invalidated, or initialize it again before this instruction.");
-	else if (log && !log->first_seq)
-		diag_suggestion(env, "Initialize R%d on every path before this instruction.", regno);
 	else
 		diag_suggestion(
 			env, "Initialize the register on every path, or initialize it again after any operation that invalidates it.");
diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h
index 490decc133f6..e155d2588a51 100644
--- a/kernel/bpf/diagnostics.h
+++ b/kernel/bpf/diagnostics.h
@@ -53,7 +53,6 @@ const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __p
 const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id);
 const char *bpf_diag_reg_type_plain(struct bpf_verifier_env *env, enum bpf_reg_type type);
 const char *bpf_diag_arg_type_plain(enum bpf_arg_type type);
-u64 bpf_diag_event_log_save(struct bpf_verifier_env *env);
 void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos);
 u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state);
 void bpf_diag_free(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index ebdf7d148c85..147a2fa9aea9 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1803,10 +1803,6 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
 						 find_header_env(scev, header));
 	}
 
-	for (i = 0; i < env->prog->len; i++)
-		if (aux[i].loop)
-			aux[i].prune_point = true;
-
 	if (env->log.level & BPF_LOG_LEVEL2) {
 		err = log_scevs(env);
 		if (err)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 5935e16730fd..fb7998654e16 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -198,8 +198,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
 	}
 }
 
-static int clean_verifier_state(struct bpf_verifier_env *env,
-				 struct bpf_verifier_state *st)
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
 {
 	int i, err;
 
@@ -225,20 +224,10 @@ static bool regs_exact(const struct bpf_reg_state *rold,
 	       check_ids(rold->map_uid, rcur->map_uid, idmap);
 }
 
-enum exact_level {
-	NOT_EXACT,
-	EXACT,
-	RANGE_WITHIN
-};
-
 /* Returns true if (rold safe implies rcur safe) */
 static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
-		    struct bpf_reg_state *rcur, struct bpf_idmap *idmap,
-		    enum exact_level exact)
+		    struct bpf_reg_state *rcur, struct bpf_idmap *idmap)
 {
-	if (exact == EXACT)
-		return regs_exact(rold, rcur, idmap);
-
 	if (rold->type == NOT_INIT)
 		/* explored state can't have used this */
 		return true;
@@ -427,8 +416,7 @@ static struct bpf_reg_state *scalar_reg_for_stack(struct bpf_verifier_env *env,
 }
 
 static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
-		      struct bpf_func_state *cur, struct bpf_idmap *idmap,
-		      enum exact_level exact)
+		      struct bpf_func_state *cur, struct bpf_idmap *idmap)
 {
 	int i, spi;
 
@@ -447,18 +435,6 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		old_type = old_slot->slot_type[im];
 		cur_slot = i < cur->allocated_stack ? bpf_stack_slot(cur, spi) : NULL;
 
-		if (exact == EXACT) {
-			u8 cur_type = cur_slot ? cur_slot->slot_type[im] : STACK_INVALID;
-
-			/* STACK_INVALID and STACK_POISON are equivalent for pruning */
-			if (old_type == STACK_POISON)
-				old_type = STACK_INVALID;
-			if (cur_type == STACK_POISON)
-				cur_type = STACK_INVALID;
-			if (!cur_slot || old_type != cur_type)
-				return false;
-		}
-
 		if (old_type == STACK_INVALID || old_type == STACK_POISON)
 			continue;
 
@@ -481,7 +457,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			old_reg = scalar_reg_for_stack(env, old_slot, im);
 			cur_reg = scalar_reg_for_stack(env, cur_slot, im);
 			if (old_reg && cur_reg) {
-				if (!regsafe(env, old_reg, cur_reg, idmap, exact))
+				if (!regsafe(env, old_reg, cur_reg, idmap))
 					return false;
 				i += (im == 0 ? BPF_REG_SIZE - 1 : 3);
 				continue;
@@ -516,8 +492,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			 * such verifier states are not equivalent.
 			 * return false to continue verification of this path
 			 */
-			if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr,
-				     idmap, exact))
+			if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr, idmap))
 				return false;
 			break;
 		case STACK_DYNPTR:
@@ -565,8 +540,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
  * Outgoing stack args are path-local state and must agree for pruning.
  */
 static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
-			   struct bpf_func_state *cur, struct bpf_idmap *idmap,
-			   enum exact_level exact)
+			   struct bpf_func_state *cur, struct bpf_idmap *idmap)
 {
 	int i, nslots, cur_cnt = cur->out_stack_arg_cnt;
 
@@ -582,7 +556,7 @@ static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *
 			  &old->stack_arg_regs[i] : &not_init;
 		cur_arg = i < cur_cnt ?
 			  &cur->stack_arg_regs[i] : &not_init;
-		if (!regsafe(env, old_arg, cur_arg, idmap, exact))
+		if (!regsafe(env, old_arg, cur_arg, idmap))
 			return false;
 	}
 
@@ -710,7 +684,7 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
  * the current state will reach 'bpf_exit' instruction safely
  */
 static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
-			      struct bpf_func_state *cur, u32 insn_idx, enum exact_level exact)
+			      struct bpf_func_state *cur, u32 insn_idx)
 {
 	u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
 	u16 i;
@@ -736,14 +710,13 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (((1 << i) & live_regs) &&
-		    !regsafe(env, &old->regs[i], &cur->regs[i],
-			     &env->idmap_scratch, exact))
+		    !regsafe(env, &old->regs[i], &cur->regs[i], &env->idmap_scratch))
 			return false;
 
-	if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
+	if (!stacksafe(env, old, cur, &env->idmap_scratch))
 		return false;
 
-	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
+	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch))
 		return false;
 
 	if (!loop_stack_safe(env, old, cur))
@@ -760,10 +733,9 @@ static void reset_idmap_scratch(struct bpf_verifier_env *env)
 	idmap->cnt = 0;
 }
 
-static bool states_equal(struct bpf_verifier_env *env,
-			 struct bpf_verifier_state *old,
-			 struct bpf_verifier_state *cur,
-			 enum exact_level exact)
+/* Everything that @cur describes is described by @old. */
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+			struct bpf_verifier_state *cur)
 {
 	u32 insn_idx;
 	int i;
@@ -798,21 +770,9 @@ static bool states_equal(struct bpf_verifier_env *env,
 		insn_idx = bpf_frame_insn_idx(old, i);
 		if (old->frame[i]->callsite != cur->frame[i]->callsite)
 			return false;
-		if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx, exact))
+		if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx))
 			return false;
 	}
 
 	return true;
 }
-
-int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	return clean_verifier_state(env, st);
-}
-
-/* Everything that @cur describes is described by @old. */
-bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
-			struct bpf_verifier_state *cur)
-{
-	return states_equal(env, old, cur, RANGE_WITHIN);
-}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fb195d20977c..285e459ef741 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -185,7 +185,10 @@ static bool bpf_global_percpu_ma_set;
  * reference remains at the end of the program, the verifier rejects it.
  */
 
-/* verifier_state + insn_idx are pushed to stack when branch is encountered */
+/*
+ * verifier_state + insn_idx are pushed to stack when branch is encountered.
+ * It's emptied after every insn, see flow_drain().
+ */
 struct bpf_verifier_stack_elem {
 	/* verifier state is 'st'
 	 * before processing instruction 'insn_idx'
@@ -197,9 +200,6 @@ struct bpf_verifier_stack_elem {
 	/* it goes there only if may_goto in front of prev_insn_idx does, see check_loop_guard() */
 	enum bpf_guard_edge guard_edge;
 	struct bpf_verifier_stack_elem *next;
-	/* length of verifier log at the time this state was pushed on stack */
-	u32 log_pos;
-	u64 diag_log_pos;
 };
 
 #define BPF_GLOBAL_PERCPU_MA_MAX_SIZE  512
@@ -1816,36 +1816,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	return 0;
 }
 
-static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
-		     int *insn_idx, bool pop_log)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_verifier_stack_elem *elem, *head = env->head;
-	int err;
-
-	if (env->head == NULL)
-		return -ENOENT;
-
-	if (cur) {
-		err = bpf_copy_verifier_state(cur, &head->st);
-		if (err)
-			return err;
-		bpf_diag_event_log_restore(env, head->diag_log_pos);
-	}
-	if (pop_log)
-		bpf_vlog_reset(&env->log, head->log_pos);
-	if (insn_idx)
-		*insn_idx = head->insn_idx;
-	if (prev_insn_idx)
-		*prev_insn_idx = head->prev_insn_idx;
-	elem = head->next;
-	bpf_free_verifier_state(&head->st, false);
-	kfree(head);
-	env->head = elem;
-	env->stack_size--;
-	return 0;
-}
-
 static bool error_recoverable_with_nospec(int err)
 {
 	/* Should only return true for non-fatal errors that are allowed to
@@ -1873,10 +1843,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	elem->insn_idx = insn_idx;
 	elem->prev_insn_idx = prev_insn_idx;
 	elem->next = env->head;
-	elem->log_pos = env->log.end_pos;
-	elem->diag_log_pos = bpf_diag_event_log_save(env);
 	env->head = elem;
-	env->stack_size++;
 	err = bpf_copy_verifier_state(&elem->st, cur);
 	if (err)
 		return ERR_PTR(-ENOMEM);
@@ -2529,10 +2496,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	elem->insn_idx = insn_idx;
 	elem->prev_insn_idx = prev_insn_idx;
 	elem->next = env->head;
-	elem->log_pos = env->log.end_pos;
-	elem->diag_log_pos = bpf_diag_event_log_save(env);
 	env->head = elem;
-	env->stack_size++;
 	/* Unlike push_stack() do not bpf_copy_verifier_state().
 	 * The caller state doesn't matter.
 	 * This is async callback. It starts in a fresh stack.
@@ -15507,7 +15471,6 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return info;
 }
 
-/* insn_aux_data->is_iter_next is not there before the main pass */
 bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn)
 {
 	struct bpf_call_arg_meta meta;
@@ -15967,7 +15930,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 	meta.fn = &desc->proto;
 
-	insn_aux->is_iter_next = bpf_is_iter_next_kfunc(&meta);
 
 	if (!insn->off &&
 	    (insn->imm == special_kfunc_list[KF_bpf_res_spin_lock] ||
@@ -21655,7 +21617,6 @@ static int flow_drain(struct bpf_verifier_env *env)
 
 	while ((elem = env->head)) {
 		env->head = elem->next;
-		env->stack_size--;
 		if (!err && elem->guard_edge)
 			err = bpf_flow_guard_edge(env, &elem->st, elem->insn_idx,
 						  elem->prev_insn_idx, elem->guard_edge);
@@ -22964,9 +22925,15 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 
 static void free_states(struct bpf_verifier_env *env)
 {
+	struct bpf_verifier_stack_elem *elem;
+
 	bpf_free_verifier_state(env->cur_state, true);
 	env->cur_state = NULL;
-	while (!pop_stack(env, NULL, NULL, false));
+	while ((elem = env->head)) {
+		env->head = elem->next;
+		bpf_free_verifier_state(&elem->st, false);
+		kfree(elem);
+	}
 	bpf_flow_reset(env);
 }
 
@@ -23323,10 +23290,10 @@ static void print_verification_stats(struct bpf_verifier_env *env)
 				env->subprog_info[i].stack_depth);
 		}
 	}
-	/* user space parses this line. There are no read marks. */
-	verbose(env, "processed %d insns (limit %d) max_states_per_insn %d "
+	/* user space parses this line. There is no limit and there are no read marks. */
+	verbose(env, "processed %d insns (limit 0) max_states_per_insn %d "
 		"total_states %d peak_states %d mark_read 0\n",
-		env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS,
+		env->insn_processed,
 		env->max_states_per_insn, env->total_states,
 		env->peak_states);
 }
-- 
2.55.0

[PATCH bpf-next v3 26/45] bpf: Update comments about paths, pruning and explored states

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

About 30 comments in verifier.c, states.c and bpf_verifier.h describe
a verifier that explores all paths of a prog and prunes a path against
explored states. The reader of regsafe() or stacksafe() learns from them
that the old state "already passed validation" and that the function
returns false "to continue verification of this path". Nothing of it is
true since the main pass is a fixpoint over the control flow graph.

Say what the code does now. States meet at the start of a basic block. The
verifier drops a state when a state that is already there covers it: if the
prog is fine with the old state, it is fine with the current one. Otherwise
it joins the two states or keeps both.

The largest change is above process_iter_next_call(). 60 lines explain how
a loop with an open-coded iterator converges: exact comparison, the depth
of the iteration, widening of scalars that are not precise. Such a loop is
a loop like any other now: what comes back to its head is joined with what
is there until nothing changes.

No functional changes.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  11 ++--
 kernel/bpf/states.c          |  59 ++++++++------------
 kernel/bpf/verifier.c        | 102 ++++++++---------------------------
 3 files changed, 49 insertions(+), 123 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 9439c8f90abd..6d385819bcf5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -58,7 +58,7 @@ struct bpf_stack_range {
 };
 
 struct bpf_reg_state {
-	/* Ordering of fields matters.  See states_equal() */
+	/* Ordering of fields matters.  See regsafe() */
 	enum bpf_reg_type type;
 	/*
 	 * Constant delta between "linked" scalars with the same ID.
@@ -438,10 +438,9 @@ struct bpf_func_state {
 	 * zero == main subprog
 	 */
 	u32 subprogno;
-	/* Every bpf_timer_start will increment async_entry_cnt.
-	 * It's used to distinguish:
-	 * void foo(void) { for(;;); }
-	 * void foo(void) { bpf_timer_set_callback(,foo); }
+	/*
+	 * The frame of an async callback has the count of frame 0 of the state
+	 * that sets the callback plus one. Frames that differ in it are not joined.
 	 */
 	u32 async_entry_cnt;
 	struct bpf_retval_range callback_ret_range;
@@ -1961,7 +1960,7 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn);
 void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 			  struct bpf_call_summary *cs);
-/* Stack effects used for state pruning; must_write implies may_write. */
+/* Stack effects used for liveness; must_write implies may_write. */
 struct arg_access_info {
 	u32 size;		/* Maximum extent; U32_MAX if unknown. */
 	u8 may_read:1;		/* Incoming contents or initialization may be needed. */
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index fb7998654e16..07b1f0becd2e 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -229,7 +229,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		    struct bpf_reg_state *rcur, struct bpf_idmap *idmap)
 {
 	if (rold->type == NOT_INIT)
-		/* explored state can't have used this */
+		/* nothing can be done with it */
 		return true;
 
 	/* Enforce that register types have to match exactly, including their
@@ -308,7 +308,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		 * ADD_CONST flags must match exactly: BPF_ADD_CONST32 and
 		 * BPF_ADD_CONST64 have different linking semantics in
 		 * sync_linked_regs() (alu32 zero-extends, alu64 does not),
-		 * so pruning across different flag types is unsafe.
+		 * so one doesn't stand for the other.
 		 */
 		if (rold->id &&
 		    (rold->id & BPF_ADD_CONST) != (rcur->id & BPF_ADD_CONST))
@@ -420,9 +420,10 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 {
 	int i, spi;
 
-	/* walk slots of the explored stack and ignore any additional
-	 * slots in the current stack, since explored(safe) state
-	 * didn't use them
+	/*
+	 * walk slots of the old stack and ignore any additional
+	 * slots in the current stack, since the old state
+	 * can't use them
 	 */
 	for (i = 0; i < old->allocated_stack; i++) {
 		struct bpf_stack_state *old_slot, *cur_slot;
@@ -441,9 +442,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		if (env->allow_uninit_stack && old_type == STACK_MISC)
 			continue;
 
-		/* explored stack has more populated slots than current stack
-		 * and these slots were used
-		 */
+		/* old stack has more populated slots than current stack */
 		if (!cur_slot)
 			return false;
 
@@ -471,10 +470,10 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		if (old_type == STACK_MISC && cur_slot->slot_type[im] == STACK_ZERO)
 			continue;
 		if (old_type != cur_slot->slot_type[im])
-			/* Ex: old explored (safe) state has STACK_SPILL in
+			/*
+			 * Ex: old state has STACK_SPILL in
 			 * this stack slot, but current has STACK_MISC ->
-			 * this verifier states are not equivalent,
-			 * return false to continue verification of this path
+			 * this verifier states are not equivalent
 			 */
 			return false;
 		if (im != BPF_REG_SIZE - 1)
@@ -482,15 +481,15 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		/* Both old and cur are having same slot_type */
 		switch (old_type) {
 		case STACK_SPILL:
-			/* when explored and current stack slot are both storing
+			/*
+			 * when old and current stack slot are both storing
 			 * spilled registers, check that stored pointers types
 			 * are the same as well.
-			 * Ex: explored safe path could have stored
+			 * Ex: old state could have stored
 			 * (bpf_reg_state) {.type = PTR_TO_STACK, .off = -8}
-			 * but current path has stored:
+			 * but current state has stored:
 			 * (bpf_reg_state) {.type = PTR_TO_STACK, .off = -16}
 			 * such verifier states are not equivalent.
-			 * return false to continue verification of this path
 			 */
 			if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr, idmap))
 				return false;
@@ -537,7 +536,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 
 /*
  * Compare stack arg slots between old and current states.
- * Outgoing stack args are path-local state and must agree for pruning.
+ * Outgoing stack args are a part of the state and must agree.
  */
 static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			   struct bpf_func_state *cur, struct bpf_idmap *idmap)
@@ -659,29 +658,17 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
 
 /* compare two verifier states
  *
- * all states stored in state_list are known to be valid, since
- * verifier reached 'bpf_exit' instruction through them
- *
- * this function is called when verifier exploring different branches of
- * execution popped from the state stack. If it sees an old state that has
- * more strict register state and more strict stack state then this execution
- * branch doesn't need to be explored further, since verifier already
- * concluded that more strict state leads to valid finish.
- *
- * Therefore two states are equivalent if register state is more conservative
- * and explored stack state is more conservative than the current one.
+ * The state that gets to an insn doesn't have to be looked at when there is
+ * a state at that insn that is more conservative in registers and in the
+ * stack: if the program is fine with the old state it's fine with the current
+ * one.
  * Example:
- *       explored                   current
+ *          old                     current
  * (slot1=INV slot2=MISC) == (slot1=MISC slot2=MISC)
  * (slot1=MISC slot2=MISC) != (slot1=INV slot2=MISC)
  *
- * In other words if current stack state (one being explored) has more
- * valid slots than old one that already passed validation, it means
- * the verifier can stop exploring and conclude that current state is valid too
- *
- * Similarly with registers. If explored state has register type as invalid
- * whereas register type in current state is meaningful, it means that
- * the current state will reach 'bpf_exit' instruction safely
+ * Similarly with registers: the type in the old state is invalid whereas
+ * the type in the current state is meaningful.
  */
 static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			      struct bpf_func_state *cur, u32 insn_idx)
@@ -746,7 +733,7 @@ bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state
 	reset_idmap_scratch(env);
 
 	/* Verification state from speculative execution simulation
-	 * must never prune a non-speculative execution one.
+	 * must never stand for a non-speculative execution one.
 	 */
 	if (old->speculative && !cur->speculative)
 		return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 285e459ef741..437a3fc03d0b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -62,19 +62,16 @@ static bool bpf_global_percpu_ma_set;
 
 /* bpf_check() is a static code analyzer that walks eBPF program
  * instruction by instruction and updates register/stack state.
- * All paths of conditional branches are analyzed until 'bpf_exit' insn.
  *
- * The first pass is depth-first-search to check that the program is a DAG.
+ * The first pass is depth-first-search over the control flow graph.
  * It rejects the following programs:
- * - larger than BPF_MAXINSNS insns
- * - if loop is present (detected via back-edge)
+ * - if loop is present (detected via back-edge) and the program is unprivileged
  * - unreachable insns exist (shouldn't be a forest. program = one function)
  * - out of bounds or malformed jumps
- * The second pass is all possible path descent from the 1st insn.
- * Since it's analyzing all paths through the program, the length of the
- * analysis is limited to 64k insn, which may be hit even if total number of
- * insn is less then 4K, but there are too many branches that change stack/regs.
- * Number of 'branches to be analyzed' is limited to 1k
+ * The second pass goes from basic block to basic block, starting at the 1st
+ * insn. States that meet at the start of a block are joined, so the length of
+ * the analysis follows the size of the program and not the number of paths
+ * through it. See flow.c.
  *
  * On entry to each instruction, each register has a type, and the instruction
  * changes the types of the registers depending on instruction semantics.
@@ -1358,7 +1355,7 @@ static bool is_stack_slot_special(const struct bpf_stack_state *stack)
  * - STACK_POISON, which truly forbids access to the slot.
  * Regardless of allow_ptr_leaks setting (i.e., privileged or unprivileged
  * mode), we won't promote STACK_INVALID to STACK_MISC. In privileged case it is
- * unnecessary as both are considered equivalent when loading data and pruning,
+ * unnecessary as both are considered equivalent when loading data and comparing states,
  * in case of unprivileged mode it will be incorrect to allow reads of invalid
  * slots.
  */
@@ -1782,9 +1779,7 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	struct bpf_func_state *dst;
 	int i, err;
 
-	/* if dst has more stack frames then src frame, free them, this is also
-	 * necessary in case of exceptional exits using bpf_throw.
-	 */
+	/* if dst has more stack frames then src frame, free them */
 	for (i = src->curframe + 1; i <= dst_state->curframe; i++) {
 		free_func_state(dst_state->frame[i]);
 		dst_state->frame[i] = NULL;
@@ -8926,69 +8921,13 @@ static void iter_stops_loops(struct bpf_verifier_env *env, int insn_idx,
  * without having to simulate infinite amount of iterations.
  *
  * In current state, we first assume that iter_next() returned NULL and
- * iterator state is set to DRAINED (BPF_ITER_STATE_DRAINED). In such
- * conditions we should not form an infinite loop and should eventually reach
- * exit.
- *
- * Besides that, we also fork current state and enqueue it for later
- * verification. In a forked state we keep iterator state as ACTIVE
- * (BPF_ITER_STATE_ACTIVE) and assume non-NULL return from iter_next(). We
- * also bump iteration depth to prevent erroneous infinite loop detection
- * later on (see iter_active_depths_differ() comment for details). In this
- * state we assume that we'll eventually loop back to another iter_next()
- * calls (it could be in exactly same location or in some other instruction,
- * it doesn't matter, we don't make any unnecessary assumptions about this,
- * everything revolves around iterator state in a stack slot, not which
- * instruction is calling iter_next()). When that happens, we either will come
- * to iter_next() with equivalent state and can conclude that next iteration
- * will proceed in exactly the same way as we just verified, so it's safe to
- * assume that loop converges. If not, we'll go on another iteration
- * simulation with a different input state, until all possible starting states
- * are validated or we reach maximum number of instructions limit.
- *
- * This way, we will either exhaustively discover all possible input states
- * that iterator loop can start with and eventually will converge, or we'll
- * effectively regress into bounded loop simulation logic and either reach
- * maximum number of instructions if loop is not provably convergent, or there
- * is some statically known limit on number of iterations (e.g., if there is
- * an explicit `if n > 100 then break;` statement somewhere in the loop).
- *
- * Iteration convergence logic in is_state_visited() relies on exact
- * states comparison, which ignores read and precision marks.
- * This is necessary because read and precision marks are not finalized
- * while in the loop. Exact comparison might preclude convergence for
- * simple programs like below:
- *
- *     i = 0;
- *     while(iter_next(&it))
- *       i++;
+ * iterator state is set to DRAINED (BPF_ITER_STATE_DRAINED).
  *
- * At each iteration step i++ would produce a new distinct state and
- * eventually instruction processing limit would be reached.
- *
- * To avoid such behavior speculatively forget (widen) range for
- * imprecise scalar registers, if those registers were not precise at the
- * end of the previous iteration and do not match exactly.
- *
- * This is a conservative heuristic that allows to verify wide range of programs,
- * however it precludes verification of programs that conjure an
- * imprecise value on the first loop iteration and use it as precise on a second.
- * For example, the following safe program would fail to verify:
- *
- *     struct bpf_num_iter it;
- *     int arr[10];
- *     int i = 0, a = 0;
- *     bpf_iter_num_new(&it, 0, 10);
- *     while (bpf_iter_num_next(&it)) {
- *       if (a == 0) {
- *         a = 1;
- *         i = 7; // Because i changed verifier would forget
- *                // it's range on second loop entry.
- *       } else {
- *         arr[i] = 42; // This would fail to verify.
- *       }
- *     }
- *     bpf_iter_num_destroy(&it);
+ * Besides that, we also fork current state. In a forked state we keep iterator
+ * state as ACTIVE (BPF_ITER_STATE_ACTIVE) and assume non-NULL return from
+ * iter_next(). This state goes around the loop like in any other loop: what
+ * comes back to the head of the loop is joined with what is there until
+ * nothing changes, see flow.c.
  */
 static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 				  struct bpf_call_arg_meta *meta)
@@ -12619,7 +12558,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	}
 	account_processed_insns(env, callee, caller);
 	/* clear everything in the callee. In case of exceptional exits using
-	 * bpf_throw, this will be done by copy_verifier_state for extra frames. */
+	 * bpf_throw, this will be done by do_check_block() for extra frames. */
 	free_func_state(callee);
 	state->frame[state->curframe--] = NULL;
 	invalidate_outgoing_stack_args(env, caller);
@@ -15338,7 +15277,7 @@ int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env,
  * Describe a helper's stack access through argument @arg (0-based, R1-R5).
  * must_write means the verifier destroys prior state throughout size bytes.
  * In other words, must_write follows the verifier's model of the
- * function's behaviour, which is safe to use for is_state_visited() pruning.
+ * function's behaviour, which is safe to use when states are cleaned and compared.
  */
 struct arg_access_info
 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
@@ -16620,7 +16559,8 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
 		alu_state |= ptr_is_dst_reg ?
 			     BPF_ALU_SANITIZE_SRC : BPF_ALU_SANITIZE_DST;
 
-		/* Limit pruning on unknown scalars to enable deep search for
+		/*
+		 * Keep states that differ in unknown scalars apart to find
 		 * potential masking differences from other program paths.
 		 */
 		if (!off_is_imm)
@@ -19251,7 +19191,7 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state,
 		if (is_null) {
 			/* We don't need id from this point
 			 * onwards anymore, thus we should better reset it,
-			 * so that state pruning has chances to take effect.
+			 * so that states have chances to be equal.
 			 */
 			__mark_reg_known_zero(reg);
 			reg->type = SCALAR_VALUE;
@@ -20716,8 +20656,8 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 	/* The side effect of the prepare_func_exit which is
 	 * being skipped is that it frees bpf_func_state.
 	 * Typically, process_bpf_exit will only be hit with
-	 * outermost exit. copy_verifier_state in pop_stack will
-	 * handle freeing of any extra bpf_func_state left over
+	 * outermost exit. do_check_block() frees the state with
+	 * any extra bpf_func_state left over
 	 * from not processing all nested function exits. We
 	 * also skip return code checks as they are not needed
 	 * for exceptional exits.
-- 
2.55.0

Re: [PATCH bpf-next v3 26/45] bpf: Update comments about paths, pruning and explored states

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: non-standard multi-line comment style in updated verifier comments
--

commit de7d08cee08981c083cd91cad28f7bba69ff01a5
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Update comments about paths, pruning and explored states

This commit updates verifier comments in include/linux/bpf_verifier.h,
kernel/bpf/states.c, and kernel/bpf/verifier.c to describe the control-flow
graph fixpoint pass and state joining instead of legacy path exploration and
pruning.

[Severity: Low]
This isn't a bug, but while this patch converts several multi-line comments in
include/linux/bpf_verifier.h, kernel/bpf/states.c (in stacksafe()), and
kernel/bpf/verifier.c (in sanitize_ptr_alu()) to the BPF multi-line comment
style with the opening /* on its own line, several other multi-line comments
updated by this patch still keep text on the opening /* line:

- kernel/bpf/states.c, in the block comment above func_states_equal()
  (starting at line 659)
- kernel/bpf/states.c, inside bpf_state_subsumes() before the old->speculative
  check (starting at line 735)
- kernel/bpf/verifier.c, in the top-level bpf_check() overview comment
  (starting at line 63)
- kernel/bpf/verifier.c, in the block comment above process_iter_next_call()
  (starting at line 8909)
- kernel/bpf/verifier.c, inside prepare_func_exit() before free_func_state()
  (starting at line 12560, which also keeps the closing */ on the same line as
  the comment text on line 12561)
- kernel/bpf/verifier.c, inside mark_ptr_or_null_reg() in the is_null branch
  (starting at line 19192)
- kernel/bpf/verifier.c, inside process_bpf_exit_full() before the
  exception_exit check (starting at line 20656)

Should these updated multi-line comments also place the opening /* and closing
*/ on their own lines to match the BPF coding style?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=26

Re: [PATCH bpf-next v3 26/45] bpf: Update comments about paths, pruning and explored states

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

> bpf: Update comments about paths, pruning and explored states

TL;DR: Nits only, comment-only patch: some comments still describe the
verifier walking all paths (loop_stack_safe() comment, regsafe() check_ids()
example); should they be reworded too?

> diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
> --- a/kernel/bpf/states.c
> +++ b/kernel/bpf/states.c

[ ... ]

> @@ -659,29 +658,17 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
>  
>  /* compare two verifier states
>   *
> - * all states stored in state_list are known to be valid, since
> - * verifier reached 'bpf_exit' instruction through them
> - *
[ ... ]
> + * The state that gets to an insn doesn't have to be looked at when there is
> + * a state at that insn that is more conservative in registers and in the
> + * stack: if the program is fine with the old state it's fine with the current
> + * one.
>   * Example:

This isn't a bug, but this patch is meant to remove comments that describe
a verifier walking all paths, and a few are still left.

The block comment above loop_stack_safe() in the same file still says:

 * The walk of a loop with two entries is meant to get to the end of the
 * loop, like today's walk of all paths.

Since the verifier no longer walks all paths, should this wording be
updated as well?

The check_ids() example in the regsafe() comment also still describes the
case in DFS-path order:

  First verification path is [1-6]: ...
  Next verification path is [1-4, 6].

Under the fixpoint pass, the two states of that example meet at the start
of a basic block instead of being verified one path after the other.  Should
that example be reworded too?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 27/45] bpf: Track a pointer to one of a set of functions

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Each '{}' of a format string in Rust ends in core::fmt::write(), which
calls the fmt() function of the argument through a pointer:

  ArgumentType::Placeholder { formatter, value, .. } => formatter(value, f)

rust-bpf compiles it to a load from the table of all fmt() functions of the
object, and callx. k2_text.o of its corpus, with serde_json, uuid, semver,
ryu and regex-automata, has 245. At the load the verifier goes on with one
function and makes a state for each of the other 244. 3 states get to the
load: 732 states that differ only in r3 meet at the next insn. The limit is
1024. Each more type with Display or Debug adds 3 states, and 100 more
types give "more than 1024 states that cannot be joined at insn N".

  if r2 < 244 goto +1
  r2 = 244
  r2 <<= 3
  r2 += r3                 ; the table, in a read-only map
  r3 = *(u64 *)(r2 + 8)
  ...
  callx r3

Do not split the state at the load. r3 is a pointer to one of a set of
functions until an insn must know the function.

     r3 = *(u64 *)(r2 + 8)              r3 = *(u64 *)(r2 + 8)
      /        |        \                         |
   r3=f1     r3=f2 ...  r3=fN             r3 = one of f1 .. fN
     |         |          |                       |
  callx r3  callx r3   callx r3               callx r3
     |         |          |                  /    |    \
    f1        f2         fN                f1    f2 ... fN

The register holds the number of the set, and equal sets have the same
number. Thus the code that compares and joins registers does not change.
The log shows R3=func(one_of=245).

callx verifies each function of the set, as before. For a function that is
verified on its own, a copy of the state of the caller waits for its exit
states. The copy is not a state at an insn. Other functions, and a helper
or a kfunc that gets the pointer as a callback, get one state for each
function. uuid_semver of k2_text.o: 732 -> 257 states at an insn and
-19 % insns.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  15 ++-
 kernel/bpf/log.c             |   2 +
 kernel/bpf/verifier.c        | 181 +++++++++++++++++++++++++++++------
 3 files changed, 169 insertions(+), 29 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6d385819bcf5..9b7c31fefe72 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -140,7 +140,11 @@ struct bpf_reg_state {
 			unsigned long raw2;
 		} raw;
 
-		u32 subprogno; /* for PTR_TO_FUNC */
+		struct { /* for PTR_TO_FUNC */
+			u32 subprogno;
+			/* if not 0: one of env->func_sets[func_set - 1], subprogno is 0 */
+			u32 func_set;
+		};
 	};
 	/* For scalar types (SCALAR_VALUE), this represents our knowledge of
 	 * the actual value.
@@ -1209,6 +1213,9 @@ struct bpf_verifier_env {
 	 */
 	struct bpf_func_ptr *func_ptrs;
 	u32 func_ptr_cnt;
+	/* What a pointer that was read from a table of functions may point to. */
+	struct bpf_func_set **func_sets;
+	u32 func_set_cnt;
 	bool has_callx;
 	/*
 	 * Call graph edges created by callx instructions. A bitmap of
@@ -1588,6 +1595,12 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
 }
 
 bool bpf_map_is_rdonly(const struct bpf_map *map);
+/* Sorted, no function is there twice */
+struct bpf_func_set {
+	u16 cnt;
+	u16 subprog[] __counted_by(cnt);
+};
+
 struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
 				       const struct bpf_map *map, u32 *cnt);
 struct bpf_func_ptr *bpf_map_range_func_ptrs(struct bpf_verifier_env *env,
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index db496253bf62..1253118dfd05 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -664,6 +664,8 @@ static void print_reg_state(struct bpf_verifier_env *env,
 		verbose_a("parent_id=%d", reg->parent_id);
 	if (type_is_non_owning_ref(reg->type))
 		verbose_a("%s", "non_own_ref");
+	if (t == PTR_TO_FUNC && reg->func_set)
+		verbose_a("one_of=%d", env->func_sets[reg->func_set - 1]->cnt);
 	if (type_is_map_ptr(t)) {
 		if (reg->map_ptr->name[0])
 			verbose_a("map=%s", reg->map_ptr->name);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 437a3fc03d0b..41500f03b41b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -7196,7 +7196,60 @@ static void mark_reg_func_ptr(struct bpf_verifier_env *env, struct bpf_reg_state
 	regs[regno].subprogno = subprog;
 }
 
-/* a read from a table of functions branches into that many states at most */
+static int cmp_u16(const void *a, const void *b)
+{
+	return *(const u16 *)a - *(const u16 *)b;
+}
+
+/* that many different sets at most: each load from a table looks through all of them */
+#define BPF_MAX_FUNC_SETS 4096
+
+/*
+ * A load from a table of functions gives one register: a pointer to one of a set of functions.
+ * One state for each function would differ in nothing else up to the call, and there every
+ * function of the set is verified.
+ * Equal sets have the same number, so registers are compared as they are.
+ */
+static int mark_reg_func_set(struct bpf_verifier_env *env, struct bpf_reg_state *regs, int regno,
+			     u16 *subprogs, u32 cnt)
+{
+	struct bpf_func_set **sets, *set;
+	u32 i, n = 0;
+
+	sort(subprogs, cnt, sizeof(*subprogs), cmp_u16, NULL);
+	for (i = 0; i < cnt; i++)
+		if (!n || subprogs[n - 1] != subprogs[i])
+			subprogs[n++] = subprogs[i];
+	mark_reg_func_ptr(env, regs, regno, n == 1 ? subprogs[0] : 0);
+	if (n == 1)
+		return 0;
+
+	for (i = 0; i < env->func_set_cnt; i++) {
+		set = env->func_sets[i];
+		if (set->cnt == n && !memcmp(set->subprog, subprogs, n * sizeof(*subprogs)))
+			break;
+	}
+	if (i == env->func_set_cnt) {
+		if (i == BPF_MAX_FUNC_SETS) {
+			verbose(env, "more than %d sets of functions\n", BPF_MAX_FUNC_SETS);
+			return -E2BIG;
+		}
+		sets = krealloc_array(env->func_sets, i + 1, sizeof(*sets), GFP_KERNEL_ACCOUNT);
+		if (!sets)
+			return -ENOMEM;
+		env->func_sets = sets;
+		set = kzalloc_flex(*set, subprog, n, GFP_KERNEL_ACCOUNT);
+		if (!set)
+			return -ENOMEM;
+		set->cnt = n;
+		memcpy(set->subprog, subprogs, n * sizeof(*subprogs));
+		sets[env->func_set_cnt++] = set;
+	}
+	regs[regno].func_set = i + 1;
+	return 0;
+}
+
+/* a read from a table of functions yields one of that many at most */
 #define BPF_MAX_FUNC_PTR_TARGETS BPF_MAX_SUBPROGS
 /* and that many pointers at most are next to what is read at variable offset */
 #define BPF_MAX_FUNC_PTR_RANGE 4096
@@ -7229,7 +7282,7 @@ static u32 count_read_offsets(struct tnum offs, u64 min_off, u64 max_off, u32 li
  * is possible at, e.g. var_off is the stride of the elements of an array.
  * A read of exactly one pointer yields PTR_TO_FUNC. When the offset is variable
  * and only pointers can be read, which is how an element of a table of
- * functions is loaded, the verification continues with each of them. Other
+ * functions is loaded, it is one of them, see mark_reg_func_set(). Other
  * reads that overlap with a pointer are rejected, because their result is not
  * known until the program is jitted.
  *
@@ -7242,7 +7295,6 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
 	struct tnum offs = tnum_add(reg->var_off, tnum_const(off));
 	struct bpf_reg_state *regs = cur_regs(env);
 	struct bpf_map *map = reg->map_ptr;
-	struct bpf_verifier_state *branch;
 	u16 *targets __free(kfree) = NULL;
 	struct bpf_func_ptr *ptrs;
 	u32 i, cnt, n = 0;
@@ -7293,16 +7345,7 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
 	if (err)
 		return err;
 
-	for (i = 0; i < n - 1; i++) {
-		branch = push_stack(env, env->insn_idx + 1, env->insn_idx,
-				    env->cur_state->speculative);
-		if (IS_ERR(branch))
-			return PTR_ERR(branch);
-		mark_reg_func_ptr(env, branch->frame[branch->curframe]->regs, value_regno,
-				  targets[i]);
-	}
-	mark_reg_func_ptr(env, regs, value_regno, targets[n - 1]);
-	return 0;
+	return mark_reg_func_set(env, regs, value_regno, targets, n);
 
 overlap:
 	verbose(env, "read of %d bytes at offset [%llu,%llu] of map '%s' overlaps with a pointer to a function\n",
@@ -11593,7 +11636,8 @@ static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier
 		if (arg_needs_caller(state, sub, &caller->regs[i]))
 			return false;
 	}
-	for (i = 0; i < bpf_in_stack_arg_cnt(sub); i++) {
+	/* callx of one function of a set asks before it is checked that the arguments are there */
+	for (i = 0; i < bpf_in_stack_arg_cnt(sub) && i < caller->out_stack_arg_cnt; i++) {
 		if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
 		    base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
 			return false;
@@ -12028,13 +12072,93 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  * PTR_TO_FUNC register identifies the callee, so from here on callx is verified
  * as a direct call of that static subprog.
  */
+static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int *insn_idx)
+{
+	int err;
+
+	err = btf_check_subprog_call(env, subprog);
+	if (err == -EFAULT)
+		return err;
+
+	err = record_callx_edge(env, cur_func(env)->subprogno, subprog);
+	if (err)
+		return err;
+	env->subprog_info[subprog].via_callx = true;
+
+	return check_static_func_call(env, subprog, insn_idx);
+}
+
+/* The insn needs to know the function: go on with each function of the set. */
+static int split_func_set(struct bpf_verifier_env *env, int regno)
+{
+	struct bpf_reg_state *reg = reg_state(env, regno);
+	struct bpf_verifier_state *branch;
+	struct bpf_func_set *set;
+	u32 i;
+
+	if (reg->type != PTR_TO_FUNC || !reg->func_set)
+		return 0;
+	set = env->func_sets[reg->func_set - 1];
+	reg->func_set = 0;
+	for (i = 0; i < set->cnt; i++) {
+		reg->subprogno = set->subprog[i];
+		if (i == set->cnt - 1)
+			break;
+		branch = push_stack(env, env->insn_idx, env->insn_idx, false);
+		if (IS_ERR(branch))
+			return PTR_ERR(branch);
+	}
+	return 0;
+}
+
+/*
+ * callx of one of the functions of a set. A function that is verified on its own
+ * gets a copy of the state that waits for it. Nothing meets at an insn.
+ */
+static int check_func_callx_set(struct bpf_verifier_env *env, int regno, int *insn_idx)
+{
+	struct bpf_verifier_state *cur = env->cur_state, *copy, *branch;
+	struct bpf_reg_state *reg = reg_state(env, regno);
+	struct bpf_func_set *set = env->func_sets[reg->func_set - 1];
+	int err = 0;
+	u32 i;
+
+	/* once, and not in every copy */
+	account_current_path(env);
+	reg->func_set = 0;
+	for (i = 0; i < set->cnt && !err; i++) {
+		reg->subprogno = set->subprog[i];
+		if (!func_is_on_its_own(env, cur, set->subprog[i])) {
+			branch = push_stack(env, *insn_idx, *insn_idx, false);
+			if (IS_ERR(branch))
+				return PTR_ERR(branch);
+			continue;
+		}
+		copy = kzalloc_obj(*copy, GFP_KERNEL_ACCOUNT);
+		if (!copy)
+			return -ENOMEM;
+		err = bpf_copy_verifier_state(copy, cur);
+		if (!err) {
+			env->cur_state = copy;
+			err = check_func_callx_one(env, set->subprog[i], insn_idx);
+			env->cur_state = cur;
+		}
+		bpf_free_verifier_state(copy, true);
+		if (err == PROCESS_BPF_EXIT)
+			err = 0;
+		else if (verifier_bug_if(!err, env, "callx: func#%d doesn't wait", set->subprog[i]))
+			err = -EFAULT;
+	}
+	/* all that is to be done has a state of its own */
+	return err ?: PROCESS_BPF_EXIT;
+}
+
 static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			    int *insn_idx)
 {
-	struct bpf_func_state *caller = cur_func(env);
 	struct bpf_reg_state *reg;
 	const char *reason;
-	int err, subprog;
+	int err;
 
 	err = check_reg_arg(env, insn->dst_reg, SRC_OP);
 	if (err)
@@ -12067,17 +12191,9 @@ static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		return err;
 
 	/* PTR_TO_FUNC is a pointer to a static subprog */
-	subprog = reg->subprogno;
-	err = btf_check_subprog_call(env, subprog);
-	if (err == -EFAULT)
-		return err;
-
-	err = record_callx_edge(env, caller->subprogno, subprog);
-	if (err)
-		return err;
-	env->subprog_info[subprog].via_callx = true;
-
-	return check_static_func_call(env, subprog, insn_idx);
+	if (reg->func_set)
+		return check_func_callx_set(env, insn->dst_reg, insn_idx);
+	return check_func_callx_one(env, reg->subprogno, insn_idx);
 }
 
 int map_set_for_each_callback_args(struct bpf_verifier_env *env,
@@ -20875,7 +20991,7 @@ static int check_arena_sp_insn(struct bpf_verifier_env *env, struct bpf_insn *in
 
 static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 {
-	int err;
+	int err, i;
 	struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
 	u8 class = BPF_CLASS(insn->code);
 
@@ -20952,6 +21068,12 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 				return check_func_callx(env, insn, &env->insn_idx);
 			if (insn->src_reg == BPF_PSEUDO_CALL)
 				return check_func_call(env, insn, &env->insn_idx);
+			/* a callback is one function */
+			for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+				err = split_func_set(env, i);
+				if (err)
+					return err;
+			}
 			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
 				return check_kfunc_call(env, insn, &env->insn_idx);
 			return check_helper_call(env, insn, &env->insn_idx);
@@ -25432,6 +25554,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->gotox_tmp_buf);
 	kvfree(env->callx_edges);
 	kvfree(env->func_ptrs);
+	for (i = 0; i < env->func_set_cnt; i++)
+		kfree(env->func_sets[i]);
+	kfree(env->func_sets);
 	bpf_diag_free(env);
 	kvfree(env->idoms);
 	kvfree(env);
-- 
2.55.0

Re: [PATCH bpf-next v3 27/45] bpf: Track a pointer to one of a set of functions

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: unconditional R1-R5 split in do_check_insn() causes state explosion
--

commit 75a36349ee42f80ce6f3c17dd0dfc5718a853221
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Track a pointer to one of a set of functions

This commit tracks a pointer read from a function table as a single
register state representing a set of target functions (func_set) instead
of branching states at the load instruction. The function set is later
expanded into individual function targets when a callx instruction or a
helper/kfunc call requires a specific function.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 437a3fc03d0bf..41500f03b41be 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -12028,13 +12072,93 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
[ ... ]
> +/* The insn needs to know the function: go on with each function of the set. */
> +static int split_func_set(struct bpf_verifier_env *env, int regno)
> +{
> +	struct bpf_reg_state *reg = reg_state(env, regno);
> +	struct bpf_verifier_state *branch;
> +	struct bpf_func_set *set;
> +	u32 i;
> +
> +	if (reg->type != PTR_TO_FUNC || !reg->func_set)
> +		return 0;
> +	set = env->func_sets[reg->func_set - 1];
> +	reg->func_set = 0;
> +	for (i = 0; i < set->cnt; i++) {
> +		reg->subprogno = set->subprog[i];
> +		if (i == set->cnt - 1)
> +			break;
> +		branch = push_stack(env, env->insn_idx, env->insn_idx, false);
> +		if (IS_ERR(branch))
> +			return PTR_ERR(branch);
> +	}
> +	return 0;
> +}
[ ... ]
> @@ -20952,6 +21068,12 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
>  				return check_func_callx(env, insn, &env->insn_idx);
>  			if (insn->src_reg == BPF_PSEUDO_CALL)
>  				return check_func_call(env, insn, &env->insn_idx);
> +			/* a callback is one function */
> +			for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
> +				err = split_func_set(env, i);
> +				if (err)
> +					return err;
> +			}
>  			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
>  				return check_kfunc_call(env, insn, &env->insn_idx);
>  			return check_helper_call(env, insn, &env->insn_idx);

[Severity: Medium]
In do_check_insn(), split_func_set() is called unconditionally across all
registers BPF_REG_1 through BPF_REG_5 before every kfunc and helper call,
without checking whether the register is actually a callback argument or
even live at env->insn_idx.

Within a basic block, helper and kfunc calls that do not invoke synchronous
callbacks do not have a flow point set, so do_check_block() steps into the
call instruction without running bpf_flow_deliver() or
bpf_clean_verifier_state():

  do_check_block()
    do_check_insn()
      split_func_set()

If a caller-saved register in R1..R5 was loaded with a func_set of size N
earlier in the block and copied to a callee-saved register R6..R9 or
spilled to the stack before a helper or kfunc call that does not overwrite
that register, the dead register still holds PTR_TO_FUNC with reg->func_set
set when reaching the call.

In that case, split_func_set() clones env->cur_state N - 1 times onto
env->head at env->insn_idx. After env->cur_state finishes the call,
flow_drain() delivers those N - 1 states to env->insn_idx, allocating N - 1
flow_state objects, cleaning the dead register in flow_arrive(), and queuing
a duplicate pending state at env->insn_idx that re-verifies the call and the
rest of the basic block a second time.

Furthermore, if a second register in R1..R5 is marked live in
live_regs_before at the call while also holding a func_set, looping over
BPF_REG_1..BPF_REG_5 splits both registers and produces all N * N
combinations at env->insn_idx. For example, bpf_timer_set_callback_proto
sets arg3_type = ARG_PTR_TO_PROG_AUX, which causes bpf_get_call_summary() to
set arg_slot_cnt = 3 and compute_insn_live_regs() to mark R3 live in
live_regs_before even though the BPF program only passes R1 and R2. If a
program loads a callback from a table of N >= 33 functions into R3 and
copies r2 = r3 before calling bpf_timer_set_callback(), splitting both R2
and R3 creates N * N >= 1089 unjoinable states at env->insn_idx, exceeding
BPF_FLOW_MAX_STATES (1024) in flow_merge() and failing verification with
-E2BIG.

Should split_func_set() only be invoked for registers that correspond to
actual callback arguments (such as inside check_func_arg() /
process_kf_arg_ptr_to_func(), or gated on the call prototype and
live_regs_before)?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=27

[PATCH bpf-next v3 28/45] bpf: callx reads what the subprogs whose address is taken read

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

core::fmt::write() is in each Rust prog that prints. It reads the options
of a '{}' and calls the fmt() function of the argument through a pointer:

  if n & 8 != 0  { arg_index = ...(template.cast_array().read()); }
  if n & 16 != 0 { opt.width = args.add(opt.width)...as_u16(); }
  if n & 32 != 0 { opt.precision = args.add(opt.precision)...as_u16(); }
  args.add(arg_index).as_ref().fmt(&mut Formatter::new(output, opt))?;

The last line is callx with two arguments, r1 and r2. The pass that finds
the live registers does not know the callee: callx counts as a read of
r1-r5. What the branches leave in r4 and r5 stays live:

  w4 &= 16
  if w4 == 0 goto +4          ; r4 = 0
  r3 <<= 4
  r4 = *(u64 *)(r10 - 24)     ; r4 = args, a pointer to arena
  ...
  callx r3

r4 is 0 or a pointer, r5 is a byte or a pointer. A number and a pointer
have no join: 4 states go to callx in the first pass over the loop, 2 in
each later pass, where 1 is enough. A subprog with callx in it also seems
to take 5 arguments: what its callers leave in r3-r5 is a part of its entry
state, and it is verified again for each such leftover.

callx calls only a static subprog whose address the prog takes: with
ld_imm64, or in a table in a read-only map. mark_recursive_subprogs() has
this set. Store it in addr_taken of bpf_subprog_info. The passes in flow.c
that look through callx ahead of the main pass read the bit already.
Nothing set it so far: they took no subprog for a target of callx.

  insn                        the registers of r1-r5 that it reads
  --------------------------  ---------------------------------------------
  call of a static subprog f  live at the first insn of f, as before
  callx, before               all
  callx, this patch           live at the first insn of a subprog that has
                              addr_taken, or declared in its BTF

bpf_compute_live_registers() finds both in one loop, until nothing changes.
zext_dst comes from the same result, thus check_func_callx_one() reports
a verifier bug if the callee has no addr_taken.

bpfland_stopping of scx_bpfland.rs: -9 % insns, -15 % states. 99 of the 225
progs of the schedulers of rust-bpf change: -3 % insns, -6 % states.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/liveness.c | 28 ++++++++++++++++++++++++++--
 kernel/bpf/verifier.c | 11 ++++++++++-
 2 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 290d6e8147de..c8f74063913c 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2732,9 +2732,12 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			}
 			def = mask_widen(def);
 			use = mask_widen(use);
-			/* callx reads the address of the callee from dst_reg */
+			/*
+			 * callx reads the address of the callee from dst_reg. What the callee
+			 * reads is added in bpf_compute_live_registers().
+			 */
 			if (bpf_is_callx(insn))
-				use |= dst;
+				use = dst;
 			break;
 		default:
 			def = 0;
@@ -2801,6 +2804,25 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 
 	changed = true;
 	while (changed) {
+		u32 args = mask_widen(GENMASK(BPF_REG_5, BPF_REG_1)), callx_use = 0, declared;
+		bool taken = false;
+
+		/*
+		 * callx calls one of the subprogs whose address is taken, see
+		 * check_func_callx_one(). It reads what they read, as computed so far.
+		 */
+		for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+			if (!env->subprog_info[subprog].addr_taken)
+				continue;
+			callx_use |= state[env->subprog_info[subprog].start].in & args;
+			/* and what its BTF declares: the BTF is unreliable when one is not set */
+			declared = env->subprog_info[subprog].arg_slot_cnt;
+			callx_use |= mask_widen(GENMASK(declared, 1)) & args;
+			taken = true;
+		}
+		if (!taken)
+			callx_use = args;
+
 		changed = false;
 		/* a cycle of calls takes a round for each link */
 		if (signal_pending(current)) {
@@ -2829,6 +2851,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			    !bpf_subprog_is_global(env, callee_subprog(env, &insns[insn_idx])))
 				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
 				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
+			if (bpf_is_callx(&insns[insn_idx]))
+				use |= callx_use;
 			new_in = (new_out & ~live->def) | use;
 			/* may_goto that is added at the head of a loop goes there */
 			t = bpf_head_guard_target(env, insn_idx);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 41500f03b41b..b0d4410076dd 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6105,6 +6105,8 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
 	}
 	for_each_set_bit(s, callx, cnt)
 		bitmap_or(calls + s * words, calls + s * words, taken, cnt);
+	for_each_set_bit(s, taken, cnt)
+		si[s].addr_taken = true;
 
 	/*
 	 * Callers of a global subprog look at can_unwind before the subprog is verified.
@@ -11366,7 +11368,8 @@ static void gen_subprog_arg_proto(const struct bpf_subprog_info *sub, const stru
 
 /*
  * The argument slots that the code of a static subprog takes, whatever its BTF says.
- * Liveness errs on the safe side: a callx in the subprog counts as a read of r1-r5.
+ * Liveness errs on the safe side: a callx in the subprog counts as a read of all that the subprogs
+ * whose address is taken read or declare.
  * But it misses r3-r5 of bpf_trace_printk() and the flags of bpf_get_branch_snapshot(), which
  * their protos do not declare. Such a subprog prints or passes a wrong number under a tracer.
  * The helpers are safe with any value there: the verifier never checks these registers.
@@ -12080,6 +12083,12 @@ static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int *
 	if (err == -EFAULT)
 		return err;
 
+	/* bpf_compute_live_registers() counts on it */
+	if (verifier_bug_if(!env->subprog_info[subprog].addr_taken, env,
+			    "callx of %s() whose address is not taken",
+			    bpf_subprog_name(env, subprog)))
+		return -EFAULT;
+
 	err = record_callx_edge(env, cur_func(env)->subprogno, subprog);
 	if (err)
 		return err;
-- 
2.55.0

[PATCH bpf-next v3 29/45] bpf: Don't take what bpf_core_cast() returns for a number

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  struct task_struct *p = bpf_core_cast(t, struct task_struct);
  x = z ? p->pid : p->tgid;

clang makes one add and one load of it:

     if w1 == 0 goto 1f
     r1 = 1856			; offsetof(struct task_struct, pid)
     goto 2f
  1: r1 = 1860			; offsetof(struct task_struct, tgid)
  2: r0 += r1
     w1 = *(u32 *)(r0 + 0)

A pointer to a kernel object takes a known offset only, so the main pass
must not join 1856 and 1860. It joins them, and rejects a prog that loaded
before states were joined:

  invalid variable offset access into struct task_struct

bpf_flow_compute_kinds() takes void * that a kfunc returns for a number,
unless the kfunc has KF_RET_NULL. That finds bpf_obj_new(),
bpf_dynptr_slice() and the like, where the verifier makes up the type.
It does that for bpf_rdonly_cast(), which bpf_core_cast() calls, and for
bpf_cast_to_kern_ctx() too, and these two cannot fail. So r0 counted as
a number, and nothing asked for a constant in r1.

Name the two kfuncs in flow_returned_kind().

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 1 +
 kernel/bpf/flow.c            | 6 +++---
 kernel/bpf/verifier.c        | 7 +++++++
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 9b7c31fefe72..78a787aaba44 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1965,6 +1965,7 @@ static inline bool bpf_is_kfunc_sleepable(struct bpf_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_SLEEPABLE;
 }
 bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta);
+bool bpf_is_cast_kfunc(const struct bpf_call_arg_meta *meta);
 struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem);
 int bpf_copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off);
 bool bpf_insn_is_cond_jump(u8 code);
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 62115e396635..d5dbe417421a 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -733,10 +733,10 @@ static enum flow_kind flow_returned_kind(struct bpf_verifier_env *env, int idx)
 		t = btf_type_skip_modifiers(meta.btf, meta.func_proto->type, NULL);
 		if (!btf_type_is_ptr(t))
 			return FLOW_IS_PLAIN;
-		/* void * is a number, unless the verifier knows the kfunc. Such kfunc may fail. */
+		/* void * is a number, unless the verifier knows the kfunc: it may fail, or casts */
 		t = btf_type_skip_modifiers(meta.btf, t->type, NULL);
-		return btf_type_is_void(t) && !(meta.kfunc_flags & KF_RET_NULL) ? FLOW_IS_PLAIN :
-										  FLOW_IS_OTHER;
+		return btf_type_is_void(t) && !(meta.kfunc_flags & KF_RET_NULL) &&
+		       !bpf_is_cast_kfunc(&meta) ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
 	}
 	if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
 		return FLOW_IS_OTHER;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b0d4410076dd..2a0d5a280a1a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14259,6 +14259,13 @@ static bool is_bpf_cast_to_kern_ctx_kfunc(const struct bpf_call_arg_meta *meta)
 	return is_kfunc_call(meta, special_kfunc_list[KF_bpf_cast_to_kern_ctx]);
 }
 
+/* void * in the prototype. What it returns points to an object of the kernel. */
+bool bpf_is_cast_kfunc(const struct bpf_call_arg_meta *meta)
+{
+	return is_kfunc_call(meta, special_kfunc_list[KF_bpf_cast_to_kern_ctx]) ||
+	       is_kfunc_call(meta, special_kfunc_list[KF_bpf_rdonly_cast]);
+}
+
 static bool is_bpf_dynptr_clone_kfunc(const struct bpf_call_arg_meta *meta)
 {
 	return is_kfunc_call(meta, special_kfunc_list[KF_bpf_dynptr_clone]);
-- 
2.55.0

[PATCH bpf-next v3 30/45] bpf: Don't widen away a register that only the loop sets

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  bpf_for(i, 0, 64) {
          if (v[i] != want)
                  continue;
          last = bpf_get_prandom_u32();
          found = true;
  }
  if (found)
          bpf_printk("%llu", last);

clang 18 does not set 'last' in front of the loop. The state that enters
the loop has found = 0 and no 'last'. The state that comes around has
found = 1 and 'last'. At the loop head the two are widened into one state:
found is 0 or 1, 'last' is not set. The prog loaded before states were
joined, and is rejected now:

  R7 !read_ok

Where paths meet outside of a loop flow_unset_differs() keeps such states
apart: nothing is gained from a join that loses a live register. Ask it at
a loop head too. Like there, the rule holds while there is room for states
at the insn.

The progs of rust-bpf take 0.1% to 1.5% more insns, the progs of selftests,
Cilium and sched-ext less than 0.1%.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index d5dbe417421a..6854e795e537 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -2328,9 +2328,9 @@ static int flow_merge(struct bpf_verifier_env *env, struct flow_set *set,
 		}
 		if (!widen && flow_cmp_pkt_range(s, t))
 			continue;
-		if (!widen && flow_has_room(env, *nr_states) &&
+		if (flow_has_room(env, *nr_states) &&
 		    (flow_unset_differs(&t->st, &s->st) ||
-		     flow_join_makes_up(flow, &t->st, &s->st)))
+		     (!widen && flow_join_makes_up(flow, &t->st, &s->st))))
 			continue;
 		/*
 		 * Around a loop too:
@@ -2516,8 +2516,8 @@ static int flow_end_walk(struct bpf_verifier_env *env, struct flow_set *set, str
 
 	list_for_each_entry_reverse(t, &set->list, node) {
 		/*
-		 * The rules that hold for what comes around a loop, see flow_merge(). And one
-		 * more: the iteration has set a register, see flow_unset_differs(). The join
+		 * The rules that hold for what comes around a loop, see flow_merge(). One of
+		 * them: the iteration has set a register, see flow_unset_differs(). The join
 		 * would be without it.
 		 */
 		if (flow_trips_behind(&t->st, &s->st) ||
-- 
2.55.0

[PATCH bpf-next v3 31/45] bpf: A base that nothing is known about can be the context

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

test_tcp_custom_syncookie.c and test_cls_redirect.c keep the context in
a struct and read c->skb->data. A function that is not inlined and does

  void *data = (void *)(long)c->skb->data;
  void *end = (void *)(long)c->skb->data_end;

  if (data + (v6 ? 40 : 20) > end)
          return 0;
  return v6 ? *(u8 *)(data + 39) : *(u8 *)(data + 19);

loaded before states were joined, and is rejected now:

  invalid access to packet, off=19 size=1, R1(id=3,off=19,r=0)

c->skb is loaded through a pointer to the stack of the caller.
bpf_flow_compute_kinds() can't tell that it is the context, and took what
a load of less than 8 bytes through such base gives for a number. Nothing
kept 20 and 40 apart in front of 'r3 += r1' then. The same if the base is
the context on some of the paths only.

Ask is_valid_access() about such load also when the base is not known:
4 bytes at the offset of data, data_meta or data_end can be a pointer.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 6854e795e537..51c7cce57b49 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -708,13 +708,18 @@ static enum flow_kind flow_loaded_kind(struct bpf_verifier_env *env, const struc
 	/* a number that is good for a load is an address in arena */
 	if (test_bit(base, k->is[FLOW_IS_VALUE]))
 		return FLOW_IS_PLAIN;
-	/* only the context has pointers that are shorter than a register */
-	if (!test_bit(base, k->is[FLOW_IS_CTX]))
-		return size < BPF_REG_SIZE ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
+	if (size == BPF_REG_SIZE && !test_bit(base, k->is[FLOW_IS_CTX]))
+		return FLOW_IS_OTHER;
 	valid = env->ops->is_valid_access &&
 		env->ops->is_valid_access(insn->off, size, BPF_READ, env->prog, &info);
 	/* it notes what the prog reads, and this is a question */
 	env->prog->tstamp_type_access = tstamp;
+	/*
+	 * Only the context has pointers that are shorter than a register. A base that nothing is
+	 * known about can be the context: it came out of a struct, or only some paths have it.
+	 */
+	if (size < BPF_REG_SIZE)
+		return valid && info.reg_type != SCALAR_VALUE ? FLOW_IS_OTHER : FLOW_IS_PLAIN;
 	return valid && info.reg_type == SCALAR_VALUE ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
 }
 
-- 
2.55.0

[PATCH bpf-next v3 32/45] bpf: Keep the frames of the caller for a subprog that loads data_end

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  static __noinline bool has(struct xdp_md *ctx, void *p, u32 n)
  {
          return p + n <= (void *)(long)ctx->data_end;
  }

  void *p = (void *)(long)ctx->data;

  if (!has(ctx, p, 14))
          return XDP_DROP;
  return *(char *)(p + 13) ? XDP_PASS : XDP_DROP;

has() checks how much of the packet is there, for its caller. A static
subprog that is verified on its own has no registers of the caller, so
the range that the compare finds does not get to r6 of the caller:

  invalid access to packet, off=13 size=1, R6(id=0,off=13,r=0)

The prog loaded before static subprogs were verified on their own. It still
loads if data_end is an argument of has(): arg_needs_caller() keeps the
frames of the caller for such a call.

Do the same for a call that passes a pointer to the packet to a subprog
that loads data_end from the context. bpf_flow_compute_kinds() asks
is_valid_access() about each load through the context, and notes it in
bpf_subprog_info.loads_pkt_end.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 2 ++
 kernel/bpf/flow.c            | 2 ++
 kernel/bpf/verifier.c        | 3 ++-
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 78a787aaba44..4c0939d91fde 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1020,6 +1020,8 @@ struct bpf_subprog_info {
 	bool stack_args_from_code: 1;
 	/* it releases what its caller holds, so it's verified in the context of such caller */
 	bool releases_inherited: 1;
+	/* it loads the end of the packet from the context */
+	bool loads_pkt_end: 1;
 	/* a call of it was verified on its own: its insns are not in insns_total of the caller */
 	bool on_its_own: 1;
 	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 51c7cce57b49..a14490511a25 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -714,6 +714,8 @@ static enum flow_kind flow_loaded_kind(struct bpf_verifier_env *env, const struc
 		env->ops->is_valid_access(insn->off, size, BPF_READ, env->prog, &info);
 	/* it notes what the prog reads, and this is a question */
 	env->prog->tstamp_type_access = tstamp;
+	if (valid && info.reg_type == PTR_TO_PACKET_END)
+		bpf_find_containing_subprog(env, insn - env->prog->insnsi)->loads_pkt_end = true;
 	/*
 	 * Only the context has pointers that are shorter than a register. A base that nothing is
 	 * known about can be the context: it came out of a struct, or only some paths have it.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 2a0d5a280a1a..1fa37faa8e69 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11602,7 +11602,8 @@ static bool arg_needs_caller(const struct bpf_verifier_state *state,
 	if (reg_is_pkt_pointer(reg) && reg->id)
 		return true;
 	/* it checks how much of the packet is there, for the caller */
-	if (reg->type == PTR_TO_PACKET_END && !sub->recursive)
+	if ((reg->type == PTR_TO_PACKET_END ||
+	     (reg->type == PTR_TO_PACKET && sub->loads_pkt_end)) && !sub->recursive)
 		return true;
 	/* or whether there is something to point to */
 	return type_may_be_null(reg->type) && !sub->recursive;
-- 
2.55.0

Re: [PATCH bpf-next v3 32/45] bpf: Keep the frames of the caller for a subprog that loads data_end

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4619 bytes --]

> bpf: Keep the frames of the caller for a subprog that loads data_end

TL;DR:
- loads_pkt_end isn't propagated to callers, so valid programs may be rejected
  when data_end is loaded through a nested subprog
- A subprog loading data and data_end without passing a packet pointer may
  also be rejected; testing it in func_is_on_its_own() is suggested

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c

[ ... ]

> @@ -714,6 +714,8 @@ static enum flow_kind flow_loaded_kind(struct bpf_verifier_env *env, const struc
>  		env->ops->is_valid_access(insn->off, size, BPF_READ, env->prog, &info);
>  	/* it notes what the prog reads, and this is a question */
>  	env->prog->tstamp_type_access = tstamp;
> +	if (valid && info.reg_type == PTR_TO_PACKET_END)
> +		bpf_find_containing_subprog(env, insn - env->prog->insnsi)->loads_pkt_end = true;

loads_pkt_end is set only on the subprog whose insn loads data_end, and
nothing passes it up the call graph.  The sibling flags changes_pkt_data,
might_sleep and might_throw are propagated to every caller by
merge_callee_effects() in bpf_check_cfg().

Can the program from the commit message still be rejected if has() is
called through one more static subprog?

  static __noinline bool has(struct xdp_md *ctx, void *p, u32 n)
  {
          return p + n <= (void *)(long)ctx->data_end;
  }
  static __noinline int parse(struct xdp_md *ctx, void *p)
  {
          return has(ctx, p, 14);
  }
  ...
  void *p = (void *)(long)ctx->data;
  if (!parse(ctx, p))
          return XDP_DROP;
  return *(char *)(p + 13) ? XDP_PASS : XDP_DROP;

At the call of parse(), arg_needs_caller() sees r1 as PTR_TO_CTX and r2 as
PTR_TO_PACKET with id 0.  parse()->loads_pkt_end is false, so parse() is
verified on its own with only its own frame.

Inside that pass has() keeps the frames, but find_good_pkt_pointers() can
only widen the registers of parse() and has().  The main prog's r6 is not
in that state, and the code that rebuilds the caller's state from the exit
state copies back only R0 and the references.  So r6 keeps r=0 and the load
fails with the same error as in the commit message:

  invalid access to packet, off=13 size=1, R6(id=0,off=13,r=0)

Before the series, static subprogs were always verified in the caller's
frames, so this program loaded.

loads_pkt_end is computed in bpf_flow_compute_kinds(), after
bpf_check_cfg() has run merge_callee_effects(), so propagating it would
need its own pass after the kinds are computed, for example over
env->subprog_topo_order, callees first.

[ ... ]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -11602,7 +11602,8 @@ static bool arg_needs_caller(const struct bpf_verifier_state *state,
>  	if (reg_is_pkt_pointer(reg) && reg->id)
>  		return true;
>  	/* it checks how much of the packet is there, for the caller */
> -	if (reg->type == PTR_TO_PACKET_END && !sub->recursive)
> +	if ((reg->type == PTR_TO_PACKET_END ||
> +	     (reg->type == PTR_TO_PACKET && sub->loads_pkt_end)) && !sub->recursive)
>  		return true;
>  	/* or whether there is something to point to */
>  	return type_may_be_null(reg->type) && !sub->recursive;

This condition keeps the caller's frames only if a PTR_TO_PACKET is one of
the arguments, but find_good_pkt_pointers() widens every register in every
frame with the same type and id as dst_reg.  A subprog that loads both
data and data_end therefore widens every id-0 packet pointer the caller
holds, whether or not it was passed:

  static __noinline bool has(struct xdp_md *ctx, u32 n)
  {
          return (void *)(long)ctx->data + n <= (void *)(long)ctx->data_end;
  }
  ...
  void *p = (void *)(long)ctx->data;
  if (!has(ctx, 14))
          return XDP_DROP;
  return *(char *)(p + 13) ? XDP_PASS : XDP_DROP;

Here has()->loads_pkt_end is set, but its live arguments are r1 as
PTR_TO_CTX and r2 as a scalar, so arg_needs_caller() returns false for
each of them.  has() is then verified without main's frames, the compare
does not reach main's r6, and p + 13 is rejected.  Before the series this
program loaded.

Would it be better to test loads_pkt_end once in func_is_on_its_own(), for
example:

  if (sub->loads_pkt_end && !sub->recursive)
          return false;

rather than ANDing it with the type of each argument?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 33/45] bpf: Keep the frames of the caller for ctx of struct_ops with a reference

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  static __noinline int pid_of(unsigned long long *ctx)
  {
          struct task_struct *task = (void *)ctx[1];

          return task->pid;
  }

  SEC("struct_ops/test_refcounted")
  int refcounted(unsigned long long *ctx)
  {
          int pid = pid_of(ctx);

          bpf_task_release((void *)ctx[1]);
          return pid & 1;
  }

An argument with __ref, like the skb of Qdisc_ops.enqueue, is a reference
that the prog holds from the start. __check_ctx_access() looks for the id
in ctx_arg_info[] among the references of the state. A static subprog that
is verified on its own has names in place of these ids:

  invalid bpf_context access off=8. Reference may already be released

The prog loaded before static subprogs were verified on their own. Keep the
frames of the caller when the context of a struct_ops prog is passed while
a reference is held.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/verifier.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1fa37faa8e69..8e7875e8bebf 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11586,9 +11586,13 @@ static u32 inherited_id(const struct bpf_verifier_state *state, u32 id)
 }
 
 /* What func_is_on_its_own() asks of an argument, be it in a register or on the stack */
-static bool arg_needs_caller(const struct bpf_verifier_state *state,
+static bool arg_needs_caller(struct bpf_verifier_env *env, const struct bpf_verifier_state *state,
 			     const struct bpf_subprog_info *sub, const struct bpf_reg_state *reg)
 {
+	/* the id of a __ref argument is kept in ctx_arg_info[], see __check_ctx_access() */
+	if (reg->type == PTR_TO_CTX && state->acquired_refs &&
+	    env->prog->type == BPF_PROG_TYPE_STRUCT_OPS)
+		return true;
 	/* only the ids of what is held have names */
 	if (reg->type != SCALAR_VALUE &&
 	    ((reg->id && !inherited_id(state, reg->id)) ||
@@ -11637,7 +11641,7 @@ static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier
 		if (caller->regs[i].type == PTR_TO_STACK ||
 		    base_type(caller->regs[i].type) == CONST_PTR_TO_DYNPTR)
 			return false;
-		if (arg_needs_caller(state, sub, &caller->regs[i]))
+		if (arg_needs_caller(env, state, sub, &caller->regs[i]))
 			return false;
 	}
 	/* callx of one function of a set asks before it is checked that the arguments are there */
@@ -11645,7 +11649,7 @@ static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier
 		if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
 		    base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
 			return false;
-		if (arg_needs_caller(state, sub, &caller->stack_arg_regs[i]))
+		if (arg_needs_caller(env, state, sub, &caller->stack_arg_regs[i]))
 			return false;
 	}
 	return true;
-- 
2.55.0

Re: [PATCH bpf-next v3 33/45] bpf: Keep the frames of the caller for ctx of struct_ops with a reference

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3188 bytes --]

> bpf: Keep the frames of the caller for ctx of struct_ops with a reference

TL;DR:
- Missing !sub->recursive check may make the verifier reject a recursive
  subprog taking struct_ops ctx while a ref is held (frame-depth E2BIG)
- Condition is broader than the __ref case: it fires for any held ref,
  including sched_ext

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -11586,9 +11586,13 @@ static u32 inherited_id(const struct bpf_verifier_state *state, u32 id)
>  /* What func_is_on_its_own() asks of an argument, be it in a register or on the stack */
> -static bool arg_needs_caller(const struct bpf_verifier_state *state,
> +static bool arg_needs_caller(struct bpf_verifier_env *env, const struct bpf_verifier_state *state,
>  			     const struct bpf_subprog_info *sub, const struct bpf_reg_state *reg)
>  {
> +	/* the id of a __ref argument is kept in ctx_arg_info[], see __check_ctx_access() */
> +	if (reg->type == PTR_TO_CTX && state->acquired_refs &&
> +	    env->prog->type == BPF_PROG_TYPE_STRUCT_OPS)
> +		return true;

Should this case also be limited by !sub->recursive, like the neighbouring
cases that keep the caller?

	if ((reg->type == PTR_TO_PACKET_END ||
	     (reg->type == PTR_TO_PACKET && sub->loads_pkt_end)) && !sub->recursive)
		return true;
	/* or whether there is something to point to */
	return type_may_be_null(reg->type) && !sub->recursive;

When func_is_on_its_own() returns false, check_static_func_call() calls
setup_func_entry(), which pushes a new frame.  A recursive subprog that
gets ctx and calls itself with ctx again still sees state->acquired_refs
as non-zero, so every level of the recursion keeps the frames again, and
states with a different curframe are never compared or joined.

If the depth is not bounded by a constant, the descent only stops at the
check in setup_func_entry():

	if (state->curframe + 1 >= MAX_CALL_FRAMES) {
		verbose(env, "the call stack of %d frames is too deep\n", ...);
		return -E2BIG;

For example, a test_refcounted struct_ops prog holds task__ref from the
start, and calls:

	static __noinline int depth(unsigned long long *ctx, int n)
	{
		return n > 0 ? depth(ctx, n - 1) : ctx[0] & 1;
	}

with an unknown n.  depth() only reads the plain int at ctx[0].  Before
this patch it was verified once on its own and the prog loaded.  Can this
now be rejected?

The condition is also wider than the __ref case the commit message
describes.  It fires for any BPF_PROG_TYPE_STRUCT_OPS prog that holds any
entry in refs[], including bpf_spin_lock and IRQ-flag entries, since
acquired_refs counts all REF_TYPE_* kinds.  That includes sched_ext
programs, which have no __ref arguments, so ctx_arg_info[] never holds a
ref_id there.

Would it be better to add && !sub->recursive, and to check whether a
ctx_arg_info[i].ref_id is still among state->refs rather than testing
acquired_refs?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 34/45] bpf: R0 is live at the exit that an added may_goto goes to

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  const volatile bool debug;

  if (debug)
          return skb->len + 7;
  while (*p)
          ;
  return 0;

GCC emits an exit for each return, and puts the one of 'if (debug)' last.
With debug = 0 it is dead code.

The verifier adds may_goto to a loop that it cannot count. If the prog
holds nothing there, may_goto ends it with 'r0 = 0; exit'. Until these
insns are added an exit of the subprog stands for them in the main pass:
bail_insn, the last one. Liveness does not visit dead code, so no register
is live there. r0 = 0 is cleaned from the state that gets to it:

  R0 !read_ok

Mark r0, and r2 if the subprog returns a pair, as live at bail_insn.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/liveness.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index c8f74063913c..e930d928afa5 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2880,6 +2880,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		 */
 		insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
 	}
+	/* the exit that stands for 'r0 = 0; exit' may be dead code, see push_budget_exit() */
+	for (i = 0; i < env->subprog_cnt; i++)
+		if (env->subprog_info[i].bail_insn >= 0)
+			insn_aux[env->subprog_info[i].bail_insn].live_regs_before |=
+				BIT(BPF_REG_0) | (bpf_ret_reg_pair(env, i) ? BIT(BPF_REG_2) : 0);
 
 	/*
 	 * Forward pass: resolve stack access through FP-derived pointers.
-- 
2.55.0

[PATCH bpf-next v3 35/45] bpf: Keep the step of a number in the union of two ranges

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  u32 n = x & 15;

  for (i = n; i >= 0; i--)
          sum += a[i];

clang counts in bytes and leaves the loop one element below the array:

     r2 <<= 2			; off = n * 4
  1: r3 = a ll
     r3 += r2
     r3 = *(u32 *)(r3 + 0)
     w1 += w3
     r2 += -4
     if r2 != -4 goto 1b

Scalar evolution does not count a loop that '!=' ends when the start is
a range and the step is not 1. So the states at the loop head are widened.
The offset does not get past -4 because it is a multiple of 4. The union
of two ranges forgets that, and a prog that loaded before is rejected:

  math between map_value pointer and register with unbounded min value

a + s * k and b + t * k are both of the form a + gcd(s, t, a - b) * k.
Compute that step in bpf_reg_union_range(). Widening drops it when a bound
moves. A step gives way to one of its divisors only, so it needs no
widening of its own.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/verifier.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8e7875e8bebf..83baaf1a6f2a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -18425,7 +18425,7 @@ void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *
 		     reg->r32.base != old->r32.base || reg->r32.size != old->r32.size;
 	s64 min = reg_smin(reg), max = reg_smax(reg);
 
-	/* Only the step is gone, see bpf_reg_union_range() */
+	/* Only the step got weaker. It comes to rest without help, see below. */
 	if (!moved && reg->var_off.value == old->var_off.value &&
 	    reg->var_off.mask == old->var_off.mask)
 		return;
@@ -18462,14 +18462,36 @@ void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *
 		reg->var_off = min >= 0 ? tnum_union(reg->var_off, tnum_range(min, max))
 					: tnum_unknown;
 	}
+	/*
+	 * No range moved: the known bits or the step got weaker. The step comes to rest without
+	 * help: it gives way to one of its divisors only.
+	 *	off = n * 4; do { sum += *(u32 *)(a + off); off -= 4; } while (off != -4);
+	 * counts on "a multiple of 4".
+	 */
+	if (moved)
+		reg_step_reset(reg);
 }
 
 void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src)
 {
+	bool acc_const = cnum64_is_const(acc->r64), src_const = cnum64_is_const(src->r64);
+	/* A constant is its own base. Its step is 0: every number divides 0. */
+	s64 a = acc_const ? cnum64_smin(acc->r64) : acc->base;
+	s64 b = src_const ? cnum64_smin(src->r64) : src->base;
+	u16 step = gcd(acc_const ? 0 : acc->step, src_const ? 0 : src->step);
+	s64 diff;
+
 	acc->r64 = cnum64_union(acc->r64, src->r64);
 	acc->r32 = cnum32_union(acc->r32, src->r32);
 	acc->var_off = tnum_union(acc->var_off, src->var_off);
-	reg_step_reset(acc);
+
+	/* a + s * k and b + t * k are both of the form a + gcd(s, t, a - b) * k */
+	if (check_sub_overflow(a, b, &diff) || (!step && (diff > U16_MAX || diff < -U16_MAX)))
+		step = 1;
+	else
+		step = (step ? gcd(step, imod(diff, step)) : abs(diff)) ?: 1;
+	acc->step = step;
+	acc->base = imod(a, step);
 	/*
 	 * No reg_bounds_sync(): every part of the result has to include that
 	 * part of both inputs, and an input is not always in sync.
-- 
2.55.0

[PATCH bpf-next v3 36/45] bpf: Keep the first iteration of a counted loop apart

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  idx = 0;
  for (i = 0; i < 100000; i++)
          idx = table[idx];       /* u8 table[256] */

Scalar evolution has the count and no formula for idx. If something looks
at such a register, like the load here, the verifier goes through the
iterations one by one. 100000 iterations are too many for that. Then idx is
not known in the loop, and a prog that loaded before is rejected:

  math between map_value pointer and register with unbounded min value

But nothing that idx is made from is what a register had at the loop head.
It is a byte in each iteration. Only the first one differs: it has the 0
from the entry.

Verify the first iteration with the state that enters the loop. The state
that returns to the loop head has idx in [0, 255]. Give its induction
variables the ranges that follow from the count, as at the entry. It stands
for all other iterations. What returns next must be within it, otherwise
the two are joined like at each loop head. The loop takes 12 insns.

expr_is_fresh() finds the registers, bpf_loop_iters.fresh_regs has them,
loop_stack_entry.first marks a state that is in the first iteration.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 15 ++++++--
 kernel/bpf/flow.c            |  4 +--
 kernel/bpf/join.c            |  1 +
 kernel/bpf/scev.c            | 68 ++++++++++++++++++++++++++++++++++--
 kernel/bpf/states.c          |  1 +
 kernel/bpf/verifier.c        | 33 ++++++++++++++++-
 6 files changed, 114 insertions(+), 8 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 4c0939d91fde..de2c5aa1567d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -351,6 +351,11 @@ struct bpf_loop_iters {
 	u32 max_header_count;   /* max number of times header is executed */
 	/* live, changed by the loop, and there is no summary of how */
 	unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/*
+	 * live, made anew in every iteration: the first one tells what it is in the rest.
+	 * Or not set at the entry, and also in forget_regs: the first one sets it.
+	 */
+	unsigned long fresh_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* [0, max] at the entry, and the loop only clears bits */
 	unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/*
@@ -395,6 +400,11 @@ struct loop_stack_entry {
 	 * flow_join_makes_up(). The test of the flag is decided in both, and decides nothing.
 	 */
 	u32 flag_apart:1;
+	/*
+	 * The first iteration of a counted loop is kept apart. The state that comes back
+	 * to the head is made to stand for the rest.
+	 */
+	u32 first:1;
 	/*
 	 * How many times the state came back to the head of the loop. With @apart that is
 	 * the iteration that it is in. A state that stands for many iterations is a join, and has
@@ -416,7 +426,7 @@ struct loop_stack_entry {
 /* The state is in a walk on patience, which may not get to the end of the loop */
 static inline bool bpf_loop_walk_may_give_up(const struct loop_stack_entry *entry)
 {
-	return entry->apart && entry->terminates && !entry->counted;
+	return entry->apart && entry->terminates && !entry->counted && !entry->first;
 }
 
 #define LOOP_STACK_SIZE 16
@@ -2161,7 +2171,8 @@ int bpf_compute_scev(struct bpf_verifier_env *env);
 int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
 			   struct bpf_loop_iters *iters);
 int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
-			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters);
+			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters,
+			bool back);
 int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 insn_idx,
 			const struct loop_stack_entry *entry);
 
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index a14490511a25..24e7f99277eb 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1891,8 +1891,8 @@ static void flow_shape_ctx(const struct bpf_verifier_state *st, struct flow_shap
 		for (i = 0; i < frame->loop_stack_cnt; i++) {
 			entry = &frame->loop_stack[i];
 			chain = jhash_1word(entry->loop_id << 1 | entry->apart, chain);
-			ctx = jhash_3words(entry->counted << 2 | entry->guarded << 1 |
-					   entry->terminates,
+			ctx = jhash_3words(entry->counted << 3 | entry->guarded << 2 |
+					   entry->first << 1 | entry->terminates,
 					   entry->max_header_count,
 					   entry->counted ? entry->trip : 0, ctx);
 		}
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 0110e5f85003..88fe3d32a6de 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -415,6 +415,7 @@ static bool join_loop_stack(const struct bpf_func_state *fa, const struct bpf_fu
 		    fa->loop_stack[i].guarded != fb->loop_stack[i].guarded ||
 		    fa->loop_stack[i].counted != fb->loop_stack[i].counted ||
 		    fa->loop_stack[i].apart != fb->loop_stack[i].apart ||
+		    fa->loop_stack[i].first != fb->loop_stack[i].first ||
 		    (fa->loop_stack[i].apart && fa->loop_stack[i].trip != fb->loop_stack[i].trip) ||
 		    fa->loop_stack[i].refs != fb->loop_stack[i].refs ||
 		    fa->loop_stack[i].max_header_count != fb->loop_stack[i].max_header_count ||
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 147a2fa9aea9..0cbf158dae6d 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1966,7 +1966,7 @@ scev_reg_for_read(struct bpf_verifier_env *env, const struct bpf_func_state *st,
  * This does not go through check_stack_write(). The buffer of a live dynptr slice, see
  * bpf_reg_state::buf, does not become a spill here. The callers skip the slots that are
  * is_unbound_scev_reg() when the loop is entered. A slot that a call in the loop may write
- * has no description: it is in forget_regs, and they skip these too.
+ * has no description: it is in forget_regs or fresh_regs, and they skip these too.
  */
 static int scev_reg_for_write(struct bpf_verifier_env *env, struct bpf_func_state *st,
 			      u32 r, struct bpf_reg_state **out)
@@ -3194,6 +3194,31 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
 		bounds->r64 = r64;
 }
 
+/*
+ * Nothing that @id is computed from is what a register had at the head of the loop, and
+ * some of it is not known: it's made anew in every iteration.
+ */
+static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
+{
+	struct scev *scev = env->scev;
+	bool opaque = false;
+	u32 order, regno;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if (order & DEPTH_LIMIT)
+			return false;
+		if (!(order & PRE))
+			continue;
+		if (is_reg(scev, id, &regno) || is_op(scev, id, UNKNOWN))
+			return false;
+		if (is_opaque(scev, id))
+			opaque = true;
+	}
+	return opaque;
+}
+
 /*
  * Does @id have no bit that register @r doesn't have at the head of the loop? 'x & e' and the
  * low bits of x lose bits only. 'x >> e', 'x / e' and 'x % e' are not greater than x as well, but
@@ -3407,6 +3432,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 		return err;
 
 	bitmap_zero(iters->forget_regs, REGS_NUM);
+	bitmap_zero(iters->fresh_regs, REGS_NUM);
 	bitmap_zero(iters->mask_regs, REGS_NUM);
 	iters->spill_base = false;
 	header_env = find_header_env(scev, insn_idx);
@@ -3520,7 +3546,23 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 			walk = true;
 			continue;
 		}
+		/*
+		 * What is loaded from memory in every iteration is the same kind of thing in all
+		 * of them. One iteration tells what. There is nothing more to find out when
+		 * iterations are looked at one by one.
+		 */
+		if (expr_is_fresh(env, header_env->reg2expr[r])) {
+			__set_bit(r, iters->fresh_regs);
+			continue;
+		}
 		__set_bit(r, iters->forget_regs);
+		/*
+		 *	for (i = 0; i < 40; i++) { if (i) sum += prev; prev = (x ^ i) * 3; }
+		 * It's not set in the first iteration only. That counts if the loop is not walked:
+		 * nothing looks at prev, or there are too many iterations.
+		 */
+		if (reg && reg->type == NOT_INIT)
+			__set_bit(r, iters->fresh_regs);
 		/*
 		 * Only a walk through the iterations tells what it is in each of them. That is not
 		 * worth it for a number that nothing looks at: a sum, a hash, what a call leaves in
@@ -3624,13 +3666,18 @@ static int union_any_reg(struct bpf_verifier_env *env, struct bpf_func_state *lo
 /*
  * Make @st, which is at the head of a counted loop, stand for all iterations. @loop_entry is the
  * state that entered the loop and @iters is what bpf_compute_loop_iters() says about it.
- * @st is a copy of @loop_entry.
+ *
+ * @st is a copy of @loop_entry, or, if the loop has iters->fresh_regs, the state that comes back to
+ * the head after the first iteration, @back. These registers are left as the first iteration
+ * made them. At the entry they are something else: 0, a pointer, not set.
  */
 int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
-			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters)
+			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters,
+			bool back)
 {
 	struct bpf_func_state *cur_func = st->frame[st->curframe];
 	struct bpf_func_state *entry_func = loop_entry->frame[loop_entry->curframe];
+	const struct bpf_reg_state *val, *entry_val;
 	struct bpf_verifier_log *log = &env->log;
 	struct scev *scev = env->scev;
 	struct bpf_reg_state *reg, *spill32;
@@ -3650,6 +3697,8 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 			scev_forget_reg(env, cur_func, r);
 			continue;
 		}
+		if (test_bit(r, iters->fresh_regs))
+			continue;
 
 		r_expr = header_env->reg2scev[r];
 		spill32 = spill32_counter(scev, cur_func, r_expr, r, &coefs);
@@ -3658,9 +3707,19 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 				 is_const_scev_reg(env, insn_idx, r)))
 			continue;
 
+		/*
+		 * A call in the loop can change the type, see bpf_clamp_scev_regs(). Then the
+		 * register is left alone. What comes back to the head next is compared with it.
+		 */
+		val = scev_reg_for_read(env, cur_func, r);
+		entry_val = scev_reg_for_read(env, entry_func, r);
+		if (back && (!val || !entry_val || !bpf_reg_union_compatible(val, entry_val)))
+			continue;
 		if (test_bit(r, iters->mask_regs)) {
 			/* The range stays. It's no longer equal to the registers with its id. */
 			err = scev_reg_for_write(env, cur_func, r, &reg);
+			if (!err && back)
+				err = bpf_reg_union(env, reg, entry_val);
 			if (err)
 				return err;
 			if (log->level & BPF_LOG_LEVEL2) {
@@ -3676,6 +3735,9 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 
 		/* If SCEV for r is (linear <reg> <slope>)*/
 		if (spill32 || (is_simple_linear(scev, r_expr, &coefs) && coefs.base_reg == r)) {
+			/* the range is derived from @loop_entry: bpf_clamp_scev_regs() */
+			if (back)
+				continue;
 			/* the other half of the slot stays as it is */
 			reg = spill32;
 			err = reg ? 0 : scev_reg_for_write(env, cur_func, r, &reg);
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 07b1f0becd2e..efc0870b85c2 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -646,6 +646,7 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
 		    old_stack[i].guarded == cur_stack[i].guarded &&
 		    old_stack[i].counted == cur_stack[i].counted &&
 		    old_stack[i].apart == cur_stack[i].apart &&
+		    old_stack[i].first == cur_stack[i].first &&
 		    (!old_stack[i].apart ||
 		     (old_stack[i].probe ? old_stack[i].trip <= cur_stack[i].trip :
 					   old_stack[i].trip == cur_stack[i].trip)))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 83baaf1a6f2a..06265d617f4c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21488,6 +21488,26 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 			*backedge = true;
 			return check_refs_taken_in_loop(env, entry) ?: maybe_clamp_scev_regs(env);
 		}
+		/*
+		 * The first iteration is over. What the loop makes anew in every iteration is
+		 * in this state. The rest is what the count says, like at the entry.
+		 */
+		if (entry->first) {
+			entry->first = false;
+			entry->apart = false;
+			entry->trip++;
+			err = bpf_compute_loop_iters(env, entry->entry_state, &iters);
+			if (err < 0)
+				return err;
+			if (verifier_bug_if(!err, env, "loop at %d is not counted any more",
+					    env->insn_idx))
+				return -EFAULT;
+			err = maybe_clamp_scev_regs(env);
+			if (err)
+				return err;
+			err = bpf_widen_scev_regs(env, cur, entry->entry_state, &iters, true);
+			return err < 0 ? err : 0;
+		}
 		loop = env->insn_aux_data[env->insn_idx].loop;
 		/* the next iteration is another kind of state, there is no cycle */
 		trip = entry->trip;
@@ -21570,7 +21590,18 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	entry->iv_range = iters.iv_range;
 	entry->entry_state = entry_state;
 	entry->terminates = true;
-	err = bpf_widen_scev_regs(env, cur, entry_state, &iters);
+	/*
+	 *	idx = 0; for (i = 0; i < 100; i++) idx = table[idx];
+	 * What the loop makes of idx is not known before its body is looked at, and what idx is
+	 * at the entry goes with i == 0 only. So the state goes through the first iteration as
+	 * it is. The state that comes back to the head has idx in 0..255.
+	 */
+	if (!bitmap_empty(iters.fresh_regs, BPF_SCEV_REGS_NUM)) {
+		entry->first = true;
+		entry->apart = true;
+		return 0;
+	}
+	err = bpf_widen_scev_regs(env, cur, entry_state, &iters, false);
 	return err < 0 ? err : 0;
 }
 
-- 
2.55.0

Re: [PATCH bpf-next v3 36/45] bpf: Keep the first iteration of a counted loop apart

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3336 bytes --]

> bpf: Keep the first iteration of a counted loop apart

TL;DR: In bpf_compute_loop_iters(), treating loaded registers as fresh skips
the walk decision for short loops, so the verifier may wrongly reject valid
programs, such as a NULL-checked kptr loop.

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 147a2fa9aea9..0cbf158dae6d 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c

[ ... ]

> @@ -3520,7 +3546,23 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
>  			walk = true;
>  			continue;
>  		}
> +		/*
> +		 * What is loaded from memory in every iteration is the same kind of thing in all
> +		 * of them. One iteration tells what. There is nothing more to find out when
> +		 * iterations are looked at one by one.
> +		 */
> +		if (expr_is_fresh(env, header_env->reg2expr[r])) {
> +			__set_bit(r, iters->fresh_regs);
> +			continue;
> +		}
>  		__set_bit(r, iters->forget_regs);

Does this continue skip the walk decision for short loops?

Before this patch, a live loop-carried register without a summary went into
forget_regs, and if bpf_flow_is_needed() wanted it, walk was set to true and
bpf_compute_loop_iters() returned 2, so the loop was walked iteration by
iteration when it was short enough.

Now a register whose reg2expr is only OPAQUE (any load that is not an fp
spill fill, for example) never sets walk, so the function returns 1 even
for a short loop.  The comment says there is nothing more to find out when
iterations are looked at one by one, but that doesn't seem to hold when the
loaded value is combined with other state.  For example:

    struct foo *t = NULL; int have = 0;
    for (i = 0; i < 4; i++) {
            if (have) sum += t->x;
            t = v->kptr;              /* reg2expr OPAQUE -> fresh */
            have = 0;
            if (t && t->y > 100) have = 1;   /* reg2scev ANY(have, 0, 1) */
    }

Before, t is in forget_regs and used as a load base, so walk = true and the
loop is walked with exact per-iteration states.  After, t is fresh, have is
a widenable ANY, and err == 1, so entry->first is set.  The state coming
back from the NULL path (t = scalar 0, have = 0) goes through
bpf_widen_scev_regs(..., back = true), where union_any_reg() makes have
[0, 1] while t is still scalar 0.  The next iteration takes "if (have)" and
dereferences a scalar, so the program is rejected.

A loop that loads from a frozen read-only map at an IV-dependent offset
(cur = tbl[i], used in the next iteration) seems to lose precision the same
way: the walk keeps each folded constant, while the summary folds only the
first iteration.

The follow-up commit 01f19b3a79be ("bpf: What a call returns in each
iteration is like what the loop loads") makes call results fresh but keeps
walking for them: "A call can return NULL in one iteration and set a flag
with it, so a loop that is short enough is still walked first."  Should
loads do the same, setting fresh_regs and still falling through to the
bpf_flow_is_needed() / walk decision, so only loops too long to walk get
the first-iteration split?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 37/45] bpf: What a call returns in each iteration is like what the loop loads

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  for (i = 0; i < 20000; i++) {
          sum += a[x];
          x = bpf_get_prandom_u32() & 15;
  }

x is loop-carried: an iteration uses what the iteration before made. The
loop is counted, and too long to go through its iterations one by one.
Scalar evolution knows nothing about r0 after a call, so x is not known in
the loop. The prog loaded before states were joined:

  math between map_value pointer and register with unbounded min value

A value that the loop loads from memory in each iteration is handled: the
first iteration is verified on its own, and tells what the value is in the
others. What a call returns is made anew in each iteration in the same way.

Give r0 after a call the expression that a load gets, with a parameter that
tells the two apart. A call can return NULL in one iteration and set a flag
with it, so a loop that is short enough is still walked first.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/scev.c | 31 ++++++++++++++++++++-----------
 1 file changed, 20 insertions(+), 11 deletions(-)

diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 0cbf158dae6d..507235ad6d76 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -852,8 +852,11 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
 		break;
 	case BPF_JMP:
 	case BPF_JMP32:
-		if (opcode == BPF_CALL)
+		if (opcode == BPF_CALL) {
 			forget_call_regs(e);
+			/* made anew, like what is loaded. The parameter tells the two apart. */
+			return replace_reg(scev, e, BPF_REG_0, expr1(scev, OPAQUE, 1));
+		}
 		/* for non-CALL there are no changes in register states */
 		break;
 	case BPF_LD:
@@ -3196,14 +3199,15 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
 
 /*
  * Nothing that @id is computed from is what a register had at the head of the loop, and
- * some of it is not known: it's made anew in every iteration.
+ * some of it is not known: it's made anew in every iteration. @call: by a call.
  */
-static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
+static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id, bool *call)
 {
 	struct scev *scev = env->scev;
 	bool opaque = false;
 	u32 order, regno;
 
+	*call = false;
 	scev->stack_sz = 0;
 	expr_stack_push(scev, id);
 	while (expr_next(scev, &id, &order)) {
@@ -3213,8 +3217,10 @@ static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
 			continue;
 		if (is_reg(scev, id, &regno) || is_op(scev, id, UNKNOWN))
 			return false;
-		if (is_opaque(scev, id))
+		if (is_opaque(scev, id)) {
 			opaque = true;
+			*call |= scev->exprs[id].params[0];
+		}
 	}
 	return opaque;
 }
@@ -3344,7 +3350,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	struct bpf_loop *loop;
 	int insn_idx = st->insn_idx;
 	u64 until, false_until;
-	bool same = true, walk = false;
+	bool same = true, walk = false, call;
 	int err, i;
 	u32 r;
 
@@ -3547,15 +3553,18 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 			continue;
 		}
 		/*
-		 * What is loaded from memory in every iteration is the same kind of thing in all
-		 * of them. One iteration tells what. There is nothing more to find out when
-		 * iterations are looked at one by one.
+		 * What is loaded from memory or returned by a call in every iteration is the same
+		 * kind of thing in all of them. One iteration tells what. There is nothing more
+		 * to find out when iterations are looked at one by one. But for what goes with it:
+		 * a call returns NULL and a flag is set. So a walk comes first for a call.
 		 */
-		if (expr_is_fresh(env, header_env->reg2expr[r])) {
+		if (expr_is_fresh(env, header_env->reg2expr[r], &call)) {
 			__set_bit(r, iters->fresh_regs);
-			continue;
+			if (!call)
+				continue;
+		} else {
+			__set_bit(r, iters->forget_regs);
 		}
-		__set_bit(r, iters->forget_regs);
 		/*
 		 *	for (i = 0; i < 40; i++) { if (i) sum += prev; prev = (x ^ i) * 3; }
 		 * It's not set in the first iteration only. That counts if the loop is not walked:
-- 
2.55.0

[PATCH bpf-next v3 38/45] bpf: Find where the jumps of a loop stop a number

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  bpf_for(i, 0, 100)
          bpf_for(j, 0, 100)
                  if (n < 8)
                          v->a[n++] = j;
  return v->a[n];

  while (off < 8000) { l = v->big[off]; if (!l) break; off += l; }

n and off grow by what the data say, so there is no formula for them.
Widening moves the bound that grows to a fixed stop: 0, S32_MAX, U32_MAX or
S64_MAX. n is [0, S32_MAX] at the loop head. The jump limits it in the
loop, nothing limits it after the loop. clang tests off at the end of the
loop, so nothing limits it at the load. Both loaded before states were
joined:

  R6 unbounded memory access, make sure to bounds check any such access

The jumps of the loop tell where the bound stops. Examine one iteration
before the main pass, from the head to the back edges, and write what is in
a register or a stack slot as 'root + constant'. A root is the value at the
head, or a value that the iteration makes. A jump that compares 'root + c'
with a constant limits the root on each edge that stays in the loop. What
returns to the head is 'root + c' again, and the limits give a stop: 8 and
7999 here. bpf_reg_widen_range() takes the nearest of all stops.

A stop is a guess. What returns to the head next is compared with the
widened state as always. A wrong stop costs one more pass through the loop.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  24 +-
 kernel/bpf/fixups.c          |   1 +
 kernel/bpf/flow.c            | 438 ++++++++++++++++++++++++++++++++++-
 kernel/bpf/join.c            |  25 +-
 kernel/bpf/verifier.c        |  36 ++-
 5 files changed, 504 insertions(+), 20 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index de2c5aa1567d..af10957af5ad 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -721,6 +721,23 @@ struct bpf_loop_exit {
 	int to; /* instruction outside the loop */
 };
 
+/* The places that the table of a loop knows: registers, then stack slots, fp-8 first */
+#define BPF_LOOP_SLOT	64
+#define BPF_LOOP_PLACES	128
+
+/*
+ * Where the bounds of the number in @place may come to rest at the head of a loop: the lowest and
+ * the highest upper bound, the highest and the lowest lower bound that the jumps of the loop
+ * let come back to the head, see flow_compute_rungs().
+ */
+struct bpf_loop_rungs {
+	s64 max[2];
+	s64 min[2];
+	bool has_max;
+	bool has_min;
+	u8 place;
+};
+
 struct bpf_loop {
 	struct bpf_backedge backedges[MAX_BACKEDGES];
 	/* edges exiting from this loop, includes edges from nested loops */
@@ -759,6 +776,10 @@ struct bpf_loop {
 	/* what gets a constant added in every iteration, in 64 and in 32 bits */
 	unsigned long linear_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	unsigned long linear32_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/* the places that insns of the loop write, as far as the insns tell */
+	unsigned long writes[BITS_TO_LONGS(BPF_LOOP_PLACES)];
+	struct bpf_loop_rungs *rungs;
+	u32 rungs_cnt;
 };
 
 struct bpf_insn_aux_data {
@@ -2012,7 +2033,8 @@ int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
 void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src);
 void bpf_reg_forget_range(struct bpf_reg_state *reg);
-void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear);
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old,
+			 const struct bpf_loop_rungs *rungs, int linear);
 int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
 		  const struct bpf_reg_state *src);
 
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 3413c7addca8..108b2aa924a9 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -928,6 +928,7 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
 	for (i = start; i < end; i++) {
 		if (aux_data[i].loop) {
 			kvfree(aux_data[i].loop->exits);
+			kvfree(aux_data[i].loop->rungs);
 			kvfree(aux_data[i].loop);
 			aux_data[i].loop = NULL;
 		}
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 24e7f99277eb..db2ea93d25b6 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1192,6 +1192,439 @@ bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r)
 	return false;
 }
 
+/*
+ * A number that grows from one iteration of a loop to the next is not followed step by step. The
+ * bound that moved is put where it may come to rest, see bpf_reg_widen_range(). The jumps of
+ * the loop tell where that is:
+ *	do { a[i] = 0; i += 3; } while (i < 30);		29 at most comes back to the head
+ *	while (i < 30) { a[i] = 0; i += 3; }			32
+ *	loop { if (...) { if (n >= 64) break; a[n++] = x; } }	64
+ *
+ * One iteration is looked at, from the head to the jumps back. What a register or a stack slot
+ * holds is written as root + constant. A root is what a place held at the head, or what was made
+ * in a place since. A copy, an extension, a spill and a fill keep the root. += and -= of a constant
+ * change the constant. Everything else makes a new root. Where paths meet two constants become
+ * a range of them, and two roots a new one.
+ * A jump that compares root + c with a constant bounds the root on each of its edges that stays in
+ * the loop. What comes back to the head in a place is root + c again: that, with the bounds of
+ * the root, is where the place may come to rest.
+ *
+ * It's a guess. The state that comes back to the head is compared with it like with any other
+ * state. So it's fine to take a 32-bit add for an add, to miss a store through a pointer, to mix up
+ * two numbers that are made in one place, and to know nothing of the edges that the verifier adds.
+ */
+struct flow_sym {
+	s32 lo, hi;
+	u8 root;
+};
+
+#define FLOW_NO_PLACE	U8_MAX
+/* How many of them the look at one loop may take. A loop that needs more has no table. */
+#define BPF_FLOW_MAX_SYMS	(4 << 20)
+
+static_assert(FLOW_SLOT == BPF_LOOP_SLOT && FLOW_PLACES == BPF_LOOP_PLACES);
+
+/* @insn makes the place *@to a copy of the place *@from, extended or cut to its low bits */
+static bool flow_copies_place(const struct bpf_insn *insn, u8 *to, u8 *from)
+{
+	u8 class = BPF_CLASS(insn->code);
+	int slot;
+
+	if (class == BPF_ALU || class == BPF_ALU64) {
+		*to = insn->dst_reg;
+		*from = insn->src_reg;
+		return flow_is_copy(insn) && *to < BPF_REG_FP && *from < BPF_REG_FP;
+	}
+	if (class != BPF_LDX && class != BPF_STX)
+		return false;
+	slot = flow_fp_slot(insn, class == BPF_LDX ? insn->src_reg : insn->dst_reg);
+	if (slot < 0)
+		return false;
+	*to = class == BPF_LDX ? insn->dst_reg : FLOW_SLOT + slot;
+	*from = class == BPF_LDX ? FLOW_SLOT + slot : insn->src_reg;
+	return *to != BPF_REG_FP && *from != BPF_REG_FP;
+}
+
+/* An ALU insn that leaves a small number as it is: an extension, or half of one */
+static bool flow_extends(const struct bpf_insn *insn)
+{
+	if (BPF_SRC(insn->code) != BPF_K)
+		return false;
+	if (flow_is_shift(insn))
+		return insn->imm == 32 || insn->imm == 48 || insn->imm == 56;
+	return BPF_OP(insn->code) == BPF_AND &&
+	       (insn->imm == 0xff || insn->imm == 0xffff || insn->imm == -1);
+}
+
+static void flow_written_places(const struct bpf_insn *insn, unsigned long *places)
+{
+	u8 class = BPF_CLASS(insn->code);
+	int p;
+
+	bitmap_zero(places, FLOW_PLACES);
+	if (class == BPF_ALU || class == BPF_ALU64 || class == BPF_LDX || bpf_is_ldimm64(insn)) {
+		__set_bit(insn->dst_reg, places);
+	} else if (class == BPF_LD || (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL)) {
+		bitmap_set(places, BPF_REG_0, BPF_REG_5 + 1);
+	} else if (class == BPF_ST || class == BPF_STX) {
+		p = bpf_atomic_load_reg(insn);
+		if (p >= 0)
+			__set_bit(p, places);
+		p = flow_mem_place(insn, insn->dst_reg);
+		if (p >= FLOW_SLOT)
+			__set_bit(p, places);
+	}
+}
+
+/*
+ * The jump at @idx compares the register *@reg with a constant: it's taken if "reg op k".
+ * The constant has 32 bits, so does what is added to it below: nothing wraps.
+ */
+static bool flow_jump_cmp(struct bpf_verifier_env *env, int idx, u8 *reg, u8 *op, s64 *k)
+{
+	const struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	const struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code);
+
+	*op = BPF_OP(insn->code);
+	if ((class != BPF_JMP && class != BPF_JMP32) || *op == BPF_JA || *op == BPF_CALL ||
+	    *op == BPF_EXIT || *op == BPF_JCOND || *op == BPF_JSET)
+		return false;
+	*reg = insn->dst_reg;
+	if (BPF_SRC(insn->code) == BPF_K) {
+		*k = insn->imm;
+	} else if (aux->const_reg_mask & BIT(insn->src_reg)) {
+		*k = aux->const_reg_vals[insn->src_reg];
+	} else if (aux->const_reg_mask & BIT(insn->dst_reg)) {
+		*k = aux->const_reg_vals[insn->dst_reg];
+		*reg = insn->src_reg;
+		*op = bpf_flip_opcode(*op);
+	} else {
+		return false;
+	}
+	if (class == BPF_JMP32)
+		*k = *op == BPF_JSGT || *op == BPF_JSGE || *op == BPF_JSLT || *op == BPF_JSLE ?
+		     (s64)(s32)*k : (s64)(u32)*k;
+	return *reg < BPF_REG_FP;
+}
+
+static void flow_add_max(struct bpf_loop_rungs *r, s64 max)
+{
+	r->max[0] = r->has_max ? min(r->max[0], max) : max;
+	r->max[1] = r->has_max ? max(r->max[1], max) : max;
+	r->has_max = true;
+}
+
+static void flow_add_min(struct bpf_loop_rungs *r, s64 min)
+{
+	r->min[0] = r->has_min ? max(r->min[0], min) : min;
+	r->min[1] = r->has_min ? min(r->min[1], min) : min;
+	r->has_min = true;
+}
+
+/* "x op k" holds. What does that say about the root of @x? */
+static void flow_note_cmp(struct bpf_loop_rungs *root, const struct flow_sym *x, u8 op, s64 k)
+{
+	bool has_max = true, has_min = true;
+	s64 max = k, min = k;
+
+	switch (op) {
+	case BPF_JEQ:
+		break;
+	case BPF_JNE:
+		/* it cuts off the end of a range */
+		max = k - 1;
+		min = k + 1;
+		break;
+	case BPF_JLT:
+		max = k - 1;
+		fallthrough;
+	case BPF_JLE:
+		/* the ranges are signed: nothing is said by a constant above S64_MAX */
+		min = 0;
+		has_max = max >= 0;
+		has_min = max >= 0;
+		break;
+	case BPF_JGT:
+		min = k + 1;
+		fallthrough;
+	case BPF_JGE:
+		has_max = false;
+		has_min = k >= 0;
+		break;
+	case BPF_JSLT:
+		max = k - 1;
+		fallthrough;
+	case BPF_JSLE:
+		has_min = false;
+		break;
+	case BPF_JSGT:
+		min = k + 1;
+		fallthrough;
+	case BPF_JSGE:
+		has_max = false;
+		break;
+	default:
+		return;
+	}
+	if (has_max)
+		flow_add_max(root, max - x->lo);
+	if (has_min)
+		flow_add_min(root, min - x->hi);
+}
+
+/* @cur is what the @n places hold in front of @insn. Make it what they hold behind it. */
+static void flow_sym_step(const struct bpf_insn *insn, struct flow_sym *cur, const u8 *idx, u32 n)
+{
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code), to, from;
+	DECLARE_BITMAP(written, FLOW_PLACES);
+	struct flow_sym *x;
+	s64 c = insn->imm;
+	int p;
+
+	/* both are among the places or none is, see flow_compute_loop_rungs() */
+	if (flow_copies_place(insn, &to, &from)) {
+		if (idx[to] != FLOW_NO_PLACE)
+			cur[idx[to]] = cur[idx[from]];
+		return;
+	}
+	if ((class == BPF_ALU || class == BPF_ALU64) && idx[insn->dst_reg] != FLOW_NO_PLACE) {
+		x = &cur[idx[insn->dst_reg]];
+		if (flow_extends(insn))
+			return;
+		if (op == BPF_SUB)
+			c = -c;
+		if ((op == BPF_ADD || op == BPF_SUB) && BPF_SRC(insn->code) == BPF_K &&
+		    x->lo + c >= S32_MIN && x->hi + c <= S32_MAX) {
+			x->lo += c;
+			x->hi += c;
+			return;
+		}
+	}
+	flow_written_places(insn, written);
+	for_each_set_bit(p, written, FLOW_PLACES)
+		if (idx[p] != FLOW_NO_PLACE)
+			cur[idx[p]] = (struct flow_sym){ .root = n + idx[p] };
+}
+
+/* One more path leads to where @to holds */
+static void flow_sym_meet(struct flow_sym *to, const struct flow_sym *from, u32 n)
+{
+	u32 j;
+
+	if (to[0].root == FLOW_NO_PLACE) {
+		memcpy(to, from, n * sizeof(*to));
+		return;
+	}
+	for (j = 0; j < n; j++) {
+		if (to[j].root != from[j].root) {
+			to[j] = (struct flow_sym){ .root = n + j };
+			continue;
+		}
+		to[j].lo = min(to[j].lo, from[j].lo);
+		to[j].hi = max(to[j].hi, from[j].hi);
+	}
+}
+
+static u8 flow_place_set(u8 *set, u8 p)
+{
+	while (set[p] != p) {
+		set[p] = set[set[p]];
+		p = set[p];
+	}
+	return p;
+}
+
+/* @at: the insns in the order. The insns of a loop are next to each other, the head first. */
+static int flow_compute_loop_rungs(struct bpf_verifier_env *env, const u32 *at, u32 total, int h)
+{
+	const struct bpf_insn *insns = env->prog->insnsi, *insn;
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop_rungs *roots = NULL, *rungs, *r;
+	u8 set[FLOW_PLACES], idx[FLOW_PLACES], to, from, reg, op;
+	u32 first = env->flow->pos[h], nr, n = 0, back_cnt = 0, cnt = 0, j, k;
+	DECLARE_BITMAP(tested, FLOW_PLACES) = {};
+	DECLARE_BITMAP(moves, FLOW_PLACES) = {};
+	struct flow_sym *syms, *cur, *back, *x;
+	struct bpf_loop *loop = aux[h].loop;
+	const u32 *pos = env->flow->pos;
+	struct bpf_iarray *succ;
+	int i, p, t, err = -ENOMEM;
+	bool carry = false;
+	s64 c;
+
+	nr = min(loop->nr_insns, total - first);
+
+	/*
+	 * The places to look at: what a jump compares with a constant, and what that is copied
+	 * from and to. One of them has to change in the loop.
+	 */
+	for (p = 0; p < FLOW_PLACES; p++)
+		set[p] = p;
+	for (k = 0; k < nr; k++) {
+		i = at[first + k];
+		if (flow_copies_place(&insns[i], &to, &from))
+			set[flow_place_set(set, to)] = flow_place_set(set, from);
+		else if (flow_jump_cmp(env, i, &reg, &op, &c))
+			__set_bit(reg, tested);
+	}
+	for_each_set_bit(p, tested, FLOW_PLACES)
+		__set_bit(flow_place_set(set, p), moves);
+	bitmap_copy(tested, moves, FLOW_PLACES);
+	bitmap_zero(moves, FLOW_PLACES);
+	for_each_set_bit(p, loop->writes, FLOW_PLACES)
+		__set_bit(flow_place_set(set, p), moves);
+	for (p = 0; p < FLOW_PLACES; p++) {
+		i = flow_place_set(set, p);
+		idx[p] = test_bit(i, tested) && test_bit(i, moves) ? n++ : FLOW_NO_PLACE;
+	}
+	if (!n || (u64)n * nr > BPF_FLOW_MAX_SYMS)
+		return 0;
+
+	/* for each insn, for the current insn, for each jump back */
+	syms = kvmalloc_array((size_t)nr + 1 + loop->backedges_cnt, n * sizeof(*syms),
+			      GFP_KERNEL_ACCOUNT);
+	/* for each root, and the result for each place */
+	roots = kvcalloc(3 * n, sizeof(*roots), GFP_KERNEL_ACCOUNT);
+	if (!syms || !roots)
+		goto out;
+	memset(syms, 0xff, (size_t)nr * n * sizeof(*syms));
+	cur = syms + (size_t)nr * n;
+	back = cur + n;
+	rungs = roots + 2 * n;
+	for (j = 0; j < n; j++)
+		syms[j] = (struct flow_sym){ .root = j };
+
+	for (k = 0; k < nr; k++) {
+		i = at[first + k];
+		insn = &insns[i];
+		if (!carry)
+			memcpy(cur, syms + (size_t)k * n, n * sizeof(*cur));
+		/* nothing but a jump back leads here */
+		for (j = 0; !carry && j < n; j++)
+			if (cur[j].root == FLOW_NO_PLACE)
+				cur[j] = (struct flow_sym){ .root = n + j };
+		/* a loop in the loop: nobody knows how many times it does what it does */
+		if (k && aux[i].loop)
+			for_each_set_bit(p, aux[i].loop->writes, FLOW_PLACES)
+				if (idx[p] != FLOW_NO_PLACE)
+					cur[idx[p]] = (struct flow_sym){ .root = n + idx[p] };
+
+		if (flow_jump_cmp(env, i, &reg, &op, &c) && idx[reg] != FLOW_NO_PLACE) {
+			x = &cur[idx[reg]];
+			if (pos[i + insn->off + 1] - first < nr)
+				flow_note_cmp(&roots[x->root], x, op, c);
+			if (pos[i + 1] - first < nr)
+				flow_note_cmp(&roots[x->root], x, bpf_rev_opcode(op), c);
+		}
+		flow_sym_step(insn, cur, idx, n);
+
+		succ = bpf_insn_successors(env, i);
+		carry = succ->cnt == 1 && !test_bit(succ->items[0], env->flow->point) &&
+			pos[succ->items[0]] == first + k + 1;
+		if (carry)
+			continue;
+		iarray_for_each(t, succ) {
+			if (t == h && back_cnt < loop->backedges_cnt)
+				memcpy(back + (size_t)back_cnt++ * n, cur, n * sizeof(*cur));
+			else if (pos[t] > pos[i] && pos[t] - first < nr)
+				flow_sym_meet(syms + (size_t)(pos[t] - first) * n, cur, n);
+			if (t == h || pos[t] > pos[i] || pos[t] - first >= nr)
+				continue;
+			/* jump back of an inner loop: what it adds moves the bounds of the root */
+			for (j = 0; j < n; j++) {
+				r = &roots[cur[j].root];
+				if (r->has_max && cur[j].hi > 0)
+					flow_add_max(r, r->max[1] + cur[j].hi);
+				if (r->has_min && cur[j].lo < 0)
+					flow_add_min(r, r->min[1] + cur[j].lo);
+			}
+		}
+	}
+
+	for (k = 0; k < back_cnt; k++) {
+		for (j = 0; j < n; j++) {
+			x = &back[(size_t)k * n + j];
+			r = &roots[x->root];
+			/* it is what it was */
+			if (x->root == j && !x->lo && !x->hi)
+				continue;
+			if (r->has_max) {
+				flow_add_max(&rungs[j], r->max[0] + x->hi);
+				flow_add_max(&rungs[j], r->max[1] + x->hi);
+			}
+			if (r->has_min) {
+				flow_add_min(&rungs[j], r->min[0] + x->lo);
+				flow_add_min(&rungs[j], r->min[1] + x->lo);
+			}
+		}
+	}
+	for (j = 0; j < n; j++)
+		cnt += rungs[j].has_max || rungs[j].has_min;
+	err = 0;
+	if (!cnt)
+		goto out;
+	loop->rungs = kvcalloc(cnt, sizeof(*loop->rungs), GFP_KERNEL_ACCOUNT);
+	if (!loop->rungs) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (p = 0; p < FLOW_PLACES; p++) {
+		if (idx[p] == FLOW_NO_PLACE || (!rungs[idx[p]].has_max && !rungs[idx[p]].has_min))
+			continue;
+		loop->rungs[loop->rungs_cnt] = rungs[idx[p]];
+		loop->rungs[loop->rungs_cnt++].place = p;
+	}
+out:
+	kvfree(syms);
+	kvfree(roots);
+	return err;
+}
+
+static int flow_compute_rungs(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	u32 total = env->cfg.cur_postorder, depth, k, *at;
+	DECLARE_BITMAP(written, FLOW_PLACES);
+	struct bpf_loop *loop;
+	int h, i, err = 0;
+
+	for (h = 0; h < env->prog->len && !aux[h].loop; h++)
+		;
+	if (h == env->prog->len)
+		return 0;
+	at = kvmalloc_array(total, sizeof(*at), GFP_KERNEL_ACCOUNT);
+	if (!at)
+		return -ENOMEM;
+	for (k = 0; k < total; k++) {
+		i = env->cfg.insn_postorder[k];
+		at[env->flow->pos[i]] = i;
+		flow_written_places(&env->prog->insnsi[i], written);
+		for (h = bpf_loop_at_index(env, i); h >= 0; h = aux[h].loop_header)
+			bitmap_or(aux[h].loop->writes, aux[h].loop->writes, written, FLOW_PLACES);
+	}
+	for (h = 0; h < env->prog->len && !err; h++) {
+		loop = aux[h].loop;
+		/* the order says little about a loop with more than one entry */
+		if (!loop || loop->irreducible || loop->backedges_overflow)
+			continue;
+		/* the main pass doesn't get that deep */
+		depth = 0;
+		for (i = aux[h].loop_header; i >= 0; i = aux[i].loop_header)
+			depth++;
+		if (depth >= LOOP_STACK_SIZE)
+			continue;
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		err = flow_compute_loop_rungs(env, at, total, h);
+	}
+	kvfree(at);
+	return err;
+}
+
 int bpf_flow_prepare(struct bpf_verifier_env *env)
 {
 	struct bpf_insn_aux_data *aux = env->insn_aux_data;
@@ -1279,7 +1712,10 @@ int bpf_flow_prepare(struct bpf_verifier_env *env)
 		return err;
 	for (i = 0; !err && i < kinds; i++)
 		err = flow_compute_need_const(env, &flow->need[i], i);
-	return err;
+	err = err ?: flow_compute_rungs(env);
+	if (err)
+		return err;
+	return 0;
 }
 
 static void flow_free_state(struct bpf_verifier_env *env, struct flow_state *s)
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 88fe3d32a6de..cc4494aef187 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -39,7 +39,7 @@ struct bpf_join_state {
 	bool widen;
 	/* the states came around this loop in the frame that is joined, NULL if not */
 	const struct bpf_loop *loop;
-	/* the register or stack slot that is joined, as scalar evolution numbers it, or -1 */
+	/* the register or stack slot that is joined, as in struct bpf_loop_rungs, or -1 */
 	int place;
 };
 
@@ -142,13 +142,26 @@ static bool range_equal(const struct bpf_reg_state *ra, const struct bpf_reg_sta
 	       ra->base == rb->base && ra->step == rb->step;
 }
 
+static const struct bpf_loop_rungs *join_rungs(const struct bpf_join_state *js)
+{
+	u32 i;
+
+	for (i = 0; js->loop && i < js->loop->rungs_cnt; i++)
+		if (js->loop->rungs[i].place == js->place)
+			return &js->loop->rungs[i];
+	return NULL;
+}
+
 /* Does every iteration add a constant to what is in the place? In 64 or in 32 bits, 0 if not */
 static int join_place_is_linear(const struct bpf_join_state *js)
 {
-	if (!js->loop || js->place < 0 || js->place >= BPF_SCEV_REGS_NUM)
+	/* the place as scalar evolution numbers it */
+	u32 r = js->place < BPF_LOOP_SLOT ? js->place : js->place - BPF_LOOP_SLOT + MAX_BPF_REG;
+
+	if (!js->loop || js->place < 0 || r >= BPF_SCEV_REGS_NUM)
 		return 0;
-	return test_bit(js->place, js->loop->linear_regs) ? 64 :
-	       test_bit(js->place, js->loop->linear32_regs) ? 32 : 0;
+	return test_bit(r, js->loop->linear_regs) ? 64 :
+	       test_bit(r, js->loop->linear32_regs) ? 32 : 0;
 }
 
 /*
@@ -172,7 +185,7 @@ static bool join_range(struct bpf_join_state *js, struct bpf_reg_state *dst,
 	 */
 	if (dst->type != SCALAR_VALUE)
 		return false;
-	bpf_reg_widen_range(dst, ra, join_place_is_linear(js));
+	bpf_reg_widen_range(dst, ra, join_rungs(js), join_place_is_linear(js));
 	return true;
 }
 
@@ -468,7 +481,7 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
 	for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
 		const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
 
-		js->place = MAX_BPF_REG + i;
+		js->place = BPF_LOOP_SLOT + i;
 		if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
 			return false;
 	}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 06265d617f4c..554713c9c0bb 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -18414,23 +18414,27 @@ void bpf_reg_forget_range(struct bpf_reg_state *reg)
  * @reg is what @old became after a trip around a cycle, @old included. The end
  * of the range that moved is going to move again. Put it where it may come
  * to rest. There are few such places, so this is done a few times at most.
- * It's put at the nearest of these places: every value between the range and the place is
+ * @rungs: where the jumps of the loop stop it, if they do.
+ * It's put at the nearest of all these places: every value between the range and the place is
  * made up, and the program is checked with it. Wherever it's put, the range only gets larger.
  * @linear: every iteration adds a constant to it, in that many bits. Then nothing but a jump
- * stops it. It goes on until it wraps, and is any number of that many bits.
+ * stops it. If there is none ahead it goes on until it wraps, and is any number of that many bits.
  */
-void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear)
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old,
+			 const struct bpf_loop_rungs *rungs, int linear)
 {
 	bool moved = reg->r64.base != old->r64.base || reg->r64.size != old->r64.size ||
 		     reg->r32.base != old->r32.base || reg->r32.size != old->r32.size;
-	s64 min = reg_smin(reg), max = reg_smax(reg);
+	s64 min = reg_smin(reg), max = reg_smax(reg), stop;
 
 	/* Only the step got weaker. It comes to rest without help, see below. */
 	if (!moved && reg->var_off.value == old->var_off.value &&
 	    reg->var_off.mask == old->var_off.mask)
 		return;
 
-	if (linear && (max > reg_smax(old) || min < reg_smin(old))) {
+	if (linear &&
+	    ((max > reg_smax(old) && !(rungs && rungs->has_max && max <= rungs->max[1])) ||
+	     (min < reg_smin(old) && !(rungs && rungs->has_min && min >= rungs->min[1])))) {
 		if (linear == 64 || min < 0 || max > U32_MAX) {
 			bpf_reg_forget_range(reg);
 			return;
@@ -18438,13 +18442,21 @@ void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *
 		min = 0;
 		max = U32_MAX;
 	}
-	if (max > reg_smax(old))
-		max = max <= 0 ? 0 :
-		      max <= S32_MAX ? S32_MAX :
-		      max <= U32_MAX ? U32_MAX : S64_MAX;
-	if (min < reg_smin(old))
-		min = min >= 0 ? 0 :
-		      min >= S32_MIN ? S32_MIN : S64_MIN;
+	if (max > reg_smax(old)) {
+		stop = max <= 0 ? 0 :
+		       max <= S32_MAX ? S32_MAX :
+		       max <= U32_MAX ? U32_MAX : S64_MAX;
+		if (rungs && rungs->has_max && max <= rungs->max[1])
+			stop = min(stop, max <= rungs->max[0] ? rungs->max[0] : rungs->max[1]);
+		max = stop;
+	}
+	if (min < reg_smin(old)) {
+		stop = min >= 0 ? 0 :
+		       min >= S32_MIN ? S32_MIN : S64_MIN;
+		if (rungs && rungs->has_min && min >= rungs->min[1])
+			stop = max(stop, min >= rungs->min[0] ? rungs->min[0] : rungs->min[1]);
+		min = stop;
+	}
 	if (min == S64_MIN && max == S64_MAX) {
 		bpf_reg_forget_range(reg);
 		return;
-- 
2.55.0

[PATCH bpf-next v3 39/45] selftests/bpf: Adjust tests to log lines that are gone

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Without this patch 114 subtests of test_progs and 4 tests of test_verifier
fail. Their progs get the old verdict, but the tests match lines of the
verifier log that the kernel does not print any more. Keep the progs. Match
the line of the new log that shows the same fact.

  [P] "bpf: Remove precision tracking"
  [V] "bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth"
  [F] "bpf: Verify programs as a fixpoint over the control flow graph"

  the tests matched        they match now
  -----------------------  ------------------------------------------------
  [P] "mark_precise: ..",  The value of the scalar where the prog uses it:
  462 lines                "R2=-8".
  [V] "N: safe", 23 lines  "from M to N: within". If no state covers the
                           other one: "joined", and the join.
  [F] "id=1" of a scalar,  No id: the verifier clears an id that only one
  12 lines                 live register has.

The first row is verifier_precision.c, _subprog_precision.c, _spill_fill.c,
_scalar_ids.c, stack_arg_precision.c and verifier/precise.c. The second row
is verifier_bounds_step.c, _search_pruning.c, _live_stack.c: the tests of
regsafe(), stacksafe() and range_within(). The target of a jump is verified
before the fall-through. Thus the tests in which the first of two states
must cover the second one swap their branches.

Three verdicts change. short_loop1 was "too large". Now one state at the
loop head covers all iterations, and it loads. 2 unprivileged progs of
verifier_value_ptr_arith.c load: r1 = [0, 3] and r1 = [0, 7] are joined in
front of 'r1 += r0', and the insn gets one limit for its mask. 8 progs are
rejected as before, with a different message. BPF_F_TEST_STATE_FREQ has no
effect. Remove it from the tests that change.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/progs/aggregate_ret_func.c  |  15 -
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |   7 -
 .../testing/selftests/bpf/progs/dynptr_fail.c |   4 +-
 .../selftests/bpf/progs/exceptions_assert.c   |  24 +-
 .../selftests/bpf/progs/stack_arg_precision.c |  48 +--
 .../selftests/bpf/progs/test_global_func15.c  |   5 -
 .../bpf/progs/verifier_array_access.c         |  14 +-
 .../bpf/progs/verifier_bounds_deduction.c     |   2 +-
 .../bpf/progs/verifier_bounds_step.c          |  73 ++--
 .../bpf/progs/verifier_linked_scalars.c       |   7 +-
 .../selftests/bpf/progs/verifier_live_stack.c |  34 +-
 .../bpf/progs/verifier_map_ret_val.c          |   2 +-
 .../bpf/progs/verifier_mem_size_reg.c         |   8 +-
 .../selftests/bpf/progs/verifier_precision.c  | 186 +++------
 .../selftests/bpf/progs/verifier_scalar_ids.c | 329 +++++-----------
 .../bpf/progs/verifier_search_pruning.c       |  40 +-
 .../selftests/bpf/progs/verifier_spill_fill.c | 224 ++++-------
 .../selftests/bpf/progs/verifier_spin_lock.c  |   4 +-
 .../selftests/bpf/progs/verifier_stack_ptr.c  |  13 +-
 .../bpf/progs/verifier_subprog_precision.c    | 353 +++---------------
 .../bpf/progs/verifier_value_or_null.c        |   2 +-
 .../bpf/progs/verifier_value_ptr_arith.c      |   6 +-
 .../selftests/bpf/progs/verifier_var_off.c    |   3 -
 .../testing/selftests/bpf/verifier/precise.c  |  69 +---
 24 files changed, 399 insertions(+), 1073 deletions(-)

diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
index f0077d951a25..2671323255cd 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -121,14 +121,6 @@ static __naked __noinline u128 static_agg_precise(void)
 SEC("tc")
 __load_if_JITed()
 __success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
-__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
 __naked int aggregate_ret_static_precise(void)
 {
 	asm volatile (
@@ -150,13 +142,6 @@ __naked int aggregate_ret_static_precise(void)
 SEC("tc")
 __load_if_JITed()
 __success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
 __naked int aggregate_ret_global_precise(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
index 0361a7b42c9b..102156175ba0 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -31,13 +31,6 @@ SEC("tc")
 __arch_x86_64 __arch_arm64
 __load_if_JITed()
 __success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
 __naked int aggregate_ret_kfunc_precise(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c
index 3770ffdeaaaa..c6b239b78325 100644
--- a/tools/testing/selftests/bpf/progs/dynptr_fail.c
+++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c
@@ -127,7 +127,7 @@ static int missing_release_callback_fn(__u32 index, void *data)
 
 /* A callback cannot return with the last dynptr for a referenced resource. */
 SEC("?raw_tp")
-__failure __msg("cannot overwrite referenced dynptr")
+__failure __msg("Unreleased reference id=1")
 int ringbuf_missing_release_callback(void *ctx)
 {
 	bpf_loop(10, missing_release_callback_fn, NULL, 0);
@@ -782,7 +782,7 @@ int dynptr_pruning_overwrite(struct __sk_buff *ctx)
 }
 
 SEC("?tc")
-__success __msg("12: safe") __log_level(2)
+__success __msg("to 12: within") __log_level(2)
 int dynptr_pruning_stacksafe(struct __sk_buff *ctx)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/exceptions_assert.c b/tools/testing/selftests/bpf/progs/exceptions_assert.c
index e4abf4172fca..67b51f728af2 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_assert.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_assert.c
@@ -29,32 +29,32 @@ check_assert(s64, ==, eq_llong_min, LLONG_MIN);
 __msg("R{{.}}=0x7fffffffffffffff")
 check_assert(s64, ==, eq_llong_max, LLONG_MAX);
 
-__msg("R{{.}}=scalar(id=1,smax=0x7ffffffe)")
+__msg("R{{.}}=scalar(smax=0x7ffffffe)")
 check_assert(s64, <, lt_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
 check_assert(s64, <, lt_zero, 0);
-__msg("R{{.}}=scalar(id=1,smax=0xffffffff7fffffff")
+__msg("R{{.}}=scalar(smax=0xffffffff7fffffff")
 check_assert(s64, <, lt_neg, INT_MIN);
 
-__msg("R{{.}}=scalar(id=1,smax=0x7fffffff)")
+__msg("R{{.}}=scalar(smax=0x7fffffff)")
 check_assert(s64, <=, le_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smax=0)")
+__msg("R{{.}}=scalar(smax=0)")
 check_assert(s64, <=, le_zero, 0);
-__msg("R{{.}}=scalar(id=1,smax=0xffffffff80000000")
+__msg("R{{.}}=scalar(smax=0xffffffff80000000")
 check_assert(s64, <=, le_neg, INT_MIN);
 
-__msg("R{{.}}=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >, gt_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >, gt_zero, 0);
-__msg("R{{.}}=scalar(id=1,smin=0xffffffff80000001")
+__msg("R{{.}}=scalar(smin=0xffffffff80000001")
 check_assert(s64, >, gt_neg, INT_MIN);
 
-__msg("R{{.}}=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >=, ge_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >=, ge_zero, 0);
-__msg("R{{.}}=scalar(id=1,smin=0xffffffff80000000")
+__msg("R{{.}}=scalar(smin=0xffffffff80000000")
 check_assert(s64, >=, ge_neg, INT_MIN);
 
 SEC("?tc")
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_precision.c b/tools/testing/selftests/bpf/progs/stack_arg_precision.c
index c9564776d733..6c6f58a67eff 100644
--- a/tools/testing/selftests/bpf/progs/stack_arg_precision.c
+++ b/tools/testing/selftests/bpf/progs/stack_arg_precision.c
@@ -23,7 +23,7 @@ int __btf_kfunc_gen(struct __sk_buff *ctx)
 }
 
 /*
- * Test precision backtracking across bpf-to-bpf call for kfunc stack arg.
+ * Test that the range of a stack arg is known across bpf-to-bpf call for kfunc stack arg.
  * subprog_call_mem_kfunc receives a size as incoming stack arg (arg6)
  * and forwards it as mem__sz (arg7) to bpf_kfunc_call_stack_arg_mem.
  */
@@ -52,52 +52,12 @@ static long subprog_call_mem_kfunc(long a, long b, long c, long d, long e, long
 }
 
 SEC("tc")
-__description("stack_arg: precision backtracking across bpf2bpf call for kfunc")
+__description("stack_arg: range across bpf2bpf call for kfunc")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 9 to 11: joined")
+__msg("13: (79) r1 = *(u64 *)(r11 +8)        ; R1=scalar(smin=umin=smin32=umin32=4,smax=umax=smax32=umax32=6,var_off=(0x4; 0x2),step=0+2)")
 __btf_func_path("btf__stack_arg_precision.bpf.o")
-__msg("mark_precise: frame1: last_idx 26 first_idx 13 subseq_idx -1")
-__msg("mark_precise: frame1: regs= stack= before 25: (b7) r5 = 5")
-__msg("mark_precise: frame1: regs= stack= before 24: (b7) r4 = 4")
-__msg("mark_precise: frame1: regs= stack= before 23: (b7) r3 = 3")
-__msg("mark_precise: frame1: regs= stack= before 22: (b7) r2 = 2")
-__msg("mark_precise: frame1: regs= stack= before 21: (b7) r1 = 1")
-__msg("mark_precise: frame1: regs= stack= before 20: (7b) *(u64 *)(r11 -16) = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 19: (7b) *(u64 *)(r11 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 18: (07) r2 += -8")
-__msg("mark_precise: frame1: regs=r1 stack= before 17: (bf) r2 = r10")
-__msg("mark_precise: frame1: regs=r1 stack= before 16: (7b) *(u64 *)(r10 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 14: (18) r2 = 0x807060504030201")
-__msg("mark_precise: frame1: regs=r1 stack= before 13: (79) r1 = *(u64 *)(r11 +8)")
-__msg("mark_precise: frame1: parent state regs= stack=:  frame1: R10=fp0")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame1: last_idx 11 first_idx 11 subseq_idx 13")
-__msg("mark_precise: frame1: regs= stack= before 11: (85) call pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx 11")
-__msg("mark_precise: frame0: regs= stack= before 9: (05) goto pc+1")
-__msg("mark_precise: frame0: regs= stack= before 8: (7a) *(u64 *)(r11 -8) = 4")
-__msg("mark_precise: frame1: last_idx 26 first_idx 13 subseq_idx -1 ")
-__msg("mark_precise: frame1: regs= stack= before 25: (b7) r5 = 5")
-__msg("mark_precise: frame1: regs= stack= before 24: (b7) r4 = 4")
-__msg("mark_precise: frame1: regs= stack= before 23: (b7) r3 = 3")
-__msg("mark_precise: frame1: regs= stack= before 22: (b7) r2 = 2")
-__msg("mark_precise: frame1: regs= stack= before 21: (b7) r1 = 1")
-__msg("mark_precise: frame1: regs= stack= before 20: (7b) *(u64 *)(r11 -16) = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 19: (7b) *(u64 *)(r11 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 18: (07) r2 += -8")
-__msg("mark_precise: frame1: regs=r1 stack= before 17: (bf) r2 = r10")
-__msg("mark_precise: frame1: regs=r1 stack= before 16: (7b) *(u64 *)(r10 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 14: (18) r2 = 0x807060504030201")
-__msg("mark_precise: frame1: regs=r1 stack= before 13: (79) r1 = *(u64 *)(r11 +8)")
-__msg("mark_precise: frame1: parent state regs= stack=:  frame1: R10=fp0")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame1: last_idx 11 first_idx 11 subseq_idx 13 ")
-__msg("mark_precise: frame1: regs= stack= before 11: (85) call pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx 11 ")
-__msg("mark_precise: frame0: regs= stack= before 10: (7a) *(u64 *)(r11 -8) = 6")
 __naked void stack_arg_precision_bpf2bpf(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/test_global_func15.c b/tools/testing/selftests/bpf/progs/test_global_func15.c
index 201cc000b3f4..497f09dc8876 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func15.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func15.c
@@ -24,12 +24,7 @@ int global_func15(struct __sk_buff *skb)
 }
 
 SEC("cgroup_skb/ingress")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
 __failure
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (b7) r0 = 1")
-/* check that branch code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7")
 __msg("At program exit the register R0 has ")
 __naked int global_func15_tricky_pruning(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_array_access.c b/tools/testing/selftests/bpf/progs/verifier_array_access.c
index 9333321c3ac8..825ff1c3ce43 100644
--- a/tools/testing/selftests/bpf/progs/verifier_array_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_array_access.c
@@ -248,7 +248,7 @@ l0_%=:	exit;						\
 SEC("socket")
 __description("invalid map access into an array with no floor check")
 __failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void array_with_no_floor_check(void)
 {
@@ -280,7 +280,7 @@ l0_%=:	exit;						\
 SEC("socket")
 __description("invalid map access into an array with a invalid max check")
 __failure __msg("invalid access to map value, value_size=48 off=44 size=8")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 max value is outside of the allowed memory range")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void with_a_invalid_max_check_1(void)
 {
@@ -541,8 +541,7 @@ l0_%=:	exit;						\
 
 SEC("socket")
 __description("valid map access into an array using constant without nullness")
-__success __retval(4) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(4)
 unsigned int an_array_with_a_constant_no_nullness(void)
 {
 	/* Need 8-byte alignment for spill tracking */
@@ -557,9 +556,7 @@ unsigned int an_array_with_a_constant_no_nullness(void)
 
 SEC("socket")
 __description("valid multiple map access into an array using constant without nullness")
-__success __retval(8) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -16) = {{(0|r[0-9])}}")
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(8)
 unsigned int multiple_array_with_a_constant_no_nullness(void)
 {
 	__u32 __attribute__((aligned(8))) key = 1;
@@ -595,8 +592,7 @@ unsigned int an_array_with_a_32bit_constant_0_no_nullness(void)
 
 SEC("socket")
 __description("valid map access into a pcpu array using constant without nullness")
-__success __retval(4) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(4)
 unsigned int a_pcpu_array_with_a_constant_no_nullness(void)
 {
 	__u32 __attribute__((aligned(8))) key = 1;
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
index 260a6df264e3..a474c9978a35 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
@@ -102,7 +102,7 @@ l0_%=:	r0 -= r1;					\
 SEC("socket")
 __description("check deducing bounds from const, 7")
 __failure __msg("dereference of modified ctx ptr")
-__msg_unpriv("R1 has pointer with unsupported alu operation")
+__msg_unpriv("dereference of modified ctx ptr")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void deducing_bounds_from_const_7(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
index d824a5d48a39..51b703058cf7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
@@ -371,14 +371,17 @@ struct {
 	__type(value, struct step_val);
 } step_map SEC(".maps");
 
-/* Old register [4..130, step 2] should prune cur register [8..68, step 4]. */
+/*
+ * Register [8..68, step 4] is within register [4..130, step 2]: one state stands for both
+ * and nothing is lost.
+ */
 SEC("socket")
 __success __log_level(2)
 __msg("7: (27) r6 *= 4                       ; R6=scalar({{.*}}umin32=8,{{.*}}umax32=68,{{.*}},step=0+4)")
 __msg("10: (27) r7 *= 2                      ; R7=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2)")
 __msg("11: (25) if r0 > 0x2a goto pc+1")
-__msg("from 11 to 13: safe")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("13: R6=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2) R10=fp0")
+__msg("processed 18 insns")
 __naked void step_prune_hit_multiple(void)
 {
 	asm volatile ("					\
@@ -394,11 +397,11 @@ __naked void step_prune_hit_multiple(void)
 	r7 += 2;					\
 	r7 *= 2;					\
 	if r0 > 42 goto 1f;	/* can't predict */	\
-	r6 = r7;		/* step=2 explored first, step=4 explored next */ \
+	r6 = r7;		/* step=2 here, step=4 on the other path */ \
 1:	r0 = r10;					\
 	r6 = -r6;					\
 	r0 += r6;					\
-	*(u8 *)(r0 + 0) = 7;	/* force r6 precise */	\
+	*(u8 *)(r0 + 0) = 7;				\
 	exit;						\
 "	:
 	: __imm(bpf_get_prandom_u32)
@@ -406,17 +409,16 @@ __naked void step_prune_hit_multiple(void)
 }
 
 /*
- * Old register [0..189, step 3] should not prune cur register [0..30, step 2].
- * Bounds and tnum containment allow pruning; step divisibility must prevent it.
+ * Register [0..30, step 2] is not within register [0..189, step 3].
+ * Bounds and tnum say that it is; step divisibility must prevent it.
+ * The states are joined and the result has no step.
  */
 SEC("socket")
 __success __log_level(2)
 __msg("6: (27) r6 *= 2                       ; R6=scalar({{.*}}smin32=0,{{.*}}umax32=30,{{.*}},step=0+2)")
 __msg("8: (27) r7 *= 3                       ; R7=scalar({{.*}}smin32=0,{{.*}}umax32=189,{{.*}},step=0+3)")
 __msg("9: (25) if r0 > 0x2a goto pc+1")
-__msg("17: (95) exit")
-__msg("from 9 to 11: R6=scalar({{.*}},step=0+2)")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("11: R6=scalar({{.*}}umax32=189,var_off=(0x0; 0xff)) R10=fp0")
 __naked void step_prune_miss_non_multiple(void)
 {
 	asm volatile ("					\
@@ -430,13 +432,13 @@ __naked void step_prune_miss_non_multiple(void)
 	r7 &= 0x3f;					\
 	r7 *= 3;					\
 	if r0 > 42 goto 1f;	/* can't predict */	\
-	r6 = r7;		/* step=3 explored first, step=2 explored next */ \
+	r6 = r7;		/* step=3 here, step=2 on the other path */ \
 1:							\
 	r2 = r10;					\
 	r6 += 1;					\
 	r6 = -r6;					\
 	r2 += r6;					\
-	*(u8 *)(r2 + 0) = 7;	/* force r6 precise */	\
+	*(u8 *)(r2 + 0) = 7;				\
 	r0 = 0;						\
 	exit;						\
 "	:
@@ -445,17 +447,17 @@ __naked void step_prune_miss_non_multiple(void)
 }
 
 /*
- * Constant current register lying on the cached line: cached is {0,3,6,...}
- * (step 3, base 0), current is the constant 6. range_within() takes the
- * constant branch: imod(6, 3) == base 0, so cur is on the line and the
- * (precise) state is pruned -> "safe".
+ * Constant register lying on the line of the other state: that is {0,3,6,...}
+ * (step 3, base 0), this is the constant 6. range_within() takes the
+ * constant branch: imod(6, 3) == base 0, so the constant is on the line and
+ * the state with the line stands for both.
  */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("14: (27) r1 *= 3")		/* cached path: step 3 line */
-__msg("16: (b7) r1 = 6")		/* current path: const 6, on the line */
-__msg("17: safe")			/* pruned at join: imod(6, 3) == 0 */
+__msg("16: (b7) r1 = 6")		/* const 6, on the line */
+__msg("14: (27) r1 *= 3")		/* step 3 line */
+__msg("from 15 to 17: joined")
+__msg("17: R1=scalar({{.*}},step=0+3) R7=")	/* imod(6, 3) == 0 */
 __naked void step_prune_hit_const_on_line(void)
 {
 	asm volatile ("					\
@@ -472,13 +474,13 @@ __naked void step_prune_hit_const_on_line(void)
 	r1 = r6;					\
 	r1 &= 0xff;					\
 	if r6 > 0 goto l_cur_%=;			\
-	r1 *= 3;			/* old: step 3 */	\
+	r1 *= 3;			/* step 3 */	\
 	goto l_join_%=;					\
 l_cur_%=:						\
-	r1 = 6;				/* cur: const on line */	\
+	r1 = 6;				/* const on line */	\
 l_join_%=:						\
 	r0 = r7;					\
-	r0 += r1;			/* r1 forced precise */	\
+	r0 += r1;					\
 	r2 = *(u8 *)(r0 + 0);				\
 l_out_%=:						\
 	r0 = 0;						\
@@ -491,22 +493,21 @@ l_out_%=:						\
 }
 
 /*
- * Constant current register NOT on the cached line: cached is {0,3,6,...}
- * (step 3, base 0), current is the constant 7. imod(7, 3) == 1 != base 0,
- * so range_within() fails and the join is traversed again. A power-of-two
+ * Constant register NOT on the line of the other state: that is {0,3,6,...}
+ * (step 3, base 0), this is the constant 7. imod(7, 3) == 1 != base 0,
+ * so range_within() fails and the states are joined. A power-of-two
  * step is avoided on purpose: with step 3 the tnum is loose enough to admit
  * 7, so imod() is
  * the sole check that rejects it. No failure shape is possible here: 7 is
- * within the line's bounds and tnum, and the cached line path already
- * verifies the whole outro, so an eager prune could not miss an error.
+ * within the line's bounds and tnum.
  */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("14: (27) r1 *= 3")		/* cached path: step 3 line */
-__msg("16: (b7) r1 = 7")		/* current path: const 7, off the line */
-/* not pruned: current continues past the join with the constant offset 7 */
-__msg("19: R0=map_value({{.*}}imm=7) R1=7")
+__msg("16: (b7) r1 = 7")		/* const 7, off the line */
+__msg("14: (27) r1 *= 3")		/* step 3 line */
+/* the result stands for 7 too: it has no step */
+__msg("from 15 to 17: joined")
+__msg("17: R1=scalar({{.*}}var_off=(0x0; 0x3ff)) R7=")
 __naked void step_prune_miss_const_off_line(void)
 {
 	asm volatile ("					\
@@ -523,13 +524,13 @@ __naked void step_prune_miss_const_off_line(void)
 	r1 = r6;					\
 	r1 &= 0xff;					\
 	if r6 > 0 goto l_cur_%=;			\
-	r1 *= 3;			/* old: step 3 */	\
+	r1 *= 3;			/* step 3 */	\
 	goto l_join_%=;					\
 l_cur_%=:						\
-	r1 = 7;				/* cur: const off line */	\
+	r1 = 7;				/* const off line */	\
 l_join_%=:						\
 	r0 = r7;					\
-	r0 += r1;			/* r1 forced precise */	\
+	r0 += r1;					\
 	r2 = *(u8 *)(r0 + 0);				\
 l_out_%=:						\
 	r0 = 0;						\
diff --git a/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c b/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c
index d571fbfc86a3..caae67feaf5b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c
+++ b/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c
@@ -668,12 +668,16 @@ __naked void add_const_base_id_pruning(void)
 	call %[bpf_get_prandom_u32];				\
 	r6 = r0;						\
 	r6 &= 1;						\
-	if r6 >= 1 goto l_true_%=;				\
+	if r6 < 1 goto l_false_%=;				\
+	goto l_true_%=;						\
 								\
+l_false_%=:							\
 	/* False branch (explored first, old state) */		\
 	call %[bpf_get_prandom_u32];				\
 	r2 = r0;						\
 	r2 &= 0xff;		/* R2 = scalar(id=A) [0,255] */	\
+	r4 = r0;						\
+	r4 &= 0xff;		/* R4 = scalar [0,255], id=0 */	\
 	r3 = r2;		/* R3 linked to R2 (id=A) */	\
 	r3 += 10;		/* R3 id=A|ADD_CONST, delta=10 */\
 	r6 = 0;							\
@@ -703,6 +707,7 @@ l_merge_%=:							\
 	r9 += r3;		/* fp-16+[0,5] vs fp-16+[0,255] */\
 	*(u8*)(r9 + 0) = r6;	/* within 16B vs past fp */	\
 l_exit_%=:							\
+	r4 = r4;		/* R4 is live at the merge */	\
 	r0 = 0;							\
 	exit;							\
 "	:
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 2ae5c60150dd..795956bd4f9f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -368,7 +368,6 @@ static __used __naked void arg_track_join_convergence_subprog(void)
 }
 
 SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
 __log_level(2)
 /* fp0-8 consumed at insn 9, dead by insn 11. stack_def at insn 4 kills slots 0,1. */
 __msg("4: (7b) *(u64 *)(r10 -8) = r0        ; def: fp0-8")
@@ -380,9 +379,9 @@ __msg("10: (b7) r0 = 0{{$}}")
 __msg("11: (05) goto pc+0{{$}}")
 __msg("12: (95) exit")
 /*
- * Checkpoint at goto +0 fires because fp0-8 is dead → state pruning.
+ * fp0-8 is dead after goto +0, so one state stands for both.
  */
-__msg("12: safe")
+__msg("from 11 to 12: within")
 __naked void caller_stack_pruning(void)
 {
 	asm volatile (
@@ -497,7 +496,6 @@ struct {
 
 SEC("socket")
 __failure __msg("invalid read from stack R2 off=-4096 size=8")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked unsigned long caller_stack_write_tail_call(void)
 {
         asm volatile (
@@ -855,7 +853,7 @@ static __used __naked void maybe_clobber_key(void)
 
 /*
  * Demonstrate that subprog arg spill/reload breaks arg tracking,
- * inflating caller stack liveness and preventing state pruning.
+ * inflating caller stack liveness and keeping states apart.
  *
  * modifier2(fp-24) has two paths: one writes a scalar to *(r1+8)
  * = caller fp-16, the other leaves it as zero.  After modifier2
@@ -871,16 +869,15 @@ static __used __naked void maybe_clobber_key(void)
  * the reader never touches it.
  *
  * Result: at modifier2's return point two states with different
- * fp-16 values cannot be pruned.
+ * fp-16 values are kept apart.
  *
- * With correct (or old dynamic) liveness fp-16 is dead at that
- * point and the states prune → "6: safe" appears in the log.
+ * With correct liveness fp-16 is dead at that point and one state is
+ * within the other one.
  */
 SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
 __log_level(2)
 __success
-__msg("6: safe")
+__msg("from 16 to 6: within")
 __naked void spill_reload_inflates_stack_liveness(void)
 {
 	asm volatile (
@@ -1366,31 +1363,32 @@ __naked void half_spi_clean_preserves_stack_zero(void)
  * conditional block with helper call and temporary stack spill,
  * spill dead after merge.
  *
- * Path A (fall-through): spill r6 to fp-8 across helper call
- * Path B (branch taken): skip the block entirely
- * At merge (insn 6): fp-8 is dead (never read after merge)
+ * Path A (branch taken, verified first): spill r6 to fp-8 across helper call
+ * Path B (fall-through): skip the block entirely
+ * At merge (insn 7): fp-8 is dead (never read after merge)
  *
  * Static liveness marks fp-8 dead at merge. clean_verifier_state()
  * converts path A's STACK_SPILL to STACK_INVALID. Path B has
- * STACK_INVALID. stacksafe() matches -> path B pruned -> "6: safe".
+ * STACK_INVALID. stacksafe() matches -> one state stands for both paths.
  */
 SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
 __success
 __log_level(2)
-__msg("6: safe")
+__msg("from 3 to 7: within")
 __naked void dead_spill_at_merge_enables_pruning(void)
 {
 	asm volatile (
 	"call %[bpf_get_prandom_u32];"
 	"r6 = 7;"
-	"if r0 != 0 goto l_skip%=;"
+	"if r0 == 0 goto l_spill%=;"
+	"goto l_skip%=;"
+"l_spill%=:"
 	/* conditional block: spill, call, reload */
 	"*(u64 *)(r10 - 8) = r6;"
 	"call %[bpf_get_prandom_u32];"
 	"r6 = *(u64 *)(r10 - 8);"
 "l_skip%=:"
-	/* fp-8 dead. Path B pruned here -> "6: safe" */
+	/* fp-8 dead. The states of both paths are the same here */
 	"r0 = r6;"
 	"exit;"
 	:
diff --git a/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c b/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
index 1639628b832d..2b65ab508596 100644
--- a/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
+++ b/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
@@ -83,7 +83,7 @@ l0_%=:	exit;						\
 SEC("socket")
 __description("sometimes access memory with incorrect alignment")
 __failure __msg("R0 invalid mem access")
-__msg_unpriv("R0 leaks addr")
+__msg_unpriv("R0 invalid mem access")
 __flag(BPF_F_STRICT_ALIGNMENT)
 __naked void access_memory_with_incorrect_alignment_2(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
index 7e24706a764e..aba9510a1c08 100644
--- a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
@@ -7,13 +7,9 @@
 
 char _license[] SEC("license") = "GPL";
 
-/*
- * The __szk size of a kfunc memory/size pair must be marked precise even when
- * the nullable buffer is passed as NULL.
- */
+/* The __szk size of a kfunc memory/size pair when the nullable buffer is passed as NULL */
 SEC("?tc")
-__success __log_level(2)
-__msg("mark_precise: frame0: regs=r4 stack= before")
+__success
 int dynptr_slice_null_buf_size_precise(struct __sk_buff *skb)
 {
 	struct bpf_dynptr dptr;
diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c
index 71ac80a984d8..f0f9b62494ca 100644
--- a/tools/testing/selftests/bpf/progs/verifier_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_precision.c
@@ -16,10 +16,7 @@ struct {
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0xfffffff8 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (87) r2 = -r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 8")
+__msg("1: (87) r2 = -r2                      ; R2=-8")
 __naked int bpf_neg(void)
 {
 	asm volatile (
@@ -36,10 +33,7 @@ __naked int bpf_neg(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (d4) r2 = le16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (d4) r2 = le16 r2                  ; R2=0")
 __naked int bpf_end_to_le(void)
 {
 	asm volatile (
@@ -57,10 +51,7 @@ __naked int bpf_end_to_le(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (dc) r2 = be16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (dc) r2 = be16 r2                  ; R2=0")
 __naked int bpf_end_to_be(void)
 {
 	asm volatile (
@@ -82,10 +73,7 @@ __naked int bpf_end_to_be(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (d7) r2 = bswap16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (d7) r2 = bswap16 r2               ; R2=0")
 __naked int bpf_end_bswap(void)
 {
 	asm volatile (
@@ -104,10 +92,8 @@ __naked int bpf_end_bswap(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (db) r2 = load_acquire((u64 *)(r10 -8))")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("2: (db) r2 = load_acquire((u64 *)(r10 -8)){{ *}}; R2=8 R10=fp0 fp-8=8")
+__msg("4: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_load_acquire(void)
 {
 	asm volatile (
@@ -115,7 +101,7 @@ __naked int bpf_load_acquire(void)
 	"*(u64 *)(r10 - 8) = r1;"
 	".8byte %[load_acquire_insn];" /* r2 = load_acquire((u64 *)(r10 - 8)); */
 	"r3 = r10;"
-	"r3 += r2;" /* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -126,10 +112,8 @@ __naked int bpf_load_acquire(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r2 = r10")
-__msg("mark_precise: frame0: regs=r1 stack= before 2: (79) r1 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (db) store_release((u64 *)(r10 -8), r1)")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("1: (db) store_release((u64 *)(r10 -8), r1){{ *}}; R1=8 R10=fp0 fp-8=8")
+__msg("4: (0f) r2 += r1                      ; R1=8 R2=fp8")
 __naked int bpf_store_release(void)
 {
 	asm volatile (
@@ -137,7 +121,7 @@ __naked int bpf_store_release(void)
 	".8byte %[store_release_insn];" /* store_release((u64 *)(r10 - 8), r1); */
 	"r1 = *(u64 *)(r10 - 8);"
 	"r2 = r10;"
-	"r2 += r1;" /* mark_precise */
+	"r2 += r1;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -150,24 +134,7 @@ __naked int bpf_store_release(void)
 #endif /* v4 instruction */
 
 SEC("?raw_tp")
-__success __log_level(2)
-/*
- * Without the bug fix there will be no history between "last_idx 3 first_idx 3"
- * and "parent state regs=" lines. "R0=6" parts are here to help anchor
- * expected log messages to the one specific mark_chain_precision operation.
- *
- * This is quite fragile: if verifier checkpointing heuristic changes, this
- * might need adjusting.
- */
-__msg("2: (47) r0 |= 0                       ; R0=6")
-__msg("3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (47) r0 |= 0")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 2")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R0=P4")
-__msg("3: R0=6")
+__success
 __naked int state_loop_first_last_equal(void)
 {
 	asm volatile (
@@ -206,10 +173,6 @@ __success __log_level(2)
 __msg("8: (bd) if r2 <= r10 goto pc+3")
 __msg("9: (35) if r1 >= 0xffe3fff8 goto pc+0")
 __msg("10: (b5) if r2 <= 0x8 goto pc+0")
-__msg("mark_precise: frame1: last_idx 10 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame1: regs=r2 stack= before 9: (35) if r1 >= 0xffe3fff8 goto pc+0")
-__msg("mark_precise: frame1: regs=r2 stack= before 8: (bd) if r2 <= r10 goto pc+3")
-__msg("mark_precise: frame1: regs=r2 stack= before 7: (05) goto pc+0")
 __naked void bpf_cond_op_r10(void)
 {
 	asm volatile (
@@ -225,9 +188,6 @@ __success __log_level(2)
 __msg("3: (bf) r3 = r10")
 __msg("4: (bd) if r3 <= r2 goto pc+1")
 __msg("5: (b5) if r2 <= 0x8 goto pc+2")
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bd) if r3 <= r2 goto pc+1")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r3 = r10")
 __naked void bpf_cond_op_not_r10(void)
 {
 	asm volatile (
@@ -245,9 +205,6 @@ SEC("lsm.s/socket_connect")
 __success __log_level(2)
 __msg("0: (b7) r0 = 1                        ; R0=1")
 __msg("1: (84) w0 = -w0                      ; R0=0xffffffff")
-__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (84) w0 = -w0")
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
 __naked int bpf_neg_2(void)
 {
 	/*
@@ -280,9 +237,6 @@ SEC("lsm.s/socket_connect")
 __success __log_level(2)
 __msg("0: (b7) r0 = 1                        ; R0=1")
 __msg("1: (87) r0 = -r0                      ; R0=-1")
-__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (87) r0 = -r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
 __naked int bpf_neg_4(void)
 {
 	/*
@@ -313,11 +267,8 @@ __naked int bpf_neg_5(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_add_precision(void)
 {
 	asm volatile (
@@ -326,7 +277,7 @@ __naked int bpf_atomic_fetch_add_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_add_insn];"	/* r2 = atomic_fetch_add(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -337,11 +288,8 @@ __naked int bpf_atomic_fetch_add_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_xchg((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_xchg((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_xchg_precision(void)
 {
 	asm volatile (
@@ -350,7 +298,7 @@ __naked int bpf_atomic_xchg_precision(void)
 	"r2 = 0;"
 	".8byte %[xchg_insn];"		/* r2 = atomic_xchg(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -361,11 +309,8 @@ __naked int bpf_atomic_xchg_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_or((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_or((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_or_precision(void)
 {
 	asm volatile (
@@ -374,7 +319,7 @@ __naked int bpf_atomic_fetch_or_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_or_insn];"	/* r2 = atomic_fetch_or(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -385,11 +330,8 @@ __naked int bpf_atomic_fetch_or_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_and((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_and((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_and_precision(void)
 {
 	asm volatile (
@@ -398,7 +340,7 @@ __naked int bpf_atomic_fetch_and_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_and_insn];"	/* r2 = atomic_fetch_and(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -409,11 +351,8 @@ __naked int bpf_atomic_fetch_and_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_xor((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_xor((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_xor_precision(void)
 {
 	asm volatile (
@@ -422,7 +361,7 @@ __naked int bpf_atomic_fetch_xor_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_xor_insn];"	/* r2 = atomic_fetch_xor(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -433,12 +372,8 @@ __naked int bpf_atomic_fetch_xor_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r0 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2){{ *}}; R0=8 R2=0 R10=fp0 fp-8=mmmmmmmm")
+__msg("6: (0f) r3 += r0                      ; R0=8 R3=fp8")
 __naked int bpf_atomic_cmpxchg_precision(void)
 {
 	asm volatile (
@@ -448,7 +383,7 @@ __naked int bpf_atomic_cmpxchg_precision(void)
 	"r2 = 0;"
 	".8byte %[cmpxchg_insn];"	/* r0 = atomic_cmpxchg(*(u64 *)(r10 - 8), r0, r2) */
 	"r3 = r10;"
-	"r3 += r0;"			/* mark_precise */
+	"r3 += r0;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -457,19 +392,15 @@ __naked int bpf_atomic_cmpxchg_precision(void)
 	: __clobber_all);
 }
 
-/* Regression test for dual precision: Both the fetched value (r2) and
- * a reread of the same stack slot (r3) are tracked for precision. After
- * the atomic operation, the stack slot is STACK_MISC. Thus, the ldx at
- * insn 4 does NOT set INSN_F_STACK_ACCESS. Precision for the stack slot
- * propagates solely through the atomic fetch's load side (insn 3).
+/*
+ * Both the fetched value (r2) and a reread of the same stack slot (r3) are used.
+ * The fetched value is known. After the atomic operation, the stack slot is
+ * STACK_MISC, so what is reread is not.
  */
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2,r3 stack= before 4: (79) r3 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("5: (bf) r4 = r2                       ; R2=8 R4=8")
+__msg("6: (0f) r4 += r3                      ; R3=scalar() R4=scalar()")
 __naked int bpf_atomic_fetch_add_dual_precision(void)
 {
 	asm volatile (
@@ -482,7 +413,7 @@ __naked int bpf_atomic_fetch_add_dual_precision(void)
 	"r4 += r3;"
 	"r4 &= 7;"
 	"r5 = r10;"
-	"r5 += r4;"			/* mark_precise */
+	"r5 += r4;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -493,12 +424,8 @@ __naked int bpf_atomic_fetch_add_dual_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0,r3 stack= before 5: (79) r3 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r0 = 8")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("6: (bf) r4 = r0                       ; R0=8 R4=8")
+__msg("7: (0f) r4 += r3                      ; R3=scalar() R4=scalar()")
 __naked int bpf_atomic_cmpxchg_dual_precision(void)
 {
 	asm volatile (
@@ -512,7 +439,7 @@ __naked int bpf_atomic_cmpxchg_dual_precision(void)
 	"r4 += r3;"
 	"r4 &= 7;"
 	"r5 = r10;"
-	"r5 += r4;"			/* mark_precise */
+	"r5 += r4;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -523,9 +450,7 @@ __naked int bpf_atomic_cmpxchg_dual_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 10: (57) r1 &= 7")
-__msg("mark_precise: frame0: regs=r1 stack= before 9: (db) r1 = atomic64_fetch_add((u64 *)(r0 +0), r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("9: (db) r1 = atomic64_fetch_add((u64 *)(r0 +0), r1){{ *}}; R0=map_value({{.*}}) R1=scalar()")
 __naked int bpf_atomic_fetch_add_map_precision(void)
 {
 	asm volatile (
@@ -540,7 +465,7 @@ __naked int bpf_atomic_fetch_add_map_precision(void)
 	".8byte %[fetch_add_insn];"	/* r1 = atomic_fetch_add(*(u64 *)(r0 + 0), r1) */
 	"r1 &= 7;"
 	"r2 = r10;"
-	"r2 += r1;"			/* mark_precise */
+	"r2 += r1;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -553,9 +478,7 @@ __naked int bpf_atomic_fetch_add_map_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (57) r0 &= 7")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (db) r0 = atomic64_cmpxchg((u64 *)(r6 +0), r0, r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("11: (db) r0 = atomic64_cmpxchg((u64 *)(r6 +0), r0, r1){{ *}}; R0=scalar() R1=0")
 __naked int bpf_atomic_cmpxchg_map_precision(void)
 {
 	asm volatile (
@@ -572,7 +495,7 @@ __naked int bpf_atomic_cmpxchg_map_precision(void)
 	".8byte %[cmpxchg_insn];"	/* r0 = atomic_cmpxchg(*(u64 *)(r6 + 0), r0, r1) */
 	"r0 &= 7;"
 	"r2 = r10;"
-	"r2 += r0;"			/* mark_precise */
+	"r2 += r0;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -585,9 +508,7 @@ __naked int bpf_atomic_cmpxchg_map_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 10: (57) r1 &= 7")
-__msg("mark_precise: frame0: regs=r1 stack= before 9: (c3) r1 = atomic_fetch_add((u32 *)(r0 +0), r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("9: (c3) r1 = atomic_fetch_add((u32 *)(r0 +0), r1){{ *}}; R0=map_value({{.*}}) R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
 __naked int bpf_atomic_fetch_add_32bit_precision(void)
 {
 	asm volatile (
@@ -602,7 +523,7 @@ __naked int bpf_atomic_fetch_add_32bit_precision(void)
 	".8byte %[fetch_add_insn];"	/* r1 = atomic_fetch_add(*(u32 *)(r0 + 0), r1) */
 	"r1 &= 7;"
 	"r2 = r10;"
-	"r2 += r1;"			/* mark_precise */
+	"r2 += r1;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -615,9 +536,7 @@ __naked int bpf_atomic_fetch_add_32bit_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (57) r0 &= 7")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (c3) r0 = atomic_cmpxchg((u32 *)(r6 +0), r0, r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("11: (c3) r0 = atomic_cmpxchg((u32 *)(r6 +0), r0, r1){{ *}}; R0=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R1=0")
 __naked int bpf_atomic_cmpxchg_32bit_precision(void)
 {
 	asm volatile (
@@ -634,7 +553,7 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void)
 	".8byte %[cmpxchg_insn];"	/* r0 = atomic_cmpxchg(*(u32 *)(r6 + 0), r0, r1) */
 	"r0 &= 7;"
 	"r2 = r10;"
-	"r2 += r0;"			/* mark_precise */
+	"r2 += r0;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -648,11 +567,9 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void)
 /*
  * Verification takes two paths: with r1 being scalar zero on path (1)
  * and with r1 being some other scalar on path (2).
- * Check that the verifier does not use checkpoints created
- * on path (1) to prune path (2).
+ * Check that the state of path (1) does not stand for path (2).
  */
 SEC("?tc")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("R1 type=scalar expected=fp")
 __naked int null_mem_arg_zero_size(void)
 {
@@ -689,11 +606,9 @@ __weak int subprog_mem_arg(int *p)
 /*
  * Verification takes two paths: with r1 being scalar zero on path (1)
  * and with r1 being some other scalar on path (2).
- * Check that the verifier does not use checkpoints created
- * on path (1) to prune path (2).
+ * Check that the state of path (1) does not stand for path (2).
  */
 SEC("?raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("R1 type=scalar expected=fp")
 __naked int null_mem_arg_global_subprog(void)
 {
@@ -711,9 +626,8 @@ __naked int null_mem_arg_global_subprog(void)
 		: __clobber_all);
 }
 
-/* Same as above, check that path with r3 == 0 does not prune the path with r3 != 0 */
+/* Same as above, check that path with r3 == 0 does not stand for the path with r3 != 0 */
 SEC("?tc")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("R3 type=scalar expected=fp")
 int null_kfunc_arg_dynptr_slice(struct __sk_buff *skb)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
index 256547048cc4..9ea61b162736 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
@@ -11,27 +11,22 @@ struct {
 	__type(value, long long);
 } map_hash_8b SEC(".maps");
 
-/* Check that precision marks propagate through scalar IDs.
+/* scalar in [0, max] that shares id with other registers, as it is in the log */
+#define LINKED(id, max, mask) \
+	"scalar(id=" id ",smin=smin32=0,smax=umax=smax32=umax32=" #max ",var_off=(0x0; " #mask "))"
+#define ALONE(max, mask) \
+	"scalar(smin=smin32=0,smax=umax=smax32=umax32=" #max ",var_off=(0x0; " #mask "))"
+
+/*
  * Registers r{0,1,2} have the same scalar ID.
- * Range information is propagated for scalars sharing same ID.
- * Check that precision mark for r0 causes precision marks for r{1,2}
- * when range information is propagated for 'if <reg> <op> <const>' insn.
+ * Range information is propagated for scalars sharing same ID
+ * for 'if <reg> <op> <const>' insn.
+ * The registers share the ID in the join of both branches too.
  */
 SEC("socket")
 __success __log_level(2)
-/* first 'if' branch */
-__msg("6: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 4: (25) if r1 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2 stack=:")
-__msg("frame0: regs=r0,r1,r2 stack= before 3: (bf) r2 = r0")
-/* second 'if' branch */
-__msg("from 4 to 5: ")
-__msg("6: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 5: (bf) r3 = r10")
-__msg("frame0: regs=r0 stack= before 4: (25) if r1 > 0x7 goto pc+0")
-/* parent state already has r{0,1,2} as precise */
-__msg("frame0: parent state regs= stack=:")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("4: (25) if r1 > 0x7 goto pc+0         ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("5: R0=" LINKED("1", 255, 0xff) " R1=" LINKED("1", 255, 0xff) " R2=" LINKED("1", 255, 0xff))
 __naked void linked_regs_bpf_k(void)
 {
 	asm volatile (
@@ -42,9 +37,6 @@ __naked void linked_regs_bpf_k(void)
 	"r1 = r0;"
 	"r2 = r0;"
 	"if r1 > 7 goto +0;"
-	/* force r0 to be precise, this eventually marks r1 and r2 as
-	 * precise as well because of shared IDs
-	 */
 	"r3 = r10;"
 	"r3 += r0;"
 	/* Mark r1 and r2 as alive. */
@@ -57,16 +49,14 @@ __naked void linked_regs_bpf_k(void)
 	: __clobber_all);
 }
 
-/* Registers r{0,1,2} share same ID when 'if r1 > ...' insn is processed,
- * check that verifier marks r{1,2} as precise while backtracking
- * 'if r1 > ...' with r0 already marked.
+/*
+ * Registers r{0,1,2} share same ID when 'if r1 > r3' insn is processed,
+ * check that range information is propagated to all of them.
  */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("frame0: regs=r0 stack= before 5: (2d) if r1 > r3 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3 stack=:")
-__msg("frame0: regs=r0,r1,r2,r3 stack= before 4: (b7) r3 = 7")
+__msg("5: (2d) if r1 > r3 goto pc+0          ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("6: R0=" LINKED("1", 255, 0xff) " R1=" LINKED("1", 255, 0xff) " R2=" LINKED("1", 255, 0xff))
 __naked void linked_regs_bpf_x_src(void)
 {
 	asm volatile (
@@ -78,9 +68,6 @@ __naked void linked_regs_bpf_x_src(void)
 	"r2 = r0;"
 	"r3 = 7;"
 	"if r1 > r3 goto +0;"
-	/* force r0 to be precise, this eventually marks r1 and r2 as
-	 * precise as well because of shared IDs
-	 */
 	"r4 = r10;"
 	"r4 += r0;"
 	/* Mark r1 and r2 as alive. */
@@ -93,16 +80,11 @@ __naked void linked_regs_bpf_x_src(void)
 	: __clobber_all);
 }
 
-/* Registers r{0,1,2} share same ID when 'if r1 > r3' insn is processed,
- * check that verifier marks r{0,1,2} as precise while backtracking
- * 'if r1 > r3' with r3 already marked.
- */
+/* Same as linked_regs_bpf_x_src(), r3 is what is used after the jump. */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("frame0: regs=r3 stack= before 5: (2d) if r1 > r3 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3 stack=:")
-__msg("frame0: regs=r0,r1,r2,r3 stack= before 4: (b7) r3 = 7")
+__msg("5: (2d) if r1 > r3 goto pc+0          ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("7: (0f) r4 += r3                      ; R3=7 R4=fp7")
 __naked void linked_regs_bpf_x_dst(void)
 {
 	asm volatile (
@@ -114,9 +96,6 @@ __naked void linked_regs_bpf_x_dst(void)
 	"r2 = r0;"
 	"r3 = 7;"
 	"if r1 > r3 goto +0;"
-	/* force r0 to be precise, this eventually marks r1 and r2 as
-	 * precise as well because of shared IDs
-	 */
 	"r4 = r10;"
 	"r4 += r3;"
 	/* Mark r1 and r2 as alive. */
@@ -131,16 +110,13 @@ __naked void linked_regs_bpf_x_dst(void)
 }
 
 /* Same as linked_regs_bpf_k, but break one of the
- * links, note that r1 is absent from regs=... in __msg below.
+ * links, note that r1 is absent from the first __msg below.
  */
 SEC("socket")
 __success __log_level(2)
-__msg("7: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 6: (bf) r3 = r10")
-__msg("frame0: parent state regs=r0 stack=:")
-__msg("frame0: regs=r0 stack= before 5: (25) if r0 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r2 stack=:")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* r1 is not there */
+__msg("5: (25) if r0 > 0x7 goto pc+0         ; R0=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7) "{{$}}")
+__msg("8: (bf) r1 = r1                       ; R1=0")
 __naked void linked_regs_broken_link(void)
 {
 	asm volatile (
@@ -155,9 +131,6 @@ __naked void linked_regs_broken_link(void)
 	 */
 	"r1 = 0;"
 	"if r0 > 7 goto +0;"
-	/* force r0 to be precise,
-	 * this eventually marks r2 as precise because of shared IDs
-	 */
 	"r3 = r10;"
 	"r3 += r0;"
 	/* Mark r1 and r2 as alive. */
@@ -170,50 +143,16 @@ __naked void linked_regs_broken_link(void)
 	: __clobber_all);
 }
 
-/* Check that precision marks propagate through scalar IDs.
- * Use the same scalar ID in multiple stack frames, check that
- * precision information is propagated up the call stack.
+/*
+ * Use the same scalar ID in multiple stack frames. A static function is verified
+ * on its own: registers of the callers are not there when r1 is compared, so it has no ID.
+ * They share their IDs after the calls like before, if there is more than one left.
  */
 SEC("socket")
 __success __log_level(2)
-__msg("17: (0f) r2 += r1")
-/* Current state */
-__msg("frame2: last_idx 17 first_idx 14 subseq_idx -1 ")
-__msg("frame2: regs=r1 stack= before 16: (bf) r2 = r10")
-__msg("frame2: parent state regs=r1 stack=")
-__msg("frame1: parent state regs= stack=")
-__msg("frame0: parent state regs= stack=")
-/* Parent state */
-__msg("frame2: last_idx 13 first_idx 13 subseq_idx 14 ")
-__msg("frame2: regs=r1 stack= before 13: (25) if r1 > 0x7 goto pc+0")
-__msg("frame2: parent state regs=r1 stack=")
-/* frame1.r{6,7} are marked because mark_precise_scalar_ids()
- * looks for all registers with frame2.r1.id in the current state
- */
-__msg("frame1: parent state regs=r6,r7 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame2: last_idx 9 first_idx 9 subseq_idx 13")
-__msg("frame2: regs=r1 stack= before 9: (85) call pc+3")
-/* frame1.r1 is marked because of backtracking of call instruction */
-__msg("frame1: parent state regs=r1,r6,r7 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame1: last_idx 8 first_idx 7 subseq_idx 9")
-__msg("frame1: regs=r1,r6,r7 stack= before 8: (bf) r7 = r1")
-__msg("frame1: regs=r1,r6 stack= before 7: (bf) r6 = r1")
-__msg("frame1: parent state regs=r1 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame1: last_idx 4 first_idx 4 subseq_idx 7")
-__msg("frame1: regs=r1 stack= before 4: (85) call pc+2")
-__msg("frame0: parent state regs=r1,r6 stack=")
-/* Parent state */
-__msg("frame0: last_idx 3 first_idx 1 subseq_idx 4")
-__msg("frame0: regs=r1,r6 stack= before 3: (bf) r6 = r0")
-__msg("frame0: regs=r0,r1 stack= before 2: (bf) r1 = r0")
-__msg("frame0: regs=r0 stack= before 1: (57) r0 &= 255")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("13: (25) if r1 > 0x7 goto pc+0        ; R1=" ALONE(7, 0x7))
+__msg("10: R0=0 R6=scalar(id={{.*}}umax32=255,{{.*}} R7=scalar(id={{.*}}umax32=255,")
+__msg("5: R0=0 R6=" ALONE(255, 0xff))
 __naked void precision_many_frames(void)
 {
 	asm volatile (
@@ -254,11 +193,6 @@ void precision_many_frames__bar(void)
 	"if r1 > 7 goto +0;"
 	"r6 = 0;" /* mark r6 as live */
 	"r7 = 0;" /* mark r7 as live */
-	/* force r1 to be precise, this eventually marks:
-	 * - bar frame r1
-	 * - foo frame r{1,6,7}
-	 * - main frame r{1,6}
-	 */
 	"r2 = r10;"
 	"r2 += r1;"
 	"r0 = 0;"
@@ -266,23 +200,13 @@ void precision_many_frames__bar(void)
 	::: __clobber_all);
 }
 
-/* Check that scalars with the same IDs are marked precise on stack as
+/*
+ * Check that range information is propagated to scalars with the same IDs on stack as
  * well as in registers.
  */
 SEC("socket")
 __success __log_level(2)
-__msg("12: (0f) r2 += r1")
-/* foo frame */
-__msg("frame1: regs=r1 stack= before 11: (bf) r2 = r10")
-__msg("frame1: regs=r1 stack= before 10: (25) if r1 > 0x7 goto pc+0")
-__msg("frame1: regs=r1 stack=-8,-16 before 9: (7b) *(u64 *)(r10 -16) = r1")
-__msg("frame1: regs=r1 stack=-8 before 8: (7b) *(u64 *)(r10 -8) = r1")
-__msg("frame1: regs=r1 stack= before 4: (85) call pc+3")
-/* main frame */
-__msg("frame0: regs=r1 stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r1")
-__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
-__msg("frame0: regs=r0 stack= before 1: (57) r0 &= 255")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("10: (25) if r1 > 0x7 goto pc+0        ; R1=scalar({{.*}}umax32=7,{{.*}} fp-8=scalar({{.*}}umax32=7,{{.*}} fp-16=scalar({{.*}}umax32=7,")
 __naked void precision_stack(void)
 {
 	asm volatile (
@@ -311,10 +235,6 @@ void precision_stack__foo(void)
 	"*(u64*)(r10 - 8) = r1;"
 	"*(u64*)(r10 - 16) = r1;"
 	"if r1 > 7 goto +0;"
-	/* force r1 to be precise, this eventually marks:
-	 * - foo frame r1,fp{-8,-16}
-	 * - main frame r1,fp{-8}
-	 */
 	"r2 = r10;"
 	"r2 += r1;"
 	"r0 = *(u64*)(r10 - 8);"
@@ -329,20 +249,9 @@ void precision_stack__foo(void)
 SEC("socket")
 __success __log_level(2)
 /* r{6,7} */
-__msg("12: (0f) r3 += r7")
-__msg("frame0: regs=r7 stack= before 11: (bf) r3 = r10")
-__msg("frame0: regs=r7 stack= before 9: (25) if r7 > 0x7 goto pc+0")
-/* ... skip some insns ... */
-__msg("frame0: regs=r6,r7 stack= before 3: (bf) r7 = r0")
-__msg("frame0: regs=r0,r6 stack= before 2: (bf) r6 = r0")
+__msg("9: (25) if r7 > 0x7 goto pc+0         ; R6=" LINKED("1", 7, 0x7) " R7=" LINKED("1", 7, 0x7) "{{$}}")
 /* r{8,9} */
-__msg("13: (0f) r3 += r9")
-__msg("frame0: regs=r9 stack= before 12: (0f) r3 += r7")
-/* ... skip some insns ... */
-__msg("frame0: regs=r9 stack= before 10: (25) if r9 > 0x7 goto pc+0")
-__msg("frame0: regs=r8,r9 stack= before 7: (bf) r9 = r0")
-__msg("frame0: regs=r0,r8 stack= before 6: (bf) r8 = r0")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("10: (25) if r9 > 0x7 goto pc+0        ; R8=" LINKED("2", 7, 0x7) " R9=" LINKED("2", 7, 0x7) "{{$}}")
 __naked void precision_two_ids(void)
 {
 	asm volatile (
@@ -360,13 +269,11 @@ __naked void precision_two_ids(void)
 	"r9 = r0;"
 	/* clear r0 id */
 	"r0 = 0;"
-	/* propagate equal scalars precision */
+	/* propagate range information to equal scalars */
 	"if r7 > 7 goto +0;"
 	"if r9 > 7 goto +0;"
 	"r3 = r10;"
-	/* force r7 to be precise, this also marks r6 */
 	"r3 += r7;"
-	/* force r9 to be precise, this also marks r8 */
 	"r3 += r9;"
 	"r6 = r6;" /* mark r6 as live */
 	"r8 = r8;" /* mark r8 as live */
@@ -378,7 +285,6 @@ __naked void precision_two_ids(void)
 
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 /*
  * check that r5 is unlinked after 'if', collect_linked_regs() can't tie
  * more than 5 registers for a single insn and the register compared by
@@ -386,9 +292,6 @@ __flag(BPF_F_TEST_STATE_FREQ)
  */
 __msg("7: (25) if r5 > 0x7 goto pc+0         ; R5=scalar(smin=")
 __msg("12: (bf) r5 = r5                      ; R5=scalar(id=2")
-/* check that r{0-4} are marked precise after 'if' */
-__msg("frame0: regs=r0 stack= before 7: (25) if r5 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3,r4 stack=:")
 __naked void linked_regs_too_many_regs(void)
 {
 	asm volatile (
@@ -410,9 +313,6 @@ __naked void linked_regs_too_many_regs(void)
 	"r4 = r4;"
 	/* make r5 appear in the log */
 	"r5 = r5;"
-	/* force r0 to be precise,
-	 * this would cause r{0-4} to be precise because of shared IDs
-	 */
 	"r7 = r10;"
 	"r7 += r0;"
 	"r0 = 0;"
@@ -424,10 +324,7 @@ __naked void linked_regs_too_many_regs(void)
 
 SEC("socket")
 __failure __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("regs=r7 stack= before 5: (3d) if r8 >= r0")
-__msg("parent state regs=r0,r7,r8")
-__msg("regs=r0,r7,r8 stack= before 4: (25) if r0 > 0x1")
+__msg("5: (3d) if r8 >= r0 goto pc+3         ; R0=scalar(umin=1) R7=scalar(id=1,umax=0xfffffffffffffffe) R8=scalar(id=1,umax=0xfffffffffffffffe)")
 __msg("div by zero")
 __naked void linked_regs_broken_link_2(void)
 {
@@ -438,13 +335,11 @@ __naked void linked_regs_broken_link_2(void)
 	"call %[bpf_get_prandom_u32];"
 	"if r0 > 1 goto +0;"
 	/* r7.id == r8.id,
-	 * thus r7 precision implies r8 precision,
-	 * which implies r0 precision because of the conditional below.
+	 * thus what is learned about r8 is learned about r7
 	 */
 	"if r8 >= r0 goto 1f;"
 	/* break id relation between r7 and r8 */
 	"r8 += r8;"
-	/* make r7 precise */
 	"if r7 == 0 goto 1f;"
 	"r0 /= 0;"
 "1:"
@@ -455,13 +350,11 @@ __naked void linked_regs_broken_link_2(void)
 	: __clobber_all);
 }
 
-/* Check that mark_chain_precision() for one of the conditional jump
- * operands does not trigger equal scalars precision propagation.
- */
+/* A conditional jump that is predicted says nothing new about its operands. */
 SEC("socket")
 __success __log_level(2)
-__msg("3: (25) if r1 > 0x100 goto pc+0")
-__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
+__msg("3: (25) if r1 > 0x100 goto pc+0       ; R1=" LINKED("1", 255, 0xff) "{{$}}")
+__msg("processed 6 insns")
 __naked void cjmp_no_linked_regs_trigger(void)
 {
 	asm volatile (
@@ -470,9 +363,7 @@ __naked void cjmp_no_linked_regs_trigger(void)
 	"r0 &= 0xff;"
 	/* tie r0.id == r1.id */
 	"r1 = r0;"
-	/* the jump below would be predicted, thus r1 would be marked precise,
-	 * this should not imply precision mark for r0
-	 */
+	/* the jump below is predicted */
 	"if r1 > 256 goto +0;"
 	"r0 = 0;"
 	"exit;"
@@ -497,7 +388,6 @@ __naked void cjmp_no_linked_regs_trigger(void)
  */
 SEC("socket")
 __failure __msg("register with unbounded min value")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void check_ids_in_regsafe(void)
 {
 	asm volatile (
@@ -539,12 +429,9 @@ __naked void check_ids_in_regsafe(void)
  *   (2) r6{.id=B}, r7{.id=A}, r8{.id=B}
  *
  * Where (2) is not safe, as "r7 > 4" check won't propagate range for it.
- * This example would be considered safe without changes to
- * mark_chain_precision() to track scalar values with equal IDs.
  */
 SEC("socket")
 __failure __msg("register with unbounded min value")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void check_ids_in_regsafe_2(void)
 {
 	asm volatile (
@@ -595,24 +482,24 @@ __naked void check_ids_in_regsafe_2(void)
  * If such IDs *are* generated the 'l1' below would be reached in
  * two states:
  *
- *   (1) r1{.id=A}, r2{.id=A}
- *   (2) r1{.id=C}, r2{.id=C}
+ *   (1) r3{.id=A}, r4{.id=A}, which is there first
+ *   (2) r3{.id=B}, r4{.id=C}
  *
- * Thus forcing 'if r1 == r2' verification twice.
+ * and (2) is not within (1).
  */
 SEC("socket")
 __success __log_level(2)
-__msg("14: (1d) if r3 == r4 goto pc+0")
-__msg("frame 0: propagating r3,r4")
-__msg("14: safe")
+__msg("from 14 to 15: within")
+__msg("15: (1d) if r3 == r4 goto pc+0        ; R3=0 R4=0")
 __msg("processed 18 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void no_scalar_id_for_const(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
 	/* unpredictable jump */
-	"if r0 > 7 goto l0_%=;"
+	"if r0 <= 7 goto l2_%=;"
+	"goto l0_%=;"
+"l2_%=:"
 	/* possibly generate same scalar ids for r3 and r4 */
 	"r1 = 0;"
 	"r1 ^= r1;" /* prevent bpf_prune_dead_branches from folding the branch */
@@ -629,7 +516,7 @@ __naked void no_scalar_id_for_const(void)
 	"r3 = r1;"
 	"r4 = r2;"
 "l1_%=:"
-	/* predictable jump, marks r3 and r4 precise */
+	/* predictable jump */
 	"if r3 == r4 goto +0;"
 	"r0 = 0;"
 	"exit;"
@@ -641,17 +528,17 @@ __naked void no_scalar_id_for_const(void)
 /* Same as no_scalar_id_for_const() but for 32-bit values */
 SEC("socket")
 __success __log_level(2)
-__msg("14: (1e) if w3 == w4 goto pc+0")
-__msg("frame 0: propagating r3,r4")
-__msg("14: safe")
+__msg("from 14 to 15: within")
+__msg("15: (1e) if w3 == w4 goto pc+0        ; R3=0 R4=0")
 __msg("processed 18 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void no_scalar_id_for_const32(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
 	/* unpredictable jump */
-	"if r0 > 7 goto l0_%=;"
+	"if r0 <= 7 goto l2_%=;"
+	"goto l0_%=;"
+"l2_%=:"
 	/* possibly generate same scalar ids for r3 and r4 */
 	"w1 = 0;"
 	"w1 ^= w1;" /* prevent bpf_prune_dead_branches from folding the branch */
@@ -668,7 +555,7 @@ __naked void no_scalar_id_for_const32(void)
 	"w3 = w1;"
 	"w4 = w2;"
 "l1_%=:"
-	/* predictable jump, marks r3 and r4 precise */
+	/* predictable jump */
 	"if w3 == w4 goto +0;"
 	"r0 = 0;"
 	"exit;"
@@ -677,20 +564,20 @@ __naked void no_scalar_id_for_const32(void)
 	: __clobber_all);
 }
 
-/* Check that unique scalar IDs are ignored when new verifier state is
- * compared to cached verifier state. For this test:
- * - cached state has no id on r1
- * - new state has a unique id on r1
+/*
+ * Check that unique scalar IDs are ignored when a verifier state is
+ * compared to the state that is there. For this test:
+ * - the state that is there had a unique id on r1
+ * - the state that arrives has no id on r1
  */
 SEC("socket")
 __success __log_level(2)
 __msg("6: (25) if r6 > 0x7 goto pc+1")
 __msg("7: (57) r1 &= 255")
-__msg("8: (bf) r2 = r10")
-__msg("from 6 to 8: safe")
-__msg("processed 12 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
-__naked void ignore_unique_scalar_ids_cur(void)
+__msg("from 7 to 8: within")
+__msg("8: R0=0 R1=" ALONE(255, 0xff))
+__msg("processed 11 insns")
+__naked void ignore_unique_scalar_ids_old(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
@@ -706,8 +593,8 @@ __naked void ignore_unique_scalar_ids_cur(void)
 	"r1 &= 0xff;"
 "l0_%=:"
 	/* get here in two states:
-	 * - first: r1 has no id (cached state)
-	 * - second: r1 has a unique id (should be considered equivalent)
+	 * - first: r1 has a unique id
+	 * - second: r1 has no id (should be considered to be within the first)
 	 */
 	"r2 = r10;"
 	"r2 += r1;"
@@ -717,20 +604,20 @@ __naked void ignore_unique_scalar_ids_cur(void)
 	: __clobber_all);
 }
 
-/* Check that unique scalar IDs are ignored when new verifier state is
- * compared to cached verifier state. For this test:
- * - cached state has a unique id on r1
- * - new state has no id on r1
+/*
+ * Check that unique scalar IDs are ignored when a verifier state is
+ * compared to the state that is there. For this test:
+ * - the state that is there has no id on r1
+ * - the state that arrives has a unique id on r1
  */
 SEC("socket")
 __success __log_level(2)
 __msg("6: (25) if r6 > 0x7 goto pc+1")
 __msg("7: (05) goto pc+1")
-__msg("9: (bf) r2 = r10")
-__msg("9: safe")
-__msg("processed 13 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
-__naked void ignore_unique_scalar_ids_old(void)
+__msg("from 7 to 9: within")
+__msg("9: R0=0 R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R10=fp0")
+__msg("processed 12 insns")
+__naked void ignore_unique_scalar_ids_cur(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
@@ -748,8 +635,8 @@ __naked void ignore_unique_scalar_ids_old(void)
 	"r1 &= 0xff;"
 "l0_%=:"
 	/* get here in two states:
-	 * - first: r1 has a unique id (cached state)
-	 * - second: r1 has no id (should be considered equivalent)
+	 * - first: r1 has no id
+	 * - second: r1 has a unique id (should be considered to be within the first)
 	 */
 	"r2 = r10;"
 	"r2 += r1;"
@@ -759,16 +646,14 @@ __naked void ignore_unique_scalar_ids_old(void)
 	: __clobber_all);
 }
 
-/* Check that two registers with 0 scalar IDs in a verified state can be mapped
- * to the same scalar ID in current state.
+/*
+ * Check that two registers with 0 scalar IDs in the state that is there can be mapped
+ * to the same scalar ID in the state that arrives.
  */
 SEC("socket")
 __success __log_level(2)
-/* The states should be equivalent on reaching insn 12.
- */
-__msg("12: safe")
-__msg("processed 17 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 10 to 11: within")
+__msg("processed 15 insns")
 __naked void two_nil_old_ids_one_cur_id(void)
 {
 	asm volatile (
@@ -783,16 +668,14 @@ __naked void two_nil_old_ids_one_cur_id(void)
 	"r7 *= 1;"
 	"r0 = 0;"
 	/* Maybe make r{6,7} IDs identical */
-	"if r6 > r7 goto l0_%=;"
-	"goto l1_%=;"
-"l0_%=:"
+	"if r6 <= r7 goto l1_%=;"
 	"r6 = r7;"
 "l1_%=:"
-	/* Mark r{6,7} precise.
+	/*
 	 * Get here in two states:
-	 * - first:  r6{.id=0}, r7{.id=0} (cached state)
+	 * - first:  r6{.id=0}, r7{.id=0}
 	 * - second: r6{.id=A}, r7{.id=A}
-	 * Verifier considers such states equivalent.
+	 * Verifier considers the second one to be within the first one.
 	 * Thus "exit;" would be verified only once.
 	 */
 	"r2 = r10;"
@@ -804,18 +687,14 @@ __naked void two_nil_old_ids_one_cur_id(void)
 	: __clobber_all);
 }
 
-/* Check that two different scalar IDs in a verified state can't be
- * mapped to the same scalar ID in current state.
+/*
+ * Check that two different scalar IDs in the state that is there can't be
+ * mapped to the same scalar ID in the state that arrives.
  */
 SEC("socket")
 __success __log_level(2)
-/* The exit instruction should be reachable from two states,
- * use two matches and "processed .. insns" to ensure this.
- */
-__msg("16: (95) exit")
-__msg("16: (95) exit")
-__msg("processed 22 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__not_msg("from 10 to 11: within")
+__msg("11: R0=0 R6=" ALONE(255, 0xff) " R7=" ALONE(255, 0xff))
 __naked void two_old_ids_one_cur_id(void)
 {
 	asm volatile (
@@ -830,24 +709,22 @@ __naked void two_old_ids_one_cur_id(void)
 	"r9 = r0;"
 	"r0 = 0;"
 	/* Maybe make r{6,7} IDs identical */
-	"if r6 > r7 goto l0_%=;"
-	"goto l1_%=;"
-"l0_%=:"
+	"if r6 <= r7 goto l1_%=;"
 	"r6 = r7;"
 "l1_%=:"
-	/* Mark r{6,7} precise.
+	/*
 	 * Get here in two states:
-	 * - first:  r6{.id=A}, r7{.id=B} (cached state)
-	 * - second: r6{.id=A}, r7{.id=A}
-	 * Currently we don't want to consider such states equivalent.
-	 * Thus "exit;" would be verified twice.
+	 * - first:  r6{.id=A}, r7{.id=B}
+	 * - second: r6{.id=B}, r7{.id=B}
+	 * Currently we don't want to consider the second one to be within
+	 * the first one. Thus they are joined.
 	 */
 	"r2 = r10;"
 	"r2 += r6;"
 	"r2 += r7;"
 	/*
 	 * keep r8 and r9 live, otherwise r6->id and r7->id
-	 * will become singular and reset to zero before if r6 > r7
+	 * will become singular and reset to zero before if r6 <= r7
 	 */
 	"r9 += r8;"
 	"exit;"
@@ -878,10 +755,6 @@ __log_level(2)
 __msg("4: (77) r1 >>= 32                     ; R1=0")
 __msg("5: (bf) r0 = r1                       ; R0=0 R1=0")
 __msg("6: (95) exit")
-__msg("from 3 to 4")
-__msg("4: (77) r1 >>= 32                     ; R1=0")
-__msg("5: (bf) r0 = r1                       ; R0=0 R1=0")
-__msg("6: (95) exit")
 /* Verify that statements to randomize upper half of r1 had not been
  * generated.
  */
diff --git a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
index f40e57251e94..9ff29a981a41 100644
--- a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
+++ b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
@@ -96,7 +96,7 @@ l1_%=:	exit;						\
 SEC("socket")
 __description("varlen_map_value_access pruning")
 __failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void varlen_map_value_access_pruning(void)
 {
@@ -110,8 +110,9 @@ __naked void varlen_map_value_access_pruning(void)
 	if r0 == 0 goto l0_%=;				\
 	r1 = *(u64*)(r0 + 0);				\
 	w2 = %[max_entries];				\
-	if r2 s> r1 goto l1_%=;				\
-	w1 = 0;						\
+	if r2 s<= r1 goto l3_%=;			\
+	goto l1_%=;					\
+l3_%=:	w1 = 0;						\
 l1_%=:	w1 <<= 2;					\
 	r0 += r1;					\
 	goto l2_%=;					\
@@ -193,7 +194,7 @@ l0_%=:	exit;						\
 
 SEC("tracepoint")
 __description("precision tracking for u32 spill/fill")
-__failure __msg("R0 min value is outside of the allowed memory range")
+__failure __msg("invalid access to map value, value_size=8 off=32 size=4")
 __naked void tracking_for_u32_spill_fill(void)
 {
 	asm volatile ("					\
@@ -202,7 +203,7 @@ __naked void tracking_for_u32_spill_fill(void)
 	w6 = 32;					\
 	if r0 == 0 goto l0_%=;				\
 	w6 = 4;						\
-l0_%=:	/* Additional insns to introduce a pruning point. */\
+l0_%=:	/* Additional insns to have one more place where paths meet. */\
 	call %[bpf_get_prandom_u32];			\
 	r3 = 0;						\
 	r3 = 0;						\
@@ -239,28 +240,21 @@ __naked void for_u32_spills_u64_fill(void)
 	call %[bpf_get_prandom_u32];			\
 	r6 = r0;					\
 	w7 = 0xffffffff;				\
-	/* Additional insns to introduce a pruning point. */\
-	r3 = 1;						\
-	r3 = 1;						\
-	r3 = 1;						\
-	r3 = 1;						\
+	r9 = 0xffffffff ll;				\
+	/* Additional insns to have a place where paths meet. */\
 	call %[bpf_get_prandom_u32];			\
 	if r0 == 0 goto l0_%=;				\
 	r3 = 1;						\
-l0_%=:	w3 /= 0;					\
+l0_%=:	r3 = 1;						\
 	/* u32 spills, u64 fill */			\
 	*(u32*)(r10 - 4) = r6;				\
 	*(u32*)(r10 - 8) = r7;				\
 	r8 = *(u64*)(r10 - 8);				\
 	/* if r8 != X goto pc+1  r8 known in fallthrough branch */\
-	if r8 != 0xffffffff goto l1_%=;			\
+	if r8 != r9 goto l1_%=;				\
 	r3 = 1;						\
-l1_%=:	/* if r8 == X goto pc+1  condition always true on first\
-	 * traversal, so starts backtracking to mark r8 as requiring\
-	 * precision. r7 marked as needing precision. r6 not marked\
-	 * since it's not tracked.			\
-	 */						\
-	if r8 == 0xffffffff goto l2_%=;			\
+l1_%=:	/* if r8 == X goto pc+1 */			\
+	if r8 == r9 goto l2_%=;				\
 	/* fails if r8 correctly marked unknown after fill. */\
 	w3 /= 0;					\
 l2_%=:	r0 = 0;						\
@@ -272,7 +266,8 @@ l2_%=:	r0 = 0;						\
 
 SEC("socket")
 __description("allocated_stack")
-__success __msg("processed 15 insns")
+/* every insn once: the state that wrote to the stack is within the state that did not */
+__success __msg("processed 14 insns")
 __success_unpriv __msg_unpriv("") __log_level(1) __retval(0)
 __naked void allocated_stack(void)
 {
@@ -337,12 +332,9 @@ l0_%=:	r1 = 42;					\
 	: __clobber_all);
 }
 
-/* Without checkpoint forcibly inserted at the back-edge a loop this
- * test would take a very long time to verify.
- */
+/* The number of trips is not known */
 SEC("kprobe")
-__failure __log_level(4)
-__msg("BPF program is too large.")
+__success
 __naked void short_loop1(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
index 4d1374885f6d..34c551f97e61 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
@@ -518,15 +518,6 @@ __msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0")
  */
 __msg("10: (71) r2 = *(u8 *)(r10 -9)         ; R2=0 R10=fp0 fp-16=0")
 __msg("11: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (71) r2 = *(u8 *)(r10 -1)")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 4: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (b7) r0 = 0")
 __naked void partial_stack_load_preserves_zeros(void)
 {
 	asm volatile (
@@ -600,8 +591,6 @@ __success
 __msg("2: (62) *(u32 *)(r10 -4) = 0          ; R10=fp0 fp-8=0000????")
 __msg("4: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0000????")
 __msg("5: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (71) r2 = *(u8 *)(r10 -1)")
 __naked void partial_stack_load_preserves_partial_zeros(void)
 {
 	asm volatile (
@@ -637,7 +626,6 @@ __naked void partial_stack_load_preserves_partial_zeros(void)
 SEC("raw_tp")
 __log_level(2)
 __success
-__msg("mark_precise: frame0: regs= stack=-8")
 __msg("R2=0")
 __naked void stack_load_preserves_mixed_zero_and_zero_spill(void)
 {
@@ -663,50 +651,21 @@ __naked void stack_load_preserves_mixed_zero_and_zero_spill(void)
 char two_byte_buf[2] SEC(".data.two_byte_buf");
 
 SEC("raw_tp")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2)
 __success
-/* make sure fp-8 is IMPRECISE fake register spill */
+/* make sure fp-8 is fake register spill */
 __msg("3: (7a) *(u64 *)(r10 -8) = 1          ; R10=fp0 fp-8=1")
-/* and fp-16 is spilled IMPRECISE const reg */
+/* and fp-16 is spilled const reg */
 __msg("5: (7b) *(u64 *)(r10 -16) = r0        ; R0=1 R10=fp0 fp-16=1")
 /* validate load from fp-8, which was initialized using BPF_ST_MEM */
 __msg("8: (79) r2 = *(u64 *)(r10 -8)         ; R2=1 R10=fp0 fp-8=1")
-__msg("9: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-/* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7a) *(u64 *)(r10 -8) = 1")
-__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("9: (0f) r1 += r2                      ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 /* validate load from fp-16, which was initialized using BPF_STX_MEM */
 __msg("12: (79) r2 = *(u64 *)(r10 -16)       ; R2=1 R10=fp0 fp-16=1")
-__msg("13: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (79) r2 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-/* now both fp-8 and fp-16 are precise, very good */
-__msg("mark_precise: frame0: parent state regs= stack=-16:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("13: (0f) r1 += r2                     ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 __naked void stack_load_preserves_const_precision(void)
 {
 	asm volatile (
-		/* establish checkpoint with state that has no stack slots;
-		 * if we bubble up to this state without finding desired stack
-		 * slot, then it's a bug and should be caught
-		 */
 		"goto +0;"
 
 		/* fp-8 is const 1 *fake* register */
@@ -716,7 +675,6 @@ __naked void stack_load_preserves_const_precision(void)
 		"r0 = 1;"
 		"*(u64 *)(r10 -16) = r0;"
 
-		/* force checkpoint to check precision marks preserved in parent states */
 		"goto +0;"
 
 		/* load single U64 from aligned FAKE_REG=1 slot */
@@ -740,48 +698,21 @@ __naked void stack_load_preserves_const_precision(void)
 }
 
 SEC("raw_tp")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2)
 __success
 /* make sure fp-8 is 32-bit FAKE subregister spill */
 __msg("3: (62) *(u32 *)(r10 -8) = 1          ; R10=fp0 fp-8=????1")
-/* but fp-16 is spilled IMPRECISE zero const reg */
+/* and fp-16 is spilled const reg */
 __msg("5: (63) *(u32 *)(r10 -16) = r0        ; R0=1 R10=fp0 fp-16=????1")
 /* validate load from fp-8, which was initialized using BPF_ST_MEM */
 __msg("8: (61) r2 = *(u32 *)(r10 -8)         ; R2=1 R10=fp0 fp-8=????1")
-__msg("9: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (62) *(u32 *)(r10 -8) = 1")
-__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("9: (0f) r1 += r2                      ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 /* validate load from fp-16, which was initialized using BPF_STX_MEM */
 __msg("12: (61) r2 = *(u32 *)(r10 -16)       ; R2=1 R10=fp0 fp-16=????1")
-__msg("13: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (61) r2 = *(u32 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-16:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("13: (0f) r1 += r2                     ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 __naked void stack_load_preserves_const_precision_subreg(void)
 {
 	asm volatile (
-		/* establish checkpoint with state that has no stack slots;
-		 * if we bubble up to this state without finding desired stack
-		 * slot, then it's a bug and should be caught
-		 */
 		"goto +0;"
 
 		/* fp-8 is const 1 *fake* SUB-register */
@@ -791,7 +722,6 @@ __naked void stack_load_preserves_const_precision_subreg(void)
 		"r0 = 1;"
 		"*(u32 *)(r10 -16) = r0;"
 
-		/* force checkpoint to check precision marks preserved in parent states */
 		"goto +0;"
 
 		/* load single U32 from aligned FAKE_REG=1 slot */
@@ -1118,41 +1048,37 @@ l0_%=:	r1 >>= 32;					\
 	: __clobber_all);
 }
 
-/* stacksafe(): check if stack spill of an imprecise scalar in old state
- * is considered equivalent to STACK_{MISC,INVALID} in cur state.
+/*
+ * stacksafe(): the state with a stack spill of a scalar is there (label 1 is
+ * looked at first), the state with STACK_{MISC,INVALID} arrives.
+ * The first one is within the second one, which stands for both.
  */
 SEC("socket")
 __success __log_level(2)
+__msg("8: R0=scalar() R10=fp0 fp-8=??mm??mm")
 __msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-/* STACK_INVALID should prevent verifier in unpriv mode from
- * considering states equivalent and force an error on second
- * verification path (entry - label 1 - label 2).
- */
+__msg("processed 10 insns")
+/* STACK_INVALID is an error in unpriv mode */
 __failure_unpriv
 __msg_unpriv("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg_unpriv("9: (95) exit")
-__msg_unpriv("8: (79) r1 = *(u64 *)(r10 -8)")
 __msg_unpriv("invalid read from stack off -8+2 size 8")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void old_imprecise_scalar_vs_cur_stack_misc(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
-	/* conjure scalar at fp-8 */
-	"r0 = 42;"
-	"*(u64*)(r10 - 8) = r0;"
-	"goto 2f;"
-"1:"
 	/* conjure STACK_{MISC,INVALID} at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u16*)(r10 - 8) = r0;"
 	"*(u16*)(r10 - 4) = r0;"
+	"goto 2f;"
+"1:"
+	/* conjure scalar at fp-8 */
+	"r0 = 42;"
+	"*(u64*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r1 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
@@ -1160,71 +1086,69 @@ __naked void old_imprecise_scalar_vs_cur_stack_misc(void)
 	: __clobber_all);
 }
 
-/* stacksafe(): check that stack spill of a precise scalar in old state
- * is not considered equivalent to STACK_MISC in cur state.
+/*
+ * stacksafe(): check that the state with STACK_MISC is not considered to be
+ * within the state with a stack spill of a known scalar, which is there.
  */
 SEC("socket")
 __success __log_level(2)
-/* verifier should visit 'if r1 == 0x2a ...' two times:
- * - once for path entry - label 2;
- * - once for path entry - label 1 - label 2.
- */
-__msg("if r1 == 0x2a goto pc+0")
-__msg("if r1 == 0x2a goto pc+0")
-__msg("processed 15 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* what is loaded is not known to be 42 */
+__msg("8: (79) r1 = *(u64 *)(r10 -8)         ; R1=scalar() R10=fp0 fp-8=mmmmmmmm")
+__msg("9: (15) if r1 == 0x2a goto pc+0       ; R1=scalar()")
+__msg("processed 12 insns")
 __naked void old_precise_scalar_vs_cur_stack_misc(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
-	/* conjure scalar at fp-8 */
-	"r0 = 42;"
-	"*(u64*)(r10 - 8) = r0;"
-	"goto 2f;"
-"1:"
 	/* conjure STACK_MISC at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u64*)(r10 - 8) = r0;"
 	"*(u32*)(r10 - 4) = r0;"
+	"goto 2f;"
+"1:"
+	/* conjure scalar at fp-8 */
+	"r0 = 42;"
+	"*(u64*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should not be considered safe on second visit */
 	"r1 = *(u64*)(r10 - 8);"
-	/* use r1 in precise context */
+	/* the value of r1 matters */
 	"if r1 == 42 goto +0;"
+	/* r0 is dead where the states meet: they differ in fp-8 only */
+	"r0 = 0;"
 	"exit;"
 	:
 	: __imm(bpf_ktime_get_ns)
 	: __clobber_all);
 }
 
-/* stacksafe(): check if STACK_MISC in old state is considered
- * equivalent to stack spill of a scalar in cur state.
+/*
+ * stacksafe(): check if the state with a stack spill of a scalar is considered
+ * to be within the state with STACK_MISC, which is there.
  */
 SEC("socket")
 __success  __log_level(2)
+__msg("from 4 to 8: within")
 __msg("8: (79) r0 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 10 insns")
 __naked void old_stack_misc_vs_cur_scalar(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
+	/* conjure scalar at fp-8 */
+	"r0 = 42;"
+	"*(u64*)(r10 - 8) = r0;"
+	"goto 2f;"
+"1:"
 	/* conjure STACK_{MISC,INVALID} at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u16*)(r10 - 8) = r0;"
 	"*(u16*)(r10 - 4) = r0;"
-	"goto 2f;"
-"1:"
-	/* conjure scalar at fp-8 */
-	"r0 = 42;"
-	"*(u64*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r0 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
@@ -1232,21 +1156,16 @@ __naked void old_stack_misc_vs_cur_scalar(void)
 	: __clobber_all);
 }
 
-/* stacksafe(): check that STACK_MISC in old state is not considered
- * equivalent to stack spill of a non-scalar in cur state.
+/*
+ * stacksafe(): the state with STACK_MISC is there, the state with a stack spill
+ * of a non-scalar arrives. A program that may read uninitialized stack may
+ * take a pointer for a number: the second state is within the first one.
  */
 SEC("socket")
 __success  __log_level(2)
-/* verifier should process exit instructions twice:
- * - once for path entry - label 2;
- * - once for path entry - label 1 - label 2.
- */
-__msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("9: (95) exit")
-__msg("from 2 to 7")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 4 to 8: within")
+__msg("8: (79) r1 = *(u64 *)(r10 -8)         ; R1=scalar()")
+__msg("processed 10 insns")
 __naked void old_stack_misc_vs_cur_ctx_ptr(void)
 {
 	asm volatile(
@@ -1255,16 +1174,16 @@ __naked void old_stack_misc_vs_cur_ctx_ptr(void)
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
+	/* conjure context pointer in fp-8 */
+	"*(u64*)(r10 - 8) = r9;"
+	"goto 2f;"
+"1:"
 	/* conjure STACK_MISC at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u64*)(r10 - 8) = r0;"
 	"*(u32*)(r10 - 4) = r0;"
-	"goto 2f;"
-"1:"
-	/* conjure context pointer in fp-8 */
-	"*(u64*)(r10 - 8) = r9;"
 "2:"
-	/* read fp-8, should not be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r1 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
@@ -1310,35 +1229,34 @@ __naked void stack_noperfmon_spill_32bit_onto_64bit_slot(void)
 }
 
 /*
- * stacksafe(): check if 32-bit scalar spill in old state is considered
- * equivalent to STACK_MISC in cur state.
+ * stacksafe(): the state with a 32-bit spill of an unknown scalar is there, the
+ * state with STACK_MISC arrives. The second one is within the first one.
  * 32-bit scalar spill creates slot[0-3] = STACK_MISC, slot[4-7] = STACK_SPILL.
  * Without 32-bit spill support in stacksafe(), the STACK_SPILL vs STACK_MISC
- * mismatch at slot[4] causes pruning to fail.
+ * mismatch at slot[4] says that it is not.
  */
 SEC("socket")
 __success __log_level(2)
+__msg("from 5 to 8: within")
 __msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 10 insns")
 __naked void old_imprecise_scalar32_vs_cur_stack_misc(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
-	/* conjure 32-bit scalar spill at fp-8 */
-	"r0 = 42;"
-	"*(u32*)(r10 - 8) = r0;"
-	"goto 2f;"
-"1:"
 	/* conjure STACK_MISC at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u16*)(r10 - 8) = r0;"
 	"*(u16*)(r10 - 6) = r0;"
+	"goto 2f;"
+"1:"
+	/* conjure 32-bit scalar spill at fp-8 */
+	"call %[bpf_ktime_get_ns];"
+	"*(u32*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r1 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
diff --git a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
index d9d7b05cf6d2..f4e7f47e4cdd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
@@ -469,9 +469,9 @@ l3_%=:	r1 = r7;					\
 
 SEC("cgroup/skb")
 __description("spin_lock: regsafe() check_ids() similar id mappings")
-__success __msg("29: safe")
+__success __msg("to 29: within")
 __failure_unpriv __msg_unpriv("")
-__log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2) __retval(0)
 __naked void check_ids_similar_id_mappings(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
index e6a9021b197b..e9194a560fbd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
@@ -587,14 +587,13 @@ __naked void stack_check_size_512_with_may_goto(void)
 #endif
 
 /*
- * Verify that old PTR_TO_STACK state is considered a super-set of
- * new PTR_TO_STACK state when the new offset is within the old range,
- * e.g. old [-72, -16] vs new {-16}.
+ * Verify that a PTR_TO_STACK state is considered a super-set of
+ * another PTR_TO_STACK state when the offset of that is within its range,
+ * e.g. [-72, -16] vs {-16}.
  */
 SEC("socket")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 /*
  * r7 is widened to [-72, -16] at the loop header (insn 5),
  * the loop body sees [-64, -8] after 'r7 += 8'.
@@ -604,9 +603,9 @@ __msg("R7=fp(smin=smin32=-64,smax=smax32=-8")
 /* 'r7 = r8' sets r7 to fp-16; its ANY SCEV is not clamped. */
 __msg("R7=fp-16 R8=fp-16 loop_stack=5")
 __not_msg("clamping r7")
-/* back-edge state is pruned because {-16} is within [-72, -16]. */
-__msg("from 8 to 5: safe")
-__msg("processed 12 insns")
+/* back-edge state goes no further because {-16} is within [-72, -16]. */
+__msg("from 8 to 5: within")
+__msg("processed 11 insns")
 __naked void stack_ptr_subrange_in_loop(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
index dc0c7034c04f..041b7a71fb3f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
@@ -39,31 +39,21 @@ static unsigned long callback_subprog()
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("7: (0f) r1 += r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r0                      ; R0=12 R1=map_value(map=.data.vals,ks=4,vs=16,imm=12)")
 __naked int subprog_result_precise(void)
 {
 	asm volatile (
 		"r6 = 3;"
-		/* pass r6 through r1 into subprog to get it back as r0;
-		 * this whole chain will have to be marked as precise later
-		 */
+		/* pass r6 through r1 into subprog to get it back as r0 */
 		"r1 = r6;"
 		"call identity_subprog;"
 		/* now use subprog's returned value (which is a
-		 * r6 -> r1 -> r0 chain), as index into vals array, forcing
-		 * all of that to be known precisely
+		 * r6 -> r1 -> r0 chain), as index into vals array
 		 */
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* here r0->r1->r6 chain is forced to be precise and has to be
-		 * propagated back to the beginning, including through the
+		/*
+		 * here r0 has to be known: the constant went through the
 		 * subprog call
 		 */
 		"r1 += r0;"
@@ -98,26 +88,18 @@ static unsigned long sneaky_fp_leaking_subprog()
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
-__msg("mark_precise: frame0: regs=r0 stack= before 10: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 9: (bf) r0 = (s8)r10")
-__msg("7: R0=scalar")
+__msg("3: R0=scalar() R6=")
+__msg("6: (0f) r1 += r0                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int fp_precise_subprog_result(void)
 {
 	asm volatile (
 		"call fp_leaking_subprog;"
 		/* use subprog's returned value (which is derived from r10=fp
-		 * register), as index into vals array, forcing all of that to
-		 * be known precisely
+		 * register, so it's unknown), as index into vals array
 		 */
 		"r0 &= 3;"
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* force precision marking */
 		"r1 += r0;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -129,31 +111,18 @@ __naked int fp_precise_subprog_result(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = (s8)r1")
-/* here r1 is marked precise, even though it's fp register, but that's fine
- * because by the time we get out of subprogram it has to be derived from r10
- * anyways, at which point we'll break precision chain
- */
-__msg("mark_precise: frame1: regs=r1 stack= before 9: (bf) r1 = r10")
-__msg("7: R0=scalar")
+__msg("3: R0=scalar() R6=")
+__msg("6: (0f) r1 += r0                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int sneaky_fp_precise_subprog_result(void)
 {
 	asm volatile (
 		"call sneaky_fp_leaking_subprog;"
 		/* use subprog's returned value (which is derived from r10=fp
-		 * register), as index into vals array, forcing all of that to
-		 * be known precisely
+		 * register, so it's unknown), as index into vals array
 		 */
 		"r0 &= 3;"
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* force precision marking */
 		"r1 += r0;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -165,35 +134,25 @@ __naked int sneaky_fp_precise_subprog_result(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("9: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0")
-__msg("mark_precise: frame0: regs=r0 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 7: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (a5) if r0 < 0x4 goto pc+1")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (85) call pc+7")
+__msg("9: (0f) r1 += r0                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int global_subprog_result_precise(void)
 {
 	asm volatile (
 		"r6 = 3;"
 		/* pass r6 through r1 into subprog to get it back as r0;
-		 * given global_identity_subprog is global, precision won't
-		 * propagate all the way back to r6
+		 * given global_identity_subprog is global, nothing is known
+		 * about what it returns
 		 */
 		"r1 = r6;"
 		"call global_identity_subprog;"
 		/* now use subprog's returned value (which is unknown now, so
-		 * we need to clamp it), as index into vals array, forcing r0
-		 * to be marked precise (with no effect on r6, though)
+		 * we need to clamp it), as index into vals array
 		 */
 		"if r0 < %[vals_arr_sz] goto 1f;"
 		"r0 = %[vals_arr_sz] - 1;"
 	"1:"
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* here r0 is forced to be precise and has to be
-		 * propagated back to the global subprog call, but it
-		 * shouldn't go all the way to mark r6 as precise
-		 */
 		"r1 += r0;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -213,11 +172,9 @@ static unsigned long loop_callback_bad()
 		"if r0 s> 1000 goto 1f;"
 		"r0 = 0;"
 	"1:"
-		"goto +0;" /* checkpoint */
+		"goto +0;"
 		/* bpf_loop() expects [0, 1] values, so branch above skipping
-		 * r0 = 0; should lead to a failure, but if exit instruction
-		 * doesn't enforce r0's precision, this callback will be
-		 * successfully verified
+		 * r0 = 0; should lead to a failure
 		 */
 		"exit;"
 		:
@@ -228,14 +185,8 @@ static unsigned long loop_callback_bad()
 
 SEC("?raw_tp")
 __failure __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame1: regs=r0 stack= before 11: (b7) r0 = 0")
-/* check that we have branch code path doing its own validation */
-__msg("from 10 to 12: frame1: R0=scalar(smin=umin=1001")
-/* check that branch code path marks r0 as precise, before failing */
-__msg("mark_precise: frame1: regs=r0 stack= before 9: (85) call bpf_get_prandom_u32#7")
-__msg("At callback return the register R0 has smin=1001 should have been in [0, 1]")
+/* 0 of the fallthrough code path doesn't stand for what the branch code path has */
+__msg("At callback return the register R0 has {{.*}} should have been in [0, 1]")
 __naked int callback_precise_return_fail(void)
 {
 	asm volatile (
@@ -256,41 +207,15 @@ __naked int callback_precise_return_fail(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-/* First simulated path does not include callback body,
- * r1 and r4 are always precise for bpf_loop() calls.
- */
 __msg("9: (85) call bpf_loop#181")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: parent state regs=r4 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r4 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: parent state regs=r1 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r1 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs=r1 stack= before 7: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs=r1 stack= before 6: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs=r1 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* r6 precision propagation */
-__msg("14: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 14 first_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 12: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (25) if r6 > 0x3 goto pc+4")
-__msg("mark_precise: frame0: regs=r0,r6 stack= before 10: (bf) r6 = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 9: (85) call bpf_loop")
-/* State entering callback body popped from states stack */
-__msg("from 9 to 17: frame1:")
 __msg("17: frame1: R10=fp0 cb")
 __msg("17: (b7) r0 = 0")
 __msg("18: (95) exit")
 __msg("returning from callee:")
 __msg("to caller at 9:")
-__msg("frame 0: propagating r1,r3,r4")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit")
-__msg("from 18 to 9: safe")
+/* the callback changes nothing for the caller */
+__msg("from 18 to 9: within")
+__msg("14: (0f) r1 += r6                     ; R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R6=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int callback_result_precise(void)
 {
 	asm volatile (
@@ -309,9 +234,7 @@ __naked int callback_result_precise(void)
 		"if r6 > 3 goto 1f;"
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the bpf_loop() call, but not beyond
-		 */
+		/* here the range of r6 has to be known */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 	"1:"
@@ -326,15 +249,7 @@ __naked int callback_result_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("7: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 7 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 10: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs= stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_callee_saved_reg_precise(void)
 {
 	asm volatile (
@@ -348,9 +263,7 @@ __naked int parent_callee_saved_reg_precise(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -362,13 +275,7 @@ __naked int parent_callee_saved_reg_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("7: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 7 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_callee_saved_reg_precise_global(void)
 {
 	asm volatile (
@@ -382,9 +289,7 @@ __naked int parent_callee_saved_reg_precise_global(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -396,36 +301,14 @@ __naked int parent_callee_saved_reg_precise_global(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-/* First simulated path does not include callback body */
-__msg("12: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 12 first_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 10: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 9: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs=r6 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (b7) r1 = 1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* State entering callback body popped from states stack */
-__msg("from 9 to 15: frame1:")
+__msg("12: (0f) r1 += r6                     ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __msg("15: frame1: R10=fp0 cb")
 __msg("15: (b7) r0 = 0")
 __msg("16: (95) exit")
 __msg("returning from callee:")
 __msg("to caller at 9:")
-/* r1, r4 are always precise for bpf_loop(),
- * r6 was marked before backtracking to callback body.
- */
-__msg("frame 0: propagating r1,r3,r4,r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0")
-__msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs= stack=:")
-__msg("from 16 to 9: safe")
+/* r6 is what it was before the callback */
+__msg("from 16 to 9: within")
 __naked int parent_callee_saved_reg_precise_with_callback(void)
 {
 	asm volatile (
@@ -442,9 +325,7 @@ __naked int parent_callee_saved_reg_precise_with_callback(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) callback call
-		 */
+		/* here r6 has to be known: the callback doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -458,19 +339,7 @@ __naked int parent_callee_saved_reg_precise_with_callback(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("9: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 6")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8:")
-__msg("mark_precise: frame0: last_idx 13 first_idx 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 13: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 12: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs= stack= before 5: (85) call pc+6")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("9: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_stack_slot_precise(void)
 {
 	asm volatile (
@@ -492,9 +361,7 @@ __naked int parent_stack_slot_precise(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -506,15 +373,7 @@ __naked int parent_stack_slot_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("9: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (85) call pc+6")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("9: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_stack_slot_precise_global(void)
 {
 	asm volatile (
@@ -536,9 +395,7 @@ __naked int parent_stack_slot_precise_global(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -550,38 +407,14 @@ __naked int parent_stack_slot_precise_global(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-/* First simulated path does not include callback body */
-__msg("14: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 14 first_idx 10")
-__msg("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 12: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 10: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs= stack=-8:")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0 subseq_idx 10")
-__msg("mark_precise: frame0: regs= stack=-8 before 9: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 8: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* State entering callback body popped from states stack */
-__msg("from 10 to 17: frame1:")
+__msg("14: (0f) r1 += r6                     ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __msg("17: frame1: R10=fp0 cb")
 __msg("17: (b7) r0 = 0")
 __msg("18: (95) exit")
 __msg("returning from callee:")
 __msg("to caller at 10:")
-/* r1, r4 are always precise for bpf_loop(),
- * fp-8 was marked before backtracking to callback body.
- */
-__msg("frame 0: propagating r1,r3,r4,fp-8")
-__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0")
-__msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181")
-__msg("mark_precise: frame0: parent state regs= stack=:")
-__msg("from 18 to 10: safe")
+/* fp-8 is what it was before the callback */
+__msg("from 18 to 10: within")
 __naked int parent_stack_slot_precise_with_callback(void)
 {
 	asm volatile (
@@ -604,9 +437,7 @@ __naked int parent_stack_slot_precise_with_callback(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the callback doesn't change fp-8 */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -621,27 +452,18 @@ __naked int parent_stack_slot_precise_with_callback(void)
 __noinline __used
 static __u64 subprog_with_precise_arg(__u64 x)
 {
-	return vals[x]; /* x is forced to be precise */
+	return vals[x]; /* x has to be known */
 }
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("8: (0f) r2 += r1")
-__msg("mark_precise: frame1: last_idx 8 first_idx 0")
-__msg("mark_precise: frame1: regs=r1 stack= before 6: (18) r2 = ")
-__msg("mark_precise: frame1: regs=r1 stack= before 5: (67) r1 <<= 2")
-__msg("mark_precise: frame1: regs=r1 stack= before 2: (85) call pc+2")
-__msg("mark_precise: frame0: regs=r1 stack= before 1: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 0: (b7) r6 = 3")
+__msg("8: (0f) r2 += r1                      ; R1=12 R2=map_value(map=.data.vals,ks=4,vs=16,imm=12)")
 __naked int subprog_arg_precise(void)
 {
 	asm volatile (
 		"r6 = 3;"
 		"r1 = r6;"
-		/* subprog_with_precise_arg expects its argument to be
-		 * precise, so r1->r6 will be marked precise from inside the
-		 * subprog
-		 */
+		/* subprog_with_precise_arg needs to know its argument */
 		"call subprog_with_precise_arg;"
 		"r0 += r6;"
 		"exit;"
@@ -666,7 +488,7 @@ static __u64 subprog_spill_reg_precise(void)
 		 * verifier log output)
 		 */
 		"*(u64 *)(r10 - 16) = r2;"
-		/* use both spills as return result to propagete precision everywhere */
+		/* use both spills as return result */
 		"r0 = *(u64 *)(r10 - 16);"
 		"r2 = *(u64 *)(r1 + 0);"
 		"r0 += r2;"
@@ -676,32 +498,14 @@ static __u64 subprog_spill_reg_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("10: (0f) r1 += r7")
-__msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8")
-__msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4")
-__msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
-__msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2")
-__msg("mark_precise: frame1: regs= stack= before 16: (79) r2 = *(u64 *)(r1 +0)")
-__msg("mark_precise: frame1: regs= stack= before 15: (79) r0 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame1: regs= stack= before 14: (7b) *(u64 *)(r10 -16) = r2")
-__msg("mark_precise: frame1: regs= stack= before 13: (7b) *(u64 *)(r1 +0) = r2")
-__msg("mark_precise: frame1: regs=r2 stack= before 6: (85) call pc+6")
-__msg("mark_precise: frame0: regs=r2 stack= before 5: (bf) r2 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 1")
+__msg("10: (0f) r1 += r7                     ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=4) R7=4")
 __naked int subprog_spill_into_parent_stack_slot_precise(void)
 {
 	asm volatile (
 		"r6 = 1;"
 
 		/* pass pointer to stack slot and r6 to subprog;
-		 * r6 will be marked precise and spilled into fp-8 slot, which
-		 * also should be marked precise
+		 * r6 will be spilled into fp-8 slot
 		 */
 		"r1 = r10;"
 		"r1 += -8;"
@@ -716,8 +520,9 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
 
 		"r7 *= 4;"
 		"r1 = %[vals];"
-		/* here r7 is forced to be precise and has to be propagated
-		 * back to the beginning, handling subprog call and logic
+		/*
+		 * here r7 has to be known: it's what the subprog spilled into
+		 * the stack slot of its caller
 		 */
 		"r1 += r7;"
 		"r0 = *(u32 *)(r1 + 0);"
@@ -730,32 +535,12 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("17: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 17 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 16: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 15: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 14: (79) r0 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 13: (7b) *(u64 *)(r7 -8) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (79) r0 = *(u64 *)(r8 +16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (7b) *(u64 *)(r8 +16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 10: (79) r0 = *(u64 *)(r7 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 8: (07) r8 += -32")
-__msg("mark_precise: frame0: regs=r0 stack= before 7: (bf) r8 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 6: (07) r7 += -8")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r7 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 21: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 20: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (85) call pc+15")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 1")
+__msg("17: (0f) r1 += r0                     ; R0=4 R1=map_value(map=.data.vals,ks=4,vs=16,imm=4)")
 __naked int stack_slot_aliases_precision(void)
 {
 	asm volatile (
 		"r6 = 1;"
-		/* pass r6 through r1 into subprog to get it back as r0;
-		 * this whole chain will have to be marked as precise later
-		 */
+		/* pass r6 through r1 into subprog to get it back as r0 */
 		"r1 = r6;"
 		"call identity_subprog;"
 		/* let's setup two registers that are aliased to r10 */
@@ -777,11 +562,11 @@ __naked int stack_slot_aliases_precision(void)
 		"r0 = *(u64 *)(r8 + 16);"
 		"*(u64 *)(r7 - 8) = r0;"
 		"r0 = *(u64 *)(r10 - 16);"
-		/* get ready to use r0 as an index into array to force precision */
+		/* get ready to use r0 as an index into array */
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* here r0->r1->r6 chain is forced to be precise and has to be
-		 * propagated back to the beginning, including through the
+		/*
+		 * here r0 has to be known: the constant went through the
 		 * subprog call and all the stack spills and loads
 		 */
 		"r1 += r0;"
@@ -789,7 +574,7 @@ __naked int stack_slot_aliases_precision(void)
 		"exit;"
 		:
 		: __imm_ptr(vals)
-		: __clobber_common, "r6"
+		: __clobber_common, "r6", "r7", "r8"
 	);
 }
 
@@ -820,15 +605,11 @@ static unsigned long identity_tail_call(void)
 SEC("?raw_tp")
 __failure __log_level(2)
 __msg("13: (85) call bpf_tail_call#12")
-__msg("mark_precise: frame1: last_idx 13 first_idx 0 subseq_idx -1 ")
 __msg("returning from callee:")
 __msg("frame1: R0=scalar() R6=3 R10=fp0")
 __msg("to caller at 4:")
 __msg("R0=scalar() R6=map_value(map=.data.vals,ks=4,vs=16) R10=fp0")
 __msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: parent state regs=r0 stack=:  R0=Pscalar() R6=map_value(map=.data.vals,ks=4,vs=16) R10=fp0")
 __msg("math between map_value pointer and register with unbounded min value is not allowed")
 __naked int subprog_result_tail_call(void)
 {
@@ -860,36 +641,22 @@ static int ld_abs_subprog(void)
 }
 
 /*
- * Buggy verifier did not properly backtrack early subprogram exit
- * modelled for BPF_LD | BPF_ABS instruction, causing a segfault.
+ * BPF_LD | BPF_ABS instruction is modelled as an early exit of the subprogram.
+ * r7 is known on both paths.
  */
 SEC("socket")
 __success
 __log_level(2)
-/* early exit path */
-__msg("3: (0f) r1 += r7")
-__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r7 stack= before 9: (20) r0 = *(u32 *)skb[0]")
-__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1")
-__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1")
-__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8")
-/* fallthrough path */
-__msg("3: (0f) r1 += r7")
-__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r7 stack= before 10: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 9: (20) r0 = *(u32 *)skb[0]")
-__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1")
-__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1")
-__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8")
+/* the early exit and the exit of the subprogram give the same state */
+__msg("from 10 to 2: within")
+__msg("3: (0f) r1 += r7                      ; R1=fp-8 R7=-8")
 __naked int ld_abs_backtrack_both_paths(void)
 {
 	asm volatile (
 		"r7 = -8;"
 		"call ld_abs_subprog;"
 		"r1 = r10;"
-		"r1 += r7;" /* mark r7 as precise */
+		"r1 += r7;"
 		"*(u64 *)(r1 + 0) = 0;"
 		"r0 = 0;"
 		"exit;"
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_or_null.c b/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
index 8ff668a242eb..2a5bdac24925 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
@@ -183,7 +183,7 @@ l1_%=:	exit;						\
 SEC("socket")
 __description("invalid map access from else condition")
 __failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void map_access_from_else_condition(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
index b3b701b44550..f9ad07704578 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
@@ -225,8 +225,7 @@ l2_%=:	r0 = 1;						\
 
 SEC("socket")
 __description("map access: known scalar += value_ptr unknown vs unknown (lt)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
 __retval(1)
 __naked void ptr_unknown_vs_unknown_lt(void)
 {
@@ -270,8 +269,7 @@ l2_%=:	r0 = 1;						\
 
 SEC("socket")
 __description("map access: known scalar += value_ptr unknown vs unknown (gt)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
 __retval(1)
 __naked void ptr_unknown_vs_unknown_gt(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index 399884911ea5..d92215c5b923 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -63,7 +63,6 @@ SEC("cgroup/skb")
 __description("variable-offset stack read preserves spilled zero")
 __success
 __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8")
 __msg("R3=0")
 __retval(0)
 __naked void stack_read_var_off_preserves_spilled_zero(void)
@@ -89,7 +88,6 @@ SEC("cgroup/skb")
 __description("variable-offset stack read preserves spilled zero across slots")
 __success
 __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8,-16")
 __msg("R3=0")
 __retval(0)
 __naked void stack_read_var_off_preserves_spilled_zero_across_slots(void)
@@ -116,7 +114,6 @@ SEC("cgroup/skb")
 __description("variable-offset stack read preserves partial spilled zero")
 __success
 __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8")
 __msg("R3=0")
 __retval(0)
 __naked void stack_read_var_off_preserves_partial_spilled_zero(void)
diff --git a/tools/testing/selftests/bpf/verifier/precise.c b/tools/testing/selftests/bpf/verifier/precise.c
index a9242103dc47..de9d35ed8a4b 100644
--- a/tools/testing/selftests/bpf/verifier/precise.c
+++ b/tools/testing/selftests/bpf/verifier/precise.c
@@ -36,26 +36,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
 	.fixup_map_array_48b = { 1 },
-	.result = VERBOSE_ACCEPT,
-	.errstr =
-	"mark_precise: frame0: last_idx 26 first_idx 20\
-	mark_precise: frame0: regs=r2 stack= before 25\
-	mark_precise: frame0: regs=r2 stack= before 24\
-	mark_precise: frame0: regs=r2 stack= before 23\
-	mark_precise: frame0: regs=r2 stack= before 22\
-	mark_precise: frame0: regs=r2 stack= before 20\
-	mark_precise: frame0: parent state regs=r2 stack=:\
-	mark_precise: frame0: last_idx 19 first_idx 10\
-	mark_precise: frame0: regs=r2 stack= before 19\
-	mark_precise: frame0: regs=r9 stack= before 18\
-	mark_precise: frame0: regs=r8,r9 stack= before 17\
-	mark_precise: frame0: regs=r0,r9 stack= before 15\
-	mark_precise: frame0: regs=r0,r9 stack= before 14\
-	mark_precise: frame0: regs=r9 stack= before 13\
-	mark_precise: frame0: regs=r9 stack= before 12\
-	mark_precise: frame0: regs=r9 stack= before 11\
-	mark_precise: frame0: regs=r9 stack= before 10\
-	mark_precise: frame0: parent state regs= stack=:",
+	.result = ACCEPT,
 },
 {
 	"precise: test 2",
@@ -95,24 +76,8 @@
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
 	.fixup_map_array_48b = { 1 },
-	.result = VERBOSE_ACCEPT,
+	.result = ACCEPT,
 	.flags = BPF_F_TEST_STATE_FREQ,
-	.errstr =
-	"26: (85) call bpf_probe_read_kernel#113\
-	mark_precise: frame0: last_idx 26 first_idx 22\
-	mark_precise: frame0: regs=r2 stack= before 25\
-	mark_precise: frame0: regs=r2 stack= before 24\
-	mark_precise: frame0: regs=r2 stack= before 23\
-	mark_precise: frame0: regs=r2 stack= before 22\
-	mark_precise: frame0: parent state regs=r2 stack=:\
-	mark_precise: frame0: last_idx 20 first_idx 20\
-	mark_precise: frame0: regs=r2 stack= before 20\
-	mark_precise: frame0: parent state regs=r2 stack=:\
-	mark_precise: frame0: last_idx 19 first_idx 17\
-	mark_precise: frame0: regs=r2 stack= before 19\
-	mark_precise: frame0: regs=r9 stack= before 18\
-	mark_precise: frame0: regs=r8,r9 stack= before 17\
-	mark_precise: frame0: parent state regs= stack=:",
 },
 {
 	"precise: cross frame pruning",
@@ -153,17 +118,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_XDP,
 	.flags = BPF_F_TEST_STATE_FREQ,
-	.errstr = "mark_precise: frame0: last_idx 5 first_idx 5\
-	mark_precise: frame0: parent state regs=r4 stack=:\
-	mark_precise: frame0: last_idx 4 first_idx 2\
-	mark_precise: frame0: regs=r4 stack= before 4\
-	mark_precise: frame0: regs=r4 stack= before 3\
-	mark_precise: frame0: last_idx 5 first_idx 5\
-	mark_precise: frame0: parent state regs=r0 stack=:\
-	mark_precise: frame0: last_idx 4 first_idx 2\
-	mark_precise: frame0: regs=r0 stack= before 4\
-	5: R0=-1 R4=0",
-	.result = VERBOSE_ACCEPT,
+	.result = ACCEPT,
 	.retval = -1,
 },
 {
@@ -182,23 +137,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_XDP,
 	.flags = BPF_F_TEST_STATE_FREQ,
-	.errstr = "mark_precise: frame0: last_idx 7 first_idx 7\
-	mark_precise: frame0: parent state regs=r4 stack=:\
-	mark_precise: frame0: last_idx 6 first_idx 4\
-	mark_precise: frame0: regs=r4 stack= before 6: (b7) r0 = -1\
-	mark_precise: frame0: regs=r4 stack= before 5: (79) r4 = *(u64 *)(r10 -8)\
-	mark_precise: frame0: regs= stack=-8 before 4: (7b) *(u64 *)(r3 -8) = r0\
-	mark_precise: frame0: parent state regs=r0 stack=:\
-	mark_precise: frame0: last_idx 3 first_idx 3\
-	mark_precise: frame0: regs=r0 stack= before 3: (55) if r3 != 0x7b goto pc+0\
-	mark_precise: frame0: regs=r0 stack= before 2: (bf) r3 = r10\
-	mark_precise: frame0: regs=r0 stack= before 1: (57) r0 &= 255\
-	mark_precise: frame0: parent state regs=r0 stack=:\
-	mark_precise: frame0: last_idx 0 first_idx 0\
-	mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7\
-	mark_precise: frame0: last_idx 7 first_idx 7\
-	mark_precise: frame0: parent state regs= stack=:",
-	.result = VERBOSE_ACCEPT,
+	.result = ACCEPT,
 	.retval = -1,
 },
 {
-- 
2.55.0

Re: [PATCH bpf-next v3 39/45] selftests/bpf: Adjust tests to log lines that are gone

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 6920 bytes --]

> selftests/bpf: Adjust tests to log lines that are gone

TL;DR:
- Commit message says progs are kept, but a spill_fill test now spills an
  unknown scalar, possibly changing pruning
- Message says BPF_F_TEST_STATE_FREQ is removed, but three precise.c tests
  keep it
- Stale test comments (nits)

> selftests/bpf: Adjust tests to log lines that are gone
>
> Without this patch 114 subtests of test_progs and 4 tests of test_verifier
> fail. Their progs get the old verdict, but the tests match lines of the
> verifier log that the kernel does not print any more. Keep the progs. Match
> the line of the new log that shows the same fact.
>
> [ ... ]
>
> The first row is verifier_precision.c, _subprog_precision.c, _spill_fill.c,
> _scalar_ids.c, stack_arg_precision.c and verifier/precise.c. The second row
> is verifier_bounds_step.c, _search_pruning.c, _live_stack.c: the tests of
> regsafe(), stacksafe() and range_within(). The target of a jump is verified
> before the fall-through. Thus the tests in which the first of two states
> must cover the second one swap their branches.
>
> Three verdicts change. short_loop1 was "too large". Now one state at the
> loop head covers all iterations, and it loads. 2 unprivileged progs of
> verifier_value_ptr_arith.c load: r1 = [0, 3] and r1 = [0, 7] are joined in
> front of 'r1 += r0', and the insn gets one limit for its mask. 8 progs are
> rejected as before, with a different message. BPF_F_TEST_STATE_FREQ has no
> effect. Remove it from the tests that change.
>
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>

The message says "Keep the progs" and that the only prog changes are the
branch swaps.  In verifier_spill_fill.c,
old_imprecise_scalar32_vs_cur_stack_misc() also changes what it spills:

-	/* conjure 32-bit scalar spill at fp-8 */
-	"r0 = 42;"
-	"*(u32*)(r10 - 8) = r0;"
 [ ... ]
+	/* conjure 32-bit scalar spill at fp-8 */
+	"call %[bpf_ktime_get_ns];"
+	"*(u32*)(r10 - 8) = r0;"

It used to spill the constant 42 and now spills an unknown scalar.  Without
precision tracking, regsafe() compares the fake register built from the slot
by range, and stacksafe() goes through scalar_reg_for_stack(), so a spilled
known constant no longer covers STACK_MISC.  Is that pruning change
intended?  Also, old_precise_scalar_vs_cur_stack_misc() gets an extra
"r0 = 0;" before the exit.

Could the commit message mention these changes instead of describing the
progs as kept?

The message also says "BPF_F_TEST_STATE_FREQ has no effect. Remove it from
the tests that change."  In verifier/precise.c three changed tests still
carry the flag:

> diff --git a/tools/testing/selftests/bpf/verifier/precise.c b/tools/testing/selftests/bpf/verifier/precise.c
> --- a/tools/testing/selftests/bpf/verifier/precise.c
> +++ b/tools/testing/selftests/bpf/verifier/precise.c

[ ... ]

> @@ -95,24 +76,8 @@
>  	},
>  	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
>  	.fixup_map_array_48b = { 1 },
> -	.result = VERBOSE_ACCEPT,
> +	.result = ACCEPT,
>  	.flags = BPF_F_TEST_STATE_FREQ,

The same applies to "precise: ST zero to stack insn is supported" and
"precise: STX insn causing spi > allocated_stack", which change
VERBOSE_ACCEPT to ACCEPT but keep .flags = BPF_F_TEST_STATE_FREQ.  Nothing
breaks, since the flag is still accepted.  Should .flags be dropped from
these three tests, or should the sentence in the commit message be limited
to the __flag() annotations used by test_loader?

[ ... ]

> diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> --- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> @@ -518,15 +518,6 @@ __msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0")
>   */
>  __msg("10: (71) r2 = *(u8 *)(r10 -9)         ; R2=0 R10=fp0 fp-16=0")
>  __msg("11: (0f) r1 += r2")
> -__msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
> -__msg("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)")
 [ ... ]
>  __naked void partial_stack_load_preserves_zeros(void)

The nine mark_precise checks of partial_stack_load_preserves_zeros() are
removed, but the comments above the remaining __msg() lines are unchanged:

	/* fp-8 is spilled IMPRECISE value zero (represented by a zero value fake reg) */
	/* validate that assigning R2 from STACK_SPILL with zero value  doesn't mark register
	 * precise immediately; if necessary, it will be marked precise later
	 */
	/* similarly, when R2 is assigned from spilled register, it is initially
	 * imprecise, but will be marked precise later once it is used in precise context
	 */

Do these comments still make sense?  They describe IMPRECISE spills and
"marked precise later", and the checks for that are gone.  The matching
comments in stack_load_preserves_const_precision() were updated ("make sure
fp-8 is IMPRECISE fake register spill" became "make sure fp-8 is fake
register spill"), so this test looks like it was missed.

[ ... ]

> diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
> --- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c

[ ... ]

> @@ -830,24 +709,22 @@ __naked void two_old_ids_one_cur_id(void)
>  	"r9 = r0;"
>  	"r0 = 0;"
>  	/* Maybe make r{6,7} IDs identical */
> -	"if r6 > r7 goto l0_%=;"
> -	"goto l1_%=;"
> -"l0_%=:"
> +	"if r6 <= r7 goto l1_%=;"
>  	"r6 = r7;"
>  "l1_%=:"
[ ... ]
>  	"r2 = r10;"
>  	"r2 += r6;"
>  	"r2 += r7;"
>  	/*
>  	 * keep r8 and r9 live, otherwise r6->id and r7->id
> -	 * will become singular and reset to zero before if r6 > r7
> +	 * will become singular and reset to zero before if r6 <= r7
>  	 */
>  	"r9 += r8;"
>  	"exit;"

Is "before if r6 <= r7" still the right place in this comment?  The old
text was correct when BPF_F_TEST_STATE_FREQ put a checkpoint at the jump and
bpf_split_cur_state() called bpf_clear_singular_ids() on the current state
before the jump ran.

With the flag gone, the only caller of bpf_clear_singular_ids() is
flow_arrive(), reached through bpf_flow_deliver() when a state arrives at
the start of a basic block:

  bpf_flow_deliver()
    flow_arrive()
      bpf_clear_singular_ids()

In this prog the first such points after the jump are 'r6 = r7' and l1, so
the ids would be reset after the jump, not before 'if r6 <= r7'.  The
reasoning in the comment is still right, only the location is stale.
Something like "... will become singular and be reset to zero when the
states arrive after the jump" would match the new behavior.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

[PATCH bpf-next v3 40/45] selftests/bpf: Adjust tests to new verdicts, messages and order

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Without this patch 143 subtests of test_progs and 3 tests of test_verifier
fail. No prog that loaded is rejected. 25 progs that were rejected load.

  [F] "bpf: Verify programs as a fixpoint over the control flow graph"
  [O] "bpf: Verify static subprogs on their own"
  [R] "bpf: Allow recursive subprogs"

  before                          now
  ------------------------------  -----------------------------------------
  "infinite loop detected",       [F] 19 progs load, most run and return 0:
  "too large", "8193 jumps"       8 iter_remake_*, loop3.bpf.o, .. One
                                  state covers all iterations, and may_goto
                                  stops the loop.
  the same, the loop has no exit  [F] "loop at insn N never exits": 6 progs
  "recursive call from",          [R] 6 progs load: callx_*_recursion,
  "frames is too deep"            bounded_recursion, .. It needs the JIT:
                                  __arch_x86_64, __arch_arm64.
  "header_count is [0..3]", then  [F] "[0..5]": two bounds are joined in
  "[0..5]"                        front of a counted loop.
  the fall-through is verified    flow_compute_order(): the target of
  first                           a jump is. Swap the branches, or expect
                                  the other one of two errors.
  "0: R1=ctx() R10=fp0"           [F] "0: R10=fp0": only what is live
  scc column, "depth=",           gone, "(limit 0)": the patches that
  "(limit 1000000)"               remove the old pass
  r1 is an argument               [O] only if the callee reads it

The last row: the callees in verifier_live_stack.c and _liveness_exp.c read
the pointer that they get. unsafe_on_2nd_iter also wrote 8 bytes to
'char buf[4]', and the verifier finds that error first: the buffer gets
8 bytes. bpf_loop_iter_limit_nested compares two sums with their maximum:
the verifier knows their range, not the set of their values.

New tests of [R] stand next to the old tests of the same limits. What it
rejects: test_global_func3.c, verifier_tailcall.c,
"calls: a timer callback calls the main prog", "arena stack: the kernel
frame of a recursion counts in the depth of the stack". What runs:
"arena stack: recursion ends in the stack", and
callx_no_recursion_after_all, which gets no check at run time.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../bpf/prog_tests/bpf_verif_scale.c          |   4 +-
 .../selftests/bpf/prog_tests/log_buf.c        |   4 +-
 .../selftests/bpf/prog_tests/log_fixup.c      |   6 +-
 .../selftests/bpf/prog_tests/reg_bounds.c     |   5 +-
 ...f_qdisc_fail__invalid_dynptr_cross_frame.c |   4 +-
 .../selftests/bpf/progs/cpumask_failure.c     |   2 +-
 tools/testing/selftests/bpf/progs/iters.c     | 114 ++++++--------
 .../selftests/bpf/progs/iters_state_safety.c  |   8 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |  12 +-
 .../bpf/progs/mem_rdonly_untrusted.c          |  16 +-
 .../selftests/bpf/progs/test_global_func3.c   | 125 ++++++++++++++++
 .../selftests/bpf/progs/timer_failure.c       |   9 +-
 .../selftests/bpf/progs/verifier_align.c      |  10 +-
 .../selftests/bpf/progs/verifier_and.c        |   4 +-
 .../bpf/progs/verifier_arena_stack.c          |  82 +++++++++++
 .../selftests/bpf/progs/verifier_bounds.c     |   6 +-
 .../selftests/bpf/progs/verifier_callx.c      |  71 +++++++--
 .../bpf/progs/verifier_callx_rodata.c         |   4 +-
 .../selftests/bpf/progs/verifier_cfg.c        |   8 +-
 .../bpf/progs/verifier_global_subprogs.c      |  11 +-
 .../bpf/progs/verifier_helper_value_access.c  |   8 +-
 .../selftests/bpf/progs/verifier_idoms.c      |   9 +-
 .../bpf/progs/verifier_iterating_callbacks.c  |  17 ++-
 .../selftests/bpf/progs/verifier_live_stack.c |   3 +-
 .../bpf/progs/verifier_liveness_exp.c         |   7 +-
 .../bpf/progs/verifier_loop_hierarchy.c       | 125 ++++++++--------
 .../selftests/bpf/progs/verifier_loop_latch.c |   2 +-
 .../selftests/bpf/progs/verifier_loops1.c     |  16 +-
 .../selftests/bpf/progs/verifier_movsx.c      |   3 +-
 .../selftests/bpf/progs/verifier_scev.c       | 139 ++++++++----------
 .../bpf/progs/verifier_subprog_insn_stats.c   |  16 +-
 .../selftests/bpf/progs/verifier_tailcall.c   |  27 ++++
 .../selftests/bpf/progs/verifier_uninit.c     |   2 +-
 .../selftests/bpf/progs/verifier_unpriv.c     |  23 +--
 .../bpf/progs/verifier_unpriv_perf.c          |   4 +-
 tools/testing/selftests/bpf/verifier/calls.c  |  46 +++++-
 36 files changed, 615 insertions(+), 337 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
index 902a70d5afeb..2dc37090d7a4 100644
--- a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
@@ -154,9 +154,9 @@ void test_verif_scale_loop2()
 	scale_test("loop2.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
 }
 
-void test_verif_scale_loop3_fail()
+void test_verif_scale_loop3()
 {
-	scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */);
+	scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
 }
 
 void test_verif_scale_loop4()
diff --git a/tools/testing/selftests/bpf/prog_tests/log_buf.c b/tools/testing/selftests/bpf/prog_tests/log_buf.c
index d6f14a232002..4f821fdc4a02 100644
--- a/tools/testing/selftests/bpf/prog_tests/log_buf.c
+++ b/tools/testing/selftests/bpf/prog_tests/log_buf.c
@@ -125,7 +125,7 @@ static void obj_load_log_buf(void)
 	ASSERT_OK_PTR(strstr(libbpf_log_buf, "libbpf: prog 'bad_prog': -- BEGIN PROG LOAD LOG --"),
 		      "libbpf_log_correct");
 	ASSERT_STREQ(obj_log_buf, "", "obj_log__empty");
-	ASSERT_STREQ(good_log_buf, "processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n",
+	ASSERT_STREQ(good_log_buf, "processed 4 insns (limit 0) max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n",
 		     "good_log_ok");
 	ASSERT_STREQ(bad_log_buf, "", "bad_log_empty");
 
@@ -179,7 +179,7 @@ static void bpf_prog_load_log_buf(void)
 	opts.log_level = 2;
 	fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "good_prog", "GPL",
 			   good_prog_insns, good_prog_insn_cnt, &opts);
-	ASSERT_OK_PTR(strstr(log_buf, "0: R1=ctx() R10=fp0"), "good_log_2");
+	ASSERT_OK_PTR(strstr(log_buf, "0: R10=fp0"), "good_log_2");
 	ASSERT_GE(fd, 0, "good_fd2");
 	if (fd >= 0)
 		close(fd);
diff --git a/tools/testing/selftests/bpf/prog_tests/log_fixup.c b/tools/testing/selftests/bpf/prog_tests/log_fixup.c
index 90a98e23be61..89a7f5947081 100644
--- a/tools/testing/selftests/bpf/prog_tests/log_fixup.c
+++ b/tools/testing/selftests/bpf/prog_tests/log_fixup.c
@@ -41,7 +41,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
 				  "struct task_struct___bad.fake_field (0:1 @ offset 4)\n",
 				  "log_buf_part2");
 		ASSERT_HAS_SUBSTR(log_buf,
-				  "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n",
+				  "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n",
 				  "log_buf_end");
 		break;
 	case TRUNC_PARTIAL:
@@ -50,7 +50,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
 				  "struct task_struct___bad.fake_field (0:1 @ offset 4)\n",
 				  "log_buf_part2");
 		/* we shouldn't get full end of BPF verifier log */
-		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n"),
+		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n"),
 			    "log_buf_end");
 		break;
 	case TRUNC_FULL:
@@ -58,7 +58,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
 		ASSERT_NULL(strstr(log_buf, "struct task_struct___bad.fake_field (0:1 @ offset 4)\n"),
 			    "log_buf_part2");
 		/* we shouldn't get full end of BPF verifier log */
-		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n"),
+		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n"),
 			    "log_buf_end");
 		break;
 	}
diff --git a/tools/testing/selftests/bpf/prog_tests/reg_bounds.c b/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
index 6231258b26cc..09f7930733be 100644
--- a/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
+++ b/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
@@ -1276,7 +1276,7 @@ static int parse_range_cmp_log(const char *log_buf, struct case_spec spec,
 		{true_pos + 1,  7, "R7=", true2_reg},
 	};
 	char buf[32];
-	const char *p = log_buf, *q;
+	const char *p, *q;
 	int i, err;
 
 	for (i = 0; i < 4; i++) {
@@ -1291,8 +1291,9 @@ static int parse_range_cmp_log(const char *log_buf, struct case_spec spec,
 		 * Different verifier passes may print
 		 *   18: (bf) r0 = r6
 		 * as well, but never followed by ';'.
+		 * The branches are in the log in the order the verifier looked at them.
 		 */
-		q = p;
+		q = log_buf;
 		while ((q = strstr(q, buf)) != NULL) {
 			const char *s = q + strlen(buf);
 
diff --git a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
index 2e23b8593af9..1d1684d2cf3b 100644
--- a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
+++ b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
@@ -26,8 +26,10 @@ int BPF_PROG(invalid_dynptr_cross_frame, struct sk_buff *skb, struct Qdisc *sch,
 	bpf_dynptr_from_skb((struct __sk_buff *)skb, 0, &ptr);
 
 	hdr = bpf_dynptr_slice(&ptr, 0, NULL, sizeof(*hdr));
-	if (!hdr)
+	if (!hdr) {
+		bpf_kfree_skb(skb);
 		return NET_XMIT_DROP;
+	}
 
 	free_skb(skb);
 
diff --git a/tools/testing/selftests/bpf/progs/cpumask_failure.c b/tools/testing/selftests/bpf/progs/cpumask_failure.c
index c89c88db39d1..715218d32105 100644
--- a/tools/testing/selftests/bpf/progs/cpumask_failure.c
+++ b/tools/testing/selftests/bpf/progs/cpumask_failure.c
@@ -45,7 +45,7 @@ int BPF_PROG(test_alloc_no_release, struct task_struct *task, u64 clone_flags)
 }
 
 SEC("tp_btf/task_newtask")
-__failure __msg("NULL pointer passed to trusted R1")
+__failure __msg("R1 type=scalar expected=ptr_, trusted_ptr_, rcu_ptr_")
 int BPF_PROG(test_alloc_double_release, struct task_struct *task, u64 clone_flags)
 {
 	struct bpf_cpumask *cpumask;
diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c
index ae845ad747bf..b9677d30e692 100644
--- a/tools/testing/selftests/bpf/progs/iters.c
+++ b/tools/testing/selftests/bpf/progs/iters.c
@@ -47,7 +47,7 @@ int iter_err_unsafe_c_loop(const void *ctx)
 }
 
 SEC("?raw_tp")
-__failure __msg("unbounded memory access")
+__failure __msg("math between map_value pointer and register with unbounded min value is not allowed")
 int iter_err_unsafe_asm_loop(const void *ctx)
 {
 	struct bpf_iter_num it;
@@ -878,19 +878,15 @@ __naked int delayed_read_mark(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
+__msg("21: (79) r8 = *(u64 *)(r0 +0)")
+__msg("misaligned stack access off -33+0 size 8")
 __naked int delayed_precision_mark(void)
 {
 	/* This is equivalent to C program below.
-	 * The test is similar to delayed_iter_mark but verifies that incomplete
-	 * precision don't fool verifier.
-	 * The call to bpf_iter_num_next() is reachable with r7 values -16 and -32.
-	 * State with r7=-16 is visited first and follows r6 != 42 ... continue branch.
-	 * At this point iterator next() call is reached with r7 that has no read
-	 * and precision marks.
-	 * Loop body with r7=-32 would only be visited if verifier would decide to continue
-	 * with second loop iteration. Absence of precision mark on r7 might affect state
-	 * equivalent logic used for iterator convergence tracking.
+	 * The test is similar to delayed_read_mark.
+	 * The call to bpf_iter_num_next() is reachable with r7 values -16 and -33.
+	 * The loop body with r7=-33 is only looked at if the verifier goes on
+	 * with the second iteration of the loop.
 	 *
 	 * r8 = 0
 	 * fp[-16] = 0
@@ -899,7 +895,7 @@ __naked int delayed_precision_mark(void)
 	 * bpf_iter_num_new(&fp[-8], 0, 10)
 	 * while (bpf_iter_num_next(&fp[-8])) {
 	 *   if (r6 != 42) {
-	 *     r7 = -32
+	 *     r7 = -33
 	 *     r6 = bpf_get_prandom_u32()
 	 *     continue;
 	 *   }
@@ -957,22 +953,17 @@ __naked int delayed_precision_mark(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("misaligned stack access off -25+0 size 8")
 __naked int loop_state_deps1(void)
 {
 	/* This is equivalent to C program below.
 	 *
 	 * The case turns out to be tricky in a sense that:
-	 * - states with c=-25 are explored only on a second iteration
-	 *   of the outer loop;
-	 * - states with read+precise mark on c are explored only on
-	 *   second iteration of the inner loop and in a state which
-	 *   is pushed to states stack first.
+	 * - c is -25 only on a second iteration of the outer loop;
+	 * - c is used only on a second iteration of the inner loop.
 	 *
-	 * Depending on the details of iterator convergence logic
-	 * verifier might stop states traversal too early and miss
-	 * unsafe c=-25 memory access.
+	 * The verifier might stop too early and miss unsafe c=-25
+	 * memory access.
 	 *
 	 *   j = iter_new();		 // fp[-16]
 	 *   a = 0;			 // r6
@@ -1082,23 +1073,18 @@ __naked int loop_state_deps1(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("34: (0f) r0 += r8")
+__msg("misaligned stack access off -25+0 size 8")
 __naked int loop_state_deps2(void)
 {
 	/* This is equivalent to C program below.
 	 *
 	 * The case turns out to be tricky in a sense that:
-	 * - states with read+precise mark on c are explored only on a second
-	 *   iteration of the first inner loop and in a state which is pushed to
-	 *   states stack first.
-	 * - states with c=-25 are explored only on a second iteration of the
-	 *   second inner loop and in a state which is pushed to states stack
-	 *   first.
+	 * - c is used only on a second iteration of the first inner loop;
+	 * - c is -25 only on a second iteration of the second inner loop.
 	 *
-	 * Depending on the details of iterator convergence logic
-	 * verifier might stop states traversal too early and miss
-	 * unsafe c=-25 memory access.
+	 * The verifier might stop too early and miss unsafe c=-25
+	 * memory access.
 	 *
 	 *   j = iter_new();             // fp[-16]
 	 *   a = 0;                      // r6
@@ -1259,49 +1245,38 @@ __naked int loop_state_deps2(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("misaligned stack access off -25+0 size 8")
 __naked int loop_state_deps3(void)
 {
 	/* This is equivalent to a C program below.
 	 *
-	 *   if (random() != 24) {       // assume false branch is placed first
+	 *   if (random() != 24) {
 	 *     i = iter_new();           // fp[-8]
 	 *     while (iter_next(i));
 	 *     iter_destroy(i);
 	 *     return;
 	 *   }
 	 *
-	 *   for (i = 10; i > 0; i--);   // increase dfs_depth for child states
+	 *   for (i = 10; i > 0; i--);
 	 *
 	 *   i = iter_new();             // fp[-8]
 	 *   b = -24;                    // r8
-	 *   for (;;) {                  // checkpoint (L)
-	 *     if (iter_next(i))         // checkpoint (N)
+	 *   for (;;) {
+	 *     if (iter_next(i))
 	 *       break;
-	 *     if (random() == 77) {     // assume false branch is placed first
+	 *     if (random() == 77) {
 	 *       *(u64 *)(r10 + b) = 7;  // this is not safe when b == -25
 	 *       iter_destroy(i);
 	 *       return;
 	 *     }
-	 *     if (random() == 42) {     // assume false branch is placed first
+	 *     if (random() == 42) {
 	 *       b = -25;
 	 *     }
 	 *   }
 	 *   iter_destroy(i);
 	 *
-	 * In case of a buggy verifier first loop might poison
-	 * env->cur_state->loop_entry with a state having 0 branches
-	 * and small dfs_depth. This would trigger NOT_EXACT states
-	 * comparison for some states within second loop.
-	 * Specifically, checkpoint (L) might be problematic if:
-	 * - branch with '*(u64 *)(r10 + b) = 7' is not explored yet;
-	 * - checkpoint (L) is first reached in state {b=-24};
-	 * - traversal is pruned at checkpoint (N) setting checkpoint's (L)
-	 *   branch count to 0, thus making it eligible for use in pruning;
-	 * - checkpoint (L) is next reached in state {b=-25},
-	 *   this would cause NOT_EXACT comparison with a state {b=-24}
-	 *   while 'b' is not marked precise yet.
+	 * What the verifier learned in the first two loops must not make it
+	 * take {b=-25} at the head of the third one for {b=-24}.
 	 */
 	asm volatile (
 		"call %[bpf_get_prandom_u32];"
@@ -1884,7 +1859,6 @@ __naked int absent_mark_in_the_middle_state2(void)
 }
 
 SEC("?raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("misaligned stack access off -31+0 size 8")
 __naked int absent_mark_in_the_middle_state3(void)
 {
@@ -2014,8 +1988,8 @@ static int loop1_wrapper(void)
  * but adapted for use with bpf_loop().
  */
 SEC("raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
-__failure __msg("math between fp pointer and register with unbounded min value is not allowed")
+/* -32 or -31 */
+__failure __msg("misaligned stack access off (0xffffffffffffffe0; 0x1)+0 size 8")
 __naked void absent_mark_in_the_middle_state4(void)
 {
 	/*
@@ -2159,8 +2133,7 @@ static int loop_cb5(int i, __u64 *ctx)
 }
 
 SEC("raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
-__failure __msg("memory access is {{.*}} and is outside of the object of size 64")
+__failure __msg("invalid access to map value, value_size=64 off=402000 size=4")
 __naked void loop_counter_precision_2nd_iter(void)
 {
 	asm volatile (
@@ -2174,9 +2147,8 @@ __naked void loop_counter_precision_2nd_iter(void)
 		"r3 += -8;"
 		"r4 = 0;"
 		/*
-		 * Explore with nr_loops=1 on a first path and nr_loops=2 on a second path.
-		 * Buggy verifier did not propagate r1 precision properly,
-		 * and thus checkpoints created for nr_loops=1 case matched nr_loops=2 case.
+		 * nr_loops is 1 on one path and 2 on the other.
+		 * The second call of the callback must not be lost.
 		 */
 		"call %[bpf_loop];"
 		"r0 = 0;"
@@ -2190,12 +2162,12 @@ __naked void loop_counter_precision_2nd_iter(void)
 
 /*
  * An iterator that is destroyed and created again inside its loop starts from
- * the beginning and the loop never ends. In the first four progs the first
- * prune point after bpf_iter_num_new() is the call of bpf_iter_num_next().
+ * the beginning and the loop never ends. bpf_iter_num_next() is not what stops such
+ * loop: the verifier adds may_goto. The progs spin until it's out of budget.
  */
 SEC("socket")
 __description("iter: remake in the loop, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_jmp(void)
 {
 	asm volatile (
@@ -2256,7 +2228,7 @@ __used __naked static void iter_remake(void)
 
 SEC("socket")
 __description("iter: remake in a callee, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_callee_jmp(void)
 {
 	asm volatile (
@@ -2295,7 +2267,7 @@ __naked void iter_remake_callee_jmp(void)
 
 SEC("socket")
 __description("iter: remake after next, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_after_next_jmp(void)
 {
 	asm volatile (
@@ -2336,7 +2308,7 @@ __naked void iter_remake_after_next_jmp(void)
 
 SEC("socket")
 __description("iter: two iterators, remake each after its next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_two_jmp(void)
 {
 	asm volatile (
@@ -2398,7 +2370,7 @@ __naked void iter_remake_two_jmp(void)
 /* Same loops with a prune point in front of bpf_iter_num_next() */
 SEC("socket")
 __description("iter: remake in the loop")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_inline(void)
 {
@@ -2439,7 +2411,7 @@ __naked void iter_remake_inline(void)
 
 SEC("socket")
 __description("iter: remake in a callee")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_callee(void)
 {
@@ -2475,7 +2447,7 @@ __naked void iter_remake_callee(void)
 
 SEC("socket")
 __description("iter: remake after next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_after_next(void)
 {
@@ -2515,7 +2487,7 @@ __naked void iter_remake_after_next(void)
 
 SEC("socket")
 __description("iter: remake when drained")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_drained(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/iters_state_safety.c b/tools/testing/selftests/bpf/progs/iters_state_safety.c
index b2ba4609a31e..fa9604f0a173 100644
--- a/tools/testing/selftests/bpf/progs/iters_state_safety.c
+++ b/tools/testing/selftests/bpf/progs/iters_state_safety.c
@@ -30,7 +30,7 @@ int force_clang_to_emit_btf_for_externs(void *ctx)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("fp-8=iter_num(id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(id=1,state=active)")
 int create_and_destroy(void *ctx)
 {
 	struct bpf_iter_num iter;
@@ -207,8 +207,8 @@ static __noinline void subprog_with_iter(void)
 SEC("?raw_tp")
 __failure
 /* ensure there was a call to subprog, which might happen without __noinline */
-__msg("returning from callee:")
-__msg("Unreleased reference id=1")
+__msg("0: (85) call pc+2")
+__msg("Unreleased reference id=2 alloc_insn=7")
 int leak_iter_from_subprog_fail(void *ctx)
 {
 	subprog_with_iter();
@@ -218,7 +218,7 @@ int leak_iter_from_subprog_fail(void *ctx)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("fp-8=iter_num(id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(id=1,state=active)")
 int valid_stack_reuse(void *ctx)
 {
 	struct bpf_iter_num iter;
diff --git a/tools/testing/selftests/bpf/progs/iters_testmod_seq.c b/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
index d00888f6687a..88ce4b7a9c33 100644
--- a/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
+++ b/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
@@ -20,8 +20,8 @@ __s64 res_empty;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_empty(const void *ctx)
 {
@@ -38,8 +38,8 @@ __s64 res_full;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_full(const void *ctx)
 {
@@ -58,8 +58,8 @@ static volatile int zero = 0;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_truncated(const void *ctx)
 {
diff --git a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
index b5cf85e2a19a..c6c1c3767a75 100644
--- a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
+++ b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
@@ -219,9 +219,9 @@ int mixed_mem_untrusted_btf_id_type(void *ctx)
 	 * returning 0.
 	 */
 	asm volatile (
-	"r7 = %[p];"
-	"if %[zero] != 0 goto +1;"
 	"r7 = %[q];"
+	"if %[zero] == 0 goto +1;"
+	"r7 = %[p];"
 	"r8 = *(u64 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
@@ -263,9 +263,9 @@ int mixed_mem_btf_id_type(void *ctx)
 	 * gotten on its own.
 	 */
 	asm volatile (
-	"r7 = %[p];"
-	"if %[zero] != 0 goto +1;"
 	"r7 = %[q];"
+	"if %[zero] == 0 goto +1;"
+	"r7 = %[p];"
 	"r8 = *(u32 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
@@ -305,9 +305,9 @@ int mixed_rdonly_mem_btf_id_type(void *ctx)
 	 * only its PTR_UNTRUSTED variant is.
 	 */
 	asm volatile (
-	"r7 = %[p];"
-	"if %[zero] != 0 goto +1;"
 	"r7 = %[q];"
+	"if %[zero] == 0 goto +1;"
+	"r7 = %[p];"
 	"r8 = *(u32 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
@@ -340,9 +340,9 @@ int mixed_mem_mem_type(void *ctx)
 	 * hence without the BPF_PROBE_MEM rewrite the other path needs.
 	 */
 	asm volatile (
-	"r7 = %[q];"
-	"if %[zero] == 0 goto +1;"
 	"r7 = %[p];"
+	"if %[zero] != 0 goto +1;"
+	"r7 = %[q];"
 	"r8 = *(u64 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
diff --git a/tools/testing/selftests/bpf/progs/test_global_func3.c b/tools/testing/selftests/bpf/progs/test_global_func3.c
index b332704b395c..b916355c5e59 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func3.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func3.c
@@ -205,6 +205,131 @@ int global_func3_32_frames(struct __sk_buff *skb)
 	return f31(skb);
 }
 
+volatile int again;
+
+static __attribute__ ((noinline))
+int calls_itself(struct __sk_buff *skb)
+{
+	if (again)
+		return (calls_itself(skb) >> 1) ^ 1;
+	return f16(skb);
+}
+
+#define FRAME(name, next)			\
+static __attribute__ ((noinline))		\
+int name(struct __sk_buff *skb)			\
+{						\
+	return next(skb);			\
+}
+
+FRAME(g1, calls_itself)
+FRAME(g2, g1)
+FRAME(g3, g2)
+FRAME(g4, g3)
+FRAME(g5, g4)
+FRAME(g6, g5)
+FRAME(g7, g6)
+FRAME(g8, g7)
+FRAME(g9, g8)
+FRAME(g10, g9)
+FRAME(g11, g10)
+FRAME(g12, g11)
+FRAME(g13, g12)
+FRAME(g14, g13)
+FRAME(g15, g14)
+FRAME(g16, g15)
+
+/*
+ * 16 frames above the subprog that calls itself and 16 below it. The call of it
+ * with nothing above it comes first.
+ */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__failure __msg("the call stack of {{[0-9]+}} frames is too deep")
+int global_func3_frames_around_recursion(struct __sk_buff *skb)
+{
+	return calls_itself(skb) + g16(skb);
+}
+
+static __attribute__ ((noinline))
+int depth(int n)
+{
+	return n > 0 ? (depth(n - 1) + 1) & 255 : 0;
+}
+
+SEC("kprobe")
+__arch_x86_64
+__arch_arm64
+__failure __msg("recursive depth() in a tracing program")
+int global_func3_recursion_in_tracing_prog(void *ctx)
+{
+	return depth(again + 7);
+}
+
+__attribute__ ((noinline))
+int redirects(struct __sk_buff *skb, int n)
+{
+	int ret = n > 0 ? redirects(skb, n - 1) : 0;
+
+	return ret + bpf_clone_redirect(skb, 1, 0);
+}
+
+/* The prog would run again on top of the frames of its recursion */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__failure __msg("recursive redirects() in a program that calls bpf_clone_redirect()")
+int global_func3_recursion_redirects(struct __sk_buff *skb)
+{
+	return redirects(skb, again + 7);
+}
+
+static __attribute__ ((noinline))
+int depth_to(int n, long *out)
+{
+	*out = depth(n);
+	return 0;
+}
+
+/* The pointer to the stack is left over in r2 at the call of depth(), which doesn't read r2 */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+int global_func3_recursion_stale_stack_pointer(struct __sk_buff *skb)
+{
+	long x = 0;
+
+	depth_to(again + 7, &x);
+	return x;
+}
+
+/* It writes to the stack of its caller, or a compiler hands down the value and no pointer */
+static __attribute__ ((noinline))
+long hands_down_its_stack(long *up, int n)
+{
+	long mine = n;
+
+	*up += 1;
+	if (n <= 0)
+		return mine;
+	return hands_down_its_stack(&mine, n - 1) + mine;
+}
+
+/* What a subprog does with the stack of its callers is not found for chains without an end */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__failure
+__msg("recursive hands_down_its_stack() passes a pointer to the stack to hands_down_its_stack()")
+int global_func3_recursion_stack_pointer(struct __sk_buff *skb)
+{
+	long x = skb->len;
+
+	return hands_down_its_stack(&x, skb->mark & 7);
+}
+
 #define PTR_FRAME(name, next)			\
 static __attribute__ ((noinline))		\
 long name(long *p)				\
diff --git a/tools/testing/selftests/bpf/progs/timer_failure.c b/tools/testing/selftests/bpf/progs/timer_failure.c
index 0538269101ca..84ea861575fc 100644
--- a/tools/testing/selftests/bpf/progs/timer_failure.c
+++ b/tools/testing/selftests/bpf/progs/timer_failure.c
@@ -43,14 +43,7 @@ static unsigned long timer_cb_ret_bad()
 }
 
 SEC("fentry/bpf_fentry_test1")
-__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before")
-__msg(": (85) call bpf_get_prandom_u32#7") /* anchor message */
-/* check that branch code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before ") __msg(": (85) call bpf_get_prandom_u32#7")
 __msg("should have been in [0, 0]")
 long BPF_PROG2(test_bad_ret, int, a)
 {
@@ -80,7 +73,7 @@ static int timer_cb_infinite_loop(void *map, int *key, struct elem *val)
 }
 
 SEC("fentry/bpf_fentry_test1")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn {{[0-9]+}} never exits")
 long BPF_PROG2(test_infinite_loop_cb, int, a)
 {
 	struct bpf_timer *timer;
diff --git a/tools/testing/selftests/bpf/progs/verifier_align.c b/tools/testing/selftests/bpf/progs/verifier_align.c
index 0083ef8b8ba7..aecc5279dbaf 100644
--- a/tools/testing/selftests/bpf/progs/verifier_align.c
+++ b/tools/testing/selftests/bpf/progs/verifier_align.c
@@ -13,7 +13,7 @@
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}} R3=2")
 __msg("1: {{.*}} R3=4")
 __msg("2: {{.*}} R3=8")
@@ -35,7 +35,7 @@ __naked void mov(void)
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}}R3=1")
 __msg("1: {{.*}}R3=2")
 __msg("2: {{.*}}R3=4")
@@ -69,7 +69,7 @@ __naked void shift(void)
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}}R3=4")
 __msg("1: {{.*}}R3=8")
 __msg("2: {{.*}}R3=10")
@@ -93,7 +93,7 @@ __naked void addsub(void)
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}}R3=7")
 __msg("1: {{.*}}R3=7")
 __msg("2: {{.*}}R3=14")
@@ -532,7 +532,7 @@ __msg("9: {{.*}} R6={{[^)]*}}var_off=(0x0; 0x3c)")
 /* Adding 14 makes R6 be (4n+2) */
 __msg("10: {{.*}} R6={{[^)]*}}var_off=(0x2; 0x7c)")
 /* Subtracting from packet pointer overflows ubounds */
-__msg("13: R5={{[^)]*}}var_off=(0xffffffffffffff82; 0x7c)")
+__msg("12: {{.*}} R5={{[^)]*}}var_off=(0xffffffffffffff82; 0x7c)")
 /* New unknown value in R7 is (4n), >= 76 */
 __msg("14: {{.*}} R7={{[^)]*}}var_off=(0x0; 0x7fc)")
 /* Adding it to packet pointer gives nice bounds again */
diff --git a/tools/testing/selftests/bpf/progs/verifier_and.c b/tools/testing/selftests/bpf/progs/verifier_and.c
index 2b4fdca162be..fb64ea9928f1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_and.c
+++ b/tools/testing/selftests/bpf/progs/verifier_and.c
@@ -38,9 +38,9 @@ __naked void invalid_and_of_negative_number(void)
 	r1 &= -4;					\
 	r1 <<= 2;					\
 	r0 += r1;					\
-l0_%=:	r1 = %[test_val_foo];				\
+	r1 = %[test_val_foo];				\
 	*(u64*)(r0 + 0) = r1;				\
-	exit;						\
+l0_%=:	exit;						\
 "	:
 	: __imm(bpf_map_lookup_elem),
 	  __imm_addr(map_hash_48b),
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
index 1d676da4b5ea..b46e30a9c67f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
@@ -199,6 +199,88 @@ __naked void call_through(void)
 	: __clobber_all);
 }
 
+unsigned long first_r12, last_r12;
+
+/* 1024 bytes in the stack in arena, nothing in the kernel stack */
+static __naked __noinline __used long goes_down(void)
+{
+	asm volatile (
+	".8byte %[adj];"
+	".8byte %[mov];"	/* r1 = r12 */
+	"r2 = %[last_r12] ll;"
+	"*(u64 *)(r2 + 0) = r1;"
+	"call goes_down;"
+	"exit;"
+	:
+	: __imm_addr(last_r12),
+	  __imm_insn(adj, ADJ(1024)),
+	  __imm_insn(mov, MOV(BPF_REG_1))
+	: __clobber_all);
+}
+
+/* The check of a recursion looks at the kernel stack. It fails before 32K of arena are used. */
+SEC("syscall")
+__description("arena stack: recursion ends in the stack")
+__success __retval(1)
+__naked void recursion(void)
+{
+	asm volatile (
+	".8byte %[adj];"
+	".8byte %[mov];"	/* r1 = r12 */
+	"r2 = %[first_r12] ll;"
+	"*(u64 *)(r2 + 0) = r1;"
+	"call goes_down;"
+	"r1 = %[last_r12] ll;"
+	"r1 = *(u64 *)(r1 + 0);"
+	"r2 = %[first_r12] ll;"
+	"r2 = *(u64 *)(r2 + 0);"
+	"r1 -= r2;"
+	"r0 = 0;"
+	"if r1 < 8192 goto +2;"
+	"if r1 > 32736 goto +1;"
+	"r0 = 1;"
+	"exit;"
+	:
+	: __imm_addr(first_r12),
+	  __imm_addr(last_r12),
+	  __imm_insn(adj, ADJ(32)),
+	  __imm_insn(mov, MOV(BPF_REG_1))
+	: __clobber_all);
+}
+
+static __naked __noinline __used long large_frame(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 1792) = 0;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __naked __noinline __used long goes_down_with_leaf(void)
+{
+	asm volatile (
+	".8byte %[adj];"
+	"call large_frame;"
+	"call goes_down_with_leaf;"
+	"exit;"
+	:
+	: __imm_insn(adj, ADJ(1024))
+	: __clobber_all);
+}
+
+/* goes_down_with_leaf() gets a frame in the kernel stack. What it calls comes on top. */
+SEC("syscall")
+__description("arena stack: the kernel frame of a recursion counts in the depth of the stack")
+__failure __msg("combined stack size of 3 calls is")
+__naked void recursion_above_large_frame(void)
+{
+	asm volatile (
+	"call goes_down_with_leaf;"
+	"exit;"
+	::: __clobber_all);
+}
+
 SEC("syscall")
 __description("arena stack: helpers don't change r12")
 __success __retval(3)
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c
index 05efa9f4a522..6fede778971c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c
@@ -1797,8 +1797,8 @@ l1_%=:	r0 = 1;				\
 
 SEC("socket")
 __description("jset on same register, scalar value unknown branch 1")
-__msg("3: (b7) r0 = 0 {{.*}} R0=0")
 __msg("5: (b7) r0 = 1 {{.*}} R0=1")
+__msg("3: (b7) r0 = 0 {{.*}} R0=0")
 __success __log_level(2)
 __naked void jset_on_same_register_3(void *ctx)
 {
@@ -1818,8 +1818,8 @@ l1_%=:	r0 = 1;				\
 
 SEC("socket")
 __description("jset on same register, scalar value unknown branch 2")
-__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __msg("6: (b7) r0 = 1 {{.*}} R0=1")
+__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __success __log_level(2)
 __naked void jset_on_same_register_4(void *ctx)
 {
@@ -1840,8 +1840,8 @@ l1_%=:	r0 = 1;				\
 
 SEC("socket")
 __description("jset on same register, scalar value unknown branch 3")
-__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __msg("6: (b7) r0 = 1 {{.*}} R0=1")
+__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __success __log_level(2)
 __naked void jset_on_same_register_5(void *ctx)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index f3f8ea67eb7e..afe4990cd027 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -243,8 +243,7 @@ __naked void callx_scratches_args(void)
 /* return value of the callee is tracked */
 SEC("socket")
 __success __log_level(2)
-__msg("to caller at 4:")
-__msg("R0=6")
+__msg("4: R0=6 R10=fp0")
 __naked void callx_retval_is_tracked(void)
 {
 	asm volatile (
@@ -541,9 +540,11 @@ static unsigned long self(void)
 	);
 }
 
-/* unbounded recursion is caught by the main verification pass */
+/* unbounded recursion is stopped at run time */
 SEC("socket")
-__failure __msg("frames is too deep")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
 __naked void callx_unbounded_recursion(void)
 {
 	asm volatile (
@@ -569,12 +570,10 @@ static unsigned long countdown(void)
 	);
 }
 
-/*
- * The depth of the recursion is known to the main verification pass,
- * but recursive calls are not allowed regardless.
- */
 SEC("socket")
-__failure __msg("recursive call from countdown() to countdown()")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
 __naked void callx_bounded_recursion(void)
 {
 	asm volatile (
@@ -621,7 +620,9 @@ static unsigned long pong(void)
 }
 
 SEC("socket")
-__failure __msg("recursive call from")
+__arch_x86_64
+__arch_arm64
+__success
 __naked void callx_mutual_recursion(void)
 {
 	asm volatile (
@@ -636,6 +637,49 @@ __naked void callx_mutual_recursion(void)
 		: __clobber_all);
 }
 
+/* calls_arg(fn) { return fn(); } */
+__naked __noinline __used
+static unsigned long calls_arg(void)
+{
+	asm volatile (
+		"callx r1;"
+		"exit;"
+	);
+}
+
+__naked __noinline __used
+static unsigned long returns_7(void)
+{
+	asm volatile (
+		"r0 = 7;"
+		"exit;"
+	);
+}
+
+/*
+ * The address of calls_arg() is taken: its callx could call it. The main pass finds that it
+ * calls returns_7(). There is no cycle, so calls_arg() doesn't check the stack at run time.
+ */
+SEC("socket")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__xlated("...")
+__xlated("exit")
+__xlated("callx r1")
+__naked void callx_no_recursion_after_all(void)
+{
+	asm volatile (
+		"r2 = %[calls_arg] ll;"
+		"r1 = %[returns_7] ll;"
+		"callx r2;"
+		"exit;"
+		:
+		: __imm_addr(calls_arg),
+		  __imm_addr(returns_7)
+		: __clobber_all);
+}
+
 __naked __noinline __used
 static unsigned long use_stack_304(void)
 {
@@ -944,11 +988,8 @@ long long vals[] SEC(".data.vals") = {1, 2, 3, 4};
 
 SEC("socket")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 11: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (8d) callx r2")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("5: R0=3 R10=fp0")
+__msg("8: (0f) r1 += r0                      ; R0=24 R1=map_value(map=.data.vals,ks=4,vs=32,imm=24)")
 __retval(4)
 __naked void callx_precision(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 642071fa7498..8bb74b579203 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -669,7 +669,9 @@ static unsigned long selfcall(void)
 }
 
 SEC("socket")
-__failure __msg("recursive call from selfcall() to selfcall()")
+__arch_x86_64
+__arch_arm64
+__success
 __naked void callx_rodata_recursion(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_cfg.c b/tools/testing/selftests/bpf/progs/verifier_cfg.c
index b429fa7a08f5..e97e89ba3d68 100644
--- a/tools/testing/selftests/bpf/progs/verifier_cfg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_cfg.c
@@ -138,8 +138,8 @@ l0_%=:	r1 = r0;					\
 
 SEC("socket")
 __description("conditional loop")
-__failure __msg("infinite loop detected")
-__msg_unpriv("back-edge")
+__success
+__failure_unpriv __msg_unpriv("back-edge")
 __naked void conditional_loop(void)
 {
 	asm volatile ("					\
@@ -175,7 +175,7 @@ l1_%=:	r8 += r4;					\
 
 SEC("socket")
 __description("unconditional loop after conditional jump")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 2 never exits")
 __failure_unpriv __msg_unpriv("back-edge from insn 3 to 2")
 __naked void uncond_loop_after_cond_jmp(void)
 {
@@ -201,7 +201,7 @@ static unsigned long never_ending_subprog()
 SEC("socket")
 __description("unconditional loop after conditional jump")
 /* infinite loop is detected *after* check_cfg() */
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 6 never exits")
 __naked void uncond_loop_in_subprog_after_cond_jmp(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
index 5e8461b2167d..100d8307c3bd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
+++ b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
@@ -557,12 +557,13 @@ __noinline int global_calls_loop(int x)
 
 /*
  * loop_cb_calls_static() -> static_calls_global() -> global_calls_loop() ->
- * bpf_loop() -> loop_cb_calls_static() is an unbounded recursion that the
- * main verification pass can't see, because it doesn't follow calls of global
- * functions. None of the functions use stack.
+ * bpf_loop() -> loop_cb_calls_static() is an unbounded recursion through a global
+ * function and a helper. It's stopped at run time. None of the functions use stack.
  */
-SEC("?raw_tp")
-__failure __msg("recursive call from global_calls_loop() to loop_cb_calls_static()")
+SEC("?socket")
+__arch_x86_64
+__arch_arm64
+__success
 int recursion_via_global_func_and_callback(const void *ctx)
 {
 	return loop_cb_calls_static(0, NULL);
diff --git a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
index c6603a118fdc..1b13d3a8dcfa 100644
--- a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
@@ -717,12 +717,12 @@ __naked void bounds_check_using_bad_access_1(void)
 	*(u64*)(r2 + 0) = r1;				\
 	r1 = %[map_hash_48b] ll;			\
 	call %[bpf_map_lookup_elem];			\
-	if r0 == 0 goto l0_%=;				\
+	if r0 == 0 goto l1_%=;				\
 	r1 = r0;					\
 	r3 = *(u32*)(r0 + 0);				\
 	if r3 < 32 goto l1_%=;				\
 	r1 += r3;					\
-l0_%=:	r0 = 0;						\
+	r0 = 0;						\
 	*(u8*)(r1 + 0) = r0;				\
 	r0 = 0;						\
 	exit;						\
@@ -775,12 +775,12 @@ __naked void bounds_check_using_bad_access_2(void)
 	*(u64*)(r2 + 0) = r1;				\
 	r1 = %[map_hash_48b] ll;			\
 	call %[bpf_map_lookup_elem];			\
-	if r0 == 0 goto l0_%=;				\
+	if r0 == 0 goto l1_%=;				\
 	r1 = r0;					\
 	r3 = *(u32*)(r0 + 0);				\
 	if r3 <= 32 goto l1_%=;				\
 	r1 += r3;					\
-l0_%=:	r0 = 0;						\
+	r0 = 0;						\
 	*(u8*)(r1 + 0) = r0;				\
 	r0 = 0;						\
 	exit;						\
diff --git a/tools/testing/selftests/bpf/progs/verifier_idoms.c b/tools/testing/selftests/bpf/progs/verifier_idoms.c
index fdee6bced77b..91d9f9aae8e9 100644
--- a/tools/testing/selftests/bpf/progs/verifier_idoms.c
+++ b/tools/testing/selftests/bpf/progs/verifier_idoms.c
@@ -10,7 +10,7 @@
  * The immediate dominator is printed by log_program() as the numeric column
  * immediately before the "<insn#>:" field of every "Program dump" line:
  *
- *   Program dump (scc? loop_header? idom insn#: live_regs_before):
+ *   Program dump (loop_header? idom insn#: live_regs_before):
  *            -1   0: ....... (b7) r0 = 0     <- idom(0) = -1 (subprog entry)
  *             0   1: ....... (07) r0 += 1    <- idom(1) = 0
  *             ^^^^^^
@@ -307,17 +307,16 @@ __naked void entry_is_loop_header(void)
 /*
  * Self-loop: insn 1 is its own predecessor. idom(1) must be the pre-header
  * (insn 0); the self-edge is ignored (idoms[pred] == -1 on the first pass, then
- * idoms_intersect(a == b) short-circuits). Program is rejected later for an
- * infinite loop, but the dump (and idoms) are printed before that.
+ * idoms_intersect(a == b) short-circuits). The loop can spin forever: the
+ * verifier adds may_goto to it.
  */
 SEC("socket")
-__failure
+__success
 __log_level(2)
 __msg("Program dump")
 __msg("{{.*}} -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
 __msg("{{.*}}  0   1: {{.*}} (a5) if r0 < 0xa goto pc-1")
 __msg("{{.*}}  1   2: {{.*}} (95) exit")
-__msg("infinite loop detected")
 __naked void self_loop(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
index 4dd98acc6436..a3e31aa25fe1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
+++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
@@ -45,7 +45,7 @@ SEC("?raw_tp")
 __failure __msg("R1 type=scalar expected=fp")
 int unsafe_on_2nd_iter(void *unused)
 {
-	char buf[4];
+	char buf[8];
 	struct buf_context loop_ctx = { .buf = buf };
 
 	bpf_loop(100, unsafe_on_2nd_iter_cb, &loop_ctx, 0);
@@ -94,7 +94,7 @@ static int loop_detection_cb(__u32 idx, struct num_context *ctx)
 }
 
 SEC("?raw_tp")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn {{[0-9]+}} never exits")
 int loop_detection(void *unused)
 {
 	struct num_context loop_ctx = { .i = 0 };
@@ -246,10 +246,9 @@ int bpf_loop_iter_limit_nested(void *unused)
 	/* This makes 'c' zero, but neither clang nor verifier know it. */
 	c /= 10;
 	/* Make sure that verifier does not visit 'impossible' states:
-	 * enumerate all possible callback visit masks.
+	 * one more call of any of the callbacks gives more than that.
 	 */
-	if (a != 0 && a != 1 && a != 11 && a != 101 && a != 111 &&
-	    b != 0 && b != 1 && b != 11 && b != 101 && b != 111)
+	if (a > 111 || b > 111)
 		asm volatile ("r0 /= 0;" ::: "r0");
 	return 1000 * a + b + c;
 }
@@ -291,9 +290,13 @@ static __naked void iter_limit_bug_cb(void)
 	);
 }
 
+/*
+ * may_goto doesn't leave the loop, so it's not what stops it. The address is greater than 10:
+ * the loop spins until may_goto that the verifier adds ends the prog.
+ */
 int tmp_var;
 SEC("socket")
-__failure __msg("infinite loop detected at insn 2")
+__success __retval(0)
 __naked void jgt_imm64_and_may_goto(void)
 {
 	asm volatile ("			\
@@ -310,7 +313,7 @@ l0_%=:	.byte 0xe5; /* may_goto */	\
 }
 
 SEC("socket")
-__failure __msg("infinite loop detected at insn 1")
+__success __retval(0)
 __naked void may_goto_self(void)
 {
 	asm volatile ("			\
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 795956bd4f9f..32bab3cb3fcb 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -2844,7 +2844,7 @@ __naked void merge_preserves_may_write(void)
 	"if r0 == 0 goto 1f;"
 	/* chain A: r2 = map value (ARG_NONE, not stack) */
 	"r1 = r10;"
-	"r1 += -16;"				/* unused fp anchor */
+	"r1 += -16;"				/* fp anchor */
 	"r2 = r0;"
 	"call mvs_mid;"
 	/* chain B: r2 = caller stack slot fp-8 */
@@ -2872,6 +2872,7 @@ static __used __naked void mvs_leaf(void)
 {
 	asm volatile (
 	"*(u64 *)(r2 + 0) = 42;"
+	"r0 = r1;"				/* a register that is not read is no argument */
 	"r0 = 0;"
 	"exit;"
 	::: __clobber_all);
diff --git a/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c b/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
index 72646fa2745e..3f2a60f19319 100644
--- a/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
+++ b/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
@@ -32,8 +32,12 @@ char _license[] SEC("license") = "GPL";
 			"r1 += -" #off ";"	\
 			"call " #fn ";"
 
-/* 50 calls, each with a distinct FP offset: -8, -16, ... -400 */
+/*
+ * 50 calls, each with a distinct FP offset: -8, -16, ... -400.
+ * The function reads r1 first: a register that the callee doesn't read is no argument.
+ */
 #define CALLS_50(fn)							\
+	"r0 = r1;"							\
 	C(fn,   8) C(fn,  16) C(fn,  24) C(fn,  32) C(fn,  40)		\
 	C(fn,  48) C(fn,  56) C(fn,  64) C(fn,  72) C(fn,  80)		\
 	C(fn,  88) C(fn,  96) C(fn, 104) C(fn, 112) C(fn, 120)		\
@@ -50,6 +54,7 @@ __naked __noinline __used
 static unsigned long exp_sub7(void)
 {
 	asm volatile (
+		"r0 = r1;"
 		"r0 = 0;"
 		"exit;"
 		::: __clobber_all);
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
index 0608e983d7fc..4f4e04d50bc7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
@@ -22,10 +22,10 @@ __success
 __log_level(2)
 __msg("loop at 1{{$}}")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
-__msg("          2   3: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("      2   3: {{.*}} (95) exit")
 __naked void loop_single(void)
 {
 	asm volatile ("					\
@@ -43,13 +43,13 @@ __log_level(2)
 __msg("loop at 1{{$}}")
 __msg("loop at 4{{$}}")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  2       0   1: {{.*}} (07) r0 += 1")
-__msg("  2   1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
-__msg("          2   3: {{.*}} (b7) r1 = 0")
-__msg("  1       3   4: {{.*}} (07) r1 += 1")
-__msg("  1   4   4   5: {{.*}} (a5) if r1 < 0xa goto pc-2")
-__msg("          5   6: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("      2   3: {{.*}} (b7) r1 = 0")
+__msg("      3   4: {{.*}} (07) r1 += 1")
+__msg("  4   4   5: {{.*}} (a5) if r1 < 0xa goto pc-2")
+__msg("      5   6: {{.*}} (95) exit")
 __naked void loop_two_independent(void)
 {
 	asm volatile ("					\
@@ -70,13 +70,13 @@ __log_level(2)
 __msg("loop at 1{{$}}")
 __msg("loop at 3, nested in 1")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")			/* outer loop header */
-__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")			/* outer loop insn */
-__msg("  1   1   2   3: {{.*}} (07) r1 += 1")			/* inner loop header */
-__msg("  1   3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")	/* inner loop insn */
-__msg("  1   1   4   5: {{.*}} (a5) if r0 < 0xa goto pc-5")	/* outer loop insn */
-__msg("          5   6: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")			/* outer loop header */
+__msg("  1   1   2: {{.*}} (b7) r1 = 0")			/* outer loop insn */
+__msg("  1   2   3: {{.*}} (07) r1 += 1")			/* inner loop header */
+__msg("  3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")	/* inner loop insn */
+__msg("  1   4   5: {{.*}} (a5) if r0 < 0xa goto pc-5")	/* outer loop insn */
+__msg("      5   6: {{.*}} (95) exit")
 __naked void loop_nested(void)
 {
 	asm volatile ("					\
@@ -95,12 +95,12 @@ SEC("socket")
 __success
 __log_level(2)
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (85) call bpf_get_prandom_u32")
-__msg("          0   1: {{.*}} (25) if r0 > 0x0 goto pc+2")
-__msg("          1   2: {{.*}} (b7) r0 = 2")
-__msg("          2   3: {{.*}} (05) goto pc+1")
-__msg("          1   4: {{.*}} (b7) r0 = 3")
-__msg("          1   5: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (85) call bpf_get_prandom_u32")
+__msg("      0   1: {{.*}} (25) if r0 > 0x0 goto pc+2")
+__msg("      1   2: {{.*}} (b7) r0 = 2")
+__msg("      2   3: {{.*}} (05) goto pc+1")
+__msg("      1   4: {{.*}} (b7) r0 = 3")
+__msg("      1   5: {{.*}} (95) exit")
 __naked void fwd_edges_no_loop(void)
 {
 	asm volatile ("					\
@@ -123,16 +123,16 @@ __msg("loop at 1{{$}}")
 __msg("loop at 3, nested in 1")
 __msg("loop at 6, nested in 1")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")
-__msg("  1   1   2   3: {{.*}} (07) r1 += 1")
-__msg("  1   3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
-__msg("  1   1   4   5: {{.*}} (b7) r2 = 0")
-__msg("  1   1   5   6: {{.*}} (07) r2 += 1")
-__msg("  1   6   6   7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
-__msg("  1   1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
-__msg("          8   9: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (b7) r1 = 0")
+__msg("  1   2   3: {{.*}} (07) r1 += 1")
+__msg("  3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
+__msg("  1   4   5: {{.*}} (b7) r2 = 0")
+__msg("  1   5   6: {{.*}} (07) r2 += 1")
+__msg("  6   6   7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
+__msg("  1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg("      8   9: {{.*}} (95) exit")
 __naked void loop_nested_siblings(void)
 {
 	asm volatile ("					\
@@ -157,16 +157,16 @@ __msg("loop at 1{{$}}")
 __msg("loop at 3, nested in 1")
 __msg("loop at 5, nested in 3")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")
-__msg("  1   1   2   3: {{.*}} (07) r1 += 1")
-__msg("  1   3   3   4: {{.*}} (b7) r2 = 0")
-__msg("  1   3   4   5: {{.*}} (07) r2 += 1")
-__msg("  1   5   5   6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
-__msg("  1   3   6   7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
-__msg("  1   1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
-__msg("          8   9: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (b7) r1 = 0")
+__msg("  1   2   3: {{.*}} (07) r1 += 1")
+__msg("  3   3   4: {{.*}} (b7) r2 = 0")
+__msg("  3   4   5: {{.*}} (07) r2 += 1")
+__msg("  5   5   6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
+__msg("  3   6   7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
+__msg("  1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg("      8   9: {{.*}} (95) exit")
 __naked void loop_three_levels(void)
 {
 	asm volatile ("					\
@@ -189,14 +189,14 @@ __success
 __log_level(2)
 __msg("loop at 1")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (bf) r1 = r0")
-__msg("  1   1   2   3: {{.*}} (25) if r1 > 0x5 goto pc+1")
-__msg("  1   1   3   4: {{.*}} (b7) r1 = 1")
-__msg("  1   1   3   5: {{.*}} (0f) r0 += r1")
-__msg("  1   1   5   6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
-__msg("          6   7: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (bf) r1 = r0")
+__msg("  1   2   3: {{.*}} (25) if r1 > 0x5 goto pc+1")
+__msg("  1   3   4: {{.*}} (b7) r1 = 1")
+__msg("  1   3   5: {{.*}} (0f) r0 += r1")
+__msg("  1   5   6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
+__msg("      6   7: {{.*}} (95) exit")
 __naked void loop_with_if_else(void)
 {
 	asm volatile ("					\
@@ -217,15 +217,15 @@ __success
 __log_level(2)
 __msg("loop at 3, irreducible")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
-__msg("          0   1: {{.*}} (b7) r1 = 0")
-__msg("          1   2: {{.*}} (25) if r0 > 0x5 goto pc+2")
-__msg("  1       2   3: {{.*}} (b7) r1 = 1")
-__msg("  1   3   3   4: {{.*}} (05) goto pc+1")
-__msg("          2   5: {{.*}} (b7) r1 = 2")
-__msg("  1   3   2   6: {{.*}} (0f) r0 += r1")
-__msg("  1   3   6   7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
-__msg("          7   8: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("      0   1: {{.*}} (b7) r1 = 0")
+__msg("      1   2: {{.*}} (25) if r0 > 0x5 goto pc+2")
+__msg("      2   3: {{.*}} (b7) r1 = 1")
+__msg("  3   3   4: {{.*}} (05) goto pc+1")
+__msg("      2   5: {{.*}} (b7) r1 = 2")
+__msg("  3   2   6: {{.*}} (0f) r0 += r1")
+__msg("  3   6   7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
+__msg("      7   8: {{.*}} (95) exit")
 __naked void loop_irreducible(void)
 {
 	asm volatile ("					\
@@ -244,10 +244,9 @@ __naked void loop_irreducible(void)
 }
 
 SEC("socket")
-__failure
+__success
 __log_level(2)
 __msg("loop at 1")
-__msg("infinite loop detected at insn 1")
 __naked void loop_self(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
index 6dd46edd8ed0..e8daf791b4c0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -120,7 +120,7 @@ BAD("down from a range, one unsigned test, one too many")
 LOOP(down_unsigned_range_bad, "if r7 > 2048 goto 9f;", "w7 += -1; if w7 < 2049 goto 1b;")
 
 /* below 0 a 32-bit counter is 0xffffffff for a 64-bit jump */
-BAD("down, a 64-bit test with a bound above 32 bits")
+NO_COUNT("down, a 64-bit test with a bound above 32 bits")
 LOOP(down_unsigned_wide_bound, "w8 = 2047; r9 = 0x100000800 ll;", "w8 += -1; if r8 < r9 goto 1b;")
 
 BAD("down, an unsigned and a signed 64-bit test")
diff --git a/tools/testing/selftests/bpf/progs/verifier_loops1.c b/tools/testing/selftests/bpf/progs/verifier_loops1.c
index 48a966cda199..18067907579e 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loops1.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loops1.c
@@ -123,7 +123,7 @@ l0_%=:	exit;						\
 
 SEC("tracepoint")
 __description("infinite loop after a conditional jump")
-__failure __msg("program is too large")
+__failure __msg("loop at insn 2 never exits")
 __naked void loop_after_a_conditional_jump(void)
 {
 	asm volatile ("					\
@@ -135,10 +135,11 @@ l0_%=:	exit;						\
 "	::: __clobber_all);
 }
 
-SEC("tracepoint")
+SEC("xdp")
 __description("bounded recursion")
-__failure
-__msg("recursive call from")
+__arch_x86_64
+__arch_arm64
+__success __retval(4)
 __naked void bounded_recursion(void)
 {
 	asm volatile ("					\
@@ -163,7 +164,7 @@ l0_%=:	call bounded_recursion__1;			\
 
 SEC("tracepoint")
 __description("infinite loop in two jumps")
-__failure __msg("loop detected")
+__failure __msg("loop at insn 1 never exits")
 __naked void infinite_loop_in_two_jumps(void)
 {
 	asm volatile ("					\
@@ -174,9 +175,10 @@ l0_%=:	if r0 < 4 goto l1_%=;				\
 "	::: __clobber_all);
 }
 
-SEC("tracepoint")
+/* It never ends on its own. may_goto that the verifier adds ends the prog. */
+SEC("xdp")
 __description("infinite loop: three-jump trick")
-__failure __msg("loop detected")
+__success __retval(0)
 __naked void infinite_loop_three_jump_trick(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_movsx.c b/tools/testing/selftests/bpf/progs/verifier_movsx.c
index 195b27a51224..2f4a678c31c2 100644
--- a/tools/testing/selftests/bpf/progs/verifier_movsx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_movsx.c
@@ -226,7 +226,8 @@ l0_%=:							\
 
 SEC("socket")
 __description("MOV32SX, S8, var_off u32_max")
-__failure __msg("infinite loop detected")
+__success __log_level(2)
+__msg("w7 = (s8)w3 {{.*}}R7=scalar({{.*}}smin32=-128,smax32=127,var_off=(0x0; 0xffffffff))")
 __failure_unpriv __msg_unpriv("back-edge from insn 2 to 0")
 __naked void mov64sx_s32_varoff_1(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 07c356fde68d..5e566933f7e4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -8,6 +8,7 @@
 #include "../../../include/linux/filter.h"
 #include "bpf_misc.h"
 #include "bpf_kfuncs.h"
+#include "bpf_compiler.h"
 
 #define COND_TEST(name, _initial, _step, _bound, msg, body)                     \
 SEC("xdp")                                                                      \
@@ -238,7 +239,7 @@ __naked void meet_agrees(void)
 }
 
 SEC("xdp")
-__failure
+__success __retval(0)
 __log_level(2)
 __msg_next("scev at header 1:")
 __msg_next("  r6=(any r6 (+ r6 1)) / ?")
@@ -1234,23 +1235,20 @@ RANGE_TEST(backedges_first_is_a_range, "", 0, 3, 1, 0, 2,
 RANGE_TEST(backedges_two_variables, "", 0, 3, 1, 0, 1,
 	   BACKEDGES("r6 += 1; r7 += 1", "r9 & 1", "r6 < 3", "r7 < 6"),
 	   __msg("header_count is [0..6] ") __msg("widening r6 to 0..8 "))
-/* Explore with a bound of 3 first, of 5 next. The second state is not equal to the first. */
+/* A bound of 3 and a bound of 5 are joined in front of the loop */
 #define PRECISE_BOUND(reg)									\
 	"r4 = 2; r5 = 2; " reg " = 5; if r7 == 0 goto +1; " reg " = 3; r0 = 0;"		\
 	BACKEDGES("r6 += 1", "r9 & 1", "r6 < r4", "r6 < r5")
-#define EXPLORED_TWICE										\
-	__flag(BPF_F_TEST_STATE_FREQ)								\
-	__msg("header_count is [0..3] ") __msg("header_count is [0..5] ")
-RANGE_TEST(backedges_bound1_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r4"), EXPLORED_TWICE)
-RANGE_TEST(backedges_bound0_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r5"), EXPLORED_TWICE)
+#define EXPLORED_ONCE	__msg("header_count is [0..5] ") __not_msg("header_count is [0..3] ")
+RANGE_TEST(backedges_bound1_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r4"), EXPLORED_ONCE)
+RANGE_TEST(backedges_bound0_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r5"), EXPLORED_ONCE)
 
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 10 ")
 __msg("loop header at 1, widening r0 to 0..9 step 1")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
 __naked void post_cond_jlt(void)
 {
 	asm volatile ("					\
@@ -1262,8 +1260,8 @@ __naked void post_cond_jlt(void)
 }
 
 /*
- * Explore with r7=10 first, r7=4096 next.
- * Should trigger invalid stack read if r7 is marked precise.
+ * r7 is 10 on one path to the loop and 4096 on the other.
+ * Should trigger invalid stack read if the larger one counts.
  *
  *   r0 = 0;
  *   r6 = 0;
@@ -1278,11 +1276,7 @@ __naked void post_cond_jlt(void)
 SEC("xdp")
 __failure
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("loop header at 6, header_count is 10 ")
-__msg("mark_precise: frame0: parent state regs=r6,r7 stack=:")
-__msg("loop header at 6, widening r0 to 0..9 step 1")
-__msg("loop header at 6, header_count is 4096 ")
+__msg("loop header at 6, header_count is [0..4096] ")
 __msg("loop header at 6, widening r0 to 0..4095 step 1")
 __msg("invalid variable-offset read from stack R2")
 __naked void mark_bound_precise(void)
@@ -1308,8 +1302,8 @@ __naked void mark_bound_precise(void)
 }
 
 /*
- * Explore with r6=4086 first, r6=0 next.
- * Should trigger invalid stack read if r6 is marked precise.
+ * r6 is 4086 on one path to the loop and 0 on the other.
+ * Should trigger invalid stack read if the smaller one counts.
  *
  *   r0 = 0;
  *   r6 = bpf_get_prandom_u32() == 42 ? 4086 : 0;
@@ -1323,11 +1317,7 @@ __naked void mark_bound_precise(void)
 SEC("xdp")
 __failure
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("loop header at 5, header_count is 10 ")
-__msg("mark_precise: frame0: parent state regs=r6 stack=:")
-__msg("loop header at 5, widening r0 to 0..9 step 1")
-__msg("loop header at 5, header_count is 4096 ")
+__msg("loop header at 5, header_count is [0..4096] ")
 __msg("loop header at 5, widening r0 to 0..4095 step 1")
 __msg("invalid variable-offset read from stack R2")
 __naked void mark_base_precise(void)
@@ -1541,7 +1531,7 @@ __msg("scev at header 1:")
 __msg_next("r6=(+ r6 1) / (linear r6 1)")
 __msg_next("*fp-8=(any *fp-8 7) / (any *fp-8 7)")
 __msg("loop header at 1, widening r6 to 0..2 step 1")
-__msg("2: (7a) *(u64 *)(r10 -8) = 7          ; R10=fp0 fp-8=7 loop_stack=1")
+__msg("3: (79) r0 = *(u64 *)(r10 -8)         ; R0=scalar() R10=fp0 loop_stack=1")
 __naked void widen_any_unallocated_slot(void)
 {
 	asm volatile ("					\
@@ -1819,11 +1809,10 @@ BOUND_IN_MAP_TEST(bound_in_data, data_bound, 16)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 3, header_count is 8")
 __msg("loop header at 3, widening r0 to 2..9 step 1")
 __msg("3: R0=scalar(smin=umin=smin32=umin32=2,smax=umax=smax32=umax32=9,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void post_cond_jlt_with_base(void)
 {
 	asm volatile ("					\
@@ -1910,11 +1899,10 @@ NO_ZEXT32_TEST(arsh_by_32, 0x7ffffffe, "r7 <<= 32; r7 s>>= 32;")
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 11")
 __msg("loop header at 1, widening r0 to 0..10 step 1")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
 __naked void post_cond_jle(void)
 {
 	asm volatile ("					\
@@ -1928,11 +1916,10 @@ __naked void post_cond_jle(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 10")
 __msg("loop header at 1, widening r0 to 0..9 step 1")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=9,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void post_cond_jge(void)
 {
 	asm volatile ("					\
@@ -1947,11 +1934,10 @@ __naked void post_cond_jge(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 11")
 __msg("loop header at 1, widening r0 to 0..10 step 1")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void pre_cond_jge(void)
 {
 	asm volatile ("					\
@@ -1966,7 +1952,6 @@ __naked void pre_cond_jge(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(+ r0 1) / (linear r0 1)")
 __msg_next(" scev at latch 2:")
@@ -1975,10 +1960,9 @@ __msg("loop header at 1, widening r0")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
 __msg("1: (07) r0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
 __msg("2: (55) if r0 != 0x3 goto pc-2")
-__msg("3: (95) exit")
 __msg("loop header at 1, clamping r0")
-__msg("from 2 to 1: safe")
-__msg("processed 5 insns")
+__msg("3: (95) exit")
+__msg("processed 4 insns")
 __naked void post_cond_jne(void)
 {
 	asm volatile ("					\
@@ -1992,16 +1976,15 @@ __naked void post_cond_jne(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(+ r0 -1) / (linear r0 -1)")
 __msg_next(" scev at latch 2:")
 __msg_next("  r0=(+ r0 -1) / (linear (+ r0 -1) -1)")
 __msg("loop header at 1, header_count is 3 ")
 __msg("loop header at 1, widening r0 to 1..3 step 1")
-__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) loop_stack=1")
+__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) R10=fp0 loop_stack=1")
 __msg("loop header at 1, clamping r0 to 1..3 step 1")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
 __naked void post_cond_jne_neg_step(void)
 {
 	asm volatile ("					\
@@ -2025,7 +2008,7 @@ __msg("loop header at 2, widening r1 to 0..4 step 1")
 __msg("loop header at 2, widening r3 to 0..4 step 1")
 __msg("3: (55) if r3 != 0x5 goto pc+1 {{.*}}; R3=5")
 __msg("loop header at 2, clamping r3 to 0..4 step 1")
-__msg("from 6 to 2: safe")
+__msg("from 6 to 2: within")
 __naked void clamp_dependent_overshoot(void)
 {
 	asm volatile ("					\
@@ -2051,7 +2034,7 @@ __msg("loop header at 2, widening r1 to 0..4 step 1")
 __msg("loop header at 2, widening r3 to 0..4 step 1")
 __msg("3: (56) if w3 != 0x5 goto pc+1 {{.*}}; R3=5")
 __msg("loop header at 2, clamping r3 to 0..4 step 1")
-__msg("from 6 to 2: safe")
+__msg("from 6 to 2: within")
 __naked void clamp_dependent_overshoot32(void)
 {
 	asm volatile ("					\
@@ -2070,7 +2053,6 @@ __naked void clamp_dependent_overshoot32(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(+ r0 1) / (linear r0 1)")
 __msg_next(" scev at latch 1:")
@@ -2081,11 +2063,10 @@ __msg("1: (15) if r0 == 0x3 goto pc+2        ; R0=scalar(smin=smin32=0,smax=umax
 __msg("2: (07) r0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
 __msg("3: (05) goto pc-3")
 __msg("loop header at 1, clamping r0")
-__msg("1: safe")
 __msg("from 1 to 4: R0=3")
 __msg("4: R0=3")
 __msg("4: (95) exit")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void pre_cond_je1(void)
 {
 	asm volatile ("					\
@@ -2100,7 +2081,6 @@ __naked void pre_cond_je1(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(zext32 (+ r0 1)) / (zext32 (linear r0 1))")
 __msg_next(" scev at latch 2:")
@@ -2109,10 +2089,9 @@ __msg("loop header at 1, widening r0")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
 __msg("1: (04) w0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
 __msg("2: (56) if w0 != 0x3 goto pc-2")
-__msg("3: (95) exit")
 __msg("loop header at 1, clamping r0")
-__msg("from 2 to 1: safe")
-__msg("processed 5 insns")
+__msg("3: (95) exit")
+__msg("processed 4 insns")
 __naked void post_cond_jne32(void)
 {
 	asm volatile ("					\
@@ -2386,7 +2365,6 @@ __naked void loop_around_301_exits(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 11:")
 __msg_next("  r0=(+ r0 1) / (linear r0 1)")
 __msg_next("  r1=(+ r1 2) / (linear r1 2)")
@@ -2396,18 +2374,15 @@ __msg_next("  r1=(+ r1 2) / (linear (+ r1 2) 2)")
 __msg("loop header at 11, widening r0")
 __msg("loop header at 11, widening r1")
 __msg("11: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=7,var_off=(0x0; 0x7)) R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=14,var_off=(0x0; 0xe),step=0+2)")
-/* loop exit */
 __msg("16: (a5) if r0 < 0x8 goto pc-6")
-__msg("exiting loop 11")
-__msg("17: (95) exit")
 /* second iteration */
 __msg("loop header at 11, clamping r0")
 __msg("loop header at 11, clamping r1")
 /* iteration convergence */
-__msg("from 16 to 11: safe")
 __not_msg("{{^}}11:")
-/* map lookup error path */
-__msg("from 7 to 17: safe")
+/* loop exit */
+__msg("exiting loop 11")
+__msg("17: (95) exit")
 __naked void correlated_regs(void)
 {
 	asm volatile ("					\
@@ -2441,15 +2416,14 @@ __naked void correlated_regs(void)
  *     k += 1
  *     k <<= 1   // make SCEV construction not possible
  *     k >>= 1
- * map[k] = 1    // make k precise
+ * map[k] = 1    // k has to be known
  */
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("19: (72) *(u8 *)(r4 +0) = 1           ; R4=map_value(id={{.*}},map=map,ks=4,vs=1024,imm=16)")
 __not_msg("19: ")
-__msg("processed 106 insns")
+__msg("processed 105 insns")
 __naked void nested_loops_precise_var1(void)
 {
 	asm volatile ("					\
@@ -2488,8 +2462,7 @@ __log_level(2)
 __msg("loop header at 2, header_count is 100 ")
 __msg("loop header at 4, header_count is [0..100] ")
 __msg("loop header at 6, header_count is [0..100] ")
-__msg("processed 20 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 14 insns")
 __naked void nested_loop_with_two_exits(void)
 {
 	asm volatile ("					\
@@ -2550,14 +2523,14 @@ __msg("loop at 5")
 __msg("loop at 6, nested in 5")
 __msg("loop at 11, irreducible")
 __msg("loop at 12, nested in 11")
-/* entry via if r0 == 5 goto d_%= false branch */
+/* entry via if r0 == 5 goto d_%= true branch */
 __msg("loop header at 12, header_count is 3 ")
 __msg("loop header at 12, widening r9 to 0..2 step 1")
-__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) loop_stack=11,12")
-/* entry via if r0 == 5 goto d_%= true branch */
+/* entry via if r0 == 5 goto d_%= false branch */
 __msg("loop header at 12, header_count is 3 ")
 __msg("loop header at 12, widening r9 to 0..2 step 1")
-__msg("from 8 to 12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 loop_stack=11,12")
+__msg("12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 iter(11)=0 loop_stack=11,12")
+__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 iter(11)=0 loop_stack=11,12")
 __naked void enter_nested_loop_from_side(void)
 {
 	asm volatile ("					\
@@ -3037,7 +3010,7 @@ __log_level(2)
 __flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 3, widening r6 to 0..9 step 1")
 __msg("loop header at 3, clamping r6 to 0..9 step 1")
-__msg("12: safe")
+__msg("from 12 to 6: within")
 __naked void clamp_callee_state_pruning(void)
 {
 	asm volatile ("					\
@@ -3204,11 +3177,14 @@ RANGE_TEST(spill32_first_is_a_range, "", 0, 3, 1, 6, 6,
 RANGE_TEST(spill32_first_is_a_range_pre, "", 0, 3, 1, 6, 6,
 	   SPILL32_LOOP(SPILL32_LOAD "if r1 >= r7 goto 2f;" SPILL32_INC "goto 1b;"),
 	   SPILL32_COUNTED(7, "0..6"))
-/* Explore with 2 in the slot first, with 0 next. The second state is not equal to the first. */
+/* 2 and 0 in the slot are joined in front of the loop */
 RANGE_TEST(spill32_first_is_precise, "", 0, 0, 1, 0, 1,
 	   "if r7 == 0 goto +1; r6 = 2;" SPILL32_LOOP(SPILL32_INC "if w1 != 5 goto 1b;"),
-	   __flag(BPF_F_TEST_STATE_FREQ)
-	   __msg("header_count is [0..3] ") __msg("header_count is [0..5] "))
+	   __msg("header_count is [0..5] ") __not_msg("header_count is [0..3] "))
+/* The slot holds 64 bits when the loop starts. Nothing reads the high half: it's dead. */
+RANGE_TEST(spill32_first_is_64bit, "", 0, 0, 1, 5, 5,
+	   "*(u64 *)(r10 - 8) = r6; 1:" LIMIT_ITERATIONS SPILL32_INC "if w1 != w7 goto 1b;",
+	   SPILL32_COUNTED(5, "0..4"))
 /* The counter makes the address of a spill. Such a loop is not summarized, as with a register. */
 PARTIAL_SPILL_TEST(spill32_is_a_stack_offset, u32, 4,
 	"can't widen *fp-8, expr is (spill32 (+ *fp-8 8)), requires exact stack-offset tracking",
@@ -3224,11 +3200,7 @@ PARTIAL_SPILL_TEST(spill32_unknown_is_a_stack_offset, u32, 1000000,
 /* A step above S16_MAX has no range, as with a register */
 PARTIAL_SPILL_TEST(spill32_large_step, u32, 4, "can't widen *fp-8, expr is (spill32 ",
 	SPILL32_0, SPILL32_LOAD "w1 += 0x8000; *(u32 *)(r10 - 8) = r1;")
-/* No formula: the slot holds 64 bits when the loop starts, */
-RANGE_TEST(spill32_first_is_64bit, "", 0, 0, 1, 5, 5,
-	   "*(u64 *)(r10 - 8) = r6; 1:" LIMIT_ITERATIONS SPILL32_INC "if w1 != w7 goto 1b;",
-	   NO_COUNT)
-/* the load extends the sign, */
+/* No formula: the load extends the sign, */
 RANGE_TEST(spill32_sext_load, "", 0, 0, 1, 5, 5,
 	   SPILL32_LOOP(SPILL32_INC "r1 = *(s32 *)(r10 - 8); if r1 != r7 goto 1b;"), NO_COUNT)
 /* the sum is made from a different slot. */
@@ -4299,12 +4271,23 @@ exit_%=:						\
  *        } while (r8 < 3);
  *    }
  *    return 0;
+ *
+ * The outer loop gets may_goto in front of the jump that leaves it, the inner loop doesn't.
+ * At run time it's may_goto that ends the prog.
  */
 SEC("xdp")
-__failure
-__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("The sequence of 8193 jumps is too complex.")
+__success __retval(0)
+__xlated("call unknown")
+__xlated("...")
+__xlated("if r0 == 0x2a goto pc+4")
+__xlated("r8 = 0")
+__xlated("r8 += 1")
+__xlated("if r8 < 0x3 goto pc-2")
+__xlated("goto pc-{{[0-9]+}}")
+__xlated("r0 = 0")
+__xlated("exit")
+__xlated("r0 = 0")
+__xlated("exit")
 __naked void nested_terminating_prune(void)
 {
 	asm volatile ("					\
@@ -4679,7 +4662,7 @@ __naked void widen_pkt_pointer_negative_offset(void)
 
 /* The access is in the second iteration only: the verifier gets to it after the clamp */
 SEC("tc")
-__failure __msg("invalid access to packet, off=56 size=2, R6(id=3,off=56,r=56)")
+__failure __msg("invalid access to packet, off=56 size=2, R6(id={{[0-9]+}},off=56,r=56)")
 __naked void clamp_pkt_pointer_step_lowers_umax(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
index 8f6082fdb5c8..e7636b86cc29 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
@@ -127,13 +127,13 @@ static __u64 stats_async_schedule(struct bpf_map *map, __u32 *key,
 SEC("?raw_tp")
 __success __log_level(4)
 /*
- * self: 9 + 7 + 2 = 18
- * totals: leaf 2, scheduler 7, main root 18
+ * self: 8 + 7 + 2 = 17
+ * totals: leaf 2, scheduler 7, main root 17
  */
-__msg("subprog 0 (stats_async_direct) main insns_self 9 insns_total 18 stack 0")
+__msg("subprog 0 (stats_async_direct) main insns_self 8 insns_total 17 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_schedule) static insns_self 7 insns_total 7 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_leaf) static insns_self 2 insns_total 2 stack 0")
-__msg("processed 18 insns")
+__msg("processed 17 insns")
 __naked int stats_async_direct(void)
 {
 	asm volatile (
@@ -194,14 +194,14 @@ static __u64 stats_async_nested_schedule(struct bpf_map *map, __u32 *key,
 SEC("?raw_tp")
 __success __log_level(4)
 /*
- * self: 9 + 7 + 7 + 2 = 25
- * totals: leaf 2, outer 7, scheduler 7, main root 25
+ * self: 8 + 7 + 7 + 2 = 24
+ * totals: leaf 2, outer 7, scheduler 7, main root 24
  */
-__msg("subprog 0 (stats_async_nested) main insns_self 9 insns_total 25 stack 0")
+__msg("subprog 0 (stats_async_nested) main insns_self 8 insns_total 24 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_nested_schedule) static insns_self 7 insns_total 7 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_outer) static insns_self 7 insns_total 7 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_nested_leaf) static insns_self 2 insns_total 2 stack 0")
-__msg("processed 25 insns")
+__msg("processed 24 insns")
 __naked int stats_async_nested(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
index 42b90b692dc5..5ba41dca35ea 100644
--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
@@ -148,6 +148,33 @@ __naked void tail_call_caller_stack_ok(void)
 "	::: __clobber_all);
 }
 
+static __noinline int tail_calls(struct __sk_buff *skb)
+{
+	bpf_tail_call_static(skb, &jmp_table, 0);
+	return skb->len;
+}
+
+static __noinline int calls_itself(struct __sk_buff *skb, int n)
+{
+	int ret = 0;
+
+	if (n)
+		ret = calls_itself(skb, n - 1);
+	return ret + tail_calls(skb);
+}
+
+/* The frames of a subprog that calls itself stay on the stack too: up to a half of it. */
+SEC("tc")
+__description("tail call from subprog below a recursive subprog")
+__arch_x86_64
+__arch_arm64
+__failure
+__msg("tail_calls are not allowed when call stack of previous frames is {{[0-9]+}} bytes")
+int tail_call_below_recursion(struct __sk_buff *skb)
+{
+	return calls_itself(skb, 2);
+}
+
 SEC("tc")
 __description("tail call from subprog with 256 bytes of caller stack")
 __failure
diff --git a/tools/testing/selftests/bpf/progs/verifier_uninit.c b/tools/testing/selftests/bpf/progs/verifier_uninit.c
index 691018a46049..345223a4525b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_uninit.c
+++ b/tools/testing/selftests/bpf/progs/verifier_uninit.c
@@ -9,7 +9,7 @@
 SEC("socket")
 __description("read uninitialized register")
 __failure __msg("R2 !read_ok")
-__msg("R2 has never been initialized on this path")
+__msg("R2 is not readable here. It may never have been initialized")
 __failure_unpriv
 __naked void read_uninitialized_register(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
index b9d96366e656..87aeef7545a0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
@@ -394,8 +394,8 @@ __naked void fill_of_different_pointers_stx(void)
 	r2 = r10;					\
 	r2 += -16;					\
 	*(u64*)(r6 + 0) = r2;				\
-l0_%=:	if r1 != 0 goto l1_%=;				\
-	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
+l0_%=:	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	r1 = *(u64*)(r6 + 0);				\
 	*(u32*)(r1 + %[__sk_buff_mark]) = r3;		\
 	r0 = 0;						\
@@ -420,8 +420,8 @@ __naked void fill_of_different_pointers_st(void)
 	r2 = r10;					\
 	r2 += -16;					\
 	*(u64*)(r6 + 0) = r2;				\
-l0_%=:	if r1 != 0 goto l1_%=;				\
-	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
+l0_%=:	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	r1 = *(u64*)(r6 + 0);				\
 	.8byte %[st_mem];				\
 	r0 = 0;						\
@@ -435,7 +435,8 @@ l1_%=:	r1 = *(u64*)(r6 + 0);				\
 
 SEC("tc")
 __description("unpriv: spill/fill of different pointers stx - ctx and sock")
-__failure __msg("type=ctx expected=sock")
+/* bpf_sk_release(skb) is an error too: "type=ctx expected=sock". This one is found first. */
+__failure __msg("R1 invalid mem access 'sock_or_null'")
 __naked void pointers_stx_ctx_and_sock(void)
 {
 	asm volatile ("					\
@@ -449,10 +450,10 @@ __naked void pointers_stx_ctx_and_sock(void)
 	r6 += -8;					\
 	r1 = r8;					\
 	/* if (skb == NULL) *target = sock; */		\
-	if r1 == 0 goto l0_%=;				\
+	if r1 != 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = skb; */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	/* struct __sk_buff *skb = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
@@ -490,10 +491,10 @@ __naked void different_pointers_stx_leak_sock(void)
 	r6 += -8;					\
 	r1 = r8;					\
 	/* if (skb == NULL) *target = sock; */		\
-	if r1 == 0 goto l0_%=;				\
+	if r1 != 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = skb; */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	/* struct __sk_buff *skb = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
@@ -526,8 +527,8 @@ __naked void stx_sock_and_ctx_read(void)
 	/* if (skb) *target = skb */			\
 	if r1 == 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = sock */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
 l1_%=:	/* struct bpf_sock *sk = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
@@ -565,8 +566,8 @@ __naked void stx_sock_and_ctx_write(void)
 	/* if (skb) *target = skb */			\
 	if r1 == 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = sock */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
 l1_%=:	/* struct bpf_sock *sk = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
index 4d77407a0a79..5363093a4a5c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
@@ -17,8 +17,8 @@ __naked void fill_of_different_pointers_ldx(void)
 	r2 = r10;					\
 	r2 += %[__imm_0];				\
 	*(u64*)(r6 + 0) = r2;				\
-l0_%=:	if r1 != 0 goto l1_%=;				\
-	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
+l0_%=:	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	r1 = *(u64*)(r6 + 0);				\
 	r1 = *(u64*)(r1 + %[sample_period]);		\
 	r0 = 0;						\
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index f9d71e01ddf4..bb2b9e8c71df 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -455,8 +455,47 @@
 	BPF_EXIT_INSN(),
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
-	.errstr = "recursive call",
+	.errstr = "calls itself",
+	.result = REJECT,
+},
+{
+	"calls: a timer callback calls the main prog",
+	.insns = {
+	/* bpf_timer_set_callback(bpf_map_lookup_elem(&timer_map, &(int){0}), cb); */
+	BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+	BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+	BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+	BPF_LD_MAP_FD(BPF_REG_1, 0),
+	BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
+	BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4),
+	BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+	BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, BPF_REG_2, BPF_PSEUDO_FUNC, 0, 4),
+	BPF_RAW_INSN(0, 0, 0, 0, 0),
+	BPF_EMIT_CALL(BPF_FUNC_timer_set_callback),
+	BPF_MOV64_IMM(BPF_REG_0, 0),
+	BPF_EXIT_INSN(),
+	/* cb */
+	BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, -14),
+	BPF_MOV64_IMM(BPF_REG_0, 0),
+	BPF_EXIT_INSN(),
+	},
+	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+	.fixup_map_timer = { 3 },
+	.errstr = "recursive call of the main program",
 	.result = REJECT,
+	/* libbpf makes the main prog global, and then the arguments of the call do not match */
+	.func_info = { { 0, 4 /* main_prog */ }, { 13, 5 /* cb */ } },
+	.func_info_cnt = 2,
+	.btf_strings = "\0int\0ctx\0main_prog\0cb",
+	.btf_types = {
+	/* 1: int            */ BTF_TYPE_INT_ENC(1, BTF_INT_SIGNED, 0, 32, 4),
+	/* 2: void*          */ BTF_PTR_ENC(0),
+	/* 3: int __(void *) */ BTF_FUNC_PROTO_ENC(1, 1),
+				BTF_FUNC_PROTO_ARG_ENC(5, 2),
+	/* 4: main_prog      */ BTF_FUNC_ENC(9, 3),
+	/* 5: cb             */ BTF_FUNC_ENC(19, 3),
+	BTF_END_RAW
+	}
 },
 {
 	"calls: conditional call",
@@ -559,8 +598,7 @@
 	BPF_EXIT_INSN(),
 	},
 	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
-	.errstr = "infinite loop detected",
-	.result = REJECT,
+	.result = ACCEPT,
 },
 {
 	"calls: using r0 returned by callee",
@@ -812,7 +850,7 @@
 	BPF_EXIT_INSN(),
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
-	.errstr = "recursive call",
+	.errstr = "calls itself",
 	.result = REJECT,
 },
 {
-- 
2.55.0

Re: [PATCH bpf-next v3 40/45] selftests/bpf: Adjust tests to new verdicts, messages and order

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

> selftests/bpf: Adjust tests to new verdicts, messages and order

TL;DR: Nits only, in selftests: comments in iters.c left stale after the
prune-point and dfs_depth notes were removed, and a possibly unneeded
bpf_compiler.h include in verifier_scev.c.

> diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c
> --- a/tools/testing/selftests/bpf/progs/iters.c
> +++ b/tools/testing/selftests/bpf/progs/iters.c

[ ... ]

> @@ -2398,7 +2370,7 @@ __naked void iter_remake_two_jmp(void)
>  /* Same loops with a prune point in front of bpf_iter_num_next() */
>  SEC("socket")
>  __description("iter: remake in the loop")
> -__failure __msg("infinite loop detected")
> +__success __retval(0)
>  __flag(BPF_F_TEST_STATE_FREQ)
>  __naked void iter_remake_inline(void)
>  {

This isn't a bug, but the prune-point sentence above iter_remake_jmp() and
the dfs_depth note in the C comment of loop_state_deps3() are removed,
while "Same loops with a prune point in front of bpf_iter_num_next()" and
the asm comment "/* loop to increase dfs_depth */" are still there.

Should these be updated as well, so the comments say what still separates
the two groups of loops?

[ ... ]

> diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
> --- a/tools/testing/selftests/bpf/progs/verifier_scev.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
> @@ -8,6 +8,7 @@
>  #include "../../../include/linux/filter.h"
>  #include "bpf_misc.h"
>  #include "bpf_kfuncs.h"
> +#include "bpf_compiler.h"
>  
>  #define COND_TEST(name, _initial, _step, _bound, msg, body)                     \
>  SEC("xdp")                                                                      \

This isn't a bug, but is the new bpf_compiler.h include needed here?
Nothing in verifier_scev.c seems to use its macros.

Was it meant to replace the open-coded #pragma block in
no_widen_dynptr_kfunc_arg() with __pragma_loop_no_unroll, or can the
include be dropped?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42160076968

Re: [PATCH bpf-next v3 40/45] selftests/bpf: Adjust tests to new verdicts, messages and order

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] selftests/bpf: ineffective __not_msg() window in correlated_regs()
--

commit 02c8d77860efe8ec36bb7092420fc94e1e92c1cc
Author: Alexei Starovoitov <ast@kernel.org>

selftests/bpf: Adjust tests to new verdicts, messages and order

This commit updates BPF selftests to match the new verifier behavior for
fixpoint verification over the control flow graph, standalone static
subprogram verification, and recursive subprograms.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
> index 07c356fde68db..5e566933f7e44 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_scev.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_scev.c

[ ... ]

> @@ -2396,18 +2374,15 @@ __msg_next("  r1=(+ r1 2) / (linear (+ r1 2) 2)")
>  __msg("loop header at 11, widening r0")
>  __msg("loop header at 11, widening r1")
>  __msg("11: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=7,var_off=(0x0; 0x7)) R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=14,var_off=(0x0; 0xe),step=0+2)")
> -/* loop exit */
>  __msg("16: (a5) if r0 < 0x8 goto pc-6")
> -__msg("exiting loop 11")
> -__msg("17: (95) exit")
>  /* second iteration */
>  __msg("loop header at 11, clamping r0")
>  __msg("loop header at 11, clamping r1")
>  /* iteration convergence */
> -__msg("from 16 to 11: safe")
>  __not_msg("{{^}}11:")
> -/* map lookup error path */
> -__msg("from 7 to 17: safe")
> +/* loop exit */
> +__msg("exiting loop 11")
> +__msg("17: (95) exit")
>  __naked void correlated_regs(void)

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], in correlated_regs() in
tools/testing/selftests/bpf/progs/verifier_scev.c, does placing
__not_msg("{{^}}11:") before __msg("exiting loop 11") make the convergence
check a no-op?

In tools/testing/selftests/bpf/test_loader.c, match_negative_msgs() only
searches for the negative pattern between the end of the preceding positive
match ("loop header at 11, clamping r1") and the start of the next positive
match ("exiting loop 11").

In kernel/bpf/verifier.c, when do_check_block() processes conditional jump
instruction 16 (if r0 < 0x8 goto pc-6), check_cond_jmp_op() pushes the
branch target (instruction 11) onto env->head and advances env->insn_idx to
17. Still within the same do_check_block() call:

1. flow_drain() delivers the 16 -> 11 backedge via
bpf_flow_deliver() -> flow_add() -> bpf_flow_loop_enter() ->
maybe_clamp_scev_regs(), which emits "loop header at 11, clamping r0" and
"loop header at 11, clamping r1", followed by flow_merge() which either
logs "from 16 to 11: within" on convergence or queues instruction 11 in
flow->pending via flow_set_pending() without executing instruction 11.

2. Immediately after flow_drain() returns, do_check_block() checks
bpf_flow_is_point(env, 17) and calls bpf_flow_deliver(env, state, 17, 16),
where flow_arrive() -> bpf_flow_loop_leave() -> loop_stack_pop() emits
"exiting loop 11" before returning to do_check().

3. Only after do_check_block() returns does do_check() call
bpf_flow_next() to dequeue the next pending block. Because flow->pos orders
instruction 11 before instruction 17, a non-converged state at instruction
11 would be logged after "exiting loop 11" and before "17: (95) exit",
outside the span checked by __not_msg("{{^}}11:").

Should __not_msg("{{^}}11:") be moved after __msg("exiting loop 11")
(between "exiting loop 11" and "17: (95) exit"), or should this match
__msg("from 16 to 11: within") as done in clamp_dependent_overshoot() and
clamp_callee_state_pruning() in the same file?

[1] https://lore.kernel.org/r/53c16606b21ca4743d6f2dcc8f31619c9e070d7ca0a729fcb0ae723bb1c04ee5@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=40

[PATCH bpf-next v3 41/45] selftests/bpf: Test states that are not joined

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  ret = lookup(&p);      /* ret = -ENOENT and p is not set, or */
  if (ret < 0)           /* ret = 0 and p is set               */
          goto drop;
  use(p);

When the compiler inlines lookup(), the two states meet in front of the
jump. A verifier that follows paths keeps them apart and loads the prog.
Their join, ret = [-2, 0], gets to use(p) without p. Thus the verifier does
not join states when a check ahead can depend on the difference. Each test
has one such difference, in the form that a real prog has.

  [J] "bpf: Add join of registers and stack slots"
  [K] "bpf: Find the registers that hold a number or a pointer to arena"
  [A] "bpf: Find the registers and stack slots that later checks depend on"
  [N] "bpf: Decide which states are not joined"
  [F] "bpf: Verify programs as a fixpoint over the control flow graph"
  [C] "bpf: Don't take what bpf_core_cast() returns for a number"
  [U] "bpf: Don't widen away a register that only the loop sets"
  [X] "bpf: A base that nothing is known about can be the context"

  verifier_..     the two states differ in                   as in
  --------------  -----------------------------------------  --------------
  jumps_ahead.c   [N] a number that a jump ahead tests, the  Cilium,
                  ranges have a gap: -155 and 0              profiler,
                  [N] two such numbers that go together      test_seg6_loop
                  [N] a register that only one state sets
                  [N] 0 or not 0, in one of two slots        xdp_synproxy
                  [N] a flag, 0 and 1, that a jump in this
                  block or the next one tests
                  [U] a register that only a loop sets
  ptr_off.c       [A] a constant that is added to a pointer
                  [J] how much of the packet is checked
                  [C] the offset of one of two fields behind
                  bpf_core_cast(), bpf_cast_to_kern_ctx()
                  [X] 20 or 40 bytes of a packet whose       test_cls_
                  context was in a struct                    redirect
  var_off.c       [A] a constant offset in the stack
  mem_size_reg.c  [A] a size that a helper or a kfunc must
                  know

States that stay multiply. The verifier must join what no check can tell
apart: -134 and -135 ([N]), NULL or not in an array that a loop reads
(bpfjailer), an offset in a map value that has only numbers ([K]). 20 flags
and 12 offsets give 2^20 and 2^12 states: the rules apply only while less
than half of each limit on states is used ([N]). 4 calls that each return
one of 33 numbers give 33^4: bpf_flow_exit() widens an exit state that was
joined 16 times.

[F]: maybe_fork_scalars() makes -1 and 0 of [-1, 0]. The state with 0 must
not go back to an insn where [-1, 0] covers it (verifier_bounds.c). 1 and 2
are joined at exit: a prog that can return 2 must not attach to ingress
(cgroup_skb_egress_ret.c).

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../bpf/prog_tests/cgroup_skb_egress_ret.c    |  23 +
 .../selftests/bpf/prog_tests/verifier.c       |   4 +
 .../bpf/progs/cgroup_skb_egress_ret.c         |  12 +
 .../selftests/bpf/progs/verifier_bounds.c     | 108 +++
 .../bpf/progs/verifier_jumps_ahead.c          | 670 ++++++++++++++++++
 .../bpf/progs/verifier_mem_size_reg.c         |  48 ++
 .../selftests/bpf/progs/verifier_ptr_off.c    | 279 ++++++++
 .../selftests/bpf/progs/verifier_var_off.c    | 203 ++++++
 8 files changed, 1347 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_ptr_off.c

diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c b/tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
new file mode 100644
index 000000000000..e261caac35e6
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "cgroup_helpers.h"
+#include "cgroup_skb_egress_ret.skel.h"
+
+void test_cgroup_skb_egress_ret(void)
+{
+	struct cgroup_skb_egress_ret *skel;
+	int cg, fd;
+
+	cg = test__join_cgroup("/cgroup_skb_egress_ret");
+	if (!ASSERT_OK_FD(cg, "join_cgroup"))
+		return;
+	skel = cgroup_skb_egress_ret__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "open_and_load"))
+		goto out;
+	fd = bpf_program__fd(skel->progs.one_or_two);
+	ASSERT_EQ(bpf_prog_attach(fd, cg, BPF_CGROUP_INET_INGRESS, 0), -EINVAL, "ingress");
+	ASSERT_OK(bpf_prog_attach(fd, cg, BPF_CGROUP_INET_EGRESS, 0), "egress");
+	cgroup_skb_egress_ret__destroy(skel);
+out:
+	close(cg);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index d41339ef00d8..2f34a34e3d54 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -69,6 +69,7 @@
 #include "verifier_iterating_callbacks.skel.h"
 #include "verifier_jeq_infer_not_null.skel.h"
 #include "verifier_jit_convergence.skel.h"
+#include "verifier_jumps_ahead.skel.h"
 #include "verifier_kfunc_packet_access.skel.h"
 #include "verifier_kfunc_uninit.skel.h"
 #include "verifier_kfunc_uninit_multi.skel.h"
@@ -107,6 +108,7 @@
 #include "verifier_precision.skel.h"
 #include "verifier_prevent_map_lookup.skel.h"
 #include "verifier_private_stack.skel.h"
+#include "verifier_ptr_off.skel.h"
 #include "verifier_ptr_to_buf.skel.h"
 #include "verifier_raw_stack.skel.h"
 #include "verifier_raw_tp_writable.skel.h"
@@ -267,6 +269,7 @@ void test_verifier_int_ptr(void)              { RUN(verifier_int_ptr); }
 void test_verifier_iterating_callbacks(void)  { RUN(verifier_iterating_callbacks); }
 void test_verifier_jeq_infer_not_null(void)   { RUN(verifier_jeq_infer_not_null); }
 void test_verifier_jit_convergence(void)      { RUN(verifier_jit_convergence); }
+void test_verifier_jumps_ahead(void)          { RUN(verifier_jumps_ahead); }
 void test_verifier_kfunc_packet_access(void)  { RUN_TESTS(verifier_kfunc_packet_access); }
 void test_verifier_kfunc_uninit(void)         { RUN_TESTS(verifier_kfunc_uninit); }
 void test_verifier_kfunc_uninit_multi(void)   { RUN_TESTS(verifier_kfunc_uninit_multi); }
@@ -317,6 +320,7 @@ void test_verifier_percpu_addr(void)          { RUN(verifier_percpu_addr); }
 void test_verifier_precision(void)            { RUN(verifier_precision); }
 void test_verifier_prevent_map_lookup(void)   { RUN(verifier_prevent_map_lookup); }
 void test_verifier_private_stack(void)        { RUN(verifier_private_stack); }
+void test_verifier_ptr_off(void)              { RUN(verifier_ptr_off); }
 void test_verifier_ptr_to_buf(void)           { RUN(verifier_ptr_to_buf); }
 void test_verifier_raw_stack(void)            { RUN(verifier_raw_stack); }
 void test_verifier_raw_tp_writable(void)      { RUN(verifier_raw_tp_writable); }
diff --git a/tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c b/tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
new file mode 100644
index 000000000000..107f61d77538
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+/* Only a prog at egress returns 2. At exit the verifier has 1 and 2 in one state. */
+SEC("cgroup_skb/egress")
+int one_or_two(struct __sk_buff *skb)
+{
+	return skb->len > 100 ? 2 : 1;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c
index 6fede778971c..28d48e5fbc4f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c
@@ -2159,6 +2159,74 @@ l0_%=:	r0 = 0;						\
 	: __clobber_all);
 }
 
+/*
+ * -1 and 0 meet at the insn that forks. The state with [-1, 0] is kept there. The
+ * pushed state is within it, so it's not there that it is pushed to.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: AND where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_at_join_rejects_oob(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u64*)(r10 - 8) = r1;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map_hash_8b] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l0_%=;				\
+	r9 = r0;					\
+	r1 = *(u64*)(r9 + 0);				\
+	r6 = -1;					\
+	if r1 == 0 goto l1_%=;				\
+	r6 = 0;						\
+l1_%=:	r6 &= 8;					\
+	/* r6 is 8 (current) or 0 (pushed) */		\
+	r6 ^= 8;					\
+	/* pushed path: r6 = 8, OOB for value_size=8 */	\
+	r9 += r6;					\
+	r0 = *(u8*)(r9 + 0);				\
+l0_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: 32-bit AND of a 32-bit range where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and32_scalar32_fork_at_join_rejects_oob(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u64*)(r10 - 8) = r1;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map_hash_8b] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l0_%=;				\
+	r9 = r0;					\
+	r1 = *(u64*)(r9 + 0);				\
+	w6 = -1;					\
+	if r1 == 0 goto l1_%=;				\
+	r6 = 0;						\
+l1_%=:	w6 &= 8;					\
+	/* r6 is 8 (current) or 0 (pushed) */		\
+	r6 ^= 8;					\
+	/* pushed path: r6 = 8, OOB for value_size=8 */	\
+	r9 += r6;					\
+	r0 = *(u8*)(r9 + 0);				\
+l0_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
 /*
  * The insn that forks is the head of a loop that scalar evolution counted. The state with
  * [-1, 0] is kept there, and the state with 0 that is split off is within it.
@@ -2200,6 +2268,46 @@ l0_%=:	r0 = 0;						\
 	: __clobber_all);
 }
 
+#define FORK						\
+	"r6 = *(u64*)(r9 + 0);"				\
+	"r6 s>>= 63;"					\
+	"r6 &= 4;"					\
+	"*(u64*)(r9 + 0) = r6;"
+#define FORK4 FORK FORK FORK FORK
+
+/*
+ * The pushed states of one block that don't differ in what is live are one state: 16 forks
+ * are not 2^16 states.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: 16 forks in a block")
+__success __retval(0)
+__log_level(4)
+__msg("processed {{[0-9][0-9]?[0-9]?}} insns")
+__naked void and_scalar_16_forks_in_a_block(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64*)(r10 - 8) = r1;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"r1 = %[map_hash_8b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto l0_%=;"
+	"r9 = r0;"
+	FORK4 FORK4 FORK4 FORK4
+"l0_%=:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
+#undef FORK4
+#undef FORK
+
 /*
  * Last jump can be detected as always taken because the intersection of R5 and
  * R7 32bit tnums produces a constant that isn't within R7's s32 bounds.
diff --git a/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c b/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
new file mode 100644
index 000000000000..66d8a7c4a6a0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
@@ -0,0 +1,670 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/*
+ * A join must not make up what neither state has where a jump ahead can see it:
+ *	ret = lookup(&p);
+ *	if (ret < 0)
+ *		goto drop;
+ *	use(p);
+ * When lookup() is inlined (ret = -ENOENT, p is not set) and (ret = 0, p is set) meet in front of
+ * the jump.
+ */
+
+SEC("socket")
+__description("jumps ahead: a register is set when ret is 0")
+__success __retval(0)
+__naked void ret_and_register(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = -2;"
+		"if r0 == 0 goto 1f;"
+		"r8 = 0;"
+		"w6 = 0;"
+	"1:"
+		"w7 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r0 = r8;"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("jumps ahead: a register is not set and ret doesn't tell")
+__failure __msg("R8 !read_ok")
+__naked void ret_does_not_tell(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = 1;"
+		"if r0 == 0 goto 1f;"
+		"r8 = 0;"
+		"w6 = 0;"
+	"1:"
+		"w7 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r0 = r8;"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ * As in bpf_host.o of cilium:
+ *	err = helper();
+ *	ret = err < 0 ? -155 : 0;
+ *	if (err < 0)
+ *		goto out;
+ *	...
+ * out:
+ *	if (ret < 0)
+ *		goto drop;
+ * The states differ in what two jumps look at, and in nothing else.
+ * (err < 0, ret = 0) is neither of them.
+ */
+SEC("socket")
+__description("jumps ahead: two numbers that go together")
+__success
+__naked void two_numbers(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = 0;"
+		"if w0 s> -1 goto 1f;"
+		"w6 = -155;"
+	"1:"
+		"w7 = 0;"
+		"if w0 s< 0 goto 2f;"
+		"r0 = 0;"
+		"exit;"
+	"2:"
+		"if w6 s< 0 goto 3f;"
+		"r0 = r8;"
+	"3:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/* -134 and -135 go the same way */
+SEC("socket")
+__description("jumps ahead: states that go the same way are joined")
+__success __log_level(2)
+__msg("from 4 to 7: joined")
+__msg("7: R6=scalar({{.*}}smin32=-135,smax32=-134{{.*}}) R10=fp0")
+__msg("from 6 to 7: R6=0 R8=0 R10=fp0")
+__naked void same_way(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = -134;"
+		"if r0 == 1 goto 1f;"
+		"w6 = -135;"
+		"if r0 == 2 goto 1f;"
+		"r8 = 0;"
+		"w6 = 0;"
+	"1:"
+		"w7 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r0 = r8;"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ *	ret = idx < 8 ? 0 : -EINVAL;
+ *	if (ret < 0)
+ *		goto drop;
+ *	use(buf[idx]);
+ * 0 and -EINVAL leave a gap, so the states are not joined, and idx keeps its range.
+ */
+SEC("socket")
+__description("jumps ahead: ret vouches for the range of another number")
+__success
+__naked void ret_and_range(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"call %[bpf_get_prandom_u32];"
+		"r7 = r0;"
+		"w6 = 0;"
+		"if r7 < 8 goto 1f;"
+		"w6 = -22;"
+	"1:"
+		"w8 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r1 += r7;"
+		"r0 = *(u8 *)(r1 + 0);"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ *	ok = i < 8 && x == 2;
+ *	if (x == 7)
+ *		n++;
+ *	if (ok)
+ *		use(buf[i]);
+ * A compiler keeps ok in a register: "w6 &= 1; if w6 == 0". (ok = 0, i >= 8), (ok = 1, i < 8) and
+ * (ok = 0, i < 8) meet in front of "if (x == 7)". 0 and 1 leave no gap, and i < 8 comes with
+ * either. They are not joined when the jump that tells 0 from 1 ends this basic block or the next.
+ */
+#define FLAG_SET(max, no, yes)			\
+	"r1 = 0;"				\
+	"*(u64 *)(r10 - 8) = r1;"		\
+	"call %[bpf_get_prandom_u32];"		\
+	"r7 = r0;"				\
+	"call %[bpf_get_prandom_u32];"		\
+	"w6 = " #no ";"				\
+	"if r7 > " #max " goto +3;"		\
+	"w6 = " #yes ";"			\
+	"if r0 == 2 goto +1;"			\
+	"w6 = " #no ";"
+#define FLAG_USE(unless)				\
+	"if " #unless " goto +4;"		\
+	"r1 = r10;"				\
+	"r1 += -8;"				\
+	"r1 += r7;"				\
+	"r0 = *(u8 *)(r1 + 0);"
+#define FLAG_PROG(name, max, no, yes, rest)	\
+__naked void name(void)				\
+{						\
+	asm volatile (				\
+	FLAG_SET(max, no, yes)			\
+	rest					\
+	"exit;"					\
+	:: __imm(bpf_get_prandom_u32) : __clobber_all); \
+}
+
+SEC("socket")
+__description("jumps ahead: a flag, another jump, the test of the flag")
+__success
+FLAG_PROG(flag_next_block, 7, 0, 1, "if r0 != 7 goto +1; r0 = 0; w6 &= 1;" FLAG_USE(w6 == 0))
+
+SEC("socket")
+__description("jumps ahead: a flag that says too much")
+__failure __msg("invalid variable-offset read from stack R1")
+FLAG_PROG(flag_next_block_bad, 8, 0, 1, "if r0 != 7 goto +1; r0 = 0; w6 &= 1;" FLAG_USE(w6 == 0))
+
+/* "f = i < 8 ? in[i] : 0; ... if (f) use(buf[i]);" A compiler makes a flag of "f == 0" */
+SEC("socket")
+__description("jumps ahead: a flag that is 1 for no")
+__success
+FLAG_PROG(flag_for_no, 7, 1, 0, "if r0 != 7 goto +1; r0 = 0; w6 &= 1;" FLAG_USE(w6 != 0))
+
+SEC("socket")
+__description("jumps ahead: a flag is tested twice")
+__success
+FLAG_PROG(flag_twice, 7, 0, 1, FLAG_USE(w6 == 0) "r0 = 0;" FLAG_USE(w6 == 0))
+
+SEC("socket")
+__description("jumps ahead: the other value of a flag that is tested twice")
+__failure __msg("math between fp pointer and register with unbounded min value")
+FLAG_PROG(flag_twice_bad, 7, 0, 1, FLAG_USE(w6 == 0) "w6 ^= 1;" FLAG_USE(w6 == 0))
+
+SEC("socket")
+__description("jumps ahead: a flag is spilled, filled, copied and extended")
+__success
+FLAG_PROG(flag_moves, 7, 0, 1, "*(u64 *)(r10 - 16) = r6; r1 = *(u64 *)(r10 - 16);"
+			       "r2 = r1; r2 <<= 32; r2 >>= 32;" FLAG_USE(r2 == 0))
+
+/* if and else: two ends of basic blocks on every way to the test */
+SEC("socket")
+__description("jumps ahead: the test of a flag is too far")
+__success __log_level(2)
+__msg("from 8 to 10: joined")
+FLAG_PROG(flag_too_far, 7, 0, 1, "if r0 != 7 goto +2; r0 = 0; goto +1; r0 = 1;"
+				 "if w6 == 0 goto +1; r0 = 2;")
+
+SEC("socket")
+__description("jumps ahead: the jump tests another register than the flag")
+__success __log_level(2)
+__msg("from 8 to 10: joined")
+FLAG_PROG(flag_not_tested, 7, 0, 1, "if r0 == 1 goto +1; r0 = r6;")
+
+/* the first values of a counter */
+SEC("socket")
+__description("jumps ahead: 1 and 2 are no flag")
+__success __log_level(2)
+__msg("from 8 to 10: joined")
+FLAG_PROG(flag_1_and_2, 7, 1, 2, "if w6 == 1 goto +1; r0 = 0;")
+
+/*
+ * A register that is set in one state and not in the other: these states are not joined,
+ * whatever it is that tells them apart later.
+ */
+SEC("socket")
+__description("set or not: a register is set when a flag is 1, flag + 4 is tested")
+__success
+__naked void flag_add(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r6 = 0;"
+	"if r0 == 0 goto 1f;"
+	"r8 = 0;"
+	"r6 = 1;"
+"1:"
+	"r1 = r6;"
+	"r1 += 4;"
+	"if r1 != 5 goto 2f;"
+	"r0 = r8;"
+	"exit;"
+"2:"
+	"r0 = 0;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* The first iteration sets r7 and the flag. The states meet at the head of the loop. */
+SEC("socket")
+__description("set or not: a loop sets a register and a flag")
+__success
+__naked void set_in_a_loop(void)
+{
+	asm volatile (
+	"r6 = 0;"
+"1:"
+	".8byte %[may_goto];"
+	"if r6 == 0 goto 2f;"
+	"r8 = r7;"
+"2:"
+	"call %[bpf_get_prandom_u32];"
+	"r7 = r0;"
+	"r6 = 1;"
+	"goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 6, 0))
+	: __clobber_all);
+}
+
+/*
+ * 20 flags, 0 or 5, all are looked at later: 2^20 ways, and 0 and 5 leave a gap.
+ * States are kept apart while there is room. After that they are joined.
+ */
+#define PICK_FLAG(slot)					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r1 = 0;"					\
+	"if r0 == 0 goto +1;"				\
+	"r1 = 5;"					\
+	"*(u64*)(r10 - " #slot ") = r1;"
+#define USE_FLAG(slot)					\
+	"r1 = *(u64*)(r10 - " #slot ");"		\
+	"if r1 == 0 goto +1;"				\
+	"r6 += 1;"
+
+SEC("socket")
+__description("jumps ahead: 20 flags that leave a gap")
+__success
+__naked void many_flags_gap(void)
+{
+	asm volatile (
+	"r6 = 0;"
+	PICK_FLAG(8) PICK_FLAG(16) PICK_FLAG(24) PICK_FLAG(32) PICK_FLAG(40) PICK_FLAG(48)
+	PICK_FLAG(56) PICK_FLAG(64) PICK_FLAG(72) PICK_FLAG(80) PICK_FLAG(88) PICK_FLAG(96)
+	PICK_FLAG(104) PICK_FLAG(112) PICK_FLAG(120) PICK_FLAG(128) PICK_FLAG(136) PICK_FLAG(144)
+	PICK_FLAG(152) PICK_FLAG(160)
+	USE_FLAG(8) USE_FLAG(16) USE_FLAG(24) USE_FLAG(32) USE_FLAG(40) USE_FLAG(48)
+	USE_FLAG(56) USE_FLAG(64) USE_FLAG(72) USE_FLAG(80) USE_FLAG(88) USE_FLAG(96)
+	USE_FLAG(104) USE_FLAG(112) USE_FLAG(120) USE_FLAG(128) USE_FLAG(136) USE_FLAG(144)
+	USE_FLAG(152) USE_FLAG(160)
+	"r0 = r6;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* 33 numbers, each with an exit of its own */
+static __naked __noinline __used int ret_exits(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = r0;"
+	"if r1 != 0 goto +2; r0 = 0; exit;"
+	"if r1 != 1 goto +2; r0 = 3; exit;"
+	"if r1 != 2 goto +2; r0 = 6; exit;"
+	"if r1 != 3 goto +2; r0 = 9; exit;"
+	"if r1 != 4 goto +2; r0 = 12; exit;"
+	"if r1 != 5 goto +2; r0 = 15; exit;"
+	"if r1 != 6 goto +2; r0 = 18; exit;"
+	"if r1 != 7 goto +2; r0 = 21; exit;"
+	"if r1 != 8 goto +2; r0 = 24; exit;"
+	"if r1 != 9 goto +2; r0 = 27; exit;"
+	"if r1 != 10 goto +2; r0 = 30; exit;"
+	"if r1 != 11 goto +2; r0 = 33; exit;"
+	"if r1 != 12 goto +2; r0 = 36; exit;"
+	"if r1 != 13 goto +2; r0 = 39; exit;"
+	"if r1 != 14 goto +2; r0 = 42; exit;"
+	"if r1 != 15 goto +2; r0 = 45; exit;"
+	"if r1 != 16 goto +2; r0 = 48; exit;"
+	"if r1 != 17 goto +2; r0 = 51; exit;"
+	"if r1 != 18 goto +2; r0 = 54; exit;"
+	"if r1 != 19 goto +2; r0 = 57; exit;"
+	"if r1 != 20 goto +2; r0 = 60; exit;"
+	"if r1 != 21 goto +2; r0 = 63; exit;"
+	"if r1 != 22 goto +2; r0 = 66; exit;"
+	"if r1 != 23 goto +2; r0 = 69; exit;"
+	"if r1 != 24 goto +2; r0 = 72; exit;"
+	"if r1 != 25 goto +2; r0 = 75; exit;"
+	"if r1 != 26 goto +2; r0 = 78; exit;"
+	"if r1 != 27 goto +2; r0 = 81; exit;"
+	"if r1 != 28 goto +2; r0 = 84; exit;"
+	"if r1 != 29 goto +2; r0 = 87; exit;"
+	"if r1 != 30 goto +2; r0 = 90; exit;"
+	"if r1 != 31 goto +2; r0 = 93; exit;"
+	"r0 = 99;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* 33^4 ways: not all of them are looked at */
+SEC("socket")
+__description("jumps ahead: one of 33 numbers out of a function, 4 times")
+__success
+__naked void rets_33_4_times(void)
+{
+	asm volatile (
+	"call ret_exits; r6 = r0;"
+	"call ret_exits; r7 = r0;"
+	"call ret_exits; r8 = r0;"
+	"call ret_exits; r9 = r0;"
+	"r0 = 0;"
+	"if r6 != 3 goto +1; r0 += 1;"
+	"if r7 != 6 goto +1; r0 += 1;"
+	"if r8 != 9 goto +1; r0 += 1;"
+	"if r9 != 12 goto +1; r0 += 1;"
+	"exit;"
+	::: __clobber_all);
+}
+
+#define PICK_ANY(slot)					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r0 <<= 32;"					\
+	"r7 = r0;"					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r0 |= r7;"					\
+	"*(u64*)(r10 - " #slot ") = r0;"
+
+/*
+ * As in bpfj_jailer_free() of bpfjailer:
+ *	for (i = 0; i < 8; i++) {
+ *		p = a[i];
+ *		if (!p)
+ *			continue;
+ *		put(p);
+ *	}
+ * The jump finds out about a[i] too, which nothing looks at again. All of a[] is live while
+ * the loop goes on, because it is read through a pointer that moves. The states have to be
+ * joined behind the jump, or there are 2^8 of them. They are, because a slot that is filled
+ * through another register than r10 doesn't count as one that a jump ahead looks at.
+ */
+SEC("socket")
+__description("0 or not: an array on the stack that a loop goes through")
+__success __log_level(4)
+__msg("max_states_per_insn {{[0-9]}} total_states")
+__naked void array_of_nulls(void)
+{
+	asm volatile (
+	"r6 = 0;"
+	PICK_ANY(8) PICK_ANY(16) PICK_ANY(24) PICK_ANY(32)
+	PICK_ANY(40) PICK_ANY(48) PICK_ANY(56) PICK_ANY(64)
+	"r8 = r10;"
+	"r8 += -64;"
+	"r7 = 0;"
+"1:"
+	"r1 = *(u64*)(r8 + 0);"
+	"if r1 == 0 goto 2f;"
+	"r6 += 1;"
+"2:"
+	"r8 += 8;"
+	"r7 += 1;"
+	"if r7 < 8 goto 1b;"
+	"r0 = r6;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * As in xdp_synproxy_kern.c: hdr->ipv4 or hdr->ipv6 is there, the other one is 0.
+ *	if (hdr->ipv4) ... else if (hdr->ipv6) ...
+ * (0, x) and (y, 0) are not joined: (?, ?) has both, and r9 is not set.
+ */
+SEC("socket")
+__description("0 or not: one of two things is there")
+__success
+__naked void one_of_two(void)
+{
+	asm volatile (
+	PICK_ANY(8)
+	"r6 = r0;"
+	"r7 = 0;"
+	"call %[bpf_get_prandom_u32];"
+	"if r0 == 0 goto 1f;"
+	"r7 = r6;"
+	"r6 = 0;"
+"1:"
+	"r0 = 0;"
+	"if r6 == 0 goto 2f;"
+	"if r7 == 0 goto 2f;"
+	"r0 = r9;"
+"2:"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * As in bpf_sock.o of cilium:
+ *	err = xlate();
+ *	if (err == -EHOSTUNREACH || err == -ENOMEM) {
+ *		bpf_set_retval(err);
+ *		return 0;
+ *	}
+ *	return 1;
+ * A compiler computes the condition where err is set. 0 and 1 touch, and it's not a jump that
+ * looks at err.
+ */
+SEC("cgroup/connect4")
+__description("jumps ahead: a flag goes together with what bpf_set_retval() takes")
+__success
+__naked void flag_and_retval(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w8 = 0;"
+		"w1 = 1;"
+		"if r0 == 0 goto 1f;"
+		"w8 = 1;"
+		"w1 = -113;"
+	"1:"
+		"w0 = 1;"
+		"if w8 == 0 goto 2f;"
+		"call %[bpf_set_retval];"
+		"w0 = 0;"
+	"2:"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32), __imm(bpf_set_retval)
+		: __clobber_all
+	);
+}
+
+char buf[8];
+
+/* As in test_seg6_loop.c without 32-bit insns: the copy is tested, the original is used */
+SEC("socket")
+__description("jumps ahead: the jump looks at a copy that two shifts extend")
+__success
+__naked void extended_copy(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"r8 = 0;"
+		"if r0 == 0 goto 1f;"
+		"r8 = 5;"
+	"1:"
+		"r7 = r8;"
+		"r7 <<= 32;"
+		"r7 >>= 32;"
+		"if r7 == 0 goto 2f;"
+		"r8 += -5;"
+		"r1 = %[buf] ll;"
+		"r1 += r8;"
+		"r0 = *(u8 *)(r1 + 0);"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32), __imm_addr(buf)
+		: __clobber_all
+	);
+}
+
+static __naked __noinline __used __u64 index_or_minus_1(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"r1 = r0;"
+		"r0 = 0xffffffff ll;"
+		"if r1 > 7 goto 1f;"
+		"r0 = r1;"
+	"1:"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ * As in profiler.inc.h without 32-bit insns:
+ *	int index = find();
+ *	if (index == -1)
+ *		return 0;
+ *	use(array[index]);
+ * The numbers meet at the exit of find(). -1 doesn't fit in the jump, it is in a register.
+ */
+SEC("socket")
+__description("jumps ahead: the caller looks at what the function returns")
+__success
+__naked void caller_looks(void)
+{
+	asm volatile (
+		"call index_or_minus_1;"
+		"r0 <<= 32;"
+		"r1 = r0;"
+		"r1 >>= 32;"
+		"r2 = 0xffffffff ll;"
+		"if r1 == r2 goto 1f;"
+		"r0 s>>= 32;"
+		"r1 = %[buf] ll;"
+		"r1 += r0;"
+		"r0 = *(u8 *)(r1 + 0);"
+		"exit;"
+	"1:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(buf)
+		: __clobber_all
+	);
+}
+
+/*
+ * A state that arrives is compared with the states that it may be related to. Its shape tells:
+ * the same pointers in the same places at the same constant offsets. These states are related
+ * and don't look alike. They have to find each other.
+ */
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, char[48]);
+} shape_map SEC(".maps");
+
+/*
+ * 16 states with R9 at 16 places of the stack get to "1:". With that many states at an insn
+ * their shapes are looked at. Then comes one more, which @last makes of the 16th. The two are
+ * related and don't have the same shape.
+ */
+#define SHAPE_16_PLACES	"8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128"
+#define SHAPE_MANY(first, last, use)					\
+	asm volatile (							\
+		first							\
+		"r1 = 0;"						\
+		".irp off," SHAPE_16_PLACES ";"				\
+		"*(u64 *)(r10 - \\off) = r1;"				\
+		".endr;"						\
+		"call %[bpf_get_prandom_u32];"				\
+		"r6 = r0;"						\
+		"call %[bpf_get_prandom_u32];"				\
+		"r7 = r0;"						\
+		"r7 &= 1;"						\
+		".irp off," SHAPE_16_PLACES ";"				\
+		"r9 = r10;"						\
+		"r9 += -\\off;"						\
+		"if r6 == \\off goto 1f;"				\
+		".endr;"						\
+		last							\
+	"1:"								\
+		use							\
+		"r0 = *(u64 *)(r9 + 0);"				\
+	"2:"								\
+		"r0 = 0;"						\
+		"exit;"							\
+		:							\
+		: __imm(bpf_get_prandom_u32), __imm(bpf_map_lookup_elem),	\
+		  __imm_addr(shape_map),				\
+		  __imm_const(data, offsetof(struct __sk_buff, data)),	\
+		  __imm_const(data_end, offsetof(struct __sk_buff, data_end))	\
+		: __clobber_all						\
+	)
+
+SEC("socket")
+__description("shape: nothing in a register stands for a pointer")
+__success __log_level(2)
+__msg("processed 135 insns")
+__naked void shape_unset(void)
+{
+	SHAPE_MANY("",
+		   "r8 = r10;",
+		   "if r7 < 2 goto +1;"
+		   "r0 = *(u64 *)(r8 - 8);");
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
index aba9510a1c08..5a4a29b30b7a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
@@ -2,11 +2,59 @@
 
 #include <vmlinux.h>
 #include <bpf/bpf_helpers.h>
+#include <bpf/bpf_endian.h>
 #include "bpf_misc.h"
 #include "bpf_kfuncs.h"
 
 char _license[] SEC("license") = "GPL";
 
+/*
+ * A size that has to be a known constant is one of two constants, and a compiler makes one call
+ * of the two. The constants must not be joined on the way to the call.
+ */
+SEC("?tc")
+__description("one of two constants: __szk, the header of IPv4 or of IPv6")
+__success
+int dynptr_slice_two_sizes(struct __sk_buff *skb)
+{
+	char buf[sizeof(struct ipv6hdr)];
+	__u32 sz = sizeof(struct iphdr);
+	struct bpf_dynptr dptr;
+	__u8 *p;
+
+	bpf_dynptr_from_skb(skb, 0, &dptr);
+
+	if (skb->protocol != bpf_htons(0x0800))
+		sz = sizeof(struct ipv6hdr);
+	p = bpf_dynptr_slice(&dptr, 14, buf, sz);
+	if (p)
+		return p[0] >> 4;
+	return 0;
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+SEC("?tc")
+__description("one of two constants: the size of bpf_ringbuf_reserve()")
+__success
+int ringbuf_reserve_two_sizes(struct __sk_buff *skb)
+{
+	__u32 sz = 8;
+	char *p;
+
+	if (skb->len > 100)
+		sz = 16;
+	p = bpf_ringbuf_reserve(&ringbuf, sz, 0);
+	if (!p)
+		return 0;
+	p[7] = 1;
+	bpf_ringbuf_discard(p, 0);
+	return 0;
+}
+
 /* The __szk size of a kfunc memory/size pair when the nullable buffer is passed as NULL */
 SEC("?tc")
 __success
diff --git a/tools/testing/selftests/bpf/progs/verifier_ptr_off.c b/tools/testing/selftests/bpf/progs/verifier_ptr_off.c
new file mode 100644
index 000000000000..b9a26bc0066a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_ptr_off.c
@@ -0,0 +1,279 @@
+// SPDX-License-Identifier: GPL-2.0
+/* One of two constants is added to a pointer. A compiler makes one add and one load of it. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_misc.h"
+#include "bpf_kfuncs.h"
+
+struct with_timer {
+	int before[4];
+	struct bpf_timer timer;
+	int after[4];
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct with_timer);
+} timers SEC(".maps");
+
+__u64 in_bss[8];
+
+#define __imm_pkt								\
+	__imm_const(data, offsetof(struct __sk_buff, data)),			\
+	__imm_const(data_end, offsetof(struct __sk_buff, data_end))
+
+/*
+ *	len = v6 ? 40 : 20;
+ *	if (data + len > data_end) return 0;
+ *	if (v6) use(data[39]);
+ * 1 and 2 touch, so nothing else keeps the states apart.
+ */
+SEC("tc")
+__description("offset of a pointer: the length of a header goes with its kind")
+__success __retval(0)
+__naked void pkt_len_and_kind(void)
+{
+	asm volatile (
+	"r6 = r1;"
+	"call %[bpf_get_prandom_u32];"
+	"r8 = 1;"
+	"r9 = 20;"
+	"if r0 == 0 goto 1f;"
+	"r8 = 2;"
+	"r9 = 40;"
+"1:	r1 = *(u32 *)(r6 + %[data]);"
+	"r3 = *(u32 *)(r6 + %[data_end]);"
+	"r2 = r1;"
+	"r2 += r9;"
+	"if r2 > r3 goto 2f;"
+	"if r8 != 2 goto 2f;"
+	"r0 = *(u8 *)(r1 + 39);"
+"2:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+/* The same with c->skb->data: the context comes out of a struct on the stack */
+SEC("tc")
+__description("offset of a pointer: the packet of a context that was in a struct")
+__success __retval(0)
+__naked void pkt_of_ctx_in_struct(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = r1;"
+	"r7 = r10;"
+	"r7 += -8;"
+	"call %[bpf_get_prandom_u32];"
+	"r8 = 1;"
+	"r9 = 20;"
+	"if r0 == 0 goto 1f;"
+	"r8 = 2;"
+	"r9 = 40;"
+"1:	r6 = *(u64 *)(r7 + 0);"
+	"r1 = *(u32 *)(r6 + %[data]);"
+	"r3 = *(u32 *)(r6 + %[data_end]);"
+	"r2 = r1;"
+	"r2 += r9;"
+	"if r2 > r3 goto 2f;"
+	"if r8 != 2 goto 2f;"
+	"r0 = *(u8 *)(r1 + 39);"
+"2:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+/* data - 1 or data + 30000. A range of the two plus 1 would vouch for 30001 bytes behind data. */
+SEC("tc")
+__description("offset of a pointer: one of two places, none of them in the packet")
+__failure __msg("invalid access to packet, off=30000 size=1, R1(id=0,off=30000,r=0)")
+__naked void pkt_two_places(void)
+{
+	asm volatile (
+	"r6 = r1;"
+	"call %[bpf_get_prandom_u32];"
+	"r1 = *(u32 *)(r6 + %[data]);"
+	"r3 = *(u32 *)(r6 + %[data_end]);"
+	"r2 = r1;"
+	"if r0 == 0 goto 1f;"
+	"r2 += 30000;"
+	"goto 2f;"
+"1:	r2 += -1;"
+"2:	r2 += 1;"
+	"if r2 > r3 goto 3f;"
+	"r0 = *(u8 *)(r1 + 30000);"
+"3:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+/* A range of offsets would go over the timer */
+SEC("socket")
+__description("offset of a pointer: in front of a timer or behind it")
+__success __retval(0)
+__naked void around_timer(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u32 *)(r10 - 4) = r1;"
+	"r2 = r10;"
+	"r2 += -4;"
+	"r1 = %[timers] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 2f;"
+	"r6 = r0;"
+	"call %[bpf_get_prandom_u32];"
+	"r1 = %[before];"
+	"if r0 == 0 goto 1f;"
+	"r1 = %[after];"
+"1:	r6 += r1;"
+	"r0 = *(u32 *)(r6 + 0);"
+"2:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm(bpf_map_lookup_elem), __imm_addr(timers),
+	  __imm_const(before, offsetof(struct with_timer, before[3])),
+	  __imm_const(after, offsetof(struct with_timer, after[0]))
+	: __clobber_all);
+}
+
+/* Nothing looks at the offset but for its ends: the states are joined */
+SEC("socket")
+__description("offset of a pointer: a map value that holds nothing but numbers")
+__success __retval(0) __log_level(2)
+__msg("from 4 to 5: R1=scalar({{.*}}smin32=umin32=8,smax=umax=smax32=umax32=48{{.*}}) R10=fp0")
+__msg("max_states_per_insn 1 ")
+__naked void plain_value(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = 8;"
+	"r2 = 0;"
+	"if r0 == 0 goto 1f;"
+	"r1 = 48;"
+"1:	r6 = %[in_bss] ll;"
+	"r6 += r1;"
+	"r0 = *(u64 *)(r6 + 0);"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_addr(in_bss)
+	: __clobber_all);
+}
+
+const volatile struct {
+	int size;
+	int kind;
+} cfgs[2] = { { 8, 1 }, { 32, 2 } };
+
+/* What is at a known offset of a read-only map is known */
+SEC("tc")
+__description("offset of a pointer: one of two elements of a read-only map")
+__success
+int size_of_two(struct __sk_buff *skb)
+{
+	char buf[32];
+
+	return bpf_skb_load_bytes(skb, 0, buf, (skb->mark ? &cfgs[0] : &cfgs[1])->size);
+}
+
+/* The prototypes of the two kfuncs say void *. The verifier makes a pointer to a struct of it. */
+SEC("tc")
+__description("offset of a pointer: one of two fields behind bpf_core_cast()")
+__success
+int field_behind_core_cast(struct __sk_buff *skb)
+{
+	struct task_struct *task = bpf_core_cast((void *)bpf_get_current_task(), struct task_struct);
+
+	return skb->mark ? task->pid : task->tgid;
+}
+
+SEC("tc")
+__description("offset of a pointer: one of two fields behind bpf_cast_to_kern_ctx()")
+__success
+int field_behind_kern_ctx(struct __sk_buff *ctx)
+{
+	struct sk_buff *skb = bpf_cast_to_kern_ctx(ctx);
+
+	return ctx->mark ? skb->len : skb->data_len;
+}
+
+static __naked __noinline __used void random(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Two states enter the loop, one with each offset. r9 is not what it was when they come around.
+ * Each of them is joined with the state at the head of the loop that has its offset.
+ */
+SEC("tp_btf/sched_switch")
+__description("offset of a pointer: two offsets enter a loop")
+__success
+__naked void two_offsets_in_a_loop(void)
+{
+	asm volatile (
+	"r6 = *(u64 *)(r1 + 8);"
+	"call %[bpf_get_prandom_u32];"
+	"r7 = %[pid];"
+	"if r0 == 0 goto 1f;"
+	"r7 = %[tgid];"
+"1:	r9 = 0;"
+"2:	if r9 == 7 goto 3f;"
+	"r1 = r6;"
+	"r1 += r7;"
+	"r0 = *(u32 *)(r1 + 0);"
+	"call random;"
+	"r9 = r0;"
+	"goto 2b;"
+"3:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm_const(pid, offsetof(struct task_struct, pid)),
+	  __imm_const(tgid, offsetof(struct task_struct, tgid))
+	: __clobber_all);
+}
+
+/* The offset is what changes in the loop: it's forgotten, and the loop ends */
+SEC("tc")
+__description("offset of a pointer: the offset changes in a loop")
+__success
+__naked void offset_changes_in_a_loop(void)
+{
+	asm volatile (
+	"r6 = r1;"
+	"r7 = 0;"
+"1:	r1 = *(u32 *)(r6 + %[data]);"
+	"r2 = *(u32 *)(r6 + %[data_end]);"
+	"r7 &= 0xff;"
+	"r1 += r7;"
+	"r3 = r1;"
+	"r3 += 4;"
+	"if r3 > r2 goto 2f;"
+	"r0 = *(u32 *)(r1 + 0);"
+"2:	call %[bpf_get_prandom_u32];"
+	"r7 += 3;"
+	"if r0 != 7 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index d92215c5b923..4cfdedca8890 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -12,6 +12,13 @@ struct {
 	__type(value, long long);
 } map_hash_8b SEC(".maps");
 
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, long long);
+} map_array_8b SEC(".maps");
+
 SEC("lwt_in")
 __description("variable-offset ctx access")
 __failure __msg("variable ctx access var_off=(0x0; 0x4)")
@@ -310,6 +317,202 @@ __naked void stack_write_clobbers_spilled_regs(void)
 	: __clobber_all);
 }
 
+/*
+ * if (x) a[0] = p; else a[1] = p; with one store. The paths meet in front of it.
+ * The offset is known on each of them.
+ */
+SEC("tc")
+__description("pointer is spilled at one of two offsets")
+__success __retval(0)
+__naked void ptr_at_one_of_two_offsets(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u32*)(r10 - 4) = r1;				\
+	r2 = r10;					\
+	r2 += -4;					\
+	r1 = %[map_array_8b] ll;			\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l2_%=;				\
+	r6 = r0;					\
+	r1 = 0;						\
+	*(u64*)(r10 - 16) = r1;				\
+	*(u64*)(r10 - 24) = r1;				\
+	call %[bpf_get_prandom_u32];			\
+	r7 = 0;						\
+	r8 = 1;						\
+	if r0 > 100 goto l0_%=;				\
+	r7 = 8;						\
+	r8 = 2;						\
+l0_%=:	r2 = r10;					\
+	r2 += -24;					\
+	r2 += r7;					\
+	*(u64*)(r2 + 0) = r6;				\
+	if r8 != 1 goto l1_%=;				\
+	r3 = *(u64*)(r10 - 24);				\
+	r0 = *(u64*)(r3 + 0);				\
+	exit;						\
+l1_%=:	r3 = *(u64*)(r10 - 16);				\
+	r0 = *(u64*)(r3 + 0);				\
+	exit;						\
+l2_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_array_8b)
+	: __clobber_all);
+}
+
+#define PICK_OFF(slot)					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r1 = 0;"					\
+	"if r0 > 100 goto +1;"				\
+	"r1 = 8;"					\
+	"*(u64*)(r10 - " #slot ") = r1;"
+
+#define USE_OFF(slot)					\
+	"r1 = *(u64*)(r10 - " #slot ");"		\
+	"r2 = r10;"					\
+	"r2 += -16;"					\
+	"r2 += r1;"					\
+	"*(u64*)(r2 + 0) = r0;"
+
+/* 4096 combinations of offsets that are known: not all of them are looked at */
+SEC("tc")
+__description("12 times one of two offsets")
+__success __retval(0)
+__naked void many_offsets(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64*)(r10 - 8) = r1;"
+	"*(u64*)(r10 - 16) = r1;"
+	PICK_OFF(24) PICK_OFF(32) PICK_OFF(40) PICK_OFF(48)
+	PICK_OFF(56) PICK_OFF(64) PICK_OFF(72) PICK_OFF(80)
+	PICK_OFF(88) PICK_OFF(96) PICK_OFF(104) PICK_OFF(112)
+	USE_OFF(24) USE_OFF(32) USE_OFF(40) USE_OFF(48)
+	USE_OFF(56) USE_OFF(64) USE_OFF(72) USE_OFF(80)
+	USE_OFF(88) USE_OFF(96) USE_OFF(104) USE_OFF(112)
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym;
+extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym;
+extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym;
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+
+#define ITER_NEW	"r1 = r10; r1 += -24; w2 = 0; w3 = 100; call %[bpf_iter_num_new];"
+#define ITER_NEXT	"r1 = r10; r1 += -24; call %[bpf_iter_num_next];"
+#define ITER_DESTROY	"r1 = r10; r1 += -24; call %[bpf_iter_num_destroy];"
+#define __imm_iter	__imm(bpf_iter_num_new), __imm(bpf_iter_num_next), \
+			__imm(bpf_iter_num_destroy)
+
+/*
+ *	u8 stack[10], depth = 0;
+ *	loop {
+ *		if (x) { if (depth >= 10) break; stack[depth++] = 1; }
+ *		else { depth--; if (depth >= 10) break; y = stack[depth]; }
+ *	}
+ * depth is in a 32-bit register. 'depth & 255' is tested, and computed anew for the index.
+ */
+#define PUSH_OR_POP					\
+		"r1 = *(u64 *)(r6 + 0);"		\
+		"if r1 == 0 goto 2f;"			\
+		"w7 = w8;"				\
+		"w7 &= 255;"				\
+		"if w7 > 9 goto 9f;"			\
+		"r1 = r8;"				\
+		"r1 &= 255;"				\
+		"r2 = r10;"				\
+		"r2 += -10;"				\
+		"r2 += r1;"				\
+		"*(u8 *)(r2 + 0) = r1;"			\
+		"w8 += 1;"				\
+		"goto 1b;"				\
+	"2:"						\
+		"w8 += -1;"				\
+		"w2 = w8;"				\
+		"w2 &= 255;"				\
+		"if w2 > 9 goto 9f;"			\
+		"r2 = r8;"				\
+		"r2 &= 255;"				\
+		"r3 = r10;"				\
+		"r3 += -10;"				\
+		"r3 += r2;"				\
+		"r9 = *(u8 *)(r3 + 0);"			\
+		"goto 1b;"
+
+SEC("tc")
+__description("depth of a stack on the stack: a copy is tested")
+__success __retval(0)
+__naked void depth_copy_is_tested(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 16) = r1;"
+		"w8 = 0;"
+		ITER_NEW
+	"1:"
+		ITER_NEXT
+		"if r0 == 0 goto 9f;"
+		PUSH_OR_POP
+	"9:"
+		ITER_DESTROY
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(zero), __imm_iter
+		: __clobber_all);
+}
+
+SEC("tc")
+__description("depth of a stack on the stack: no bound")
+__failure __msg("invalid write to stack R2 off=0 size=1")
+__naked void depth_without_a_bound(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 16) = r1;"
+		"w8 = 0;"
+		ITER_NEW
+	"1:"
+		ITER_NEXT
+		"if r0 == 0 goto 9f;"
+		"r1 = r8;"
+		"r1 &= 255;"
+		"r2 = r10;"
+		"r2 += -10;"
+		"r2 += r1;"
+		"*(u8 *)(r2 + 0) = r1;"
+		"w8 += 1;"
+		"goto 1b;"
+	"9:"
+		ITER_DESTROY
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_iter
+		: __clobber_all);
+}
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+	bpf_iter_num_new(0, 0, 0);
+	bpf_iter_num_next(0);
+	bpf_iter_num_destroy(0);
+}
+
 SEC("sockops")
 __description("indirect variable-offset stack access, unbounded")
 __failure __msg("invalid unbounded variable-offset write to stack R4")
-- 
2.55.0

[PATCH bpf-next v3 42/45] selftests/bpf: Test static subprogs that are verified on their own

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier followed each call into the subprog, with the frames of the
callers in the state. Now it verifies most static subprogs for the states
at their first insn, without these frames. A call waits, and each exit
state goes to all calls that wait for this entry state. What can go wrong:
a call does not get a result that is its own, or the subprog changes what
a caller holds and the caller does not see it.

  [E] "bpf: Add the sets of states at an insn and the entries of functions"
  [M] "bpf: Move states along edges, into loops and subprogs and out of
      them"
  [B] "bpf: Make the state after a call from the caller's state and an
      exit state"
  [O] "bpf: Verify static subprogs on their own"
  [D] "bpf: Keep the frames of the caller for a subprog that loads
      data_end"
  [R] "bpf: Keep the frames of the caller for ctx of struct_ops with
      a reference"

  verifier_..         the case                                        pins
  ------------------  ----------------------------------------------  ----
  calls_in_a_row.c    20 calls pass kind 3, one call passes kind 1:   [E]
                      the 20 entry states are joined, the one stays
                      'x = inc(x); x = inc(x);' loads and returns 2   [M]
                      3 entry states meet in the subprog in 2 steps:  [E]
                      the exit state is for all 3 calls, if they      [B]
                      wait together or come one after the other
                      The argument is r6 + 10, linked to r6 of the    [O]
                      caller: the 10 is not for the subprog's links
  subprog_callers.c   The subprog gives up the lock or the RCU        [O]
                      section of its caller and takes it again.
                      3 calls wait when the verifier finds it out.
                      It is called, and it is a timer callback too.   [O]
                      Recursive subprogs change packet data or        [O]
                      a tree in the callback of bpf_rbtree_add().
                      Ids at the return: an inner map from each of    [B]
                      2 calls, a reference 2 calls down, too many ids.
                      Two calls pass packet pointers whose offsets    [O]
                      have different ranges.
                      The subprog loads data_end itself and checks    [D]
                      the packet pointer of its caller.
  struct_ops_         The subprog reads an argument with __ref from   [R]
  refcounted_subprog  ctx while the prog holds the reference.
  stack_arg.c         The stack arguments that a subprog got are not  [O]
                      the ones that it passes on. A subprog does the
                      NULL check of a stack argument for its caller.
                      A stack argument is linked in the same way.     [O]

15 of the 25 progs are unsafe and must be rejected: only one call gets
a number that is out of bounds, only the code after one call uses what the
subprog took away.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../prog_tests/test_struct_ops_refcounted.c   |   2 +
 .../selftests/bpf/prog_tests/verifier.c       |   4 +
 .../bpf/progs/struct_ops_refcounted_subprog.c |  30 +
 .../bpf/progs/verifier_calls_in_a_row.c       | 223 +++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  | 127 ++++
 .../bpf/progs/verifier_subprog_callers.c      | 543 ++++++++++++++++++
 6 files changed, 929 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_callers.c

diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
index db79ac9b98d3..e33e4cd1e157 100644
--- a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
+++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
@@ -1,6 +1,7 @@
 #include <test_progs.h>
 
 #include "struct_ops_refcounted.skel.h"
+#include "struct_ops_refcounted_subprog.skel.h"
 #include "struct_ops_refcounted_fail__ref_leak.skel.h"
 #include "struct_ops_refcounted_fail__global_subprog.skel.h"
 #include "struct_ops_refcounted_fail__tail_call.skel.h"
@@ -9,6 +10,7 @@
 void test_struct_ops_refcounted(void)
 {
 	RUN_TESTS(struct_ops_refcounted);
+	RUN_TESTS(struct_ops_refcounted_subprog);
 	RUN_TESTS(struct_ops_refcounted_fail__ref_leak);
 	RUN_TESTS(struct_ops_refcounted_fail__global_subprog);
 	RUN_TESTS(struct_ops_refcounted_fail__tail_call);
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 2f34a34e3d54..3f3b9b48523f 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -36,6 +36,7 @@
 #include "verifier_btf_flex_array.skel.h"
 #include "verifier_btf_unreliable_prog.skel.h"
 #include "verifier_call_large_imm.skel.h"
+#include "verifier_calls_in_a_row.skel.h"
 #include "verifier_callx.skel.h"
 #include "verifier_callx_rodata.skel.h"
 #include "verifier_cfg.skel.h"
@@ -130,6 +131,7 @@
 #include "verifier_stack_arg_order.skel.h"
 #include "verifier_stack_ptr.skel.h"
 #include "verifier_store_release.skel.h"
+#include "verifier_subprog_callers.skel.h"
 #include "verifier_subprog_insn_stats.skel.h"
 #include "verifier_subprog_precision.skel.h"
 #include "verifier_subprog_topo.skel.h"
@@ -236,6 +238,7 @@ void test_verifier_btf_ctx_access(void)       { RUN(verifier_btf_ctx_access); }
 void test_verifier_btf_flex_array(void)       { RUN(verifier_btf_flex_array); }
 void test_verifier_btf_unreliable_prog(void)  { RUN(verifier_btf_unreliable_prog); }
 void test_verifier_call_large_imm(void)       { RUN(verifier_call_large_imm); }
+void test_verifier_calls_in_a_row(void)       { RUN(verifier_calls_in_a_row); }
 void test_verifier_callx(void)                { RUN_JITED(verifier_callx); }
 void test_verifier_callx_rodata(void)         { RUN_JITED(verifier_callx_rodata); }
 void test_verifier_cfg(void)                  { RUN(verifier_cfg); }
@@ -342,6 +345,7 @@ void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }
 void test_verifier_stack_arg_order(void)      { RUN(verifier_stack_arg_order); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
 void test_verifier_store_release(void)        { RUN(verifier_store_release); }
+void test_verifier_subprog_callers(void)      { RUN(verifier_subprog_callers); }
 void test_verifier_subprog_insn_stats(void)   { RUN(verifier_subprog_insn_stats); }
 void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
 void test_verifier_subprog_topo(void)        { RUN(verifier_subprog_topo); }
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c b/tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
new file mode 100644
index 000000000000..c5524bce975b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+static __noinline int pid_of(unsigned long long *ctx)
+{
+	struct task_struct *task = (void *)ctx[1];
+
+	return task->pid;
+}
+
+/* A static subprog loads the referenced argument from ctx, like the main prog. */
+SEC("struct_ops/test_refcounted")
+__success
+int refcounted_subprog(unsigned long long *ctx)
+{
+	int pid = pid_of(ctx);
+
+	bpf_task_release((void *)ctx[1]);
+	return pid & 1;
+}
+
+SEC(".struct_ops.link")
+struct bpf_testmod_ops testmod_refcounted_subprog = {
+	.test_refcounted = (void *)refcounted_subprog,
+};
diff --git a/tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c b/tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
new file mode 100644
index 000000000000..401256bc34b0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
@@ -0,0 +1,223 @@
+// SPDX-License-Identifier: GPL-2.0
+/* A static subprog that is called from many places */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* gcc makes a copy of a function for an argument that is the same constant in most calls */
+#if defined(__GNUC__) && !defined(__clang__)
+#define __noclone __attribute__((noclone))
+#else
+#define __noclone
+#endif
+
+volatile int sel;
+char arr[64];
+
+#define SITES_4(call, n)  call(n) call(n + 1) call(n + 2) call(n + 3)
+#define SITES_20(call)    SITES_4(call, 0) SITES_4(call, 4) SITES_4(call, 8) \
+			  SITES_4(call, 12) SITES_4(call, 16)
+
+/*
+ * The states of the 20 calls are joined at the entry of the subprog one by one: x grows. The state
+ * of the first call stays apart from them, however many joins it takes.
+ *
+ * kind 1: idx is an index. kind 3: idx is something else. 2 is not there.
+ */
+static __noinline __noclone int use(int kind, unsigned int idx, int x)
+{
+	if (kind == 1)
+		return arr[idx] + x;
+	return idx + x;
+}
+
+SEC("socket")
+__description("20 calls in a row, a number that a jump tests")
+__success
+int many_sites_gap(void *ctx)
+{
+	int r = use(1, 5, 100);
+
+#define USE(n) if (sel == n) r += use(3, 1000, n);
+	SITES_20(USE)
+	return r;
+}
+
+/* The second call gets what the first call returns */
+static __noinline __noclone int inc(int x)
+{
+	return x + 1;
+}
+
+SEC("socket")
+__description("x = inc(x), two times in a row")
+__success __retval(2)
+int inc_two_times(void *ctx)
+{
+	int x = 0;
+
+	x = inc(x);
+	x = inc(x);
+	return x;
+}
+
+char four[4] SEC(".data.four") = { 10, 11, 12, 13 };
+
+/*
+ * For the caller the argument is r6 + 10. The subprog has no r6: r1 is linked to nothing there.
+ * r1 is 6 .. 9 and r2 is r1 + 4, that is 10 .. 13, not r1 + 4 - 10.
+ */
+#define FOUR_AT_R1_PLUS_4(off)			\
+	"r2 = r1;"				\
+	"r2 += 4;"				\
+	"r0 = 0;"				\
+	"if r1 < 6 goto 1f;"			\
+	"if r1 > 9 goto 1f;"			\
+	"r3 = %[four] ll;"			\
+	"r3 += r2;"				\
+	"r0 = *(u8 *)(r3 " #off ");"		\
+"1:"						\
+	"exit;"
+
+static __naked __noinline __used int four_at_0_to_3(void)
+{
+	asm volatile (FOUR_AT_R1_PLUS_4(- 10) :: __imm_addr(four) : __clobber_all);
+}
+
+static __naked __noinline __used int four_at_10_to_13(void)
+{
+	asm volatile (FOUR_AT_R1_PLUS_4(+ 0) :: __imm_addr(four) : __clobber_all);
+}
+
+#define CALL_WITH_R6_PLUS_10(func)		\
+	"call %[bpf_get_prandom_u32];"		\
+	"r6 = r0;"				\
+	"r1 = r6;"				\
+	"r1 += 10;"				\
+	"call " #func ";"			\
+	"r0 += r6;"				\
+	"exit;"
+
+SEC("socket")
+__description("the argument is linked to a register of the caller")
+__success
+__naked void arg_is_linked(void)
+{
+	asm volatile (CALL_WITH_R6_PLUS_10(four_at_0_to_3)
+		      :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+SEC("socket")
+__description("the argument is linked to a register of the caller, out of bounds")
+__failure __msg("invalid access to map value, value_size=4 off=10 size=1")
+__naked void arg_is_linked_bad(void)
+{
+	asm volatile (CALL_WITH_R6_PLUS_10(four_at_10_to_13)
+		      :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * r1 is 1, 3 or 5, and the jumps look at it: the states of three calls stay apart at the first
+ * insn. Those of 1 and 3 meet behind the second jump. What becomes of them meets that of 5 behind
+ * the first one. From there on one state stands for the three, and what comes out is for
+ * all three calls.
+ * Returns r2 + 3, r2 + 2, r2.
+ */
+static __naked __noinline __used int three_kinds(void)
+{
+	asm volatile (
+	"if r1 == 5 goto 2f;"
+	"if r1 == 3 goto 1f;"
+	"r2 += 1;"
+"1:"
+	"r2 += 2;"
+"2:"
+	"r0 = r2;"
+	"exit;"
+	::: __clobber_all);
+}
+
+#define CALL(kind, v)				\
+	"r1 = " #kind ";"			\
+	"r2 = " #v ";"				\
+	"call three_kinds;"
+
+#define LOAD					\
+	"r1 = %[four] ll;"			\
+	"r1 += r0;"				\
+	"r0 = *(u8 *)(r1 + 0);"
+
+/* The three calls wait at the same time */
+#define BRANCHES(v1, load1, v3, load3, v5, load5)	\
+	asm volatile (				\
+	"r6 = *(u32 *)(r1 + %[len]);"		\
+	"if r6 == 1 goto 1f;"			\
+	"if r6 == 3 goto 3f;"			\
+	CALL(5, v5)				\
+	load5					\
+	"exit;"					\
+"1:"						\
+	CALL(1, v1)				\
+	load1					\
+	"exit;"					\
+"3:"						\
+	CALL(3, v3)				\
+	load3					\
+	"exit;"					\
+	:					\
+	: __imm_addr(four),			\
+	  __imm_const(len, offsetof(struct __sk_buff, len))	\
+	: __clobber_all)
+
+SEC("tc")
+__description("three calls wait, their states meet in two steps")
+__success __retval(10)
+__naked void three_calls_wait(void)
+{
+	BRANCHES(0, LOAD, 0, LOAD, 0, LOAD);
+}
+
+SEC("tc")
+__description("three calls wait, their states meet in two steps, the first one gets 7")
+__failure __msg("invalid access to map value, value_size=4 off=7 size=1")
+__naked void three_calls_wait_bad_1(void)
+{
+	BRANCHES(4, LOAD, 0, "", 0, "");
+}
+
+#define ADD	LOAD "r6 += r0;"
+
+/* A call is done with before the next one is made */
+#define ROW(k1, v1, load1, k2, v2, load2, k3, v3, load3)	\
+	asm volatile (				\
+	"r6 = 0;"				\
+	CALL(k1, v1)				\
+	load1					\
+	CALL(k2, v2)				\
+	load2					\
+	CALL(k3, v3)				\
+	load3					\
+	"r0 = r6;"				\
+	"exit;"					\
+	:					\
+	: __imm_addr(four)			\
+	: __clobber_all)
+
+SEC("tc")
+__description("three calls in a row, their states meet in two steps")
+__success __retval(35)
+__naked void three_calls_in_a_row(void)
+{
+	ROW(1, 0, ADD, 3, 0, ADD, 5, 0, ADD);
+}
+
+SEC("tc")
+__description("three calls in a row, the second one gets 6")
+__failure __msg("invalid access to map value, value_size=4 off=6 size=1")
+__naked void three_calls_in_a_row_bad_2(void)
+{
+	ROW(1, 0, "", 3, 4, ADD, 5, 0, "");
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index 146cdf81bbc4..e3f5e4d14b75 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -445,6 +445,133 @@ __naked void stack_arg_shared_store(void)
 	);
 }
 
+/* What it stores is what it got: the path that stores nothing looks the same at the call. */
+static __naked __noinline __used long stores_same_on_one_path(long a, long b, long c, long d,
+							      long e, long f, long g)
+{
+	asm volatile (
+	"r6 = *(u64 *)(r11 + 8);"
+	"r7 = *(u64 *)(r11 + 16);"
+	"if r1 == 0 goto l0_%=;"
+	"*(u64 *)(r11 - 8) = r6;"
+	"*(u64 *)(r11 - 16) = r7;"
+"l0_%=:"
+	"r1 = 0;"
+	"r2 = 0;"
+	"r3 = 0;"
+	"r4 = 0;"
+	"r5 = 0;"
+	"call subprog_7args;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: incoming args are not taken for outgoing args, same values")
+__failure __msg("callee expects 7 args, stack arg1 is not initialized")
+__naked void incoming_for_outgoing_same(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = r0;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"*(u64 *)(r11 - 8) = 6;"
+	"*(u64 *)(r11 - 16) = 7;"
+	"call stores_same_on_one_path;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/* r1 is 6 .. 9 and r2 is r1 + 4, that is 10 .. 13: r9, which is not set, is not read */
+static __naked __noinline __used long sixth_plus_4(long a, long b, long c, long d, long e, long f)
+{
+	asm volatile (
+	"r1 = *(u64 *)(r11 + 8);"
+	"r2 = r1;"
+	"r2 += 4;"
+	"r0 = 0;"
+	"if r1 < 6 goto l0_%=;"
+	"if r1 > 9 goto l0_%=;"
+	"if r2 > 9 goto l0_%=;"
+	"r0 = r9;"
+"l0_%=:"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* For the caller the argument is r6 + 10. The subprog has no r6. */
+SEC("tc")
+__description("stack_arg: the argument is linked to a register of the caller")
+__success
+__naked void stack_arg_is_linked(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r6 = r0;"
+	"r1 = r6;"
+	"r1 += 10;"
+	"*(u64 *)(r11 - 8) = r1;"
+	"r1 = 1;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"call sixth_plus_4;"
+	"r0 += r6;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+static __naked __noinline __used long sixth_is_there(long a, long b, long c, long d, long e,
+						     long *f)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r11 + 8);"
+	"if r0 == 0 goto l0_%=;"
+	"r0 = 1;"
+"l0_%=:"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: subprog does the NULL check for its caller")
+__success
+__naked void null_check_of_stack_arg(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"r1 = %[map_hash_8b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"r6 = r0;"
+	"r1 = 1;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"*(u64 *)(r11 - 8) = r6;"
+	"call sixth_is_there;"
+	"if r0 == 0 goto l0_%=;"
+	"r0 = *(u64 *)(r6 + 0);"
+"l0_%=:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
 /* r1 - r5 and all 19 slots. Its prototype has no parameters: what it takes is seen in its code. */
 __naked __noinline __used
 static unsigned long subprog_24args(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_callers.c b/tools/testing/selftests/bpf/progs/verifier_subprog_callers.c
new file mode 100644
index 000000000000..48338c0ae467
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_callers.c
@@ -0,0 +1,543 @@
+// SPDX-License-Identifier: GPL-2.0
+/* A static subprog and its callers: what they hold, where they run */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+#define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8)))
+
+struct elem {
+	struct bpf_list_node l;
+	__u64 data;
+};
+
+private(A) struct bpf_spin_lock lock_a;
+private(A) struct bpf_list_head head_a __contains(elem, l);
+
+int pid;
+
+static __noinline void relock_a(void)
+{
+	bpf_spin_unlock(&lock_a);
+	bpf_spin_lock(&lock_a);
+}
+
+/* Somebody else can pop and free the node while the lock is not held. */
+SEC("tc")
+__description("subprog drops the lock of its caller for a while, non-owning reference")
+__failure __msg("invalid mem access 'scalar'")
+int relock_non_owning_ref(struct __sk_buff *skb)
+{
+	struct elem *n;
+
+	n = bpf_obj_new(typeof(*n));
+	if (!n)
+		return 0;
+	bpf_spin_lock(&lock_a);
+	bpf_list_push_front(&head_a, &n->l);
+	relock_a();
+	n->data = 42;
+	bpf_spin_unlock(&lock_a);
+	return 0;
+}
+
+static __noinline void rcu_window(void)
+{
+	bpf_rcu_read_unlock();
+	bpf_rcu_read_lock();
+}
+
+SEC("?fentry.s/" SYS_PREFIX "sys_getpgid")
+__description("subprog leaves the RCU section of its caller for a while")
+__failure __msg("R1 must be a rcu pointer")
+int rcu_window_rcu_ptr(void *ctx)
+{
+	struct task_struct *task = bpf_get_current_task_btf(), *parent;
+
+	bpf_rcu_read_lock();
+	parent = task->real_parent;
+	if (!parent)
+		goto out;
+	rcu_window();
+	parent = bpf_task_acquire(parent);
+	if (parent) {
+		pid = parent->pid;
+		bpf_task_release(parent);
+	}
+out:
+	bpf_rcu_read_unlock();
+	return 0;
+}
+
+struct timer_elem {
+	struct bpf_timer timer;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer_elem);
+} timers SEC(".maps");
+
+static __noinline int leaves_preempt_off(void *map, int *key, struct timer_elem *val)
+{
+	bpf_preempt_disable();
+	return 0;
+}
+
+/* The call and the callback have one frame each. The callback returns to the kernel. */
+SEC("tc")
+__description("subprog is called, then it is made a timer callback")
+__failure __msg("cannot be used inside bpf_preempt_disable-ed region")
+int call_then_timer_callback(struct __sk_buff *skb)
+{
+	struct timer_elem *val = bpf_map_lookup_elem(&timers, &(int){0});
+
+	if (!val)
+		return 0;
+	leaves_preempt_off(NULL, NULL, NULL);
+	bpf_preempt_enable();
+	bpf_timer_set_callback(&val->timer, leaves_preempt_off);
+	return 0;
+}
+
+/* return p + 14 > ctx->data_end ? -22 : 0; */
+static __naked __noinline __used int has_eth(struct xdp_md *ctx, void *p)
+{
+	asm volatile (
+	"r3 = *(u32 *)(r1 + %[data_end]);"
+	"r2 += 14;"
+	"r0 = -22;"
+	"if r2 > r3 goto 1f;"
+	"r0 = 0;"
+"1:	exit;"
+	:
+	: __imm_const(data_end, offsetof(struct xdp_md, data_end))
+	: __clobber_all);
+}
+
+/* The caller learns how much of the packet is there. */
+SEC("?xdp")
+__description("subprog compares a packet pointer of its caller with the end that it loads itself")
+__success
+__naked void subprog_loads_data_end(void)
+{
+	asm volatile (
+	"r6 = *(u32 *)(r1 + %[data]);"
+	"r2 = r6;"
+	"call has_eth;"
+	"if r0 != 0 goto 1f;"
+	"r0 = *(u8 *)(r6 + 13);"
+"1:	r0 = 1;"
+	"exit;"
+	:
+	: __imm_const(data, offsetof(struct xdp_md, data))
+	: __clobber_all);
+}
+
+struct tree_elem {
+	long key;
+	struct bpf_rb_node node;
+};
+
+private(D) struct bpf_spin_lock lock_d;
+private(D) struct bpf_rb_root root_d __contains(tree_elem, node);
+
+static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct tree_elem *elem_a = container_of(a, struct tree_elem, node);
+	struct tree_elem *elem_b = container_of(b, struct tree_elem, node);
+
+	return elem_a->key < elem_b->key;
+}
+
+static __noinline long removes_and_recurses(struct tree_elem *e, int n)
+{
+	struct bpf_rb_node *node;
+
+	if (n <= 0)
+		return 0;
+	node = bpf_rbtree_remove(&root_d, &e->node);
+	if (node)
+		bpf_rbtree_add(&root_d, node, less);
+	n = removes_and_recurses(e, n - 1);
+	barrier_var(n);
+	return n + 1;
+}
+
+static bool less_changes_tree(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct tree_elem *elem_a = container_of(a, struct tree_elem, node);
+	struct tree_elem *elem_b = container_of(b, struct tree_elem, node);
+
+	removes_and_recurses(elem_b, 1);
+	return elem_a->key < elem_b->key;
+}
+
+/* bpf_rbtree_add() walks the tree while it calls the callback. */
+SEC("?tc")
+__description("recursive subprog changes the tree, it is called from the callback of rbtree_add")
+__arch_x86_64
+__arch_arm64
+__failure __msg("bpf_rbtree_remove not allowed in rbtree cb")
+int recursion_removes_in_rbtree_callback(void *ctx)
+{
+	struct tree_elem *e;
+
+	e = bpf_obj_new(typeof(*e));
+	if (!e)
+		return 0;
+	bpf_spin_lock(&lock_d);
+	bpf_rbtree_add(&root_d, &e->node, less_changes_tree);
+	bpf_spin_unlock(&lock_d);
+	return 0;
+}
+
+/*
+ * What a subprog does is what its callees do. The walk over the graph is done with second()
+ * before it is done with first(), which second() calls: the prog calls second() and then
+ * first(), so first() is looked at first, and in it the call that comes last.
+ */
+static __noinline int pulls_data(struct __sk_buff *skb)
+{
+	return bpf_skb_pull_data(skb, 0);
+}
+
+static __noinline int pulls_first(struct __sk_buff *skb, int n);
+
+static __noinline int pulls_second(struct __sk_buff *skb, int n)
+{
+	if (n <= 0)
+		return 0;
+	return pulls_first(skb, n - 1) + 1;
+}
+
+static __noinline int pulls_first(struct __sk_buff *skb, int n)
+{
+	int r = pulls_data(skb);
+
+	return r + pulls_second(skb, n);
+}
+
+SEC("?tc")
+__description("recursive static subprogs, the one that is explored last changes packet data")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid mem access 'scalar'")
+int recursion_changes_pkt_data(struct __sk_buff *skb)
+{
+	char *p = (void *)(long)skb->data, *end = (void *)(long)skb->data_end;
+	int r;
+
+	if (p + 1 > end)
+		return 0;
+	r = pulls_second(skb, 1);
+	*p = 42;
+	r += pulls_first(skb, 0);
+	return r & 1;
+}
+
+static __naked __noinline __used long returns_third(void)
+{
+	asm volatile (
+	"r0 = r1;"
+	"r0 = r2;"
+	"r0 = r3;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* What comes back has the range of r6, not the range of the pointer of the first call. */
+#define PKT_POINTERS_OF_TWO_RANGES(func)	\
+	asm volatile (				\
+	"r9 = *(u32 *)(r1 + %[data]);"		\
+	"r8 = *(u32 *)(r1 + %[data_end]);"	\
+	"call %[bpf_get_prandom_u32];"		\
+	"r7 = r0;"				\
+	"r0 &= 0xff;"				\
+	"r1 = 0;"				\
+	"r2 = 3;"				\
+	"r3 = r9;"				\
+	"r3 += r0;"				\
+	"call " #func ";"			\
+	"r7 &= 1;"				\
+	"r6 = r9;"				\
+	"r6 += r7;"				\
+	"r1 = 0;"				\
+	"r2 = 3;"				\
+	"r3 = r6;"				\
+	"call " #func ";"			\
+	"if r0 == 0 goto 1f;"			\
+	"if r0 > r8 goto 1f;"			\
+	"r0 = *(u8 *)(r6 + 100);"		\
+"1:"						\
+	"r0 = 0;"				\
+	"exit;"					\
+	:					\
+	: __imm(bpf_get_prandom_u32),		\
+	  __imm_const(data, offsetof(struct __sk_buff, data)),		\
+	  __imm_const(data_end, offsetof(struct __sk_buff, data_end))	\
+	: __clobber_all)
+
+SEC("tc")
+__description("two calls, the offset of the packet pointer has a smaller range in the second one")
+__failure __msg("invalid access to packet")
+__naked void pkt_pointers_of_two_ranges(void)
+{
+	PKT_POINTERS_OF_TWO_RANGES(returns_third);
+}
+
+static __naked __noinline __used long returns_third_r2_calls_down(void)
+{
+	asm volatile (
+	"if r2 == 0 goto 1f;"
+	"r2 += -1;"
+	"call returns_third_r2_calls_down;"
+	"exit;"
+"1:"
+	"r0 = r3;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* A recursive subprog that gets such a pointer needs the frames of its callers too. */
+SEC("?tc")
+__description("two calls of a recursive subprog, the second packet pointer has the smaller range")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to packet")
+__naked void pkt_pointers_of_two_ranges_recursion(void)
+{
+	PKT_POINTERS_OF_TWO_RANGES(returns_third_r2_calls_down);
+}
+
+static __naked __noinline __used long tcp_sock_of(void)
+{
+	asm volatile (
+	"call %[bpf_tcp_sock];"
+	"exit;"
+	:
+	: __imm(bpf_tcp_sock)
+	: __clobber_all);
+}
+
+static __naked __noinline __used long calls_tcp_sock_of(void)
+{
+	asm volatile (
+	"call tcp_sock_of;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* The caller of tcp_sock_of() has the reference from its own caller. */
+SEC("tc")
+__description("reference goes down two calls, what is made from it is used after the release")
+__failure __msg("R7 invalid mem access 'scalar'")
+__naked void reference_two_calls_down(void)
+{
+	asm volatile (
+	"r2 = 0;"
+	"*(u64 *)(r10 - 8) = r2;"
+	"*(u64 *)(r10 - 16) = r2;"
+	"r2 = r10;"
+	"r2 += -16;"
+	"r3 = 12;"
+	"r4 = 0;"
+	"r5 = 0;"
+	"call %[bpf_sk_lookup_tcp];"
+	"if r0 == 0 goto 1f;"
+	"r6 = r0;"
+	"r1 = r6;"
+	"call calls_tcp_sock_of;"
+	"r7 = r0;"
+	"r1 = r6;"
+	"call %[bpf_sk_release];"
+	"if r7 == 0 goto 1f;"
+	"r0 = *(u32 *)(r7 + 0);"
+"1:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_sk_lookup_tcp),
+	  __imm(bpf_sk_release)
+	: __clobber_all);
+}
+
+struct inner_timers {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer_elem);
+} inner_timers SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
+	__uint(max_entries, 2);
+	__type(key, int);
+	__array(values, struct inner_timers);
+} outer_timers SEC(".maps");
+
+static __naked __noinline __used long inner_map(void)
+{
+	asm volatile (
+	"*(u32 *)(r10 - 4) = r1;"
+	"r2 = r10;"
+	"r2 += -4;"
+	"r1 = %[outer_timers] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(outer_timers)
+	: __clobber_all);
+}
+
+#define TWO_INNER_MAPS				\
+	"r1 = 0;"				\
+	"call inner_map;"			\
+	"if r0 == 0 goto 1f;"			\
+	"r6 = r0;"				\
+	"r1 = 1;"				\
+	"call inner_map;"			\
+	"if r0 == 0 goto 1f;"			\
+	"r7 = r0;"
+
+/* The timer is in the map in r0. It is fine only if that is the map in r6. */
+#define INIT_TIMER_WITH_R6			\
+	"r1 = 0;"				\
+	"*(u32 *)(r10 - 4) = r1;"		\
+	"r2 = r10;"				\
+	"r2 += -4;"				\
+	"r1 = r0;"				\
+	"call %[bpf_map_lookup_elem];"		\
+	"if r0 == 0 goto 1f;"			\
+	"r1 = r0;"				\
+	"r2 = r6;"				\
+	"r3 = 1;"				\
+	"call %[bpf_timer_init];"		\
+"1:"						\
+	"r0 = 0;"				\
+	"exit;"
+
+#define INNER_MAP_IMMS				\
+	__imm(bpf_get_prandom_u32),		\
+	__imm(bpf_map_lookup_elem),		\
+	__imm(bpf_timer_init)
+
+/* Each call returns another map. */
+SEC("tc")
+__description("subprog returns an inner map, timer of one map is set up with another one")
+__failure __msg("doesn't match map pointer in R2")
+__naked void inner_maps_of_two_calls(void)
+{
+	asm volatile (
+	TWO_INNER_MAPS
+	INIT_TIMER_WITH_R6
+	:
+	: INNER_MAP_IMMS
+	: __clobber_all);
+}
+
+/* As many as there is room for ids at the return, and one more. */
+static __naked __noinline __used void saves_irq_flags_many_times(void)
+{
+	asm volatile (
+	".rept 9153;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call %[bpf_local_irq_save];"
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	".endr;"
+	"exit;"
+	:
+	: __imm(bpf_local_irq_save)
+	: __clobber_all);
+}
+
+SEC("tc")
+__description("subprog comes back with more ids than there is room for")
+__failure __msg("too many ids at the call at insn")
+__log_level(4)
+int too_many_ids_at_the_return(void *ctx)
+{
+	unsigned long flags;
+
+	saves_irq_flags_many_times();
+	bpf_local_irq_save(&flags);
+	bpf_local_irq_restore(&flags);
+	return 0;
+}
+
+static __naked __noinline __used void leaves_rcu_section(void)
+{
+	asm volatile (
+	"call %[bpf_rcu_read_unlock];"
+	"call %[bpf_rcu_read_lock];"
+	"exit;"
+	:
+	: __imm(bpf_rcu_read_unlock),
+	  __imm(bpf_rcu_read_lock)
+	: __clobber_all);
+}
+
+static __naked __noinline __used int returns_zero(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/*
+ * Three calls wait for the subprog. When it's known what the subprog does to the RCU section of
+ * its callers each of them stops to wait and is verified in the context of its caller.
+ * Then one more call waits.
+ */
+#define RCU_BRANCHES(then1, then2, then3)	\
+	asm volatile (				\
+	"r6 = *(u32 *)(r1 + %[len]);"		\
+	"call %[bpf_rcu_read_lock];"		\
+	"if r6 == 1 goto 1f;"			\
+	"if r6 == 2 goto 2f;"			\
+	"call leaves_rcu_section;"		\
+	then3					\
+	"goto 3f;"				\
+"1:"						\
+	"call leaves_rcu_section;"		\
+	then1					\
+	"goto 3f;"				\
+"2:"						\
+	"call leaves_rcu_section;"		\
+	then2					\
+"3:"						\
+	"call %[bpf_rcu_read_unlock];"		\
+	"call returns_zero;"			\
+	"exit;"					\
+	:					\
+	: __imm(bpf_rcu_read_lock),		\
+	  __imm(bpf_rcu_read_unlock),		\
+	  __imm_const(len, offsetof(struct __sk_buff, len))	\
+	: __clobber_all)
+
+SEC("tc")
+__description("three calls stop to wait")
+__success __retval(0)
+__naked void three_calls_stop_to_wait(void)
+{
+	RCU_BRANCHES("", "", "");
+}
+
+SEC("tc")
+__description("three calls stop to wait, what follows the first one")
+__failure __msg("R7 !read_ok")
+__naked void three_calls_stop_to_wait_bad_1(void)
+{
+	RCU_BRANCHES("r0 = r7;", "", "");
+}
+
+char _license[] SEC("license") = "GPL";
-- 
2.55.0

[PATCH bpf-next v3 43/45] selftests/bpf: Test loops that are counted, walked or widened

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier has three methods for a loop:
 - Counted: scalar evolution gives the number of iterations and a range for
   each register that has a closed form. The body is verified one time.
 - Walked: iteration by iteration, with the real values.
 - Widened: what returns to the head is joined with the state at the head.
   A bound that grows moves to a fixed stop, until nothing changes.
The wrong method rejects a valid prog or costs thousands of insns. A wrong
state at the head loads an unsafe prog.

  [W] "bpf: Add union and widening of the range of a register"
  [J] "bpf: Add join of registers and stack slots"
  [A] "bpf: Find the registers and stack slots that later checks depend on"
  [M] "bpf: Move states along edges, into loops and subprogs and out of
      them"
  [L] "bpf: Enter and leave loops in the main pass, build flow.c"
  [C] "bpf: Keep the count of a loop when a register has no closed form"
  [G] "bpf: Tell the passes over the graph where an added may_goto goes"
  [S] "bpf: Keep the step of a number in the union of two ranges"
  [1] "bpf: Keep the first iteration of a counted loop apart"
  [R] "bpf: What a call returns in each iteration is like what the loop
      loads"
  [T] "bpf: Find where the jumps of a loop stop a number"

  verifier_..        the loop                     the verifier must
  -----------------  ---------------------------  -------------------------
  loop_need.c        is counted, 'x = x * 3 + 1'  [C] walk it if a jump, an
                     has no closed form           access or a call ahead
                                                  depends on x ([A]), else
                                                  verify it one time
                     is too long for a walk,      [1] verify the first
                     'idx = table[idx]',          iteration on its own
                     'x = rnd() & 15'             [R] the same
  loop_as_is.c       is not counted               [M] join the state that
                                                  ends a walk with a state
                                                  of this walk only
                     'if (..) n--', from 3        [W] stop n at 0
                     'off -= 4' until off is -4   [S] keep the step of 4
                     'if (n < 8) a[n++] = j',     [T] stop n at 8, off at
                     'off += l' below 8000        7999
  loop_walk.c        has two entries (bpfjailer)  [L] walk it to its end
                     each iteration decides       [L] go on with the walk
                     a jump that leaves the loop
                     'while (rnd()) n++'          [L] give up at once
                     has two iterations, a flag   [L] walk both
                     for a counter, and a call
                     only a flag that went both   [L] give up
                     ways decides its jump
                     32 states enter, each needs  [L] count the insns of
                     a walk of 300 insns          each walk, not of all
                     is too long for a walk       [L] not try it again
  loop_latch.c       ends when a flag is false:   [G] count it by the jump
                     'more = ++i < N && cond',    that clears the flag
                     20 insns, a jump or calls
                     in front of 'while (more)'
                     from that jump a path leads  [G] not count it
                     to another back edge, or
                     into a nested loop
                     the flag is a constant only  [G] not count it
                     without an added may_goto
  value_ptr_arith.c  'p = &v[p->next]'            [J] keep [0, 31] next to
                                                  0: it covers the rest
                     'p += 1'                     [J] reject
  scev.c             takes a reference in each    [L] reject, or load
                     iteration, or in one
  iterating_         bpf_loop() is the head of    [L] verify the callback
  callbacks.c        a walked loop                in the last iteration too

In most tests the number is an index after the loop, where no jump limits
it: its range at the head decides.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/verifier.c       |   6 +
 .../bpf/progs/verifier_iterating_callbacks.c  |  53 +++
 .../selftests/bpf/progs/verifier_loop_as_is.c | 294 ++++++++++++++
 .../selftests/bpf/progs/verifier_loop_latch.c |  74 +++-
 .../selftests/bpf/progs/verifier_loop_need.c  | 371 ++++++++++++++++++
 .../selftests/bpf/progs/verifier_loop_walk.c  | 306 +++++++++++++++
 .../selftests/bpf/progs/verifier_scev.c       | 130 ++++++
 .../bpf/progs/verifier_value_ptr_arith.c      |  55 +++
 8 files changed, 1287 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_need.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_walk.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 3f3b9b48523f..978d6d7773ca 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -84,7 +84,10 @@
 #include "verifier_liveness_exp.skel.h"
 #include "verifier_load_acquire.skel.h"
 #include "verifier_loop_hierarchy.skel.h"
+#include "verifier_loop_as_is.skel.h"
 #include "verifier_loop_latch.skel.h"
+#include "verifier_loop_need.skel.h"
+#include "verifier_loop_walk.skel.h"
 #include "verifier_loops1.skel.h"
 #include "verifier_lwt.skel.h"
 #include "verifier_map_in_map.skel.h"
@@ -286,7 +289,10 @@ void test_verifier_linked_scalars(void)       { RUN(verifier_linked_scalars); }
 void test_verifier_live_stack(void)           { RUN(verifier_live_stack); }
 void test_verifier_liveness_exp(void)         { RUN(verifier_liveness_exp); }
 void test_verifier_loop_hierarchy(void)       { RUN(verifier_loop_hierarchy); }
+void test_verifier_loop_as_is(void)           { RUN(verifier_loop_as_is); }
 void test_verifier_loop_latch(void)           { RUN(verifier_loop_latch); }
+void test_verifier_loop_need(void)            { RUN(verifier_loop_need); }
+void test_verifier_loop_walk(void)            { RUN(verifier_loop_walk); }
 void test_verifier_loops1(void)               { RUN(verifier_loops1); }
 void test_verifier_lwt(void)                  { RUN(verifier_lwt); }
 void test_verifier_map_in_map(void)           { RUN(verifier_map_in_map); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
index a3e31aa25fe1..e30f7676081f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
+++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
@@ -290,6 +290,59 @@ static __naked void iter_limit_bug_cb(void)
 	);
 }
 
+static __naked __noinline __used void set_flag_cb(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"*(u64 *)(r2 + 0) = r1;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all
+	);
+}
+
+/*
+ * bpf_loop() is the head of a loop whose iterations are looked at one by one, because of r1.
+ * The callback comes back to it in the same iteration, the last one included.
+ */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callback_returns_to_loop_head(void)
+{
+	asm volatile (
+	"r7 = 0;"
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r1 = 1;"
+	"r2 = %[set_flag_cb] ll;"
+	"r3 = r10;"
+	"r3 += -8;"
+	"r4 = 0;"
+	"1:"
+	"call %[bpf_loop];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"if r1 == 0 goto 2f;"
+	"if r7 != 3 goto 2f;"
+	"r10 = 0;"
+	"2:"
+	"r7 += 1;"
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r1 = r7;"
+	"r2 = %[set_flag_cb] ll;"
+	"r3 = r10;"
+	"r3 += -8;"
+	"r4 = 0;"
+	"if r7 < 4 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_addr(set_flag_cb),
+	  __imm(bpf_loop)
+	: __clobber_all
+	);
+}
+
 /*
  * may_goto doesn't leave the loop, so it's not what stops it. The address is greater than 10:
  * the loop spins until may_goto that the verifier adds ends the prog.
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_as_is.c b/tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
new file mode 100644
index 000000000000..ce8e5b41958a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
@@ -0,0 +1,294 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The state that enters a loop that is not counted is left as it is. So is the state that ends
+ * a walk through the iterations of such a loop. What stands for many iterations is a join of states
+ * that were seen, not a guess that is written into one of them.
+ */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u64[9]);
+} map SEC(".maps");
+
+#define LOOKUP(reg)							\
+	"r1 = 0;"							\
+	"*(u32 *)(r10 - 4) = r1;"					\
+	"r2 = r10;"							\
+	"r2 += -4;"							\
+	"r1 = %[map] ll;"						\
+	"call %[bpf_map_lookup_elem];"					\
+	#reg " = r0;"							\
+	"if r0 == 0 goto 9f;"
+
+#define END								\
+"9:"									\
+	"r0 = 0;"							\
+	"exit;"								\
+	:								\
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)			\
+	: __clobber_all
+
+#define OK(text)	SEC("socket") __description("as is: " text) __success __retval(0)
+#define BAD(text)	SEC("socket") __description("as is: " text) __failure
+
+/*
+ *	do { n += 3; } while (v[0]);
+ * Two iterations are looked at as they are. The second one ends the walk: its jump goes both
+ * ways. Its state is joined with the one before, nothing stops n, so n is any number.
+ * What comes back next is within that. The number of insns tells that there is no fourth pass.
+ * The loop gets may_goto when the walk ends, and all is done again: two times 24 insns.
+ */
+OK("the state that ends a walk is joined with the one before")
+__log_level(1) __msg("processed 48 insns")
+__naked void two_passes(void)
+{
+	asm volatile (
+	LOOKUP(r7)
+	"r6 = 0;"
+"1:"
+	"r6 += 3;"
+	"r1 = *(u32 *)(r7 + 0);"
+	"if r1 != 0 goto 1b;"
+	"r6 &= 63;"
+	"r7 += r6;"
+	"r0 = *(u8 *)(r7 + 8);"
+	END);
+}
+
+/*
+ *	x = (v[0] & 1) << 62; do { x |= x >> 1; if (x > 1 << 62) break; } while (v[1]);
+ * The range at the head is 0..1 << 62 in every pass, the lower half is 0, and one more bit is not
+ * known: bit 61, 60 .. 32. That is not followed bit by bit, 30 passes and more in a nest of loops:
+ * the known bits are what the range says after the first one.
+ */
+OK("known bits are not lost one by one")
+__log_level(1) __msg("processed 74 insns")
+__naked void bits_one_by_one(void)
+{
+	asm volatile (
+	LOOKUP(r7)
+	"r6 = *(u64 *)(r7 + 0);"
+	"r6 &= 1;"
+	"r6 <<= 62;"
+	"r8 = 1;"
+	"r8 <<= 62;"
+"1:"
+	"r1 = r6;"
+	"r1 >>= 1;"
+	"r6 |= r1;"
+	"if r6 > r8 goto 9f;"
+	"r1 = *(u64 *)(r7 + 8);"
+	"if r1 != 0 goto 1b;"
+	"r6 >>= 60;"
+	"r7 += r6;"
+	"r0 = *(u8 *)(r7 + 0);"
+	END);
+}
+
+/*
+ *	if (x < 1000000) return; do { if (x && !v[2]) x--; } while (v[1]);
+ * x is 1000000..U64_MAX, then 999999..U64_MAX. As a signed number it's S64_MIN..S64_MAX all the
+ * time: no signed bound moves, and the range does, a million times if nothing is done about it.
+ */
+OK("a range moves and its signed bounds do not")
+__log_level(1) __msg("processed 68 insns")
+__naked void unsigned_bound_moves(void)
+{
+	asm volatile (
+	LOOKUP(r7)
+	"r6 = *(u64 *)(r7 + 0);"
+	"if r6 < 1000000 goto 9f;"
+"1:"
+	"if r6 == 0 goto 2f;"
+	"r1 = *(u64 *)(r7 + 16);"
+	"if r1 != 0 goto 2f;"
+	"r6 += -1;"
+"2:"
+	"r1 = *(u64 *)(r7 + 8);"
+	"if r1 != 0 goto 1b;"
+	"r6 &= 7;"
+	"r7 += r6;"
+	"r0 = *(u8 *)(r7 + 0);"
+	END);
+}
+
+/*
+ *	n = 3; while (v[0]) { if (v[0] != 1 && n) n--; }
+ * n goes from 3 to 0. The bound that moves at the head of the loop is put at 0, the nearest of the
+ * fixed stops, and that holds. Without it n goes on to fall, and n & 63 is up to 63.
+ */
+OK("a number that falls stops at 0")
+__naked void stops_at_0(void)
+{
+	asm volatile (
+	LOOKUP(r6)
+	"r7 = 3;"
+"1:"
+	"r1 = *(u64 *)(r6 + 0);"
+	"if r1 == 0 goto 3f;"
+	"if r1 == 1 goto 2f;"
+	"if r7 == 0 goto 2f;"
+	"r7 += -1;"
+"2:"
+	"goto 1b;"
+"3:"
+	"r7 &= 63;"
+	"r6 += r7;"
+	"r0 = *(u8 *)(r6 + 68);"
+	END);
+}
+
+/*
+ *	n = 0; while (v[0]) { if (v[0] != 1 && n < 8) n++; }
+ *	((u8 *)v)[n + off]
+ * n grows in some iterations only, and no edge of the jump that looks at it leaves the loop. n is
+ * an index behind the loop, where no jump has cut it: what counts is its range at the head.
+ * 7 goes on to the add, so 8 comes back at most.
+ * @load, @store: n lives in a register, or in 32 bits of a stack slot.
+ */
+#define GROWS(name, load, store, off)					\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r6)							\
+	"r7 = 0;"							\
+	store								\
+"1:"									\
+	"r1 = *(u64 *)(r6 + 0);"					\
+	"if r1 == 0 goto 3f;"						\
+	"if r1 == 1 goto 2f;"						\
+	load								\
+	"if r7 >= 8 goto 2f;"						\
+	"r7 += 1;"							\
+	store								\
+"2:"									\
+	"goto 1b;"							\
+"3:"									\
+	load								\
+	"r6 += r7;"							\
+	"r0 = *(u8 *)(r6 + " #off ");"					\
+	END);								\
+}
+
+OK("a jump of the loop stops a number at 8")
+GROWS(stops_at_8, "", "", 63)
+
+BAD("a jump of the loop stops a number at 8, one byte too far")
+__msg("invalid access to map value, value_size=72 off=72 size=1")
+GROWS(stops_at_8_bad, "", "", 64)
+
+OK("a jump of the loop stops a number in a stack slot at 8")
+GROWS(slot_stops_at_8, "r7 = *(u32 *)(r10 - 16);", "*(u32 *)(r10 - 16) = r7;", 63)
+
+/* 8 comes back to the head of the inner loop, and goes from there to the head of the outer one */
+OK("a jump of the inner loop stops a number at 8")
+int inner_loop_stops_at_8(void *ctx)
+{
+	__u32 key = 0;
+	__u64 n = 0, *v;
+	int i, j;
+
+	v = bpf_map_lookup_elem(&map, &key);
+	if (!v)
+		return 0;
+	bpf_for(i, 0, 100) {
+		bpf_for(j, 0, 100) {
+			if (*(volatile __u64 *)v == 1)
+				continue;
+			if (n < 8)
+				v[n++] = 0;
+		}
+	}
+	return v[n];
+}
+
+/*
+ *	for (i = n; i >= 0; i--) sum += v[i];
+ * A compiler counts in bytes and leaves the loop one element below the array:
+ *	off = n * 8; do { sum += *(u64 *)(v + off); off -= @step; } while (off != -8);
+ * n is a range, so the loop is not counted. Only -8 leaves it. The offset doesn't get past -8
+ * because it is a multiple of 8, and that has to be known at the head of the loop.
+ */
+#define DOWN(name, step)						\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r7)							\
+	"r6 = *(u32 *)(r7 + 64);"					\
+	"if r6 > 7 goto 9f;"						\
+	"r6 <<= 3;"							\
+"1:"									\
+	"r2 = r7;"							\
+	"r2 += r6;"							\
+	"r0 = *(u64 *)(r2 + 0);"					\
+	"r6 += -" #step ";"						\
+	"if r6 != -8 goto 1b;"						\
+	END);								\
+}
+
+OK("down by 8 bytes, left at -8")
+DOWN(down_by_8, 8)
+
+/* 16, 4, -8 is fine, 8, -4 is not */
+BAD("down by 12 bytes, left at -8")
+__msg("min value is negative")
+DOWN(down_by_12, 12)
+
+static __naked __noinline __used void nothing(void)
+{
+	asm volatile ("r0 = 0; exit;" ::: __clobber_all);
+}
+
+/*
+ *	if (v[0]) { n = 7; have = 1; }
+ *	nothing();
+ *	i = 0; do { if (v[0] != 3) i++; } while (i < 8 && v[0] != 2);
+ *	if (have) return n;
+ * (have = 0, n is not set) and (have = 1, n = 7) wait for the function. What comes out of it goes
+ * to both at once, and the insn behind the call is the head of the loop: two walks begin, and no
+ * insn is looked at in between. The state that ends one of them is joined with a state of
+ * that walk. With a state of the other one it would be (have = 0 or 1, n is not set).
+ */
+#define TWO_WALKS(name, test)						\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r8)							\
+	"r9 = 0;"							\
+	"r1 = *(u32 *)(r8 + 0);"					\
+	"if r1 != 0 goto 1f;"						\
+	"r7 = 0;"							\
+	"goto 2f;"							\
+"1:"									\
+	"r6 = 7;"							\
+	"r7 = 1;"							\
+"2:"									\
+	"call nothing;"							\
+"3:"									\
+	"r1 = *(u32 *)(r8 + 0);"					\
+	"if r1 == 3 goto +1;"						\
+	"r9 += 1;"							\
+	"if r9 > 7 goto 4f;"						\
+	"if r1 == 1 goto 3b;"						\
+	"if r1 != 2 goto 3b;"						\
+"4:"									\
+	test								\
+	"r0 = r6;"							\
+	"exit;"								\
+	END);								\
+}
+
+OK("two walks begin at the same time")
+TWO_WALKS(two_walks_at_once, "if r7 == 0 goto 9f;")
+
+/* n is read when it is not set */
+BAD("two walks begin at the same time, the register is not set")
+__msg("R6 !read_ok")
+TWO_WALKS(two_walks_at_once_bad, "if r7 != 0 goto 9f;")
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
index e8daf791b4c0..f8c2eb4e9866 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -21,6 +21,16 @@ struct {
 	__type(value, char[NR]);
 } map SEC(".maps");
 
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+
+/* BTF of a kfunc that only asm refers to is not generated */
+void __kfunc_btf_root(void)
+{
+	bpf_rcu_read_lock();
+	bpf_rcu_read_unlock();
+}
+
 #define OK(text, count)								\
 	SEC("socket") __description("latch: " text) __success __log_level(2)	\
 	__msg("header_count is " count)
@@ -53,7 +63,8 @@ __naked void name(void)								\
 9:	r0 = 0;									\
 	exit;									\
 "	:									\
-	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map), \
+	  __imm(bpf_rcu_read_lock), __imm(bpf_rcu_read_unlock)			\
 	: __clobber_all);							\
 }
 
@@ -144,7 +155,48 @@ OK("a flag, a call behind it", "[0..2048]")
 LOOP(flag_call, "w8 = -2048;",
      "w8 += 1; w9 = 1; if w8 == 0 goto 2f; w9 = 0; 2: call %[bpf_get_prandom_u32]; if w9 != 1 goto 1b;")
 
-/* in these the loop goes on when the counter gets to 0 */
+OK("a flag that is set in front of another jump", "[0..2048]")
+LOOP(flag_far, "w8 = -2048;",
+     "w8 += 1; w2 = 1; if r7 != 0 goto 3f; r3 = r6;"
+     "3: if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
+
+/*
+ * 'do { ...; more = ++i < N && cond; work; } while (more);': a compiler keeps the flag when the
+ * work is more than some 10 insns. The flag has two reasons to be 0.
+ */
+#define MORE(count, mid)							\
+	count "w9 = 1; if w7 != 0 goto 2f; w9 = 0; 2: " mid "w9 &= 1; if w9 != 0 goto 1b;"
+#define UP_TO(n)	"w8 += 1; w9 = 0; if w8 > " #n " goto 2f;"
+#define WORK4		"r1 = *(u8 *)(r10 - 4); r1 += 1; *(u8 *)(r10 - 4) = r1; r1 = r6;"
+#define WORK20		WORK4 WORK4 WORK4 WORK4 WORK4
+#define RND		"call %[bpf_get_prandom_u32];"
+
+OK("more: work", "[0..2048]")
+LOOP(more_work, "w8 = 0;", MORE(UP_TO(2047), WORK20))
+
+BAD("more: work, one too many")
+LOOP(more_work_bad, "w8 = 0;", MORE(UP_TO(2048), WORK20))
+
+OK("more: work with a jump in it", "[0..2048]")
+LOOP(more_jump, "w8 = 0;", MORE(UP_TO(2047), WORK4 "if r7 > 5 goto 4f;" WORK20 "4:"))
+
+OK("more: work with calls in it", "[0..2048]")
+LOOP(more_calls, "w8 = 0;", MORE(UP_TO(2047), RND WORK20 RND))
+
+OK("more: the counter is spilled", "[0..2048]")
+LOOP(more_spill, "r8 = 0; *(u64 *)(r10 - 16) = r8;",
+     MORE("r8 = *(u64 *)(r10 - 16); w8 += 1; w9 = 0; *(u64 *)(r10 - 16) = r8;"
+	  "if w8 > 2047 goto 2f;", RND "w8 = w0;" WORK20 RND))
+
+#define NOT_TAKEN(n)	"w8 += 1; if w8 < " #n " goto 4f; w9 = 0; goto 2f; 4:"
+
+OK("more: the edge that is not taken leaves", "[0..2048]")
+LOOP(more_not_taken, "w8 = 0;", MORE(NOT_TAKEN(2048), WORK20))
+
+BAD("more: the edge that is not taken leaves, one too many")
+LOOP(more_not_taken_bad, "w8 = 0;", MORE(NOT_TAKEN(2049), WORK20))
+
+/* in these the loop goes on when the counter gets to its last value */
 NO_COUNT("a flag that a call overwrites")
 LOOP(flag_call_r0, "w8 = -2048;",
      "w8 += 1; w0 = 1; if w8 == 0 goto 2f; w0 = 0; 2: call %[bpf_get_prandom_u32]; if w0 != 1 goto 1b;")
@@ -158,6 +210,24 @@ LOOP(flag_changed, "w8 = -2048;", FLAG("w2 &= w7;"))
 NO_COUNT("a flag that is set again on one path")
 LOOP(flag_other_jump, "w8 = -2048;", FLAG("if r7 != 0 goto 4f; w2 = 0; 4:"))
 
+NO_COUNT("more: a back edge that doesn't test the flag")
+LOOP(more_back_edge, "w8 = 0;",
+     MORE(UP_TO(2047), "if w7 == 3 goto 5f;") "goto 9f; 5: if w8 == 2048 goto 1b;")
+
+/* A nested loop can get may_goto. It goes where a jump of that loop goes, whatever the flag is. */
+NO_COUNT("more: the way out leads through a nested loop")
+LOOP(more_nested, "w8 = 0; call %[bpf_rcu_read_lock];",
+     "w8 += 1; w9 = 0; if w8 > 2047 goto 2f; w9 = 1;"
+     "2: if w9 == 5 goto 4f; if w9 != 0 goto 3f; if w9 == 0 goto 8f; 3: goto 2b;"
+     "4: if w8 <= w7 goto 1b; 8: call %[bpf_rcu_read_unlock];")
+
+/* May_goto at the head of the first loop goes to 7 with w9 = 1. The program gets there with 0. */
+NO_COUNT("more: a flag that is a constant without may_goto")
+LOOP(more_stale, "call %[bpf_rcu_read_lock];" RND "w4 = w0; w8 = 0; w9 = 0;"
+     "5: if w4 == 0 goto 6f; w9 = 0; if w4 == 1 goto 7f; w9 = 1; goto 5b; 6: w9 = 1; goto 5b; 7:",
+     "w5 = w9; w8 += 1; if w8 > 2047 goto 2f; w5 = 1;"
+     "2: if w8 > w7 goto 8f; if w5 != 0 goto 1b; 8: call %[bpf_rcu_read_unlock];")
+
 NO_COUNT("a flag, another way to the jump that counts")
 LOOP(flag_two_ways, "w8 = -2048;",
      "w8 += 1; w2 = 0; if r7 != 0 goto 3f; w2 = 1; 3: if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_need.c b/tools/testing/selftests/bpf/progs/verifier_loop_need.c
new file mode 100644
index 000000000000..3888d8b072f2
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_need.c
@@ -0,0 +1,371 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A counted loop changes a number in a way that has no formula:
+ *	for (i = 0; i < 1000; i++) x = x * 3 + 1;
+ * The verifier can go through the iterations one by one to find out what x is. It does so only
+ * if something ahead looks at x: a jump, an access to memory, a call. Otherwise x is not
+ * known in the loop and behind it, and the loop is looked at once.
+ *
+ * What looks at a number is found ahead of the main pass, and it is a guess. A use that is missed
+ * rejects a program that would load after a walk. It never loads a program that is not safe.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* a map value of 64 bytes that holds nothing but numbers */
+u64 vals[8];
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, 1);
+} arena SEC(".maps");
+
+#define OK(text)	SEC("socket") __description("need: " text) __success
+#define BAD(text)	SEC("socket") __description("need: " text) __failure
+#define NOT_WALKED	__log_level(2) __msg("nothing looks at it")
+#define WALKED		__log_level(2) __not_msg("nothing looks at it")
+
+/* The read is not allowed */
+#define UNLESS(cond)	"if " cond " goto +1; r0 = *(u64 *)(r10 + 8);"
+
+/* r6 is 1 or 2, and is multiplied by 3 in each of @n iterations. @use comes behind the loop. */
+#define TIMES_3_IN(init, n, use)					\
+	asm volatile (							\
+	init								\
+	"call %[bpf_get_prandom_u32];"					\
+	"r6 = r0;"							\
+	"r6 &= 1;"							\
+	"r6 += 1;"							\
+	"r7 = 0;"							\
+"1:	r6 *= 3;"							\
+	"r7 += 1;"							\
+	"if r7 < " #n " goto 1b;"					\
+	use								\
+	"exit;"								\
+	:: __imm(bpf_get_prandom_u32), __imm_addr(vals) : __clobber_all)
+#define TIMES_3(n, use)	TIMES_3_IN("", n, use "r0 = 0;")
+
+OK("a number that is only stored is not followed")
+NOT_WALKED __msg("processed 11 insns")
+__naked void stored_only(void)
+{
+	TIMES_3(1000, "*(u64 *)(r10 - 8) = r6;");
+}
+
+OK("a jump looks at the number")
+WALKED
+__naked void jump(void)
+{
+	TIMES_3(5, UNLESS("r6 <= 486"));
+}
+
+BAD("a jump looks at the number, which is too large")
+__msg("invalid read from stack")
+__naked void jump_too_large(void)
+{
+	TIMES_3(5, UNLESS("r6 <= 485"));
+}
+
+OK("the number is an offset in a map value that holds numbers")
+WALKED
+__naked void value_offset(void)
+{
+	/* 27 or 54 */
+	TIMES_3(3, "r1 = %[vals] ll; r1 += r6; r0 = *(u8 *)(r1 + 9);");
+}
+
+#define VALUE_AT(reg, off)	"r1 = %[vals] ll; r1 += " #reg "; r0 = *(u8 *)(r1 + " #off ");"
+
+__naked __noinline __used
+static unsigned long value_at_plus_9(void)
+{
+	asm volatile ("r2 = r1;" VALUE_AT(r2, 9) "exit;" :: __imm_addr(vals) : __clobber_all);
+}
+
+OK("the number is an offset in a map value in a function")
+WALKED
+__naked void value_offset_argument(void)
+{
+	TIMES_3(3, "r1 = r6; call value_at_plus_9;");
+}
+
+__naked __noinline __used
+static unsigned long times_27(void)
+{
+	TIMES_3_IN("", 3, "r0 = r6;");
+}
+
+OK("what a function returns is an offset in a map value")
+WALKED
+__naked void value_offset_returned(void)
+{
+	asm volatile ("call times_27;" VALUE_AT(r0, 9) "exit;"
+		      :: __imm_addr(vals) : __clobber_all);
+}
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+struct pair {
+	__u64 lo;	/* r0 */
+	__u64 hi;	/* r2 */
+};
+
+__naked __noinline __used
+static struct pair zero_and_times_27(void)
+{
+	TIMES_3_IN("", 3, "r0 = 0; r2 = r6;");
+}
+
+OK("the second half of what a function returns is an offset in a map value")
+__load_if_JITed() WALKED
+__naked void value_offset_returned_in_r2(void)
+{
+	asm volatile ("call zero_and_times_27;" VALUE_AT(r2, 9) "exit;"
+		      :: __imm_addr(vals) : __clobber_all);
+}
+#endif
+
+__naked __noinline __used
+static unsigned long times_27_there(void)
+{
+	TIMES_3_IN("r8 = r1;", 3, "*(u64 *)(r8 + 0) = r6; r0 = 0;");
+}
+
+#define TIMES_27_AT_FP_8	"r1 = r10; r1 += -8; call times_27_there; r2 = *(u64 *)(r10 - 8);"
+
+OK("a function stores the number in the stack of its caller")
+WALKED
+__naked void stored_for_the_caller(void)
+{
+	asm volatile (TIMES_27_AT_FP_8 VALUE_AT(r2, 9) "exit;"
+		      :: __imm_addr(vals) : __clobber_all);
+}
+
+OK("the number is an offset in the stack")
+WALKED
+__naked void stack_offset(void)
+{
+	/* fp-9 or fp-18 */
+	TIMES_3(2, "*(u64 *)(r10 - 8) = r7; *(u64 *)(r10 - 16) = r7; *(u64 *)(r10 - 24) = r7;"
+		   "r6 = -r6; r1 = r10; r1 += r6; r0 = *(u8 *)(r1 + 0);");
+}
+
+__naked __noinline __used
+static unsigned long times_9(void)
+{
+	TIMES_3_IN("", 2, "r0 = r6;");
+}
+
+/* fp-9 or fp-18, plus @off */
+#define STACK_AT_MINUS_R0(off)						\
+	"*(u64 *)(r10 - 8) = r0; *(u64 *)(r10 - 16) = r0; *(u64 *)(r10 - 24) = r0;"	\
+	"r0 = -r0; r1 = r10; r1 += r0; r0 = *(u8 *)(r1 + " #off ");"
+
+OK("what a function returns is an offset in the stack")
+WALKED
+__naked void stack_offset_returned(void)
+{
+	asm volatile ("call times_9;" STACK_AT_MINUS_R0(0) "exit;" ::: __clobber_all);
+}
+
+/* r6 is 1, and is multiplied by 3 in each of 2 iterations */
+#define NINE(use)							\
+	asm volatile (							\
+	"r6 = 1;"							\
+	"r7 = 0;"							\
+"1:	r6 *= 3;"							\
+	"r7 += 1;"							\
+	"if r7 < 2 goto 1b;"						\
+	use								\
+	"r0 = 0;"							\
+	"exit;"								\
+	:: __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard),	\
+	   __imm_addr(ringbuf) : __clobber_all)
+
+OK("the number is part of a size that has to be a constant")
+WALKED
+__naked void constant_argument(void)
+{
+	NINE("r1 = %[ringbuf] ll; r2 = 7; r2 += r6; r3 = 0; call %[bpf_ringbuf_reserve];"
+	     "if r0 == 0 goto +3; r1 = r0; r2 = 0; call %[bpf_ringbuf_discard];");
+}
+
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+/* Nothing checks where a pointer to arena points to */
+OK("an address in arena is not followed")
+NOT_WALKED
+__naked void arena_address(void)
+{
+	asm volatile (
+	"r1 = %[arena] ll;"
+	"r6 = 1;"
+	"r7 = 0;"
+"1:	r1 = addr_space_cast(r6, 0x0, 0x1);"
+	"r0 = *(u64 *)(r1 + 0);"
+	"r6 *= 3;"
+	"r7 += 1;"
+	"if r7 < 1000 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(arena) : __clobber_all);
+}
+#endif
+
+/*
+ *	p = NULL; for (i = 0; i < 3; i++) { if (have) n += *p; p = &val; have = 1; }
+ * No jump and no add looks at p.
+ */
+OK("the base of a load is 0 in the first iteration")
+WALKED
+__naked void base_of_a_load(void)
+{
+	asm volatile (
+	"r6 = %[vals] ll;"
+	"r7 = 0;"
+	"r8 = 0;"
+	"r9 = 0;"
+"1:	if r9 == 0 goto 2f;"
+	"r0 = *(u64 *)(r8 + 0);"
+"2:	r8 = r6;"
+	"r9 = 1;"
+	"r7 += 1;"
+	"if r7 < 3 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(vals) : __clobber_all);
+}
+
+/* r8 goes on by 0 or 8 bytes. Nothing tells how far but the iterations. */
+OK("a pointer that moves is followed")
+WALKED
+__naked void pointer(void)
+{
+	asm volatile (
+	"r8 = %[vals] ll;"
+	"r7 = 0;"
+"1:	r1 = r7;"
+	"r1 &= 1;"
+	"r1 <<= 3;"
+	"r8 += r1;"
+	"r7 += 1;"
+	"if r7 < 14 goto 1b;"
+	"r0 = *(u64 *)(r8 + 0);"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(vals) : __clobber_all);
+}
+
+/* The sets have 64 slots of the stack. What is deeper is followed. */
+OK("a jump looks at a slot below fp-512")
+__load_if_large_stack()
+WALKED
+__naked void deep_slot(void)
+{
+	asm volatile (
+	"r6 = 2;"
+	"*(u64 *)(r10 - 520) = r6;"
+	"r7 = 0;"
+"1:	r6 = *(u64 *)(r10 - 520);"
+	"r6 *= 3;"
+	"*(u64 *)(r10 - 520) = r6;"
+	"r7 += 1;"
+	"if r7 < 5 goto 1b;"
+	"r6 = *(u64 *)(r10 - 520);"
+	UNLESS("r6 == 486")
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* The iterator is not a number. It is at its end after 3 of 5 iterations. */
+OK("a slot that holds an iterator is followed")
+WALKED
+__naked void iterator_slot(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"r2 = 0;"
+	"r3 = 3;"
+	"call %[bpf_iter_num_new];"
+	"r7 = 0;"
+"1:	r1 = r10;"
+	"r1 += -8;"
+	"call %[bpf_iter_num_next];"
+	"r7 += 1;"
+	"if r7 < 5 goto 1b;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call %[bpf_iter_num_destroy];"
+	"r0 = 0;"
+	"exit;"
+	:: __imm(bpf_iter_num_new), __imm(bpf_iter_num_next), __imm(bpf_iter_num_destroy)
+	: __clobber_all);
+}
+
+/*
+ *	for (i = 0; i < n; i++) { if (i != skip) sum += prev; prev = (x ^ i) * 3; }
+ * prev is not set when the loop is entered. A compiler doesn't set what is not read.
+ */
+#define SUM_OF_PREV(skip, n, use)					\
+	asm volatile (							\
+	"call %[bpf_get_prandom_u32];"					\
+	"r6 = r0;"							\
+	"r7 = 0;"							\
+	"r9 = 0;"							\
+"1:	if r7 == " #skip " goto 2f;"					\
+	"r9 += r8;"							\
+"2:	r8 = r6;"							\
+	"r8 ^= r7;"							\
+	"r8 *= 3;"							\
+	"r7 += 1;"							\
+	"if r7 < " #n " goto 1b;"					\
+	use								\
+	"r0 = 0;"							\
+	"exit;"								\
+	:: __imm(bpf_get_prandom_u32), __imm_addr(vals) : __clobber_all)
+
+OK("what is not set at the entry is set from the second iteration on")
+NOT_WALKED __msg("processed 20 insns")
+__naked void set_in_the_first_iteration(void)
+{
+	SUM_OF_PREV(0, 40, "*(u64 *)(r10 - 8) = r9;");
+}
+
+BAD("what is not set at the entry is read in the first iteration")
+__msg("R8 !read_ok")
+__naked void read_in_the_first_iteration(void)
+{
+	SUM_OF_PREV(1, 40, "*(u64 *)(r10 - 8) = r9;");
+}
+
+/* What a call returns is made anew in every iteration, like what is loaded */
+OK("a long loop carries what a call returns")
+int call_in_a_long_loop(void *ctx)
+{
+	u64 sum = 0;
+	u32 x = 0;
+	int i;
+
+	for (i = 0; i < 20000; i++) {
+		sum += vals[x];
+		x = bpf_get_prandom_u32() & 7;
+	}
+	return sum;
+}
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+	bpf_iter_num_new(0, 0, 0);
+	bpf_iter_num_next(0);
+	bpf_iter_num_destroy(0);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_walk.c b/tools/testing/selftests/bpf/progs/verifier_loop_walk.c
new file mode 100644
index 000000000000..1a6180aceacc
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_walk.c
@@ -0,0 +1,306 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A loop that is not counted is looked at iteration by iteration for a while: some thousand insns.
+ * Each such walk counts the insns that are looked at in it, not those of the walks next to it.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, u32);
+	__type(value, char[2048]);
+} map SEC(".maps");
+
+/* 0 and 2 stay apart up to the jump that looks at them */
+#define SET(slot)								\
+	"call %[bpf_get_prandom_u32]; r1 = 0; if r0 == 0 goto +1; r1 = 2;"	\
+	"*(u64 *)(r10 - " #slot ") = r1;"
+#define TEST(slot)	"r1 = *(u64 *)(r10 - " #slot "); if r1 == 1 goto 9f;"
+#define PAD4		"r3 = r6; r3 += 1; r4 = r3; r4 += 1;"
+
+/*
+ * 32 states enter the loop one after the other. The counter has 8 bits, which has no formula. The
+ * offset is 1, 2, 4 ...: only a walk tells that it stays in the value. One walk takes 300 insns.
+ */
+#define WALKS(name, trips)							\
+__naked void name(void)								\
+{										\
+	asm volatile ("								\
+	r1 = 0;									\
+	*(u32 *)(r10 - 4) = r1;							\
+	r2 = r10;								\
+	r2 += -4;								\
+	r1 = %[map] ll;								\
+	call %[bpf_map_lookup_elem];						\
+	if r0 == 0 goto 9f;							\
+	r6 = r0;								\
+"	SET(16) SET(24) SET(32) SET(40) SET(48) "				\
+	w8 = 0;									\
+	r9 = 1;									\
+1:	r1 = r6;								\
+	r1 += r9;								\
+	r2 = 0;									\
+	*(u8 *)(r1 + 0) = r2;							\
+	r9 <<= 1;								\
+"	PAD4 PAD4 PAD4 PAD4 "							\
+	w8 += 1;								\
+	w2 = w8;								\
+	w2 &= 255;								\
+	if w2 < " #trips " goto 1b;						\
+"	TEST(16) TEST(24) TEST(32) TEST(40) TEST(48) "				\
+9:	r0 = 0;									\
+	exit;									\
+"	:									\
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __clobber_all);							\
+}
+
+SEC("socket")
+__description("walk: 32 walks at a time")
+__success
+WALKS(walks_32, 11)
+
+/*
+ *	for (i = 7; i >= 0; i--) { if (i >= n) continue; a[i] = 0; }
+ * A compiler tests 'i >= n' in front of the loop too and jumps into the middle of it. Such a loop
+ * has no count. It's walked to its end whichever way its jumps go: no may_goto is added.
+ */
+SEC("socket")
+__description("walk: a loop with two entries")
+__success __retval(0)
+__arch_x86_64
+__arch_arm64
+__xlated("w3 += -1")
+__xlated("if w3 s< 0x0 goto pc+2")
+__xlated("if w3 < w2 goto pc-6")
+__xlated("goto pc-4")
+__xlated("exit")
+__naked void two_entries(void)
+{
+	asm volatile ("								\
+	call %[bpf_get_prandom_u32];						\
+	r2 = r0;								\
+	r2 &= 7;								\
+	r0 = 0;									\
+	w3 = 7;									\
+	if w3 >= w2 goto 2f;							\
+1:	r1 = r10;								\
+	r1 += r3;								\
+	*(u8 *)(r1 - 8) = r0;							\
+2:	w3 += -1;								\
+	if w3 s< 0 goto 3f;							\
+	if w3 < w2 goto 1b;							\
+	goto 2b;								\
+3:	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	for (i = 0; (u8)i < 11; i++) { if (!rnd()) break; v[1 << i] = 0; }
+ * One jump that leaves the loop goes both ways in each iteration, the other one is decided in
+ * each iteration: the loop counts. The walk goes on. Only a walk tells what the offset is.
+ */
+SEC("socket")
+__description("walk: goes on while an iteration decides a jump that leaves the loop")
+__success
+__naked void walk_goes_on(void)
+{
+	asm volatile ("								\
+	r1 = 0;									\
+	*(u32 *)(r10 - 4) = r1;							\
+	r2 = r10;								\
+	r2 += -4;								\
+	r1 = %[map] ll;								\
+	call %[bpf_map_lookup_elem];						\
+	if r0 == 0 goto 9f;							\
+	r6 = r0;								\
+	w8 = 0;									\
+	r9 = 1;									\
+1:	call %[bpf_get_prandom_u32];						\
+	if r0 == 0 goto 9f;							\
+	r1 = r6;								\
+	r1 += r9;								\
+	*(u8 *)(r1 + 0) = r0;							\
+	r9 <<= 1;								\
+	w8 += 1;								\
+	w2 = w8;								\
+	w2 &= 255;								\
+	if w2 < 11 goto 1b;							\
+9:	r0 = 0;									\
+	exit;									\
+"	:									\
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __clobber_all);
+}
+
+static __naked __noinline __used int zero_or_one(void)
+{
+	asm volatile ("								\
+	call %[bpf_get_prandom_u32];						\
+	r0 &= 1;								\
+	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	for (i = 0; i < 2; i++) off += zero_or_one();
+ * A compiler makes a flag of i: there is nothing to count. Both trips are walked although the loop
+ * calls a function. Only a walk tells that off is 2 at most.
+ */
+#define TWO_TRIPS(name, size)							\
+__naked void name(void)								\
+{										\
+	asm volatile ("								\
+	r6 = 1;									\
+	r7 = 0;									\
+1:	call zero_or_one;							\
+	r7 += r0;								\
+	r1 = r6;								\
+	r6 = 0;									\
+	if r1 != 0 goto 1b;							\
+	r1 = r10;								\
+	r1 += r7;								\
+	r0 = 0;									\
+	*(u8 *)(r1 - " #size ") = r0;						\
+	exit;									\
+"	::: __clobber_all);							\
+}
+
+SEC("socket")
+__description("walk: two trips of a loop with a call")
+__success __retval(0)
+TWO_TRIPS(two_trips, 3)
+
+SEC("socket")
+__description("walk: two trips of a loop with a call, off by one")
+__failure __msg("invalid variable-offset write to stack R1")
+TWO_TRIPS(two_trips_off_by_one, 2)
+
+/*
+ *	while (rnd()) n++;
+ * The second iteration decides nothing. The walk gives up at once: less than 100 insns.
+ */
+SEC("socket")
+__description("walk: gives up when an iteration decides no jump that leaves the loop")
+__success __log_level(4) __msg("processed {{..}} insns")
+__naked void walk_gives_up(void)
+{
+	asm volatile ("								\
+	r6 = 0;									\
+1:	call %[bpf_get_prandom_u32];						\
+	if r0 == 0 goto 2f;							\
+	r6 += 1;								\
+	goto 1b;								\
+2:	r0 = 0;									\
+	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	do { n++; more = rnd(); ... } while (more);
+ * A compiler keeps the flag. States with 0 and with 1 are not joined in front of its test, so the
+ * test is decided in each of them. That decides nothing: the walk gives up as above.
+ */
+SEC("socket")
+__description("walk: gives up when only a flag that went both ways decides the jump")
+__success __log_level(4) __msg("processed {{..}} insns")
+__naked void walk_gives_up_flag(void)
+{
+	asm volatile ("								\
+	r6 = 0;									\
+1:	call %[bpf_get_prandom_u32];						\
+	r6 += 1;								\
+	w7 = 1;									\
+	if r0 != 0 goto 2f;							\
+	w7 = 0;									\
+2:	w7 &= 1;								\
+	if w7 != 0 goto 1b;							\
+	r0 = 0;									\
+	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	for (off = 1; !done; off <<= 1) { v[off] = 0; done = off > 512; if (rnd()) n++; }
+ * done is known in every iteration. Another flag went both ways, and was tested before it: the
+ * walk goes on. Only a walk tells what the offset is.
+ */
+SEC("socket")
+__description("walk: goes on when the flag that leaves the loop is known")
+__success
+__naked void walk_goes_on_flag(void)
+{
+	asm volatile ("								\
+	r1 = 0;									\
+	*(u32 *)(r10 - 4) = r1;							\
+	r2 = r10;								\
+	r2 += -4;								\
+	r1 = %[map] ll;								\
+	call %[bpf_map_lookup_elem];						\
+	if r0 == 0 goto 9f;							\
+	r6 = r0;								\
+	r9 = 1;									\
+1:	r1 = r6;								\
+	r1 += r9;								\
+	r2 = 0;									\
+	*(u8 *)(r1 + 0) = r2;							\
+	w8 = 1;									\
+	if r9 > 512 goto 2f;							\
+	w8 = 0;									\
+2:	r9 <<= 1;								\
+	call %[bpf_get_prandom_u32];						\
+	w7 = 1;									\
+	if r0 != 0 goto 3f;							\
+	w7 = 0;									\
+3:	if w7 == 0 goto 4f;							\
+	r2 = 0;									\
+4:	if w8 == 0 goto 1b;							\
+9:	r0 = 0;									\
+	exit;									\
+"	:									\
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __clobber_all);
+}
+
+static __naked __noinline __used int long_loop(void)
+{
+	asm volatile ("								\
+	w3 = 0;									\
+1:	w3 += 1;								\
+	w2 = w3;								\
+	w2 &= 0xffff;								\
+	if w2 < 5000 goto 1b;							\
+	r0 = 0;									\
+	*(u64 *)(r1 + 0) = r0;							\
+	exit;									\
+"	::: __clobber_all);
+}
+
+/*
+ * The walk doesn't get to the end of the loop. The function is looked at three times, for three
+ * pointers. There is no walk the second and the third time: less than 10000 insns in all.
+ */
+SEC("socket")
+__description("walk: no other try for a loop that is too long")
+__success __log_level(4) __msg("processed {{....}} insns")
+__naked void walk_fails_once(void)
+{
+	asm volatile ("								\
+	r1 = r10;								\
+	r1 += -8;								\
+	call long_loop;								\
+	r1 = r10;								\
+	r1 += -16;								\
+	call long_loop;								\
+	r1 = r10;								\
+	r1 += -24;								\
+	call long_loop;								\
+	r0 = 0;									\
+	exit;									\
+"	::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 5e566933f7e4..ace2c1676919 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -4411,6 +4411,136 @@ __naked void post_cond_jlt32_copy(void)
 	: __clobber_all);
 }
 
+/*
+ * The loop loads r6 again in every iteration. It's 0 at the entry and a byte in all iterations
+ * but the first one: the first iteration is verified on its own, the state that comes back to
+ * the head stands for the rest.
+ */
+SEC("socket")
+__success __retval(0)
+__log_level(2)
+__msg("header_count is 10 ")
+__msg("can't widen r6, expr is _")
+__not_msg("widening r8")
+__msg("clamping r8 to 0..9 step 1")
+__naked void index_loaded_in_previous_iteration(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r7 = r0;					\
+	r6 = 0;						\
+	r8 = 0;						\
+1:	r1 = r7;					\
+	r1 += r6;					\
+	r0 = *(u8 *)(r1 + 0);				\
+	r6 = *(u8 *)(r7 + 0);				\
+	r8 += 1;					\
+	if r8 < 10 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * What the first iteration makes of r6 is a guess about the rest. A byte in the first
+ * iteration, 16 bits in the others: the state that comes back to the head next is not within it.
+ */
+SEC("socket")
+__failure __msg("R1 unbounded memory access")
+__naked void first_iteration_loads_less(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 4f;				\
+	r7 = r0;					\
+	r6 = 0;						\
+	r8 = 0;						\
+1:	r1 = r7;					\
+	r1 += r6;					\
+	r0 = *(u8 *)(r1 + 0);				\
+	if r8 != 0 goto 2f;				\
+	r6 = *(u8 *)(r7 + 0);				\
+	goto 3f;					\
+2:	r6 = *(u16 *)(r7 + 0);				\
+3:	r8 += 1;					\
+	if r8 < 10 goto 1b;				\
+4:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * Every iteration takes a reference, only the last one is released. The state that comes back
+ * to the head of the loop has a reference that nothing holds.
+ */
+SEC("socket")
+__failure __msg("Unreleased reference")
+__naked void loop_takes_reference_every_time(void)
+{
+	asm volatile (
+	"r6 = 0;"
+"1:"
+	"r1 = %[ringbuf] ll;"
+	"r2 = 8;"
+	"r3 = 0;"
+	"call %[bpf_ringbuf_reserve];"
+	"if r0 == 0 goto 2f;"
+	"r8 = r0;"
+	"r6 += 1;"
+	"if r6 < 10 goto 1b;"
+	"r1 = r8;"
+	"r2 = 0;"
+	"call %[bpf_ringbuf_discard];"
+"2:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard), __imm_addr(ringbuf)
+	: __clobber_all);
+}
+
+/*
+ * The first iteration that has something to say takes a reference, the rest use it. The state
+ * that comes back to the head has a reference more than the state that entered, once.
+ */
+__u64 nr_vals, vals[64];
+
+SEC("socket")
+__success __retval(0)
+int loop_takes_reference_once(void *ctx)
+{
+	__u64 *sum = NULL;
+	int i;
+
+	bpf_for(i, 0, nr_vals) {
+		if (i >= 64 || !vals[i])
+			continue;
+		if (!sum) {
+			sum = bpf_ringbuf_reserve(&ringbuf, sizeof(*sum), 0);
+			if (!sum)
+				break;
+			*sum = 0;
+		}
+		*sum += vals[i];
+	}
+	if (sum)
+		bpf_ringbuf_submit(sum, 0);
+	return 0;
+}
+
 /*
  * r6 is a pointer to ringbuf memory with a step of 1. bpf_ringbuf_discard() in the first iteration
  * makes a number of it. The offsets 0..1 of the pointer are not the range of this number: it's an
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
index f9ad07704578..260cbc22fbc4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
@@ -859,6 +859,61 @@ l0_%=:	r0 = 1;						\
 	: __clobber_all);
 }
 
+/*
+ * A list in a map value: it starts at element 0 and goes on with the element that the current one
+ * names. The offset is 0 in front of the loop and [0, 31] after every trip. The second of
+ * these states at the head of the loop stands for all that follow.
+ */
+SEC("tc")
+__description("map access: the offset of a value pointer grows in the first trip of a loop")
+__success
+__naked void loop_offset_settles(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r2 = r10; r2 += -8;"
+	"r1 = %[map_array_48b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 2f;"
+	"r6 = r0;"
+	"r2 = r6;"
+"1:	r5 = *(u8 *)(r2 + 0);"
+	"r5 &= 31;"
+	"r2 = r6; r2 += r5;"
+	"may_goto 2f;"
+	"goto 1b;"
+"2:	r0 = 0; exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_array_48b)
+	: __clobber_all);
+}
+
+SEC("tc")
+__description("map access: the offset of a value pointer grows in every trip of a loop")
+__failure __msg("invalid access to map value, value_size=48 off=48 size=1")
+__naked void loop_offset_grows(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r2 = r10; r2 += -8;"
+	"r1 = %[map_array_48b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 2f;"
+	"r2 = r0;"
+"1:	r5 = *(u8 *)(r2 + 0);"
+	"r2 += 1;"
+	"may_goto 2f;"
+	"goto 1b;"
+"2:	r0 = 0; exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_array_48b)
+	: __clobber_all);
+}
+
 SEC("socket")
 __description("map access: value_ptr += known scalar, 4")
 __success __success_unpriv __retval(1)
-- 
2.55.0

[PATCH bpf-next v3 44/45] selftests/bpf: Test progs that run out of budget

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier does not prove that a prog ends. It adds may_goto to each loop
that it cannot bound, and a check to the entry of each recursive subprog.
A prog is out of budget when may_goto times out or the check fails. No
other test gets there: a mistake in the added code rejects nothing, it
locks up a cpu, or the prog goes on along an edge that nobody verified.
Load and run such progs. Each invocation must return, and the return value
shows where the prog went on.

  [G] "bpf: Add may_goto to the loops that the verifier marks"
  [P] "bpf: Order the insns, find what stops a loop and where may_goto
      can go"
  [F] "bpf: Verify programs as a fixpoint over the control flow graph"
  [R] "bpf: Limit depth and time of recursive subprogs at run time",
      "bpf: Allow recursive subprogs"
  [E] "bpf: R0 is live at the exit that an added may_goto goes to"

  the prog                                        out of budget
  ----------------------------------------------  -------------------------
  a loop, the prog holds nothing                  [G] the prog ends
  the same, the last exit is dead code            [E] the prog ends
  a loop under a lock                             [G] the loop exits
  the same, the code after it needs p != NULL     [G] reject
  a loop in a subprog                             [G] the loop exits
  the same, the call has unwind_goto              [G] the subprog unwinds
  a landing pad does not release a lock           [G] reject
  all registers are live at the added may_goto    [G] none of them changes
  a loop has no exit, a nest of 17 loops          [F] [P] reject
  may_goto of the prog does not leave the loop,   [P] may_goto at the head,
  a jump to itself, a loop with two entries       the prog ends
  an iterator does not stop the loop around it    [F] the prog ends
  recursion is too deep, no unwind_goto           [R] the subprog returns 0
  the kernel stack where such a recursion ends    [R] a half of it is free
  recursion is too deep, calls have unwind_goto   [R] the landing pad runs
  recursion is not deep, but makes 2^60 calls     [R] the prog returns
  recursion through bpf_loop(), or in a loop      [R] the prog returns
  recursion through a global subprog and a table  [R] reject
  a timer callback arms itself again              [R] no recursion

The edge of an added may_goto is an edge of the prog. 10 more progs are
unsafe only on this edge: a register that only the loop sets, a bound of
the next loop, a loop that is counted for one state or call and gets
may_goto for a different one ([P], [F]). The verifier must reject them.

verifier_budget.c is serial, like timed_may_goto: its progs spin for 250 ms
in an RCU read-side critical section, one after the other. It takes 5 sec.
verifier_callx.c gets a test of packet pointers after callx of an argument.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/verifier.c       |    3 +
 .../selftests/bpf/progs/verifier_budget.c     | 1898 +++++++++++++++++
 .../selftests/bpf/progs/verifier_callx.c      |   33 +
 .../bpf/progs/verifier_callx_rodata.c         |   51 +
 4 files changed, 1985 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_budget.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 978d6d7773ca..71ccd1e94bb2 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -31,6 +31,7 @@
 #include "verifier_bpf_get_stack.skel.h"
 #include "verifier_bpf_trap.skel.h"
 #include "verifier_bswap.skel.h"
+#include "verifier_budget.skel.h"
 #include "verifier_btf_array_access.skel.h"
 #include "verifier_btf_ctx_access.skel.h"
 #include "verifier_btf_flex_array.skel.h"
@@ -236,6 +237,8 @@ void test_verifier_bounds_mix_sign_unsign(void) { RUN(verifier_bounds_mix_sign_u
 void test_verifier_bpf_get_stack(void)        { RUN(verifier_bpf_get_stack); }
 void test_verifier_bpf_trap(void)             { RUN(verifier_bpf_trap); }
 void test_verifier_bswap(void)                { RUN(verifier_bswap); }
+/* Serial: many progs spin for 250 ms in an RCU read-side critical section, see timed_may_goto.c */
+void serial_test_verifier_budget(void)        { RUN(verifier_budget); }
 void test_verifier_btf_array_access(void)     { RUN(verifier_btf_array_access); }
 void test_verifier_btf_ctx_access(void)       { RUN(verifier_btf_ctx_access); }
 void test_verifier_btf_flex_array(void)       { RUN(verifier_btf_flex_array); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_budget.c b/tools/testing/selftests/bpf/progs/verifier_budget.c
new file mode 100644
index 000000000000..4118d8152f37
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_budget.c
@@ -0,0 +1,1898 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/*
+ * The verifier adds may_goto to a loop that it cannot count and a check of
+ * the stack to a recursive subprog. What the program does when it is out of budget:
+ * - it unwinds, when unwind_goto follows every call of the subprog and the subprog
+ *   holds nothing of its own. The pads of the callers are verified.
+ * - the main prog and an async callback end
+ * - otherwise it leaves the loop, or returns 0 from the subprog
+ *
+ * It takes unwinding, per-cpu insns and timed may_goto in JIT. Some of the progs never end
+ * on their own.
+ */
+#define unwind_goto(off) ".8byte %[insn_unwind_goto_" #off "];"
+#define unwind_exit ".8byte %[insn_unwind_exit];"
+
+#define __imm_unwind_goto(off) \
+	__imm_insn(insn_unwind_goto_##off, \
+		   BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, off, 0))
+#define __imm_unwind_exit \
+	__imm_insn(insn_unwind_exit, \
+		   BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+
+extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym;
+extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym;
+extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym;
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+unsigned long one = 1;
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, long);
+} empty SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 2);
+	__type(key, int);
+	__type(value, long[2]);
+} pairs SEC(".maps");
+
+SEC("socket")
+__description("budget: a loop in the main prog ends it")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in the main prog ends it, the last exit is dead code")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main_dead_exit(void)
+{
+	asm volatile (
+		"r6 = 0;"
+		"if r6 != 0 goto 2f;"
+		"r6 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+	"2:"
+		"r0 = 2;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in the main prog that holds something is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main_rcu(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"call %[bpf_rcu_read_lock];"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+/* 0 is not what this prog may return */
+SEC("cgroup/recvmsg4")
+__description("budget: a loop in the main prog that cannot return 0 is left")
+__arch_x86_64
+__arch_arm64
+__success
+__naked void loop_in_main_retval(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+/*
+ * do { p = lookup(); } while (!p); return *p;
+ * may_goto must not go where the loop goes when it's over: p may be NULL. Something is held, so
+ * it cannot end the prog either: there is no way out.
+ */
+SEC("socket")
+__description("budget: something is held and the code behind the loop counts on its exit")
+__arch_x86_64
+__arch_arm64
+__failure __msg("R0 invalid mem access 'map_value_or_null'")
+__naked void exit_of_loop_checks_null_rcu(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u32 *)(r10 - 4) = r1;"
+		"call %[bpf_rcu_read_lock];"
+	"1:"
+		"r2 = r10;"
+		"r2 += -4;"
+		"r1 = %[empty] ll;"
+		"call %[bpf_map_lookup_elem];"
+		"if r0 == 0 goto 1b;"
+		"r6 = *(u64 *)(r0 + 0);"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(empty),
+		  __imm(bpf_map_lookup_elem),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long spins(void)
+{
+	asm volatile (
+		"r2 = %[one] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(one)
+	);
+}
+
+/* the same, for the tests that call it another way */
+__naked __noinline __used
+static unsigned long spins_caught(void)
+{
+	asm volatile (
+		"r2 = %[one] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(one)
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog is left when there is no unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog(void)
+{
+	asm volatile (
+		"call spins;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog unwinds when there is unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(42)
+__naked void loop_in_subprog_caught(void)
+{
+	asm volatile (
+		"call spins_caught;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long spins_caught3(void)
+{
+	asm volatile (
+		"r2 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+	);
+}
+
+/* Nothing in the subprog unwinds. The pad is looked at because of may_goto that is added. */
+SEC("socket")
+__description("budget: the pad forgets what the caller holds")
+__arch_x86_64
+__arch_arm64
+__failure __msg("cannot be used inside bpf_rcu_read_lock-ed region")
+__naked void pad_forgets(void)
+{
+	asm volatile (
+		"call %[bpf_rcu_read_lock];"
+		"call spins_caught3;"
+		unwind_goto(4)
+		"r6 = r0;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = r6;"
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(4),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+/* There is no pad for what the subprog took itself */
+__naked __noinline __used
+static unsigned long spins_locked(void)
+{
+	asm volatile (
+		"call %[bpf_rcu_read_lock];"
+		"r2 = %[one] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog that holds something is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog_locked(void)
+{
+	asm volatile (
+		"call spins_locked;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+/* r1: how deep to go. Returns the sum of 1 .. r1 */
+__naked __noinline __used
+static unsigned long sum_to(void)
+{
+	asm volatile (
+		"if r1 != 0 goto 1f;"
+		"r0 = 0;"
+		"exit;"
+	"1:"
+		"r6 = r1;"
+		"r1 += -1;"
+		"call sum_to;"
+		"r0 += r6;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion")
+__arch_x86_64
+__arch_arm64
+__success __retval(55)
+__naked void recursion(void)
+{
+	asm volatile (
+		"r1 = 10;"
+		"call sum_to;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * The one that is called too deep returns 0: 1000 + 999 + ... down to where it was: more than
+ * 1000, less than 500500
+ */
+SEC("socket")
+__description("budget: recursion that is too deep returns 0 when there is no unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_too_deep(void)
+{
+	asm volatile (
+		"r1 = 1000;"
+		"call sum_to;"
+		"r1 = r0;"
+		"r0 = 0;"
+		"if r1 <= 1000 goto +2;"
+		"if r1 >= 500500 goto +1;"
+		"r0 = 1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+unsigned long lowest_r10;
+
+__naked __noinline __used
+static unsigned long goes_down(void)
+{
+	asm volatile (
+		"r2 = %[lowest_r10] ll;"
+		"*(u64 *)(r2 + 0) = r10;"
+		"call goes_down;"
+		"exit;"
+		:
+		: __imm_addr(lowest_r10)
+		: __clobber_all
+	);
+}
+
+/*
+ * The stack of a task is aligned to its size, 16K at least. r10 of the last frame whose body
+ * runs is right above the middle.
+ */
+SEC("socket")
+__description("budget: recursion ends in the middle of the kernel stack")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_ends_in_the_middle(void)
+{
+	asm volatile (
+		"call goes_down;"
+		"r1 = %[lowest_r10] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r1 += 64;"
+		"r1 &= 8191;"
+		"r0 = 1;"
+		"if r1 < 256 goto +1;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(lowest_r10)
+		: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long has_large_frame(void)
+{
+	asm volatile (
+		"*(u64 *)(r10 - 1024) = r1;"
+		"call has_large_frame;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* The limit on the stack of a chain of calls has the frame of a recursive subprog in it */
+SEC("socket")
+__description("budget: one frame of a recursive subprog counts in the depth of the stack")
+__arch_x86_64
+__arch_arm64
+__failure __msg("combined stack size of 2 calls is 2560. Too large")
+__naked void recursion_with_large_frames(void)
+{
+	asm volatile (
+		"*(u64 *)(r10 - 1536) = r1;"
+		"call has_large_frame;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long sum_to_caught(void)
+{
+	asm volatile (
+		"if r1 != 0 goto 1f;"
+		"r0 = 0;"
+		"exit;"
+	"1:"
+		"r6 = r1;"
+		"r1 += -1;"
+		"call sum_to_caught;"
+		unwind_goto(2)
+		"r0 += r6;"
+		"exit;"
+		"r0 = 0;"
+		unwind_exit
+		:
+		: __imm_unwind_goto(2),
+		  __imm_unwind_exit
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion with unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(55)
+__naked void recursion_caught(void)
+{
+	asm volatile (
+		"r1 = 10;"
+		"call sum_to_caught;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion that is too deep unwinds")
+__arch_x86_64
+__arch_arm64
+__success __retval(42)
+__naked void recursion_too_deep_caught(void)
+{
+	asm volatile (
+		"r1 = 1000;"
+		"call sum_to_caught;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+/* r1: how deep to go. It calls itself twice: 2^r1 calls */
+__naked __noinline __used
+static unsigned long calls_itself_twice(void)
+{
+	asm volatile (
+		"if r1 != 0 goto 1f;"
+		"r0 = 1;"
+		"exit;"
+	"1:"
+		"r6 = r1;"
+		"r1 += -1;"
+		"call calls_itself_twice;"
+		"r7 = r0;"
+		"r1 = r6;"
+		"r1 += -1;"
+		"call calls_itself_twice;"
+		"r0 += r7;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion that is not deep, 2^10 calls")
+__arch_x86_64
+__arch_arm64
+__success __retval(1024)
+__naked void recursion_tree(void)
+{
+	asm volatile (
+		"r1 = 10;"
+		"call calls_itself_twice;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* 61 frames fit in the stack. The check begins with may_goto: the time is limited too. */
+SEC("socket")
+__description("budget: recursion that is not deep, 2^60 calls")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void recursion_tree_forever(void)
+{
+	asm volatile (
+		"r1 = 60;"
+		"call calls_itself_twice;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* The time slice is for an invocation of the prog, not for the call that starts a recursion */
+SEC("socket")
+__description("budget: a loop that calls a subprog that calls itself 2^60 times")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_calls_recursion_tree(void)
+{
+	asm volatile (
+		"r8 = %[one] ll;"
+	"1:"
+		"r1 = 60;"
+		"call calls_itself_twice;"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/*
+ * Every frame loops forever and calls itself in every trip. The call that finds no room in the
+ * stack returns at once. When the time slice is over every frame that is on the stack leaves
+ * its loop.
+ */
+__naked __noinline __used
+static unsigned long calls_itself_in_a_loop(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+	"1:"
+		"call calls_itself_in_a_loop;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(one)
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog that calls itself in a loop that never ends")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_in_a_loop(void)
+{
+	asm volatile (
+		"call calls_itself_in_a_loop;"
+		"r0 = 1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * calls_loop() -> bpf_loop() -> loop_cb() -> calls_loop(): the frames of the helper are on
+ * the stack too
+ */
+static int loop_cb(__u32 i, void *ctx);
+
+static __noinline int calls_loop(int x)
+{
+	bpf_loop(1, loop_cb, NULL, zero);
+	return 0;
+}
+
+static __noinline int loop_cb(__u32 i, void *ctx)
+{
+	return calls_loop(i);
+}
+
+SEC("socket")
+__description("budget: recursion through a helper that calls a callback")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+int recursion_through_callback(void *ctx)
+{
+	return calls_loop(0);
+}
+
+struct timer {
+	struct bpf_timer t;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer);
+} timers SEC(".maps");
+
+__naked __noinline __used
+static int arms_itself(void)
+{
+	asm volatile (
+		"r1 = r3;"
+		"r2 = %[arms_itself] ll;"
+		"call %[bpf_timer_set_callback];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(arms_itself), __imm(bpf_timer_set_callback)
+		: __clobber_all
+	);
+}
+
+/* The helper keeps the callback for later. There is nothing to count, on any architecture. */
+SEC("socket")
+__description("budget: a callback that arms itself again is not a recursion")
+__success
+__xlated("...")
+__xlated("exit")
+__xlated("r1 = r3")
+__naked void callback_arms_itself(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u32 *)(r10 - 4) = r1;"
+		"r2 = r10;"
+		"r2 += -4;"
+		"r1 = %[timers] ll;"
+		"call %[bpf_map_lookup_elem];"
+		"if r0 == 0 goto 1f;"
+		"r1 = r0;"
+		"r2 = %[arms_itself] ll;"
+		"call %[bpf_timer_set_callback];"
+	"1:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(timers), __imm_addr(arms_itself), __imm(bpf_map_lookup_elem),
+		  __imm(bpf_timer_set_callback)
+		: __clobber_all
+	);
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, long);
+} hash SEC(".maps");
+
+static __noinline int sleeps_at_the_bottom(int n, void *user_ptr)
+{
+	char buf[8];
+
+	if (n)
+		return sleeps_at_the_bottom(n - 1, user_ptr) + 1;
+	return bpf_copy_from_user(buf, sizeof(buf), user_ptr);
+}
+
+static int calls_what_sleeps(void *map, void *key, void *value, void *ctx)
+{
+	return sleeps_at_the_bottom(1, (void *)zero) & 1;
+}
+
+SEC("syscall")
+__description("budget: recursion in a sleepable prog can sleep")
+__arch_x86_64
+__arch_arm64
+__success
+int recursion_sleeps(void *ctx)
+{
+	return sleeps_at_the_bottom(1, (void *)zero) & 1;
+}
+
+/* The frames of the callers are not there when a recursive subprog is verified */
+SEC("syscall")
+__description("budget: recursion in a callback that runs under rcu_read_lock() cannot sleep")
+__arch_x86_64
+__arch_arm64
+__failure __msg("sleepable helper bpf_copy_from_user#148 in callback running under rcu_read_lock")
+int recursion_sleeps_in_rcu_callback(void *ctx)
+{
+	bpf_for_each_map_elem(&hash, calls_what_sleeps, NULL, 0);
+	return 0;
+}
+
+/*
+ * may_goto of the program stops the loop that it leaves. This one goes to itself
+ * when it is out of budget. The loop spins until may_goto that is added ends the prog.
+ */
+SEC("socket")
+__description("budget: may_goto that doesn't leave the loop")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_stays_in_loop(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+	"1:"
+		".8byte %[may_goto_self];"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm_insn(may_goto_self, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, -1, 0))
+		: __clobber_all
+	);
+}
+
+/* The jump to itself has to go through may_goto that is added in front of it */
+SEC("socket")
+__description("budget: a conditional jump to itself")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void jump_to_itself(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r1 = *(u64 *)(r6 + 0);"
+	"1:"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/* A cycle with two entries: there is no insn that every way into it goes through */
+SEC("socket")
+__description("budget: a loop with two entries")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_with_two_entries(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 2f;"
+	"1:"
+		"r2 = *(u64 *)(r6 + 0);"
+		"if r2 == 0 goto 3f;"
+	"2:"
+		"r3 = *(u64 *)(r6 + 0);"
+		"if r3 != 0 goto 1b;"
+	"3:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/*
+ * bpf_iter_num_next() stops the loop that the program leaves when there is nothing left.
+ * This one goes on. The iterator is held, so the loop is left when it is out of budget.
+ */
+SEC("socket")
+__description("budget: a loop that goes on when its iterator is drained")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_after_iter_is_drained(void)
+{
+	asm volatile (
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 10;"
+		"call %[bpf_iter_num_new];"
+		"r6 = %[one] ll;"
+	"1:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy)
+		: __clobber_all
+	);
+}
+
+/*
+ * bpf_iter_num_next() stops the loops that the program entered after it made the iterator.
+ * The loop around bpf_for() makes a new one in every trip, so it gets may_goto. It ends on its own
+ * after a time that is much longer than the budget.
+ */
+unsigned long trips, sum;
+unsigned long many = 1UL << 32;
+
+SEC("socket")
+__description("budget: the iterator of bpf_for() doesn't stop the loop around it")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+int loop_around_bpf_for(void *ctx)
+{
+	volatile unsigned long *cnt = &trips, *max = &many;
+	int i;
+
+	do {
+		bpf_for(i, 0, 2)
+			sum += i;
+		(*cnt)++;
+	} while (*cnt < *max);
+	return 1;
+}
+
+/*
+ * The first time the outer loop counts on bpf_iter_num_next(): its trips are joined, r7 is not
+ * known and the store is refused. All is done again, the loop is walked, r7 is 27 and the store is
+ * dead code. Where the first time got to doesn't count.
+ */
+SEC("socket")
+__description("budget: an insn that is reached only before all is done again is dead code")
+__arch_x86_64
+__arch_arm64
+__success
+__xlated("...")
+__xlated("if r6 < 0x3 goto pc-{{[0-9]+}}")
+__xlated("r0 = 0")
+__xlated("exit")
+__naked void insn_of_dropped_round(void)
+{
+	asm volatile (
+		"r6 = 0;"
+		"r7 = 1;"
+	"1:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 10;"
+		"call %[bpf_iter_num_new];"
+	"2:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"if r0 != 0 goto 2b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		"r7 *= 3;"
+		"r6 += 1;"
+		"if r6 == 1 goto 1b;"
+		"if r6 < 3 goto 1b;"
+		"if r7 == 27 goto 3f;"
+		"r1 = 0;"
+		"*(u64 *)(r1 + 0) = r1;"
+	"3:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy)
+		: __clobber_all
+	);
+}
+
+/*
+ * The loop at 3: spins. It has no jump that every trip goes through, so may_goto is at its head.
+ * The first way out of it, from 2: to 1:, is the back edge of the loop at 1:, which
+ * has no may_goto of its own: bpf_iter_num_next() stops it. From the head of 3: to 1: there is
+ * no bpf_iter_num_next().
+ */
+SEC("socket")
+__description("budget: may_goto at the head of a loop leaves the loops around it too")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void guard_at_head_of_nested_loop(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r9 = %[zero] ll;"
+		"r7 = 0;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 2;"
+		"call %[bpf_iter_num_new];"
+		"call %[bpf_rcu_read_lock];"
+		"r7 += 1;"
+		"goto 1f;"
+	"2:"
+		"r7 += 1;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 != 0 goto 4f;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 != 0 goto 4f;"
+	"1:"
+		"r7 += 1;"
+		"r1 = *(u64 *)(r6 + 0);"
+	"5:"
+		"if r1 == 0 goto 5b;"
+	"3:"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 != 0 goto +0;"
+	"4:"
+		"r7 += 1;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 == 0 goto 3b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"if r0 == 0 goto 6f;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 == 0 goto 2b;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 4b;"
+	"6:"
+		"call %[bpf_rcu_read_unlock];"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm_addr(zero),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+#define NEST(n, body)	#n ": call %[bpf_get_prandom_u32];" body "if r0 == " #n " goto " #n "b;"
+#define NEST4(a, b, c, d, body)	NEST(a, NEST(b, NEST(c, NEST(d, body))))
+
+/* It's refused before anything goes up the nest from each of its insns */
+SEC("socket")
+__description("budget: a nest of 17 loops")
+__failure __msg("Too many nested loops at")
+__naked void nest_of_17_loops(void)
+{
+	asm volatile (
+		NEST4(1, 2, 3, 4, NEST4(5, 6, 7, 8, NEST4(9, 10, 11, 12,
+		NEST4(13, 14, 15, 16, NEST(17, "")))))
+		"r0 = 0;"
+		"exit;"
+		:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * The loops below spin and have no jump that every trip goes through, so may_goto is at the head.
+ * It goes to the code behind the loop, past what the loop makes for that code, with the registers
+ * and the stack as they are at the head.
+ *
+ * r6 is 5 on every edge of the program that gets behind the loop. At the head it's 7.
+ */
+__naked __noinline __used
+static unsigned long spins_and_makes(void)
+{
+	asm volatile (
+		"r8 = %[one] ll;"
+		"r6 = 7;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"r6 = 5;"
+		"if r1 != 2 goto 1b;"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop keeps a register that the loop sets")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void guard_at_head_keeps_reg(void)
+{
+	asm volatile (
+		"call spins_and_makes;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* Only the loop sets r6 */
+__naked __noinline __used
+static unsigned long spins_and_makes_unset(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"r6 = 5;"
+		"if r1 != 2 goto 1b;"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, a register that only the loop sets")
+__failure __msg("R6 !read_ok")
+__naked void guard_at_head_unset_reg(void)
+{
+	asm volatile (
+		"call spins_and_makes_unset;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* fp-8 is 0 on every edge of the program that gets behind the loop. It's dead at the head. */
+__naked __noinline __used
+static unsigned long spins_and_makes_slot(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+		"r7 = 1;"
+		"*(u64 *)(r10 - 8) = r7;"
+		"r7 = 0;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"*(u64 *)(r10 - 8) = r7;"
+		"if r1 != 2 goto 1b;"
+		"r0 = *(u64 *)(r10 - 8);"
+		"r0 <<= 3;"
+		"r8 += r0;"
+		"r0 = *(u64 *)(r8 + 0);"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, a stack slot that is dead there")
+__failure __msg("slot poisoned by dead code elimination")
+__naked void guard_at_head_dead_slot(void)
+{
+	asm volatile (
+		"call spins_and_makes_slot;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * The low half of r7 is live at the head of the loop, all of it where may_goto goes. On the edges
+ * of the program that get there it's 3. 'w7 &= 7' has to clear the upper half.
+ */
+__naked __noinline __used
+static unsigned long spins_with_half_of_reg(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+		"r7 = *(u64 *)(r8 + 0);"
+		"w7 &= 7;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if w1 == w7 goto 1b;"
+		"r7 = 3;"
+		"if r1 == 2 goto 2f;"
+		"w7 = 5;"
+		"goto 1b;"
+	"2:"
+		"r0 = r7;"
+		"r0 >>= 32;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, one half of a register is live there")
+__arch_x86_64
+__arch_arm64
+__flag(BPF_F_TEST_RND_HI32)
+__success __retval(0)
+__naked void guard_at_head_half_of_reg(void)
+{
+	asm volatile (
+		"call spins_with_half_of_reg;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * From the head of a loop to the code behind it is not an edge of the program. What holds on
+ * every edge of the program that gets there doesn't hold when may_goto does.
+ */
+const volatile unsigned long four = 4;
+char five_bytes[5] SEC(".data.five_bytes");
+
+/*
+ * r7 points to 'four' on every edge. At the head of the first loop it points to @at_head.
+ * The second loop goes on while r6 is less than what r7 points to.
+ */
+#define SPINS_THEN_COUNTS(at_head)			\
+		"r8 = %[zero] ll;"			\
+		"r7 = %[" #at_head "] ll;"		\
+	"1:"						\
+		"r2 = *(u64 *)(r7 + 0);"		\
+		"r1 = *(u64 *)(r8 + 0);"		\
+		"if r1 == 1 goto 1b;"			\
+		"r7 = %[four] ll;"			\
+		"if r1 == 2 goto 2f;"			\
+		"r7 = %[" #at_head "] ll;"		\
+		"goto 1b;"				\
+	"2:"						\
+		"r6 = 0;"				\
+	"3:"						\
+		"r2 = *(u64 *)(r7 + 0);"		\
+		"if r6 >= r2 goto 4f;"			\
+		"r1 = %[five_bytes] ll;"		\
+		"r1 += r6;"				\
+		"r3 = 1;"				\
+		"*(u8 *)(r1 + 0) = r3;"			\
+		"r6 += 1;"				\
+		"goto 3b;"				\
+	"4:"						\
+		"r0 = r6;"				\
+		"exit;"					\
+		:					\
+		: __imm_addr(zero),			\
+		  __imm_addr(one),			\
+		  __imm_addr(four),			\
+		  __imm_addr(five_bytes)		\
+		: __clobber_all
+
+__naked __noinline __used
+static unsigned long spins_then_counts_to_anything(void)
+{
+	asm volatile (SPINS_THEN_COUNTS(one));
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, another bound for the next loop")
+__arch_x86_64
+__arch_arm64
+__failure __msg("math between map_value pointer and register with unbounded min value")
+__naked void guard_at_head_bound_of_next_loop(void)
+{
+	asm volatile (
+		"call spins_then_counts_to_anything;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * r7 is fp-8 on every edge and fp-16 at the head of the first loop. The second loop
+ * counts in fp-16 and stores a number that nobody knows through r7.
+ */
+__naked __noinline __used
+static unsigned long spins_then_counts_on_stack(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+		"r3 = 0;"
+		"*(u64 *)(r10 - 8) = r3;"
+		"*(u64 *)(r10 - 16) = r3;"
+		"r7 = r10;"
+		"r7 += -16;"
+	"1:"
+		"r2 = *(u64 *)(r7 + 0);"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"r7 = r10;"
+		"r7 += -8;"
+		"if r1 == 2 goto 2f;"
+		"r7 = r10;"
+		"r7 += -16;"
+		"goto 1b;"
+	"2:"
+		"r3 = 0;"
+		"*(u64 *)(r10 - 16) = r3;"
+	"3:"
+		"r6 = *(u64 *)(r10 - 16);"
+		"if r6 == 4 goto 4f;"
+		"r1 = %[five_bytes] ll;"
+		"r1 += r6;"
+		"r3 = 1;"
+		"*(u8 *)(r1 + 0) = r3;"
+		"r6 += 1;"
+		"*(u64 *)(r10 - 16) = r6;"
+		"r3 = *(u64 *)(r8 + 0);"
+		"*(u64 *)(r7 + 0) = r3;"
+		"goto 3b;"
+	"4:"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(zero),
+		  __imm_addr(five_bytes)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, another pointer to the stack")
+__arch_x86_64
+__arch_arm64
+__failure __msg("math between map_value pointer and register with unbounded min value")
+__naked void guard_at_head_stack_pointer(void)
+{
+	asm volatile (
+		"call spins_then_counts_on_stack;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * Two states that have no join enter a loop. One has no count, so the loop gets may_goto.
+ * The other one has a count. may_goto is there when it runs too, and a loop in front
+ * may have used up the time slice. Then it leaves the loop in the first trip.
+ *
+ * r7 = the value, 16 bytes, and r8 = count, or r7 = 0 and r8 = anything
+ */
+#define VALUE_AND_COUNT_OR_NEITHER(count)		\
+		"r1 = 0;"				\
+		"*(u32 *)(r10 - 4) = r1;"		\
+		"r2 = r10;"				\
+		"r2 += -4;"				\
+		"r1 = %[pairs] ll;"			\
+		"call %[bpf_map_lookup_elem];"		\
+		"if r0 == 0 goto 9f;"			\
+		"r9 = %[zero] ll;"			\
+		"r2 = *(u64 *)(r9 + 0);"		\
+		"if r2 == 0 goto 3f;"			\
+		"r7 = r0;"				\
+		"r8 = " #count ";"			\
+		"goto 4f;"				\
+	"3:"						\
+		"r7 = 0;"				\
+		"r8 = *(u64 *)(r9 + 0);"		\
+	"4:"
+
+/* It's in the value when r6 is what it is at the end of the loop that was counted: last */
+#define LOAD_IF_VALUE(last)				\
+		"if r7 == 0 goto 9f;"			\
+		"r2 = " #last ";"			\
+		"r2 -= r6;"				\
+		"r2 &= 1;"				\
+		"r2 <<= 4;"				\
+		"r7 += r2;"				\
+		"r0 = *(u64 *)(r7 + 0);"		\
+		"exit;"					\
+	"9:"						\
+		"r0 = 0;"				\
+		"exit;"					\
+		:					\
+		: __imm_addr(pairs),			\
+		  __imm_addr(zero),			\
+		  __imm(bpf_map_lookup_elem)		\
+		: __clobber_all
+
+__naked __noinline __used
+static unsigned long counted_or_not(void)
+{
+	asm volatile (
+		VALUE_AND_COUNT_OR_NEITHER(10)
+		"r6 = 0;"
+	"1:"
+		"r6 += 1;"
+		"if r6 < r8 goto 1b;"
+		LOAD_IF_VALUE(10)
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto that another state needs, the loop is counted")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void guard_in_counted_loop(void)
+{
+	asm volatile (
+		"call counted_or_not;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* 2 4 8 16 is not counted. Constants decide the jump. */
+__naked __noinline __used
+static unsigned long walked_or_not(void)
+{
+	asm volatile (
+		VALUE_AND_COUNT_OR_NEITHER(16)
+		"r6 = 1;"
+	"1:"
+		"r6 <<= 1;"
+		"if r6 < r8 goto 1b;"
+		"r6 >>= 1;"
+		LOAD_IF_VALUE(8)
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto that another state needs, constants decide the exit of the loop")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void guard_in_walked_loop(void)
+{
+	asm volatile (
+		"call walked_or_not;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* i = 0; do { i++; } while (i < n); return v ? v[((10 - i) & 1) * 2] : 0; */
+__naked __noinline __used
+static unsigned long counts(unsigned long n, long *v)
+{
+	asm volatile (
+		"r0 = 0;"
+	"1:"
+		"r0 += 1;"
+		"if r0 < r1 goto 1b;"
+		"if r2 == 0 goto 2f;"
+		"r3 = 10;"
+		"r3 -= r0;"
+		"r3 &= 1;"
+		"r3 <<= 4;"
+		"r2 += r3;"
+		"r0 = *(u64 *)(r2 + 0);"
+	"2:"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+#define LOOKUP_PAIR					\
+		"r1 = 0;"				\
+		"*(u32 *)(r10 - 4) = r1;"		\
+		"r2 = r10;"				\
+		"r2 += -4;"				\
+		"r1 = %[pairs] ll;"			\
+		"call %[bpf_map_lookup_elem];"		\
+		"if r0 == 0 goto 9f;"			\
+		"r6 = r0;"
+
+SEC("socket")
+__description("budget: a subprog is called without a count, then with a count")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void two_callers(void)
+{
+	asm volatile (
+		LOOKUP_PAIR
+		"r1 = %[zero] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r2 = 0;"
+		"call counts;"
+		"r1 = 10;"
+		"r2 = r6;"
+		"call counts;"
+	"9:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(pairs),
+		  __imm_addr(zero),
+		  __imm(bpf_map_lookup_elem)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog is called with a count, then without a count")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void two_callers_counted_first(void)
+{
+	asm volatile (
+		LOOKUP_PAIR
+		"r1 = 10;"
+		"r2 = r6;"
+		"call counts;"
+		"r1 = %[zero] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r2 = 0;"
+		"call counts;"
+	"9:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(pairs),
+		  __imm_addr(zero),
+		  __imm(bpf_map_lookup_elem)
+		: __clobber_all
+	);
+}
+
+/* A global subprog is verified after the main prog */
+__naked __noinline __used
+int counts_anything(void)
+{
+	asm volatile (
+		"r1 = %[zero] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r2 = 0;"
+		"call counts;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog is called with a count, then without a count by a global subprog")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void two_callers_two_passes(void)
+{
+	asm volatile (
+		LOOKUP_PAIR
+		"r1 = 10;"
+		"r2 = r6;"
+		"call counts;"
+		"call counts_anything;"
+	"9:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(pairs),
+		  __imm(bpf_map_lookup_elem)
+		: __clobber_all
+	);
+}
+
+/*
+ * if (x) lock(); do { p = lookup(); } while (!p); if (x) { unlock(); return 0; } return *p;
+ * looks(0) holds nothing: may_goto unwinds. looks(1) cannot, so may_goto leaves the loop, in
+ * looks(0) too, and p is NULL.
+ */
+__naked __noinline __used
+static unsigned long looks(unsigned long x)
+{
+	asm volatile (
+		"r6 = r1;"
+		"if r6 == 0 goto 1f;"
+		"call %[bpf_rcu_read_lock];"
+	"1:"
+		"r1 = 0;"
+		"*(u32 *)(r10 - 4) = r1;"
+		"r2 = r10;"
+		"r2 += -4;"
+		"r1 = %[empty] ll;"
+		"call %[bpf_map_lookup_elem];"
+		"if r0 == 0 goto 1b;"
+		"if r6 == 0 goto 2f;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = 0;"
+		"exit;"
+	"2:"
+		"r0 = *(u64 *)(r0 + 0);"
+		"exit;"
+		:
+		: __imm_addr(empty),
+		  __imm(bpf_map_lookup_elem),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog finds out late that something is held")
+__arch_x86_64
+__arch_arm64
+__failure __msg("R0 invalid mem access 'map_value_or_null'")
+__naked void unwinds_then_cannot(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"call looks;"
+		unwind_goto(5)
+		"r1 = 1;"
+		"call looks;"
+		unwind_goto(2)
+		"r0 = 0;"
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(5),
+		  __imm_unwind_goto(2)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop without an exit")
+__arch_x86_64
+__arch_arm64
+__failure __msg("never exits")
+__naked void loop_without_exit(void)
+{
+	asm volatile (
+		"r0 = 0;"
+	"1:"
+		"r0 += 1;"
+		"goto 1b;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * may_goto keeps the address of its count in a register that is dead. What is live in front
+ * of the insn that gets may_goto is live in front of may_goto.
+ */
+SEC("socket")
+__description("budget: may_goto in front of a jump keeps R0")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void guard_keeps_r0(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"r0 = 7;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+#define ALL_REGS_LIVE_IN_LOOP(var)			\
+		"r9 = %[" #var "] ll;"			\
+		"r0 = 1;"				\
+		"r2 = 2;"				\
+		"r3 = 3;"				\
+		"r4 = 4;"				\
+		"r5 = 5;"				\
+		"r6 = 6;"				\
+		"r7 = 7;"				\
+		"r8 = 8;"				\
+	"1:"						\
+		"r1 = *(u64 *)(r9 + 0);"		\
+		"if r1 != 0 goto 1b;"			\
+		"r0 += r1;"				\
+		"r0 += r2;"				\
+		"r0 += r3;"				\
+		"r0 += r4;"				\
+		"r0 += r5;"				\
+		"r0 += r6;"				\
+		"r0 += r7;"				\
+		"r0 += r8;"				\
+		"r9 = *(u64 *)(r9 + 0);"		\
+		"r0 += r9;"				\
+		"exit;"					\
+		:					\
+		: __imm_addr(var)			\
+		: __clobber_all
+
+/* No register is dead: R0 waits in the stack */
+SEC("socket")
+__description("budget: may_goto where all registers are live")
+__arch_x86_64
+__arch_arm64
+__success __retval(36)
+__xlated("...")
+__xlated("*(u64 *)(r10 -8) = r0")
+__xlated("r0 = 0x{{.*}}")
+__xlated("r0 = &(void __percpu *)(r0)")
+__xlated("r13 = *(u64 *)(r0 +0)")
+__xlated("...")
+__xlated("*(u64 *)(r0 +0) = r13")
+__xlated("r0 = *(u64 *)(r10 -8)")
+__xlated("if r1 != 0x0 goto pc-{{[0-9]+}}")
+__naked void guard_all_regs_live(void)
+{
+	asm volatile (
+		ALL_REGS_LIVE_IN_LOOP(zero)
+	);
+}
+
+__naked __noinline __used
+static unsigned long spins_all_regs_live(void)
+{
+	asm volatile (
+		ALL_REGS_LIVE_IN_LOOP(one)
+	);
+}
+
+/* The time is up: the loop is left with r1 = 1 and all the other registers as they are */
+SEC("socket")
+__description("budget: may_goto where all registers are live, out of budget")
+__arch_x86_64
+__arch_arm64
+__success __retval(38)
+__naked void guard_all_regs_live_spins(void)
+{
+	asm volatile (
+		"call spins_all_regs_live;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * A loop that has no count is looked at trip by trip for a while. A state that is within a state of
+ * an earlier trip is dropped, wherever that is. Then nothing says that the loop ends, it gets
+ * may_goto, and all is done again. None of the loops below ends on its own.
+ */
+SEC("socket")
+__description("budget: every second trip leaves everything as it was")
+__arch_x86_64
+__arch_arm64
+__success __retval(0) __log_level(2)
+__msg("processed 25 insns")
+__naked void trips_of_period_two(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r7 = 1;"
+	"1:"
+		"r7 ^= 3;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 == 0 goto 2f;"
+		"if r7 != 7 goto 1b;"
+	"2:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/*
+ * r7 is 0 at the head in the first trip and 5 in the second one. It's dead at 2, where the second
+ * trip is dropped. Nothing comes back to the head that is within what is there.
+ */
+SEC("socket")
+__description("budget: a trip is dropped in the middle of the loop")
+__arch_x86_64
+__arch_arm64
+__success __retval(0) __log_level(2)
+__msg("processed 33 insns")
+__naked void trip_dropped_in_the_middle(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r7 = 0;"
+	"1:"
+		"if r7 == 7 goto 3f;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 == 0 goto 3f;"
+		"if r1 == 2 goto 2f;"
+		"r1 = 0;"
+	"2:"
+		"r7 = 5;"
+		"goto 1b;"
+	"3:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/* n ? zero_at_the_bottom(n - 1) : zero */
+__naked __noinline __used
+static unsigned long zero_at_the_bottom(unsigned long n)
+{
+	asm volatile (
+		"if r1 == 0 goto 1f;"
+		"r1 += -1;"
+		"call zero_at_the_bottom;"
+		"exit;"
+	"1:"
+		"r1 = %[zero] ll;"
+		"r0 = *(u64 *)(r1 + 0);"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+/*
+ * The call of a recursive subprog waits for what comes out of it. The call of the second trip is
+ * within the one that waits. A recursive subprog that has no time or no stack left returns 0 at
+ * once, so that doesn't stop the loop.
+ */
+SEC("socket")
+__description("budget: a trip is dropped at a call that waits")
+__arch_x86_64
+__arch_arm64
+__success __retval(0) __log_level(2)
+__msg("processed 27 insns")
+__naked void trip_dropped_at_a_call(void)
+{
+	asm volatile (
+		"r7 = 0;"
+	"1:"
+		"if r7 == 7 goto 2f;"
+		"r1 = 3;"
+		"call zero_at_the_bottom;"
+		"r7 = 5;"
+		"if r0 == 0 goto 1b;"
+	"2:"
+		"r0 = 1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * Every trip goes through bpf_iter_num_next(), so the loop is taken for bounded at first, and
+ * through may_goto of the program. The iterator is made in the loop, and may_goto stays in it:
+ * neither of them stops it. The program doesn't end on its own.
+ */
+SEC("socket")
+__description("budget: may_goto that stays in the loop doesn't stop it")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_stays_in_the_loop(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"r6 = *(u64 *)(r6 + 0);"
+		"if r6 != 0 goto 4f;"
+	"1:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 2;"
+		"call %[bpf_iter_num_new];"
+	"2:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"if r0 != 0 goto 2b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		".8byte %[may_goto];"
+		"if r6 == 0 goto 1b;"
+	"4:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy),
+		  __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 0 /* offset */, 0))
+		: __clobber_all
+	);
+}
+
+/* No state is within another one: the walk gets to the end of the loop, no may_goto */
+SEC("socket")
+__description("budget: ten trips that differ")
+__success __retval(10) __log_level(2)
+__msg("processed 34 insns")
+__xlated("r8 += 1")
+__xlated("r7 <<= 1")
+__xlated("if r7 != 0x400 goto pc-3")
+__naked void ten_trips_that_differ(void)
+{
+	asm volatile (
+		"r7 = 1;"
+		"r8 = 0;"
+	"1:"
+		"r8 += 1;"
+		"r7 <<= 1;"
+		"if r7 != 1024 goto 1b;"
+		"r0 = r8;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * BTF FUNC records are not generated for kfuncs referenced
+ * from inline assembly. These records are necessary for
+ * libbpf to link the program. The function below is a hack
+ * to ensure that BTF FUNC records are generated.
+ */
+void __kfunc_btf_root(void)
+{
+	bpf_rcu_read_lock();
+	bpf_rcu_read_unlock();
+	bpf_iter_num_new(0, 0, 0);
+	bpf_iter_num_next(0);
+	bpf_iter_num_destroy(0);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index afe4990cd027..5f0d9e545656 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -637,6 +637,39 @@ __naked void callx_mutual_recursion(void)
 		: __clobber_all);
 }
 
+static __noinline __used int pulls_data(struct __sk_buff *skb)
+{
+	return bpf_skb_pull_data(skb, 0);
+}
+
+/*
+ * The prog loads the address of the function, apply() calls it. The packet
+ * pointer of the prog is gone after the call of apply().
+ */
+SEC("tc")
+__failure __msg("R6 invalid mem access 'scalar'")
+__naked void callx_fn_as_arg_changes_pkt_data(void)
+{
+	asm volatile (
+		"r2 = r1;"
+		"r6 = *(u32 *)(r1 + %[data]);"
+		"r7 = *(u32 *)(r1 + %[data_end]);"
+		"r0 = 0;"
+		"r3 = r6;"
+		"r3 += 1;"
+		"if r3 > r7 goto 1f;"
+		"r1 = %[pulls_data] ll;"
+		"call apply;"
+		"*(u8 *)(r6 + 0) = 42;"
+		"r0 = 0;"
+	"1:	exit;"
+		:
+		: __imm_addr(pulls_data),
+		  __imm_const(data, offsetof(struct __sk_buff, data)),
+		  __imm_const(data_end, offsetof(struct __sk_buff, data_end))
+		: __clobber_all);
+}
+
 /* calls_arg(fn) { return fn(); } */
 __naked __noinline __used
 static unsigned long calls_arg(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 8bb74b579203..2e7e65949497 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -198,6 +198,57 @@ __naked void callx_rodata_array_of_structs_data(void)
 		: __clobber_all);
 }
 
+int calls_what_calls_it(void);
+
+__naked __noinline __used
+static unsigned long calls_global(void)
+{
+	asm volatile (
+		"call calls_what_calls_it;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all);
+}
+
+__naked __noinline __used
+int calls_what_calls_it(void)
+{
+	asm volatile (
+		RODATA(cb, ".quad calls_global;")
+		"r2 = cb_%= ll;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 1;"
+		"r3 = 0;"
+		"r4 = 0;"
+		"call %[bpf_loop];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_loop)
+		: __clobber_all);
+}
+
+/*
+ * No subprog takes the address of the callback, and calls of global functions are not
+ * followed. It's a recursion all the same, and nothing checks the stack.
+ */
+SEC("socket")
+__failure __msg("recursive call from calls_global() to calls_what_calls_it()")
+__naked void callx_rodata_callback_recursion(void)
+{
+	asm volatile (
+		FUNC_TABLE2(tbl, add1, add2)
+		"call calls_what_calls_it;"
+		/* libbpf resolves the pointers for progs that have callx */
+		"r2 = tbl_%= ll;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 1;"
+		"callx r2;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all);
+}
+
 /* but not the data and the pointer with the same load */
 SEC("socket")
 __failure __msg("overlaps with a pointer to a function")
-- 
2.55.0

[PATCH bpf-next v3 45/45] selftests/bpf: Test callx of one of a set of functions

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

'ops[i](x)' in C, and each '{}' of a format string in Rust, is a load from
a table of functions at a variable offset, and callx. The load gives one
register that points to one of N functions.

  [S] "bpf: Track a pointer to one of a set of functions"
  [L] "bpf: callx reads what the subprogs whose address is taken read"

  file, test                     the rule that it pins
  -----------------------------  ------------------------------------------
  verifier_callx_rodata.c, callx_rodata_..
   one_of_two                    [S] The log has R2=func(one_of=2). callx
                                 calls the function that the index selects.
   callback_one_of_two           [S] bpf_loop() gets the register: each
                                 function is verified as the callback.
   one_of_two_at_loop_head       [S] callx is the head of a loop. The prog
                                 is unsafe in the fourth iteration with one
                                 of the two functions: rejected.
  verifier_callx_many.c, callx_rodata_..
   400_functions                 [S] 3 states that have no join get to the
                                 load: 3 states at an insn. One state for
                                 each function is 1200, the limit is 1024.
   4400_sets                     [S] More than 4096 sets: rejected.
  verifier_stack_arg.c
   callx_one_of_two_not_written  [S] The caller does not store the stack
                                 arguments that both functions take:
                                 rejected.
  verifier_callx.c
   callx_reads_declared_args     [L] r2 is live at callx: the callee does
                                 not read it, but its BTF declares it.

one_of_two, 400_functions and 4400_sets fail without [S], and
callx_reads_declared_args fails without [L]. The other three tests pass in
both cases. They examine the places where a set becomes one function.

libbpf adds the functions of a table to each prog of the object that has
callx. Thus the 400 functions are in a file of their own.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_callx.c      |  24 ++++
 .../selftests/bpf/progs/verifier_callx_many.c | 113 ++++++++++++++++++
 .../bpf/progs/verifier_callx_rodata.c         | 112 +++++++++++++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  |  54 +++++++++
 5 files changed, 305 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_many.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 71ccd1e94bb2..0db43480cd99 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -39,6 +39,7 @@
 #include "verifier_call_large_imm.skel.h"
 #include "verifier_calls_in_a_row.skel.h"
 #include "verifier_callx.skel.h"
+#include "verifier_callx_many.skel.h"
 #include "verifier_callx_rodata.skel.h"
 #include "verifier_cfg.skel.h"
 #include "verifier_cgroup_inv_retcode.skel.h"
@@ -246,6 +247,7 @@ void test_verifier_btf_unreliable_prog(void)  { RUN(verifier_btf_unreliable_prog
 void test_verifier_call_large_imm(void)       { RUN(verifier_call_large_imm); }
 void test_verifier_calls_in_a_row(void)       { RUN(verifier_calls_in_a_row); }
 void test_verifier_callx(void)                { RUN_JITED(verifier_callx); }
+void test_verifier_callx_many(void)           { RUN_JITED(verifier_callx_many); }
 void test_verifier_callx_rodata(void)         { RUN_JITED(verifier_callx_rodata); }
 void test_verifier_cfg(void)                  { RUN(verifier_cfg); }
 void test_verifier_cgroup_inv_retcode(void)   { RUN(verifier_cgroup_inv_retcode); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index 5f0d9e545656..a2bc466791f3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -985,6 +985,30 @@ __naked void callx_callee_reads_caller_stack_nested(void)
 		: __clobber_all);
 }
 
+__naked __noinline __used
+static unsigned long reads_one_of_two(long a, long b)
+{
+	asm volatile (
+		"r0 = r1;"
+		"exit;"
+		::: __clobber_all);
+}
+
+/* r2 is live at callx: the BTF of the callee, which a tracer counts on, declares it */
+SEC("socket")
+__success __retval(1) __log_level(2)
+__msg(".123...... (8d) callx r3")
+__naked void callx_reads_declared_args(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = reads_one_of_two ll;"
+		"callx r3;"
+		"exit;"
+		::: __clobber_all);
+}
+
 /* registers that are constant before callx are not constant after it */
 SEC("socket")
 __success __retval(1)
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_many.c b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
new file mode 100644
index 000000000000..23c8948c24b8
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * callx through a table of many functions. libbpf appends the functions that read-only
+ * data points to to every prog of the object that has callx, hence a file of its own.
+ */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../../../include/linux/filter.h"
+
+#ifdef CAN_USE_CALLX
+
+/*
+ * Read-only data with pointers to functions, where the compiler puts tables
+ * of functions, structures of operations and vtables. libbpf resolves
+ * a pointer to the offset of the function in the program and the kernel
+ * recognizes it by that value.
+ */
+#define DATA(SECTION, NAME, ...)				\
+	".pushsection " SECTION ",@progbits;"			\
+	".balign 8;"						\
+	#NAME "_%=:"						\
+	__VA_ARGS__						\
+	".type " #NAME "_%=, @object;"				\
+	".size " #NAME "_%=, .-" #NAME "_%=;"			\
+	".popsection;"
+
+#define RODATA(NAME, ...) DATA(".rodata,\"a\"", NAME, __VA_ARGS__)
+
+#define REP10(M, p) M(p##0) M(p##1) M(p##2) M(p##3) M(p##4) M(p##5) M(p##6) M(p##7) M(p##8) M(p##9)
+#define REP100(M, p) REP10(M, p##0) REP10(M, p##1) REP10(M, p##2) REP10(M, p##3) REP10(M, p##4) \
+		     REP10(M, p##5) REP10(M, p##6) REP10(M, p##7) REP10(M, p##8) REP10(M, p##9)
+#define REP400(M) REP100(M, 0) REP100(M, 1) REP100(M, 2) REP100(M, 3)
+
+#define MANY_FUNC(n)						\
+__naked __noinline __used static unsigned long many##n(void)	\
+{								\
+	asm volatile ("r0 = r1; r0 += 1; exit;");		\
+}
+#define MANY_QUAD(n) ".quad many" #n ";"
+
+REP400(MANY_FUNC)
+
+/* many<n> or what is 'mask' bytes behind it in the table: 400 sets that were not there before */
+#define SET_LOAD(n) "r2 = *(u64 *)(r7 + 8 * (1" #n " - 1000));"
+#define SETS400(mask) "r7 = r6; r7 &= " #mask "; r7 += r8;" REP400(SET_LOAD)
+
+SEC("socket")
+__failure __msg("more than 4096 sets of functions")
+__naked void callx_rodata_4400_sets(void)
+{
+	asm volatile (
+		RODATA(twice, REP400(MANY_QUAD) REP400(MANY_QUAD))
+		"call %[bpf_get_prandom_u32];"
+		"r6 = r0;"
+		"r8 = twice_%= ll;"
+		SETS400(8) SETS400(16) SETS400(32) SETS400(64) SETS400(128) SETS400(256)
+		SETS400(512) SETS400(1024) SETS400(2048) SETS400(24) SETS400(40)
+		"r1 = 10;"
+		"callx r2;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
+/* 3 states that have no join get to a table of 400 functions: 3 states, and not 1200 */
+SEC("socket")
+__success __retval(11)
+__naked void callx_rodata_400_functions(void)
+{
+	asm volatile (
+		RODATA(big, REP400(MANY_QUAD))
+		"r7 = r1;"
+		"call %[bpf_get_prandom_u32];"
+		"if r0 == 1 goto 1f;"
+		"r7 = r10;"
+		"if r0 == 2 goto 1f;"
+		"r7 = 0;"
+	"1:"
+		"call %[bpf_get_prandom_u32];"
+		"r0 &= 511;"
+		"if r0 > 399 goto 2f;"
+		"r0 <<= 3;"
+		"r2 = big_%= ll;"
+		"r2 += r0;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 10;"
+		"callx r2;"
+		"if r7 != 0 goto +0;"
+		"exit;"
+	"2:"
+		"r0 = 11;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
+#else
+
+SEC("socket")
+__skip("callx needs arch JIT and compiler support")
+__success
+int dummy(void *ctx)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 2e7e65949497..3545c2a2e8f1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -171,6 +171,118 @@ __naked void callx_rodata_array_of_structs(void)
 		: __clobber_all);
 }
 
+/* one state with one of two functions, and not two states */
+SEC("socket")
+__success __retval(11)
+__log_level(2)
+__msg("r2 = *(u64 *)(r2 +0)          ; R2=func(one_of=2)")
+__naked void callx_rodata_one_of_two(void)
+{
+	asm volatile (
+		FUNC_TABLE2(tbl, add1, add2)
+		"call %[bpf_get_prandom_u32];"
+		"r6 = r0;"
+		"r6 &= 1;"
+		"r3 = r6;"
+		"r3 <<= 3;"
+		"r2 = tbl_%= ll;"
+		"r2 += r3;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 10;"
+		"callx r2;"
+		"r0 -= r6;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_stop(void)
+{
+	asm volatile (
+		"r0 = 1;"
+		"exit;"
+	);
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_go_on(void)
+{
+	asm volatile (
+		"r0 = 0;"
+		"exit;"
+	);
+}
+
+/* a helper needs to know the callback */
+SEC("socket")
+__success __retval(0)
+__naked void callx_rodata_callback_one_of_two(void)
+{
+	asm volatile (
+		FUNC_TABLE2(cbs, loop_cb_stop, loop_cb_go_on)
+		"call %[bpf_get_prandom_u32];"
+		"r0 &= 1;"
+		"r0 <<= 3;"
+		"r2 = cbs_%= ll;"
+		"r2 += r0;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r6 = r2;"
+		"r1 = 2;"
+		"r3 = 0;"
+		"r4 = 0;"
+		"call %[bpf_loop];"
+		/* libbpf resolves the pointers for progs that have callx */
+		"callx r6;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32), __imm(bpf_loop)
+		: __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long not_three(void)
+{
+	asm volatile (
+		"r0 = 0;"
+		"if r1 != 3 goto +1;"
+		"r10 = 0;"
+		"exit;"
+	);
+}
+
+/*
+ * callx is the head of a loop whose iterations are looked at one by one, because of r1.
+ * The states with one function each are in the iteration of the state with both.
+ */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callx_rodata_one_of_two_at_loop_head(void)
+{
+	asm volatile (
+		FUNC_TABLE2(tbl, add1, not_three)
+		"call %[bpf_get_prandom_u32];"
+		"r0 &= 1;"
+		"r0 <<= 3;"
+		"r6 = tbl_%= ll;"
+		"r6 += r0;"
+		"r6 = *(u64 *)(r6 + 0);"
+		"r7 = 0;"
+		"r1 = 0;"
+	"l_%=:"
+		"callx r6;"
+		"r7 += 1;"
+		"r1 = r7;"
+		"if r7 < 4 goto l_%=;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
 /* and the data of the element that the index selects can be read */
 SEC("socket")
 __success __retval(16)
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index e3f5e4d14b75..c642675dd92b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -572,6 +572,60 @@ __naked void null_check_of_stack_arg(void)
 	: __clobber_all);
 }
 
+static __naked __noinline __used long adds_arg6_arg7(long a, long b, long c, long d, long e,
+						     long f, long g)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r11 + 8);"
+	"r1 = *(u64 *)(r11 + 16);"
+	"r0 += r1;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __naked __noinline __used long subs_arg6_arg7(long a, long b, long c, long d, long e,
+						     long f, long g)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r11 + 8);"
+	"r1 = *(u64 *)(r11 + 16);"
+	"r0 -= r1;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: callx of one of two functions, args are not written")
+__failure __msg("callee expects 7 args, stack arg1 is not initialized")
+__naked void callx_one_of_two_not_written(void)
+{
+	asm volatile (
+	".pushsection .rodata,\"a\",@progbits;"
+	".balign 8;"
+"two_funcs_%=:"
+	".quad adds_arg6_arg7;"
+	".quad subs_arg6_arg7;"
+	".type two_funcs_%=, @object;"
+	".size two_funcs_%=, .-two_funcs_%=;"
+	".popsection;"
+	"call %[bpf_get_prandom_u32];"
+	"r0 &= 1;"
+	"r0 <<= 3;"
+	"r2 = two_funcs_%= ll;"
+	"r2 += r0;"
+	"r6 = *(u64 *)(r2 + 0);"
+	"r1 = 1;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"callx r6;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
 /* r1 - r5 and all 19 slots. Its prototype has no parameters: what it takes is seen in its code. */
 __naked __noinline __used
 static unsigned long subprog_24args(void)
-- 
2.55.0

Re: [PATCH bpf-next v3 45/45] selftests/bpf: Test callx of one of a set of functions

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] selftests/bpf: missing __load_if_JITed() in callx_one_of_two_not_written()
--

commit 64c4b960a70c853d9b8785754e157dac9645cef4
Author: Alexei Starovoitov <ast@kernel.org>

selftests/bpf: Test callx of one of a set of functions

This commit adds BPF verifier selftests covering indirect function calls
(callx) where a register holds a pointer to one of a set of functions loaded
from a read-only table, as well as declared argument liveness at callx.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
> index e3f5e4d14b759..c642675dd92bc 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
> @@ -572,6 +572,60 @@ __naked void null_check_of_stack_arg(void)
>  	: __clobber_all);
>  }
>  
> +static __naked __noinline __used long adds_arg6_arg7(long a, long b, long c, long d, long e,
> +						     long f, long g)
> +{
> +	asm volatile (
> +	"r0 = *(u64 *)(r11 + 8);"
> +	"r1 = *(u64 *)(r11 + 16);"
> +	"r0 += r1;"
> +	"exit;"
> +	::: __clobber_all);
> +}
> +
> +static __naked __noinline __used long subs_arg6_arg7(long a, long b, long c, long d, long e,
> +						     long f, long g)
> +{
> +	asm volatile (
> +	"r0 = *(u64 *)(r11 + 8);"
> +	"r1 = *(u64 *)(r11 + 16);"
> +	"r0 -= r1;"
> +	"exit;"
> +	::: __clobber_all);
> +}
> +
> +SEC("tc")
> +__description("stack_arg: callx of one of two functions, args are not written")
> +__failure __msg("callee expects 7 args, stack arg1 is not initialized")
> +__naked void callx_one_of_two_not_written(void)
> +{
> +	asm volatile (
> +	".pushsection .rodata,\"a\",@progbits;"
> +	".balign 8;"
> +"two_funcs_%=:"
> +	".quad adds_arg6_arg7;"
> +	".quad subs_arg6_arg7;"
> +	".type two_funcs_%=, @object;"
> +	".size two_funcs_%=, .-two_funcs_%=;"
> +	".popsection;"
> +	"call %[bpf_get_prandom_u32];"
> +	"r0 &= 1;"
> +	"r0 <<= 3;"
> +	"r2 = two_funcs_%= ll;"
> +	"r2 += r0;"
> +	"r6 = *(u64 *)(r2 + 0);"
> +	"r1 = 1;"
> +	"r2 = 2;"
> +	"r3 = 3;"
> +	"r4 = 4;"
> +	"r5 = 5;"
> +	"callx r6;"
> +	"exit;"
> +	:
> +	: __imm(bpf_get_prandom_u32)
> +	: __clobber_all);
> +}

[Severity: Medium]
As reported by Sashiko on v2 [1], should callx_one_of_two_not_written() be
marked with __load_if_JITed() (and guarded with #ifdef CAN_USE_CALLX)?

In tools/testing/selftests/bpf/prog_tests/verifier.c,
test_verifier_stack_arg() runs the suite via RUN(verifier_stack_arg) rather
than RUN_JITED(verifier_stack_arg):

    void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }

When the BPF JIT is disabled (for example, bpf_jit_enable=0 when
CONFIG_BPF_JIT_ALWAYS_ON is not set), subtests in verifier_stack_arg.c
without __load_if_JITed() are still loaded.

During verification of callx_one_of_two_not_written(), reading the function
pointer from .rodata and executing callx r6 both invoke
require_callx_jit(env) before reaching check_func_callx_set() and
check_outgoing_stack_args():

    do_check_all()
      -> check_map_mem_read()
        -> check_func_ptr_read()
          -> require_callx_jit()

and:

    do_check_all()
      -> check_func_callx()
        -> require_callx_jit()

In kernel/bpf/verifier.c:require_callx_jit():

    static int require_callx_jit(struct bpf_verifier_env *env)
    {
            if (!env->prog->jit_requested) {
                    verbose(env, "JIT is required to use callx\n");
                    return -EOPNOTSUPP;
            }

Without JIT enabled, wouldn't the verifier reject this program with "JIT is
required to use callx" instead of reaching the expected "callee expects 7
args, stack arg1 is not initialized" error?

[1] https://lore.kernel.org/r/20261105204513.2230344-36-alexei.starovoitov@gmail.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106035111.3315285-1-alexei.starovoitov@gmail.com?part=45

Re: [PATCH bpf-next v3 00/45] bpf: A verifier that joins states

[Eduard Zingerman]

On Fri, 2026-11-06 at 03:50 +0000, Alexei Starovoitov wrote:
> From: Alexei Starovoitov <ast@kernel.org>
> 
> The verifier follows each path of a prog, and each branch can double the
> number of paths. The work grows with the paths, not with the prog:
>   insns processed                                    today    this series
>   rust-bpf  scx_beerland/beerland_init_task        1000001 *        17893
>   rust-bpf  scx_qmap/qmap_init_cids                 131088 *          196
>   scx       scx_eevdf/eevdf_dispatch                350041          15199
>   Cilium    bpf_host/cil_to_netdev, Oct 2026         64658           7472
>   * "BPF program is too large", "The sequence of 8193 jumps is too complex"
> 
> Authors cut their progs until they fit. Nobody can cut core, alloc and the
> crates of Rust: of the 653 progs of https://github.com/4ast/rust-bpf 187
> load today, 170 hit one of these limits, 164 get "infinite loop detected"
> and 122 "recursive call". Now 648 load, and no corpus of C progs loses one.
> For rust-bpf both kernels have 4 kfuncs on top, see Numbers.
> 
> The idea
> ========
> Do not follow paths. Where paths meet, join their states: r1=1 and r1=5
> give r1=[1,5]. Repeat until no state changes. States stay apart only if
> they have no join, or if a later check depends on the difference.
>   loops      counted by scalar evolution: one state covers all iterations
>              walked: a state per iteration, while constants decide an exit
>              all other loops: join at the head, widen, add may_goto
>   subprogs   A static subprog is verified for the states at its entry, not
>              for each chain of calls. Recursion is the same, with widening.
>   run time   may_goto limits the time of loops and of recursion. r10 limits
>              the depth: it must be in the upper half of its kernel stack.
> 
> What users see
> ==============
>  - The limits of 1M processed insns and 8192 jumps are gone. The limits are
>    on states, for a prog of N insns: 1024 at an insn, 16 * N + 4096 at
>    a time, 256 * N + 64K in all.
>  - "infinite loop detected" is gone: the verifier adds may_goto. A loop
>    that has no exit gets "loop at insn N never exits".
>  - may_goto is also added to 5 progs of the table below that load today:
>    loop1, loop2, loop5, tcp_autolowat_enqueue_rcvq (GCC), chaos_enqueue.
>  - Recursion is allowed on x86-64 and arm64. Patch 22 lists the exceptions.
>  - All added may_goto of an invocation share its time slice of 250 ms. Then
>    the prog returns 0. A subprog that cannot unwind, or a frame that holds
>    a lock or a reference, leaves the loop. The stderr stream has the cause.
>  - A join loses the relation between scalars. This is rejected now:
>      if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
>      x = arr[n - i];                                    n - i is [-3, 5]
>  - A cgroup_skb prog whose r0 can be 2 or 3 attaches to egress only. That
>    holds for a range too: 'return 1' and 'return 2' are joined to [1, 2].
>  - The log has no "mark_precise:", "N: safe", "depth=" and no scc column.
>    Level 2 shows "from 13 to 9: within", "joined" or "widened" where states
>    meet. The statistics line keeps its format: "limit 0", "mark_read 0".
>  - BPF_F_TEST_STATE_FREQ is accepted and has no effect.
> 
> Numbers
> =======
> A: bpf-next 8424858a5de0. B: this series, the same objects. The selftests
> that the series adds or changes are not counted. None of them is lost.
> 
> Corpus                 Progs  Lost  Gained  Insns  States   Mem  Time
> ---------------------  -----  ----  ------  -----  ------  ----  ----
> selftests               3043     0       1   -63%   +120%  -18%  -27%
> selftests, no_alu32     2905     0       1   -66%    +86%  -27%  -34%
> selftests, cpuv4        2914     0       1   -64%    +96%  -27%  -32%
> selftests, bpf_gcc      2733     0       5   -69%    +96%  -16%  -24%
> 37 builds of Cilium     5189     0       0   -75%    +11%   -6%  -31%
> sched-ext/scx            495     0       0   -89%    -34%  -66%  -67%
> scx, as BPF CI has it    428     0       0   -85%    -22%  -38%  -39%
> tools/sched_ext          240     0       0   -69%    +37%   +3%  -11%
> public progs of Meta     511     0       0   -85%    -35%  -55%  -48%
> all                    18458     0       8   -74%    +10%  -19%  -36%
> 
> Progs load in A and in B. The per cent compare sums over these progs:
> 75.1M -> 19.2M processed insns, 61 -> 39 s. Mem is mem_peak of veristat.
> Time is the best of 3 runs on kernels without debug options. B counts each
> state that arrives at a basic block as a state, A only those that it saves.
> Gained: loop3 in the 4 builds, 4 progs that A rejects only in GCC's code.
> Of the progs that take 1000 insns or more in B, two take over 5% more than
> in A. loop5, 82 -> 4182: a walk up to its limit of 4096 insns, then a join.
> res_spin_lock_test_held_lock_max, 2356 -> 4088: states that hold different
> numbers of locks have no join.
> 
> rust-bpf               Progs  load in A  load in B  B: insns   time
> 16 schedulers            225         52        225      1.6M  1.9 s
> crates                   428        135        423      7.3M  9.4 s
> For rust-bpf A and B have the kfuncs of patches 30-31 of [1] on top. They
> are in no series now. No released compiler emits the code of rust-bpf.
> 
> How to read the series
> ======================
> Start with patch 7, the overview. Then 4-5, 13-14, 15, 17 and 19.
>   1-2    what is added to a prog: may_goto, the checks of recursion
>   3-5    union and widening of a range, join.c: the join of two states
>   6-10   data structures of flow.c, passes ahead of the main pass: order of
>          the insns, places for may_goto, what later checks depend on
>   11-14  the sets of states, iterations, what is not joined, the merge
>   15-18  how states move: edges, calls and exits, loops, added edges
>   19     the switch. The check of one insn against one state is the same.
>   20-22  more counted loops, static subprogs on their own, recursion
>   23-26  removal of pruning, precision tracking and SCCs: 3382 lines
>   27-28  callx: a pointer to one of a set of functions, live registers
>   29-38  code that needs more than that: casts, the context in a struct,
>          subprogs that check for their caller, loops that are not counted
>   39-45  selftests: 39-40 adjust tests, 41-45 add tests
> Each patch builds. Patches 1-18 add the code that patch 19 switches to.
> Some selftests fail from patch 19 to 39.

Hi Alexei,

regarding the Meta veristat failures. Please find distilled examples here:
https://github.com/eddyz87/bpf/tree/ast-new-verifier-v3-meta-tests
Applied as commits on top of your series.
All slop but seem to add up.

...