[PATCH bpf-next v4 00/46] bpf: A verifier that joins states

[PATCH bpf-next v4 00/46] bpf: A verifier that joins states

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier follows each path of a prog, and each branch can double the
number of paths. The work grows with the paths, not with the prog:
  insns processed                                    today    this series
  rust-bpf  scx_beerland/beerland_init_task        1000001 *        16430
  rust-bpf  scx_qmap/qmap_init_cids                 131088 *          196
  scx       scx_eevdf/eevdf_dispatch                350041          15322
  Cilium    bpf_host/cil_to_netdev, Oct 2026         64658           7538
  * "BPF program is too large", "The sequence of 8193 jumps is too complex"

Authors cut their progs until they fit. Nobody can cut core, alloc and the
crates of Rust: of the 653 progs of https://github.com/4ast/rust-bpf 187
load today, 170 hit one of these limits, 164 get "infinite loop detected"
and 122 "recursive call". Now 648 load, and the corpora of C progs below
lose none. For rust-bpf both kernels have 4 kfuncs on top, see Numbers.

Loops in BPF progs are written for the verifier, not for the reader:

  bpf_qdisc_fifo.c       bpf_for(i, 0, sch->q.qlen) {
  arena_htab.c           for (i = zero; i < 100000 && can_loop; i++) {
  rbtree_search_kptr.c   while (rb_n && can_loop) {
  tailcall_bpf2bpf2.c    barrier_var(ret); return ret;

Now plain C loads: 'for (i = 0; i < 100000; i++)', 'while (rb_n)'. The last
patch has 34 progs of the selftests without these macros. bpf-next rejects
all of them, five in some flavours of test_progs only.

The idea
========
Do not follow paths. Where paths meet, join their states: r1=1 and r1=5
give r1=[1,5]. Repeat until no state changes. States stay apart only if
they have no join, or if a later check depends on the difference.
  loops      counted by scalar evolution: one state covers all iterations
             walked: a state per iteration, while constants decide an exit
             all other loops: join at the head, widen, add may_goto
  subprogs   A static subprog is verified for the states at its entry, not
             for each chain of calls. Recursion is the same, with widening.
  run time   may_goto limits the time of loops and of recursion. r10 limits
             the depth: it must be in the upper half of its kernel stack.

What users see
==============
 - The limits of 1M processed insns and 8192 jumps are gone. The limits are
   on states, for a prog of N insns: 1024 at an insn, 16 * N + 4096 at
   a time, 256 * N + 64K in all.
 - "infinite loop detected" is gone: the verifier adds may_goto. A loop
   that has no exit gets "loop at insn N never exits".
 - A loop is counted by a jump in the middle of its body, by a jump that
   sets a flag, by a copy of the counter.
 - may_goto is also added to 5 progs of the table below that load today:
   loop1, loop2, loop5, tcp_autolowat_enqueue_rcvq (GCC), chaos_enqueue
   (as of v2).
 - Recursion is allowed on x86-64 and arm64. Patch 22 lists the exceptions.
 - All added may_goto of an invocation share its time slice of 250 ms. Then
   the prog returns 0. A subprog that cannot unwind, or a frame that holds
   a lock or a reference, leaves the loop. If the code behind the loop
   reads what only the loop sets, the subprog returns 0. The stderr stream
   has the cause.
 - A subprog that returns void can have a tail call or LD_ABS. LLVM 23
   makes 'void' of an 'int' that no caller reads.
 - With CAP_PERFMON a part of a pointer can be spilled, and 32-bit add to
   a pointer and ops on two pointers make a number. With it "invalid size
   of register spill" and "32-bit pointer arithmetic prohibited" are gone.
 - A join loses the relation between scalars. This is rejected now:
     if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
     x = arr[n - i];                                    n - i is [-3, 5]
 - A cgroup_skb prog whose r0 can be 2 or 3 attaches to egress only. That
   holds for a range too: 'return 1' and 'return 2' are joined to [1, 2].
 - The log has no "mark_precise:", "N: safe", "depth=" and no scc column.
   Level 2 shows "from 13 to 9: within", "joined" or "widened" where states
   meet. The statistics line keeps its format: "limit 0", "mark_read 0".
 - BPF_F_TEST_STATE_FREQ is accepted and has no effect.

Numbers
=======
The tables are those of v2. A: bpf-next 8424858a5de0. B: the series, the
same objects. The selftests that the series adds or changes are not
counted. None of them is lost. v4 against v3 on these corpora and rust-bpf:
no prog is lost, insns +0.8%, states +1.0%, mem -0.9%.

Corpus                 Progs  Lost  Gained  Insns  States   Mem  Time
---------------------  -----  ----  ------  -----  ------  ----  ----
selftests               3043     0       1   -63%   +120%  -18%  -27%
selftests, no_alu32     2905     0       1   -66%    +86%  -27%  -34%
selftests, cpuv4        2914     0       1   -64%    +96%  -27%  -32%
selftests, bpf_gcc      2733     0       5   -69%    +96%  -16%  -24%
37 builds of Cilium     5189     0       0   -75%    +11%   -6%  -31%
sched-ext/scx            495     0       0   -89%    -34%  -66%  -67%
scx, as BPF CI has it    428     0       0   -85%    -22%  -38%  -39%
tools/sched_ext          240     0       0   -69%    +37%   +3%  -11%
public progs of Meta     511     0       0   -85%    -35%  -55%  -48%
all                    18458     0       8   -74%    +10%  -19%  -36%

Progs load in A and in B. The per cent compare sums over these progs:
75.1M -> 19.2M processed insns, 61 -> 39 s. Mem is mem_peak of veristat.
Time is the best of 3 runs on kernels without debug options. B counts each
state that arrives at a basic block as a state, A only those that it saves.
Gained: loop3 in the 4 builds, 4 progs that A rejects only in GCC's code.
Of the progs that take 1000 insns or more in B, two take over 5% more than
in A. loop5, 82 -> 4182: a walk up to its limit of 4096 insns, then a join.
res_spin_lock_test_held_lock_max, 2356 -> 4088: states that hold different
numbers of locks have no join.

rust-bpf               Progs  load in A  load in B  B: insns   time
16 schedulers            225         52        225      1.6M  1.9 s
crates                   428        135        423      8.3M  9.4 s
The insns are those of v4, like the four rows at the top of this letter.
For rust-bpf A and B have the kfuncs of patches 30-31 of [1] on top. They
are in no series now. No released compiler emits the code of rust-bpf.

How to read the series
======================
Start with patch 7, the overview. Then 4-5, 13-14, 15, 17 and 19.
  1-2    what is added to a prog: may_goto, the checks of recursion
  3-5    union and widening of a range, join.c: the join of two states
  6-10   data structures of flow.c, passes ahead of the main pass: order of
         the insns, places for may_goto, what later checks depend on
  11-14  the sets of states, iterations, what is not joined, the merge
  15-18  how states move: edges, calls and exits, loops, added edges
  19     the switch. The check of one insn against one state is the same.
  20-22  more counted loops, static subprogs on their own, recursion
  23-26  removal of pruning, precision tracking and SCCs: 3382 lines
  27-28  callx: a pointer to one of a set of functions, live registers
  29-31  loops that are too long for a walk: the first iteration, what
         a call returns, where the jumps of the loop stop a number
  32-34  more loops with a count: a jump in the middle of the body,
         a copy of the counter
  35-37  rules of today's verifier that plain C runs into: subprogs that
         return void, parts of pointers, 32-bit add to a pointer
  38-46  selftests: 38-39 adjust tests, 40-46 add tests
Each patch builds. Patches 1-18 add the code that patch 19 switches to.
Some selftests fail from patch 19 to 38.

Follow-ups
==========
 - Documentation/bpf/verifier.rst: the algorithm, with examples.
 - Liveness: no spilled pointer is kept for reads that see only its bytes.
 - No second pass over the prog when a loop gets may_goto late: half of the
   insns for rust-bpf.
 - bpf_prog_info tells that an invocation of the prog was cut short.

v3 -> v4
========
 - What comes around a loop, or back from the callback of bpf_loop(), is
   widened only with the state that it comes from. v3 took all states at
   the head: after 'ret = 0; if (..) { denied++; ret = -EPERM; }' in front
   of bpf_for() it widened ret, and rejected 'return ret' of an LSM prog
   (Eduard). The rule of flow_merge() on constant offsets of pointers in
   a loop is gone (patches 14, 15, 17, 19).
 - A jump in the middle of the body counts a loop, also when four other
   exits are nearer to the back edge. Ranges are one iteration shorter
   behind it. v3 rejected an index that is carried through bpf_for() and
   that only the count of such a jump bounds (Eduard, patches 32-33).
 - 'while (buf[i++] && i < N)' is counted: its exit tests a copy of the
   counter (Eduard, patch 34).
 - A counted loop joins and widens what has no closed form. v3 forgot it if
   the loop was too long for a walk: 'if (n < 8) v[n++] = 0;' in two loops
   of 100 iterations was rejected with 'for' and loaded with bpf_for()
   (patch 20).
 - The exits of a loop by way of a flag are found in one pass against the
   edges. v3 searched from each edge, 4096 insns per back edge: with 30 ifs
   in front of 'while (more)' the loop lost its count (patch 18).
 - may_goto at the head of a loop leaves the subprog if the code behind the
   loop reads what only the loop sets: glob_match() of arena_strsearch
   needs no cond_break (patch 19).
 - Widening: a sum that no jump of the loop depends on becomes an unknown
   scalar in two passes, not in five. A bound that leaves a fixed stop and
   32 bits is not tried at S64_MAX (patches 3, 4).
 - Patches 35-37 are new, with their tests in 45: what plain C runs into in
   today's verifier.
 - Patch 46 is new: 34 progs of the selftests in plain C that bpf-next
   rejects, five in some flavours of test_progs only.
 - Patches 29-35 of v3 are folded into the patches that add the code that
   they change: 3, 9, 14, 15, 19 and 21. 36-38 of v3 are 29-31.
 - Rebased to bpf-next 3e1821848412.

v2 -> v3
========
 - A scalar argument of a static subprog loses delta together with its id.
   v2 kept delta, took wrong bounds in the callee and let an access out of
   bounds load (sashiko).
 - A loop with calls is walked for two iterations. v2 widened it at once
   and rejected strobemeta.h with two maps or two strings.
 - The frame of a recursive subprog grows before the stack of a chain of
   calls is checked, not after (bpf-ci, sashiko).
 - A loop that ends when a flag is false is counted by the jump that clears
   the flag, also when code stands between that jump and the test. States
   with 0 and 1 in a flag are not joined in front of its test.
 - Patches 29-38 of v3 are new. Each loads code that bpf-next loads and v2
   rejects: casts, the context in a struct, r0 at a dead exit (bpf-ci,
   sashiko), a register that only a loop sets (bpf-ci), loops that count
   down in bytes, loop-carried values in long loops, and numbers that only
   a jump of the loop stops (Ihor).

v1 -> v2
========
v1 is part 3 of [1]. Its patches for the verifier of today and the time
slice of may_goto are in bpf-next.
 - Recursion: may_goto and r10, no per-cpu counters of bytes and calls.
 - flow.c comes in 9 patches and join.c in 2. In v1 each was one patch.
 - No public prog of Meta is lost (Ihor): widening goes to fixed stops, and
   constants that are added to a pointer to the packet are not joined.
 - A state is compared only with the states of a related shape (Ihor).
 - bpf_iter_*_next() stops a loop only if the iterator was created before
   the loop (Ihor). A loop with two entries is walked.
 - Offloaded progs get neither may_goto nor recursion (Ihor).
 - The failures of v1 in BPF CI (s390x, arm64, gcc) are fixed.

[1] https://lore.kernel.org/bpf/20261026025943.404450-1-alexei.starovoitov@gmail.com/

Signed-off-by: Alexei Starovoitov <ast@kernel.org>

Alexei Starovoitov (46):
  bpf: Add may_goto to the loops that the verifier marks
  bpf: Limit depth and time of recursive subprogs at run time
  bpf: Add union and widening of the range of a register
  bpf: Add join of registers and stack slots
  bpf: Add join of verifier states
  bpf: Tell which arguments of a call a join must leave alone
  bpf: Add the data structures of the main pass over the control flow
    graph
  bpf: Order the insns, find what stops a loop and where may_goto can go
  bpf: Find the registers that hold a number or a pointer to arena
  bpf: Find the registers and stack slots that later checks depend on
  bpf: Add the sets of states at an insn and the entries of functions
  bpf: Compare states of different iterations of a loop
  bpf: Decide which states are not joined
  bpf: Merge the state that arrives into the set at an insn
  bpf: Move states along edges, into loops and subprogs and out of them
  bpf: Make the state after a call from the caller's state and an exit
    state
  bpf: Enter and leave loops in the main pass, build flow.c
  bpf: Tell the passes over the graph where an added may_goto goes
  bpf: Verify programs as a fixpoint over the control flow graph
  bpf: Keep the count of a loop when a register has no closed form
  bpf: Verify static subprogs on their own
  bpf: Allow recursive subprogs
  bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth
  bpf: Remove precision tracking
  bpf: Remove prune points, levels of state comparison and pop_stack()
  bpf: Update comments about paths, pruning and explored states
  bpf: Track a pointer to one of a set of functions
  bpf: callx reads what the subprogs whose address is taken read
  bpf: Keep the first iteration of a counted loop apart
  bpf: What a call returns in each iteration is like what the loop loads
  bpf: Find where the jumps of a loop stop a number
  bpf: A loop is one iteration shorter behind the jump that counts it
  bpf: Give the places of a back edge to the jumps that count
  bpf: Count a loop whose exit tests a copy of the counter
  bpf: Allow tail calls and LD_ABS in subprogs that return void
  bpf: A part of a pointer that is spilled is a number
  bpf: 32-bit add to a pointer and pointer op pointer make a number
  selftests/bpf: Adjust tests to log lines that are gone
  selftests/bpf: Adjust tests to new verdicts, messages and order
  selftests/bpf: Test states that are not joined
  selftests/bpf: Test static subprogs that are verified on their own
  selftests/bpf: Test loops that are counted, walked or widened
  selftests/bpf: Test progs that run out of budget
  selftests/bpf: Test callx of one of a set of functions
  selftests/bpf: Test pointers that become numbers and subprogs that
    return void
  selftests/bpf: Test progs without bpf_for(), zero, can_loop and such

 Documentation/bpf/bpf_design_QA.rst           |    7 +-
 Documentation/bpf/kfuncs.rst                  |    2 +-
 Documentation/bpf/linux-notes.rst             |    4 +-
 include/linux/bpf.h                           |    6 +
 include/linux/bpf_verifier.h                  |  602 +--
 include/linux/filter.h                        |   16 +
 kernel/bpf/Makefile                           |    2 +-
 kernel/bpf/backtrack.c                        | 1052 ------
 kernel/bpf/cfg.c                              |  297 +-
 kernel/bpf/check_btf.c                        |   10 +-
 kernel/bpf/const_fold.c                       |  270 +-
 kernel/bpf/core.c                             |   19 +
 kernel/bpf/diagnostics.c                      |   13 -
 kernel/bpf/diagnostics.h                      |    1 -
 kernel/bpf/fixups.c                           |  333 ++
 kernel/bpf/flow.c                             | 3319 +++++++++++++++++
 kernel/bpf/join.c                             |  664 ++++
 kernel/bpf/liveness.c                         |  114 +-
 kernel/bpf/log.c                              |   14 +-
 kernel/bpf/loops.c                            |  176 +-
 kernel/bpf/scev.c                             |  460 ++-
 kernel/bpf/states.c                           | 1181 +-----
 kernel/bpf/verifier.c                         | 3168 ++++++++++------
 .../bpf/prog_tests/bpf_verif_scale.c          |    4 +-
 .../bpf/prog_tests/cgroup_skb_egress_ret.c    |   23 +
 .../selftests/bpf/prog_tests/log_buf.c        |    4 +-
 .../selftests/bpf/prog_tests/log_fixup.c      |    6 +-
 .../selftests/bpf/prog_tests/plain_c.c        |  160 +
 .../selftests/bpf/prog_tests/reg_bounds.c     |    5 +-
 .../prog_tests/test_struct_ops_refcounted.c   |    2 +
 .../selftests/bpf/prog_tests/verifier.c       |   19 +
 .../selftests/bpf/progs/aggregate_ret_func.c  |   15 -
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |    7 -
 ...f_qdisc_fail__invalid_dynptr_cross_frame.c |    4 +-
 .../bpf/progs/cgroup_skb_egress_ret.c         |   12 +
 .../bpf/progs/compute_live_registers.c        |    8 +-
 .../selftests/bpf/progs/cpumask_failure.c     |    2 +-
 .../testing/selftests/bpf/progs/dynptr_fail.c |    9 +-
 .../selftests/bpf/progs/exceptions_assert.c   |   24 +-
 tools/testing/selftests/bpf/progs/iters.c     |  114 +-
 .../selftests/bpf/progs/iters_state_safety.c  |    8 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |   12 +-
 .../bpf/progs/mem_rdonly_untrusted.c          |   16 +-
 .../selftests/bpf/progs/plain_c_arena_htab.c  |  222 ++
 .../bpf/progs/plain_c_arena_htab_asm.c        |    5 +
 .../selftests/bpf/progs/plain_c_arena_list.c  |  181 +
 .../selftests/bpf/progs/plain_c_arena_race.c  |   45 +
 .../bpf/progs/plain_c_arena_strsearch.c       |  263 ++
 .../bpf/progs/plain_c_bpf_ma_ttrace.c         |   53 +
 .../bpf/progs/plain_c_bpf_nop_bench.c         |   75 +
 .../bpf/progs/plain_c_bpf_qdisc_fifo.c        |  132 +
 .../bpf/progs/plain_c_bpf_qdisc_fq.c          |  728 ++++
 .../bpf/progs/plain_c_cpumask_success.c       |   83 +
 .../bpf/progs/plain_c_dynptr_success.c        |  175 +
 .../selftests/bpf/progs/plain_c_iters.c       |  147 +
 .../bpf/progs/plain_c_mem_rdonly_untrusted.c  |   24 +
 .../bpf/progs/plain_c_mptcp_subflow.c         |  116 +
 .../bpf/progs/plain_c_rbtree_search.c         |  136 +
 .../bpf/progs/plain_c_rbtree_search_kptr.c    |  264 ++
 .../bpf/progs/plain_c_strobemeta_bpf_loop.c   |  583 +++
 .../bpf/progs/plain_c_tailcall_bpf2bpf2.c     |   40 +
 .../plain_c_tailcall_bpf2bpf_hierarchy1.c     |   44 +
 .../plain_c_tailcall_bpf2bpf_hierarchy2.c     |   81 +
 .../plain_c_tailcall_bpf2bpf_hierarchy3.c     |   74 +
 .../bpf/progs/plain_c_tailcall_callback.c     |   80 +
 .../bpf/progs/plain_c_test_wakeup_source.c    |   93 +
 .../plain_c_tracing_stack_arg_tailcall_tgt.c  |   55 +
 .../bpf/progs/plain_c_verifier_arena_large.c  |  122 +
 .../plain_c_verifier_iterating_callbacks.c    |   25 +
 .../bpf/progs/plain_c_verifier_loop_as_is.c   |   40 +
 .../bpf/progs/plain_c_xdp_lb_bench.c          |  723 ++++
 .../selftests/bpf/progs/stack_arg_precision.c |   48 +-
 .../bpf/progs/struct_ops_refcounted_subprog.c |   30 +
 .../selftests/bpf/progs/test_global_func15.c  |    5 -
 .../selftests/bpf/progs/test_global_func3.c   |  125 +
 .../selftests/bpf/progs/timer_failure.c       |    9 +-
 .../selftests/bpf/progs/verifier_align.c      |   10 +-
 .../selftests/bpf/progs/verifier_and.c        |    4 +-
 .../bpf/progs/verifier_arena_stack.c          |   82 +
 .../bpf/progs/verifier_array_access.c         |   18 +-
 .../selftests/bpf/progs/verifier_bounds.c     |  114 +-
 .../bpf/progs/verifier_bounds_deduction.c     |    2 +-
 .../bpf/progs/verifier_bounds_step.c          |   73 +-
 .../selftests/bpf/progs/verifier_budget.c     | 1915 ++++++++++
 .../bpf/progs/verifier_calls_in_a_row.c       |  223 ++
 .../selftests/bpf/progs/verifier_callx.c      |  128 +-
 .../selftests/bpf/progs/verifier_callx_many.c |  113 +
 .../bpf/progs/verifier_callx_rodata.c         |  167 +-
 .../selftests/bpf/progs/verifier_cfg.c        |    8 +-
 .../bpf/progs/verifier_global_subprogs.c      |   11 +-
 .../bpf/progs/verifier_helper_value_access.c  |    8 +-
 .../selftests/bpf/progs/verifier_idoms.c      |    9 +-
 .../bpf/progs/verifier_iterating_callbacks.c  |   70 +-
 .../bpf/progs/verifier_jumps_ahead.c          |  670 ++++
 .../selftests/bpf/progs/verifier_ld_ind.c     |   31 +-
 .../bpf/progs/verifier_linked_scalars.c       |    7 +-
 .../selftests/bpf/progs/verifier_live_stack.c |   37 +-
 .../bpf/progs/verifier_liveness_exp.c         |    7 +-
 .../selftests/bpf/progs/verifier_loop_as_is.c |  513 +++
 .../bpf/progs/verifier_loop_hierarchy.c       |  125 +-
 .../selftests/bpf/progs/verifier_loop_latch.c |  308 +-
 .../selftests/bpf/progs/verifier_loop_need.c  |  372 ++
 .../selftests/bpf/progs/verifier_loop_walk.c  |  306 ++
 .../selftests/bpf/progs/verifier_loops1.c     |   16 +-
 .../bpf/progs/verifier_map_ret_val.c          |    2 +-
 .../bpf/progs/verifier_mem_size_reg.c         |   52 +-
 .../selftests/bpf/progs/verifier_movsx.c      |    3 +-
 .../selftests/bpf/progs/verifier_precision.c  |  186 +-
 .../selftests/bpf/progs/verifier_ptr_off.c    |  279 ++
 .../selftests/bpf/progs/verifier_scalar_ids.c |  329 +-
 .../selftests/bpf/progs/verifier_scev.c       |  272 +-
 .../bpf/progs/verifier_search_pruning.c       |   40 +-
 .../selftests/bpf/progs/verifier_spill_fill.c |  271 +-
 .../selftests/bpf/progs/verifier_spin_lock.c  |    4 +-
 .../selftests/bpf/progs/verifier_stack_arg.c  |  181 +
 .../selftests/bpf/progs/verifier_stack_ptr.c  |   13 +-
 .../bpf/progs/verifier_subprog_callers.c      |  543 +++
 .../bpf/progs/verifier_subprog_insn_stats.c   |   16 +-
 .../bpf/progs/verifier_subprog_precision.c    |  353 +-
 .../selftests/bpf/progs/verifier_tailcall.c   |   96 +
 .../selftests/bpf/progs/verifier_uninit.c     |    2 +-
 .../selftests/bpf/progs/verifier_unpriv.c     |   27 +-
 .../bpf/progs/verifier_unpriv_perf.c          |    4 +-
 .../bpf/progs/verifier_value_illegal_alu.c    |   42 +-
 .../bpf/progs/verifier_value_or_null.c        |    2 +-
 .../bpf/progs/verifier_value_ptr_arith.c      |   65 +-
 .../selftests/bpf/progs/verifier_var_off.c    |  206 +-
 tools/testing/selftests/bpf/verifier/calls.c  |   46 +-
 .../testing/selftests/bpf/verifier/precise.c  |   69 +-
 129 files changed, 19983 insertions(+), 5477 deletions(-)
 delete mode 100644 kernel/bpf/backtrack.c
 create mode 100644 kernel/bpf/flow.c
 create mode 100644 kernel/bpf/join.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/plain_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_htab.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_htab_asm.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_list.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_race.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_strsearch.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_ma_ttrace.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_nop_bench.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fifo.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fq.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_cpumask_success.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_dynptr_success.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_iters.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_mem_rdonly_untrusted.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_mptcp_subflow.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_rbtree_search.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_rbtree_search_kptr.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_strobemeta_bpf_loop.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf2.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy1.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy2.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy3.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_callback.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tracing_stack_arg_tailcall_tgt.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_verifier_arena_large.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_verifier_iterating_callbacks.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_verifier_loop_as_is.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_xdp_lb_bench.c
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_budget.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_many.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_need.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_walk.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_ptr_off.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_callers.c

-- 
2.55.0

[PATCH bpf-next v4 01/46] bpf: Add may_goto to the loops that the verifier marks

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

SpinLock::lock() of rust-bpf (scx_common.rs), here in central_enqueue:

  while self.locked.compare_exchange(0, 1, Acquire, Relaxed).is_err() {
      while self.locked.load(Relaxed) != 0 { core::hint::spin_loop(); }
  }

  infinite loop detected at insn 127

111 of the 225 progs of the schedulers of rust-bpf get this message. Memory
decides when the loop ends: no analysis finds a bound. The same holds for
a lookup in a BTreeMap. C has bpf_for(), can_loop, bpf_loop(). Nobody can
put them into core and alloc.

With one state for all iterations the verifier can still show that each of
them is safe. Then the kernel must stop the loop at run time. may_goto does
that: after 250 ms it jumps to its target. That jump is a new edge in the
prog, and must be verified. One target for all loops is not possible:

  lock(), see above                 pick_idle_cpu_in() of scx_p2dq.rs
  Not to the exit of the loop:        let cores = core.then(..::idle_smt);
  the caller goes on as if it         cpus.filter(..).find(..)
  had the lock. Unwind instead:     Not out of the function: the code
                                    after the loop releases the cpumask.
  may_goto L
  insn                              may_goto T
  ...                               if ... goto T
  last insn of the subprog
  L: r0 = 0
  unwind_exit, main prog: exit      T is outside of the loop

Thus the main pass selects the edge for each loop, verifies it, and
records it in insn_aux_data. bpf_insert_loop_guards() adds the insns after
the main pass. No code sets loop_guard in this patch: no prog changes.
 - unwind_exit is for a subprog whose callers have cleanup code for it
   (bail_unwinds): their destructors run, the code after the call does not.
 - The inner loop of lock() goes around without a jump that leaves. Then
   may_goto is at the loop head (guard_at_header): 142 of the 225 progs.
 - may_goto has a 16-bit offset. If L is more than 2048 insns away it gets
   'r0 = 0; exit' of its own, and one more jump in each iteration.

Progs without such a loop get the same code. 174 of the 225 progs have one.
An added may_goto costs what a may_goto of the prog costs. It is 10 insns,
14 if no register is dead: thus it gets the live registers of the insn that
follows it. That, and one L per subprog, not per loop: 6 % less jited code.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  33 ++++++
 kernel/bpf/fixups.c          | 219 +++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |   4 +
 3 files changed, 256 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d6e41444ab14..7e268540dae9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -815,6 +815,10 @@ struct bpf_insn_aux_data {
 	u64 cast_arena:1; /* addr_space_cast of an address in arena */
 	u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
 	u64 bb_end:1; /* last instruction of a basic block */
+	u64 loop_guard:1; /* may_goto is added in front of this instruction */
+	u64 guard_place:1; /* it's the place for may_goto if the loop needs one */
+	u64 guard_at_header:1; /* there is no better place than the head of the loop */
+	u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
 	u64 need_scev:1;
 	u64 alu_state:5; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
@@ -878,6 +882,8 @@ struct bpf_insn_aux_data {
 	 * For a loop header, identifies the parent loop instead.
 	 */
 	s32 loop_header;
+	/* where may_goto in front of this instruction goes */
+	s32 guard_to;
 	/* additional information about the loop if this instruction is a loop header */
 	struct bpf_loop *loop;
 };
@@ -1021,6 +1027,8 @@ struct bpf_subprog_info {
 	 * a hidden pointer to the result, a pair in two registers, no dead arguments.
 	 */
 	bool stack_args_from_code: 1;
+	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
+	bool bail_unwinds: 1;
 	u8 code_stack_args;
 	u16 stack_arg_cnt; /* incoming + max outgoing */
 	u16 max_out_stack_arg_cnt;
@@ -1521,6 +1529,7 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_split_cur_state(struct bpf_verifier_env *env);
@@ -1845,6 +1854,30 @@ struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *en
 const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);
 int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
+
+/*
+ * Where the program goes from may_goto and from the checks that the verifier adds:
+ * - May_goto in front of a jump goes where the jump goes, with the registers and the stack
+ *   as they are in front of the jump. bpf_insn_successors() has that edge.
+ * - A subprog that is out of budget goes to bail_insn, an exit. No such edge.
+ * - May_goto in front of the head of a loop goes to where the loops around it are left: from
+ *   @idx to what is returned here, -1 if there is none. No such edge either. The registers
+ *   and the stack are what they are in front of @idx.
+ *
+ * The main pass looks at the states that go there. What is computed from the graph before
+ * the main pass has to take these edges into account where the main pass believes it:
+ * - Against the direction of the edges, liveness: a register that is live at the target is
+ *   live at @idx. Stack that is dead at @idx stays dead: a read of it at the target is
+ *   rejected.
+ * - In the direction of the edges, bpf_compute_const_regs() and compute_subprog_args(): what
+ *   holds on all edges of the graph to the target doesn't hold on this one.
+ */
+static inline int bpf_head_guard_target(const struct bpf_verifier_env *env, int idx)
+{
+	const struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+
+	return aux->guard_at_header ? aux->guard_to : -1;
+}
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index f95af2ca3591..6fd79c2fbdc1 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -587,6 +587,225 @@ static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
 	return 0;
 }
 
+static bool jumps_to_itself(const struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+	return (class == BPF_JMP || class == BPF_JMP32) && op != BPF_CALL && op != BPF_EXIT &&
+	       insn->code != (BPF_JMP32 | BPF_JA) && insn->off == -1;
+}
+
+/* How a subprog leaves when it is out of budget, see push_budget_exit() */
+static struct bpf_insn budget_exit_insn(struct bpf_verifier_env *env, struct bpf_subprog_info *sub)
+{
+	if (!sub->bail_unwinds)
+		return BPF_EXIT_INSN();
+	sub->can_unwind = true;
+	env->prog->jit_required = 1;
+	return BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0);
+}
+
+/*
+ * may_goto has 16 bits for the offset, and what is between it and its target is going
+ * to grow: may_goto itself becomes 10 insns, 14 when it has to save a register.
+ */
+#define BPF_GUARD_NEAR	2048
+
+static bool guard_leaves_subprog(const struct bpf_insn_aux_data *aux)
+{
+	return aux->loop_guard && !aux->guard_no_unwind;
+}
+
+/*
+ * 'r0 = 0; exit' behind the last insn of the subprog, for all its guards that are near.
+ * Returns the number of insns that are going to be added.
+ */
+static int append_budget_exit(struct bpf_verifier_env *env, int s)
+{
+	struct bpf_subprog_info *sub = &env->subprog_info[s];
+	int last = (sub + 1)->start - 1, n = 0, i;
+	struct bpf_insn buf[4], *insn;
+
+	for (i = max_t(int, sub->start, last + 1 - BPF_GUARD_NEAR); i <= last; i++)
+		if (guard_leaves_subprog(&env->insn_aux_data[i]))
+			break;
+	if (i > last)
+		return 0;
+
+	buf[n++] = env->prog->insnsi[last];
+	buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+	if (bpf_ret_reg_pair(env, s))
+		buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
+	buf[n++] = budget_exit_insn(env, sub);
+	insn = bpf_patch_list_add(env, last, buf, n);
+	return IS_ERR(insn) ? PTR_ERR(insn) : n - 1;
+}
+
+/*
+ * Add may_goto in front of the insns that the main pass marked, see
+ * check_loop_guard(). It verified the edge from the insn to where may_goto goes.
+ */
+int bpf_insert_loop_guards(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int len = env->prog->len, i, r, s, to, n, jmp, err = 0;
+	struct bpf_insn *patch, *insn;
+	int *target, *leave, *moved;
+	bool indirect;
+	u16 *zero;
+	s64 off;
+
+	for (i = 0; i < len; i++)
+		if (aux[i].loop_guard)
+			break;
+	if (i == len)
+		return 0;
+
+	/*
+	 * Where the subprog has 'r0 = 0; exit', 0 if it has none. And how far that moved
+	 * the subprog: guard_to is from the time before.
+	 */
+	leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
+	if (!leave)
+		return -ENOMEM;
+	moved = leave + env->subprog_cnt;
+	for (s = 0; s < env->subprog_cnt && !err; s++) {
+		leave[s] = append_budget_exit(env, s);
+		if (leave[s] < 0)
+			err = leave[s];
+	}
+	/* A guard cannot jump into a patch that is in the list: 'r0 = 0' has no index yet. */
+	err = err ?: bpf_patch_list_commit(env);
+	if (err) {
+		kvfree(leave);
+		return err;
+	}
+	aux = env->insn_aux_data;
+	len = env->prog->len;
+	for (s = 0, n = 0; s < env->subprog_cnt; s++) {
+		moved[s] = n;
+		n += leave[s];
+		if (!leave[s])
+			continue;
+		i = env->subprog_info[s + 1].start;
+		leave[s] = i - leave[s];
+		/*
+		 * The list gives what is known about an insn to the last insn of its patch,
+		 * but for indirect_target. Here the original insn is the first one.
+		 */
+		indirect = aux[leave[s] - 1].indirect_target;
+		swap(aux[leave[s] - 1], aux[i - 1]);
+		aux[leave[s] - 1].indirect_target = indirect;
+		aux[i - 1].indirect_target = false;
+		/* the last insn may be dead code */
+		while (--i >= leave[s])
+			aux[i].seen = 1;
+	}
+
+	/* where the guard of the insn goes */
+	target = kvcalloc(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+	zero = kvcalloc(len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+	patch = kvcalloc(8, sizeof(*patch), GFP_KERNEL_ACCOUNT);
+	if (!target || !zero || !patch) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < len; i++) {
+		s = bpf_find_containing_subprog(env, i) - env->subprog_info;
+		target[i] = aux[i].loop_guard ? aux[i].guard_to + moved[s] : -1;
+		if (guard_leaves_subprog(&aux[i])) {
+			/* see push_budget_exit() */
+			if (leave[s] && leave[s] - i <= BPF_GUARD_NEAR) {
+				target[i] = leave[s];
+			} else {
+				target[i] = -2;
+				zero[i] = BIT(BPF_REG_0);
+				if (bpf_ret_reg_pair(env, s))
+					zero[i] |= BIT(BPF_REG_2);
+			}
+		}
+	}
+
+	for (i = 0; i < len; i++) {
+		to = target[i];
+		if (to == -1)
+			continue;
+		/*
+		 *	may_goto to		or	may_goto +1
+		 *	insn				goto +N
+		 *					r0 = 0, ...
+		 *					exit
+		 *					insn
+		 */
+		n = 0;
+		if (zero[i]) {
+			patch[n++] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0);
+			patch[n++] = BPF_JMP_A(hweight16(zero[i]) + 1);
+			for (r = BPF_REG_0; r < BPF_REG_FP; r++)
+				if (zero[i] & BIT(r))
+					patch[n++] = BPF_MOV64_IMM(r, 0);
+		}
+		jmp = n++;
+		/* What is behind moves with this patch. The other patches are not there yet. */
+		if (to == -2)
+			off = 0;
+		else if (to > i)
+			off = to - i + n - jmp - 1;
+		else
+			off = to - (i + jmp) - 1;
+		if (off > S16_MAX || off < S16_MIN) {
+			verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
+			err = -ERANGE;
+			break;
+		}
+		patch[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
+		if (to == -2)
+			patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env, i));
+		patch[n] = env->prog->insnsi[i];
+		/* A jump to itself is a back edge. It goes through may_goto like the others. */
+		if (jumps_to_itself(&patch[n]))
+			patch[n].off -= n;
+		else
+			err = bpf_adj_moved_insn(&patch[n], n);
+		if (err) {
+			verbose(env, "insn %d cannot be patched due to 16-bit range\n", i);
+			break;
+		}
+		n++;
+		insn = bpf_patch_list_add(env, i, patch, n);
+		if (IS_ERR(insn)) {
+			err = PTR_ERR(insn);
+			break;
+		}
+	}
+	err = err ?: bpf_patch_list_commit(env);
+	if (err)
+		goto out;
+	/*
+	 * bpf_do_misc_fixups() keeps the address of the count of may_goto in a register that is
+	 * dead at the may_goto. A new insn has no live_regs_before: all registers would be dead.
+	 * What is live in front of the insn is live in front of its guard. Nothing else is:
+	 *  - The guard of a jump goes where the jump goes.
+	 *  - What is live where the guard at the head of a loop goes is live at the head,
+	 *    see bpf_head_guard_target().
+	 *  - On the way out of the subprog R0, and R2 of a pair, are set to 0.
+	 */
+	aux = env->insn_aux_data;
+	for (i = 0, n = 0; i < len; i++) {
+		if (target[i] == -1)
+			continue;
+		r = zero[i] ? 3 + hweight16(zero[i]) : 1;
+		aux[i + n].live_regs_before = aux[i + n + r].live_regs_before;
+		n += r;
+	}
+out:
+	kvfree(leave);
+	kvfree(target);
+	kvfree(zero);
+	kvfree(patch);
+	return err;
+}
+
 /* Clean up dynamically allocated fields of aux data for instructions [start, ...] */
 void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
 {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1b58a1796a2e..eb51f7f04153 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -24673,9 +24673,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret == 0 && env->callx_edges)
 		ret = sort_subprogs_topo(env);
 
+	/* the driver numbers the insns like the program that it was given */
 	if (ret == 0 && bpf_prog_is_offloaded(env->prog->aux))
 		ret = bpf_prog_offload_finalize(env);
 
+	if (ret == 0)
+		ret = bpf_insert_loop_guards(env);
+
 skip_full_check:
 	kvfree(env->explored_states);
 
-- 
2.55.0

Re: [PATCH bpf-next v4 01/46] bpf: Add may_goto to the loops that the verifier marks

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] bpf: bpf_insert_loop_guards() logs shifted insn index instead of
  orig_idx
--

commit 4c080c1282306b111e989d48659533beacc7c4d9
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Add may_goto to the loops that the verifier marks

This commit introduces bpf_insert_loop_guards() to insert may_goto
instructions and subprogram exit sequences for loops marked by the
verifier during the main pass.

> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index f95af2ca3591b..6fd79c2fbdc1e 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -587,6 +587,225 @@ static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
[ ... ]
> +int bpf_insert_loop_guards(struct bpf_verifier_env *env)
> +{
[ ... ]
> +	for (s = 0; s < env->subprog_cnt && !err; s++) {
> +		leave[s] = append_budget_exit(env, s);
> +		if (leave[s] < 0)
> +			err = leave[s];
> +	}
> +	/* A guard cannot jump into a patch that is in the list: 'r0 = 0' has no index yet. */
> +	err = err ?: bpf_patch_list_commit(env);
[ ... ]
> +		if (off > S16_MAX || off < S16_MIN) {
> +			verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
> +			err = -ERANGE;
> +			break;
> +		}
> +		patch[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
> +		if (to == -2)
> +			patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env, i));
> +		patch[n] = env->prog->insnsi[i];
> +		/* A jump to itself is a back edge. It goes through may_goto like the others. */
> +		if (jumps_to_itself(&patch[n]))
> +			patch[n].off -= n;
> +		else
> +			err = bpf_adj_moved_insn(&patch[n], n);
> +		if (err) {
> +			verbose(env, "insn %d cannot be patched due to 16-bit range\n", i);
> +			break;
> +		}

[Severity: Low]
Should these two verbose() messages log aux[i].orig_idx instead of i?

The earlier bpf_patch_list_commit() call inserts append_budget_exit()
sequences at the end of subprograms, which shifts instruction indices i in
later subprograms away from their original positions in the user-supplied
program.

In addition, calling bpf_insert_loop_guards() in bpf_check() before
check_max_stack_depth() shifts the raw instruction indices reported by
check_exception_cb(), check_recursion(), and check_callbacks() when
check_max_stack_depth() rejects a program.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=1

[PATCH bpf-next v4 02/46] bpf: Limit depth and time of recursive subprogs at run time

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Rust recurses where C has a loop. scx_flatcg.rs of rust-bpf has

  struct CgrpCtx { parent: Option<Arc<CgrpCtx>>, ... }

The drop of the last Arc of a cgroup drops the Arc of its parent:
Arc::drop_slow() calls itself, and 8 of the 15 progs are rejected:
"recursive call from drop_slow<..>() to drop_slow<..>()". quicksort() under
sort_unstable_by() in scx_rusty.rs, BTreeMap, serde_json and regex-automata
recurse too: 100 of the 225 progs of the schedulers of rust-bpf, 100 of its
437 tests. The verifier cannot know how deep the calls go, or how many
there are.

Limit both at run time. Each subprog on a cycle of the call graph starts
with two checks:

  entry:  may_goto bail                                 limits the time
          if (r10 is in the upper half of the stack)    limits the depth
                  goto body
          bpf_prog_report_recursion()
  bail:   r0 = 0; exit
  body:   ...

Both are needed. The depth does not limit the time:

  int fib(int n) { return n < 2 ? n : fib(n - 1) + fib(n - 2); }

fib(90) is never deeper than 90 frames and makes 10^19 calls. There is no
loop in it, so no may_goto of a loop bounds it. The cycle is in the call
graph, and it goes through the first insn of fib(). With may_goto there
fib(90) ends after 250 ms, the time slice that the loops and recursions of
an invocation share. The time does not limit the depth:
'int f(int n) { return f(n + 1) + 1; }' is out of stack in less than 1 ms.

r10 is an address in the kernel stack that the prog runs on. Thus the check
sees all that is in this stack, not only the frames of this prog, without
per-cpu state. On arm64 a stack is aligned to its size: one bit of r10
tells the half. On x86-64 r10 is compared with the tops of the task stack
and the irq stack. In any other stack, like in an NMI that interrupts the
kernel, the check fails. Other archs get no recursion. The stack in arena
has no check: the verifier makes the frame in the kernel stack so large
that a half of that stack is full before the stack in arena. It does so in
check_call_depth(): the limit on the stack of a chain of calls has to see
the frames that the JIT gets.

No subprog is marked as recursive yet. A call runs 12 more insns on x86-64
(17 on the irq stack), 8 on arm64: +0.2 to +0.5 nsec on x86-64.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |   3 +
 include/linux/filter.h       |  16 +++++
 kernel/bpf/core.c            |  19 ++++++
 kernel/bpf/fixups.c          | 113 +++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |  49 +++++++++++++++
 6 files changed, 202 insertions(+)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index b011acc22c50..cdf3c9b90987 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1944,6 +1944,8 @@ struct bpf_prog_aux {
 	bool has_prologue;
 	bool might_sleep;
 	bool can_unwind;
+	/* frames of recursive subprogs can be on the stack when it is called */
+	bool in_recursion;
 	bool kprobe_write_ctx;
 	/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
 	bool uses_arena_insns;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7e268540dae9..2f935061b8fd 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -995,6 +995,8 @@ struct bpf_subprog_info {
 	bool has_tail_call: 1;
 	bool might_throw: 1;
 	bool can_unwind: 1; /* has unwind_exit */
+	bool recursive: 1; /* on a cycle of the call graph */
+	bool in_recursion: 1; /* recursive, or a recursive subprog gets to it */
 	bool via_callx: 1; /* callx calls it */
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
@@ -1532,6 +1534,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
+int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
 int bpf_split_cur_state(struct bpf_verifier_env *env);
 int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 
diff --git a/include/linux/filter.h b/include/linux/filter.h
index a58ad2318412..61448be53f8d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1401,6 +1401,21 @@ bool bpf_jit_supports_large_stack(void);
 bool bpf_jit_supports_timed_may_goto(void);
 bool bpf_jit_supports_fsession(void);
 
+/*
+ * Whether a recursion in @prog can be limited at run time:
+ *  - r10 tells how much of the kernel stack is used, on the architectures whose stacks
+ *    emit_stack_check() knows.
+ *  - All may_goto of the prog share one time slice. That is the timed form, which only JITed
+ *    progs get, see bpf_do_misc_fixups(). The other form counts in each frame.
+ *  - A walk of the stack finds the prog when there is no room, to tell user space, see
+ *    bpf_prog_report_recursion().
+ */
+static inline bool bpf_prog_has_invocation_budget(const struct bpf_prog *prog)
+{
+	return (IS_ENABLED(CONFIG_X86_64) || IS_ENABLED(CONFIG_ARM64)) && prog->jit_requested &&
+	       bpf_jit_supports_timed_may_goto() && bpf_jit_supports_exceptions();
+}
+
 struct bpf_jit_arg_abi {
 	/* Argument registers of the kernel convention. */
 	u8 nr_arg_regs;
@@ -1438,6 +1453,7 @@ u64 arch_bpf_timed_may_goto(void);
 u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *);
 void bpf_timed_may_goto_save(const struct bpf_prog *prog, struct bpf_timed_may_goto *saved);
 void bpf_timed_may_goto_restore(const struct bpf_prog *prog, struct bpf_timed_may_goto *saved);
+u64 bpf_prog_report_recursion(void);
 bool bpf_helper_changes_pkt_data(enum bpf_func_id func_id);
 
 static inline bool bpf_dump_raw_ok(const struct cred *cred)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 8e2fb66940c7..f88a1c041a7a 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3956,6 +3956,25 @@ static noinline void bpf_prog_report_may_goto_violation(void)
 #endif
 }
 
+/* Called by a recursive subprog that finds no room, see bpf_insert_recursion_checks() */
+u64 bpf_prog_report_recursion(void)
+{
+#ifdef CONFIG_BPF_SYSCALL
+	struct bpf_stream_stage ss;
+	struct bpf_prog *prog;
+
+	prog = bpf_prog_find_from_stack();
+	/* The calls that follow get here too. One report is enough until user space reads it. */
+	if (!prog || atomic_read(&prog->aux->stream[BPF_STDERR - 1]->capacity))
+		return 0;
+	bpf_stream_stage(ss, prog, BPF_STDERR, ({
+		bpf_stream_printk(ss, "ERROR: No room in the stack for a call of a recursive function\n");
+		bpf_stream_dump_stack(ss);
+	}));
+#endif
+	return 0;
+}
+
 u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *p)
 {
 	u64 time = ktime_get_mono_fast_ns();
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 6fd79c2fbdc1..3413c7addca8 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -806,6 +806,118 @@ int bpf_insert_loop_guards(struct bpf_verifier_env *env)
 	return err;
 }
 
+/*
+ * if (r10 is in the upper half of the kernel stack) goto +@off
+ *
+ * r10 of a subprog without a private stack is an address in the kernel stack that it runs on.
+ * arm64: the stacks of tasks, of interrupts and of SDEI events have THREAD_SIZE bytes and are
+ * aligned to twice that size. One bit of an address tells the half.
+ * x86-64: the stack for interrupts is aligned to a page only. But the cpu has the top of the stack
+ * of the task and the top of its stack for interrupts. In any other stack (NMI, #DB, #MC, #DF,
+ * #VC) the check fails: they have half of the size.
+ * R0 has nothing at the entry of a subprog.
+ */
+static int emit_stack_check(struct bpf_insn *buf, int off)
+{
+	int n = 0;
+
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+	const void __percpu *top[] = { &cpu_current_top_of_stack, &hardirq_stack_ptr };
+	int i;
+
+	static_assert(IRQ_STACK_SIZE == THREAD_SIZE);
+	for (i = 0; i < 2; i++) {
+		buf[n++] = BPF_MOV64_IMM(BPF_REG_0, (u32)(__force unsigned long)top[i]);
+		buf[n++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
+		buf[n++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0);
+		buf[n++] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_10);
+		buf[n++] = BPF_JMP_IMM(BPF_JLT, BPF_REG_0, THREAD_SIZE / 2, i ? off : off + 5);
+	}
+#else
+	buf[n++] = BPF_JMP_IMM(BPF_JSET, BPF_REG_10, THREAD_SIZE / 2, off);
+#endif
+	return n;
+}
+
+/*
+ * How deep a recursion goes is not known, so it is limited at run time. In the same
+ * spirit as may_goto: the program goes on for as long as there is room and takes a way
+ * out that was verified when there is none.
+ *
+ * Every subprog on a cycle of the call graph begins with:
+ *
+ *	may_goto bail;
+ *	if (r10 is in the upper half of the kernel stack) goto body;
+ *	bpf_prog_report_recursion();
+ * bail:
+ *	r0 = 0;
+ *	exit;
+ * body:
+ *
+ * It is the stack as it is: whatever is there is counted, be it the callers of the prog in the
+ * kernel, another prog, a trampoline or a helper that calls a callback. An invocation in
+ * another stack has nothing to do with it.
+ *
+ * The lower half is for what is below r10: the frame of the subprog, then what it calls.
+ * That is the next recursive subprog, which doesn't get past its check, or a chain of calls that
+ * check_max_stack_depth() looked at together with that frame.
+ *
+ * A subprog that calls itself twice doesn't have to go deep to go on forever: hence may_goto.
+ * Once the time slice of the invocation is over every call returns at once, and the frames
+ * that are there come to an end.
+ */
+int bpf_insert_recursion_checks(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	struct bpf_insn buf[24], nop = BPF_JMP_A(0), *insn;
+	bool found = false;
+	int s, n, jmp, err;
+
+	for (s = 1; s < env->subprog_cnt; s++)
+		found |= si[s].recursive;
+	if (!found)
+		return 0;
+	env->prog->jit_required = 1;
+
+	for (s = 1; s < env->subprog_cnt; s++) {
+		if (!si[s].recursive)
+			continue;
+		n = 0;
+		/* The exit drops the frame in the stack in arena. Take it before the check. */
+		if (is_arena_sp_adj(&env->prog->insnsi[si[s].start])) {
+			buf[n++] = env->prog->insnsi[si[s].start];
+			/* one insn for one: done at once, the list copies the nop */
+			insn = bpf_patch_list_add(env, si[s].start, &nop, 1);
+			if (IS_ERR(insn))
+				return PTR_ERR(insn);
+		}
+		jmp = n++;
+		n += emit_stack_check(buf + n, bpf_ret_reg_pair(env, s) ? 4 : 3);
+		/* The arguments are not needed any more */
+		buf[n++] = BPF_EMIT_KERNEL_CALL(bpf_prog_report_recursion);
+		buf[jmp] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, n - jmp - 1, 0);
+		buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+		if (bpf_ret_reg_pair(env, s))
+			buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
+		buf[n++] = budget_exit_insn(env, &si[s]);
+		/* calls get to the check, jumps to the first insn do not */
+		err = bpf_patch_list_insert(env, si[s].start, buf, n);
+		if (err)
+			return err;
+	}
+	err = bpf_patch_list_commit(env);
+	if (err)
+		return err;
+	/* may_goto keeps the address of its count in a register that is dead: R0 */
+	for (s = 1; s < env->subprog_cnt; s++) {
+		if (!si[s].recursive)
+			continue;
+		n = si[s].start + is_arena_sp_adj(&env->prog->insnsi[si[s].start]);
+		env->insn_aux_data[n].live_regs_before = (u16)~BIT(BPF_REG_0);
+	}
+	return 0;
+}
+
 /* Clean up dynamically allocated fields of aux data for instructions [start, ...] */
 void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
 {
@@ -1878,6 +1990,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
 		func[i]->aux->can_unwind = env->subprog_info[i].can_unwind;
+		func[i]->aux->in_recursion = env->subprog_info[i].in_recursion;
 		func[i]->aux->token = prog->aux->token;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index eb51f7f04153..c16a20ab664d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6112,6 +6112,47 @@ static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_gra
 	} while (changed);
 }
 
+/*
+ * The stack in arena has no check of its own, see bpf_insert_recursion_checks(). A frame
+ * of a recursive subprog takes at least the same part of a half of the kernel stack as of the
+ * room in the stack in arena: there is room in arena for the frames that pass the check. The
+ * last of them and the one that doesn't pass come on top.
+ *
+ * Returns 1 when frames in the kernel stack grew.
+ */
+static int pad_recursive_frames(struct bpf_verifier_env *env, struct bpf_call_graph *cg,
+				struct bpf_subprog_call_depth_info *dinfo)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	u32 arena_frame = 0, depth = 0, room, frame;
+	int s;
+
+	for_each_set_bit(s, cg->live, cg->cnt) {
+		depth = max_t(u32, depth, dinfo[s].arena_depth + si[s].arena_stack_depth);
+		if (si[s].recursive)
+			arena_frame = max(arena_frame, si[s].arena_stack_depth);
+	}
+	if (!arena_frame)
+		return 0;
+	if (depth + 2 * arena_frame >= BPF_ARENA_STACK_MAX)
+		goto no_room;
+	room = BPF_ARENA_STACK_MAX - depth - 2 * arena_frame;
+	env->max_arena_stack_depth = BPF_ARENA_STACK_MAX;
+
+	for_each_set_bit(s, cg->live, cg->cnt) {
+		if (!si[s].recursive)
+			continue;
+		frame = DIV_ROUND_UP(si[s].arena_stack_depth * (THREAD_SIZE / 2), room);
+		if (frame > env->stack_limit)
+			goto no_room;
+		si[s].stack_depth = max(si[s].stack_depth, frame);
+	}
+	return 1;
+no_room:
+	verbose(env, "no room in the stack in arena for recursion\n");
+	return -EACCES;
+}
+
 static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
 {
 	struct bpf_subprog_info *si = env->subprog_info;
@@ -6122,6 +6163,11 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
 	if (!dinfo)
 		return -ENOMEM;
 	compute_call_depth(env, cg, dinfo);
+	err = pad_recursive_frames(env, cg, dinfo);
+	if (err < 0)
+		goto out;
+	if (err)
+		compute_call_depth(env, cg, dinfo);
 
 	for_each_set_bit(s, cg->live, cg->cnt) {
 		/* A caller of it is over a limit. That one is reported, not what it calls. */
@@ -24696,6 +24742,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret == 0)
 		ret = bpf_optimize_bpf_loop(env);
 
+	if (ret == 0)
+		ret = bpf_insert_recursion_checks(env);
+
 	if (ret == 0)
 		bpf_lower_cast_mov(env);
 
-- 
2.55.0

[PATCH bpf-next v4 03/46] bpf: Add union and widening of the range of a register

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

format_cgrp_path() of scx_layered, prog layered_runnable:

  u32 len = 0;
  bpf_for(level, 1, max_level + 1) {
          if (level > 1 && len < MAX_PATH - 1)
                  path[len++] = '/';
          ...
          len += ret - 1;
  }

The verifier follows each path, and its work grows with the number of
paths. To make it grow with the size of the prog, the states that meet at
an insn must become one state. Then the loop head has one state for all
iterations. len in it is the union of what arrives: 0 from above, more
after each pass through the loop, and the passes do not end. Widening ends
them: the bound that moved is put far out at once, and the jumps in the
loop make the range small again. The compiler tests len in 32 bits:

  len at the head  after 'if w7 > 0xffe'    path[len++] = '/'
  [0, S64_MAX]     [0, 0x7fffffff00000ffe]  R2 unbounded memory access
  [0, U32_MAX]     [0, 4094]                ok

A 32-bit compare cannot bring back what widening to 64 bits threw away.
Thus bpf_reg_widen_range() moves a bound to the nearest of a few stops: 0,
S32_MAX, U32_MAX, S64_MAX above it, 0, S32_MIN, S64_MIN below. A stop is
a guess, and a guess that fails costs one more pass. No jump holds sum in
'sum += a[i]; if (sum == 77) ..'. After S32_MAX it can still have 32 bits:
U32_MAX holds it. A bound that was at S32_MAX, U32_MAX or S32_MIN and
leaves 32 bits gets no other stop: [0, S64_MAX] wraps in the next pass. The
number is unknown: btree_set_ops of rust-bpf, 251722 -> 230969 insns.

rusty_dispatch of rust-bpf: 'r9 += 1; if r9 < r8 goto' with r8 from memory.
2, [2, S32_MAX], [2, U32_MAX], unknown: 4 passes. Scalar evolution knows
that each iteration adds a constant to r9: @linear. Such a number goes on
until it wraps. It is unknown at once, or [0, U32_MAX] if it is computed in
32 bits: 2 passes, 788 -> 536 insns.

bpf_reg_union_range() is the union, without the reg_bounds_sync() of
bpf_reg_union(): an input is not always in sync. It keeps the step of
a number. clang counts 'for (i = n; i >= 0; i--) sum += a[i];' in bytes:
'r2 += -4; if r2 != -4 goto'. n is a range, so '!=' gives no count. r2
stops at -4 only because it is a multiple of 4. a + s * k and b + t * k are
both of the form a + gcd(s, t, a - b) * k. A step gives way to one of its
divisors only: it needs no widening. Nothing calls this code yet.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  11 ++++
 kernel/bpf/verifier.c        | 108 +++++++++++++++++++++++++++++++++++
 2 files changed, 119 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 2f935061b8fd..79faf084cbcd 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1341,6 +1341,7 @@ struct bpf_verifier_env {
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
 	struct scev *scev;
+	struct bpf_join_state *join;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1531,6 +1532,13 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+
+/* join.c */
+int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
+		    const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
+		    bool widen);
+void bpf_join_free(struct bpf_verifier_env *env);
+
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
@@ -2059,6 +2067,9 @@ void bpf_trim_pkt_range(struct bpf_reg_state *reg, s64 smax);
 int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 		      struct cnum64 r64, struct cnum32 r32, u16 base, u16 step);
 bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
+void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src);
+void bpf_reg_forget_range(struct bpf_reg_state *reg);
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear);
 int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
 		  const struct bpf_reg_state *src);
 
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c16a20ab664d..801e4607a462 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -18189,6 +18189,114 @@ bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_re
 	       a->id == b->id && a->parent_id == b->parent_id && a->map_uid == b->map_uid;
 }
 
+/* Nothing is known about the number or the variable part of the offset anymore. */
+void bpf_reg_forget_range(struct bpf_reg_state *reg)
+{
+	reg->var_off = tnum_unknown;
+	__mark_reg_unbounded(reg);
+	reg_step_reset(reg);
+}
+
+/*
+ * @reg is what @old became after a trip around a cycle, @old included. The end
+ * of the range that moved is going to move again. Put it where it may come
+ * to rest. There are few such places, so this is done a few times at most.
+ * It's put at the nearest of these places: every value between the range and the place is
+ * made up, and the program is checked with it. Wherever it's put, the range only gets larger.
+ * @linear: every iteration adds a constant to it, in that many bits. Then nothing but a jump
+ * stops it. It goes on until it wraps, and is any number of that many bits.
+ */
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear)
+{
+	bool moved = reg->r64.base != old->r64.base || reg->r64.size != old->r64.size ||
+		     reg->r32.base != old->r32.base || reg->r32.size != old->r32.size;
+	s64 min = reg_smin(reg), max = reg_smax(reg);
+
+	/* Only the step got weaker. It comes to rest without help, see below. */
+	if (!moved && reg->var_off.value == old->var_off.value &&
+	    reg->var_off.mask == old->var_off.mask)
+		return;
+
+	if (linear && (max > reg_smax(old) || min < reg_smin(old))) {
+		if (linear == 64 || min < 0 || max > U32_MAX) {
+			bpf_reg_forget_range(reg);
+			return;
+		}
+		min = 0;
+		max = U32_MAX;
+	}
+	if (max > reg_smax(old))
+		max = max <= 0 ? 0 :
+		      max <= S32_MAX ? S32_MAX :
+		      max <= U32_MAX ? U32_MAX : S64_MAX;
+	if (min < reg_smin(old))
+		min = min >= 0 ? 0 :
+		      min >= S32_MIN ? S32_MIN : S64_MIN;
+	if (min == S64_MIN && max == S64_MAX) {
+		bpf_reg_forget_range(reg);
+		return;
+	}
+	/*
+	 * A fixed stop is a guess. A number of 32 bits that gets past S32_MAX comes to rest at
+	 * U32_MAX. What gets past S32_MAX, U32_MAX or S32_MIN and out of 32 bits is stopped by
+	 * nothing:
+	 *	sum += a[i]; if (sum == 77) ...
+	 * It wraps in the next pass, and the other bound is gone as well.
+	 */
+	if ((max == S64_MAX && (reg_smax(old) == S32_MAX || reg_smax(old) == U32_MAX)) ||
+	    (min == S64_MIN && reg_smin(old) == S32_MIN)) {
+		bpf_reg_forget_range(reg);
+		return;
+	}
+	/* every part has to include what it did before, hence no reg_bounds_sync() */
+	reg->r64 = cnum64_union(reg->r64, cnum64_from_srange(min, max));
+	if (min >= 0 && max <= U32_MAX) {
+		reg->r32 = cnum32_union(reg->r32, cnum32_from_urange(min, max));
+		reg->var_off = tnum_union(reg->var_off, tnum_range(min, max));
+	} else if (min >= S32_MIN && max <= S32_MAX) {
+		reg->r32 = cnum32_union(reg->r32, cnum32_from_srange(min, max));
+		reg->var_off = tnum_unknown;
+	} else {
+		reg->r32 = CNUM32_UNBOUNDED;
+		reg->var_off = min >= 0 ? tnum_union(reg->var_off, tnum_range(min, max))
+					: tnum_unknown;
+	}
+	/*
+	 * No range moved: the known bits or the step got weaker. The step comes to rest without
+	 * help: it gives way to one of its divisors only.
+	 *	off = n * 4; do { sum += *(u32 *)(a + off); off -= 4; } while (off != -4);
+	 * counts on "a multiple of 4".
+	 */
+	if (moved)
+		reg_step_reset(reg);
+}
+
+void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src)
+{
+	bool acc_const = cnum64_is_const(acc->r64), src_const = cnum64_is_const(src->r64);
+	/* A constant is its own base. Its step is 0: every number divides 0. */
+	s64 a = acc_const ? cnum64_smin(acc->r64) : acc->base;
+	s64 b = src_const ? cnum64_smin(src->r64) : src->base;
+	u16 step = gcd(acc_const ? 0 : acc->step, src_const ? 0 : src->step);
+	s64 diff;
+
+	acc->r64 = cnum64_union(acc->r64, src->r64);
+	acc->r32 = cnum32_union(acc->r32, src->r32);
+	acc->var_off = tnum_union(acc->var_off, src->var_off);
+
+	/* a + s * k and b + t * k are both of the form a + gcd(s, t, a - b) * k */
+	if (check_sub_overflow(a, b, &diff) || (!step && (diff > U16_MAX || diff < -U16_MAX)))
+		step = 1;
+	else
+		step = (step ? gcd(step, imod(diff, step)) : abs(diff)) ?: 1;
+	acc->step = step;
+	acc->base = imod(a, step);
+	/*
+	 * No reg_bounds_sync(): every part of the result has to include that
+	 * part of both inputs, and an input is not always in sync.
+	 */
+}
+
 /*
  * Extends acc's scalar range to include src's scalar range.
  * Caller must ensure that other kinds of metadata is compatible.
-- 
2.55.0

Re: [PATCH bpf-next v4 03/46] bpf: Add union and widening of the range of a register

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

> bpf: Add union and widening of the range of a register

TL;DR: Not a bug, but bpf_reg_union_range() duplicates the union lines of
bpf_reg_union() and computes the step differently; the review asks whether
bpf_reg_union() could reuse it so the step logic lives in one place.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src)
> +{
> +	bool acc_const = cnum64_is_const(acc->r64), src_const = cnum64_is_const(src->r64);
> +	/* A constant is its own base. Its step is 0: every number divides 0. */
> +	s64 a = acc_const ? cnum64_smin(acc->r64) : acc->base;
> +	s64 b = src_const ? cnum64_smin(src->r64) : src->base;
> +	u16 step = gcd(acc_const ? 0 : acc->step, src_const ? 0 : src->step);
> +	s64 diff;
> +
> +	acc->r64 = cnum64_union(acc->r64, src->r64);
> +	acc->r32 = cnum32_union(acc->r32, src->r32);
> +	acc->var_off = tnum_union(acc->var_off, src->var_off);

this isn't a bug, but these three union lines repeat the ones in
bpf_reg_union() word for word, and the step is then computed with a
different rule (gcd(s, t, a - b) here, versus gcd(s, t) with a fallback to
step 1 when the bases disagree in bpf_reg_union()).

Could bpf_reg_union() be written as the compatibility check plus
bpf_reg_union_range() plus reg_bounds_sync() and the sanity check, so the
step-union logic lives in one place?

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

[PATCH bpf-next v4 04/46] bpf: Add join of registers and stack slots

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

ipv6_hdrlen_offset() of Cilium skips extension headers, unrolled:

  for (i = 0; i < IPV6_MAX_HEADERS; i++) {
          int hdrlen = ipv6_skip_exthdr(ctx, &newnh, l3_off + len);
          ...  8, (opthdr->hdrlen + 2) << 2 or (opthdr->hdrlen + 1) << 3
          len += hdrlen;
  }

The verifier follows each path. Three lengths for each header: the insn
behind this loop is verified 76 times, and each time the rest of the prog
follows, where more paths fork. tail_nodeport_nat_ingress_ipv6 of bpf_xdp,
as of May 2026, has 4095 insns, the verifier processes 182050.

Add the join of two states: one state that covers the two. Here hdrlen is
[8, 2048]. A caller that replaces two states with their join verifies the
code that follows one time. This prog: 3 times in place of 76, 7932 insns.
This is the first half of join.c: ids, registers, stack slots.

Only numbers are joined: scalars, and the variable offsets of pointers. A
join loses information. Where the prog needs it there is no join, and the
caller keeps the two states:
 - map value + 8 and map value + 16: two fields, not a range
   scx_rusty: create_save_cpumask(&kptr) for three kptrs in .data
   "R1 doesn't have constant offset. kptr has to be at the constant offset"
 - packet pointers that are checked up to different ends
   Cilium: l4_csum_replace(ctx, l4_off + csum->offset, ..)
   "invalid access to packet": 206 progs of bpf_xdp in 37 builds of Cilium
 - a scalar and a pointer, pointers to two objects: of no use to a prog

Each path makes its own ids. An id in the join stands for a pair: the id
in state a and the id in state b. The pairs must be one to one, as in
check_ids(). If not, a scalar loses its id, and a pointer has no join.
 - 'bpf_for(i, 0, 10) sum += i;' of iters_num: each bpf_iter_num_next()
   returns a pointer with a new id. If ids had to be equal, the main pass
   says "more than 1024 states that cannot be joined at insn 16".
 - scx_pandemonium: 'if (.. && (u32)src_cpu < MAX_CPUS) pair_warm_ns[..]'
   tests r5 and indexes with its copy in fp-88. Paths meet between copy
   and test. A join without ids: "R3 unbounded memory access".

With 'widen', at a loop head, a scalar that grew is widened. If each
iteration adds a constant to it, it becomes an unknown scalar of 64 or 32
bits at once: scev_compute_linear_regs() sets linear_regs, linear32_regs of
struct bpf_loop for that. So does sum of bpf_for() above: it is a register
plus or minus something (expr_is_sum()), and no jump of the loop depends on
it (tested_regs). 0, [0, S32_MAX], [0, U32_MAX], unknown are 4 passes
through the body, this is 2: 59 -> 43 insns. If a jump depends on the sum
the stops stay. A pointer whose offset grew has no join: no later
comparison gives it a bound.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  11 +
 kernel/bpf/join.c            | 400 +++++++++++++++++++++++++++++++++++
 kernel/bpf/scev.c            |  75 +++++++
 3 files changed, 486 insertions(+)
 create mode 100644 kernel/bpf/join.c

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 79faf084cbcd..749ebb1ad489 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -760,6 +760,17 @@ struct bpf_loop {
 	 * loops.
 	 */
 	unsigned long store_base_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/*
+	 * Loop-entry registers used to compute the operands of conditional jumps, in this and
+	 * nested loops.
+	 */
+	unsigned long tested_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/*
+	 * what gets a constant added in every iteration, in 64 and in 32 bits. A sum that is
+	 * not among tested_regs is taken for that too: nothing stops it either
+	 */
+	unsigned long linear_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	unsigned long linear32_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 };
 
 struct bpf_insn_aux_data {
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
new file mode 100644
index 000000000000..fe513b7a8e88
--- /dev/null
+++ b/kernel/bpf/join.c
@@ -0,0 +1,400 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Join of verifier states.
+ *
+ * bpf_states_join(dst, a, b) computes the state that describes every concrete
+ * state described by 'a' or by 'b'. With @widen a number that is larger in the
+ * result than in 'a' gets the bound that moved put further out, so that a sequence
+ * of states at a loop head ends. A pointer whose offset is larger has no join.
+ *
+ * Numbers and offsets of pointers are joined. A pointer and a number, pointers
+ * to different objects, different references and locks are not: there is no
+ * join and the states stay apart.
+ *
+ * The contract is checked on every call: both inputs have to be within the
+ * result, see bpf_state_subsumes(). A violation is a verifier bug.
+ *
+ * Inputs have to be at the same insn and cleaned with
+ * bpf_clean_verifier_state(): dead registers and stack slots are the same on
+ * both sides then, but for dynptrs, iterators and irq flags: they stay.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/cnum.h>
+#include <linux/tnum.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+
+struct join_idpair {
+	u32 a;
+	u32 b;
+	bool bad;
+};
+
+struct bpf_join_state {
+	struct bpf_verifier_env *env;
+	struct join_idpair pairs[BPF_ID_MAP_SIZE];
+	u32 npairs;
+	bool pairs_overflow;
+	bool widen;
+	/* the states came around this loop in the frame that is joined, NULL if not */
+	const struct bpf_loop *loop;
+	/* the register or stack slot that is joined, as scalar evolution numbers it, or -1 */
+	int place;
+};
+
+static struct bpf_join_state *join_state(struct bpf_verifier_env *env)
+{
+	if (!env->join) {
+		env->join = kvzalloc_obj(*env->join, GFP_KERNEL_ACCOUNT);
+		if (env->join)
+			env->join->env = env;
+	}
+	return env->join;
+}
+
+void bpf_join_free(struct bpf_verifier_env *env)
+{
+	kvfree(env->join);
+	env->join = NULL;
+}
+
+/*
+ * An id in the result stands for the pair (id in 'a', id in 'b'). For the
+ * result to be within both inputs the pairs have to form a bijection,
+ * see check_ids(). The first pass over all locations records the pairs. An id
+ * seen with two partners makes all pairs it takes part in bad.
+ * The second pass keeps the id of 'a' for good pairs.
+ */
+static void jid_mark_bad(struct bpf_join_state *js, u32 a, u32 b)
+{
+	u32 i;
+
+	for (i = 0; i < js->npairs; i++)
+		if ((a && js->pairs[i].a == a) || (b && js->pairs[i].b == b))
+			js->pairs[i].bad = true;
+}
+
+static void jid_note(struct bpf_join_state *js, u32 ida, u32 idb, bool consistent)
+{
+	u32 a = ida & ~BPF_ADD_CONST, b = idb & ~BPF_ADD_CONST;
+	struct join_idpair *p;
+	bool other = false;
+	u32 i;
+
+	if (!a || !b)
+		return;
+	for (i = 0; i < js->npairs; i++) {
+		p = &js->pairs[i];
+		if (p->a == a && p->b == b) {
+			if (!consistent)
+				jid_mark_bad(js, a, b);
+			return;
+		}
+		if (p->a == a || p->b == b)
+			other = true;
+	}
+	if (other) {
+		jid_mark_bad(js, a, b);
+		consistent = false;
+	}
+	if (js->npairs == ARRAY_SIZE(js->pairs)) {
+		js->pairs_overflow = true;
+		return;
+	}
+	p = &js->pairs[js->npairs++];
+	p->a = a;
+	p->b = b;
+	p->bad = !consistent;
+}
+
+static bool jid_good(struct bpf_join_state *js, u32 ida, u32 idb)
+{
+	u32 a = ida & ~BPF_ADD_CONST, b = idb & ~BPF_ADD_CONST;
+	u32 i;
+
+	if (!a || !b || js->pairs_overflow)
+		return false;
+	if ((ida ^ idb) & BPF_ADD_CONST)
+		return false;
+	for (i = 0; i < js->npairs; i++)
+		if (js->pairs[i].a == a && js->pairs[i].b == b)
+			return !js->pairs[i].bad;
+	return false;
+}
+
+/* Ids that have to match: both zero, or a good pair */
+static bool jid_strict_ok(struct bpf_join_state *js, u32 ida, u32 idb)
+{
+	if (!ida && !idb)
+		return true;
+	return jid_good(js, ida, idb);
+}
+
+static bool range_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	return ra->r64.base == rb->r64.base && ra->r64.size == rb->r64.size &&
+	       ra->r32.base == rb->r32.base && ra->r32.size == rb->r32.size &&
+	       ra->var_off.value == rb->var_off.value && ra->var_off.mask == rb->var_off.mask &&
+	       ra->base == rb->base && ra->step == rb->step;
+}
+
+/* Does every iteration add a constant to what is in the place? In 64 or in 32 bits, 0 if not */
+static int join_place_is_linear(const struct bpf_join_state *js)
+{
+	if (!js->loop || js->place < 0 || js->place >= BPF_SCEV_REGS_NUM)
+		return 0;
+	return test_bit(js->place, js->loop->linear_regs) ? 64 :
+	       test_bit(js->place, js->loop->linear32_regs) ? 32 : 0;
+}
+
+/*
+ * Join of the range of two registers of the same type, numbers or pointers
+ * with variable offset. @dst is a copy of @ra. Returns false if there is no join.
+ */
+static bool join_range(struct bpf_join_state *js, struct bpf_reg_state *dst,
+		       const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	if (range_equal(ra, rb))
+		return true;
+	bpf_reg_union_range(dst, rb);
+	if (!js->widen || range_equal(dst, ra))
+		return true;
+	/*
+	 * The bound of a number that grows around a cycle is put where it may come to rest. If
+	 * that is far out the number gets a bound again from the next comparison. The offset of
+	 * a pointer doesn't: there is no access through a pointer at an offset that is not
+	 * known. So there is nothing to gain from that join. If the offset stops to grow
+	 * the next state is within the last one.
+	 */
+	if (dst->type != SCALAR_VALUE)
+		return false;
+	bpf_reg_widen_range(dst, ra, join_place_is_linear(js));
+	return true;
+}
+
+static bool reg_fixed_part_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	return !memcmp(ra, rb, offsetof(struct bpf_reg_state, var_off));
+}
+
+static bool reg_value_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	return !memcmp(ra, rb, offsetof(struct bpf_reg_state, id));
+}
+
+static bool scalar_link_consistent(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+	if ((ra->id ^ rb->id) & BPF_ADD_CONST)
+		return false;
+	if ((ra->id & BPF_ADD_CONST) && ra->delta != rb->delta)
+		return false;
+	return true;
+}
+
+/*
+ * Join two registers, regsafe() documents what each type needs.
+ * Pass 1 records id pairs, pass 2 computes @dst, which is a copy of @ra on entry.
+ * Returns false if there is no join.
+ */
+static bool join_reg(struct bpf_join_state *js, struct bpf_reg_state *dst,
+		     const struct bpf_reg_state *ra, const struct bpf_reg_state *rb, int pass)
+{
+	if (ra->type == NOT_INIT || rb->type == NOT_INIT) {
+		if (pass == 2 && ra->type != NOT_INIT)
+			*dst = *rb;
+		return true;
+	}
+
+	if (ra->type != rb->type)
+		return false;
+
+	switch (base_type(ra->type)) {
+	case SCALAR_VALUE:
+		if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
+			return false;
+		if (pass == 1) {
+			jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
+			return true;
+		}
+		join_range(js, dst, ra, rb);
+		/* a link that exists on one side only is dropped */
+		if (ra->id && !jid_good(js, ra->id, rb->id)) {
+			dst->id = 0;
+			dst->delta = 0;
+		}
+		return true;
+	case PTR_TO_MAP_KEY:
+	case PTR_TO_MAP_VALUE:
+	case PTR_TO_MEM:
+	case PTR_TO_BUF:
+	case PTR_TO_TP_BUFFER:
+		if (!reg_fixed_part_equal(ra, rb))
+			return false;
+		/*
+		 * Two places in an object are two pointers: what is at a place
+		 * and how much is left behind it is lost in a range of them.
+		 */
+		if (!js->widen && tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+		    ra->var_off.value != rb->var_off.value)
+			return false;
+		break;
+	case PTR_TO_PACKET_META:
+	case PTR_TO_PACKET:
+		if ((ra->range < 0 || rb->range < 0) && ra->range != rb->range)
+			return false;
+		/* how much of the packet was checked goes with numbers elsewhere in the state */
+		if (!js->widen && ra->range != rb->range)
+			return false;
+		/*
+		 * Displacements between pointers that share an id are kept. No id is an id too:
+		 * the start of the packet plus a constant, see find_good_pkt_pointers().
+		 */
+		if (ra->r64.base != rb->r64.base)
+			return false;
+		if (pass == 1) {
+			jid_note(js, ra->id, rb->id, true);
+			return true;
+		}
+		if (!jid_strict_ok(js, ra->id, rb->id))
+			return false;
+		if (rb->range != ra->range)
+			dst->range = min(ra->range, rb->range);
+		return join_range(js, dst, ra, rb);
+	case PTR_TO_ARENA:
+		/* The offset is not tracked. */
+		return true;
+	case PTR_TO_INSN:
+		if (!reg_fixed_part_equal(ra, rb))
+			return false;
+		return pass == 1 || join_range(js, dst, ra, rb);
+	default:
+		if (!reg_value_equal(ra, rb))
+			return false;
+		break;
+	}
+
+	if (pass == 1) {
+		jid_note(js, ra->id, rb->id, true);
+		jid_note(js, ra->parent_id, rb->parent_id, true);
+		jid_note(js, ra->map_uid, rb->map_uid, true);
+		return true;
+	}
+	if (!jid_strict_ok(js, ra->id, rb->id) ||
+	    !jid_strict_ok(js, ra->parent_id, rb->parent_id) ||
+	    !jid_strict_ok(js, ra->map_uid, rb->map_uid))
+		return false;
+	return join_range(js, dst, ra, rb);
+}
+
+static bool slot_is_special(const struct bpf_stack_state *s)
+{
+	u8 t = s->slot_type[BPF_REG_SIZE - 1];
+
+	return t == STACK_DYNPTR || t == STACK_ITER || t == STACK_IRQ_FLAG;
+}
+
+static bool stype_is_invalid(u8 t)
+{
+	return t == STACK_INVALID || t == STACK_POISON;
+}
+
+static bool slot_has_spill(const struct bpf_stack_state *s)
+{
+	int i;
+
+	for (i = 0; i < BPF_REG_SIZE; i++)
+		if (s->slot_type[i] == STACK_SPILL)
+			return true;
+	return false;
+}
+
+static bool join_special_slot(struct bpf_join_state *js, const struct bpf_stack_state *sa,
+			      const struct bpf_stack_state *sb, int pass)
+{
+	const struct bpf_reg_state *ra = &sa->spilled_ptr, *rb = &sb->spilled_ptr;
+
+	if (memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type)))
+		return false;
+	switch (sa->slot_type[BPF_REG_SIZE - 1]) {
+	case STACK_DYNPTR:
+		if (ra->dynptr.type != rb->dynptr.type ||
+		    ra->dynptr.first_slot != rb->dynptr.first_slot)
+			return false;
+		break;
+	case STACK_ITER:
+		if (ra->type != rb->type || ra->iter.btf != rb->iter.btf ||
+		    ra->iter.btf_id != rb->iter.btf_id || ra->iter.state != rb->iter.state ||
+		    ra->iter.loops != rb->iter.loops)
+			return false;
+		break;
+	case STACK_IRQ_FLAG:
+		if (ra->irq.kfunc_class != rb->irq.kfunc_class)
+			return false;
+		break;
+	}
+	if (pass == 1) {
+		jid_note(js, ra->id, rb->id, true);
+		jid_note(js, ra->parent_id, rb->parent_id, true);
+		return true;
+	}
+	return jid_strict_ok(js, ra->id, rb->id) &&
+	       jid_strict_ok(js, ra->parent_id, rb->parent_id);
+}
+
+/*
+ * Join of one 8 byte stack slot, @ds is a copy of @sa on entry.
+ * stacksafe() documents which slot is within which.
+ */
+static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
+		      const struct bpf_stack_state *sa, const struct bpf_stack_state *sb, int pass)
+{
+	static const struct bpf_stack_state invalid_slot;
+	bool a_ptr, b_ptr;
+	u8 res[BPF_REG_SIZE];
+	int i;
+
+	if (!sb)
+		sb = &invalid_slot;
+
+	if (slot_is_special(sa) || slot_is_special(sb))
+		return join_special_slot(js, sa, sb, pass);
+
+	if (!memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type)) &&
+	    sa->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL)
+		return join_reg(js, &ds->spilled_ptr, &sa->spilled_ptr, &sb->spilled_ptr, pass);
+
+	a_ptr = slot_has_spill(sa) && sa->spilled_ptr.type != SCALAR_VALUE;
+	b_ptr = slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE;
+	/* a pointer and something else */
+	if (a_ptr || b_ptr)
+		return false;
+
+	/* the prog can't read uninitialized stack: slots of different kinds have no join */
+	if (!js->env->allow_uninit_stack)
+		return !memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type));
+
+	/* scalar spills, MISC, ZERO, INVALID, POISON: no ids are kept */
+	if (pass == 1)
+		return true;
+
+	for (i = 0; i < BPF_REG_SIZE; i++) {
+		u8 x = sa->slot_type[i], y = sb->slot_type[i];
+
+		if (x == STACK_SPILL)
+			x = STACK_MISC;
+		if (y == STACK_SPILL)
+			y = STACK_MISC;
+		if (stype_is_invalid(x) && stype_is_invalid(y))
+			res[i] = sa->slot_type[i];
+		else if (stype_is_invalid(x) || stype_is_invalid(y))
+			res[i] = STACK_INVALID;
+		else if (x == y)
+			res[i] = x;
+		else
+			res[i] = STACK_MISC;
+	}
+	memcpy(ds->slot_type, res, sizeof(res));
+	bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
+	return true;
+}
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index bb1c67176d78..9629a6bf80b1 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1220,6 +1220,22 @@ static void collect_store_base_regs(struct bpf_verifier_env *env,
 			or_expr_regs(env, cur_env->reg2expr[r], mask);
 }
 
+/*
+ * Record the loop-entry registers that the operands of the conditional jump at 'idx' are
+ * computed from, best effort like or_expr_regs().
+ */
+static void collect_tested_regs(struct bpf_verifier_env *env, struct env *cur_env, int idx,
+				unsigned long *mask)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+
+	if (!bpf_insn_is_cond_jump(insn->code) || BPF_OP(insn->code) == BPF_JCOND)
+		return;
+	or_expr_regs(env, cur_env->reg2expr[insn->dst_reg], mask);
+	if (BPF_SRC(insn->code) == BPF_X)
+		or_expr_regs(env, cur_env->reg2expr[insn->src_reg], mask);
+}
+
 /* Find the topmost loop header containing idx inside cur_header, or -1 if none. */
 static int topmost_nested_loop(struct bpf_verifier_env *env, int idx, int cur_header)
 {
@@ -1325,6 +1341,7 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 			return -ENOMEM;
 		/* The freshly allocated environment has all expressions unknown. */
 		bitmap_fill(cur_loop->store_base_regs, REGS_NUM);
+		bitmap_fill(cur_loop->tested_regs, REGS_NUM);
 		header_env->empty = false;
 		return 0;
 	}
@@ -1364,6 +1381,9 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 			/* Pull the nested loop's stack-store base dependencies up. */
 			for_each_set_bit(r, nested_loop->store_base_regs, BPF_SCEV_REGS_NUM)
 				or_expr_regs(env, cur_env->reg2expr[r], cur_loop->store_base_regs);
+			/* And what its jumps depend on. */
+			for_each_set_bit(r, nested_loop->tested_regs, BPF_SCEV_REGS_NUM)
+				or_expr_regs(env, cur_env->reg2expr[r], cur_loop->tested_regs);
 			forget_non_invariants(env, cur_env, idx);
 			if (log_level2)
 				log_env_changes(env, LOG_AT_TRANSFER, old_env, cur_env, idx);
@@ -1386,6 +1406,7 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 				if (log_level2)
 					memcpy(old_env, cur_env, sizeof(*old_env));
 				collect_store_base_regs(env, cur_env, idx, cur_loop->store_base_regs);
+				collect_tested_regs(env, cur_env, idx, cur_loop->tested_regs);
 				err = transfer(env, cur_env, idx);
 				if (err)
 					goto out;
@@ -1676,6 +1697,57 @@ static int compute_header_scevs(struct bpf_verifier_env *env, struct env *header
 	}
 	return 0;
 }
+
+/*
+ * 'sum += a[i]', 'if (..) { len -= n; len -= 2; }': is 'id', what an iteration makes of
+ * register 'r', the register plus or minus something, or the register as it was?
+ * 'low32': it's computed in 32 bits.
+ */
+static bool expr_is_sum(struct scev *scev, u32 id, u32 r, bool low32, int depth)
+{
+	u32 a, b;
+
+	if (is_reg(scev, id, &a))
+		return a == r;
+	if (!depth--)
+		return false;
+	if (low32 && is_zext32(scev, id, &a))
+		return expr_is_sum(scev, a, r, low32, depth);
+	if (is_any(scev, id, &a, &b))
+		return expr_is_sum(scev, a, r, low32, depth) &&
+		       expr_is_sum(scev, b, r, low32, depth);
+	if (is_add(scev, id, &a, &b) && expr_is_sum(scev, b, r, low32, depth))
+		return true;
+	return (is_add(scev, id, &a, &b) || is_binop(scev, BPF_SUB, id, &a, &b)) &&
+	       expr_is_sum(scev, a, r, low32, depth);
+}
+
+/*
+ * Find the registers and stack slots that nothing but a jump stops: what gets a constant added in
+ * every iteration of the loop, and a sum that no jump of the loop depends on.
+ */
+static void scev_compute_linear_regs(struct scev *scev, struct bpf_loop *loop,
+				     struct env *header_env)
+{
+	u32 r, same, base, slope, id;
+	bool low32;
+	int bits;
+
+	for (r = 0; r < REGS_NUM; r++) {
+		id = header_env->reg2expr[r];
+		bits = 0;
+		if (is_linear_or_linear32(scev, header_env->reg2scev[r], &base, &slope, &low32) &&
+		    is_reg(scev, base, &same) && same == r)
+			bits = low32 ? 32 : 64;
+		/* what the loop leaves alone is no sum */
+		else if (!is_reg(scev, id, &same) && !test_bit(r, loop->tested_regs))
+			bits = expr_is_sum(scev, id, r, false, EXPR_STACK_DEPTH) ? 64 :
+			       expr_is_sum(scev, id, r, true, EXPR_STACK_DEPTH) ? 32 : 0;
+		if (bits)
+			__set_bit(r, bits == 32 ? loop->linear32_regs : loop->linear_regs);
+	}
+}
+
 static int instantiate_header_scevs(struct scev *scev, u32 id, void *priv)
 {
 	u32 reg, *reg2scev = priv;
@@ -1784,6 +1856,9 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
 					   find_loop_env(scev, header, idx));
 		if (err)
 			return err;
+		if (aux[idx].loop)
+			scev_compute_linear_regs(scev, aux[idx].loop,
+						 find_header_env(scev, header));
 	}
 
 	for (i = 0; i < env->prog->len; i++)
-- 
2.55.0

Re: [PATCH bpf-next v4 04/46] bpf: Add join of registers and stack slots

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] bpf: join_reg() zeroes dst->delta under explore_alu_limits,
  failing regsafe()
- [Medium] bpf: loss of STACK_POISON in join_slot() when joining stack
  slots
--

commit b2b35d77649236d79410bf6f82762993660918fc
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Add join of registers and stack slots

This commit introduces state joining for registers and stack slots, along
with SCEV tracking of linear registers for loop-head widening.

> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> new file mode 100644
> index 0000000000000..fe513b7a8e884
> --- /dev/null
> +++ b/kernel/bpf/join.c

[ ... ]

> +static bool join_reg(struct bpf_join_state *js, struct bpf_reg_state *dst,
> +		     const struct bpf_reg_state *ra, const struct bpf_reg_state *rb, int pass)
> +{
> +	if (ra->type == NOT_INIT || rb->type == NOT_INIT) {
> +		if (pass == 2 && ra->type != NOT_INIT)
> +			*dst = *rb;
> +		return true;
> +	}
> +
> +	if (ra->type != rb->type)
> +		return false;
> +
> +	switch (base_type(ra->type)) {
> +	case SCALAR_VALUE:
> +		if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
> +			return false;
> +		if (pass == 1) {
> +			jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
> +			return true;
> +		}
> +		join_range(js, dst, ra, rb);
> +		/* a link that exists on one side only is dropped */
> +		if (ra->id && !jid_good(js, ra->id, rb->id)) {
> +			dst->id = 0;
> +			dst->delta = 0;
> +		}
> +		return true;

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], when js->env->explore_alu_limits
is set, can zeroing dst->delta here make the joined register fail regsafe()
against ra and rb?

When ra and rb have matching non-zero deltas (BPF_ADD_CONST) and identical
ranges, reg_value_equal(ra, rb) succeeds, but if !jid_good(js, ra->id, rb->id)
drops the link in pass 2 (for instance, when one path sets BPF_ADD_CONST64
and the other sets BPF_ADD_CONST32, or on an ID pairing conflict), dst->delta
is cleared to 0.  Because s32 delta sits before id in struct bpf_reg_state,
regsafe() under explore_alu_limits compares delta via memcmp(...,
offsetof(struct bpf_reg_state, id)), causing bpf_state_subsumes() to fail,
triggering verifier_bug() and rejecting the program with -EFAULT.

[ ... ]

> +static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
> +		      const struct bpf_stack_state *sa, const struct bpf_stack_state *sb, int pass)
> +{

[ ... ]

> +	for (i = 0; i < BPF_REG_SIZE; i++) {
> +		u8 x = sa->slot_type[i], y = sb->slot_type[i];
> +
> +		if (x == STACK_SPILL)
> +			x = STACK_MISC;
> +		if (y == STACK_SPILL)
> +			y = STACK_MISC;
> +		if (stype_is_invalid(x) && stype_is_invalid(y))
> +			res[i] = sa->slot_type[i];
> +		else if (stype_is_invalid(x) || stype_is_invalid(y))
> +			res[i] = STACK_INVALID;

[Severity: Medium]
Does this overwrite STACK_POISON with STACK_INVALID when joining a poisoned
stack byte with STACK_INVALID, STACK_MISC, STACK_ZERO, or a scalar
STACK_SPILL?

For example, when only the low 4-byte half of a slot is live,
__clean_func_state() leaves the dead upper half of a 64-bit constant-zero
scalar spill as STACK_SPILL in one state while marking it STACK_POISON in
another.  Degrading STACK_SPILL to STACK_MISC and joining it with
STACK_POISON produces STACK_INVALID, which allows subsequent reads of dead
poisoned stack slots to succeed under allow_uninit_stack instead of being
rejected.  Checking whether x == STACK_POISON or y == STACK_POISON first and
setting res[i] = STACK_POISON would preserve poison tracking while remaining
compatible with stacksafe().

[1] https://lore.kernel.org/r/ee77c73bff0b56a40098080a5335d5fd5e4d8592ac15afe769c1c4661f026576@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=4

[PATCH bpf-next v4 05/46] bpf: Add join of verifier states

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

central_dispatch() of tools/sched_ext/scx_central.bpf.c:

  bpf_for(cpu, 0, nr_cpu_ids) {
          ...
          if (dispatch_to_cpu(cpu))
                  *gimme = false;
  }
  ...
  dispatch_to_cpu(central_cpu);

To verify dispatch_to_cpu() one time, not for each call, the states of the
two calls must meet at its first insn. Numbers, like cpu, can be joined.
But at one call the iterator of bpf_for() is held, at the other nothing.
One state cannot stand for the two: the code after the return destroys the
iterator for one of them only. A state is more than registers and slots.

Add the second half of join.c and build it. bpf_states_join() joins two
states at one insn, frame by frame. There is no join, and the caller keeps
the two states, when one of these differs:

  frames                number, subprog, callsite, kind of callback
  locks, RCU, preempt   the counts, the lock
  references            number and types. The ids must pair one to one
  held since the entry  entry_refs, entry_rcu_locks, entry_preempt_locks
  loop stack            the loops, the state that entered them, the count,
                        the iteration if iterations are kept apart
  flags                 speculative, sleepable, unwound, ...

 - A register, its spill and the reference have one id. Thus the pairs of
   ids are collected over all frames and references first, and the join
   is made in a second pass.
 - Only live registers count, and the caller removes dead stack slots with
   bpf_clean_verifier_state(). What is dead does not prevent a join.
 - A subprog that is verified without its callers has what they hold in
   its state, as above. entry_* tell how much of it is still held.
 - Such a subprog has its incoming stack arguments in the slots for
   outgoing ones, until it has loaded them. The two kinds are not joined.
   quicksort(right, scratch, limit, pivot_ref, is_less) of Rust's core is
   recursive and takes 7 words: bpfland_init of rust-bpf fails without it.

bpf_states_join() examines each result with the comparison that pruning
uses: the join must cover state a and state b. If it does not, that is a
verifier bug and the prog is rejected, not loaded. The fields that join.c
compares come with this patch. Nothing sets them, nothing calls the join.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  65 ++++++++-
 kernel/bpf/Makefile          |   2 +-
 kernel/bpf/join.c            | 247 +++++++++++++++++++++++++++++++++++
 kernel/bpf/states.c          |  12 ++
 4 files changed, 324 insertions(+), 2 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 749ebb1ad489..922f01ee8136 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -120,6 +120,11 @@ struct bpf_reg_state {
 			/* packing following two fields to fit iter state into 16 bytes */
 			enum bpf_iter_state state:2;
 			int depth:30;
+			/*
+			 * The number of loops, in all frames, that the program was in
+			 * when it made the iterator, see iter_stops_loops().
+			 */
+			u32 loops:30;
 		} iter;
 
 		/* For irq stack slots */
@@ -363,12 +368,35 @@ struct loop_stack_entry {
 	struct bpf_verifier_state *entry_state;
 	u32 min_header_count;
 	u32 max_header_count;
-	u32 loop_id:31;
+	u32 loop_id:23;
 	u32 terminates:1;
+	/* the loop had may_goto when the state entered it, see check_loop_guard() */
+	u32 guarded:1;
+	/*
+	 * The iterations are kept apart: states with different @trip are not
+	 * joined for as long as they are in the loop.
+	 */
+	u32 apart:1;
+	/* max_header_count follows from the state at the entry, not from patience */
+	u32 counted:1;
+	/*
+	 * How many times the state came back to the head of the loop. With @apart that is
+	 * the iteration that it is in. A state that stands for many iterations is a join, and has
+	 * the largest number of the states that it is made of.
+	 */
+	u32 trip;
+	/* the number of references that the state had when it entered the loop */
+	u32 refs;
 	int iv_reg;
 	struct cnum64 iv_range;
 };
 
+/* The state is in a walk on patience, which may not get to the end of the loop */
+static inline bool bpf_loop_walk_may_give_up(const struct loop_stack_entry *entry)
+{
+	return entry->apart && entry->terminates && !entry->counted;
+}
+
 #define LOOP_STACK_SIZE 16
 
 /* state of the program:
@@ -426,6 +454,14 @@ struct bpf_func_state {
 	 * subprogram (loops never cross subprogram boundaries).
 	 */
 	u32 loop_stack_cnt;
+	/*
+	 * How much of what was held when the frame was entered is held ever since. For
+	 * references that is the first entry_refs of refs[]. What is released and taken again
+	 * doesn't count. More than that is for this frame to release.
+	 */
+	u32 entry_refs;
+	u32 entry_rcu_locks;
+	u32 entry_preempt_locks;
 
 	/* The following fields should be last. See copy_func_state() */
 	/* The state of the stack. Each element of the array describes BPF_REG_SIZE
@@ -558,8 +594,15 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	/* a global function or an extension, and what it calls: exit of frame 0 is not the end */
+	bool callers_unknown;
 	/* The callee left with unwind_exit. Next insn is unwind_goto. */
 	bool unwound;
+	/*
+	 * A function that is verified on its own changed packet data. Its callers
+	 * and their packet pointers are not in the state, see bpf_flow_return().
+	 */
+	bool changed_pkt_data;
 
 	/*
 	 * First and last insn idx of this verifier state.
@@ -620,6 +663,16 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
 	return NULL;
 }
 
+/*
+ * The function that is verified on its own has no caller to look at. Its incoming arguments
+ * are in stack_arg_regs of frame 0 until it is done with loading them. Until then it has
+ * no outgoing arguments.
+ */
+static inline bool bpf_stack_args_are_incoming(const struct bpf_func_state *frame)
+{
+	return !frame->frameno && !frame->no_stack_arg_load;
+}
+
 /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
 #define bpf_for_each_spilled_reg(iter, frame, reg, mask)			\
 	for (iter = 0, reg = bpf_get_spilled_reg(iter, frame, mask);		\
@@ -754,6 +807,8 @@ struct bpf_loop {
 	bool exits_overflow;
 	/* @exits of a loop in this one is cut. Then @exits of this loop can lack exits too. */
 	bool nested_exits_overflow;
+	/* and it is going to be there, see bpf_flow_loop_gets_guards() */
+	bool guarded;
 	/*
 	 * Loop-entry registers used to compute stack addresses for loads,
 	 * stores and calls requiring fixed stack offsets, in this and nested
@@ -893,6 +948,11 @@ struct bpf_insn_aux_data {
 	 * For a loop header, identifies the parent loop instead.
 	 */
 	s32 loop_header;
+	/*
+	 * bpf_iter_*_next(): the innermost loop around it that counted on it and that it doesn't
+	 * stop, -1 if none was found. It doesn't stop the loops around that one either.
+	 */
+	s32 not_stopped;
 	/* where may_goto in front of this instruction goes */
 	s32 guard_to;
 	/* additional information about the loop if this instruction is a loop header */
@@ -1543,6 +1603,9 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+			struct bpf_verifier_state *cur);
 
 /* join.c */
 int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 269810b128c4..3e0b096cfc9f 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index fe513b7a8e88..4da051767e1d 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -398,3 +398,250 @@ static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
 	bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
 	return true;
 }
+
+static bool join_loop_stack(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
+{
+	u32 i;
+
+	if (fa->loop_stack_cnt != fb->loop_stack_cnt)
+		return false;
+	/* what is known about a counted loop comes from the state it was entered with */
+	for (i = 0; i < fa->loop_stack_cnt; i++)
+		if (fa->loop_stack[i].loop_id != fb->loop_stack[i].loop_id ||
+		    fa->loop_stack[i].terminates != fb->loop_stack[i].terminates ||
+		    fa->loop_stack[i].guarded != fb->loop_stack[i].guarded ||
+		    fa->loop_stack[i].counted != fb->loop_stack[i].counted ||
+		    fa->loop_stack[i].apart != fb->loop_stack[i].apart ||
+		    (fa->loop_stack[i].apart && fa->loop_stack[i].trip != fb->loop_stack[i].trip) ||
+		    fa->loop_stack[i].refs != fb->loop_stack[i].refs ||
+		    fa->loop_stack[i].max_header_count != fb->loop_stack[i].max_header_count ||
+		    fa->loop_stack[i].entry_state != fb->loop_stack[i].entry_state)
+			return false;
+	return true;
+}
+
+static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
+		      const struct bpf_func_state *fa, const struct bpf_func_state *fb,
+		      u16 live_regs, int pass)
+{
+	static const struct bpf_reg_state not_init = { .type = NOT_INIT };
+	struct bpf_verifier_env *env = js->env;
+	u32 i, nslots_b;
+	bool in_a, in_b;
+
+	if (pass == 1) {
+		if (fa->callsite != fb->callsite || fa->frameno != fb->frameno ||
+		    fa->subprogno != fb->subprogno)
+			return false;
+		if (fa->in_callback_fn != fb->in_callback_fn ||
+		    fa->in_async_callback_fn != fb->in_async_callback_fn ||
+		    fa->in_exception_callback_fn != fb->in_exception_callback_fn ||
+		    fa->in_rcu_callback_fn != fb->in_rcu_callback_fn ||
+		    fa->async_entry_cnt != fb->async_entry_cnt ||
+		    memcmp(&fa->callback_ret_range, &fb->callback_ret_range,
+			   sizeof(fa->callback_ret_range)))
+			return false;
+		if (fa->entry_refs != fb->entry_refs ||
+		    fa->entry_rcu_locks != fb->entry_rcu_locks ||
+		    fa->entry_preempt_locks != fb->entry_preempt_locks)
+			return false;
+		if (!join_loop_stack(fa, fb))
+			return false;
+	}
+
+	for (i = 0; i < MAX_BPF_REG; i++) {
+		if (!(live_regs & BIT(i))) {
+			if (pass == 2 && i < BPF_REG_FP)
+				bpf_mark_reg_not_init(env, &fd->regs[i]);
+			continue;
+		}
+		js->place = i;
+		if (!join_reg(js, &fd->regs[i], &fa->regs[i], &fb->regs[i], pass))
+			return false;
+	}
+
+	nslots_b = bpf_stack_nr_slots(fb);
+	for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
+		const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
+
+		js->place = MAX_BPF_REG + i;
+		if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
+			return false;
+	}
+	js->place = -1;
+	/* Slots that exist only in 'b' are STACK_INVALID in the result. */
+	for (i = bpf_stack_nr_slots(fa); pass == 1 && i < nslots_b; i++) {
+		const struct bpf_stack_state *sb = bpf_stack_slot(fb, i);
+
+		if (slot_is_special(sb) ||
+		    (slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE))
+			return false;
+	}
+
+	if (fa->out_stack_arg_cnt < fb->out_stack_arg_cnt)
+		return false;
+	/* incoming arguments are not joined with outgoing ones: that side has no outgoing ones */
+	in_a = bpf_stack_args_are_incoming(fa) && !bpf_stack_args_are_incoming(fb);
+	in_b = bpf_stack_args_are_incoming(fb) && !bpf_stack_args_are_incoming(fa);
+	for (i = 0; i < fa->out_stack_arg_cnt; i++) {
+		const struct bpf_reg_state *rb;
+
+		if (in_a) {
+			if (pass == 2)
+				bpf_mark_reg_not_init(env, &fd->stack_arg_regs[i]);
+			continue;
+		}
+		rb = i < fb->out_stack_arg_cnt && !in_b ? &fb->stack_arg_regs[i] : &not_init;
+		if (!join_reg(js, &fd->stack_arg_regs[i], &fa->stack_arg_regs[i], rb, pass))
+			return false;
+	}
+
+	if (pass == 2) {
+		fd->no_stack_arg_load = fa->no_stack_arg_load || fb->no_stack_arg_load;
+		fd->callback_depth = min(fa->callback_depth, fb->callback_depth);
+		for (i = 0; i < fd->loop_stack_cnt; i++)
+			fd->loop_stack[i].trip = max(fd->loop_stack[i].trip, fb->loop_stack[i].trip);
+	}
+	return true;
+}
+
+/* Things that can't be different in two states that are joined */
+static bool join_check_state(const struct bpf_verifier_state *a,
+			     const struct bpf_verifier_state *b)
+{
+	int i;
+
+	if (a->curframe != b->curframe || a->insn_idx != b->insn_idx)
+		return false;
+	if (a->speculative != b->speculative || a->in_sleepable != b->in_sleepable ||
+	    a->callers_unknown != b->callers_unknown ||
+	    a->unwound != b->unwound || a->changed_pkt_data != b->changed_pkt_data)
+		return false;
+	if (a->active_locks != b->active_locks ||
+	    a->active_preempt_locks != b->active_preempt_locks ||
+	    a->active_rcu_locks != b->active_rcu_locks ||
+	    a->active_lock_ptr != b->active_lock_ptr ||
+	    !a->active_irq_id != !b->active_irq_id ||
+	    !a->active_lock_id != !b->active_lock_id)
+		return false;
+	if (a->acquired_refs != b->acquired_refs)
+		return false;
+	for (i = 0; i < a->acquired_refs; i++) {
+		if (a->refs[i].type != b->refs[i].type)
+			return false;
+		switch (a->refs[i].type) {
+		case REF_TYPE_PTR:
+		case REF_TYPE_IRQ:
+			break;
+		default:
+			if (a->refs[i].ptr != b->refs[i].ptr)
+				return false;
+		}
+	}
+	return true;
+}
+
+static void join_refs_note(struct bpf_join_state *js, const struct bpf_verifier_state *a,
+			   const struct bpf_verifier_state *b)
+{
+	int i;
+
+	jid_note(js, a->active_irq_id, b->active_irq_id, true);
+	jid_note(js, a->active_lock_id, b->active_lock_id, true);
+	for (i = 0; i < a->acquired_refs; i++) {
+		jid_note(js, a->refs[i].id, b->refs[i].id, true);
+		if (a->refs[i].type == REF_TYPE_PTR)
+			jid_note(js, a->refs[i].parent_id, b->refs[i].parent_id, true);
+	}
+}
+
+static bool join_refs_check(struct bpf_join_state *js, const struct bpf_verifier_state *a,
+			    const struct bpf_verifier_state *b)
+{
+	int i;
+
+	if (!jid_strict_ok(js, a->active_irq_id, b->active_irq_id) ||
+	    !jid_strict_ok(js, a->active_lock_id, b->active_lock_id))
+		return false;
+	for (i = 0; i < a->acquired_refs; i++) {
+		if (!jid_strict_ok(js, a->refs[i].id, b->refs[i].id))
+			return false;
+		if (a->refs[i].type == REF_TYPE_PTR &&
+		    !jid_strict_ok(js, a->refs[i].parent_id, b->refs[i].parent_id))
+			return false;
+	}
+	return true;
+}
+
+/* The result has to describe everything that an input describes */
+static int join_check(struct bpf_join_state *js, struct bpf_verifier_state *dst,
+		      const struct bpf_verifier_state *in, const char *which)
+{
+	struct bpf_verifier_env *env = js->env;
+
+	if (bpf_state_subsumes(env, dst, (struct bpf_verifier_state *)in))
+		return 0;
+	/* check_ids() has no room left: that says nothing about the join */
+	if (env->idmap_scratch.cnt == BPF_ID_MAP_SIZE)
+		return 1;
+	bpf_log(&env->log, "join:");
+	print_verifier_state(env, dst, dst->curframe, true);
+	bpf_log(&env->log, "input %s:", which);
+	print_verifier_state(env, in, in->curframe, true);
+	verifier_bug(env, "join of states at insn %u does not subsume input '%s'",
+		     dst->insn_idx, which);
+	return -EFAULT;
+}
+
+/*
+ * Compute in @dst the state that describes everything that @a or @b describe.
+ * @dst has to be a valid state (zeroed is fine) different from @a and @b, its
+ * previous content is released. Bookkeeping fields are copied from @a.
+ *
+ * Returns 0 on success, 1 if there is no join, negative error otherwise.
+ * When 1 is returned the content of @dst is undefined, but it's a valid state.
+ */
+int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
+		    const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
+		    bool widen)
+{
+	struct bpf_join_state *js = join_state(env);
+	const struct bpf_func_state *frame;
+	int pass, fr, err;
+
+	if (!js)
+		return -ENOMEM;
+	js->widen = widen;
+	js->npairs = 0;
+	js->pairs_overflow = false;
+
+	if (!join_check_state(a, b))
+		return 1;
+	err = bpf_copy_verifier_state(dst, a);
+	if (err)
+		return err;
+
+	for (pass = 1; pass <= 2; pass++) {
+		if (pass == 1)
+			join_refs_note(js, a, b);
+		else if (!join_refs_check(js, a, b))
+			return 1;
+		for (fr = 0; fr <= a->curframe; fr++) {
+			u32 insn_idx = bpf_frame_insn_idx((struct bpf_verifier_state *)a, fr);
+			u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
+			u32 cnt;
+
+			/* not the states that get to the head through a call of the function */
+			frame = a->frame[fr];
+			cnt = frame->loop_stack_cnt;
+			js->loop = widen && fr == a->curframe && cnt &&
+				   frame->loop_stack[cnt - 1].loop_id == insn_idx ?
+				   env->insn_aux_data[insn_idx].loop : NULL;
+			if (!join_func(js, dst->frame[fr], a->frame[fr], b->frame[fr],
+				       live_regs, pass))
+				return 1;
+		}
+	}
+
+	return join_check(js, dst, a, "a") ?: join_check(js, dst, b, "b");
+}
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 50480c42bcad..4136a708a78e 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -1765,3 +1765,15 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 	 */
 	return bpf_split_cur_state(env);
 }
+
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+	return clean_verifier_state(env, st);
+}
+
+/* Everything that @cur describes is described by @old. */
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+			struct bpf_verifier_state *cur)
+{
+	return states_equal(env, old, cur, RANGE_WITHIN);
+}
-- 
2.55.0

Re: [PATCH bpf-next v4 05/46] bpf: Add join of verifier states

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] bpf: join_reg() zeroing dst->delta breaks explore_alu_limits
  in join_check()
- [Medium] bpf: asymmetric unallocated and poisoned stack handling in
  join_func()
--

commit 81dc6f947daf4ec051f2b11325ec95b20b615ab2
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Add join of verifier states

This commit completes the state-joining implementation in join.c by adding
frame, reference, loop stack, and whole-state join logic along with a
subsumption check on the resulting state.

> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> index fe513b7a8e884..4da051767e1d9 100644
> --- a/kernel/bpf/join.c
> +++ b/kernel/bpf/join.c
> @@ -398,3 +398,250 @@ static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
>  	bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
>  	return true;
>  }

[ ... ]

> +	nslots_b = bpf_stack_nr_slots(fb);
> +	for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
> +		const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
> +
> +		js->place = MAX_BPF_REG + i;
> +		if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
> +			return false;
> +	}
> +	js->place = -1;
> +	/* Slots that exist only in 'b' are STACK_INVALID in the result. */
> +	for (i = bpf_stack_nr_slots(fa); pass == 1 && i < nslots_b; i++) {
> +		const struct bpf_stack_state *sb = bpf_stack_slot(fb, i);
> +
> +		if (slot_is_special(sb) ||
> +		    (slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE))
> +			return false;
> +	}

[Severity: Medium]
When !js->env->allow_uninit_stack, does this loop drop live initialized
stack slots that exist only in fb?

For slots beyond bpf_stack_nr_slots(fa), the loop only rejects special
slots and non-scalar spills.  If fb has a live scalar STACK_SPILL,
STACK_MISC, or STACK_ZERO slot at i >= bpf_stack_nr_slots(fa), the join
succeeds (since stacksafe() only checks up to fd->allocated_stack, which is
copied from fa) and leaves slot i unallocated in fd, causing later reads on
path b to fail verification with -EACCES.  When fa and fb are passed in the
opposite order, join_slot() rejects the join at:

kernel/bpf/join.c:join_slot() {
	...
	/* the prog can't read uninitialized stack: slots of different kinds have no join */
	if (!js->env->allow_uninit_stack)
		return !memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type));
}

Conversely, when bpf_stack_nr_slots(fa) > bpf_stack_nr_slots(fb) and
bpf_clean_verifier_state() has marked dead slots in fa as STACK_POISON,
join_slot() substitutes invalid_slot (all STACK_INVALID) for the missing
slot in fb, and the memcmp() above rejects the join because STACK_POISON !=
STACK_INVALID, even though both represent dead/unallocated stack bytes.

[ ... ]

> +/* The result has to describe everything that an input describes */
> +static int join_check(struct bpf_join_state *js, struct bpf_verifier_state *dst,
> +		      const struct bpf_verifier_state *in, const char *which)
> +{
> +	struct bpf_verifier_env *env = js->env;
> +
> +	if (bpf_state_subsumes(env, dst, (struct bpf_verifier_state *)in))
> +		return 0;
> +	/* check_ids() has no room left: that says nothing about the join */
> +	if (env->idmap_scratch.cnt == BPF_ID_MAP_SIZE)
> +		return 1;
> +	bpf_log(&env->log, "join:");
> +	print_verifier_state(env, dst, dst->curframe, true);
> +	bpf_log(&env->log, "input %s:", which);
> +	print_verifier_state(env, in, in->curframe, true);
> +	verifier_bug(env, "join of states at insn %u does not subsume input '%s'",
> +		     dst->insn_idx, which);
> +	return -EFAULT;
> +}

[Severity: Medium]
Can join_check() trigger a false verifier_bug() when env->explore_alu_limits
is set and join_reg() drops an inconsistent scalar link?

In struct bpf_reg_state, delta lies before id.  In join_reg(), two
SCALAR_VALUE registers with matching non-zero delta and identical ranges
pass reg_value_equal(ra, rb) when js->env->explore_alu_limits is true, but
if their IDs do not form a consistent bijection across the states, pass 2
clears both dst->id and dst->delta:

kernel/bpf/join.c:join_reg() {
	...
	case SCALAR_VALUE:
		if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
			return false;
		if (pass == 1) {
			jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
			return true;
		}
		join_range(js, dst, ra, rb);
		/* a link that exists on one side only is dropped */
		if (ra->id && !jid_good(js, ra->id, rb->id)) {
			dst->id = 0;
			dst->delta = 0;
		}
		return true;
}

When join_check() then calls bpf_state_subsumes(), regsafe() checks:

kernel/bpf/states.c:regsafe() {
	...
	case SCALAR_VALUE:
		if (env->explore_alu_limits) {
			/* explore_alu_limits disables tnum_in() and range_within()
			 * logic and requires everything to be strict
			 */
			return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
			       check_scalar_ids(rold->id, rcur->id, idmap);
		}
}

Because dst->delta is now 0 while the input register still has a non-zero
delta, memcmp() fails and join_check() hits verifier_bug() and returns
-EFAULT.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=5

[PATCH bpf-next v4 06/46] bpf: Tell which arguments of a call a join must leave alone

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

get_some_addr() of the selftest mem_rdonly_untrusted:

  if (bpf_get_prandom_u32())
          return bpf_rdonly_cast(0, bpf_core_type_id_kernel(struct sock));
  else
          return bpf_rdonly_cast(0, 0);

The compiler makes one call of the two, with r2 = 0 or the type id. The
paths meet in front of the call, and a join makes a range of two constants:

  R2 must be a known constant

cil_sock4_connect() and cil_sock4_sendmsg() of Cilium's bpf_sock:

  if (err == -EHOSTUNREACH || err == -ENOMEM) {
          try_set_retval(err);

The compiler decides the 'if' where err is set, and keeps a flag. All
paths meet, then the flag is tested. In the join err is [-113, 1]:

  At bpf_set_retval the register R1 has .. should have been in [-4095, 0]

The progs are valid. A pass that joins states must know, ahead of a call,
which registers it must not join.

antistall_set() of scx_layered:

  bpf_for_each(scx_dsq, __p, dsq_id, 0) {

bpf_iter_scx_dsq_next() ends this loop: it needs no may_goto. The pass that
finds the places for may_goto must know that, before the main pass sets
is_iter_next in insn_aux_data. With may_goto in each such loop layered_init
and mitosis_init are rejected on the added edge, and the schedulers in
tools/sched_ext take 48 % more insns.

  bpf_call_const_arg_regs()    a helper or a kfunc takes only a known
                               constant in these registers
  bpf_call_checked_arg_regs()  it compares them with a range: sizes of
                               memory, the argument of bpf_set_retval()
  bpf_insn_is_iter_next()      the call is bpf_iter_*_next()

The first two go by the types of the arguments in struct bpf_func_proto,
like check_func_arg(), for r1 - r5. bpf_set_retval() is checked elsewhere
and is added by hand. Nothing calls the three in this patch.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  3 ++
 kernel/bpf/verifier.c        | 62 ++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 922f01ee8136..7fda41a4bf25 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1614,6 +1614,9 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
 void bpf_join_free(struct bpf_verifier_env *env);
 
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
+u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
+u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
+bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 801e4607a462..ea6dd39982f2 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -9215,6 +9215,13 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type)
 	return type == ARG_MEM_SIZE || type == ARG_MEM_SIZE_OR_ZERO;
 }
 
+/* A number that check_func_arg() takes only when it is a known constant */
+static bool arg_type_is_const(enum bpf_arg_type type)
+{
+	return type == ARG_CONST_SCALAR || type == ARG_CONST_MEM_SIZE ||
+	       type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
+}
+
 static bool arg_type_is_raw_mem(enum bpf_arg_type type)
 {
 	/*
@@ -15379,6 +15386,61 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return info;
 }
 
+/* insn_aux_data->is_iter_next is not there before the main pass */
+bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	struct bpf_call_arg_meta meta;
+
+	return bpf_pseudo_kfunc_call(insn) &&
+	       !bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) &&
+	       bpf_is_iter_next_kfunc(&meta);
+}
+
+/*
+ * check_func_arg() goes by the types of the arguments in the proto of a helper or a kfunc.
+ * So does what has to be known ahead of the call: the registers with arguments of a kind.
+ */
+static u16 call_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn,
+			 bool (*is)(enum bpf_arg_type type))
+{
+	const struct btf_func_model *fm = NULL;
+	const struct bpf_func_proto *fn;
+	struct bpf_kfunc_desc *desc;
+	u32 i, slot = 0;
+	u16 mask = 0;
+
+	if (bpf_pseudo_kfunc_call(insn)) {
+		desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+		if (!desc)
+			return 0;
+		fn = &desc->proto;
+		fm = &desc->func_model;
+	} else if (bpf_get_helper_proto(env, insn->imm, &fn) < 0) {
+		return 0;
+	}
+	for (i = 0; i < ARRAY_SIZE(fn->arg_type) && fn->arg_type[i] != ARG_UNUSED &&
+		    slot < MAX_BPF_FUNC_REG_ARGS; i++) {
+		if (is(fn->arg_type[i]))
+			mask |= BIT(BPF_REG_1 + slot);
+		slot += fm ? btf_func_model_arg_slots(fm, i) : 1;
+	}
+	return mask;
+}
+
+/* The registers in which the helper or the kfunc takes nothing but a known constant */
+u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	return call_arg_regs(env, insn, arg_type_is_const);
+}
+
+/* The registers in which it takes a number that has to be in a range */
+u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	if (bpf_helper_call(insn) && insn->imm == BPF_FUNC_set_retval)
+		return BIT(BPF_REG_1);
+	return call_arg_regs(env, insn, arg_type_is_mem_size);
+}
+
 /*
  * Describe a kfunc's stack access through argument slot @arg (0-based).
  * As for helpers, must_write describes destruction of prior verifier state,
-- 
2.55.0

Re: [PATCH bpf-next v4 06/46] bpf: Tell which arguments of a call a join must leave alone

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2408 bytes --]

> bpf: Tell which arguments of a call a join must leave alone

TL;DR: bpf_call_checked_arg_regs() may miss the range-checked bpf_throw()
cookie in R1, so a later join could widen it and wrongly reject a valid
program at exit.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +/* The registers in which it takes a number that has to be in a range */
> +u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
> +{
> +	if (bpf_helper_call(insn) && insn->imm == BPF_FUNC_set_retval)
> +		return BIT(BPF_REG_1);
> +	return call_arg_regs(env, insn, arg_type_is_mem_size);
> +}

bpf_set_retval() is not the only call whose scalar argument is range
checked outside check_func_arg().  check_kfunc_call() does the same for
the cookie of bpf_throw():

    if (meta.func_id == special_kfunc_list[KF_bpf_throw]) {
        ...
        /* In the case of the default callback, the cookie value passed
         * to bpf_throw becomes the return value of the program.
         */
        if (!env->exception_callback_subprog) {
            err = check_return_code(env, BPF_REG_1, "R1");

check_return_code() rejects R1 when it is outside the return range of the
program, for example [0, 1] for BPF_PROG_TYPE_CGROUP_SKB.

For bpf_throw, call_arg_regs() sees one plain u64 argument, which
get_kfunc_arg_type() classifies as ARG_SCALAR, so it returns 0 and nothing
adds BIT(BPF_REG_1).

The consumer added later in the series, flow_tested_regs() in
kernel/bpf/flow.c (80384e102a64 "bpf: Find the registers and stack slots
that later checks depend on"), marks R0 at exits only, and nothing in
flow.c at the end of the series mentions bpf_throw.  So the same pattern
as the bpf_set_retval() case in the commit message (a flag tested before
bpf_throw(cookie), with the paths joined first) would widen the cookie and
reject a valid program with "At program exit the register R1 has ...
should have been in [0, 1]".

Should bpf_call_checked_arg_regs() return BIT(BPF_REG_1) for the bpf_throw
kfunc when env->exception_callback_subprog is not set, as it does for
bpf_set_retval()?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

[PATCH bpf-next v4 07/46] bpf: Add the data structures of the main pass over the control flow graph

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier follows each path of a prog from the entry to exit. It ends a
path early only where an earlier path left a state that covers the current
one. A live value that differs from path to path defeats that. Then the
work grows with the number of paths, not with the size of the prog:

  for b in bytes { h = (h ^ *b as u64).wrapping_mul(0x100000001b3); }

In this FNV hash h is a new number in each iteration, so no state covers
the next one. Branches that set different values do the same to C code.

  insns processed                                     today   this series
  rust-bpf   hash_fnv_bytes                         1000001 *         160
  rust-bpf   scx_qmap/qmap_init_cids                 131088 *         196
  selftests  test_verif_scale3/balancer_ingress      821107         57155
  scx        scx_eevdf/eevdf_dispatch                350041         15322
  Cilium     bpf_host/cil_to_netdev, Oct 2026         64658          7538
  * "BPF program is too large", "The sequence of 8193 jumps is too complex"

flow.c is a new main pass: a data-flow analysis over the control flow
graph that runs to a fixpoint. Where two paths meet, it joins their states
into one state that covers both: r1=1 and r1=5 give r1=[1,5]. At the head
of a loop it widens what grows: h becomes an unknown scalar, and one more
pass over the body ends the loop. A join can lose a fact that a later check
depends on. Thus a basic block has a set of states, not one state:

  a state arrives at a basic block        the verifier
  a state of the set covers it            drops it
  it has a join with a state of the set   replaces that state with the join
  all other cases                         adds it to the set

The verifier verifies a block with each new state of its set and sends the
results to the successors of the block. The pass ends when no set has a new
state. The check of one insn does not change.

Add the head comment, the limits and the data structures:

  struct flow_state   a state in a set, or the state of a call that waits
  struct flow_set     the states at one insn
  struct flow_entry   the calls that wait for a subprog, its exit states
  struct flow_need    what the checks after an insn depend on

The limits are on states and grow with the prog. For a prog of N insns:
1024 at an insn, 16 * N + 4096 at a time, 256 * N + 64K in all.
Nothing builds flow.c until its last function is there.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 181 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 181 insertions(+)
 create mode 100644 kernel/bpf/flow.c

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
new file mode 100644
index 000000000000..01681957855d
--- /dev/null
+++ b/kernel/bpf/flow.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * The main pass of the verifier as a fixpoint over the control flow graph.
+ *
+ * Every insn that starts a basic block has a set of states. The state that
+ * arrives there is
+ * - dropped, if it is within a state of the set,
+ * - joined with a state of the set, if there is a join, see bpf_states_join(),
+ *   that doesn't make up what neither of them has, see flow_join_makes_up(),
+ * - added to the set otherwise.
+ * What is new in the set is pending. Pending states are processed in the order
+ * of the program in which a loop is done with before what follows it, see
+ * flow_compute_order(). The pass ends when nothing is pending.
+ *
+ * Loops. The state that enters a loop is made to stand for all iterations with
+ * what scalar evolution knows about the loop, see bpf_flow_loop_enter().
+ * The state that comes back to the loop head is expected to be within it.
+ * If it is not, the two are joined, what grew is forgotten, and the loop is
+ * done again.
+ *
+ * Functions. A static function that gets nothing but numbers and pointers to
+ * what is not the stack is verified on its own, for the states at its entry
+ * and not for every chain of calls. The call waits. What comes out of the
+ * function goes to all calls that wait for the states it comes from.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/filter.h>
+#include <linux/jhash.h>
+#include <linux/rbtree.h>
+#include <linux/slab.h>
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+/*
+ * States at an insn that have no join: kinds of values, references, chains of calls.
+ * A state that arrives is compared with those that it may be related to, see flow_may_relate().
+ * At worst that is all of them, and the work is the square of it.
+ */
+#define BPF_FLOW_MAX_STATES	1024
+/*
+ * All states in the sets at a time: that many for every insn of the prog, and some. Not counted:
+ * calls that wait, what came out of a function, the states that loops were entered with.
+ */
+#define BPF_FLOW_LIVE_PER_INSN	16
+#define BPF_FLOW_LIVE_BASE	4096
+/* All states that were ever there. It bounds the work by the size of the prog. */
+#define BPF_FLOW_TOTAL_PER_INSN	256
+#define BPF_FLOW_TOTAL_BASE	65536
+/*
+ * What comes out of a function is joined that many times. Then it is widened, like what
+ * comes back to a loop head: the exit states have to stop changing.
+ */
+#define BPF_FLOW_MAX_JOINS	16
+/* With fewer states at an insn it's faster to compare them than to look at their shapes. */
+#define BPF_FLOW_MIN_SHAPES	8
+/* bpf_loop() with that many iterations at most is walked. */
+#define BPF_FLOW_MAX_CALLBACKS	8
+/* What a callback returns to the helper with is joined that many times before it is widened. */
+#define BPF_FLOW_CALLBACK_JOINS	4
+
+enum flow_cycle {
+	FLOW_NO_CYCLE,
+	/* what grew is forgotten */
+	FLOW_CYCLE,
+	/* after a while. A flag that a callback sets doesn't grow any further. */
+	FLOW_CALLBACK_CYCLE,
+};
+
+/* See flow_may_relate() */
+enum flow_word {
+	FLOW_CHAIN,
+	FLOW_CTX,
+	FLOW_LAYOUT,
+	FLOW_PTRS,
+	FLOW_OFFS,
+	FLOW_SHAPE_WORDS,
+};
+
+struct flow_shape {
+	u32 word[FLOW_SHAPE_WORDS];
+	/* how much of the packet is checked, the sum over the pointers to the packet */
+	u32 pkt_range;
+	/* what comes after FLOW_CTX is computed */
+	bool places;
+};
+
+struct flow_state {
+	struct bpf_verifier_state st;
+	/* states at the insn, see struct flow_set, calls of struct flow_entry */
+	struct list_head node;
+	struct rb_node rb;		/* pending states */
+	int prev_insn_idx;
+	u32 seq;
+	struct flow_shape shape;
+	u16 joins;
+	bool pending;
+	/* the order of pending states, see flow_set_pending() */
+	u8 key_len;
+	u32 key[MAX_CALL_FRAMES + 1];
+};
+
+/* what came out of a function that is verified on its own */
+struct flow_exit {
+	struct list_head node;
+	struct bpf_verifier_state st;
+	u16 joins;
+};
+
+/*
+ * Every call that waits makes an entry of the function: the state at its first insn and what
+ * becomes of it. Entries whose states met are one set: what comes out of the function for one of
+ * them is for the calls of all of them. One entry stands for the set and has its lists.
+ */
+struct flow_entry {
+	struct flow_entry *up;		/* towards the one that stands for the set: up == itself */
+	struct list_head calls;		/* struct flow_state: the calls that wait */
+	struct list_head exits;		/* struct flow_exit: what came out */
+};
+
+struct flow_func {
+	int exit_idx;			/* all exits are compared as if they were here */
+};
+
+/* States at an insn, in the order in which they came. */
+struct flow_set {
+	struct list_head list;
+};
+
+struct flow_united {
+	u32 a, b;
+};
+
+struct flow_kept {
+	struct list_head node;
+	struct bpf_verifier_state st;
+};
+
+/* What the checks ahead of an insn look at, see enum flow_need_kind */
+struct flow_need {
+	u64 slots;			/* of the stack, fp-8 is bit 0 */
+	u16 regs;
+};
+
+/* see flow_compute_need_const() */
+enum flow_need_kind {
+	FLOW_NEED_ARG,
+	FLOW_NEED_STACK_OFF,
+	FLOW_NEED_PTR_OFF,
+	FLOW_NEED_JUMP,
+	FLOW_NEED_VALUE_OFF,
+	FLOW_NEED_BASE,
+	FLOW_NEED_JUMP_ALU,
+	FLOW_NEED_KINDS,
+};
+
+struct bpf_flow {
+	/* computed once */
+	u32 len;			/* of the prog during the pass */
+	u32 nr_funcs;
+	u32 *pos;			/* place of the insn in the order */
+	u32 *rank;			/* callers go first */
+	struct flow_need *need[FLOW_NEED_KINDS];
+	unsigned long *point;		/* insn starts a basic block */
+	unsigned long *keep;		/* states that were processed are compared with */
+
+	/* of the pass */
+	struct flow_set *states;	/* per insn */
+	struct flow_set *enter;		/* per insn: the states that enter the loop at its head */
+	u32 *nr_states;			/* per insn: the two sets are counted together */
+	struct rb_root_cached pending;
+	struct flow_func *funcs;
+	struct flow_entry **entries;	/* bpf_verifier_state.flow_entry is the index, 0 is none */
+	u32 entry_cnt, entry_cap;
+	struct flow_united *united;
+	u32 united_cnt, united_cap;
+	struct list_head kept;
+	struct bpf_verifier_state tmp;
+	u32 seq;
+	u32 live_states;
+};
-- 
2.55.0

[PATCH bpf-next v4 08/46] bpf: Order the insns, find what stops a loop and where may_goto can go

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

A fixpoint shows that each iteration of a loop is safe, not that it ends:

  do { i++; sum += PT_REGS_RC(ctx); } while (i < 0x100000000ULL);

Today selftest loop3 hits the limit of 1M insns. The main pass widens i and
is done, so the verifier has to add a may_goto. That is an edge that the
prog does not have. Its place decides which progs load:

1. None, if each iteration executes a may_goto that leaves the loop, or
bpf_iter_*_next() of an iterator that was created before the loop.

  bpf_repeat(1000000) {
          map_val = bpf_map_lookup_elem(&hash_map, &key);
          if (map_val) { found = true; break; }
  }
  if (found) *map_val = 123;

An added may_goto goes where 'break' goes, without the test. Selftest
iters/iter_tricky_but_fine: "R6 invalid mem access 'map_value_or_null'".
layered_init of scx_layered and mitosis_init of scx_mitosis fail alike.

2. In front of a conditional jump that leaves the loop, to its target: the
prog gets there with the same registers and stack. One jump of the loop
itself, bpf_insn_leaves_loop(), that each iteration goes through is enough:
the 'while' of loop3. Else each back edge gets one: a may_goto at the head
skips the body, and the first exit can read what the body sets.

3. At the head, if a back edge has none. The compiler makes one loop of

  while self.locked.compare_exchange(0, 1, Acquire, Relaxed).is_err() {
      while self.locked.load(Relaxed) != 0 { core::hint::spin_loop(); }
  }

and the inner back edge has none. Without this place the main pass says
"loop at insn 127 never exits" for 142 of the 225 progs of the schedulers
of rust-bpf. This edge can skip what stops an outer loop: it leaves all.

The added insn executes in each state. Thus a loop has it for all states or
for none. The main pass can find out late: it first goes through loop3
iteration by iteration, as today, and gives up. Then the pass is repeated.

The main pass takes the blocks that have a new state in the order of
flow_compute_order(): reverse post order, but a loop stays together. Else
the code behind a 'break' is verified before the loop is done, then again.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   1 +
 kernel/bpf/flow.c            | 335 +++++++++++++++++++++++++++++++++++
 kernel/bpf/loops.c           |   8 +
 3 files changed, 344 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7fda41a4bf25..332701cdf3ee 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1617,6 +1617,7 @@ int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
+bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 01681957855d..0f5acdf1cb94 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -179,3 +179,338 @@ struct bpf_flow {
 	u32 seq;
 	u32 live_states;
 };
+
+/*
+ * The order. Reverse post order is a topological order of the graph without
+ * back edges, but it may put what follows a loop between the insns of
+ * the loop. Keep the insns of a loop together, the head first: in the list of
+ * its parent the loop takes the place of the insn of it that comes first.
+ *
+ * Every cycle has a back edge of the depth first search in it and
+ * bpf_compute_loops() makes the target of such edge a loop head. So every edge
+ * that goes backwards in the order is in a loop whose head is at or before its
+ * target, loops that are not reducible included.
+ */
+static int flow_compute_order(struct bpf_verifier_env *env, struct bpf_flow *flow)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int len = env->prog->len, i, s, n, h, top, depth;
+	int *next, *head, *tail, *stack;
+	unsigned long *linked;
+	u32 pos = 0;
+	int err = -ENOMEM;
+
+	/* lists of members are indexed by loop head, len is the function */
+	next = kvmalloc_array(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+	head = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+	tail = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+	stack = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+	linked = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+	if (!next || !head || !tail || !stack || !linked)
+		goto out;
+
+	for (i = 0; i <= len; i++)
+		head[i] = tail[i] = -1;
+
+	for (s = 0; s < env->subprog_cnt; s++) {
+		int first = env->subprog_info[s].postorder_start;
+		int last = env->subprog_info[s + 1].postorder_start - 1;
+
+		head[len] = tail[len] = -1;
+		for (i = last; i >= first; i--) {
+			n = env->cfg.insn_postorder[i];
+			/* n, then the heads of the loops around it that are not in a list yet */
+			depth = 0;
+			for (h = n; h >= 0 && !test_bit(h, linked); h = aux[h].loop_header)
+				stack[depth++] = h;
+			while (depth) {
+				h = stack[--depth];
+				top = aux[h].loop_header >= 0 ? aux[h].loop_header : len;
+				__set_bit(h, linked);
+				next[h] = -1;
+				if (tail[top] < 0)
+					head[top] = h;
+				else
+					next[tail[top]] = h;
+				tail[top] = h;
+			}
+		}
+
+		depth = 0;
+		n = head[len];
+		while (n >= 0 || depth) {
+			if (n < 0) {
+				n = next[stack[--depth]];
+				continue;
+			}
+			flow->pos[n] = pos++;
+			if (aux[n].loop && head[n] >= 0) {
+				stack[depth++] = n;
+				n = head[n];
+			} else {
+				n = next[n];
+			}
+		}
+	}
+	err = 0;
+out:
+	kvfree(next);
+	kvfree(head);
+	kvfree(tail);
+	kvfree(stack);
+	kvfree(linked);
+	return err;
+}
+
+/* Does the insn at @idx stop the loop @h? */
+static bool flow_is_bound(struct bpf_verifier_env *env, int h, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	int t;
+
+	/* Once it is out of budget it's a jump. It has to leave the loop. */
+	if (bpf_is_may_goto_insn(insn)) {
+		t = bpf_loop_at_index(env, idx + insn->off + 1);
+		return t != h && !bpf_is_nested_loop(env, t, h);
+	}
+	if (!bpf_insn_is_iter_next(env, insn))
+		return false;
+	/* unless the main pass finds that it doesn't, see iter_stops_loops() */
+	t = env->insn_aux_data[idx].not_stopped;
+	return t < 0 || bpf_is_nested_loop(env, h, t);
+}
+
+/*
+ * What stops a loop at run time is may_goto that leaves it, or bpf_iter_*_next() of
+ * an iterator that is older than the loop, that every trip goes through. A loop without
+ * it gets may_goto unless the number of its trips turns out to be known.
+ *
+ * The place for it is in front of a conditional jump that leaves the loop, with
+ * the same target. The registers and the stack are what they are when
+ * the program leaves the loop there. One such jump that every trip goes
+ * through is enough. Otherwise every back edge needs one.
+ */
+static void flow_set_guard(struct bpf_verifier_env *env, int h, int n)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int t = n + env->prog->insnsi[n].off + 1;
+	int i = bpf_loop_at_index(env, t);
+
+	if (i == h || bpf_is_nested_loop(env, i, h))
+		t = n + 1;
+	aux[n].guard_place = true;
+	aux[n].guard_to = t;
+	aux[h].loop->has_guards = true;
+}
+
+static void flow_place_guards(struct bpf_verifier_env *env, int h, u32 *seen)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *loop = aux[h].loop;
+	int i, n, o, t, found = -1;
+
+	if (loop->backedges_overflow)
+		goto header;
+
+	/*
+	 * The head of a loop with more than one entry doesn't dominate what is in the loop.
+	 * Then an insn that dominates a back edge can be passed once, on the way in, and
+	 * never again. Every trip goes through the jump that is the back edge, and
+	 * through the head.
+	 */
+	if (loop->irreducible) {
+		for (i = 0; i < loop->backedges_cnt; i++)
+			if (!loop->backedges[i].latches_cnt ||
+			    loop->backedges[i].latches[0] != loop->backedges[i].from)
+				goto header;
+		for (i = 0; i < loop->backedges_cnt; i++)
+			flow_set_guard(env, h, loop->backedges[i].from);
+		return;
+	}
+
+	/* seen[n]: the number of back edges that n dominates */
+	for (i = 0; i < loop->backedges_cnt; i++)
+		for (n = loop->backedges[i].from; n >= 0; n = n == h ? -1 : env->idoms[n])
+			seen[n]++;
+	for (n = loop->backedges[0].from; n >= 0 && found < 0; n = n == h ? -1 : env->idoms[n])
+		if (seen[n] == loop->backedges_cnt && bpf_insn_leaves_loop(env, h, n))
+			found = n;
+	for (i = 0; i < loop->backedges_cnt; i++)
+		for (n = loop->backedges[i].from; n >= 0; n = n == h ? -1 : env->idoms[n])
+			seen[n] = 0;
+	if (found >= 0) {
+		flow_set_guard(env, h, found);
+		return;
+	}
+
+	for (i = 0; i < loop->backedges_cnt; i++)
+		if (!loop->backedges[i].latches_cnt)
+			goto header;
+	for (i = 0; i < loop->backedges_cnt; i++)
+		flow_set_guard(env, h, loop->backedges[i].latches[0]);
+	return;
+
+header:
+	/*
+	 * In front of a jump may_goto goes where the jump goes. From the head to where the loop
+	 * is left is not an edge of the program: it skips what is in the loop. If that is what
+	 * stops a loop around this one, bpf_iter_*_next() or i++, and the exit leads back to
+	 * the head of that loop, the program goes round and round from may_goto to may_goto.
+	 * So leave all loops.
+	 *
+	 * What is derived from the graph in the direction of the edges has to know about this
+	 * one, see bpf_head_guard_target().
+	 */
+	for (o = h; aux[o].loop_header >= 0; o = aux[o].loop_header)
+		;
+	/* a loop that has no exit can only be stopped together with the program */
+	if (!aux[o].loop->exits_cnt)
+		return;
+	t = aux[o].loop->exits[0].to;
+	aux[h].guard_place = true;
+	aux[h].guard_at_header = true;
+	aux[h].guard_to = t;
+	loop->has_guards = true;
+}
+
+/* Does every trip of the loop @h go through an insn that stops it? */
+static bool flow_loop_is_bounded(struct bpf_verifier_env *env, int h)
+{
+	struct bpf_loop *loop = env->insn_aux_data[h].loop;
+	int i, n;
+
+	/* see flow_place_guards() about loops with more than one entry */
+	if (loop->backedges_overflow || loop->irreducible)
+		return false;
+	for (i = 0; i < loop->backedges_cnt; i++) {
+		for (n = loop->backedges[i].from; n >= 0 && n != h; n = env->idoms[n])
+			if (flow_is_bound(env, h, n))
+				break;
+		if (n < 0 || !flow_is_bound(env, h, n))
+			return false;
+	}
+	return true;
+}
+
+/*
+ * The main pass got to bpf_iter_*_next() at @insn_idx in a state in which it doesn't stop
+ * the loop @h around it. What is assumed about the loops stays as it is until the pass
+ * is over, so that all states are treated alike.
+ */
+void bpf_flow_next_is_no_bound(struct bpf_verifier_env *env, int insn_idx, int h)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
+
+	if (aux->not_stopped >= 0 && !bpf_is_nested_loop(env, h, aux->not_stopped))
+		return;
+	aux->not_stopped = h;
+	env->restart = true;
+}
+
+/*
+ * True if a loop was taken for bounded in the pass and is not. The pass is done again then.
+ * A loop that has something else that stops it, like the outer one of two nested
+ * bpf_for(), is not affected. Every time at least one loop is found out for good.
+ */
+bool bpf_flow_loops_lost_bound(struct bpf_verifier_env *env)
+{
+	struct bpf_loop *loop;
+	bool lost = false;
+	int h;
+
+	for (h = 0; h < env->prog->len; h++) {
+		loop = env->insn_aux_data[h].loop;
+		if (!loop || !loop->rt_bounded || flow_loop_is_bounded(env, h))
+			continue;
+		loop->rt_bounded = false;
+		lost = true;
+	}
+	return lost;
+}
+
+/*
+ * A state is in the loop @h without a count, so the loop gets may_goto. At run time may_goto is
+ * there for everybody, also for those who know the number of trips or got to the end of the loop
+ * trip by trip. It can be out of budget in the first trip, because of a loop before this one.
+ *
+ * So a loop has may_goto or it has not, and a state takes the edges of may_goto if the loop had
+ * it when the state entered. If a state entered without it, with a count or on a walk that may
+ * give up, all is done again, and the loop has may_goto from the start.
+ */
+void bpf_flow_loop_gets_guards(struct bpf_verifier_env *env, int h)
+{
+	struct bpf_loop *loop = env->insn_aux_data[h].loop;
+
+	if (loop->guarded)
+		return;
+	loop->guarded = true;
+	if (loop->no_guard_edges)
+		env->guards_late = true;
+}
+
+/* The main prog and the global functions are about to be verified, maybe not for the first time */
+void bpf_flow_new_round(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int i;
+
+	env->restart = false;
+	env->guards_late = false;
+	for (i = 0; i < env->prog->len; i++) {
+		aux[i].guard_edge_dropped = false;
+		if (aux[i].loop)
+			aux[i].loop->no_guard_edges = false;
+	}
+}
+
+/*
+ * Which loops have something that stops them at run time, and where may_goto goes in the others.
+ * Returns 1 if a place is at the head of a loop.
+ */
+int bpf_flow_place_guards(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int len = env->prog->len, h, i, depth, ret = 0;
+	struct bpf_loop *loop;
+	u32 *seen;
+
+	seen = kvcalloc(len, sizeof(*seen), GFP_KERNEL_ACCOUNT);
+	if (!seen)
+		return -ENOMEM;
+
+	for (i = 0; i < len; i++) {
+		aux[i].not_stopped = -1;
+		depth = 0;
+		for (h = bpf_loop_at_index(env, i); h >= 0; h = aux[h].loop_header) {
+			/* the main pass doesn't get that deep */
+			if (++depth > LOOP_STACK_SIZE) {
+				verbose(env, "Too many nested loops at %d\n", i);
+				kvfree(seen);
+				return -E2BIG;
+			}
+			aux[h].loop->nr_insns++;
+			if (bpf_pseudo_call(&env->prog->insnsi[i]) ||
+			    bpf_is_callx(&env->prog->insnsi[i]))
+				aux[h].loop->has_calls = true;
+		}
+	}
+
+	for (h = 0; h < len; h++) {
+		loop = aux[h].loop;
+		if (!loop)
+			continue;
+		/* a deep nest of loops takes long */
+		if (signal_pending(current)) {
+			ret = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		loop->rt_bounded = flow_loop_is_bounded(env, h);
+		/* also when it's bounded: the main pass may find that it's not */
+		flow_place_guards(env, h, seen);
+		if (aux[h].guard_at_header)
+			ret = 1;
+	}
+	kvfree(seen);
+	return ret;
+}
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index cc15712f70d4..729e0055f86f 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -540,6 +540,14 @@ bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n)
 	       (f != h && !bpf_is_nested_loop(env, f, h));
 }
 
+/* The same for an insn that is in the loop @h itself */
+bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n)
+{
+	if (bpf_loop_at_index(env, n) != h)
+		return false;
+	return bpf_jump_leaves_loop(env, h, n);
+}
+
 int bpf_compute_loops(struct bpf_verifier_env *env)
 {
 	int i, j, k, s, t, iloop, sloop, err = 0, len = env->prog->len;
-- 
2.55.0

[PATCH bpf-next v4 09/46] bpf: Find the registers that hold a number or a pointer to arena

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The allocator of tools/sched_ext/scx_sdt.bpf.c looks for a free slot:

  if ((word & 0xffffffff) == 0) { num += 32; word >>= 32; }
  if ((word & 0xffff) == 0) { num += 16; word >>= 16; }
  ...                                         six times, this is ffs()
  return (i * 64) + ffs(freeslots);

The main pass joins the two states behind each 'if': num is [0, 32], then
[0, 48]. It must not join two constants that the prog adds to a pointer to
the packet, the context or a kernel object: these need the exact offset.
The states meet before the add, and there the add is only 'r0 += r3'. If r0
can be such a pointer, num stays a constant and each 'if' doubles the
states: up to 256 at an insn, not 2. sdt_init_task: 4897 insns, not 666.

bpf_flow_compute_kinds() goes forward through the control flow graph,
before the main pass. It finds the registers, stack slots and stack
arguments that hold, on each path to an insn:
 - plain_regs: a scalar, or a pointer to arena: it may point anywhere,
 - value_regs: that, or a pointer to a map value that has only scalars and
   is not read-only: only the lowest and the highest offset count,
 - the context: a load through it can give a pointer to the packet.

  ALU op other than 64-bit add, sub, mov     scalar, addr_space_cast too
  64-bit add, sub                            what both operands are
  load through a scalar or a map value       scalar
  load through ctx, or of less than 8 bytes  is_valid_access() tells
  spill and fill through r10, r11            what the prog stored
  helper, kfunc                              from the prototype
  static subprog                             true at all calls, all exits
  global subprog, callback, target of callx  only the context

test_cls_redirect.c has the context in a struct. In a subprog c->skb comes
from the stack of the caller: nothing is known about it.
  data = c->skb->data; if (data + (v6 ? 40 : 20) > end) return 0;
So 4 bytes at the offset of data can be a pointer. Else 20 and 40 are
joined: "invalid access to packet". loads_pkt_end notes a subprog that
loads data_end, for the main pass. void * from a kfunc is a scalar, but not
with KF_RET_NULL and not from bpf_rdonly_cast(), bpf_cast_to_kern_ctx().
  p = bpf_core_cast(t, struct task_struct); x = z ? p->pid : p->tgid;
is one 'r0 += r1' with 1856 or 1860. Joined:
"invalid variable offset access into struct task_struct".

The result only selects the states that the main pass does not join:
a mistake rejects a valid prog or makes more states. No unsafe prog loads.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   3 +
 kernel/bpf/flow.c            | 314 +++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |   7 +
 3 files changed, 324 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 332701cdf3ee..700f5bb41f83 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1100,6 +1100,8 @@ struct bpf_subprog_info {
 	 * a hidden pointer to the result, a pair in two registers, no dead arguments.
 	 */
 	bool stack_args_from_code: 1;
+	/* it loads the end of the packet from the context */
+	bool loads_pkt_end: 1;
 	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
 	bool bail_unwinds: 1;
 	u8 code_stack_args;
@@ -2113,6 +2115,7 @@ static inline bool bpf_is_kfunc_sleepable(struct bpf_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_SLEEPABLE;
 }
 bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta);
+bool bpf_is_cast_kfunc(const struct bpf_call_arg_meta *meta);
 struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem);
 int bpf_copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off);
 bool bpf_insn_is_cond_jump(u8 code);
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 0f5acdf1cb94..f039cc949854 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -514,3 +514,317 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)
 	kvfree(seen);
 	return ret;
 }
+
+/*
+ * The places of a function that hold a value: the registers, the arguments on the stack that it
+ * got and those that it hands over, the slots of the stack that struct flow_need has a bit for.
+ */
+enum {
+	FLOW_ARG_IN	= 16,
+	FLOW_ARG_OUT	= FLOW_ARG_IN + MAX_STACK_ARG_SLOTS,
+	FLOW_SLOT	= 64,
+	FLOW_PLACES	= 128,
+};
+static_assert(FLOW_ARG_OUT + MAX_STACK_ARG_SLOTS <= FLOW_SLOT);
+
+enum flow_kind {
+	FLOW_IS_PLAIN,	/* a number or a pointer to arena: nothing checks where it points to */
+	FLOW_IS_VALUE,	/* that, or a pointer to a map value that holds nothing but numbers */
+	FLOW_IS_CTX,	/* the context, as the prog or a global function got it */
+	FLOW_KINDS,
+	FLOW_IS_OTHER = FLOW_KINDS,
+};
+
+/* What the places hold for sure in front of an insn */
+struct flow_kinds {
+	unsigned long is[FLOW_KINDS][BITS_TO_LONGS(FLOW_PLACES)];
+};
+
+static void flow_kind_set(struct flow_kinds *k, u32 place, enum flow_kind kind)
+{
+	int i;
+
+	for (i = 0; i < FLOW_KINDS; i++)
+		__assign_bit(place, k->is[i], i == kind);
+	if (kind == FLOW_IS_PLAIN)
+		__set_bit(place, k->is[FLOW_IS_VALUE]);
+}
+
+static void flow_kind_copy(struct flow_kinds *to, u32 place, const struct flow_kinds *from,
+			   u32 from_place)
+{
+	int i;
+
+	for (i = 0; i < FLOW_KINDS; i++)
+		__assign_bit(place, to->is[i], test_bit(from_place, from->is[i]));
+}
+
+/* What holds on both ways to an insn. Returns true if that is less than before. */
+static bool flow_kinds_meet(struct flow_kinds *to, const struct flow_kinds *from)
+{
+	unsigned long lost = 0;
+	int i, j;
+
+	for (i = 0; i < FLOW_KINDS; i++) {
+		for (j = 0; j < ARRAY_SIZE(to->is[i]); j++) {
+			lost |= to->is[i][j] & ~from->is[i][j];
+			to->is[i][j] &= from->is[i][j];
+		}
+	}
+	return lost;
+}
+
+/* The first insn of a function that is called from @k */
+static bool flow_kinds_enter(struct flow_kinds *to, const struct flow_kinds *k)
+{
+	struct flow_kinds e = {};
+	int i;
+
+	/* r12 points to arena. The rest is not set. */
+	bitmap_fill(e.is[FLOW_IS_PLAIN], FLOW_PLACES);
+	bitmap_fill(e.is[FLOW_IS_VALUE], FLOW_PLACES);
+	flow_kind_set(&e, BPF_REG_FP, FLOW_IS_OTHER);
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+		flow_kind_copy(&e, i, k, i);
+	for (i = 0; i < MAX_STACK_ARG_SLOTS; i++)
+		flow_kind_copy(&e, FLOW_ARG_IN + i, k, FLOW_ARG_OUT + i);
+	return flow_kinds_meet(to, &e);
+}
+
+/* The place that @insn loads from or stores to through r10 or r11, -1 if it's something else */
+static int flow_mem_place(const struct bpf_insn *insn, u8 base)
+{
+	int n = (abs(insn->off) - 1) / BPF_REG_SIZE;
+
+	if (!insn->off)
+		return -1;
+	if (base == BPF_REG_FP && insn->off < 0 && n < 64)
+		return FLOW_SLOT + n;
+	if (base == BPF_REG_PARAMS && n < MAX_STACK_ARG_SLOTS)
+		return (insn->off < 0 ? FLOW_ARG_OUT : FLOW_ARG_IN) + n;
+	return -1;
+}
+
+static enum flow_kind flow_map_value_kind(const struct bpf_map *map)
+{
+	if (map->map_type == BPF_MAP_TYPE_ARENA)
+		return FLOW_IS_PLAIN;
+	/* what is at a known offset of a read-only map is known: a number, a string, a function */
+	if (map->map_flags & BPF_F_RDONLY_PROG)
+		return FLOW_IS_OTHER;
+	return IS_ERR_OR_NULL(map->record) && !map->inner_map_meta ? FLOW_IS_VALUE : FLOW_IS_OTHER;
+}
+
+/* What @insn loads through the register @base */
+static enum flow_kind flow_loaded_kind(struct bpf_verifier_env *env, const struct flow_kinds *k,
+				       const struct bpf_insn *insn, u8 base)
+{
+	struct bpf_insn_access_aux info = { .reg_type = SCALAR_VALUE };
+	int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
+	bool tstamp = env->prog->tstamp_type_access, valid;
+
+	/* a number that is good for a load is an address in arena */
+	if (test_bit(base, k->is[FLOW_IS_VALUE]))
+		return FLOW_IS_PLAIN;
+	if (size == BPF_REG_SIZE && !test_bit(base, k->is[FLOW_IS_CTX]))
+		return FLOW_IS_OTHER;
+	valid = env->ops->is_valid_access &&
+		env->ops->is_valid_access(insn->off, size, BPF_READ, env->prog, &info);
+	/* it notes what the prog reads, and this is a question */
+	env->prog->tstamp_type_access = tstamp;
+	if (valid && info.reg_type == PTR_TO_PACKET_END)
+		bpf_find_containing_subprog(env, insn - env->prog->insnsi)->loads_pkt_end = true;
+	/*
+	 * Only the context has pointers that are shorter than a register. A base that nothing is
+	 * known about can be the context: it came out of a struct, or only some paths have it.
+	 */
+	if (size < BPF_REG_SIZE)
+		return valid && info.reg_type != SCALAR_VALUE ? FLOW_IS_OTHER : FLOW_IS_PLAIN;
+	return valid && info.reg_type == SCALAR_VALUE ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
+}
+
+/* What a helper or a kfunc returns */
+static enum flow_kind flow_returned_kind(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	const struct bpf_func_proto *fn;
+	struct bpf_call_arg_meta meta;
+	const struct btf_type *t;
+
+	if (bpf_pseudo_kfunc_call(insn)) {
+		if (bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) < 0)
+			return FLOW_IS_OTHER;
+		t = btf_type_skip_modifiers(meta.btf, meta.func_proto->type, NULL);
+		if (!btf_type_is_ptr(t))
+			return FLOW_IS_PLAIN;
+		/* void * is a number, unless the verifier knows the kfunc: it may fail, or casts */
+		t = btf_type_skip_modifiers(meta.btf, t->type, NULL);
+		return btf_type_is_void(t) && !(meta.kfunc_flags & KF_RET_NULL) &&
+		       !bpf_is_cast_kfunc(&meta) ? FLOW_IS_PLAIN : FLOW_IS_OTHER;
+	}
+	if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
+		return FLOW_IS_OTHER;
+	if (fn->ret_type == RET_INTEGER || fn->ret_type == RET_VOID)
+		return FLOW_IS_PLAIN;
+	if (base_type(fn->ret_type) == RET_PTR_TO_MAP_VALUE &&
+	    (aux->const_reg_map_mask & BIT(BPF_REG_1)))
+		return flow_map_value_kind(env->used_maps[aux->const_reg_vals[BPF_REG_1]]);
+	return FLOW_IS_OTHER;
+}
+
+/*
+ * @k is what holds in front of the insn at @idx. Make it what holds behind it.
+ * @ret: the registers that are FLOW_IS_PLAIN at all exits of a subprog.
+ * Returns true if less is known than before at the entry or about the exits of a subprog.
+ */
+static bool flow_kinds_step(struct bpf_verifier_env *env, struct flow_kinds *kinds, u16 *ret,
+			    struct flow_kinds *k, int idx)
+{
+	struct bpf_subprog_info *sub = env->subprog_info;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+	u8 dst = insn->dst_reg, src = insn->src_reg;
+	bool wide = BPF_SIZE(insn->code) == BPF_DW && BPF_MODE(insn->code) == BPF_MEM &&
+		    !(insn->off % BPF_REG_SIZE);
+	bool lost = false;
+	u16 regs = U16_MAX;
+	int i, s;
+
+	if (class == BPF_ALU64 && (op == BPF_ADD || op == BPF_SUB)) {
+		__clear_bit(dst, k->is[FLOW_IS_CTX]);
+		for (i = 0; BPF_SRC(insn->code) == BPF_X && i < FLOW_IS_CTX; i++)
+			if (!test_bit(src, k->is[i]))
+				__clear_bit(dst, k->is[i]);
+	} else if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && !insn->off) {
+		flow_kind_copy(k, dst, k, src);
+	} else if (class == BPF_ALU || class == BPF_ALU64) {
+		/*
+		 * addr_space_cast too. It leaves a pointer that is not in arena as it is, and
+		 * a compiler casts what it cannot know. Nearly all of that is in arena.
+		 */
+		flow_kind_set(k, dst, FLOW_IS_PLAIN);
+	} else if (class == BPF_LDX) {
+		s = flow_mem_place(insn, src);
+		if (wide && s >= 0)
+			flow_kind_copy(k, dst, k, s);
+		else
+			flow_kind_set(k, dst, flow_loaded_kind(env, k, insn, src));
+	} else if (class == BPF_STX || class == BPF_ST) {
+		i = bpf_atomic_load_reg(insn);
+		if (i >= 0)
+			flow_kind_set(k, i, flow_loaded_kind(env, k, insn,
+						insn->imm == BPF_LOAD_ACQ ? src : dst));
+		s = flow_mem_place(insn, dst);
+		if (class == BPF_STX && wide && s >= 0)
+			flow_kind_copy(k, s, k, src);
+		else if (s >= 0)
+			flow_kind_set(k, s, FLOW_IS_PLAIN);
+	} else if (bpf_is_ldimm64(insn)) {
+		if (src == BPF_PSEUDO_MAP_VALUE || src == BPF_PSEUDO_MAP_IDX_VALUE)
+			flow_kind_set(k, dst, flow_map_value_kind(
+					env->used_maps[env->insn_aux_data[idx].map_index]));
+		else
+			flow_kind_set(k, dst, src ? FLOW_IS_OTHER : FLOW_IS_PLAIN);
+	} else if (insn->code == (BPF_JMP | BPF_EXIT) && !bpf_is_unwind_exit(insn)) {
+		s = bpf_find_containing_subprog(env, idx) - sub;
+		lost = ret[s] & ~k->is[FLOW_IS_PLAIN][0];
+		ret[s] &= k->is[FLOW_IS_PLAIN][0];
+	} else if (class == BPF_LD || (class == BPF_JMP && op == BPF_CALL)) {
+		s = bpf_pseudo_call(insn) ? bpf_find_subprog(env, idx + insn->imm + 1) : -1;
+		if (s >= 0 && !bpf_subprog_is_global(env, s)) {
+			lost = flow_kinds_enter(&kinds[sub[s].start], k);
+			regs = ret[s];
+		} else if (bpf_is_callx(insn)) {
+			regs = 0;
+		}
+		/* a subprog may return a pair in R0 and R2, the rest is not set */
+		regs |= ~(BIT(BPF_REG_0) | BIT(BPF_REG_2));
+		for (i = BPF_REG_0; i <= BPF_REG_5; i++)
+			flow_kind_set(k, i, regs & BIT(i) ? FLOW_IS_PLAIN : FLOW_IS_OTHER);
+		for (i = 0; i < MAX_STACK_ARG_SLOTS; i++)
+			flow_kind_set(k, FLOW_ARG_OUT + i, FLOW_IS_PLAIN);
+		if (bpf_helper_call(insn) || bpf_pseudo_kfunc_call(insn))
+			flow_kind_set(k, BPF_REG_0, flow_returned_kind(env, idx));
+	}
+	return lost;
+}
+
+/*
+ * Find the registers that hold a number or a pointer to arena, and those that point to the value
+ * of a map that holds nothing but numbers at worst, whichever way the prog gets to an insn:
+ * plain_regs and value_regs of struct bpf_insn_aux_data. The others may hold any pointer.
+ *
+ * Forwards, from "all is known" down, until nothing changes. Arguments go from the callers to
+ * the callees and results the other way, so the subprogs are looked at in both orders in turn.
+ *
+ * Nothing depends on it but which states are kept apart. A pointer that is taken for
+ * a number costs a rule. A number that is taken for a pointer costs states. So it's fine to
+ * miss a store through a pointer to the stack, and to take what callx calls for unknown.
+ *
+ * The edges that the verifier adds, see bpf_head_guard_target(): may_goto at the head of a loop
+ * takes what holds there to its target. A subprog that is out of budget goes to an exit with 0
+ * in place of the result, a number.
+ */
+int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *sub = env->subprog_info;
+	int len = env->prog->len, cnt = env->subprog_cnt, i, j, n, s, t, idx, err = 0;
+	struct flow_kinds *kinds, k = {};
+	struct bpf_iarray *succ;
+	bool lost = true;
+	u16 *ret;
+
+	kinds = kvmalloc_array(len, sizeof(*kinds), GFP_KERNEL_ACCOUNT);
+	ret = kvmalloc_array(env->subprog_cnt, sizeof(*ret), GFP_KERNEL_ACCOUNT);
+	if (!kinds || !ret) {
+		kvfree(kinds);
+		kvfree(ret);
+		return -ENOMEM;
+	}
+	memset(kinds, 0xff, len * sizeof(*kinds));
+	memset(ret, 0xff, env->subprog_cnt * sizeof(*ret));
+
+	/* what the kernel, a helper, callx or another prog calls: only the context is known */
+	for (s = 0; s < env->subprog_cnt; s++) {
+		if (s && !bpf_subprog_is_global(env, s) && !sub[s].addr_taken)
+			continue;
+		flow_kinds_enter(&kinds[sub[s].start], &k);
+		for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++)
+			if (sub[s].args[i].arg_type == ARG_PTR_TO_CTX ||
+			    (!s && !i && env->prog->type != BPF_PROG_TYPE_EXT))
+				flow_kind_set(&kinds[sub[s].start], BPF_REG_1 + i, FLOW_IS_CTX);
+	}
+
+	for (n = 0; lost; n++) {
+		lost = false;
+		/* a chain of calls takes a round for each link */
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		for (j = 0; j < cnt; j++) {
+			/* it has callees first */
+			s = env->subprog_topo_order[n & 1 ? j : cnt - 1 - j];
+			for (i = sub[s + 1].postorder_start; i-- > sub[s].postorder_start;) {
+				idx = env->cfg.insn_postorder[i];
+				k = kinds[idx];
+				lost |= flow_kinds_step(env, kinds, ret, &k, idx);
+				succ = bpf_insn_successors(env, idx);
+				iarray_for_each(t, succ)
+					lost |= flow_kinds_meet(&kinds[t], &k);
+				t = bpf_head_guard_target(env, idx);
+				if (t >= 0)
+					lost |= flow_kinds_meet(&kinds[t], &kinds[idx]);
+			}
+		}
+	}
+	for (i = 0; i < len; i++) {
+		env->insn_aux_data[i].plain_regs = kinds[i].is[FLOW_IS_PLAIN][0];
+		env->insn_aux_data[i].value_regs = kinds[i].is[FLOW_IS_VALUE][0];
+	}
+	kvfree(kinds);
+	kvfree(ret);
+	return err;
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ea6dd39982f2..d42c67d6d70b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14110,6 +14110,13 @@ static bool is_bpf_cast_to_kern_ctx_kfunc(const struct bpf_call_arg_meta *meta)
 	return is_kfunc_call(meta, special_kfunc_list[KF_bpf_cast_to_kern_ctx]);
 }
 
+/* void * in the prototype. What it returns points to an object of the kernel. */
+bool bpf_is_cast_kfunc(const struct bpf_call_arg_meta *meta)
+{
+	return is_kfunc_call(meta, special_kfunc_list[KF_bpf_cast_to_kern_ctx]) ||
+	       is_kfunc_call(meta, special_kfunc_list[KF_bpf_rdonly_cast]);
+}
+
 static bool is_bpf_dynptr_clone_kfunc(const struct bpf_call_arg_meta *meta)
 {
 	return is_kfunc_call(meta, special_kfunc_list[KF_bpf_dynptr_clone]);
-- 
2.55.0

Re: [PATCH bpf-next v4 09/46] bpf: Find the registers that hold a number or a pointer to arena

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] bpf: missing propagation of loads_pkt_end to callers in
  bpf_flow_compute_kinds()
--

commit 56842cec67f7566ca6cea05c44634e53a6e064fc
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Find the registers that hold a number or a pointer to arena

This commit adds bpf_flow_compute_kinds() to track which registers and stack
slots hold plain scalars, map value pointers, or the context before each
instruction, and records subprograms that load the packet end pointer in
sub->loads_pkt_end.

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> index 0f5acdf1cb946..f039cc9498542 100644
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c
> @@ -514,3 +514,317 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)

[ ... ]

> +/* What @insn loads through the register @base */
> +static enum flow_kind flow_loaded_kind(struct bpf_verifier_env *env, const struct flow_kinds *k,
> +				       const struct bpf_insn *insn, u8 base)
> +{
> +	struct bpf_insn_access_aux info = { .reg_type = SCALAR_VALUE };
> +	int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
> +	bool tstamp = env->prog->tstamp_type_access, valid;
> +
> +	/* a number that is good for a load is an address in arena */
> +	if (test_bit(base, k->is[FLOW_IS_VALUE]))
> +		return FLOW_IS_PLAIN;
> +	if (size == BPF_REG_SIZE && !test_bit(base, k->is[FLOW_IS_CTX]))
> +		return FLOW_IS_OTHER;
> +	valid = env->ops->is_valid_access &&
> +		env->ops->is_valid_access(insn->off, size, BPF_READ, env->prog, &info);
> +	/* it notes what the prog reads, and this is a question */
> +	env->prog->tstamp_type_access = tstamp;
> +	if (valid && info.reg_type == PTR_TO_PACKET_END)
> +		bpf_find_containing_subprog(env, insn - env->prog->insnsi)->loads_pkt_end = true;

[ ... ]

> +	} else if (class == BPF_LD || (class == BPF_JMP && op == BPF_CALL)) {
> +		s = bpf_pseudo_call(insn) ? bpf_find_subprog(env, idx + insn->imm + 1) : -1;
> +		if (s >= 0 && !bpf_subprog_is_global(env, s)) {
> +			lost = flow_kinds_enter(&kinds[sub[s].start], k);
> +			regs = ret[s];
> +		} else if (bpf_is_callx(insn)) {
> +			regs = 0;
> +		}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v3 [1], does loads_pkt_end need to
be propagated from a static callee to its caller?

flow_loaded_kind() sets loads_pkt_end only on the subprog that directly
contains the ctx->data_end load, and neither flow_kinds_step() nor
merge_callee_effects() passes it up the call graph.  In commit "bpf: Verify
static subprogs on their own", func_is_on_its_own() and arg_needs_caller()
check sub->loads_pkt_end on the immediate callee to decide whether a static
subprog that receives a packet pointer must be verified in its caller's
frames.

If a caller passes ctx and a packet pointer to a static subprog that
delegates the ctx->data_end load and bounds check to a nested static
subprog, the outer subprog has loads_pkt_end == false and is verified
without its caller's frame.  find_good_pkt_pointers() in the inner subprog
then never widens the packet pointer in the outer caller's frame, causing
valid packet accesses after the call to be rejected.

[1] https://lore.kernel.org/r/c5cd652a5014ef6157278403838481eb2f4a8c5add5e6de5a2837b3121f30074@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=9

[PATCH bpf-next v4 10/46] bpf: Find the registers and stack slots that later checks depend on

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The join of two constants is not a constant, and some checks need one. The
two states meet before the insn that checks. Thus the verifier must know,
where they meet, that a later check depends on a register or a stack slot.
A table per kind of check, FLOW_NEED_*, tells the rules that refuse a join:

ARG        an argument that must be a constant
    if (x) return bpf_rdonly_cast(0, ID); return bpf_rdonly_cast(0, 0);
  The compiler makes one call of it: "R2 must be a known constant" in
  selftest mem_rdonly_untrusted/mixed_mem_type.
STACK_OFF  an add to a pointer to the stack
    if (... || depth >= BT_MAXLVL_PRINT) return 0; ind = stack[depth];
  The compiler tests one copy of depth and adds another. bt_print() of scx:
  "invalid variable-offset read from stack R3"
PTR_OFF    an add to a pointer to the packet, the context, a kernel object,
           a map value that is read-only or has a timer, a lock, a pointer
    if (data + hdr_len + inner_ip_offset > data_end) return TC_ACT_UNSPEC;
    inner_protocol = inner_iph->protocol;
  With [20, 40] as the offset the test tells nothing about data + hdr_len:
  "invalid access to packet, off=48 size=1" in katran/tcdecapinfo.
JUMP       a jump that compares with a constant, an argument or a return
           value that the verifier compares with a range
    if (ret < 0) ..; to_sport = state->to_sport;
  ret is a negative constant where state can be NULL. As a part of a range
  it passes the test: "R3 invalid mem access 'map_value_or_null'" in
  tail_handle_snat_fwd_ipv4 of Cilium.

Three more tables are only for bpf_flow_is_needed(): does a check depend
on a value that a counted loop changes and that has no closed form? If not,
the value is widened at the loop head, and the verifier has no reason to
walk the iterations one by one:

    for (int i = 0; i < STACK_MAX_LEN; ++i) {
      if (!frame_ptr || !get_frame_data(frame_ptr, pidData, &frame, &sym))

Each iteration fills the 24 stack slots of sym anew, and no check looks at
them. pyperf600_nounroll takes 1281 insns, 154K with a walk of all 600.

VALUE_OFF  PTR_OFF, STACK_OFF, and an offset in each other map value
BASE       the address of a load or a store
JUMP_ALU   JUMP, and the values that the tested value is calculated from

flow_compute_need_const() goes backward through the control flow graph
until nothing changes, like the calculation of live registers. A use that
it misses can reject a valid prog. It cannot accept an unsafe prog.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 324 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 324 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index f039cc949854..f5a91552b090 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -515,6 +515,74 @@ int bpf_flow_place_guards(struct bpf_verifier_env *env)
 	return ret;
 }
 
+/* The slot that @insn spills to or fills from, if it's one that struct flow_need has a bit for */
+static int flow_fp_slot(const struct bpf_insn *insn, u8 base)
+{
+	u8 mode = BPF_MODE(insn->code);
+
+	if (base != BPF_REG_FP || (mode != BPF_MEM && mode != BPF_MEMSX) ||
+	    insn->off >= 0 || insn->off < -64 * BPF_REG_SIZE || insn->off % BPF_REG_SIZE)
+		return -1;
+	return -insn->off / BPF_REG_SIZE - 1;
+}
+
+/*
+ * A number is added to a pointer to the stack. A store at an offset that is not known
+ * forgets what the slots it may hit hold, pointers above all.
+ */
+static u16 flow_stack_off_regs(struct bpf_verifier_env *env, int idx)
+{
+	u16 ptrs = env->insn_aux_data[idx].stack_ptrs;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 op = BPF_OP(insn->code);
+
+	if (BPF_CLASS(insn->code) != BPF_ALU64 || BPF_SRC(insn->code) != BPF_X ||
+	    (op != BPF_ADD && op != BPF_SUB))
+		return 0;
+	if ((ptrs & BIT(insn->dst_reg)) && !(ptrs & BIT(insn->src_reg)))
+		return BIT(insn->src_reg);
+	if (op == BPF_ADD && (ptrs & BIT(insn->src_reg)) && !(ptrs & BIT(insn->dst_reg)))
+		return BIT(insn->dst_reg);
+	return 0;
+}
+
+/*
+ * A jump that compares a register with a constant, which may be in the other register.
+ * A call does the same: the verifier compares a size, or what bpf_set_retval() takes,
+ * with what is allowed. So does the exit of the prog, see flow_compute_need_const().
+ */
+static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	u16 consts = aux->const_reg_mask | aux->const_reg_wide_mask;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+	if (bpf_helper_call(insn) || bpf_pseudo_kfunc_call(insn))
+		return bpf_call_checked_arg_regs(env, insn);
+	if ((class != BPF_JMP && class != BPF_JMP32) ||
+	    op == BPF_JA || op == BPF_CALL || op == BPF_EXIT || op == BPF_JCOND)
+		return 0;
+	if (BPF_SRC(insn->code) == BPF_K || (consts & BIT(insn->src_reg)))
+		return BIT(insn->dst_reg);
+	return consts & BIT(insn->dst_reg) ? BIT(insn->src_reg) : 0;
+}
+
+/* With or without sign extension */
+static bool flow_is_copy(const struct bpf_insn *insn)
+{
+	return BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_X && !insn->imm &&
+	       (!insn->off || insn->off == 8 || insn->off == 16 || insn->off == 32);
+}
+
+static bool flow_is_shift(const struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+	return (class == BPF_ALU || class == BPF_ALU64) &&
+	       (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH);
+}
+
 /*
  * The places of a function that hold a value: the registers, the arguments on the stack that it
  * got and those that it hands over, the slots of the stack that struct flow_need has a bit for.
@@ -828,3 +896,259 @@ int bpf_flow_compute_kinds(struct bpf_verifier_env *env)
 	kvfree(ret);
 	return err;
 }
+
+/*
+ * A number is added to what may be a pointer whose offset the verifier looks at:
+ * - An object of the kernel, the context, a socket take a known offset only.
+ * - The packet: the offset was compared with the end of the packet before the states met.
+ * - The value of a map that has a timer, a lock, a pointer in the middle.
+ * - The value of a read-only map: what is loaded from a known offset is known.
+ * Not a value that holds nothing but numbers. Only the lowest and the highest offset
+ * count there, and the join of two constants has both. @any_value: that one too.
+ */
+static u16 flow_ptr_off_regs(struct bpf_verifier_env *env, int idx, bool any_value)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u16 both = BIT(insn->dst_reg) | BIT(insn->src_reg), ptrs;
+	u8 op = BPF_OP(insn->code);
+
+	if (BPF_CLASS(insn->code) != BPF_ALU64 || BPF_SRC(insn->code) != BPF_X ||
+	    (op != BPF_ADD && op != BPF_SUB) || (aux->stack_ptrs & both))
+		return 0;
+	ptrs = both & ~(any_value ? aux->plain_regs : aux->value_regs);
+	if (op == BPF_SUB)
+		ptrs &= BIT(insn->dst_reg);
+	/* the one that may be the pointer makes the other one the number */
+	return ptrs && ptrs != both ? both ^ ptrs : ptrs;
+}
+
+/* The base of a load or a store, unless it's a number or a pointer to arena for sure */
+static u16 flow_base_regs(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code);
+	u16 regs = 0;
+
+	if (class == BPF_LDX || bpf_atomic_is_load_acq(insn))
+		regs = BIT(insn->src_reg);
+	else if (class == BPF_ST || class == BPF_STX)
+		regs = BIT(insn->dst_reg);
+	return regs & ~env->insn_aux_data[idx].plain_regs & GENMASK(BPF_REG_9, BPF_REG_0);
+}
+
+/*
+ * Some arguments of kfuncs and helpers have to be known constants: the id of a type,
+ * the size of what is reserved in a ring buffer, the size of a slice of a dynptr.
+ * A compiler makes one call of
+ *	if (x) p = bpf_rdonly_cast(q, ID); else p = bpf_rdonly_cast(q, 0);
+ * and one store of
+ *	if (x) a[n++] = p; else b[m++] = p;
+ * where a and b are on the stack.
+ * and one load of
+ *	x = ipv6 ? *(u16 *)(pkt + 54) : *(u16 *)(pkt + 34);	y = z ? task->pid : task->tgid;
+ * The join of two constants is not a constant. Find the registers and the slots of the stack
+ * that end up in such arguments, or in offsets in the stack or from a pointer,
+ * like live registers are found: backwards, until nothing changes.
+ * States in which they differ are not joined.
+ * Nothing depends on it but what is known: it's fine to miss a store through a pointer.
+ *
+ * FLOW_NEED_JUMP: those that a jump ahead compares with a constant, see flow_join_makes_up().
+ * As they are, or extended: without 32-bit insns a compiler tests
+ *	r7 = r8; r7 <<= 32; r7 >>= 32;
+ * and goes on with r8.
+ * A slot that is filled through another pointer to the stack than r10 is not one of them, and
+ * flow_null_differs() counts on that.
+ *
+ * The last three kinds keep no states apart. They are for bpf_flow_is_needed(), which asks
+ * whether anything ahead looks at a number that a loop changes from iteration to iteration.
+ * FLOW_NEED_VALUE_OFF: FLOW_NEED_PTR_OFF, and the offset in a map value that holds nothing but
+ * numbers. It has to be in the value. It can be an argument, or what a function returns:
+ *	return v->hist[count_bits(x)];
+ * So can an offset in the stack. FLOW_NEED_STACK_OFF doesn't follow it through a return, so this
+ * table has it too.
+ * FLOW_NEED_BASE: the base of a load or a store, as it is, behind a cast, moved by a constant:
+ *	p = NULL; for (i = 0; i < 3; i++) { if (p) n += p->len; p = next(); }
+ * FLOW_NEED_JUMP_ALU: FLOW_NEED_JUMP, and what the tested number is made of step by step:
+ *	for (i = 0; i < 3; i++) x /= 10; if (x < 8) ...
+ * In x op= y that is x. y is not followed.
+ * What is stored through a pointer to the stack other than r10 counts as tested. It's a result
+ * for the caller, and no table follows it from the store to the fill:
+ *	count_bits(x, &n); return v->hist[n];
+ * These two go through callx, to every function whose address is taken.
+ */
+static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_need **pneed,
+				   enum flow_need_kind kind)
+{
+	struct bpf_insn *insns = env->prog->insnsi, *insn;
+	struct flow_need in, out, *need;
+	struct bpf_iarray *succ;
+	bool jump = kind == FLOW_NEED_JUMP || kind == FLOW_NEED_JUMP_ALU;
+	bool callx = kind == FLOW_NEED_BASE || kind == FLOW_NEED_JUMP_ALU;
+	bool changed = true, any = false, keep, from_src, reg;
+	int i, s, idx, slot, err = 0;
+	u16 *gen, callx_regs = 0, pair = BIT(BPF_REG_0) | BIT(BPF_REG_2);
+	u8 *ret, class, op;
+
+	need = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
+	gen = kvcalloc(env->prog->len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+	/* the callers need what the subprog returns: r0, and r2 of a pair */
+	ret = kvcalloc(env->subprog_cnt, sizeof(u8), GFP_KERNEL_ACCOUNT);
+	*pneed = need;
+	if (!need || !gen || !ret) {
+		kvfree(gen);
+		kvfree(ret);
+		return -ENOMEM;
+	}
+	for (i = 0; i < env->prog->len; i++) {
+		if (jump)
+			gen[i] = flow_tested_regs(env, i);
+		else if (kind == FLOW_NEED_STACK_OFF)
+			gen[i] = flow_stack_off_regs(env, i);
+		else if (kind == FLOW_NEED_PTR_OFF || kind == FLOW_NEED_VALUE_OFF)
+			gen[i] = flow_ptr_off_regs(env, i, kind == FLOW_NEED_VALUE_OFF);
+		else if (kind == FLOW_NEED_BASE)
+			gen[i] = flow_base_regs(env, i);
+		else if (bpf_helper_call(&insns[i]) || bpf_pseudo_kfunc_call(&insns[i]))
+			gen[i] = bpf_call_const_arg_regs(env, &insns[i]);
+		if (kind == FLOW_NEED_JUMP_ALU && BPF_CLASS(insns[i].code) == BPF_STX &&
+		    insns[i].dst_reg != BPF_REG_FP &&
+		    (env->insn_aux_data[i].stack_ptrs & BIT(insns[i].dst_reg)))
+			gen[i] = BIT(insns[i].src_reg);
+		if (kind == FLOW_NEED_VALUE_OFF)
+			gen[i] |= flow_stack_off_regs(env, i);
+		any |= gen[i];
+	}
+	if (jump) {
+		/* the verifier compares what the prog and its callbacks return with a range */
+		ret[0] = BIT(BPF_REG_0);
+		for (i = 0; i < env->prog->len; i++) {
+			if (!bpf_pseudo_func(&insns[i]))
+				continue;
+			s = bpf_find_subprog(env, i + insns[i].imm + 1);
+			if (s >= 0)
+				ret[s] = BIT(BPF_REG_0);
+		}
+		any = true;
+	}
+	while (any && changed) {
+		changed = false;
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		for (s = 0; callx && s < env->subprog_cnt; s++)
+			if (env->subprog_info[s].addr_taken)
+				callx_regs |= need[env->subprog_info[s].start].regs;
+		for (i = 0; i < env->cfg.cur_postorder; i++) {
+			idx = env->cfg.insn_postorder[i];
+			insn = &insns[idx];
+			class = BPF_CLASS(insn->code);
+			succ = bpf_insn_successors(env, idx);
+			out.regs = 0;
+			out.slots = 0;
+			for (s = 0; s < succ->cnt; s++) {
+				out.regs |= need[succ->items[s]].regs;
+				out.slots |= need[succ->items[s]].slots;
+			}
+			in = out;
+			if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) {
+				in.regs &= ~GENMASK(BPF_REG_5, BPF_REG_0);
+				if (bpf_is_callx(insn))
+					in.regs |= callx_regs & GENMASK(BPF_REG_5, BPF_REG_1);
+				s = idx + insn->imm + 1;
+				if (bpf_pseudo_call(insn) &&
+				    !bpf_subprog_is_global(env, bpf_find_subprog(env, s))) {
+					if (kind != FLOW_NEED_PTR_OFF)
+						in.regs |= need[s].regs &
+							   GENMASK(BPF_REG_5, BPF_REG_1);
+					s = bpf_find_subprog(env, s);
+					/*
+					 * Not for an index where states are kept apart: that is
+					 * one of many numbers.
+					 */
+					if ((kind == FLOW_NEED_ARG || kind == FLOW_NEED_VALUE_OFF ||
+					     jump) &&
+					    (out.regs & pair & ~ret[s])) {
+						ret[s] |= out.regs & pair;
+						changed = true;
+					}
+				}
+			} else if (insn->code == (BPF_JMP | BPF_EXIT)) {
+				s = bpf_find_containing_subprog(env, idx) - env->subprog_info;
+				in.regs |= ret[s];
+			} else if (class == BPF_LDX || class == BPF_LD) {
+				slot = class == BPF_LDX ? flow_fp_slot(insn, insn->src_reg) : -1;
+				if (slot >= 0 && (out.regs & BIT(insn->dst_reg)))
+					in.slots |= BIT_ULL(slot);
+				in.regs &= ~BIT(insn->dst_reg);
+			} else if (class == BPF_STX || class == BPF_ST) {
+				slot = flow_fp_slot(insn, insn->dst_reg);
+				if (slot >= 0 && (out.slots & BIT_ULL(slot))) {
+					in.slots &= ~BIT_ULL(slot);
+					if (class == BPF_STX)
+						in.regs |= BIT(insn->src_reg);
+				}
+			} else if ((class == BPF_ALU || class == BPF_ALU64) &&
+				   (out.regs & BIT(insn->dst_reg))) {
+				op = BPF_OP(insn->code);
+				reg = BPF_SRC(insn->code) == BPF_X;
+				/* the result is known when what it is made of is */
+				keep = op != BPF_MOV;
+				from_src = reg;
+				if (jump)
+					from_src = flow_is_copy(insn);
+				if (kind == FLOW_NEED_JUMP)
+					keep = flow_is_shift(insn) && !reg;
+				if (kind == FLOW_NEED_BASE) {
+					keep = !reg && (op == BPF_ADD || op == BPF_SUB);
+					from_src = reg && op == BPF_MOV;
+				}
+				if (!keep)
+					in.regs &= ~BIT(insn->dst_reg);
+				if (from_src)
+					in.regs |= BIT(insn->src_reg);
+			}
+			in.regs |= gen[idx];
+			if (in.regs != need[idx].regs || in.slots != need[idx].slots) {
+				need[idx] = in;
+				changed = true;
+			}
+		}
+	}
+	/*
+	 * An offset that is an argument: not through the call, see above, and not where the calls
+	 * meet. A function would be looked at once for each offset that it's called with.
+	 */
+	for (s = 0; kind == FLOW_NEED_PTR_OFF && s < env->subprog_cnt; s++)
+		need[env->subprog_info[s].start].regs = 0;
+	kvfree(gen);
+	kvfree(ret);
+	if (!any) {
+		kvfree(need);
+		*pneed = NULL;
+	}
+	return err;
+}
+
+/*
+ * Does anything ahead of the insn at @idx look at what @r holds there? @r is a register or a stack
+ * slot of the function as scev.c counts them. See flow_compute_need_const(): it's a guess.
+ */
+bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r)
+{
+	const struct flow_need *need;
+	int i;
+
+	/* no table has a bit for it */
+	if (r >= MAX_BPF_REG + 64)
+		return true;
+	for (i = 0; i < FLOW_NEED_KINDS; i++) {
+		need = env->flow->need[i];
+		if (need && (r < MAX_BPF_REG ? need[idx].regs & BIT(r) :
+					       need[idx].slots & BIT_ULL(r - MAX_BPF_REG)))
+			return true;
+	}
+	return false;
+}
-- 
2.55.0

[PATCH bpf-next v4 11/46] bpf: Add the sets of states at an insn and the entries of functions

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest test_verif_scale1 has these lines 90 times. jhash() is static:

  ptr = data + i;
  if (ptr + nh_off > data_end) break;
  ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++);

Today the verifier verifies jhash() again for each state at each call:
530362 insns. The main pass verifies a static subprog for the states at its
first insn, and the call waits for the states at the exits. Each call that
waits makes an entry. Its number is in the entry state and in all states
that come from it. 4095 states arrive at the 90 calls. At the first insn of
jhash() a state of a different call covers 4005 of them. Then each exit
state is for the calls of the two:

  call A  -->  entry 1  --+
                          +-->  one set: calls { A, B }, exit states
  call B  -->  entry 2  --+

Thus the entries are disjoint sets, with union and find: 50765 insns. If
they stay apart, the main pass hits its limit of states at an insn:
"more than 1024 states that cannot be joined at insn 903". The same for
pandemonium_enqueue of scx_pandemonium and 100 of the 225 progs of the
schedulers of rust-bpf.

All new states are in one tree. flow_set_pending() gives them an order:
 1. Callers are before the subprogs that they call: all calls get to
    a subprog before it is verified.
 2. Then the position of the call in each frame, then of the insn. Thus
    a callee that keeps the frames of its callers, like a callback, is
    complete before the code after the call.
 3. A walk verifies a loop iteration by iteration, as today. An iteration
    is complete before the next one:

  for (int i = 0; i < MAX_AFFINITY_CANDIDATES; i++) {
      if (*val >= nr_cpu_ids) continue;
      if (++checked >= affinity_search_online) break;

'continue' is a back edge in the middle of the loop. By position alone it
runs ahead through all iterations, and each is done again when the rest of
the one before it arrives: pandemonium_select_cpu, 130073 insns, not 26758.

bpf_flow_prepare() marks the insns where a state goes into a set, and those
where a second state can arrive. Only there the set keeps a state that the
verifier used: to keep them all doubles the peak of states of Cilium, scx.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 324 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 324 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index f5a91552b090..3996220f89f0 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1152,3 +1152,327 @@ bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r)
 	}
 	return false;
 }
+
+int bpf_flow_prepare(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_insn *insns = env->prog->insnsi;
+	/* the rest is for loops */
+	int len = env->prog->len, kinds = FLOW_NEED_VALUE_OFF, i, s, err;
+	struct bpf_iarray *succ;
+	struct bpf_flow *flow;
+	u8 *preds;
+
+	flow = kzalloc_obj(*flow, GFP_KERNEL_ACCOUNT);
+	if (!flow)
+		return -ENOMEM;
+	env->flow = flow;
+	INIT_LIST_HEAD(&flow->kept);
+	flow->pending = RB_ROOT_CACHED;
+
+	flow->pos = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	flow->rank = kvcalloc(env->subprog_cnt + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	flow->point = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+	flow->keep = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+	flow->states = kvmalloc_array(len, sizeof(*flow->states), GFP_KERNEL_ACCOUNT);
+	flow->enter = kvmalloc_array(len, sizeof(*flow->enter), GFP_KERNEL_ACCOUNT);
+	flow->nr_states = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	flow->funcs = kvcalloc(env->subprog_cnt + 1, sizeof(*flow->funcs), GFP_KERNEL_ACCOUNT);
+	preds = kvcalloc(len, 1, GFP_KERNEL_ACCOUNT);
+	if (!flow->pos || !flow->rank || !flow->point || !flow->keep || !flow->states ||
+	    !flow->enter ||
+	    !flow->nr_states || !flow->funcs || !preds) {
+		kvfree(preds);
+		return -ENOMEM;
+	}
+	for (i = 0; i < len; i++) {
+		INIT_LIST_HEAD(&flow->states[i].list);
+		INIT_LIST_HEAD(&flow->enter[i].list);
+	}
+	for (i = 0; i <= env->subprog_cnt; i++)
+		flow->funcs[i].exit_idx = -1;
+	/* from here on there is something for bpf_flow_reset() to look at */
+	flow->len = len;
+	flow->nr_funcs = env->subprog_cnt + 1;
+
+	/* subprog_topo_order[] has callees first */
+	for (i = 0; i < env->subprog_cnt; i++)
+		flow->rank[env->subprog_topo_order[i]] = env->subprog_cnt - i;
+
+	for (i = 0; i < len; i++) {
+		succ = bpf_insn_successors(env, i);
+		iarray_for_each(s, succ) {
+			if (preds[s] < 2)
+				preds[s]++;
+			if (aux[i].bb_end)
+				__set_bit(s, flow->point);
+		}
+		/* where the callees come back to */
+		if (bpf_pseudo_call(&insns[i]) || bpf_is_callx(&insns[i])) {
+			if (i + 1 < len) {
+				__set_bit(i + 1, flow->point);
+				__set_bit(i + 1, flow->keep);
+			}
+		}
+		if (aux[i].calls_callback) {
+			__set_bit(i, flow->point);
+			__set_bit(i, flow->keep);
+		}
+		if (aux[i].loop) {
+			__set_bit(i, flow->point);
+			__set_bit(i, flow->keep);
+			kinds = FLOW_NEED_KINDS;
+		}
+		if (bpf_is_ldimm64(&insns[i]))
+			i++;
+	}
+	for (i = 0; i < len; i++)
+		if (preds[i] > 1)
+			__set_bit(i, flow->keep);
+	for (i = 0; i < env->subprog_cnt; i++) {
+		__set_bit(env->subprog_info[i].start, flow->point);
+		__set_bit(env->subprog_info[i].start, flow->keep);
+	}
+	kvfree(preds);
+
+	err = flow_compute_order(env, flow);
+	if (err)
+		return err;
+	for (i = 0; !err && i < kinds; i++)
+		err = flow_compute_need_const(env, &flow->need[i], i);
+	return err;
+}
+
+static void flow_free_state(struct bpf_verifier_env *env, struct flow_state *s)
+{
+	bpf_free_verifier_state(&s->st, false);
+	kfree(s);
+	env->flow->live_states--;
+}
+
+/* What a pass leaves behind. */
+void bpf_flow_reset(struct bpf_verifier_env *env)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *s, *sn;
+	struct flow_kept *k, *kn;
+	struct flow_exit *x, *xn;
+	struct flow_entry *e;
+	int i;
+
+	if (!flow || !flow->states || !flow->enter)
+		return;
+	for (i = 0; i < flow->len; i++) {
+		list_for_each_entry_safe(s, sn, &flow->states[i].list, node) {
+			list_del(&s->node);
+			flow_free_state(env, s);
+		}
+		list_for_each_entry_safe(s, sn, &flow->enter[i].list, node) {
+			list_del(&s->node);
+			flow_free_state(env, s);
+		}
+		flow->nr_states[i] = 0;
+	}
+	flow->pending = RB_ROOT_CACHED;
+	for (i = 0; i < flow->nr_funcs && flow->funcs; i++)
+		flow->funcs[i].exit_idx = -1;
+	for (i = 1; i <= flow->entry_cnt; i++) {
+		e = flow->entries[i];
+		list_for_each_entry_safe(s, sn, &e->calls, node) {
+			bpf_free_verifier_state(&s->st, false);
+			kfree(s);
+		}
+		list_for_each_entry_safe(x, xn, &e->exits, node) {
+			bpf_free_verifier_state(&x->st, false);
+			kfree(x);
+		}
+		kfree(e);
+	}
+	list_for_each_entry_safe(k, kn, &flow->kept, node) {
+		bpf_free_verifier_state(&k->st, false);
+		kfree(k);
+	}
+	INIT_LIST_HEAD(&flow->kept);
+	bpf_free_verifier_state(&flow->tmp, false);
+	memset(&flow->tmp, 0, sizeof(flow->tmp));
+	flow->entry_cnt = 0;
+	flow->united_cnt = 0;
+}
+
+void bpf_flow_free(struct bpf_verifier_env *env)
+{
+	struct bpf_flow *flow = env->flow;
+	int i;
+
+	if (!flow)
+		return;
+	bpf_flow_reset(env);
+	kvfree(flow->pos);
+	kvfree(flow->rank);
+	for (i = 0; i < FLOW_NEED_KINDS; i++)
+		kvfree(flow->need[i]);
+	kvfree(flow->point);
+	kvfree(flow->keep);
+	kvfree(flow->states);
+	kvfree(flow->enter);
+	kvfree(flow->nr_states);
+	kvfree(flow->funcs);
+	kvfree(flow->entries);
+	kvfree(flow->united);
+	kfree(flow);
+	env->flow = NULL;
+}
+
+bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx)
+{
+	return test_bit(insn_idx, env->flow->point);
+}
+
+/* A copy of @st that lives as long as the pass. */
+struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
+					       const struct bpf_verifier_state *st)
+{
+	struct flow_kept *k = kzalloc_obj(*k, GFP_KERNEL_ACCOUNT);
+
+	if (!k)
+		return NULL;
+	list_add(&k->node, &env->flow->kept);
+	if (bpf_copy_verifier_state(&k->st, st))
+		return NULL;
+	return &k->st;
+}
+
+/*
+ * Both are in a loop whose iterations are kept apart: an iteration is done with before
+ * the next one begins. A back edge in the middle of the loop would run ahead otherwise and
+ * leave the rest of every iteration behind, see flow_earlier_trip(). The state that stands
+ * for all iterations comes last.
+ */
+static int flow_cmp_trips(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
+{
+	const struct loop_stack_entry *ea, *eb;
+	u32 i, ta, tb;
+
+	for (i = 0; i < fa->loop_stack_cnt && i < fb->loop_stack_cnt; i++) {
+		ea = &fa->loop_stack[i];
+		eb = &fb->loop_stack[i];
+		if (ea->loop_id != eb->loop_id)
+			break;
+		ta = ea->apart ? ea->trip : U32_MAX;
+		tb = eb->apart ? eb->trip : U32_MAX;
+		if (ta != tb)
+			return ta < tb ? -1 : 1;
+	}
+	return 0;
+}
+
+static bool flow_less(struct rb_node *na, const struct rb_node *nb)
+{
+	const struct flow_state *a = rb_entry(na, struct flow_state, rb);
+	const struct flow_state *b = rb_entry(nb, struct flow_state, rb);
+	int cmp;
+	u32 i;
+
+	for (i = 0; i < a->key_len && i < b->key_len; i++) {
+		/* key[i] is the place in frame[i - 1] */
+		if (i) {
+			cmp = flow_cmp_trips(a->st.frame[i - 1], b->st.frame[i - 1]);
+			if (cmp)
+				return cmp < 0;
+		}
+		if (a->key[i] != b->key[i])
+			return a->key[i] < b->key[i];
+	}
+	if (a->key_len != b->key_len)
+		return a->key_len < b->key_len;
+	return a->seq < b->seq;
+}
+
+/*
+ * Callers first, so that the calls of a function get to it before it is looked
+ * at. Then by the place of the call in every frame: what is called in the frames of
+ * its callers is done with before what follows the call. What follows the call of
+ * a function that is verified on its own goes on with every state that comes out.
+ */
+static void flow_set_pending(struct bpf_flow *flow, struct flow_state *s)
+{
+	const struct bpf_verifier_state *st = &s->st;
+	u32 i;
+
+	s->key[0] = flow->rank[st->frame[0]->subprogno];
+	/* the place in the function makes the rank of the function redundant below */
+	for (i = 0; i < st->curframe && i + 2 < ARRAY_SIZE(s->key); i++)
+		s->key[i + 1] = flow->pos[st->frame[i + 1]->callsite];
+	s->key[i + 1] = flow->pos[st->insn_idx];
+	s->key_len = i + 2;
+	s->pending = true;
+	rb_add_cached(&s->rb, &flow->pending, flow_less);
+}
+
+static void flow_unlink(struct bpf_verifier_env *env, struct flow_state *s)
+{
+	struct bpf_flow *flow = env->flow;
+
+	if (s->pending)
+		rb_erase_cached(&s->rb, &flow->pending);
+	s->pending = false;
+	list_del(&s->node);
+	flow->nr_states[s->st.insn_idx]--;
+}
+
+/* The entry that stands for the set that entry @id is in */
+static struct flow_entry *flow_entry_find(struct bpf_flow *flow, u32 id)
+{
+	struct flow_entry *e = flow->entries[id];
+
+	while (e->up != e) {
+		e->up = e->up->up;
+		e = e->up;
+	}
+	return e;
+}
+
+static int flow_entry_new(struct bpf_flow *flow)
+{
+	struct flow_entry **entries, *e;
+	u32 cap;
+
+	if (flow->entry_cnt + 1 >= flow->entry_cap) {
+		cap = max(64U, flow->entry_cap * 2);
+		entries = kvrealloc(flow->entries, cap * sizeof(*entries), GFP_KERNEL_ACCOUNT);
+		if (!entries)
+			return -ENOMEM;
+		flow->entries = entries;
+		flow->entry_cap = cap;
+	}
+	e = kmalloc_obj(*e, GFP_KERNEL_ACCOUNT);
+	if (!e)
+		return -ENOMEM;
+	e->up = e;
+	INIT_LIST_HEAD(&e->calls);
+	INIT_LIST_HEAD(&e->exits);
+	flow->entries[++flow->entry_cnt] = e;
+	return flow->entry_cnt;
+}
+
+/* It delivers states, so it is done when nothing is in the middle of that. */
+static int flow_unite(struct bpf_flow *flow, u32 a, u32 b)
+{
+	struct flow_united *u;
+	u32 cap;
+
+	if (!a || !b || flow_entry_find(flow, a) == flow_entry_find(flow, b))
+		return 0;
+	if (flow->united_cnt == flow->united_cap) {
+		cap = max(16U, flow->united_cap * 2);
+		u = kvrealloc(flow->united, cap * sizeof(*u), GFP_KERNEL_ACCOUNT);
+		if (!u)
+			return -ENOMEM;
+		flow->united = u;
+		flow->united_cap = cap;
+	}
+	u = &flow->united[flow->united_cnt++];
+	u->a = a;
+	u->b = b;
+	return 0;
+}
-- 
2.55.0

[PATCH bpf-next v4 12/46] bpf: Compare states of different iterations of a loop

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest loop1, nested_loops():

  for (j = 0; j < 300; j++)
      for (i = 0; i < j; i++) {
          ...
          sum += i * m;
      }

Scalar evolution counts both loops, but has no closed form for sum, and the
verifier checks sum: it is the return value. It walks such a loop: it
verifies the iterations one by one, each with its own values, as it does
today. A join of j=5 and j=6 ends that. Thus a state has the number of its
iteration in each loop that is walked, and flow_trips_behind() tells the
merge, the code that puts a state into a set, which states not to join.
But then the states of all iterations stay in the sets, up to the limit:

  more than 1024 states that cannot be joined at insn 10

A walk only goes forward: what follows a state of iteration N is of
iteration N or later. flow_earlier_trip() finds the states that are two or
more iterations behind, and the merge frees them. nested_loops() then has
21 states at a time and takes 9954 insns (today: 5505 and 316200).
lavd_init and qmap_init of rust-bpf fail in the same way.

Crate spin, Mutex::lock():

  while self.is_locked() { R::relax(); }

No iteration changes what the verifier sees: each ends in the state of the
one before, with a larger number. Thus in a walk of a loop without a count
a state of an earlier iteration can cover a later one. flow_trips_near():
the merge drops a state that a state of its own iteration or of the two
before it covers. If only states of one iteration are compared, 164 of the
225 progs of the schedulers of rust-bpf take more insns, 9 % in sum.

The drop closes a cycle: the earlier state leads to the later one and now
stands for it. "No state returns to the head, so the loop ends at run time"
does not hold any more. Thus flow_cycle_closed() gives the loop a may_goto.
It does nothing if the loop is counted, or if may_goto or bpf_iter_*_next()
stops each iteration. It rejects a loop that has no place for may_goto.

flow_within() is 'covers' for the merge: the comparison of pruning, also of
the states that entered each counted loop: its ranges come from them.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 173 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 3996220f89f0..19b502bcb57c 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1476,3 +1476,176 @@ static int flow_unite(struct bpf_flow *flow, u32 a, u32 b)
 	u->b = b;
 	return 0;
 }
+
+/*
+ * What is known about a counted loop comes from the state it was entered
+ * with. @t stands for @s only if that holds for these states too.
+ */
+static bool flow_loops_within(struct bpf_verifier_env *env, const struct bpf_verifier_state *t,
+			      const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *lt, *ls;
+	u32 fr, i;
+
+	for (fr = 0; fr <= t->curframe; fr++) {
+		if (t->frame[fr]->loop_stack_cnt != s->frame[fr]->loop_stack_cnt)
+			return false;
+		for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+			lt = &t->frame[fr]->loop_stack[i];
+			ls = &s->frame[fr]->loop_stack[i];
+			if (!lt->terminates || lt->entry_state == ls->entry_state)
+				continue;
+			if (!ls->terminates ||
+			    !bpf_state_subsumes(env, lt->entry_state, ls->entry_state))
+				return false;
+		}
+	}
+	return true;
+}
+
+static bool flow_within(struct bpf_verifier_env *env, struct bpf_verifier_state *t,
+			struct bpf_verifier_state *s)
+{
+	return t->curframe == s->curframe &&
+	       t->frame[0]->subprogno == s->frame[0]->subprogno &&
+	       /* an async callback returns to the kernel, a call of the function to its caller */
+	       !t->flow_entry == !s->flow_entry &&
+	       bpf_state_subsumes(env, t, s) && flow_loops_within(env, t, s);
+}
+
+/*
+ * How many iterations is @t behind @s, in the loop in which that is the most? Only the loops
+ * whose iterations are kept apart are looked at.
+ * 0 if both are in the same iteration of every loop, U32_MAX if @t is ahead of @s in a loop.
+ */
+static u32 flow_trips_behind(const struct bpf_verifier_state *t, const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *lt, *ls;
+	u32 fr, i, behind = 0;
+
+	if (t->curframe != s->curframe)
+		return U32_MAX;
+	for (fr = 0; fr <= t->curframe; fr++) {
+		if (t->frame[fr]->loop_stack_cnt != s->frame[fr]->loop_stack_cnt)
+			return U32_MAX;
+		for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+			lt = &t->frame[fr]->loop_stack[i];
+			ls = &s->frame[fr]->loop_stack[i];
+			if (!lt->apart && !ls->apart)
+				continue;
+			if (lt->trip > ls->trip)
+				return U32_MAX;
+			behind = max(behind, ls->trip - lt->trip);
+		}
+	}
+	return behind;
+}
+
+/*
+ * A state is compared with the states of its own iteration and of the two before it. The older ones
+ * are gone, see flow_earlier_trip(), but for what is pending or waits: every iteration of a loop
+ * calls a recursive function, and what comes out of it goes to all the calls at once.
+ */
+static bool flow_trips_near(const struct bpf_verifier_state *t, const struct bpf_verifier_state *s)
+{
+	return flow_trips_behind(t, s) <= 2;
+}
+
+/*
+ * @s is within @t, and from now on @t stands for it.
+ *
+ * A walk on patience gets to the end of a loop when no state comes back to the head any more.
+ * Then the loop ends at run time too: the number of the iteration grows at the head, nothing makes
+ * it smaller, and there is a largest one.
+ * If @t is of an earlier iteration than @s that is not true any more. What becomes of @t got to @s
+ * and is @t again: as far as the verifier can see the program goes around for ever. A spin lock
+ * is such a loop. So the loop gets may_goto.
+ *
+ * It's asked for every loop that @t is behind in, not only for the outermost one. This is the only
+ * way for the number of a loop to get smaller. In a walk it doesn't change in another way but at
+ * the head: a state never stands for a state of an earlier iteration, see loop_stack_safe(), and it
+ * takes the place of, or is joined with, states of its own iterations only, see flow_merge().
+ * Where the iterations are not kept apart a join has the larger number, and a state may stand
+ * for one with a smaller number. Such loop has a count or may_goto, or something stops it, since
+ * the state entered it.
+ */
+static int flow_cycle_closed(struct bpf_verifier_env *env, const struct bpf_verifier_state *t,
+			     const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *lt, *ls;
+	struct bpf_loop *loop;
+	u32 fr, i;
+
+	for (fr = 0; fr <= t->curframe; fr++) {
+		for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+			lt = &t->frame[fr]->loop_stack[i];
+			ls = &s->frame[fr]->loop_stack[i];
+			if (lt->trip >= ls->trip)
+				continue;
+			/* counted: what comes back is expected to be within what is at the head */
+			if (lt->terminates && !bpf_loop_walk_may_give_up(lt))
+				continue;
+			loop = env->insn_aux_data[lt->loop_id].loop;
+			/* it goes around for as long as the iterator has elements or may_goto budget */
+			if (loop->rt_bounded)
+				continue;
+			if (!loop->has_guards) {
+				verbose(env, "loop at insn %d never exits\n", lt->loop_id);
+				return -EINVAL;
+			}
+			bpf_flow_loop_gets_guards(env, lt->loop_id);
+		}
+	}
+	return 0;
+}
+
+/*
+ * @t is of an earlier iteration of a loop whose iterations are kept apart than @s.
+ * What comes after @s is of later iterations too, so @t is not going to be
+ * of use. If something of its iteration does come it is looked at again.
+ *
+ * Not so where a state stands for many iterations. States of different kinds are at the head of
+ * such loop. One of them comes to rest early, and what comes around is within it for as long as
+ * the others take.
+ */
+static bool flow_earlier_trip(const struct bpf_verifier_state *t,
+			      const struct bpf_verifier_state *s)
+{
+	const struct bpf_func_state *ft, *fs;
+	u32 fr, i;
+
+	if (t->curframe != s->curframe)
+		return false;
+	for (fr = 0; fr <= t->curframe; fr++) {
+		ft = t->frame[fr];
+		fs = s->frame[fr];
+		if (ft->subprogno != fs->subprogno || ft->callsite != fs->callsite ||
+		    ft->loop_stack_cnt != fs->loop_stack_cnt)
+			return false;
+		for (i = 0; i < ft->loop_stack_cnt; i++) {
+			if (ft->loop_stack[i].loop_id != fs->loop_stack[i].loop_id ||
+			    ft->loop_stack[i].apart != fs->loop_stack[i].apart)
+				return false;
+			if (ft->loop_stack[i].apart &&
+			    ft->loop_stack[i].trip != fs->loop_stack[i].trip)
+				return ft->loop_stack[i].trip + 1 < fs->loop_stack[i].trip;
+		}
+	}
+	return false;
+}
+
+/* Can flow_earlier_trip() be true for a state and @s? */
+static bool flow_has_earlier_trips(const struct bpf_verifier_state *s)
+{
+	const struct loop_stack_entry *entry;
+	u32 fr, i;
+
+	for (fr = 0; fr <= s->curframe; fr++) {
+		for (i = 0; i < s->frame[fr]->loop_stack_cnt; i++) {
+			entry = &s->frame[fr]->loop_stack[i];
+			if (entry->apart && entry->trip > 1)
+				return true;
+		}
+	}
+	return false;
+}
-- 
2.55.0

[PATCH bpf-next v4 13/46] bpf: Decide which states are not joined

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Cilium, nat.h, in tail_handle_snat_fwd_ipv4:

  if (ret < 0) ..; to_sport = state->to_sport;

(ret=-155, state=map_value_or_null) and (ret=0, state=map_value) meet in
front of that. Numbers always have a join. But ret=[-155, 0] with
map_value_or_null has ret=0 without state, as no path has. It passes the
jump: "R3 invalid mem access 'map_value_or_null'".

A join must not make up what a later check can see. A number is tested if
a later jump compares it with a constant, or the verifier checks its range
at a call or at exit. States that have a join stay apart if they differ in:

 - A tested number, and the ranges leave a gap: -155 and 0, not 0 and 1.
   113 progs of Cilium, profiler3 and test_seg6_loop without alu32.
 - Two tested numbers, each without a common value: (err >= 0, ret=0) and
   (err < 0, ret=DROP). The join has err < 0 with ret=0.
 - A flag, 0 and 1, that a jump in this basic block or the next one tests:
   'ok = i < 64 && v->in[i] == 2; if (v->x == 7) v->n++;
    if (ok) v->w[i] = 1;'
   clang keeps ok in a register. (ok=0, i>63), (ok=1, i<=63), (ok=0, i<=63)
   meet. The join has ok=1 with i>63: "R0 unbounded memory access".
 - A register that only one of them sets. scx_layered/layered_runnable:
   'scoped_guard(rcu) { if (...) return result; nsid = ...; }' meets at the
   unlock with a flag of the compiler, then reads nsid: "R7 !read_ok".
 - Two tested numbers, (0, x) and (y, 0). xdp_synproxy_kern and
   test_tcp_custom_syncookie: 'if (hdr->ipv4) ... else if (hdr->ipv6) ...'
   after the packet changed: "R1 type=scalar expected=fp, pkt, ...".
 - A constant that becomes a stack offset. bt_print(), 5 selftests of scx:
   'if (... || depth >= BT_MAXLVL_PRINT) return 0; ind = stack[depth];'
   A copy of depth is tested: "invalid variable-offset read from stack".
 - A constant that is added to a pointer. katran/tcdecapinfo, 20 or 40 in
   'if (data + hdr_len + inner_ip_offset > data_end)'. With [20, 40] the
   jump says nothing about data + hdr_len: "invalid access to packet".
 - A constant that a kfunc or helper must know. One call is made of
   'c ? bpf_rdonly_cast(0, id) : bpf_rdonly_cast(0, 0)' in
   mem_rdonly_untrusted/mixed_mem_type: "R2 must be a known constant".
 - The number of calls of the callback of bpf_loop(N, ...), N <= 8:
   'bpf_loop(1, cb, &ctx, 0); return choice_arr[ctx.i];' in
   verifier_iterating_callbacks: "invalid access to map value".

The rules only refuse joins: no unsafe prog loads because of them. They
cost states: the first one 10 % of Cilium's insns, K flags 2^K states. Thus
most hold only while each limit is less than half used: flow_has_room().

The shape of a state is a filter in front of all that: hashes of its calls
and loops, of what it holds, of the places, objects and constant offsets of
its pointers. The load of test_cls_redirect_subprogs, 301 states at an
insn, takes three times as long without the shape.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 566 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 559 insertions(+), 7 deletions(-)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 19b502bcb57c..feaa5949a2a1 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -148,6 +148,7 @@ enum flow_need_kind {
 	FLOW_NEED_STACK_OFF,
 	FLOW_NEED_PTR_OFF,
 	FLOW_NEED_JUMP,
+	FLOW_NEED_FLAG,
 	FLOW_NEED_VALUE_OFF,
 	FLOW_NEED_BASE,
 	FLOW_NEED_JUMP_ALU,
@@ -568,6 +569,19 @@ static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
 	return consts & BIT(insn->dst_reg) ? BIT(insn->src_reg) : 0;
 }
 
+/* A jump that compares a register with 0 or 1 */
+static u16 flow_flag_regs(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+
+	if (BPF_SRC(insn->code) != BPF_K || !bpf_insn_is_cond_jump(insn->code) ||
+	    BPF_OP(insn->code) == BPF_JCOND)
+		return 0;
+	if (BPF_OP(insn->code) == BPF_JSET ? !(insn->imm & 1) : (u32)insn->imm > 1)
+		return 0;
+	return BIT(insn->dst_reg);
+}
+
 /* With or without sign extension */
 static bool flow_is_copy(const struct bpf_insn *insn)
 {
@@ -960,6 +974,15 @@ static u16 flow_base_regs(struct bpf_verifier_env *env, int idx)
  * A slot that is filled through another pointer to the stack than r10 is not one of them, and
  * flow_null_differs() counts on that.
  *
+ * FLOW_NEED_FLAG: those of which a jump tells 0 from 1, see flow_join_makes_up(). As they are,
+ * extended, or masked: a bool is tested as "w1 &= 1; if w1 != 0". Not through a call.
+ * The jump ends the basic block of the insn, or a block that this block leads to. Not further:
+ * every state that is kept apart goes through all that is on the way to the jump once more.
+ * So there are two rounds. In the first one nothing comes from the blocks ahead. What it finds
+ * at the first insn of a block is what the second one takes at the end of the blocks that lead
+ * there. In both a block depends on nothing that the round itself computes for another block,
+ * so the order in which the insns are looked at doesn't matter.
+ *
  * The last three kinds keep no states apart. They are for bpf_flow_is_needed(), which asks
  * whether anything ahead looks at a number that a loop changes from iteration to iteration.
  * FLOW_NEED_VALUE_OFF: FLOW_NEED_PTR_OFF, and the offset in a map value that holds nothing but
@@ -981,10 +1004,11 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 				   enum flow_need_kind kind)
 {
 	struct bpf_insn *insns = env->prog->insnsi, *insn;
-	struct flow_need in, out, *need;
+	struct flow_need in, out, *need, *from, *ahead = NULL;
 	struct bpf_iarray *succ;
 	bool jump = kind == FLOW_NEED_JUMP || kind == FLOW_NEED_JUMP_ALU;
 	bool callx = kind == FLOW_NEED_BASE || kind == FLOW_NEED_JUMP_ALU;
+	bool flag = kind == FLOW_NEED_FLAG, second = false;
 	bool changed = true, any = false, keep, from_src, reg;
 	int i, s, idx, slot, err = 0;
 	u16 *gen, callx_regs = 0, pair = BIT(BPF_REG_0) | BIT(BPF_REG_2);
@@ -994,8 +1018,11 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 	gen = kvcalloc(env->prog->len, sizeof(u16), GFP_KERNEL_ACCOUNT);
 	/* the callers need what the subprog returns: r0, and r2 of a pair */
 	ret = kvcalloc(env->subprog_cnt, sizeof(u8), GFP_KERNEL_ACCOUNT);
+	if (flag)
+		ahead = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
 	*pneed = need;
-	if (!need || !gen || !ret) {
+	if (!need || !gen || !ret || (flag && !ahead)) {
+		kvfree(ahead);
 		kvfree(gen);
 		kvfree(ret);
 		return -ENOMEM;
@@ -1003,6 +1030,8 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 	for (i = 0; i < env->prog->len; i++) {
 		if (jump)
 			gen[i] = flow_tested_regs(env, i);
+		else if (flag)
+			gen[i] = flow_flag_regs(env, i);
 		else if (kind == FLOW_NEED_STACK_OFF)
 			gen[i] = flow_stack_off_regs(env, i);
 		else if (kind == FLOW_NEED_PTR_OFF || kind == FLOW_NEED_VALUE_OFF)
@@ -1031,6 +1060,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 		}
 		any = true;
 	}
+again:
 	while (any && changed) {
 		changed = false;
 		if (signal_pending(current)) {
@@ -1048,9 +1078,10 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 			succ = bpf_insn_successors(env, idx);
 			out.regs = 0;
 			out.slots = 0;
+			from = flag && env->insn_aux_data[idx].bb_end ? ahead : need;
 			for (s = 0; s < succ->cnt; s++) {
-				out.regs |= need[succ->items[s]].regs;
-				out.slots |= need[succ->items[s]].slots;
+				out.regs |= from[succ->items[s]].regs;
+				out.slots |= from[succ->items[s]].slots;
 			}
 			in = out;
 			if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) {
@@ -1058,7 +1089,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 				if (bpf_is_callx(insn))
 					in.regs |= callx_regs & GENMASK(BPF_REG_5, BPF_REG_1);
 				s = idx + insn->imm + 1;
-				if (bpf_pseudo_call(insn) &&
+				if (bpf_pseudo_call(insn) && !flag &&
 				    !bpf_subprog_is_global(env, bpf_find_subprog(env, s))) {
 					if (kind != FLOW_NEED_PTR_OFF)
 						in.regs |= need[s].regs &
@@ -1097,10 +1128,12 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 				/* the result is known when what it is made of is */
 				keep = op != BPF_MOV;
 				from_src = reg;
-				if (jump)
+				if (jump || flag)
 					from_src = flow_is_copy(insn);
-				if (kind == FLOW_NEED_JUMP)
+				if (kind == FLOW_NEED_JUMP || flag)
 					keep = flow_is_shift(insn) && !reg;
+				if (flag && op == BPF_AND && !reg)
+					keep = insn->imm & 1;
 				if (kind == FLOW_NEED_BASE) {
 					keep = !reg && (op == BPF_ADD || op == BPF_SUB);
 					from_src = reg && op == BPF_MOV;
@@ -1117,12 +1150,18 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 			}
 		}
 	}
+	if (flag && !err && !second) {
+		memcpy(ahead, need, env->prog->len * sizeof(*need));
+		second = changed = true;
+		goto again;
+	}
 	/*
 	 * An offset that is an argument: not through the call, see above, and not where the calls
 	 * meet. A function would be looked at once for each offset that it's called with.
 	 */
 	for (s = 0; kind == FLOW_NEED_PTR_OFF && s < env->subprog_cnt; s++)
 		need[env->subprog_info[s].start].regs = 0;
+	kvfree(ahead);
 	kvfree(gen);
 	kvfree(ret);
 	if (!any) {
@@ -1649,3 +1688,516 @@ static bool flow_has_earlier_trips(const struct bpf_verifier_state *s)
 	}
 	return false;
 }
+
+/* @flag: 0 and 1 */
+static bool flow_two_consts(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb,
+			    bool flag)
+{
+	return ra->type == SCALAR_VALUE && rb->type == SCALAR_VALUE &&
+	       tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+	       ra->var_off.value != rb->var_off.value &&
+	       (!flag || (ra->var_off.value | rb->var_off.value) == 1);
+}
+
+/* Constants that have to stay constants, see flow_compute_need_const() */
+static bool flow_needed_consts_differ(const struct bpf_flow *flow, enum flow_need_kind kind,
+				      const struct bpf_verifier_state *a,
+				      const struct bpf_verifier_state *b)
+{
+	const struct flow_need *needs = flow->need[kind], *need;
+	const struct bpf_stack_state *sa, *sb;
+	const struct bpf_func_state *fa, *fb;
+	bool flag = kind == FLOW_NEED_FLAG;
+	u32 fr, i, nr;
+
+	if (!needs || a->curframe != b->curframe)
+		return false;
+	for (fr = 0; fr <= a->curframe; fr++) {
+		/* the callers need what they need when the call returns */
+		need = &needs[fr == a->curframe ? a->insn_idx : a->frame[fr + 1]->callsite + 1];
+		fa = a->frame[fr];
+		fb = b->frame[fr];
+		for (i = BPF_REG_0; need->regs && i < BPF_REG_FP; i++)
+			if ((need->regs & BIT(i)) &&
+			    flow_two_consts(&fa->regs[i], &fb->regs[i], flag))
+				return true;
+		nr = min(bpf_stack_nr_slots(fa), bpf_stack_nr_slots(fb));
+		for (i = 0; i < nr && i < 64 && need->slots >> i; i++) {
+			sa = bpf_stack_slot(fa, i);
+			sb = bpf_stack_slot(fb, i);
+			if ((need->slots & BIT_ULL(i)) &&
+			    bpf_is_spilled_scalar_reg(sa) && bpf_is_spilled_scalar_reg(sb) &&
+			    flow_two_consts(&sa->spilled_ptr, &sb->spilled_ptr, flag))
+				return true;
+		}
+	}
+	return false;
+}
+
+/* regsafe() and join_reg() say what each type needs */
+static void flow_shape_ptr(struct flow_shape *sh, u32 place, const struct bpf_reg_state *reg)
+{
+	u32 ids = (reg->id != 0) | (reg->parent_id != 0) << 1 | (reg->map_uid != 0) << 2;
+	u32 len = offsetof(struct bpf_reg_state, var_off);
+	bool has_off = false, var_off = false;
+	u64 off = reg->var_off.value;
+
+	switch (base_type(reg->type)) {
+	case NOT_INIT:
+	case SCALAR_VALUE:
+		return;
+	case PTR_TO_ARENA:
+		/* the offset is not tracked */
+		len = sizeof(reg->type);
+		ids = 0;
+		break;
+	case PTR_TO_PACKET_META:
+	case PTR_TO_PACKET:
+		len = sizeof(reg->type);
+		ids &= 1;
+		/* behind the end of the packet, or at the end: the same in both states */
+		if (reg->range < 0)
+			ids |= -reg->range << 1;
+		else
+			sh->pkt_range += reg->range;
+		if (reg->id) {
+			/* displacements between pointers that share an id are kept */
+			has_off = true;
+			off = reg->r64.base;
+		} else if (reg->range >= 0) {
+			/* join_reg() takes two such constants for two places in the packet */
+			has_off = true;
+			var_off = !tnum_is_const(reg->var_off) || off > MAX_PACKET_OFF;
+		}
+		break;
+	case PTR_TO_INSN:
+		/* a range of offsets is a set of targets */
+		ids = 0;
+		break;
+	case PTR_TO_MAP_KEY:
+	case PTR_TO_MAP_VALUE:
+	case PTR_TO_MEM:
+	case PTR_TO_BUF:
+	case PTR_TO_TP_BUFFER:
+	case PTR_TO_STACK:
+	case PTR_TO_BTF_ID:
+		has_off = true;
+		var_off = !tnum_is_const(reg->var_off);
+		break;
+	default:
+		len = offsetof(struct bpf_reg_state, id);
+	}
+	sh->word[FLOW_LAYOUT] = jhash_1word(place << 1 | var_off, sh->word[FLOW_LAYOUT]);
+	sh->word[FLOW_PTRS] = jhash2((const u32 *)reg, len / sizeof(u32),
+				     sh->word[FLOW_PTRS] + ids);
+	if (has_off && !var_off)
+		sh->word[FLOW_OFFS] = jhash_2words(lower_32_bits(off), upper_32_bits(off),
+						   sh->word[FLOW_OFFS]);
+}
+
+/* stacksafe() and join_special_slot() say what each kind needs */
+static void flow_shape_special(struct flow_shape *sh, u32 place, const struct bpf_stack_state *slot)
+{
+	const struct bpf_reg_state *reg = &slot->spilled_ptr;
+	u32 kind = slot->slot_type[BPF_REG_SIZE - 1] << 2 | (reg->id != 0);
+	u32 *ptrs = &sh->word[FLOW_PTRS];
+
+	switch (slot->slot_type[BPF_REG_SIZE - 1]) {
+	case STACK_DYNPTR:
+		kind |= (reg->parent_id != 0) << 1;
+		*ptrs = jhash_3words(kind, reg->dynptr.type, reg->dynptr.first_slot, *ptrs);
+		break;
+	case STACK_ITER:
+		*ptrs = jhash_3words(lower_32_bits((unsigned long)reg->iter.btf),
+				     upper_32_bits((unsigned long)reg->iter.btf),
+				     reg->iter.btf_id, *ptrs + kind);
+		*ptrs = jhash_3words(reg->type, reg->iter.state, reg->iter.loops, *ptrs);
+		break;
+	case STACK_IRQ_FLAG:
+		*ptrs = jhash_2words(kind, reg->irq.kfunc_class, *ptrs);
+		break;
+	}
+	sh->word[FLOW_LAYOUT] = jhash_1word(place << 1, sh->word[FLOW_LAYOUT]);
+}
+
+/* The part of the shape that is made of the registers and the stack, when it is asked for */
+static void flow_shape_places(struct bpf_verifier_env *env, struct flow_state *s)
+{
+	struct bpf_verifier_state *st = &s->st;
+	const struct bpf_stack_state *slot;
+	const struct bpf_func_state *frame;
+	struct flow_shape *sh = &s->shape;
+	u32 fr, i, args;
+	u16 live;
+
+	sh->places = true;
+	for (fr = 0; fr <= st->curframe; fr++) {
+		frame = st->frame[fr];
+		/* a state that does a call again is not cleaned, dead registers are not compared */
+		live = env->insn_aux_data[bpf_frame_insn_idx(st, fr)].live_regs_before;
+		for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+			if (live & BIT(i))
+				flow_shape_ptr(sh, fr << 10 | i, &frame->regs[i]);
+		for (i = 0; i < bpf_stack_nr_slots(frame); i++) {
+			slot = bpf_stack_slot(frame, i);
+			switch (slot->slot_type[BPF_REG_SIZE - 1]) {
+			case STACK_SPILL:
+				flow_shape_ptr(sh, fr << 10 | (i + 16), &slot->spilled_ptr);
+				break;
+			case STACK_DYNPTR:
+			case STACK_ITER:
+			case STACK_IRQ_FLAG:
+				flow_shape_special(sh, fr << 10 | (i + 16), slot);
+				break;
+			}
+		}
+		/* incoming arguments are not compared with outgoing ones: other places */
+		args = 16 + MAX_BPF_STACK_SLOTS;
+		if (bpf_stack_args_are_incoming(frame))
+			args += MAX_STACK_ARG_SLOTS;
+		for (i = 0; i < frame->out_stack_arg_cnt; i++)
+			flow_shape_ptr(sh, fr << 10 | (i + args), &frame->stack_arg_regs[i]);
+	}
+}
+
+/* The part of the shape that is not */
+static void flow_shape_ctx(const struct bpf_verifier_state *st, struct flow_shape *sh)
+{
+	const struct bpf_reference_state *ref;
+	const struct loop_stack_entry *entry;
+	const struct bpf_func_state *frame;
+	u32 chain = st->curframe, ctx, fr, i;
+
+	ctx = jhash_3words(st->in_sleepable | st->callers_unknown << 1 |
+			   st->unwound << 2 | st->changed_pkt_data << 3 |
+			   (st->active_irq_id != 0) << 4 | (st->active_lock_id != 0) << 5,
+			   st->acquired_refs, st->active_locks, 0);
+	ctx = jhash_2words(st->active_preempt_locks, st->active_rcu_locks, ctx);
+	ctx = jhash_2words(lower_32_bits((unsigned long)st->active_lock_ptr),
+			   upper_32_bits((unsigned long)st->active_lock_ptr), ctx);
+	for (i = 0; i < st->acquired_refs; i++) {
+		ref = &st->refs[i];
+		ctx = jhash_1word(ref->type, ctx);
+		if (ref->type != REF_TYPE_PTR && ref->type != REF_TYPE_IRQ)
+			ctx = jhash_2words(lower_32_bits((unsigned long)ref->ptr),
+					   upper_32_bits((unsigned long)ref->ptr), ctx);
+	}
+	for (fr = 0; fr <= st->curframe; fr++) {
+		frame = st->frame[fr];
+		chain = jhash_3words(frame->subprogno, frame->callsite, frame->loop_stack_cnt,
+				     chain);
+		ctx = jhash_2words(frame->entry_refs, frame->in_rcu_callback_fn, ctx);
+		ctx = jhash_2words(frame->entry_rcu_locks, frame->entry_preempt_locks, ctx);
+		for (i = 0; i < frame->loop_stack_cnt; i++) {
+			entry = &frame->loop_stack[i];
+			chain = jhash_1word(entry->loop_id << 1 | entry->apart, chain);
+			ctx = jhash_3words(entry->counted << 2 | entry->guarded << 1 |
+					   entry->terminates,
+					   entry->max_header_count,
+					   entry->counted ? entry->trip : 0, ctx);
+		}
+	}
+	memset(sh, 0, sizeof(*sh));
+	sh->word[FLOW_CHAIN] = chain;
+	sh->word[FLOW_CTX] = ctx;
+}
+
+/* Are the words up to @last the same? */
+static __always_inline bool flow_same_shape(const struct flow_state *a, const struct flow_state *b,
+					    enum flow_word last)
+{
+	u32 i;
+
+	for (i = 0; i <= last; i++)
+		if (a->shape.word[i] != b->shape.word[i])
+			return false;
+	return true;
+}
+
+/*
+ * A state that arrives is compared with the states at the insn: is it within one of them, see
+ * bpf_state_subsumes(), does it have a join with one of them, see bpf_states_join()? Most of
+ * them have nothing to do with it. The shape of a state tells that before the registers are
+ * compared or, for a join, copied. It has what must be the same in two states that are related:
+ *
+ * FLOW_CHAIN	the chain of calls and the loops that every frame is in
+ * FLOW_CTX	what is known about every loop, the iteration of a counted one,
+ *		the flags of the state and of its frames, locks and references
+ * FLOW_LAYOUT	which registers, stack slots and stack arguments hold a pointer, a dynptr,
+ *		an iterator or an irq flag, which of the pointers are at a constant offset
+ * FLOW_PTRS	what the pointers point to and which of their ids are set, the kind of the others
+ * FLOW_OFFS	the constant offsets
+ *
+ * Nothing about numbers is in it.
+ *
+ * States that differ in one of the first two words are not related. Neither are states of
+ * the same layout that differ in one of the last two: a pointer is related to a pointer to the same
+ * thing only, and two constant offsets are two places in it.
+ * States of different layout may be related, whatever the last two words are: nothing in
+ * a register or in a slot stands for a pointer there, and a range of offsets stands for a constant,
+ * or is joined with it.
+ *
+ * A word is a hash. It is a filter and proves nothing: the states that get through it go
+ * through all the comparisons. So it's fine to look at the first two words only: @places.
+ */
+static bool flow_may_relate(struct bpf_verifier_env *env, struct flow_state *t,
+			    struct flow_state *s, bool places)
+{
+	if (!flow_same_shape(t, s, FLOW_CTX))
+		return false;
+	if (!places)
+		return true;
+	if (!t->shape.places)
+		flow_shape_places(env, t);
+	if (!s->shape.places)
+		flow_shape_places(env, s);
+	return !flow_same_shape(t, s, FLOW_LAYOUT) || flow_same_shape(t, s, FLOW_OFFS);
+}
+
+/*
+ * A pointer to the packet stands for one with at least as many checked bytes, and without
+ * widening it's joined with one that has as many. In states of the same layout that holds for
+ * the sums too. Returns 0 if the sums are the same, or say nothing.
+ */
+static int flow_cmp_pkt_range(const struct flow_state *t, const struct flow_state *s)
+{
+	if (!t->shape.places || !s->shape.places || !flow_same_shape(t, s, FLOW_LAYOUT) ||
+	    t->shape.pkt_range == s->shape.pkt_range)
+		return 0;
+	return t->shape.pkt_range < s->shape.pkt_range ? -1 : 1;
+}
+
+/*
+ * bpf_loop(N, ...) with a small N: the calls of the callback are looked at one by one, like
+ * the iterations of a loop that is walked. The call stops to enter the callback when
+ * callback_depth gets to N.
+ */
+static bool flow_callback_depth_differs(struct bpf_verifier_env *env,
+					const struct bpf_verifier_state *a,
+					const struct bpf_verifier_state *b)
+{
+	u32 max = BPF_FLOW_MAX_CALLBACKS, fr;
+	const struct bpf_func_state *fa, *fb;
+	struct bpf_insn *insn;
+
+	if (a->curframe != b->curframe)
+		return false;
+	/* at the call, and in the callback and what it calls */
+	for (fr = 0; fr <= a->curframe; fr++) {
+		insn = &env->prog->insnsi[fr == a->curframe ? a->insn_idx :
+					  a->frame[fr + 1]->callsite];
+		if (!bpf_helper_call(insn) || insn->imm != BPF_FUNC_loop)
+			continue;
+		fa = a->frame[fr];
+		fb = b->frame[fr];
+		if (fa->callback_depth != fb->callback_depth &&
+		    fa->regs[BPF_REG_1].type == SCALAR_VALUE &&
+		    reg_umax(&fa->regs[BPF_REG_1]) <= max &&
+		    fb->regs[BPF_REG_1].type == SCALAR_VALUE &&
+		    reg_umax(&fb->regs[BPF_REG_1]) <= max)
+			return true;
+	}
+	return false;
+}
+
+static const struct bpf_reg_state *flow_slot_number(const struct bpf_func_state *frame, u32 i)
+{
+	const struct bpf_stack_state *slot = bpf_stack_slot(frame, i);
+
+	if (slot->slot_type[0] != STACK_SPILL || slot->slot_type[BPF_REG_SIZE - 1] != STACK_SPILL ||
+	    slot->spilled_ptr.type != SCALAR_VALUE)
+		return NULL;
+	return &slot->spilled_ptr;
+}
+
+/* Returns true when the answer is known */
+static bool flow_null_pair(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb,
+			   bool *a_null, bool *b_null)
+{
+	if (tnum_equals_const(ra->var_off, 0) && !tnum_is_const(rb->var_off)) {
+		*a_null = true;
+		if (reg_umin(rb) == 1 && reg_umax(rb) == U64_MAX)
+			return true;
+	}
+	if (tnum_equals_const(rb->var_off, 0) && !tnum_is_const(ra->var_off)) {
+		*b_null = true;
+		if (reg_umin(ra) == 1 && reg_umax(ra) == U64_MAX)
+			return true;
+	}
+	return *a_null && *b_null;
+}
+
+/*
+ * Two things of which one is there: "if (hdr->ipv4) ... else if (hdr->ipv6)". The one that
+ * is not there is 0. What the other one is may be unknown, e.g. a pointer to the packet
+ * after the packet was changed. (0, x) is not within (y, 0) and the join (?, ?) forgets that
+ * it's one or the other.
+ * The same for a number that was compared with 0: 0 on one path, not 0 on the other.
+ *
+ * Only where a jump ahead can see it: FLOW_NEED_JUMP, see flow_join_makes_up(). What no jump
+ * looks at again sends no state another way, and every such number doubles the states.
+ *
+ *	for (i = 0; i < n; i++) { p = a[i]; if (!p) continue; put(p); }
+ * a[] is on the stack. The jump finds out about the slot that p was filled from too. Nothing
+ * looks at a[i] again, but the loop reads through a pointer that moves, so all of a[] is live
+ * until the loop is done: 2^n states if they stay apart.
+ * They don't, because FLOW_NEED_JUMP has only what is filled through r10, see flow_fp_slot().
+ * It cannot tell a[i] from a[i + 1]. With what is filled through other pointers to the stack in
+ * it, all of a[] is tested in all of the loop.
+ */
+static bool flow_null_differs(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
+			      const struct bpf_verifier_state *b)
+{
+	const struct bpf_func_state *fa, *fb;
+	const struct bpf_reg_state *ra, *rb;
+	const struct flow_need *need = flow->need[FLOW_NEED_JUMP];
+	bool a_null = false, b_null = false;
+	u32 i;
+
+	if (!need || a->curframe != b->curframe)
+		return false;
+	need += a->insn_idx;
+	fa = a->frame[a->curframe];
+	fb = b->frame[b->curframe];
+	for (i = BPF_REG_0; i < BPF_REG_FP; i++) {
+		ra = &fa->regs[i];
+		rb = &fb->regs[i];
+		if ((need->regs & BIT(i)) &&
+		    ra->type == SCALAR_VALUE && rb->type == SCALAR_VALUE &&
+		    flow_null_pair(ra, rb, &a_null, &b_null))
+			return true;
+	}
+	for (i = 0; i < 64 && i < bpf_stack_nr_slots(fa) && i < bpf_stack_nr_slots(fb); i++) {
+		if (!(need->slots & BIT_ULL(i)))
+			continue;
+		ra = flow_slot_number(fa, i);
+		rb = flow_slot_number(fb, i);
+		if (ra && rb && flow_null_pair(ra, rb, &a_null, &b_null))
+			return true;
+	}
+	return false;
+}
+
+static const struct bpf_reg_state *flow_number_at(const struct bpf_func_state *frame, u8 loc)
+{
+	const struct bpf_reg_state *reg;
+
+	if (loc <= BPF_REG_FP)
+		reg = &frame->regs[loc];
+	else if (loc - BPF_REG_FP - 1 < bpf_stack_nr_slots(frame) &&
+		 bpf_is_spilled_scalar_reg(bpf_stack_slot(frame, loc - BPF_REG_FP - 1)))
+		reg = &bpf_stack_slot(frame, loc - BPF_REG_FP - 1)->spilled_ptr;
+	else
+		return NULL;
+	return reg->type == SCALAR_VALUE ? reg : NULL;
+}
+
+/*
+ * A register is set in one of the states and not in the other. Only live registers are in
+ * a state, so something ahead may read it. The state in which it's not set doesn't get there:
+ * a jump sends it another way. The join would not know which way to go, and would get
+ * there without the register. There is nothing to gain from such a join but a state less.
+ */
+static bool flow_unset_differs(const struct bpf_verifier_state *a,
+			       const struct bpf_verifier_state *b)
+{
+	u32 fr, i;
+
+	for (fr = 0; fr <= a->curframe; fr++)
+		for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+			if ((a->frame[fr]->regs[i].type == NOT_INIT) !=
+			    (b->frame[fr]->regs[i].type == NOT_INIT))
+				return true;
+	return false;
+}
+
+static bool flow_disjoint(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+	return cnum64_is_empty(cnum64_intersect(a->r64, b->r64)) ||
+	       cnum32_is_empty(cnum32_intersect(a->r32, b->r32)) ||
+	       ((a->var_off.value ^ b->var_off.value) & ~(a->var_off.mask | b->var_off.mask));
+}
+
+/* The smallest range that has both has numbers that are in neither */
+static bool flow_gap(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+	if (cnum64_is_empty(cnum64_intersect(a->r64, b->r64)) &&
+	    a->r64.size + b->r64.size + 1 > a->r64.size &&
+	    cnum64_union(a->r64, b->r64).size > a->r64.size + b->r64.size + 1)
+		return true;
+	return cnum32_is_empty(cnum32_intersect(a->r32, b->r32)) &&
+	       cnum32_union(a->r32, b->r32).size > (u64)a->r32.size + b->r32.size + 1;
+}
+
+/*
+ * Numbers always have a join. It must not make up what neither state has where a jump ahead
+ * can see it. FLOW_NEED_JUMP has the registers and the stack slots that a jump ahead compares
+ * with a constant, or the verifier does at a call or at the exit.
+ *
+ *	if (ret < 0) goto err; if (ret > 0) goto skip; use(p);
+ * ret = [-2, -1] and ret = [1, 2], both without p, meet in front of that. [-2, 2] has 0, gets
+ * through both jumps, and there is no p. So two ranges that leave a gap are not joined. Ranges
+ * that touch, 0 and 1, -1 and 0, or have something in common make up no number.
+ *
+ *	err = update(); ret = err < 0 ? DROP : 0; ... if (err < 0) goto out; ... if (ret < 0) ...
+ * (err >= 0, ret = 0) and (err < 0, ret = DROP) meet. The join has err < 0 with ret = 0. So
+ * two numbers, each with nothing in common between the states, are not joined either.
+ *
+ *	fatal = err == -ENOMEM; ... if (fatal) bpf_set_retval(err);
+ * (fatal = 0, err = 1) and (fatal = 1, err = -ENOMEM) meet. The same: the join has
+ * fatal = 1 with err = 1, which bpf_set_retval() doesn't take.
+ *
+ *	ok = i < 64 && in[i]; if (x) n++; if (ok) a[i] = 1;
+ * (ok = 0, i >= 64), (ok = 0, i < 64) and (ok = 1, i < 64) meet. 0 and 1 touch, and i < 64 comes
+ * with either, so none of the above keeps them apart. The join has ok = 1 with i >= 64. A flag is
+ * made to send states different ways. So 0 and 1 are not joined where a jump that is not far
+ * tells one from the other: FLOW_NEED_FLAG.
+ *
+ * The states go on to the jumps on their own. Behind the last one the numbers are dead, and what
+ * is left is joined.
+ */
+static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
+			       const struct bpf_verifier_state *b)
+{
+	const struct bpf_reg_state *ra, *rb;
+	const struct flow_need *tested = flow->need[FLOW_NEED_JUMP], *need;
+	u32 fr, i, id = 0;
+	bool one = false;
+
+	if (!tested)
+		return false;
+	for (fr = 0; fr <= a->curframe; fr++) {
+		/* the callers go on when the call returns */
+		need = &tested[fr == a->curframe ? a->insn_idx : a->frame[fr + 1]->callsite + 1];
+		for (i = BPF_REG_0; i < BPF_REG_FP + 1 + 64; i++) {
+			if (i <= BPF_REG_FP ? !(need->regs & BIT(i)) :
+					      !(need->slots & BIT_ULL(i - BPF_REG_FP - 1)))
+				continue;
+			ra = flow_number_at(a->frame[fr], i);
+			rb = flow_number_at(b->frame[fr], i);
+			if (!ra || !rb || !flow_disjoint(ra, rb))
+				continue;
+			if (flow_gap(ra, rb))
+				return true;
+			/* copies of a number are one number */
+			if (one && (!id || id != (ra->id & ~BPF_ADD_CONST)))
+				return true;
+			one = true;
+			id = ra->id & ~BPF_ADD_CONST;
+		}
+	}
+	return flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b);
+}
+
+/*
+ * States that have a join are kept apart while there is room: half of every limit.
+ * After that they are joined, which may lose what the prog needs, and does not run into the limit.
+ */
+static bool flow_has_room(const struct bpf_verifier_env *env, u32 nr_states)
+{
+	u64 len = env->prog->len;
+
+	return nr_states < BPF_FLOW_MAX_STATES / 2 &&
+	       env->flow->live_states < (BPF_FLOW_LIVE_PER_INSN * len + BPF_FLOW_LIVE_BASE) / 2 &&
+	       env->total_states < (BPF_FLOW_TOTAL_PER_INSN * len + BPF_FLOW_TOTAL_BASE) / 2;
+}
-- 
2.55.0

[PATCH bpf-next v4 14/46] bpf: Merge the state that arrives into the set at an insn

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

scx_flow, flow_dispatch() is a row of blocks like this one:
  if (nq0 > 0) {
      node_moved = flow_move_one_hint(node_dsq, cpu, &visits, nq0);
      if (node_moved > left)
          node_moved = left;
      left -= node_moved;
  }
Each way through a block leaves other ranges in left, visits, node_moved.
No check behind the block depends on the way. Today's verifier follows all
paths: 70716 insns for a prog of 1117. With one state where the ways meet
it takes 3127. flow_merge() puts a state S into the set at an insn:

  a state of the set covers S ?  ------ yes -->  drop S
     | no
  S covers a state T of the set ?  ---- yes -->  drop T
     |
  S has a join with a state T,  ------- yes -->  drop T, S = join(S, T),
  and no rule refuses it ?                       start again
     | no
  add S to the set: it is new, the verifier goes on from it

A state is dropped only if a state of the set covers it. Thus the verified
states cover each execution of the prog.

S can come around a cycle: to the head of a loop, to the first insn of
a recursive subprog, from a callback to its helper. There a bound can grow
by one step each time: i=[0,1], [0,2], ... Thus the join is a widening: it
gives up the bound that grew. S and T differ in what the cycle changes, so
the rules that refuse a join are off. These stay on:
 - A constant that becomes a stack offset: bt_print() of scx has
   'stack[depth++] = ind + 1' and 'ind = stack[depth]' in bpf_for().
 - A live register that only S has set. In clang 18's code of
   'bpf_for(..) { if (..) continue; last = rnd(); found = true; }' and
   'if (found) use(last);' the join has no 'last': "R7 !read_ok".
 - The calls of the callback of bpf_loop(N <= 8). What other callbacks come
   back with is joined 4 times, then widened.
All that holds only if S comes from T:
  ret = 0; if (file->f_flags & O_TRUNC) { denied++; ret = -EPERM; }
  bpf_for(i, 0, 100) { sum += a[i]; } total = sum; return ret;
Two states enter the loop. Next to -EPERM the 0 that comes around looks
like a number that grew, and an LSM prog is rejected:
"R0 has smin=0 smax=4294967295 should have been in [-4095, 0]".
Thus a state that enters a loop, or gets to bpf_loop(), has a number, and
so has what becomes of it. S is widened with a T of its number only.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 277 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 277 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index feaa5949a2a1..0774e54f8b2c 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -2189,6 +2189,22 @@ static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_ver
 	return flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b);
 }
 
+/*
+ * @s came back to the head of a loop, or from a callback to the helper that calls it. @t is there.
+ * Do they come from two states that entered the cycle?
+ */
+static bool flow_entries_differ(const struct bpf_verifier_state *t,
+				const struct bpf_verifier_state *s, enum flow_cycle cycle)
+{
+	const struct bpf_func_state *ft = t->frame[t->curframe], *fs = s->frame[s->curframe];
+	u32 cnt = fs->loop_stack_cnt;
+
+	if (cycle == FLOW_CALLBACK_CYCLE)
+		return ft->callback_entry_id != fs->callback_entry_id;
+	return cnt && ft->loop_stack_cnt == cnt &&
+	       ft->loop_stack[cnt - 1].entry_id != fs->loop_stack[cnt - 1].entry_id;
+}
+
 /*
  * States that have a join are kept apart while there is room: half of every limit.
  * After that they are joined, which may lose what the prog needs, and does not run into the limit.
@@ -2201,3 +2217,264 @@ static bool flow_has_room(const struct bpf_verifier_env *env, u32 nr_states)
 	       env->flow->live_states < (BPF_FLOW_LIVE_PER_INSN * len + BPF_FLOW_LIVE_BASE) / 2 &&
 	       env->total_states < (BPF_FLOW_TOTAL_PER_INSN * len + BPF_FLOW_TOTAL_BASE) / 2;
 }
+
+/* The state of @set behind @t, the first one if @t is NULL, that @s may be related to */
+static struct flow_state *flow_next_related(struct bpf_verifier_env *env,
+					    const struct flow_set *set, struct flow_state *s,
+					    struct flow_state *t, bool places)
+{
+	t = t ? list_next_entry(t, node) : list_first_entry(&set->list, struct flow_state, node);
+	list_for_each_entry_from(t, &set->list, node)
+		if (flow_may_relate(env, t, s, places))
+			return t;
+	return NULL;
+}
+
+static void flow_link(struct bpf_verifier_env *env, struct flow_set *set, struct flow_state *s)
+{
+	list_add_tail(&s->node, &set->list);
+	env->flow->nr_states[s->st.insn_idx]++;
+}
+
+static void flow_sweep_trip(struct bpf_verifier_env *env, struct flow_state *t,
+			    const struct flow_state *s)
+{
+	if (t->pending || !flow_earlier_trip(&t->st, &s->st))
+		return;
+	flow_unlink(env, t);
+	flow_free_state(env, t);
+}
+
+/* States of earlier iterations than @s go away, see flow_earlier_trip() */
+static void flow_sweep_trips(struct bpf_verifier_env *env, struct flow_set *set,
+			     const struct flow_state *s)
+{
+	struct flow_state *t, *tn;
+
+	if (!flow_has_earlier_trips(&s->st))
+		return;
+	list_for_each_entry_safe(t, tn, &set->list, node)
+		if (flow_same_shape(t, s, FLOW_CHAIN))
+			flow_sweep_trip(env, t, s);
+}
+
+/*
+ * Put *@ps into @set, one of the sets of states at its insn. Returns
+ * 1 if it is within a state of the set. It is gone then.
+ * 0 if it is in the set now, as it was or joined with states of the set.
+ * @cycle: it came around a cycle, what grew is forgotten.
+ */
+static int flow_merge(struct bpf_verifier_env *env, struct flow_set *set,
+		      struct flow_state **ps, enum flow_cycle cycle)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *s = *ps, *t, *tn;
+	int insn_idx = s->st.insn_idx, err;
+	u32 *nr_states = &flow->nr_states[insn_idx];
+	bool places, widen_all, widen;
+	/* the set of the states that enter a loop is a stop on the way, see bpf_flow_deliver() */
+	bool log = (env->log.level & BPF_LOG_LEVEL2) && set == &flow->states[insn_idx];
+	const char *what = NULL;
+	int prev = s->prev_insn_idx;
+
+again:
+	if (signal_pending(current)) {
+		err = -EAGAIN;
+		goto err_free;
+	}
+	if (need_resched())
+		cond_resched();
+
+	/* again after a join: an offset may not be a constant any more, a pointer may be gone */
+	flow_shape_ctx(&s->st, &s->shape);
+	places = *nr_states >= BPF_FLOW_MIN_SHAPES;
+	for (t = flow_next_related(env, set, s, NULL, places); t;
+	     t = flow_next_related(env, set, s, t, places)) {
+		if (flow_cmp_pkt_range(t, s) > 0 || !flow_trips_near(&t->st, &s->st) ||
+		    !flow_within(env, &t->st, &s->st))
+			continue;
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "from %d to %d: within\n", prev, insn_idx);
+		err = flow_cycle_closed(env, &t->st, &s->st) ?:
+		      flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+		flow_free_state(env, s);
+		return err ?: 1;
+	}
+
+	flow_sweep_trips(env, set, s);
+
+	widen_all = cycle == FLOW_CYCLE;
+	for (t = flow_next_related(env, set, s, NULL, places); t; t = tn) {
+		tn = flow_next_related(env, set, s, t, places);
+		/*
+		 * Iterations that are kept apart have no join. Neither does a state take the place
+		 * of a state of another iteration that it covers: that one is done with, or has its
+		 * own way to go, see flow_cycle_closed().
+		 * The shape is a hash, and what follows goes through the frames of both.
+		 */
+		if (flow_trips_behind(&t->st, &s->st))
+			continue;
+		widen = widen_all || (cycle == FLOW_CALLBACK_CYCLE &&
+				      max(s->joins, t->joins) >= BPF_FLOW_CALLBACK_JOINS);
+		/*
+		 *	ret = 0; if (...) { denied++; ret = -EPERM; }
+		 *	bpf_for(i, 0, n) { ... } return ret;
+		 * Two states enter the loop. What comes around it with 0 is for the state with 0.
+		 * Next to -EPERM it looks like a number that grew, and would be forgotten. The same
+		 * for the header of a packet at pkt + 34 and at pkt + 54. No rule has to tell which
+		 * numbers that holds for: a state knows where it comes from.
+		 * If it has no join with that state it is at the head on its own from now on.
+		 * The same for bpf_loop() in place of the loop.
+		 */
+		if (widen && flow_has_room(env, *nr_states) &&
+		    flow_entries_differ(&t->st, &s->st, cycle))
+			continue;
+		if (!widen && flow_has_room(env, *nr_states) &&
+		    flow_null_differs(flow, &t->st, &s->st))
+			continue;
+		if (!widen && flow_needed_consts_differ(flow, FLOW_NEED_ARG, &t->st, &s->st))
+			continue;
+		if (flow_callback_depth_differs(env, &t->st, &s->st))
+			continue;
+		/* what grows in a loop is not replaced with what is larger, it is forgotten */
+		if (!widen && flow_cmp_pkt_range(s, t) <= 0 && flow_within(env, &s->st, &t->st)) {
+			err = flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+			if (err)
+				goto err_free;
+			/*
+			 * What a callback comes back with and covers what is there grew, like
+			 * what is joined with it. It counts, or nothing ends "*ctx += 1".
+			 */
+			s->joins = max(s->joins, t->joins) + (cycle == FLOW_CALLBACK_CYCLE);
+			flow_unlink(env, t);
+			flow_free_state(env, t);
+			what = what ?: "joined";
+			continue;
+		}
+		if (!widen && flow_cmp_pkt_range(s, t))
+			continue;
+		if (flow_has_room(env, *nr_states) &&
+		    (flow_unset_differs(&t->st, &s->st) ||
+		     (!widen && flow_join_makes_up(flow, &t->st, &s->st))))
+			continue;
+		/*
+		 * Around a loop too:
+		 *	u8 stack[10], depth = 0;
+		 *	loop {
+		 *		if (...) { if (depth >= 10) break; stack[depth++] = x; }
+		 *		else { depth--; if (depth >= 10) break; x = stack[depth]; }
+		 *	}
+		 * depth is in a 32-bit register. A compiler tests 'depth & 255' and computes it
+		 * anew for the index. What is found out about a copy says nothing about
+		 * a range, and everything about a constant.
+		 */
+		if (flow_has_room(env, *nr_states) &&
+		    flow_needed_consts_differ(flow, FLOW_NEED_STACK_OFF, &t->st, &s->st))
+			continue;
+		if (!widen && flow_has_room(env, *nr_states) &&
+		    flow_needed_consts_differ(flow, FLOW_NEED_PTR_OFF, &t->st, &s->st))
+			continue;
+		err = bpf_states_join(env, &flow->tmp, &t->st, &s->st, widen);
+		if (err < 0)
+			goto err_free;
+		if (err)
+			continue;
+		what = widen ? "widened" : "joined";
+		err = flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+		if (err)
+			goto err_free;
+		flow->tmp.flow_entry = t->st.flow_entry;
+		flow->tmp.last_insn_idx = s->prev_insn_idx;
+		s->joins = max(s->joins, t->joins) + 1;
+		bpf_free_verifier_state(&s->st, false);
+		s->st = flow->tmp;
+		memset(&flow->tmp, 0, sizeof(flow->tmp));
+		flow_unlink(env, t);
+		flow_free_state(env, t);
+		/* the join may be within, or have a join with, what is left in the set */
+		goto again;
+	}
+
+	if (*nr_states >= BPF_FLOW_MAX_STATES) {
+		verbose(env, "more than %d states that cannot be joined at insn %d\n",
+			BPF_FLOW_MAX_STATES, insn_idx);
+		err = -E2BIG;
+		goto err_free;
+	}
+	if (flow->live_states > BPF_FLOW_LIVE_PER_INSN * env->prog->len + BPF_FLOW_LIVE_BASE ||
+	    env->total_states >
+	    BPF_FLOW_TOTAL_PER_INSN * (u64)env->prog->len + BPF_FLOW_TOTAL_BASE) {
+		verbose(env, "too many states for %d insns: %d, %d at a time\n",
+			env->prog->len, env->total_states, flow->live_states);
+		err = -E2BIG;
+		goto err_free;
+	}
+	/*
+	 * The state is printed when its turn comes. That says enough about a state that
+	 * is new, and about a state that goes on to the next insn.
+	 */
+	if (prev >= 0 && insn_idx == prev + 1 + bpf_is_ldimm64(&env->prog->insnsi[prev]) &&
+	    what && !strcmp(what, "joined"))
+		what = NULL;
+	if (log && what)
+		verbose(env, "from %d to %d: %s\n", prev, insn_idx, what);
+	flow_link(env, set, s);
+	env->max_states_per_insn = max(env->max_states_per_insn, *nr_states);
+	return 0;
+
+err_free:
+	flow_free_state(env, s);
+	return err;
+}
+
+static struct flow_state *flow_new_state(struct bpf_verifier_env *env, int prev_insn_idx)
+{
+	struct flow_state *s = kzalloc_obj(*s, GFP_KERNEL_ACCOUNT);
+	struct bpf_flow *flow = env->flow;
+
+	if (!s)
+		return NULL;
+	s->prev_insn_idx = prev_insn_idx;
+	s->seq = ++flow->seq;
+	env->total_states++;
+	flow->live_states++;
+	env->peak_states = max(env->peak_states, flow->live_states);
+	return s;
+}
+
+/* What @st holds is taken and @st is left empty. What doesn't matter at @insn_idx goes. */
+static struct flow_state *flow_arrive(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+				      int insn_idx, int prev_insn_idx, bool *enters)
+{
+	struct bpf_verifier_state *cur = env->cur_state;
+	int cur_idx = env->insn_idx, err;
+	struct flow_state *s;
+
+	if (insn_idx < 0 || insn_idx >= env->prog->len) {
+		verbose(env, "invalid insn idx %d insn_cnt %d\n", insn_idx, env->prog->len);
+		return ERR_PTR(-EFAULT);
+	}
+
+	s = flow_new_state(env, prev_insn_idx);
+	if (!s)
+		return ERR_PTR(-ENOMEM);
+	s->st = *st;
+	memset(st, 0, sizeof(*st));
+	s->st.insn_idx = insn_idx;
+	s->st.last_insn_idx = prev_insn_idx;
+
+	env->cur_state = &s->st;
+	env->insn_idx = insn_idx;
+	err = bpf_flow_loop_leave(env, enters);
+	if (!err)
+		err = bpf_clean_verifier_state(env, &s->st);
+	env->cur_state = cur;
+	env->insn_idx = cur_idx;
+	if (err) {
+		flow_free_state(env, s);
+		return ERR_PTR(err);
+	}
+	/* when the registers that it was shared with are dead, or are those of the caller */
+	bpf_clear_singular_ids(env, &s->st);
+	return s;
+}
-- 
2.55.0

[PATCH bpf-next v4 15/46] bpf: Move states along edges, into loops and subprogs and out of them

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest test_verif_scale1 has these lines 90 times. jhash() is static:

  ptr = data + i;
  if (ptr + nh_off > data_end) break;
  ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++);

The 90th call is reached with i = 0 .. 89, in 90 states. All calls: 4095.
jhash() sees one of 90 pointers. If a state in jhash() has the frames of
its callers, as today, states of different calls have no join:
"more than 1024 states that cannot be joined at insn 903". Thus a state in
a static subprog has one frame, and the caller's state waits at the call:

  bpf_flow_call()   saves the caller. The entry state goes to the set at
                    the first insn of the subprog, like any state.
  bpf_flow_exit()   a state at exit goes to all calls that wait for its
                    entry state. The exit states are a set too.

When the merge drops or joins an entry state, its calls wait for the state
that took its place, and flow_do_united() gives them its exit states. 4005
of the 4095 calls end so: 50765 insns, today 530362. Else the limit stops
this test, 100 progs of the schedulers of rust-bpf and 124 of its tests.

The exit states of an entry are joined like the states at an insn, and the
joins must end. Thus an exit state that was joined more than 16 times is
widened. The entry and the exits of a recursive subprog are widened at
once.

Loops. Scalar evolution derives the count and the ranges of a loop from the
state that enters it. Thus the states that enter are a set of their own.
A copy of each new one, with these ranges, goes to the set at the head.
When a walk ends before the loop does, one state stands for the rest:
flow_end_walk() joins, with widening, the state that came back to the head
and the state one iteration before. They differ in what the loop changes.
There is no join if the iteration set a register for the first time, or
changed a constant that becomes an offset of the stack or of a pointer.

bpf_flow_deliver() sends a state along an edge, bpf_flow_next() gives the
next new state. A state that gets to bpf_loop() gets its number there: what
the callback comes back with is widened with the state that it comes from.
An added may_goto is an edge that the prog does not have:
bpf_flow_guard_edge() sends a state along it only if the loop has may_goto
by now. Else it drops the state and notes that. If the loop gets may_goto
later, the prog is verified again.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/flow.c | 392 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 392 insertions(+)

diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 0774e54f8b2c..3d7ec3305e64 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -2478,3 +2478,395 @@ static struct flow_state *flow_arrive(struct bpf_verifier_env *env, struct bpf_v
 	bpf_clear_singular_ids(env, &s->st);
 	return s;
 }
+
+/* Is that where may_goto in front of @insn_idx goes, as far as it's known? */
+static bool flow_guard_edge_is_real(struct bpf_verifier_env *env, int insn_idx, u8 edge)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+
+	return aux[bpf_loop_at_index(env, insn_idx)].loop->guarded &&
+	       aux[insn_idx].guard_no_unwind == (edge == BPF_GUARD_LEAVES_LOOP);
+}
+
+/*
+ * @st gets to @insn_idx if the loop has may_goto in front of @prev_insn_idx and that is where it
+ * goes, see check_loop_guard(). If that is not so, or not known yet, @st is dropped. If it turns
+ * out to be so later all is done again.
+ */
+int bpf_flow_guard_edge(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			int insn_idx, int prev_insn_idx, enum bpf_guard_edge edge)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+
+	if (flow_guard_edge_is_real(env, prev_insn_idx, edge)) {
+		aux[prev_insn_idx].loop_guard = true;
+		return bpf_flow_deliver(env, st, insn_idx, prev_insn_idx);
+	}
+	/* it's not going to leave the subprog any more */
+	if (edge == BPF_GUARD_LEAVES_SUBPROG && aux[prev_insn_idx].guard_no_unwind)
+		return 0;
+	aux[prev_insn_idx].guard_edge_dropped = true;
+	aux[bpf_loop_at_index(env, prev_insn_idx)].loop->no_guard_edges = true;
+	return 0;
+}
+
+/*
+ * @s came back to the head of a loop, and the walk through the iterations ends, see
+ * bpf_flow_loop_enter(). From now on a state stands for many iterations. The first such state is
+ * the join of @s with the state that it comes from, one iteration ago. Both were seen, and what
+ * differs between them is what the loop changes. If they have no join @s goes on as it is.
+ * A state of another walk, which began with another state, has nothing to do with @s.
+ */
+static int flow_end_walk(struct bpf_verifier_env *env, struct flow_set *set, struct flow_state *s)
+{
+	struct bpf_func_state *frame = s->st.frame[s->st.curframe];
+	struct loop_stack_entry *entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	u32 top = frame->loop_stack_cnt - 1, walk = entry->walk_id;
+	struct bpf_flow *flow = env->flow;
+	int insn_idx = s->st.insn_idx, err;
+	struct flow_state *t;
+
+	list_for_each_entry_reverse(t, &set->list, node) {
+		/*
+		 * Some rules of flow_merge() hold here too. One of them: the iteration has set
+		 * a register, see flow_unset_differs(). The join would be without it.
+		 */
+		if (flow_trips_behind(&t->st, &s->st) ||
+		    t->st.frame[t->st.curframe]->loop_stack[top].walk_id != walk ||
+		    (flow_has_room(env, flow->nr_states[insn_idx]) &&
+		     (flow_unset_differs(&t->st, &s->st) ||
+		      flow_needed_consts_differ(flow, FLOW_NEED_STACK_OFF, &t->st, &s->st) ||
+		      flow_needed_consts_differ(flow, FLOW_NEED_PTR_OFF, &t->st, &s->st))))
+			continue;
+		err = bpf_states_join(env, &flow->tmp, &t->st, &s->st, true);
+		if (err < 0)
+			return err;
+		if (err)
+			continue;
+		err = flow_unite(flow, s->st.flow_entry, t->st.flow_entry);
+		if (err)
+			return err;
+		flow->tmp.flow_entry = t->st.flow_entry;
+		flow->tmp.last_insn_idx = s->prev_insn_idx;
+		bpf_free_verifier_state(&s->st, false);
+		s->st = flow->tmp;
+		memset(&flow->tmp, 0, sizeof(flow->tmp));
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "from %d to %d: widened\n", s->prev_insn_idx, insn_idx);
+		break;
+	}
+
+	frame = s->st.frame[s->st.curframe];
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	entry->guarded = env->insn_aux_data[insn_idx].loop->guarded;
+	entry->apart = false;
+	entry->probe = false;
+	entry->decided = false;
+	entry->terminates = false;
+	entry->trip++;
+	entry->max_header_count = 0;
+	return 0;
+}
+
+/*
+ * @s goes into the set of the states at its insn. It is taken.
+ * @enters: it enters a loop.
+ */
+static int flow_add(struct bpf_verifier_env *env, struct flow_state *s, enum flow_cycle cycle,
+		    bool enters)
+{
+	struct bpf_verifier_state *cur = env->cur_state;
+	int cur_idx = env->insn_idx, insn_idx = s->st.insn_idx, err;
+	const struct loop_stack_entry *entry;
+	const struct bpf_func_state *frame;
+	bool backedge = false, drop = false, ends_walk = false;
+	struct bpf_flow *flow = env->flow;
+
+	env->cur_state = &s->st;
+	env->insn_idx = insn_idx;
+	err = bpf_flow_loop_enter(env, &backedge, &drop, &ends_walk);
+	env->cur_state = cur;
+	env->insn_idx = cur_idx;
+	if (!err && ends_walk)
+		err = flow_end_walk(env, &flow->states[insn_idx], s);
+	if (!err && enters) {
+		frame = s->st.frame[s->st.curframe];
+		entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+		/* it has a count, see bpf_flow_loop_gets_guards() */
+		if (entry->terminates && !entry->guarded && !bpf_loop_walk_may_give_up(entry))
+			env->insn_aux_data[insn_idx].loop->no_guard_edges = true;
+	}
+	if (err || drop) {
+		flow_free_state(env, s);
+		return err;
+	}
+
+	err = flow_merge(env, &flow->states[insn_idx], &s, backedge ? FLOW_CYCLE : cycle);
+	if (err)
+		return err < 0 ? err : 0;
+	flow_set_pending(flow, s);
+	return 0;
+}
+
+/*
+ * @st gets to @insn_idx. What it holds is taken and @st is left empty.
+ */
+int bpf_flow_deliver(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+		     int insn_idx, int prev_insn_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct bpf_subprog_info *sub;
+	struct flow_state *s, *e;
+	enum flow_cycle cycle;
+	bool enters = false;
+	int err;
+
+	s = flow_arrive(env, st, insn_idx, prev_insn_idx, &enters);
+	if (IS_ERR(s))
+		return PTR_ERR(s);
+
+	/*
+	 * Cycles that have no loop head: the callback that came back to the helper
+	 * that calls it, and calls that go through the entry of a function.
+	 */
+	sub = bpf_find_containing_subprog(env, insn_idx);
+	cycle = FLOW_NO_CYCLE;
+	if (sub->recursive && sub->start == insn_idx)
+		cycle = FLOW_CYCLE;
+	else if (env->insn_aux_data[insn_idx].calls_callback && prev_insn_idx >= 0 &&
+		 env->prog->insnsi[prev_insn_idx].code == (BPF_JMP | BPF_EXIT))
+		cycle = FLOW_CALLBACK_CYCLE;
+	else if (env->insn_aux_data[insn_idx].calls_callback)
+		s->st.frame[s->st.curframe]->callback_entry_id = ++env->loop_entry_cnt;
+
+	/*
+	 * What is known about a loop is derived from the state that enters it. So
+	 * the states that enter are a set of their own, and what gets to
+	 * the set at the head of the loop is derived from what is new in it.
+	 */
+	if (enters) {
+		err = flow_merge(env, &flow->enter[insn_idx], &s, cycle);
+		if (err)
+			return err < 0 ? err : 0;
+		cycle = FLOW_NO_CYCLE;
+		e = s;
+		s = flow_new_state(env, prev_insn_idx);
+		if (!s)
+			return -ENOMEM;
+		err = bpf_copy_verifier_state(&s->st, &e->st);
+		if (err) {
+			flow_free_state(env, s);
+			return err;
+		}
+	}
+	return flow_add(env, s, cycle, enters);
+}
+
+static void flow_free_call(struct flow_state *w)
+{
+	list_del(&w->node);
+	bpf_free_verifier_state(&w->st, false);
+	kfree(w);
+}
+
+/* @x came out of the function that the call @w waits for. @w may be gone afterwards. */
+static int flow_apply_exit(struct bpf_verifier_env *env, struct flow_state *w,
+			   struct flow_exit *x)
+{
+	int call_idx = w->st.insn_idx, err;
+	struct bpf_verifier_state st = {};
+	struct flow_state *s;
+
+	err = bpf_copy_verifier_state(&st, &w->st);
+	if (!err)
+		err = bpf_flow_return(env, &st, &x->st, call_idx);
+	if (!err)
+		err = bpf_flow_deliver(env, &st, call_idx + 1, call_idx);
+	bpf_free_verifier_state(&st, false);
+	if (err != 1)
+		return err;
+
+	/*
+	 * The call is verified in the context of the caller: it doesn't wait any more, it is
+	 * done again. Not with bpf_flow_deliver(): the state was at the insn before, it may
+	 * be there still.
+	 */
+	s = flow_new_state(env, w->st.last_insn_idx);
+	if (!s)
+		return -ENOMEM;
+	s->st = w->st;
+	memset(&w->st, 0, sizeof(w->st));
+	flow_free_call(w);
+	flow_shape_ctx(&s->st, &s->shape);
+	flow_link(env, &env->flow->states[call_idx], s);
+	flow_set_pending(env->flow, s);
+	return 0;
+}
+
+/* What came out for the set of @from goes to the calls that wait for the set of @to */
+static int flow_apply_exits(struct bpf_verifier_env *env, struct flow_entry *from,
+			    struct flow_entry *to)
+{
+	struct flow_state *w, *wn;
+	struct flow_exit *x;
+	int err;
+
+	list_for_each_entry(x, &from->exits, node) {
+		list_for_each_entry_safe(w, wn, &to->calls, node) {
+			err = flow_apply_exit(env, w, x);
+			if (err)
+				return err;
+		}
+	}
+	return 0;
+}
+
+static int flow_do_united(struct bpf_verifier_env *env)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_entry *a, *b;
+	struct flow_united u;
+	int err;
+
+	while (flow->united_cnt) {
+		u = flow->united[--flow->united_cnt];
+		a = flow_entry_find(flow, u.a);
+		b = flow_entry_find(flow, u.b);
+		if (a == b)
+			continue;
+		err = flow_apply_exits(env, a, b) ?: flow_apply_exits(env, b, a);
+		if (err)
+			return err;
+		list_splice_tail_init(&a->calls, &b->calls);
+		list_splice_tail_init(&a->exits, &b->exits);
+		a->up = b;
+	}
+	return 0;
+}
+
+/*
+ * The call at @call_idx in @caller of a function that is verified on its own,
+ * @entry is the state at its first insn. Both are taken.
+ */
+int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		  struct bpf_verifier_state *entry, int subprog, int call_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *w;
+	int id;
+
+	w = kzalloc_obj(*w, GFP_KERNEL_ACCOUNT);
+	if (!w)
+		return -ENOMEM;
+	w->st = *caller;
+	memset(caller, 0, sizeof(*caller));
+	w->st.insn_idx = call_idx;
+
+	id = flow_entry_new(flow);
+	if (id < 0) {
+		bpf_free_verifier_state(&w->st, false);
+		kfree(w);
+		return id;
+	}
+	list_add_tail(&w->node, &flow->entries[id]->calls);
+
+	entry->flow_entry = id;
+	if (env->log.level & BPF_LOG_LEVEL2)
+		verbose(env, "call at %d waits for func#%d\n", call_idx, subprog);
+	return bpf_flow_deliver(env, entry, env->subprog_info[subprog].start, call_idx);
+}
+
+/* @st came to exit of the function that is verified on its own. It is taken. */
+int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_func *f = &flow->funcs[st->frame[0]->subprogno];
+	u32 id = st->flow_entry, cnt = 0;
+	struct flow_exit *x, *o, *on;
+	struct flow_state *w, *wn;
+	struct flow_entry *entry;
+	u16 joins = 0;
+	int err;
+
+	st->insn_idx = exit_idx;
+	err = bpf_clean_verifier_state(env, st);
+	if (err)
+		return err;
+	if (f->exit_idx < 0)
+		f->exit_idx = exit_idx;
+	st->insn_idx = f->exit_idx;
+	entry = flow_entry_find(flow, id);
+
+	list_for_each_entry_safe(o, on, &entry->exits, node) {
+		if (bpf_state_subsumes(env, &o->st, st))
+			return 0;
+		if (flow_has_room(env, ++cnt) && flow_join_makes_up(flow, &o->st, st))
+			continue;
+		/* what comes out of a recursive function gets into it again */
+		err = bpf_states_join(env, &flow->tmp, &o->st, st,
+				      env->subprog_info[st->frame[0]->subprogno].recursive ||
+				      o->joins > BPF_FLOW_MAX_JOINS);
+		if (err < 0)
+			return err;
+		if (err)
+			continue;
+		joins = max(joins, o->joins) + 1;
+		bpf_free_verifier_state(st, false);
+		*st = flow->tmp;
+		memset(&flow->tmp, 0, sizeof(flow->tmp));
+		st->flow_entry = id;
+		list_del(&o->node);
+		bpf_free_verifier_state(&o->st, false);
+		kfree(o);
+	}
+
+	x = kzalloc_obj(*x, GFP_KERNEL_ACCOUNT);
+	if (!x)
+		return -ENOMEM;
+	x->st = *st;
+	memset(st, 0, sizeof(*st));
+	x->joins = joins;
+	list_add_tail(&x->node, &entry->exits);
+
+	list_for_each_entry_safe(w, wn, &entry->calls, node) {
+		err = flow_apply_exit(env, w, x);
+		if (err)
+			return err;
+	}
+	return 0;
+}
+
+/*
+ * Make the next pending state the current one. env->cur_state has to be empty.
+ * Returns -ENOENT when nothing is pending.
+ */
+int bpf_flow_next(struct bpf_verifier_env *env, int *prev_insn_idx)
+{
+	struct bpf_flow *flow = env->flow;
+	struct flow_state *s;
+	struct rb_node *n;
+	int err;
+
+	do {
+		err = flow_do_united(env);
+		if (err)
+			return err;
+	} while (flow->united_cnt);
+	n = rb_first_cached(&flow->pending);
+	if (!n)
+		return -ENOENT;
+	s = rb_entry(n, struct flow_state, rb);
+	rb_erase_cached(&s->rb, &flow->pending);
+	s->pending = false;
+
+	env->insn_idx = s->st.insn_idx;
+	*prev_insn_idx = s->prev_insn_idx;
+	if (test_bit(s->st.insn_idx, flow->keep))
+		return bpf_copy_verifier_state(env->cur_state, &s->st);
+	*env->cur_state = s->st;
+	memset(&s->st, 0, sizeof(s->st));
+	s->st.insn_idx = env->insn_idx;
+	flow_unlink(env, s);
+	flow_free_state(env, s);
+	return 0;
+}
-- 
2.55.0

Re: [PATCH bpf-next v4 15/46] bpf: Move states along edges, into loops and subprogs and out of them

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2816 bytes --]

> bpf: Move states along edges, into loops and subprogs and out of them

TL;DR: Speculative exit states may lose their speculative flag when returning
from a subprog, so programs that previously got a nospec fix may now be
rejected (stricter, not unsafe); suggests carrying the flag over.

> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> --- a/kernel/bpf/flow.c
> +++ b/kernel/bpf/flow.c

[ ... ]

> +int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx)
> +{

[ ... ]

> +	x = kzalloc_obj(*x, GFP_KERNEL_ACCOUNT);
> +	if (!x)
> +		return -ENOMEM;
> +	x->st = *st;
> +	memset(st, 0, sizeof(*st));
> +	x->joins = joins;
> +	list_add_tail(&x->node, &entry->exits);
> +
> +	list_for_each_entry_safe(w, wn, &entry->calls, node) {
> +		err = flow_apply_exit(env, w, x);

Can a speculative exit state lose its speculative flag here?

bpf_flow_exit() accepts any state that reaches BPF_EXIT in frame 0 with a
flow_entry, including states pushed by sanitize_speculative_path().  Those
states are kept apart from the non-speculative ones by bpf_state_subsumes()
and join_check_state(), so they end up as their own flow_exit.

flow_apply_exit() then builds the continuation from a copy of w->st.  That
copy is never speculative, because func_is_on_its_own() returns false when
state->speculative, and bpf_flow_return() copies R0 and the refs from the
exit but not the speculative flag.  The result is delivered to call_idx + 1
as an ordinary state.

Before this series, a speculative path that reached the callee's exit went
on in the caller through prepare_func_exit() with speculative still set,
so do_check_block() could set insn_aux->nospec for a failure that was
recoverable with nospec:

    error_recoverable_with_nospec(err) && state->speculative

With this code the same value reaches the caller in a non-speculative
state, so the error rejects the program.  For example, a loader with
CAP_BPF but without CAP_PERFMON calls a static subprog with a constant
flag, and inside it "if (flag == 1) return 0; return 1000;" is decided, so
the fallthrough is pushed speculatively and exits with R0 = 1000.  If the
caller uses the returned value as a map value offset, that access used to
get a nospec and now fails with "invalid access to map value".

This makes the verifier stricter rather than unsafe.  Would carrying the
flag into the continuation work, for example:

    st.speculative |= x->st.speculative;

before bpf_flow_deliver() in flow_apply_exit()?  No later commit in the
series appears to carry the flag across the return.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

[PATCH bpf-next v4 16/46] bpf: Make the state after a call from the caller's state and an exit state

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

match_one() of scx_layered runs in three nested bpf_for() loops and calls
a static subprog, get_pid_nr_ns():

  scoped_guard(rcu) { ... nspid = get_pid_nr_ns(p_pid, pid_ns);

It is verified for the states at its first insn, not for each chain of
calls. One exit state is the result for all calls that wait:

  2: call f  --- { r1=scalar } --->  +---------------------+
  7: call f  --- { r1=scalar } --->  | f: verified 1 time  |
  3: ...     <-- { r0=[0,7] } -----  |                     |
  8: ...     <-- { r0=[0,7] } -----  +---------------------+

That works only if calls that differ in ids give the same entry state. Here
the callers hold three iterators, and each bpf_iter_num_new() has a new id.
What is held is a part of the entry state: it decides what the subprog may
do. In the subprog the id of refs[i] of the caller becomes a name,
BPF_INHERITED_ID + i. A name stands for one object of the caller. Thus
check_ids() and the join do not rename it: it is equal only to itself.

If a call with something held keeps the frames of the callers instead,
layered_runnable takes 5219 insns, not 4459, and recursion has no end.
core::fmt of Rust is recursive, and progs format a message while they hold
a reference. 27 progs of the schedulers of rust-bpf fail:
"the call stack of 33 frames is too deep".

bpf_flow_return() makes the state after the call:
 - r0, and r2 of a pair, come from the exit state. r1-r5 and the outgoing
   stack arguments are not set.
 - What is held is what the caller held and what the subprog took. The
   names become the ids of the caller again.
 - All other ids get new values for each call. If not, two calls in
   sequence return the same object.

Selftest test_spin_lock, lock_static_subprog_unlock():

  bpf_spin_lock(&lockA);
  ret = static_subprog_unlock(ctx);

An exit state does not tell what a release does to the caller. Thus the
frame counts what it holds without interruption since its entry. If that is
less than the caller held, the function returns 1, and the call is verified
again with the frames of the caller. Also: rcu_read_lock, preempt_lock. The
same if it changed the packet through a function that is its argument.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  18 +++++
 kernel/bpf/join.c            |   3 +
 kernel/bpf/states.c          |   4 +
 kernel/bpf/verifier.c        | 138 +++++++++++++++++++++++++++++++++++
 4 files changed, 163 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 700f5bb41f83..75f5c30027f8 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1100,6 +1100,8 @@ struct bpf_subprog_info {
 	 * a hidden pointer to the result, a pair in two registers, no dead arguments.
 	 */
 	bool stack_args_from_code: 1;
+	/* it releases what its caller holds, so it's verified in the context of such caller */
+	bool releases_inherited: 1;
 	/* it loads the end of the packet from the context */
 	bool loads_pkt_end: 1;
 	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
@@ -1134,6 +1136,20 @@ struct bpf_id_pair {
 	u32 cur;
 };
 
+/*
+ * A function that is verified on its own, see func_is_on_its_own(), has no ids of its caller.
+ * It has names for the ids of what is held: BPF_INHERITED_ID + i is the id of refs[i] at the call.
+ * - A name depends only on the place. Each id in the state at the entry is replaced with its name.
+ *   The ids of the caller can be names that it got from its own caller: they are replaced too.
+ * - A name stands for the same thing in all states of the function. It is equal only to itself:
+ *   check_ids() and jid_note() do not rename it. So two calls share a state at the entry, and what
+ *   comes out for it, only if the same places have the same names.
+ * - bpf_flow_return() puts the ids of the call that waits back. All other ids get new values.
+ * - env->id_gen stays below BPF_MAX_ID. So no id is a name or has a bit of BPF_ADD_CONST.
+ */
+#define BPF_INHERITED_ID	0x20000000
+#define BPF_MAX_ID		(BPF_INHERITED_ID / 2)
+
 struct bpf_idmap {
 	u32 tmp_id_gen;
 	u32 cnt;
@@ -1615,6 +1631,8 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
 		    bool widen);
 void bpf_join_free(struct bpf_verifier_env *env);
 
+int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		    const struct bpf_verifier_state *exit, int call_idx);
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
 u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 4da051767e1d..0110e5f85003 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -84,6 +84,9 @@ static void jid_note(struct bpf_join_state *js, u32 ida, u32 idb, bool consisten
 
 	if (!a || !b)
 		return;
+	/* a name of what the caller has, see BPF_INHERITED_ID */
+	if (a != b && (a | b) >= BPF_INHERITED_ID)
+		consistent = false;
 	for (i = 0; i < js->npairs; i++) {
 		p = &js->pairs[i];
 		if (p->a == a && p->b == b) {
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 4136a708a78e..9d3a86b5820e 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -350,6 +350,10 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
 	if (old_id == 0) /* cur_id == 0 as well */
 		return true;
 
+	/* a name of what the caller has, see BPF_INHERITED_ID */
+	if (((old_id | cur_id) & ~BPF_ADD_CONST) >= BPF_INHERITED_ID)
+		return old_id == cur_id;
+
 	for (i = 0; i < idmap->cnt; i++) {
 		if (map[i].old == old_id)
 			return map[i].cur == cur_id;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d42c67d6d70b..5e66d6768fcf 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11757,8 +11757,146 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return 0;
 }
 
+
 static int process_bpf_exit_full(struct bpf_verifier_env *env,
 				 bool *do_print_state, bool exception_exit);
+static void account_current_path(struct bpf_verifier_env *env);
+
+/*
+ * The function can be verified on its own, for the state at its entry, when
+ * what it does depends on its arguments only and it cannot get to what its
+ * callers have: nothing points to their stack.
+ *
+ * What the callers hold, references, locks and so on, is a part of that state:
+ * it matters for what the function may do. Calls that differ in ids only
+ * have to get to the same state, so the ids of what is held are replaced with
+ * BPF_INHERITED_ID + the position for the function, and put back at the return.
+ */
+
+static u32 fresh_id(struct bpf_verifier_env *env, struct bpf_idmap *idmap, u32 id)
+{
+	u32 flags = id & BPF_ADD_CONST, i;
+
+	id &= ~BPF_ADD_CONST;
+	if (!id)
+		return 0;
+	for (i = 0; i < idmap->cnt; i++)
+		if (idmap->map[i].old == id)
+			return idmap->map[i].cur | flags;
+	if (idmap->cnt == BPF_ID_MAP_SIZE)
+		return 0;
+	idmap->map[idmap->cnt].old = id;
+	idmap->map[idmap->cnt].cur = ++env->id_gen;
+	return idmap->map[idmap->cnt++].cur | flags;
+}
+
+/*
+ * @caller waits at the call at @call_idx and @exit is what came out of the
+ * function. Make @caller what it is after the call.
+ * Returns 1 if the call has to be verified in the context of the caller.
+ */
+int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		    const struct bpf_verifier_state *exit, int call_idx)
+{
+	struct bpf_func_state *frame = caller->frame[caller->curframe];
+	const struct bpf_func_state *callee = exit->frame[0];
+	struct bpf_idmap *idmap = &env->idmap_scratch;
+	u32 caller_rcu = caller->active_rcu_locks, caller_preempt = caller->active_preempt_locks;
+	struct bpf_reference_state *held;
+	struct bpf_reg_state *reg;
+	u32 i, j, nregs, nheld, kept = 0, id;
+	int err;
+
+	if (exit->unwound && !bpf_is_unwind_goto(&env->prog->insnsi[call_idx + 1])) {
+		verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
+			call_idx);
+		return -EINVAL;
+	}
+	caller->unwound = exit->unwound;
+
+	/*
+	 * bpf_check_cfg() takes the subprog that loads the address of a function for the
+	 * one that calls it. This one got the address from its caller. What it means for
+	 * the packet pointers of the caller is found out in the context of the caller.
+	 */
+	if (exit->changed_pkt_data) {
+		env->subprog_info[callee->subprogno].changes_pkt_data = true;
+		return 1;
+	}
+
+	for (i = 0; i < CALLER_SAVED_REGS; i++)
+		bpf_mark_reg_not_init(env, &frame->regs[caller_saved[i]]);
+	for (i = 0; i < frame->out_stack_arg_cnt; i++)
+		bpf_mark_reg_not_init(env, &frame->stack_arg_regs[i]);
+
+	/* what is held now is what was held at the call and what the function took */
+	held = caller->refs;
+	nheld = caller->acquired_refs;
+	caller->refs = NULL;
+	caller->acquired_refs = 0;
+	err = copy_reference_state(caller, exit);
+	if (err) {
+		kfree(held);
+		return err;
+	}
+
+	/*
+	 * The same state comes out for every call. Ids in it have to be new
+	 * every time, or two calls in a row return the same object.
+	 */
+	idmap->cnt = 0;
+	for (i = 0; i < nheld; i++) {
+		idmap->map[idmap->cnt].old = BPF_INHERITED_ID + i;
+		idmap->map[idmap->cnt++].cur = held[i].id;
+	}
+	/*
+	 * What is taken is added at the end. What is released leaves no hole. Only what the
+	 * function held all the time is kept: the state at its exit looks the same when it
+	 * released a lock and took it again, or took another lock that has no id either.
+	 */
+	for (i = 0; i < nheld && i < callee->entry_refs; i++) {
+		/* see inherited_id() */
+		for (j = 0; held[j].id != held[i].id; j++)
+			;
+		id = held[i].id ? BPF_INHERITED_ID + j : 0;
+		if (caller->refs[i].id != id || caller->refs[i].type != held[i].type)
+			break;
+		caller->refs[i] = held[i];
+		kept++;
+	}
+	kfree(held);
+	/* What it means for the caller is found out in the context of the caller. */
+	if (kept != nheld || callee->entry_rcu_locks < caller_rcu ||
+	    callee->entry_preempt_locks < caller_preempt) {
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "func#%d releases what its caller holds at insn %d\n",
+				callee->subprogno, call_idx);
+		env->subprog_info[callee->subprogno].releases_inherited = true;
+		return 1;
+	}
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		reg = &frame->regs[ret_regs[i]];
+		*reg = callee->regs[ret_regs[i]];
+		reg->id = fresh_id(env, idmap, reg->id);
+		reg->parent_id = fresh_id(env, idmap, reg->parent_id);
+		reg->map_uid = fresh_id(env, idmap, reg->map_uid);
+	}
+	/* those that were held are the first */
+	for (i = nheld; i < caller->acquired_refs; i++) {
+		caller->refs[i].id = fresh_id(env, idmap, caller->refs[i].id);
+		if (caller->refs[i].type == REF_TYPE_PTR)
+			caller->refs[i].parent_id = fresh_id(env, idmap, caller->refs[i].parent_id);
+	}
+	caller->active_irq_id = fresh_id(env, idmap, caller->active_irq_id);
+	caller->active_lock_id = fresh_id(env, idmap, caller->active_lock_id);
+	/* fresh_id() had no room for an id, or there are no ids left */
+	if (idmap->cnt == BPF_ID_MAP_SIZE || env->id_gen >= BPF_MAX_ID) {
+		verbose(env, "too many ids at the call at insn %d\n", call_idx);
+		return -E2BIG;
+	}
+	return 0;
+}
 
 /*
  * Call of a static subprog. The callee is verified in the context of
-- 
2.55.0

Re: [PATCH bpf-next v4 16/46] bpf: Make the state after a call from the caller's state and an exit state

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] bpf: missing exit->speculative propagation in bpf_flow_return()
--

commit 96816fe78c2229631b095b8d13a8d312f8d07492
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Make the state after a call from the caller's state and an exit state

This commit adds bpf_flow_return() and BPF_INHERITED_ID to construct the
caller's post-call verifier state from the waiting caller state and the
callee's exit state when verifying a static subprogram on its own.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index d42c67d6d70b2..5e66d6768fcf7 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -11757,8 +11757,146 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
[ ... ]
> +int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
> +		    const struct bpf_verifier_state *exit, int call_idx)
> +{
> +	struct bpf_func_state *frame = caller->frame[caller->curframe];
> +	const struct bpf_func_state *callee = exit->frame[0];
> +	struct bpf_idmap *idmap = &env->idmap_scratch;
> +	u32 caller_rcu = caller->active_rcu_locks, caller_preempt = caller->active_preempt_locks;
> +	struct bpf_reference_state *held;
> +	struct bpf_reg_state *reg;
> +	u32 i, j, nregs, nheld, kept = 0, id;
> +	int err;
> +
> +	if (exit->unwound && !bpf_is_unwind_goto(&env->prog->insnsi[call_idx + 1])) {
> +		verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
> +			call_idx);
> +		return -EINVAL;
> +	}
> +	caller->unwound = exit->unwound;

[Severity: High]
Should bpf_flow_return() also propagate exit->speculative to
caller->speculative?

When a subprogram verified on its own spawns a speculative path via
sanitize_speculative_path() that reaches BPF_EXIT, caller->speculative is
false in the waiting caller state.  Leaving caller->speculative false on
return treats the speculative return path as architectural execution in the
caller: instructions reached only along that path are marked seen (skipping
dead-code sanitization), speculative return bounds in R0/R2 are joined into
the caller's architectural state, and recoverable speculative errors in the
caller are rejected instead of being mitigated with a speculation barrier.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=16

[PATCH bpf-next v4 17/46] bpf: Enter and leave loops in the main pass, build flow.c

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The main pass joins the states that meet at an insn. At the head of a loop
with a join alone real progs are rejected. bpf_flow_loop_enter() has three
methods for a state that enters a loop, and tries them in this order:
1. Count. scx_lavd/lavd_dispatch:
  for (int j = 0; j < LAVD_CPDOM_MAX_NR; j++) {
      ...
      cpdom_id = get_neighbor_id(cpdomc, i, j);
Scalar evolution finds at most 128 iterations. The state gets j = [0, 127]
and covers all of them: one pass through the body, no may_goto, as today.

2. Walk: each iteration has its own state. Selftest netif_receive_skb:
  for (i = 0; i < len; i++) {
      delta = s1[i] - s2[i];
      if (delta || s1[i] == 0 || s2[i] == 0)
          break;
The compiler tests 'i - 9 < 247', unsigned: no count. After a join i is
unknown: "math between map_value pointer and register with unbounded .."
In a walk i is a constant and decides its test.
flow_try_walk() starts a walk. It goes on while each iteration but the
first decides a jump that leaves the loop, for at most 4096 insns, or two
iterations of a longer loop. A loop with calls gets two iterations: its
size says nothing about the subprogs that it calls. strobemeta.h with
NO_UNROLL, SUBPROGS and two maps:
  for (int i = 0; i < STROBE_MAX_MAPS; ++i)
      payload_off = read_map_var(cfg, i, ..., data, payload_off);
The compiler makes a flag of i: no count. Only a walk bounds payload_off.
A loop with two entries is walked whatever its jumps do. bpfjailer:
  for (int i = BPFJ_MAX_POD_PER_PID - 1; i >= 0; --i) {
      if (i >= num_pods)
          continue;
The compiler copies the test in front of the loop and jumps into its
middle. Scalar evolution says "unsupported loop: irreducible". With a join
14 progs get a may_goto in a loop of 8 iterations.

3. Join, widen, add may_goto. Test hash_fnv_bytes of rust-bpf:
  for b in bytes {
      h = (h ^ *b as u64).wrapping_mul(0x100000001b3);
The length is unknown and h differs in each iteration. Today that is
"BPF program is too large. Processed 1000001 insn". Now the walk ends in
the second iteration, flow.c widens h to an unknown scalar, and the state
that returns next is covered: 160 insns. That shows that each iteration is
safe, not that the loop ends. Thus the loop gets a may_goto. If there is
no place for one: "loop at insn N never exits".

bpf_flow_loop_leave() removes the loops that an edge leaves. flow.c and
the two functions call each other, so flow.o is built from here on, and
do_check() does not use it yet. A loop that leaks a reference in each
iteration has no fixpoint: check_refs_taken_in_loop() rejects a state that
returns to the head with a reference that no register and no slot holds.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  80 +++++++++++
 kernel/bpf/Makefile          |   2 +-
 kernel/bpf/verifier.c        | 262 +++++++++++++++++++++++++++++++++++
 3 files changed, 343 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 75f5c30027f8..6745c92ad586 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -364,6 +364,13 @@ struct bpf_loop_iters {
 	struct cnum64 iv_range;
 };
 
+/* Where may_goto that the verifier adds goes, see check_loop_guard() */
+enum bpf_guard_edge {
+	BPF_NO_GUARD_EDGE,
+	BPF_GUARD_LEAVES_SUBPROG,
+	BPF_GUARD_LEAVES_LOOP,
+};
+
 struct loop_stack_entry {
 	struct bpf_verifier_state *entry_state;
 	u32 min_header_count;
@@ -379,6 +386,10 @@ struct loop_stack_entry {
 	u32 apart:1;
 	/* max_header_count follows from the state at the entry, not from patience */
 	u32 counted:1;
+	/* the walk goes on for as long as each iteration decides a jump that leaves the loop */
+	u32 probe:1;
+	/* this one did */
+	u32 decided:1;
 	/*
 	 * How many times the state came back to the head of the loop. With @apart that is
 	 * the iteration that it is in. A state that stands for many iterations is a join, and has
@@ -387,6 +398,17 @@ struct loop_stack_entry {
 	u32 trip;
 	/* the number of references that the state had when it entered the loop */
 	u32 refs;
+	/*
+	 * Which of the states that entered the loop this one comes from. What comes back to
+	 * the head is for that state, see flow_merge(). A join has the number of one of the two.
+	 */
+	u32 entry_id;
+	/*
+	 * A walk on patience: which one it is, and how many insns may be looked at in it, see
+	 * env->walk_insns. The size of the loop says nothing about the functions that it calls.
+	 */
+	u32 walk_id;
+	u32 walk_patience;
 	int iv_reg;
 	struct cnum64 iv_range;
 };
@@ -447,6 +469,8 @@ struct bpf_func_state {
 	 *                           | number of simulations is tracked in frame N
 	 */
 	u32 callback_depth;
+	/* Which of the states that got to the call this one comes from, see flow_merge() */
+	u32 callback_entry_id;
 	/* Instructions processed in this frame and callees on the current path. */
 	u32 insns_subtotal;
 	/*
@@ -603,6 +627,11 @@ struct bpf_verifier_state {
 	 * and their packet pointers are not in the state, see bpf_flow_return().
 	 */
 	bool changed_pkt_data;
+	/*
+	 * Not zero in a function that is verified on its own: the call that the
+	 * state at its entry came from, see bpf_flow_call().
+	 */
+	u32 flow_entry;
 
 	/*
 	 * First and last insn idx of this verifier state.
@@ -807,8 +836,23 @@ struct bpf_loop {
 	bool exits_overflow;
 	/* @exits of a loop in this one is cut. Then @exits of this loop can lack exits too. */
 	bool nested_exits_overflow;
+	/* every trip goes through an insn that stops the loop, see flow_is_bound() */
+	bool rt_bounded;
+	/* insns in the loop, nested loops included */
+	u32 nr_insns;
+	/*
+	 * a walk on patience whose jumps were all decided, or the two trips of a loop with calls,
+	 * didn't get to the end of the loop: there is no other try
+	 */
+	bool walk_failed;
+	/* functions of the program are called in the loop */
+	bool has_calls;
+	/* not rt_bounded: there are places for may_goto, see guard_to in bpf_insn_aux_data */
+	bool has_guards;
 	/* and it is going to be there, see bpf_flow_loop_gets_guards() */
 	bool guarded;
+	/* a state went through it without the edges of may_goto, see bpf_flow_loop_gets_guards() */
+	bool no_guard_edges;
 	/*
 	 * Loop-entry registers used to compute stack addresses for loads,
 	 * stores and calls requiring fixed stack offsets, in this and nested
@@ -868,6 +912,9 @@ struct bpf_insn_aux_data {
 	DECLARE_BITMAP(may_write_mask, MAX_BPF_STACK_SLOTS);
 	DECLARE_BITMAP(live_stack_before, MAX_BPF_STACK_SLOTS);
 	u16 stack_ptrs; /* bitmask of regs that may hold a frame pointer here (arg_track) */
+	/* regs that hold a number or a pointer to arena; that, or a pointer to a plain map value */
+	u16 plain_regs;
+	u16 value_regs;
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
 	u64 nospec:1; /* do not execute this instruction speculatively */
@@ -885,6 +932,7 @@ struct bpf_insn_aux_data {
 	u64 guard_place:1; /* it's the place for may_goto if the loop needs one */
 	u64 guard_at_header:1; /* there is no better place than the head of the loop */
 	u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
+	u64 guard_edge_dropped:1; /* an edge of may_goto from here was dropped */
 	u64 need_scev:1;
 	u64 alu_state:5; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
@@ -1068,6 +1116,7 @@ struct bpf_subprog_info {
 	bool can_unwind: 1; /* has unwind_exit */
 	bool recursive: 1; /* on a cycle of the call graph */
 	bool in_recursion: 1; /* recursive, or a recursive subprog gets to it */
+	bool addr_taken: 1; /* callx calls nothing else */
 	bool via_callx: 1; /* callx calls it */
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
@@ -1283,6 +1332,11 @@ struct bpf_verifier_env {
 	bool test_state_freq;		/* test verifier with different pruning frequency */
 	bool test_reg_invariants;	/* fail verification on register invariants violations */
 	bool uses_arena_sp;		/* the program has insns with BPF_REG_ARENA_SP */
+	bool restart;			/* see bpf_flow_next_is_no_bound() */
+	bool guards_late;		/* see bpf_flow_loop_gets_guards() */
+	u32 loop_entry_cnt;		/* see loop_stack_entry.entry_id, callback_entry_id */
+	u32 walk_cnt;			/* see loop_stack_entry.walk_id */
+	u32 *walk_insns;		/* the insns that were looked at in each of them */
 	u32 max_arena_stack_depth;	/* of all call chains */
 	struct bpf_verifier_state *cur_state; /* current verifier state */
 	/* Search pruning optimization, array of list_heads for
@@ -1430,6 +1484,7 @@ struct bpf_verifier_env {
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
 	struct scev *scev;
+	struct bpf_flow *flow;
 	struct bpf_join_state *join;
 };
 
@@ -1631,6 +1686,31 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
 		    bool widen);
 void bpf_join_free(struct bpf_verifier_env *env);
 
+/* flow.c */
+int bpf_flow_prepare(struct bpf_verifier_env *env);
+void bpf_flow_reset(struct bpf_verifier_env *env);
+void bpf_flow_free(struct bpf_verifier_env *env);
+bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx);
+struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
+					       const struct bpf_verifier_state *st);
+int bpf_flow_deliver(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+		     int insn_idx, int prev_insn_idx);
+int bpf_flow_guard_edge(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			int insn_idx, int prev_insn_idx, enum bpf_guard_edge edge);
+int bpf_flow_next(struct bpf_verifier_env *env, int *prev_insn_idx);
+int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+		  struct bpf_verifier_state *entry, int subprog, int call_idx);
+int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx);
+void bpf_flow_next_is_no_bound(struct bpf_verifier_env *env, int insn_idx, int h);
+bool bpf_flow_loops_lost_bound(struct bpf_verifier_env *env);
+void bpf_flow_loop_gets_guards(struct bpf_verifier_env *env, int h);
+void bpf_flow_new_round(struct bpf_verifier_env *env);
+int bpf_flow_place_guards(struct bpf_verifier_env *env);
+int bpf_flow_compute_kinds(struct bpf_verifier_env *env);
+bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r);
+/* verifier.c, for flow.c */
+int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters);
+int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop, bool *ends_walk);
 int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
 		    const struct bpf_verifier_state *exit, int call_idx);
 int bpf_insert_loop_guards(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 3e0b096cfc9f..ab3ba42596ec 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o flow.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5e66d6768fcf..5faea922699a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21368,6 +21368,268 @@ static int handle_loop_entry_exit(struct bpf_verifier_env *env)
 	return 0;
 }
 
+/*
+ * The current state came back to the head of a loop. If it has a reference more than the states
+ * that are there it is neither within one of them nor joined with one. It goes around again:
+ *	bpf_for(i, 0, n) { if (!p) p = bpf_ringbuf_reserve(...); ... }
+ * That ends when the loop stops taking references, or when there is no place left to keep them.
+ * A reference that no live register and no live stack slot holds cannot be released any more.
+ * There is nothing to wait for.
+ *
+ * Only what was taken in the loop is looked at. New references are added at the end. What callers
+ * hold is not in the state of a function that is verified on its own.
+ */
+static int check_refs_taken_in_loop(struct bpf_verifier_env *env,
+				    const struct loop_stack_entry *entry)
+{
+	u32 mask = (1 << STACK_SPILL) | (1 << STACK_DYNPTR) | (1 << STACK_ITER);
+	struct bpf_verifier_state *st = env->cur_state;
+	struct bpf_stack_state *stack;
+	struct bpf_func_state *frame;
+	struct bpf_reg_state *reg;
+	bool held;
+	int i, j, id;
+
+	for (i = entry->refs; i < st->acquired_refs; i++) {
+		if (st->refs[i].type != REF_TYPE_PTR)
+			continue;
+		id = st->refs[i].id;
+		held = false;
+		for (j = 0; j < st->acquired_refs; j++)
+			if (st->refs[j].type == REF_TYPE_PTR && st->refs[j].parent_id == id)
+				held = true;
+		bpf_for_each_reg_in_vstate_mask(st, frame, reg, stack, mask, ({
+			if (reg->id == id || reg->parent_id == id)
+				held = true;
+		}));
+		if (held)
+			continue;
+		verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", id, st->refs[i].insn_idx);
+		verbose(env, "nothing holds it in the next iteration of the loop at insn %d\n",
+			entry->loop_id);
+		return -EINVAL;
+	}
+	return 0;
+}
+
+/* How many insns it may take to go through the iterations of a loop one by one. */
+#define BPF_FLOW_MAX_WALK	100000
+/* The same when it is not known that the walk gets to the end of the loop. */
+#define BPF_FLOW_MAX_TRY	4096
+
+/* How many times the current insn is looked at because of the loops around it that are walked. */
+static u64 flow_walk_factor(const struct bpf_verifier_state *st)
+{
+	const struct loop_stack_entry *e;
+	u64 factor = 1;
+	u32 fr, i;
+
+	for (fr = 0; fr <= st->curframe; fr++) {
+		for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++) {
+			e = &st->frame[fr]->loop_stack[i];
+			/* the walk of a loop with two entries pays as it goes, see env->walk_insns */
+			if (e->counted || e->probe)
+				factor = min_t(u64, factor * max(e->max_header_count, 1U), U32_MAX);
+		}
+	}
+	return factor;
+}
+
+/*
+ * The current state enters the loop of @entry, for which there is no count. It's found out on the
+ * way whether the loop has an end: the walk goes on for as long as the jumps that leave the loop
+ * are decided. Returns 1 if the iterations are kept apart from now on, 0 if that takes too long.
+ */
+static int flow_try_walk(struct bpf_verifier_env *env, struct loop_stack_entry *entry)
+{
+	struct bpf_loop *loop = env->insn_aux_data[entry->loop_id].loop;
+	u64 factor = flow_walk_factor(env->cur_state);
+	u32 patience, *walk_insns;
+
+	patience = env->allow_uninit_stack && !loop->rt_bounded &&
+		   (!loop->has_calls || loop->has_guards) ? BPF_FLOW_MAX_TRY : 0;
+	if (loop->walk_failed)
+		patience = 0;
+	/* Two trips for a loop of any length. Many a loop is left in the second one. */
+	if (patience && factor == 1)
+		patience = max_t(u64, patience, 2ULL * loop->nr_insns);
+	if (factor * loop->nr_insns > patience)
+		return 0;
+	entry->max_header_count = max(patience / (u32)(factor * loop->nr_insns), 1U);
+	/*
+	 * A compiler makes a flag of the counter of
+	 *	for (i = 0; i < 2; i++) off += f(i);
+	 * and there is nothing to count. In the second trip the flag decides the jump.
+	 * A loop with calls gets these two trips, whatever they take, and no more: its size says
+	 * nothing about the functions that it calls. It gets may_goto at once, so that nothing is
+	 * done again when the second trip comes back to the head.
+	 */
+	if (loop->has_calls) {
+		entry->max_header_count = min(entry->max_header_count, 2U);
+		patience = U32_MAX;
+		bpf_flow_loop_gets_guards(env, entry->loop_id);
+		entry->guarded = true;
+	}
+	entry->terminates = true;
+	/* scalar evolution has nothing to say about a loop with two entries */
+	entry->probe = !loop->irreducible;
+	entry->walk_id = env->walk_cnt + 1;
+	if (is_power_of_2(entry->walk_id)) {
+		walk_insns = kvrealloc(env->walk_insns, 2 * entry->walk_id * sizeof(u32),
+				       GFP_KERNEL_ACCOUNT);
+		if (!walk_insns)
+			return -ENOMEM;
+		env->walk_insns = walk_insns;
+	}
+	env->walk_cnt++;
+	env->walk_insns[entry->walk_id] = 0;
+	entry->walk_patience = patience;
+	entry->apart = true;
+	return 1;
+}
+
+/*
+ * The current state gets to env->insn_idx. Leave the loops that the insn is not in.
+ * @enters: the insn is the head of a loop that the state is not in yet.
+ */
+int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters)
+{
+	struct bpf_func_state *frame = cur_func(env);
+	int err;
+
+	err = loop_stack_pop(env);
+	if (err)
+		return err;
+	*enters = env->insn_aux_data[env->insn_idx].loop &&
+		  (!frame->loop_stack_cnt ||
+		   frame->loop_stack[frame->loop_stack_cnt - 1].loop_id != env->insn_idx);
+	return 0;
+}
+
+/*
+ * Enter the loops that env->insn_idx is in.
+ *
+ * When the number of iterations follows from the state that enters a loop at its head
+ * induction variables get the ranges that they have in that many iterations, and the loop
+ * needs no may_goto.
+ * The rest of the state is left as it is. What comes back to the head is joined with it.
+ * @ends_walk: the state came back to the head and is the last one that is kept apart from
+ * the states of other iterations.
+ */
+int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop, bool *ends_walk)
+{
+	struct bpf_verifier_state *cur = env->cur_state, *entry_state;
+	const struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *entry;
+	struct bpf_loop_iters iters;
+	struct bpf_loop *loop;
+	bool pushed, decided;
+	u32 trip;
+	u64 per_trip;
+	int err;
+
+	err = loop_stack_push(env, &pushed);
+	if (err)
+		return err;
+
+	if (frame->loop_stack_cnt == 0)
+		return 0;
+
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	if (env->insn_idx != entry->loop_id)
+		return 0;
+
+	/*
+	 * No edge goes from a call to itself: the call goes on with one function of a set, see
+	 * split_func_set(), in the same iteration.
+	 */
+	if (!pushed && cur->last_insn_idx == env->insn_idx &&
+	    BPF_CLASS(insn->code) == BPF_JMP && BPF_OP(insn->code) == BPF_CALL)
+		return 0;
+
+	/* The callback came back to the helper that calls it, in the same iteration. */
+	if (frame->callback_depth)
+		return 0;
+
+	/* If nothing was pushed and env->insn_idx is a loop header, we've taken a backedge. */
+	if (!pushed) {
+		if (!entry->apart) {
+			entry->trip++;
+			*backedge = true;
+			return check_refs_taken_in_loop(env, entry) ?: maybe_clamp_scev_regs(env);
+		}
+		loop = env->insn_aux_data[env->insn_idx].loop;
+		/* the next iteration is another kind of state, there is no cycle */
+		trip = entry->trip;
+		decided = !entry->probe || entry->decided || !trip;
+		entry->decided = false;
+		if (decided && ++entry->trip < entry->max_header_count &&
+		    (entry->counted ||
+		     env->walk_insns[entry->walk_id] <= entry->walk_patience))
+			return 0;
+		/* there is no such iteration at run time */
+		if (entry->counted) {
+			*drop = true;
+			return 0;
+		}
+		/*
+		 * The walk doesn't get to the end of the loop. From now on what comes back to
+		 * the head is joined with what is there. The loop gets may_goto.
+		 */
+		if (!loop->rt_bounded && !loop->has_guards) {
+			verbose(env, "loop at insn %d never exits\n", env->insn_idx);
+			return -EINVAL;
+		}
+		if (!loop->rt_bounded)
+			bpf_flow_loop_gets_guards(env, env->insn_idx);
+		/* the jumps were decided, it's the loop that is too long. Or it had its two trips */
+		if (decided || loop->has_calls)
+			loop->walk_failed = true;
+		/* the iteration that it went through: its state is at the head */
+		entry->trip = trip;
+		*ends_walk = true;
+		return 0;
+	}
+
+	loop = env->insn_aux_data[env->insn_idx].loop;
+	err = bpf_compute_loop_iters(env, cur, &iters);
+	if (err < 0)
+		return err;
+
+	/* @per_trip: the insns of one iteration, times the walks around the loop. */
+	per_trip = flow_walk_factor(cur) * loop->nr_insns;
+	/* The walk has to get to the end of the loop, so it is known to have one. Or not: */
+	if (!err) {
+		err = flow_try_walk(env, entry);
+		if (err)
+			return err < 0 ? err : 0;
+		if (!loop->rt_bounded && !loop->has_guards) {
+			verbose(env, "loop at insn %d never exits\n", env->insn_idx);
+			return -EINVAL;
+		}
+		if (!loop->rt_bounded) {
+			bpf_flow_loop_gets_guards(env, env->insn_idx);
+			entry->guarded = true;
+		}
+		return 0;
+	}
+
+	/* What the ranges are derived from for as long as the loop is looked at. */
+	entry_state = bpf_flow_keep_state(env, cur);
+	if (!entry_state)
+		return -ENOMEM;
+
+	entry->min_header_count = iters.min_header_count;
+	entry->max_header_count = iters.max_header_count;
+	entry->iv_reg = iters.iv_reg;
+	entry->iv_range = iters.iv_range;
+	entry->entry_state = entry_state;
+	entry->terminates = true;
+	err = bpf_widen_scev_regs(env, cur, entry_state, &iters);
+	return err < 0 ? err : 0;
+}
+
 static int do_check(struct bpf_verifier_env *env)
 {
 	bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
-- 
2.55.0

[PATCH bpf-next v4 18/46] bpf: Tell the passes over the graph where an added may_goto goes

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

A spin lock in Rust, test atomic_spin_lock of rust-bpf:

  for _ in 0..n {
      while lock.compare_exchange_weak(...).is_err() { spin_loop(); }
      data += 1; lock.store(false, Release);
  }

The compiler makes one loop with two back edges. No jump that leaves the
loop is in each iteration. Today: "infinite loop detected at insn 41". The
only place for may_goto that each iteration passes is the head. Its target
is an exit of all loops around it. 142 of the 225 progs of the schedulers
of rust-bpf have such a loop. The prog has no edge from the head to the
exit. The main pass verifies the state that goes there, but it relies on
passes that see only the control flow graph:

  bpf_compute_const_regs()  r7 == &ro on all edges to the exit, &rw at the
                            head. 'for (i = 0; i < *r7; i++)' behind the
                            exit gets its count from ro.
  compute_subprog_args()    The same with fp-8, fp-16: a store through r7.
  live registers            What the loop sets on each path to the exit is
                            dead at the head: a read behind it is rejected.

Thus call bpf_flow_place_guards() directly after the loops are known. No
constant is known at the target. compute_subprog_args() joins what it knows
at the head into the target. What is live at the target is live at the
head, may_goto or not: r1 in compute_live_registers/may_goto.

  do { ..; more = ++i < 256 && v->in[i & 63]; 10 insns; } while (more);
clang keeps the flag: 'if w6 > 255' sets it and stays in the loop. Today's
bpf_edge_leaves_loop() loses it at 'w1 &= 1': no count, 15611 insns, and
"R7 unbounded memory access" with one state for all iterations. Ask once,
for all edges: with which number in which register is the loop sure to be
left from this insn on? bpf_compute_leave_conds() goes against the edges:

  if w1 != 0 goto head   w1 is 0      does not go to the head
  w1 &= 1                w1 is even   what the insn turns into 0
  l: ..                  w1 is even   nothing writes w1
  w1 = 1                 never        1 is not even

An edge leaves the loop if the constants at its jump meet the condition at
its target: w1 is 0 at 'if w6 > 255 goto l'. 86 insns. Each insn has one
turn. A condition is a set of numbers per register. const_fold.c tells what
an insn makes of a number. Its constants must know the added edges.
A nested loop is not entered: may_goto there goes where a jump of that loop
goes, whatever the constants say.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h                  |   8 +
 kernel/bpf/const_fold.c                       | 264 +++++++++++++++++-
 kernel/bpf/liveness.c                         |  41 ++-
 kernel/bpf/loops.c                            | 145 ++++------
 kernel/bpf/scev.c                             |  12 +-
 kernel/bpf/verifier.c                         |  15 +
 .../bpf/progs/compute_live_registers.c        |   8 +-
 .../selftests/bpf/progs/verifier_loop_latch.c |   6 +-
 8 files changed, 387 insertions(+), 112 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6745c92ad586..42499a6f7e96 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -817,6 +817,8 @@ struct bpf_backedge {
 	 */
 	int latches[MAX_LATCHES];
 	u8 latches_cnt;
+	/* bit k: the loop is left on the edge of latches[k] that is taken, not on the other one */
+	u8 taken_leaves;
 };
 
 struct bpf_loop_exit {
@@ -934,6 +936,8 @@ struct bpf_insn_aux_data {
 	u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
 	u64 guard_edge_dropped:1; /* an edge of may_goto from here was dropped */
 	u64 need_scev:1;
+	/* a jump in a loop: the edge that is not taken, bit 0, or the other one leaves the loop */
+	u64 edge_leaves:2;
 	u64 alu_state:5; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
 	 * pattern for a bpf_fastcall call.
@@ -1483,6 +1487,8 @@ struct bpf_verifier_env {
 	struct bpf_iarray *succ;
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
+	/* per insn, while bpf_compute_leave_conds() runs */
+	struct bpf_leave_cond *leave_conds;
 	struct scev *scev;
 	struct bpf_flow *flow;
 	struct bpf_join_state *join;
@@ -1718,7 +1724,9 @@ u16 bpf_call_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 u16 bpf_call_checked_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
+int bpf_compute_leave_conds(struct bpf_verifier_env *env);
 bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
+int bpf_find_latches(struct bpf_verifier_env *env, bool flags);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
 int bpf_split_cur_state(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index eac2bf80274c..d01d0cc0dde5 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -250,7 +250,7 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
 	struct const_arg_info ci_out[MAX_BPF_REG];
 	struct bpf_iarray *succ;
 	bool changed;
-	int i, r;
+	int i, r, t;
 
 	/* kvzalloc zeroes memory, so all entries start as CONST_ARG_UNVISITED (0) */
 	ci_in = kvzalloc_objs(*ci_in, insn_cnt, GFP_KERNEL_ACCOUNT);
@@ -264,6 +264,12 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
 		for (r = 0; r < MAX_BPF_REG; r++)
 			ci_in[start][r] = unknown;
 	}
+	/* The same from the head of a loop to where the loop is left, once the loops are known. */
+	for (i = 0; i < insn_cnt; i++) {
+		t = bpf_head_guard_target(env, i);
+		for (r = 0; r < MAX_BPF_REG && t >= 0; r++)
+			ci_in[t][r] = unknown;
+	}
 
 redo:
 	changed = false;
@@ -344,6 +350,262 @@ static int eval_const_branch(u8 opcode, u64 dst_val, u64 src_val)
 	}
 }
 
+/* 1: the registers @ci take the conditional jump @insn, 0: they don't, -1: not known */
+static int const_branch_taken(const struct bpf_insn *insn, const struct const_arg_info *ci)
+{
+	u64 dst = ci[insn->dst_reg].val, src = insn->imm;
+
+	if (!ci_is_const(&ci[insn->dst_reg]))
+		return -1;
+	if (BPF_SRC(insn->code) == BPF_X) {
+		if (!ci_is_const(&ci[insn->src_reg]))
+			return -1;
+		src = ci[insn->src_reg].val;
+	}
+	/* see bpf_prune_dead_branches() */
+	if (BPF_CLASS(insn->code) == BPF_JMP32) {
+		dst = (s32)dst;
+		src = (s32)src;
+	}
+	return eval_const_branch(insn->code, dst, src);
+}
+
+/*
+ * When is the program that stands in front of an insn of a loop sure to leave the loop?
+ * Always, or when one register holds a number of a set, whatever the other registers hold.
+ * A set has one bit per range of numbers of 32 bits. A flag is a small number: each of 0 - 31
+ * is a range. An error code or a size is not: 32 - 63, 64 - 127, ... are the other ranges.
+ */
+struct bpf_leave_cond {
+	u64 vals[MAX_BPF_REG];
+	bool always;
+};
+
+#define LEAVE_SMALL	32
+#define LEAVE_RANGES	(LEAVE_SMALL + 32 - ilog2(LEAVE_SMALL))
+
+static const struct bpf_leave_cond leave_never;
+static const struct bpf_leave_cond leave_always = {
+	.vals = { [0 ... MAX_BPF_REG - 1] = U64_MAX },
+	.always = true,
+};
+
+static u64 leave_val_bit(u64 val)
+{
+	if (val > U32_MAX)
+		return 0;
+	return BIT_ULL(val < LEAVE_SMALL ? val : ilog2(val) + LEAVE_SMALL - ilog2(LEAVE_SMALL));
+}
+
+/* The condition for the loop at @h in front of @t, where an insn of that loop goes on */
+static const struct bpf_leave_cond *leave_cond_at(struct bpf_verifier_env *env, int h, int t)
+{
+	int l = bpf_loop_at_index(env, t);
+
+	if (l != h)
+		return bpf_is_nested_loop(env, l, h) ? &leave_never : &leave_always;
+	return t == h || !env->leave_conds ? &leave_never : &env->leave_conds[t];
+}
+
+/* The same when the jump at @n goes on at @t: always, if the constants at @n meet it */
+static const struct bpf_leave_cond *leave_cond_edge(struct bpf_verifier_env *env, int h, int n,
+						    int t)
+{
+	const struct bpf_insn_aux_data *aux = &env->insn_aux_data[n];
+	const struct bpf_leave_cond *p = leave_cond_at(env, h, t);
+	int r;
+
+	for (r = 0; r < ARRAY_SIZE(aux->const_reg_vals); r++)
+		if ((aux->const_reg_mask & BIT(r)) &&
+		    (p->vals[r] & leave_val_bit(aux->const_reg_vals[r])))
+			return &leave_always;
+	return p;
+}
+
+/* Is what the insn at @idx makes of @reg == @val in the set @vals? @ci: nothing is known. */
+static bool insn_makes(struct bpf_verifier_env *env, struct const_arg_info *ci, int idx,
+		       int reg, u64 val, u64 vals)
+{
+	struct const_arg_info unknown = { .state = CONST_ARG_UNKNOWN, .val = 0 };
+	struct bpf_insn *insns = env->prog->insnsi;
+	struct const_arg_info *dst = &ci[insns[idx].dst_reg];
+	bool ret;
+
+	if (reg >= 0) {
+		ci[reg].state = CONST_ARG_CONST;
+		ci[reg].val = val;
+	}
+	const_reg_xfer(env, ci, &insns[idx], insns, idx);
+	ret = ci_is_const(dst) && (vals & leave_val_bit(dst->val));
+	*dst = unknown;
+	if (reg >= 0)
+		ci[reg] = unknown;
+	return ret;
+}
+
+/* @p holds behind the insn at @idx. Make it what holds in front of it. */
+static void leave_cond_insn(struct bpf_verifier_env *env, int idx, struct bpf_leave_cond *p)
+{
+	struct bpf_insn *insns = env->prog->insnsi, *insn = &insns[idx];
+	int r, v, dst = insn->dst_reg, src = insn->src_reg;
+	struct const_arg_info ci[MAX_BPF_REG] = {};
+	u8 class = BPF_CLASS(insn->code);
+	u64 vals;
+
+	/* Nothing is known about a register that is no constant although all were constants */
+	for (r = 0; r < MAX_BPF_REG; r++)
+		ci[r].state = CONST_ARG_CONST;
+	const_reg_xfer(env, ci, insn, insns, idx);
+	for (r = 0; r < MAX_BPF_REG; r++) {
+		if (!ci_is_const(&ci[r]))
+			p->vals[r] = 0;
+		ci[r].state = CONST_ARG_UNKNOWN;
+	}
+	if ((class != BPF_ALU && class != BPF_ALU64 && class != BPF_LD) ||
+	    dst >= MAX_BPF_REG || !p->vals[dst])
+		return;
+	vals = p->vals[dst];
+	p->vals[dst] = 0;
+	if (insn_makes(env, ci, idx, -1, 0, vals)) {
+		*p = leave_always;
+		return;
+	}
+	for (v = 0; v < LEAVE_SMALL && class != BPF_LD; v++) {
+		if (insn_makes(env, ci, idx, dst, v, vals))
+			p->vals[dst] |= BIT_ULL(v);
+		if (BPF_SRC(insn->code) == BPF_X && src != dst && src < MAX_BPF_REG &&
+		    insn_makes(env, ci, idx, src, v, vals))
+			p->vals[src] |= BIT_ULL(v);
+	}
+}
+
+/*
+ * const_branch_taken() for all of @lo - @hi in the register that @insn compares with a number.
+ * The range has numbers of 32 bits of one sign. So '<' and the like treat all of them alike
+ * if they treat both ends alike. '==' and '!=' single out the number that they compare with.
+ * '&': @lo is the bit that all of them have, @hi has the bits that any of them has.
+ */
+static int range_branch_taken(const struct bpf_insn *insn, struct const_arg_info *ci, u64 lo,
+			      u64 hi)
+{
+	u64 vals[] = { lo, hi, clamp_t(u64, (u32)insn->imm, lo, hi) };
+	int i, taken[ARRAY_SIZE(vals)];
+
+	for (i = 0; i < ARRAY_SIZE(vals); i++) {
+		ci[insn->dst_reg].val = vals[i];
+		taken[i] = const_branch_taken(insn, ci);
+	}
+	return taken[0] == taken[1] && taken[0] == taken[2] ? taken[0] : -1;
+}
+
+/*
+ * @p is what is enough on both edges of the jump at @idx. If the jump compares a register
+ * with a number then a number in the register goes one way: enough on that edge is enough.
+ */
+static void leave_cond_jump(struct bpf_verifier_env *env, int h, int idx,
+			    struct bpf_leave_cond *p)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	struct const_arg_info ci[MAX_BPF_REG] = {};
+	u64 t, f, lo, go = 0, stay = 0;
+	int i, taken;
+
+	if (BPF_SRC(insn->code) != BPF_K)
+		return;
+	t = leave_cond_edge(env, h, idx, idx + insn->off + 1)->vals[insn->dst_reg];
+	f = leave_cond_edge(env, h, idx, idx + 1)->vals[insn->dst_reg];
+	if (t == f)
+		return;
+	ci[insn->dst_reg].state = CONST_ARG_CONST;
+	for (i = 0; i < LEAVE_RANGES; i++) {
+		lo = i < LEAVE_SMALL ? i : BIT_ULL(i - LEAVE_SMALL + ilog2(LEAVE_SMALL));
+		taken = range_branch_taken(insn, ci, lo, i < LEAVE_SMALL ? lo : 2 * lo - 1);
+		if (taken > 0)
+			go |= BIT_ULL(i);
+		else if (!taken)
+			stay |= BIT_ULL(i);
+	}
+	p->vals[insn->dst_reg] |= (t & go) | (f & stay);
+}
+
+/*
+ * A compiler makes
+ *	w1 = 0; if w7 > 255 goto l; ...; w1 = 1; l: ...; w1 &= 1; if w1 != 0 goto head;
+ * of 'do { ...; more = ++i < 256 && ...; ...; } while (more);'. The jump that leaves the loop
+ * looks at 1 or 0. The jump that counts stays in the loop on both edges.
+ *
+ * So find for every insn of a loop when the loop is left from there on. Go against the edges:
+ *	if w1 != 0 goto head	w1 is 0		the numbers that don't go to the head
+ *	w1 &= 1			w1 is even	the numbers that the insn turns into 0
+ *	l: ...			w1 is even	nothing writes w1
+ *	w1 = 1			never		1 is not even
+ * w1 is 0 at the jump that counts and 0 is even: the edge to l leaves the loop.
+ *
+ * An insn that is not in the loop: always. The head: never. More than one successor: what is
+ * enough for each of them. Every insn has one turn, in post order. A cycle without the head is
+ * a nested loop, so the successors in the loop itself had their turn. One that didn't: never.
+ * The jumps on the way know constants too: 'r7 = 0; if w1 != 0 goto a; ...; a: if r7 == 2 goto
+ * head' leaves the loop when w1 is not 0.
+ *
+ * Every edge that the program has at run time counts, the edges of may_goto that the verifier
+ * adds too. May_goto of the loop and of the loops around it leaves the loop. A nested loop can
+ * get one in front of a jump that the numbers decide the other way. So a nested loop: never.
+ *
+ * What is kept is the answer for the two edges of each jump: insn_aux_data->edge_leaves.
+ */
+int bpf_compute_leave_conds(struct bpf_verifier_env *env)
+{
+	const struct bpf_leave_cond *q;
+	struct bpf_leave_cond *p;
+	struct bpf_iarray *succ;
+	int i, r, s, h, idx;
+	bool jump;
+	u64 any;
+
+	env->leave_conds = kvcalloc(env->prog->len, sizeof(*p), GFP_KERNEL_ACCOUNT);
+	if (!env->leave_conds)
+		return -ENOMEM;
+	for (i = 0; i < env->cfg.cur_postorder; i++) {
+		idx = env->cfg.insn_postorder[i];
+		h = bpf_loop_at_index(env, idx);
+		if (h < 0)
+			continue;
+		p = &env->leave_conds[idx];
+		*p = leave_always;
+		jump = bpf_insn_is_cond_jump(env->prog->insnsi[idx].code);
+		succ = bpf_insn_successors(env, idx);
+		iarray_for_each(s, succ) {
+			q = jump ? leave_cond_edge(env, h, idx, s) : leave_cond_at(env, h, s);
+			for (r = 0; r < MAX_BPF_REG; r++)
+				p->vals[r] &= q->vals[r];
+			p->always &= q->always;
+			if (jump && q->always)
+				env->insn_aux_data[idx].edge_leaves |= BIT(s != idx + 1);
+		}
+		if (jump) {
+			leave_cond_jump(env, h, idx, p);
+			continue;
+		}
+		for (r = 0, any = 0; r < MAX_BPF_REG; r++)
+			any |= p->vals[r];
+		if (any && !p->always)
+			leave_cond_insn(env, idx, p);
+	}
+	kvfree(env->leave_conds);
+	env->leave_conds = NULL;
+	return 0;
+}
+
+/*
+ * Does the program leave the loop at @h when the conditional jump at @n goes on at @t?
+ * @h is the innermost loop around @n.
+ */
+bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t)
+{
+	return leave_cond_at(env, h, t)->always ||
+	       (env->insn_aux_data[n].edge_leaves & BIT(t != n + 1));
+}
+
 /*
  * Rewrite conditional branches with constant outcomes into unconditional
  * jumps using register values resolved by bpf_compute_const_regs() pass.
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 525c0faff1a6..6a2ee407412b 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1054,6 +1054,22 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in
 	return true;
 }
 
+/* What leaves @idx gets to @target */
+static bool arg_track_join_all(struct bpf_verifier_env *env, int idx, int target,
+			       struct arg_track *in, struct arg_track *stack_in,
+			       const struct arg_track *out, const struct arg_track *stack_out,
+			       u32 nslots)
+{
+	bool changed = false;
+	int r;
+
+	for (r = 0; r < MAX_AT_TRACK_REGS; r++)
+		changed |= arg_track_join(env, idx, target, r, &in[r], out[r]);
+	for (r = 0; r < nslots; r++)
+		changed |= arg_track_join(env, idx, target, -r - 1, &stack_in[r], stack_out[r]);
+	return changed;
+}
+
 /*
  * Compute the result when an ALU op destroys offset precision.
  * If a single arg is identifiable, preserve it with OFF_IMPRECISE.
@@ -2069,6 +2085,7 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
 		int i = idx - start;
 		struct bpf_insn *insn = &insns[idx];
 		struct bpf_iarray *succ;
+		int ti;
 
 		if (!arg_is_visited(&at_in[i][0]) && !arg_is_visited(&at_in[i][1]))
 			continue;
@@ -2076,6 +2093,12 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
 		memcpy(at_out, at_in[i], sizeof(at_out));
 		memcpy(at_stack_out, &at_stack_in[(size_t)i * nslots], nslots * sizeof(*at_stack_out));
 
+		ti = bpf_head_guard_target(env, idx) - start;
+		if (ti >= 0)
+			changed |= arg_track_join_all(env, idx, start + ti, at_in[ti],
+						      &at_stack_in[(size_t)ti * nslots],
+						      at_out, at_stack_out, nslots);
+
 		arg_track_xfer(env, insn, idx, at_out, at_stack_out, nslots,
 			       at_stack_arg_entry, instance, callsites);
 		arg_track_log(env, insn, idx, at_in[i], &at_stack_in[(size_t)i * nslots], at_out,
@@ -2085,21 +2108,14 @@ static noinline_for_stack int compute_subprog_args(struct bpf_verifier_env *env,
 		succ = bpf_insn_successors(env, idx);
 		for (int s = 0; s < succ->cnt; s++) {
 			int target = succ->items[s];
-			int ti;
 
 			/* Filter: stay within the subprogram's range */
 			if (target < start || target >= end)
 				continue;
 			ti = target - start;
-
-			for (r = 0; r < MAX_AT_TRACK_REGS; r++)
-				changed |= arg_track_join(env, idx, target, r,
-							  &at_in[ti][r], at_out[r]);
-
-			for (r = 0; r < nslots; r++)
-				changed |= arg_track_join(env, idx, target, -r - 1,
-							  &at_stack_in[(size_t)ti * nslots + r],
-							  at_stack_out[r]);
+			changed |= arg_track_join_all(env, idx, target, at_in[ti],
+						      &at_stack_in[(size_t)ti * nslots],
+						      at_out, at_stack_out, nslots);
 		}
 	}
 	if (changed)
@@ -2780,6 +2796,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			u32 new_out = 0;
 			u32 new_in = 0;
 			u32 use;
+			int t;
 
 			succ = bpf_insn_successors(env, insn_idx);
 			for (int s = 0; s < succ->cnt; ++s)
@@ -2794,6 +2811,10 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
 				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
 			new_in = (new_out & ~live->def) | use;
+			/* may_goto that is added at the head of a loop goes there */
+			t = bpf_head_guard_target(env, insn_idx);
+			if (t >= 0)
+				new_in |= state[t].in;
 			if (new_out != live->out || new_in != live->in) {
 				live->in = new_in;
 				live->out = new_out;
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 729e0055f86f..0f3b0411003b 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -304,73 +304,6 @@ int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx)
 	return aux[idx].loop ? idx : aux[idx].loop_header;
 }
 
-/* @known: the registers that hold the constant @val behind @insn. False: a jump or the like. */
-static bool track_consts(const struct bpf_insn *insn, u16 *known, s32 *val)
-{
-	u8 class = BPF_CLASS(insn->code);
-
-	if (class == BPF_ST || (class == BPF_STX && BPF_MODE(insn->code) == BPF_MEM))
-		return true;
-	/* r6 - r9 are what they were when the call returns */
-	if (insn->code == (BPF_JMP | BPF_CALL)) {
-		*known &= ~GENMASK(BPF_REG_5, BPF_REG_0);
-		return true;
-	}
-	if (class != BPF_ALU && class != BPF_ALU64 && class != BPF_LDX)
-		return false;
-	*known &= ~BIT(insn->dst_reg);
-	if (class != BPF_LDX && BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_K) {
-		*known |= BIT(insn->dst_reg);
-		val[insn->dst_reg] = insn->imm;
-	}
-	return true;
-}
-
-/*
- * Does the program leave the loop at @h when the conditional jump at @n goes on at @t?
- * It does when @t is not in the loop. A compiler also makes
- *	w2 = 1; if w1 == 0 goto l; w2 = 0; l: ...; if w2 != 1 goto head;
- * of 'while (n--)'. So follow the constants that are set next to the jump for as long as there
- * is one way to go.
- */
-bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t)
-{
-	struct bpf_insn *insns = env->prog->insnsi, *insn;
-	s32 val[MAX_BPF_REG];
-	u16 known = 0;
-	int i, l;
-	u8 op;
-
-	/* the insns in front of the jump, for as long as nothing else leads to the next one */
-	for (i = n; i && i > n - 4 && !env->insn_aux_data[i].loop && env->idoms[i] == i - 1; i--)
-		;
-	for (; i < n; i++)
-		if (!track_consts(&insns[i], &known, val))
-			known = 0;
-	for (i = 0; i < 16; i++, t++) {
-		l = bpf_loop_at_index(env, t);
-		if (l != h && !bpf_is_nested_loop(env, l, h))
-			return true;
-		insn = &insns[t];
-		op = BPF_OP(insn->code);
-		if (t == h) {
-			return false;
-		} else if (insn->code == (BPF_JMP | BPF_JA)) {
-			t += insn->off;
-		} else if (is_cond_jmp_insn(insn)) {
-			/* a 32-bit and a 64-bit jump agree on a number that is not negative */
-			if (!(known & BIT(insn->dst_reg)) || val[insn->dst_reg] < 0 ||
-			    BPF_SRC(insn->code) != BPF_K || (op != BPF_JEQ && op != BPF_JNE))
-				return false;
-			if ((val[insn->dst_reg] == insn->imm) == (op == BPF_JEQ))
-				t += insn->off;
-		} else if (!track_consts(insn, &known, val)) {
-			return false;
-		}
-	}
-	return false;
-}
-
 /*
  * Collect conditional jumps within n's loop that dominate 'n' and leave
  * the loop on one edge, see bpf_edge_leaves_loop(), nearest first.
@@ -395,6 +328,8 @@ static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
 			    (backedge->latches_cnt || bpf_jump_leaves_loop(env, n_loop, n))) {
 				if (backedge->latches_cnt == MAX_LATCHES)
 					return;
+				if (t_exits_loop)
+					backedge->taken_leaves |= BIT(backedge->latches_cnt);
 				backedge->latches[backedge->latches_cnt++] = n;
 			}
 		}
@@ -404,6 +339,55 @@ static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
 	}
 }
 
+/*
+ * Without @flags: the jumps that leave a loop themselves. That tells where the verifier may add
+ * may_goto. It takes the constants that hold with these edges to follow a flag.
+ */
+int bpf_find_latches(struct bpf_verifier_env *env, bool flags)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_backedge *backedge;
+	struct bpf_loop *loop;
+	int i, j, k, err;
+
+	if (flags) {
+		err = bpf_compute_leave_conds(env);
+		if (err)
+			return err;
+	}
+	for (i = 0; i < env->prog->len; i++) {
+		loop = aux[i].loop;
+		if (!loop)
+			continue;
+		for (j = 0; j < loop->backedges_cnt; j++) {
+			backedge = &loop->backedges[j];
+			backedge->latches_cnt = 0;
+			backedge->taken_leaves = 0;
+			/*
+			 * In theory, the backedge->from and it's latch can reside in
+			 * an inner loop of `i`, e.g.:
+			 *
+			 *  1:  r7 += 1;
+			 *  2:  r6 += 1;
+			 *      if r7 == 2 goto 1b;
+			 *      if r6 < 2 goto 2b;
+			 *
+			 * For now, let's assume that latches are unknown for such cases.
+			 * (GCC/LLVM handle this by inserting artificial cfg nodes).
+			 */
+			if (bpf_loop_at_index(env, backedge->from) != i)
+				continue;
+			find_dominating_conditions(env, backedge->from, backedge);
+			for (k = 1; k < backedge->latches_cnt; k++)
+				if ((env->log.level & BPF_LOG_LEVEL2) &&
+				    !bpf_jump_leaves_loop(env, i, backedge->latches[k]))
+					bpf_log(&env->log, "loop at %d, exit by way of a flag at %d\n",
+						i, backedge->latches[k]);
+		}
+	}
+	return 0;
+}
+
 /*
  * As described in "A New Algorithm for Identifying Loops in Decompilation" by Wei et al,
  * adapted to be non-recursive.
@@ -572,30 +556,9 @@ int bpf_compute_loops(struct bpf_verifier_env *env)
 		if (err)
 			goto out;
 	}
-	/* find latches */
-	for (i = 0; i < len; i++) {
-		loop = aux[i].loop;
-		if (!loop)
-			continue;
-		for (j = 0; j < loop->backedges_cnt; j++) {
-			backedge = &loop->backedges[j];
-			/*
-			 * In theory, the backedge->from and it's latch can reside in
-			 * an inner loop of `i`, e.g.:
-			 *
-			 *  1:  r7 += 1;
-			 *  2:  r6 += 1;
-			 *      if r7 == 2 goto 1b;
-			 *      if r6 < 2 goto 2b;
-			 *
-			 * For now, let's assume that latches are unknown for such cases.
-			 * (GCC/LLVM handle this by inserting artificial cfg nodes).
-			 */
-			if (bpf_loop_at_index(env, backedge->from) != i)
-				continue;
-			find_dominating_conditions(env, backedge->from, backedge);
-		}
-	}
+	err = bpf_find_latches(env, false);
+	if (err)
+		goto out;
 	/* find exits */
 	for (i = 0; i < len; i++) {
 		iloop = aux[i].loop ? i : aux[i].loop_header;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 9629a6bf80b1..d62e087f811d 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2101,13 +2101,12 @@ static bool loop_invariant(struct scev *scev, u32 id)
 
 static int match_linear_latch(struct bpf_verifier_env *env,
 			      u32 header,
-			      u32 latch_idx,
+			      u32 latch_idx, bool taken_leaves,
 			      struct linear_latch *latch)
 {
 	struct bpf_insn *insn = &env->prog->insnsi[latch_idx];
 	struct scev *scev = env->scev;
 	struct env *latch_env;
-	u32 true_branch_tgt;
 	u32 src_reg_scev;
 	u32 dst_reg_scev;
 	u32 t, l, r, op;
@@ -2116,8 +2115,7 @@ static int match_linear_latch(struct bpf_verifier_env *env,
 	op = BPF_OP(insn->code);
 	latch->jmp32 = BPF_CLASS(insn->code) == BPF_JMP32;
 	/* Flip the condition if true branch jumps out of the loop. */
-	true_branch_tgt = latch_idx + bpf_jmp_offset(insn) + 1;
-	if (bpf_edge_leaves_loop(env, header, latch_idx, true_branch_tgt))
+	if (taken_leaves)
 		op = bpf_rev_opcode(op);
 	switch (op) {
 	case BPF_JSLT:
@@ -3347,7 +3345,8 @@ static int count_backedge(struct bpf_verifier_env *env, struct bpf_func_state *s
 
 	for (j = 0; j < backedge->latches_cnt; j++) {
 		latch_idx = backedge->latches[j];
-		linear_latch = match_linear_latch(env, insn_idx, latch_idx, &latch);
+		linear_latch = match_linear_latch(env, insn_idx, latch_idx,
+						  backedge->taken_leaves & BIT(j), &latch);
 		if (linear_latch < 0)
 			return linear_latch;
 		if (!linear_latch) {
@@ -3363,6 +3362,9 @@ static int count_backedge(struct bpf_verifier_env *env, struct bpf_func_state *s
 					insn_idx, latch_idx);
 			continue;
 		}
+		/* a flag can have other reasons to end the loop */
+		if (!bpf_jump_leaves_loop(env, insn_idx, latch_idx))
+			tmp.min_header_count = 0;
 		if (!found || tmp.max_header_count < iters->max_header_count) {
 			iters->max_header_count = tmp.max_header_count;
 			iters->iv_reg = tmp.iv_reg;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5faea922699a..62a58f38f4e2 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -25268,6 +25268,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/*
+	 * It took the constants to remove dead branches and find the loops. Now it's known
+	 * which edges the verifier may add, see bpf_head_guard_target(). What follows
+	 * believes the constants.
+	 */
+	ret = bpf_flow_place_guards(env);
+	if (ret > 0)
+		ret = bpf_compute_const_regs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
+	ret = bpf_find_latches(env, true);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index ba43304836ca..f5c505dfd464 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -316,11 +316,15 @@ __naked void gotol(void)
 }
 #endif
 
+/*
+ * If the verifier had to add may_goto to this loop it would be at the head, insn 0, and go to
+ * insn 3. What is live at 3 is live at 0.
+ */
 SEC("socket")
 __log_level(2)
-__msg("0: .......... (b7) r1 = 1")
+__msg("0: .1........ (b7) r1 = 1")
 __msg("1: .1........ (e5) may_goto pc+1")
-__msg("2: .......... (05) goto pc-3")
+__msg("2: .1........ (05) goto pc-3")
 __msg("3: .1........ (bf) r0 = r1")
 __msg("4: 0......... (95) exit")
 __naked void may_goto(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
index 1fcefce9ab01..6dd46edd8ed0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -130,17 +130,17 @@ LOOP(down_two_tests, "w8 = 4; r6 += 2043;",
 /* 'while (n--)': the jump that counts sets a flag, the jump that leaves the loop tests it */
 #define FLAG(mid)	"w8 += 1; w2 = 1; if w8 == 0 goto 2f; w2 = 0; 2: " mid " if w2 != 1 goto 1b;"
 
-OK("a flag", "2048")
+OK("a flag", "[0..2048]")
 LOOP(flag, "w8 = -2048;", FLAG("r3 = r6;"))
 
 BAD("a flag, one too many")
 LOOP(flag_bad, "w8 = -2049;", FLAG("r3 = r6;"))
 
-OK("a flag, a store behind it", "2048")
+OK("a flag, a store behind it", "[0..2048]")
 LOOP(flag_store, "w8 = -2048;",
      "w8 += 1; w2 = 1; *(u32 *)(r10 - 4) = r8; if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
 
-OK("a flag, a call behind it", "2048")
+OK("a flag, a call behind it", "[0..2048]")
 LOOP(flag_call, "w8 = -2048;",
      "w8 += 1; w9 = 1; if w8 == 0 goto 2f; w9 = 0; 2: call %[bpf_get_prandom_u32]; if w9 != 1 goto 1b;")
 
-- 
2.55.0

Re: [PATCH bpf-next v4 18/46] bpf: Tell the passes over the graph where an added may_goto goes

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] bpf: missing head guard target edge in update_insn() stack
  liveness
- [High] bpf: bpf_prune_dead_branches() folds jumps before head guards
  are placed
--

commit b45291925cc857048f8bb1e4b926985f761611df
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Tell the passes over the graph where an added may_goto goes

This commit runs bpf_flow_place_guards() right after loop detection so
constant folding, subprog argument tracking, and register liveness
account for synthetic may_goto edges from loop headers, and adds
backward condition tracking in bpf_compute_leave_conds() to identify
loop exits controlled by flags.

> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index 525c0faff1a66..6a2ee407412b8 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c
[ ... ]
> @@ -2780,6 +2796,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>  			u32 new_out = 0;
>  			u32 new_in = 0;
>  			u32 use;
> +			int t;
>  
>  			succ = bpf_insn_successors(env, insn_idx);
>  			for (int s = 0; s < succ->cnt; ++s)
> @@ -2794,6 +2811,10 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>  				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
>  				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
>  			new_in = (new_out & ~live->def) | use;
> +			/* may_goto that is added at the head of a loop goes there */
> +			t = bpf_head_guard_target(env, insn_idx);
> +			if (t >= 0)
> +				new_in |= state[t].in;
>  			if (new_out != live->out || new_in != live->in) {
>  				live->in = new_in;
>  				live->out = new_out;

[Severity: High]
Does update_insn() need the same bpf_head_guard_target() edge for
backward stack-slot liveness (FM_LIVE_BEFORE)?

compute_subprog_args() now propagates stack pointers and spills across
bpf_head_guard_target(), and bpf_compute_live_registers() unions
state[t].in into new_in at the loop header, but update_insn() still
computes FM_LIVE_BEFORE only from normal CFG successors minus
must_write:

kernel/bpf/liveness.c:update_insn() {
	...
	for (s = 0; s < succ->cnt; ++s) {
		unsigned long *succ_live;

		succ_live = rel_mask(fm, relative_idx(instance, succ->items[s]), FM_LIVE_BEFORE);
		for (w = 0; w < fm->words; w++)
			new_after[w] |= succ_live[w];
	}
	...
	for (w = 0; w < fm->words; w++) {
		unsigned long new_before = (new_after[w] & ~must_write[w]) | may_read[w];
}

If a stack slot is initialized before the loop header, overwritten on
every normal path through the loop to the exit target t, and read at or
after t, FM_LIVE_BEFORE is cleared at the loop header.  When a state
arrives at the header in flow_arrive(), __clean_func_state() sees
bpf_stack_slot_alive() == false, clears spilled_ptr to NOT_INIT, and
marks the slot STACK_POISON before check_loop_guard() delivers the state
along the head-guard edge to t.  Reads of that slot after t then fail
verification as reading a poisoned slot, or if the slot held a spilled
pointer before the header and a scalar on normal loop exits, join_slot()
at t can merge the poisoned slot with the scalar into STACK_INVALID when
allow_uninit_stack is set, allowing the spilled pointer bytes on the
may_goto path to be read as a scalar.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 5faea922699af..62a58f38f4e22 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -25268,6 +25268,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
>  	if (ret < 0)
>  		goto skip_full_check;
>  
> +	/*
> +	 * It took the constants to remove dead branches and find the loops. Now it's known
> +	 * which edges the verifier may add, see bpf_head_guard_target(). What follows
> +	 * believes the constants.
> +	 */
> +	ret = bpf_flow_place_guards(env);
> +	if (ret > 0)
> +		ret = bpf_compute_const_regs(env);
> +	if (ret < 0)
> +		goto skip_full_check;
> +
> +	ret = bpf_find_latches(env, true);
> +	if (ret < 0)
> +		goto skip_full_check;
> +
>  	ret = bpf_compute_live_registers(env);
>  	if (ret < 0)
>  		goto skip_full_check;

[Severity: High]
Can re-running bpf_compute_const_regs() after bpf_flow_place_guards()
leave behind conditional jumps that bpf_prune_dead_branches() already
rewrote into unconditional jumps using constants that only hold on
normal loop exits?

Before bpf_compute_loops() and bpf_flow_place_guards() run, the first
bpf_compute_const_regs() pass has no head-guard edges, so a register (or
a read-only map load through it, as in the r7 == &ro vs &rw case from
the commit message) that is constant on all normal exits to t is marked
constant at and after t.  bpf_prune_dead_branches() then permanently
overwrites conditional jumps testing that register at or after t in
env->prog->insnsi with BPF_JMP_A and recomputes postorder.  Re-running
bpf_compute_const_regs() after bpf_flow_place_guards() clears
const_reg_mask at t, but the in-place BPF_JMP_A rewrites remain in the
program, so a state taking the header may_goto to t will unconditionally
follow the branch decided for normal exits.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=18

[PATCH bpf-next v4 19/46] bpf: Verify programs as a fixpoint over the control flow graph

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  for i in 0..n { v.push(i * i); }
Test vec_push_pop of rust-bpf: "The sequence of 8193 jumps is too complex."
The verifier follows each path from the first insn to exit, and each branch
can double the number of paths. 123 tests of rust-bpf hit this limit or
1M insns. C progs are cut until they fit.

Do not follow paths. Where two paths meet, make one state that covers both,
a join: r1=1 and r1=5 give r1=[1,5]. What is safe for the join is safe for
both. Repeat until no state changes. The work depends on the size of the
prog, not on the number of paths: 1761 insns with this patch, and 120 of
the 123 load. do_check() takes the next new state from flow.c and checks
one basic block. The check of an insn is the same.

What user space sees:
 - The limits of 1M processed insns and 8192 jumps are gone, from
   bpf_design_QA.rst too. flow.c limits states, for a prog of N insns:
     1024 at an insn      "more than 1024 states that cannot be joined"
     16 * N + 4096 at a time, 256 * N + 64K in all    "too many states"
 - "infinite loop detected" is gone: 111 of the 225 progs of the schedulers
   of rust-bpf. A fixpoint shows that each iteration is safe, not that the
   loop ends:
     counted, or walked to its end                      nothing is added
     each iteration has may_goto or bpf_iter_*_next()   nothing is added
     no exit                                            rejected
     all other loops                                    may_goto
 - An added may_goto is a new edge. check_loop_guard() verifies it. It goes
   to 'r0 = 0; exit': the probe loop of hashbrown exits only with a bucket,
   and the code after the loop uses it: "R6 invalid mem access 'scalar'".
   Until these insns are added the last exit stands for them. With GCC that
   can be dead code, 'if (debug) return ..': r0 is marked as live there.
   If the frame holds more than at its entry, or is a subprog that cannot
   unwind, may_goto goes to the exit of the loop: the prog releases there
   what it holds. From the head of a loop that is no edge of the prog. In
   glob_match() of arena_strsearch without cond_break only the body sets
   r0 for the 'exit' behind the loop: "R0 !read_ok". A subprog that holds
   nothing is left with 'r0 = 0; exit' then. Its caller is verified with 0.
 - A walk goes on while each iteration decides a jump that leaves the loop.
   A jump that is decided only because states are kept apart by a flag does
   not count: it repeats the jump that set the flag, which went both ways.
   lexical_numbers of rust-bpf: 41610 insns, 98980 if it counts.
 - 1 and 2 are joined at exit, and a test for 'r0 is 2 or 3' misses [1, 2].
   Thus a cgroup_skb prog whose r0 can be 2 or 3 attaches to egress only.
 - Static subprogs and recursion are as before. Pruning, jmp_history and
   precision marks are only disabled here.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 Documentation/bpf/bpf_design_QA.rst |   7 +-
 Documentation/bpf/kfuncs.rst        |   2 +-
 include/linux/bpf_verifier.h        |  12 +
 kernel/bpf/backtrack.c              |   7 +
 kernel/bpf/const_fold.c             |   6 +
 kernel/bpf/flow.c                   |  21 +-
 kernel/bpf/liveness.c               |   5 +
 kernel/bpf/log.c                    |   5 +
 kernel/bpf/states.c                 |  50 ++-
 kernel/bpf/verifier.c               | 656 ++++++++++++++++++++--------
 10 files changed, 570 insertions(+), 201 deletions(-)

diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst
index 98d70b8a2533..f86b272920ae 100644
--- a/Documentation/bpf/bpf_design_QA.rst
+++ b/Documentation/bpf/bpf_design_QA.rst
@@ -92,11 +92,8 @@ Q: What are the verifier limits?
 A: The only limit known to the user space is BPF_MAXINSNS (4096).
 It's the maximum number of instructions that the unprivileged bpf
 program can have. The verifier has various internal limits.
-Like the maximum number of instructions that can be explored during
-program analysis. Currently, that limit is set to 1 million.
-Which essentially means that the largest program can consist
-of 1 million NOP instructions. There is a limit to the maximum number
-of subsequent branches, a limit to the number of nested bpf-to-bpf
+The largest program can consist of 1 million instructions.
+There is a limit to the number of nested bpf-to-bpf
 calls, a limit to the number of the verifier states per instruction,
 a limit to the number of maps used by the program.
 All these limits can be hit with a sufficiently complex program.
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index ac9e3a175378..c7da603cab77 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -159,7 +159,7 @@ An example is given below::
 Here, bpf_obj_new uses local_type_id argument to find out the size of that type
 ID in program's BTF and return a sized pointer to it. Each type ID will have a
 distinct size, hence it is crucial to treat each such call as distinct when
-values don't match during verifier state pruning checks.
+values don't match when the verifier compares and joins states.
 
 Hence, whenever a constant scalar argument is accepted by a kfunc which is not a
 size parameter, and the value of the constant matters for program safety, __k
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 42499a6f7e96..83cd9983a0b2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -390,6 +390,11 @@ struct loop_stack_entry {
 	u32 probe:1;
 	/* this one did */
 	u32 decided:1;
+	/*
+	 * The state is kept apart from one with the other value of a flag, see
+	 * flow_join_makes_up(). The test of the flag is decided in both, and decides nothing.
+	 */
+	u32 flag_apart:1;
 	/*
 	 * How many times the state came back to the head of the loop. With @apart that is
 	 * the iteration that it is in. A state that stands for many iterations is a join, and has
@@ -934,6 +939,7 @@ struct bpf_insn_aux_data {
 	u64 guard_place:1; /* it's the place for may_goto if the loop needs one */
 	u64 guard_at_header:1; /* there is no better place than the head of the loop */
 	u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
+	u64 guard_unset_reg:1; /* where it goes a register is read that is not set here */
 	u64 guard_edge_dropped:1; /* an edge of may_goto from here was dropped */
 	u64 need_scev:1;
 	/* a jump in a loop: the edge that is not taken, bit 0, or the other one leaves the loop */
@@ -1157,9 +1163,14 @@ struct bpf_subprog_info {
 	bool releases_inherited: 1;
 	/* it loads the end of the packet from the context */
 	bool loads_pkt_end: 1;
+	/* a call of it was verified on its own: its insns are not in insns_total of the caller */
+	bool on_its_own: 1;
 	/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
 	bool bail_unwinds: 1;
+	/* some call of it is not followed by unwind_goto */
+	bool plain_call: 1;
 	u8 code_stack_args;
+	int bail_insn; /* an exit of it, for the main pass. -1 if there is none */
 	u16 stack_arg_cnt; /* incoming + max outgoing */
 	u16 max_out_stack_arg_cnt;
 };
@@ -1713,6 +1724,7 @@ void bpf_flow_loop_gets_guards(struct bpf_verifier_env *env, int h);
 void bpf_flow_new_round(struct bpf_verifier_env *env);
 int bpf_flow_place_guards(struct bpf_verifier_env *env);
 int bpf_flow_compute_kinds(struct bpf_verifier_env *env);
+u16 bpf_flow_flag_regs(struct bpf_verifier_env *env, int idx);
 bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r);
 /* verifier.c, for flow.c */
 int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index a85ee6ee3776..a6e69c1f5ff6 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -16,6 +16,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
 	struct bpf_jmp_history_entry *p;
 	size_t alloc_size;
 
+	/* FLOW: nothing looks at the history */
+	return 0;
+
 	/* combine instruction flags if we already recorded this instruction */
 	if (env->cur_hist_ent) {
 		/* atomic instructions push insn_flags twice, for READ and
@@ -861,6 +864,10 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
 	struct bpf_func_state *func;
 	bool tmp, skip_first = true;
 	struct bpf_stack_state *ss;
+
+	/* FLOW: there are no precision marks */
+	bt_reset(bt);
+	return 0;
 	struct bpf_reg_state *reg;
 	int i, fr, err;
 
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index d01d0cc0dde5..3c8c1b64088b 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -263,6 +263,12 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
 
 		for (r = 0; r < MAX_BPF_REG; r++)
 			ci_in[start][r] = unknown;
+		/*
+		 * The subprog gets there from where it is out of budget too, see
+		 * push_budget_exit(). That is not an edge of the graph.
+		 */
+		for (r = 0; r < MAX_BPF_REG && env->subprog_info[i].bail_insn >= 0; r++)
+			ci_in[env->subprog_info[i].bail_insn][r] = unknown;
 	}
 	/* The same from the head of a loop to where the loop is left, once the loops are known. */
 	for (i = 0; i < insn_cnt; i++) {
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 3d7ec3305e64..866835eef0cc 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -570,7 +570,7 @@ static u16 flow_tested_regs(struct bpf_verifier_env *env, int idx)
 }
 
 /* A jump that compares a register with 0 or 1 */
-static u16 flow_flag_regs(struct bpf_verifier_env *env, int idx)
+u16 bpf_flow_flag_regs(struct bpf_verifier_env *env, int idx)
 {
 	struct bpf_insn *insn = &env->prog->insnsi[idx];
 
@@ -1031,7 +1031,7 @@ static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_nee
 		if (jump)
 			gen[i] = flow_tested_regs(env, i);
 		else if (flag)
-			gen[i] = flow_flag_regs(env, i);
+			gen[i] = bpf_flow_flag_regs(env, i);
 		else if (kind == FLOW_NEED_STACK_OFF)
 			gen[i] = flow_stack_off_regs(env, i);
 		else if (kind == FLOW_NEED_PTR_OFF || kind == FLOW_NEED_VALUE_OFF)
@@ -2152,13 +2152,18 @@ static bool flow_gap(const struct bpf_reg_state *a, const struct bpf_reg_state *
  * with either, so none of the above keeps them apart. The join has ok = 1 with i >= 64. A flag is
  * made to send states different ways. So 0 and 1 are not joined where a jump that is not far
  * tells one from the other: FLOW_NEED_FLAG.
+ *	while (more) { ...; more = next(&x); }
+ * That jump can be the one that leaves a loop. It is decided in both states, one stays and one
+ * leaves. It's the jump that set the flag, which went both ways, once more: no sign that the
+ * loop gets anywhere. A walk on patience has to know, see mark_loop_exit_decided().
  *
  * The states go on to the jumps on their own. Behind the last one the numbers are dead, and what
  * is left is joined.
  */
-static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
-			       const struct bpf_verifier_state *b)
+static bool flow_join_makes_up(const struct bpf_flow *flow, struct bpf_verifier_state *a,
+			       struct bpf_verifier_state *b)
 {
+	struct bpf_func_state *fa = a->frame[a->curframe], *fb = b->frame[b->curframe];
 	const struct bpf_reg_state *ra, *rb;
 	const struct flow_need *tested = flow->need[FLOW_NEED_JUMP], *need;
 	u32 fr, i, id = 0;
@@ -2186,7 +2191,13 @@ static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_ver
 			id = ra->id & ~BPF_ADD_CONST;
 		}
 	}
-	return flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b);
+	if (!flow_needed_consts_differ(flow, FLOW_NEED_FLAG, a, b))
+		return false;
+	for (i = 0; i < fa->loop_stack_cnt; i++)
+		fa->loop_stack[i].flag_apart = fa->loop_stack[i].probe;
+	for (i = 0; i < fb->loop_stack_cnt; i++)
+		fb->loop_stack[i].flag_apart = fb->loop_stack[i].probe;
+	return true;
 }
 
 /*
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 6a2ee407412b..60296eb880db 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2837,6 +2837,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		 */
 		insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
 	}
+	/* the exit that stands for 'r0 = 0; exit' may be dead code, see push_budget_exit() */
+	for (i = 0; i < env->subprog_cnt; i++)
+		if (env->subprog_info[i].bail_insn >= 0)
+			insn_aux[env->subprog_info[i].bail_insn].live_regs_before |=
+				BIT(BPF_REG_0) | (bpf_ret_reg_pair(env, i) ? BIT(BPF_REG_2) : 0);
 
 out:
 	kvfree(state);
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 916408c63355..e1c9b56cd13f 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -802,6 +802,11 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 		verbose(env, " cb");
 	if (state->in_async_callback_fn)
 		verbose(env, " async_cb");
+	/* the iterations of the loops in which they are kept apart */
+	for (i = 0; i < state->loop_stack_cnt; i++)
+		if (state->loop_stack[i].apart)
+			verbose(env, " iter(%d)=%u", state->loop_stack[i].loop_id,
+				state->loop_stack[i].trip);
 	if (state->loop_stack_cnt) {
 		verbose(env, " loop_stack=");
 		for (i = 0; i < state->loop_stack_cnt; i++)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 9d3a86b5820e..4b2506f3e3e2 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -840,6 +840,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			    old_reg->iter.btf != cur_reg->iter.btf ||
 			    old_reg->iter.btf_id != cur_reg->iter.btf_id ||
 			    old_reg->iter.state != cur_reg->iter.state ||
+			    old_reg->iter.loops != cur_reg->iter.loops ||
 			    /* ignore {old_reg,cur_reg}->iter.depth, see above */
 			    !check_ids(old_reg->id, cur_reg->id, idmap))
 				return false;
@@ -872,16 +873,19 @@ static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *
 			   struct bpf_func_state *cur, struct bpf_idmap *idmap,
 			   enum exact_level exact)
 {
-	int i, nslots;
+	int i, nslots, cur_cnt = cur->out_stack_arg_cnt;
 
-	nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt);
+	/* incoming arguments are not compared with outgoing ones, the other way is ruled out */
+	if (bpf_stack_args_are_incoming(cur) && !bpf_stack_args_are_incoming(old))
+		cur_cnt = 0;
+	nslots = max(old->out_stack_arg_cnt, cur_cnt);
 	for (i = 0; i < nslots; i++) {
 		struct bpf_reg_state *old_arg, *cur_arg;
 		struct bpf_reg_state not_init = { .type = NOT_INIT };
 
 		old_arg = i < old->out_stack_arg_cnt ?
 			  &old->stack_arg_regs[i] : &not_init;
-		cur_arg = i < cur->out_stack_arg_cnt ?
+		cur_arg = i < cur_cnt ?
 			  &cur->stack_arg_regs[i] : &not_init;
 		if (!regsafe(env, old_arg, cur_arg, idmap, exact))
 			return false;
@@ -940,6 +944,23 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
 	return true;
 }
 
+/*
+ * In a loop that is walked without a count a state of an earlier iteration stands for a state of
+ * a later one, see flow_cycle_closed(). Never the other way round.
+ *
+ * In a loop that is counted and walked the state that gets to iteration max_header_count is
+ * dropped: there is no such iteration at run time. The number in a state is the number of
+ * iterations that the program has done, and only states with the same number are compared. They
+ * differ in what is counted anyway.
+ *
+ * The walk of a loop with two entries is meant to get to the end of the loop, like today's walk of
+ * all paths. A state that is within the state of the iteration before it is on its way there:
+ *	for (i = 7; i >= 0; i--) { if (i >= n) continue; ... }
+ * i is in 0..6, then in 0..5. So here too only states with the same number are compared.
+ *
+ * Where the iterations are not kept apart a state stands for many of them, and the number says
+ * nothing about what it describes.
+ */
 static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			    struct bpf_func_state *cur)
 {
@@ -953,7 +974,13 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
 	for (i = 0; i < old->loop_stack_cnt; i++) {
 		if (old_stack[i].loop_id == cur_stack[i].loop_id &&
 		    old_stack[i].max_header_count == cur_stack[i].max_header_count &&
-		    old_stack[i].terminates == cur_stack[i].terminates)
+		    old_stack[i].terminates == cur_stack[i].terminates &&
+		    old_stack[i].guarded == cur_stack[i].guarded &&
+		    old_stack[i].counted == cur_stack[i].counted &&
+		    old_stack[i].apart == cur_stack[i].apart &&
+		    (!old_stack[i].apart ||
+		     (old_stack[i].probe ? old_stack[i].trip <= cur_stack[i].trip :
+					   old_stack[i].trip == cur_stack[i].trip)))
 			continue;
 		return false;
 	}
@@ -999,9 +1026,19 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (!old->no_stack_arg_load && cur->no_stack_arg_load)
 		return false;
 
+	if (memcmp(&old->callback_ret_range, &cur->callback_ret_range,
+		   sizeof(old->callback_ret_range)))
+		return false;
+
 	if (old->in_rcu_callback_fn != cur->in_rcu_callback_fn)
 		return false;
 
+	/* what the frame did to what its callers hold is a part of what it tells them */
+	if (old->entry_refs != cur->entry_refs ||
+	    old->entry_rcu_locks != cur->entry_rcu_locks ||
+	    old->entry_preempt_locks != cur->entry_preempt_locks)
+		return false;
+
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (((1 << i) & live_regs) &&
 		    !regsafe(env, &old->regs[i], &cur->regs[i],
@@ -1047,10 +1084,13 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->speculative && !cur->speculative)
 		return false;
 
+	if (old->callers_unknown != cur->callers_unknown)
+		return false;
+
 	if (old->in_sleepable != cur->in_sleepable)
 		return false;
 
-	if (old->unwound != cur->unwound)
+	if (old->unwound != cur->unwound || old->changed_pkt_data != cur->changed_pkt_data)
 		return false;
 
 	if (!refsafe(old, cur, &env->idmap_scratch))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 62a58f38f4e2..0e35f4454c80 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -194,13 +194,14 @@ struct bpf_verifier_stack_elem {
 	struct bpf_verifier_state st;
 	int insn_idx;
 	int prev_insn_idx;
+	/* it goes there only if may_goto in front of prev_insn_idx does, see check_loop_guard() */
+	enum bpf_guard_edge guard_edge;
 	struct bpf_verifier_stack_elem *next;
 	/* length of verifier log at the time this state was pushed on stack */
 	u32 log_pos;
 	u64 diag_log_pos;
 };
 
-#define BPF_COMPLEXITY_LIMIT_JMP_SEQ	8192
 #define BPF_COMPLEXITY_LIMIT_STATES	64
 
 #define BPF_GLOBAL_PERCPU_MA_MAX_SIZE  512
@@ -1038,6 +1039,16 @@ static bool in_rcu_cs(struct bpf_verifier_env *env);
 
 static bool is_kfunc_rcu_protected(struct bpf_call_arg_meta *meta);
 
+/* The number of loops that the frames of @st below @frameno are in */
+static u32 nr_loops_below(const struct bpf_verifier_state *st, u32 frameno)
+{
+	u32 fr, nr = 0;
+
+	for (fr = 0; fr < frameno; fr++)
+		nr += st->frame[fr]->loop_stack_cnt;
+	return nr;
+}
+
 static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 				 struct bpf_call_arg_meta *meta,
 				 struct bpf_reg_state *reg, int insn_idx,
@@ -1070,6 +1081,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 		st->iter.btf = btf;
 		st->iter.btf_id = btf_id;
 		st->iter.state = BPF_ITER_STATE_ACTIVE;
+		st->iter.loops = nr_loops_below(env->cur_state, env->cur_state->curframe + 1);
 		st->iter.depth = 0;
 
 		for (j = 0; j < BPF_REG_SIZE; j++)
@@ -1605,9 +1617,14 @@ static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx)
 
 static void release_reference_state(struct bpf_verifier_state *state, int idx)
 {
-	int last_idx;
+	int last_idx, fr;
 	size_t rem;
 
+	/* the frames that were entered with it don't hold it ever since */
+	for (fr = 0; fr <= state->curframe; fr++)
+		if (idx < state->frame[fr]->entry_refs)
+			state->frame[fr]->entry_refs--;
+
 	/* IRQ state requires the relative ordering of elements remaining the
 	 * same, since it relies on the refs array to behave as a stack, so that
 	 * it can detect out-of-order IRQ restore. Hence use memmove to shift
@@ -1622,6 +1639,20 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx)
 	return;
 }
 
+/* bpf_rcu_read_unlock() or bpf_preempt_enable() is done, see release_reference_state() */
+static void release_context_state(struct bpf_verifier_state *state)
+{
+	struct bpf_func_state *frame;
+	int fr;
+
+	for (fr = 0; fr <= state->curframe; fr++) {
+		frame = state->frame[fr];
+		frame->entry_rcu_locks = min(frame->entry_rcu_locks, state->active_rcu_locks);
+		frame->entry_preempt_locks = min(frame->entry_preempt_locks,
+						 state->active_preempt_locks);
+	}
+}
+
 static bool find_reference_state(struct bpf_verifier_state *state, int id)
 {
 	int i;
@@ -1789,7 +1820,10 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 		return err;
 	dst_state->speculative = src->speculative;
 	dst_state->in_sleepable = src->in_sleepable;
+	dst_state->callers_unknown = src->callers_unknown;
 	dst_state->unwound = src->unwound;
+	dst_state->flow_entry = src->flow_entry;
+	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
 	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
@@ -1918,11 +1952,6 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	if (err)
 		return ERR_PTR(-ENOMEM);
 	elem->st.speculative |= speculative;
-	if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
-		verbose(env, "The sequence of %d jumps is too complex.\n",
-			env->stack_size);
-		return ERR_PTR(-E2BIG);
-	}
 	if (elem->st.parent) {
 		++elem->st.parent->branches;
 		/* WARN_ON(branches > 2) technically makes sense here,
@@ -2601,12 +2630,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	elem->diag_log_pos = bpf_diag_event_log_save(env);
 	env->head = elem;
 	env->stack_size++;
-	if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
-		verbose(env,
-			"The sequence of %d jumps is too complex for async cb.\n",
-			env->stack_size);
-		return ERR_PTR(-E2BIG);
-	}
 	/* Unlike push_stack() do not bpf_copy_verifier_state().
 	 * The caller state doesn't matter.
 	 * This is async callback. It starts in a fresh stack.
@@ -3459,7 +3482,17 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
 	bool seen = false;
 	int i;
 
+	for (i = 0; i < env->subprog_cnt; i++)
+		env->subprog_info[i].bail_insn = -1;
+
 	for (i = 0; i < env->prog->len; i++, insn++) {
+		if (bpf_is_ldimm64(insn)) {
+			i++;
+			insn++;
+			continue;
+		}
+		if (insn->code == (BPF_JMP | BPF_EXIT))
+			bpf_find_containing_subprog(env, i)->bail_insn = i;
 		if (bpf_is_unwind_exit(insn)) {
 			bpf_find_containing_subprog(env, i)->can_unwind = true;
 			seen = true;
@@ -4626,6 +4659,23 @@ static int check_stack_write(struct bpf_verifier_env *env,
 	return err;
 }
 
+/*
+ * The function that is verified on its own has no caller to look at. Its incoming
+ * arguments are in its stack_arg_regs until it is done with loading them.
+ */
+static void end_stack_arg_loads(struct bpf_verifier_env *env, struct bpf_func_state *state)
+{
+	int i;
+
+	if (state->no_stack_arg_load)
+		return;
+	state->no_stack_arg_load = true;
+	if (state->frameno)
+		return;
+	for (i = 0; i < state->out_stack_arg_cnt; i++)
+		bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]);
+}
+
 /*
  * Write a value to the outgoing stack arg area.
  * off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7).
@@ -4651,6 +4701,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		return -EINVAL;
 	}
 
+	end_stack_arg_loads(env, state);
 	err = grow_stack_arg_slots(env, state, spi + 1);
 	if (err)
 		return err;
@@ -4670,7 +4721,6 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
 	}
 	bpf_diag_mod_end(env);
-	state->no_stack_arg_load = true;
 	return bpf_push_jmp_history(env, env->cur_state,
 				    INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
 }
@@ -4699,7 +4749,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
 		return -EACCES;
 	}
 
-	caller = vstate->frame[vstate->curframe - 1];
+	caller = vstate->frame[vstate->curframe ? vstate->curframe - 1 : 0];
 	arg = bpf_get_spilled_stack_arg(spi, caller);
 	if (!arg) {
 		verbose(env, "invalid read from uninitialized stack arg off %d\n", off);
@@ -9077,6 +9127,38 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
 	return &bpf_stack_slot(cur_st->frame[iter_frameno], iter_spi)->spilled_ptr;
 }
 
+/*
+ * bpf_iter_*_next() that every trip of a loop goes through stops the loop if the iterator is
+ * older than the loop: every trip takes an element, and there are only so many. It doesn't stop
+ * - the loop that makes the iterator, which starts from the beginning in every trip then:
+ *	while (v->x) { bpf_for(i, 0, 10) ...; }
+ *   Of the loops that the program is in, the ones that it entered before it made the iterator
+ *   are among the first iter.loops.
+ * - any loop if nothing was left already: the program was told so and did not leave.
+ * A loop with a known number of trips doesn't count on the iterator. Nor does a walk on patience:
+ * there is none in a loop that is taken for bounded, see flow_try_walk().
+ */
+static void iter_stops_loops(struct bpf_verifier_env *env, int insn_idx,
+			     const struct bpf_reg_state *iter)
+{
+	const struct bpf_verifier_state *st = env->cur_state;
+	struct bpf_func_state *frame = st->frame[st->curframe];
+	u32 below = nr_loops_below(st, st->curframe);
+	struct loop_stack_entry *entry;
+	bool stops;
+	int i;
+
+	/* the loops of this subprog: flow_is_bound() doesn't look into the functions it calls */
+	for (i = frame->loop_stack_cnt - 1; i >= 0; i--) {
+		entry = &frame->loop_stack[i];
+		stops = iter->iter.state != BPF_ITER_STATE_DRAINED && below + i >= iter->iter.loops;
+		if (entry->terminates || stops)
+			continue;
+		bpf_flow_next_is_no_bound(env, insn_idx, entry->loop_id);
+		return;
+	}
+}
+
 /* process_iter_next_call() is called when verifier gets to iterator's next
  * "method" (e.g., bpf_iter_num_next() for numbers iterator) call. We'll refer
  * to it as just "iter_next()" in comments below.
@@ -9158,7 +9240,7 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
 static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 				  struct bpf_call_arg_meta *meta)
 {
-	struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
+	struct bpf_verifier_state *cur_st = env->cur_state, *queued_st;
 	struct bpf_func_state *cur_fr = cur_st->frame[cur_st->curframe], *queued_fr;
 	struct bpf_reg_state *cur_iter, *queued_iter;
 
@@ -9173,20 +9255,9 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 		return -EFAULT;
 	}
 
+	iter_stops_loops(env, insn_idx, cur_iter);
+
 	if (cur_iter->iter.state == BPF_ITER_STATE_ACTIVE) {
-		/* Because iter_next() call is a checkpoint is_state_visitied()
-		 * should guarantee parent state with same call sites and insn_idx.
-		 */
-		if (!cur_st->parent || cur_st->parent->insn_idx != insn_idx ||
-		    !same_callsites(cur_st->parent, cur_st)) {
-			verifier_bug(env, "bad parent state for iter next call");
-			return -EFAULT;
-		}
-		/* Note cur_st->parent in the call below, it is necessary to skip
-		 * checkpoint created for cur_st by is_state_visited()
-		 * right at this instruction.
-		 */
-		prev_st = find_prev_entry(env, cur_st->parent, insn_idx);
 		/* branch out active iter state */
 		queued_st = push_stack(env, insn_idx + 1, insn_idx, false);
 		if (IS_ERR(queued_st))
@@ -9195,8 +9266,6 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 		queued_iter = get_iter_from_state(queued_st, meta);
 		queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
 		queued_iter->iter.depth++;
-		if (prev_st)
-			widen_imprecise_scalars(env, prev_st, queued_st);
 
 		queued_fr = queued_st->frame[queued_st->curframe];
 		mark_ptr_not_null_reg(&queued_fr->regs[BPF_REG_0]);
@@ -11523,6 +11592,9 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
 			callsite,
 			state->curframe + 1 /* frameno within this callchain */,
 			subprog /* subprog number within this prog */);
+	callee->entry_refs = state->acquired_refs;
+	callee->entry_rcu_locks = state->active_rcu_locks;
+	callee->entry_preempt_locks = state->active_preempt_locks;
 	err = set_callee_state_cb(env, caller, callee, callsite);
 	if (err)
 		goto err_out;
@@ -11757,6 +11829,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return 0;
 }
 
+enum {
+	PROCESS_BPF_EXIT = 1,
+	INSN_IDX_UPDATED = 2,
+};
 
 static int process_bpf_exit_full(struct bpf_verifier_env *env,
 				 bool *do_print_state, bool exception_exit);
@@ -11899,9 +11975,9 @@ int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *cal
 }
 
 /*
- * Call of a static subprog. The callee is verified in the context of
- * the caller, hence set up a new frame and continue from the first
- * instruction of the callee.
+ * Call of a static subprog. Unless the callee is verified on its own it is
+ * verified in the context of the caller, hence set up a new frame and
+ * continue from the first instruction of the callee.
  */
 static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
 				  int *insn_idx)
@@ -11930,7 +12006,7 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
 				    callee_info->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS));
 	if (stack_arg_cnt > caller_info->stack_arg_cnt)
 		caller_info->stack_arg_cnt = stack_arg_cnt;
-
+	/* btf_check_subprog_call() doesn't get to it when BTF is not reliable */
 	err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS + callee_incoming,
 					bpf_subprog_name(env, subprog), NULL, NULL);
 	if (err)
@@ -12448,7 +12524,15 @@ static bool is_rbtree_lock_required_kfunc(u32 btf_id);
 
 static void account_processed_insn(struct bpf_verifier_env *env)
 {
+	struct bpf_verifier_state *st = env->cur_state;
 	struct bpf_func_state *frame = cur_func(env);
+	u32 fr, i;
+
+	/* every walk on patience that the state is in pays for the insn */
+	for (fr = 0; fr <= st->curframe; fr++)
+		for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++)
+			if (bpf_loop_walk_may_give_up(&st->frame[fr]->loop_stack[i]))
+				env->walk_insns[st->frame[fr]->loop_stack[i].walk_id]++;
 
 	env->insn_processed++;
 	frame->insns_subtotal++;
@@ -12542,7 +12626,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	}
 
 	caller = state->frame[state->curframe - 1];
-	if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx])) {
+	if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx]) || state->unwound) {
 		/* the helper or the kfunc that called it doesn't expect that */
 		if (callee->in_callback_fn) {
 			verbose(env, "callback cannot unwind\n");
@@ -12561,6 +12645,16 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EACCES;
 		}
 
+		/* It's called any number of times: what it takes it has to release. */
+		for (i = callee->entry_refs; i < state->acquired_refs; i++) {
+			if (state->refs[i].type != REF_TYPE_PTR)
+				continue;
+			verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
+				state->refs[i].id, state->refs[i].insn_idx);
+			bpf_diag_leak(env, state->refs[i].id, state->refs[i].insn_idx, *insn_idx);
+			return -EINVAL;
+		}
+
 		/* we are going to rely on register's precise value */
 		err = mark_chain_precision(env, BPF_REG_0);
 		if (err)
@@ -16145,6 +16239,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			return -EINVAL;
 		}
 		env->cur_state->active_rcu_locks--;
+		release_context_state(env->cur_state);
 		bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_RCU, false,
 					env->cur_state->active_rcu_locks);
 		if (!in_rcu_cs(env))
@@ -16162,6 +16257,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			return -EINVAL;
 		}
 		env->cur_state->active_preempt_locks--;
+		release_context_state(env->cur_state);
 		bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_PREEMPT, false,
 					env->cur_state->active_preempt_locks);
 		if (!in_rcu_cs(env))
@@ -19613,6 +19709,25 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
 	}
 }
 
+/* It's known which way the jump at env->insn_idx goes */
+static void mark_loop_exit_decided(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+	struct bpf_func_state *frame = st->frame[st->curframe];
+	bool flag = bpf_flow_flag_regs(env, env->insn_idx);
+	struct loop_stack_entry *entry;
+	u32 i;
+
+	for (i = 0; i < frame->loop_stack_cnt; i++) {
+		entry = &frame->loop_stack[i];
+		/* only because the states are kept apart. That holds for the first such jump. */
+		if (entry->flag_apart && flag)
+			entry->flag_apart = false;
+		else if (entry->apart && entry->probe &&
+			 bpf_jump_leaves_loop(env, entry->loop_id, env->insn_idx))
+			entry->decided = true;
+	}
+}
+
 static int check_cond_jmp_op(struct bpf_verifier_env *env,
 			     struct bpf_insn *insn, int *insn_idx)
 {
@@ -19725,6 +19840,8 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	env->true_reg1 = *dst_reg;
 	env->true_reg2 = *src_reg;
 	pred = is_branch_taken(env, dst_reg, src_reg, opcode, is_jmp32);
+	if (pred >= 0)
+		mark_loop_exit_decided(env, this_branch);
 	if (pred >= 0) {
 		/* If we get here with a dst_reg pointer type it is because
 		 * above is_branch_taken() special cased the 0 comparison.
@@ -20337,7 +20454,7 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	}
 
 	if (!tnum_is_unknown(enforce_attach_type_range) &&
-	    tnum_in(enforce_attach_type_range, reg->var_off))
+	    reg_smax(reg) > 1)
 		env->prog->enforce_expected_attach_type = 1;
 	return 0;
 }
@@ -20833,10 +20950,6 @@ static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type typ
 	return 0;
 }
 
-enum {
-	PROCESS_BPF_EXIT = 1,
-	INSN_IDX_UPDATED = 2,
-};
 
 static int process_bpf_exit_full(struct bpf_verifier_env *env,
 				 bool *do_print_state,
@@ -21152,7 +21265,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 			}
 			mark_reg_scratched(env, BPF_REG_0);
 			if (bpf_in_stack_arg_cnt(&env->subprog_info[cur_func(env)->subprogno]))
-				cur_func(env)->no_stack_arg_load = true;
+				end_stack_arg_loads(env, cur_func(env));
 			if (bpf_is_callx(insn))
 				return check_func_callx(env, insn, &env->insn_idx);
 			if (insn->src_reg == BPF_PSEUDO_CALL)
@@ -21195,6 +21308,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 	return -EFAULT;
 }
 
+static int flow_try_walk(struct bpf_verifier_env *env, struct loop_stack_entry *entry);
+
 /*
  * Push the loop entered at insn_idx onto the stack. Usually this adds a single
  * entry on top of its enclosing loop. However, an edge may enter an inner loop
@@ -21209,7 +21324,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
  * Such a bypass makes the enclosing loop irreducible, so its header is missing
  * from the stack and both headers (1) and (2) need to be pushed onto stack.
  * Only the innermost loop (the one actually entered at insn_idx) carries SCEV bounds;
- * the bypassed ancestors are irreducible and pushed as non-terminating.
+ * the bypassed ancestors are irreducible: walked, see flow_try_walk(), or pushed as
+ * non-terminating.
  *
  * Assumes loop_stack_pop() has already truncated the stack to the common ancestor
  * of the bpf_loop_at_index(env->insn_idx) and whatever was at the top of the loop stack.
@@ -21222,7 +21338,7 @@ static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
 	u32 missing_headers[LOOP_STACK_SIZE];
 	u32 cnt = frame->loop_stack_cnt;
 	u32 num_missing = 0;
-	int h;
+	int h, err;
 
 	*pushed = false;
 
@@ -21246,7 +21362,26 @@ static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
 
 	for (; num_missing; num_missing--, cnt++) {
 		h = missing_headers[num_missing - 1];
-		loop_stack[cnt] = (struct loop_stack_entry){ .loop_id = h };
+		/* wherever it is entered: a loop can have more than one entry */
+		if (!aux[h].loop->exits_cnt) {
+			verbose(env, "loop at insn %d never exits\n", h);
+			return -EINVAL;
+		}
+		loop_stack[cnt] = (struct loop_stack_entry){
+			.loop_id = h,
+			.refs = env->cur_state->acquired_refs,
+			.entry_id = ++env->loop_entry_cnt,
+		};
+		/* there is no count for a loop that is not entered at its head */
+		if (h != env->insn_idx && !aux[h].loop->rt_bounded) {
+			frame->loop_stack_cnt = cnt;
+			err = flow_try_walk(env, &loop_stack[cnt]);
+			if (err < 0)
+				return err;
+			if (!err)
+				bpf_flow_loop_gets_guards(env, h);
+		}
+		loop_stack[cnt].guarded = aux[h].loop->guarded;
 		if (env->log.level & BPF_LOG_LEVEL2)
 			verbose(env, "entering loop %d\n", h);
 		*pushed = true;
@@ -21315,59 +21450,6 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
 	return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry);
 }
 
-static int handle_loop_entry_exit(struct bpf_verifier_env *env)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_func_state *frame = cur_func(env);
-	struct loop_stack_entry *entry;
-	struct bpf_loop_iters iters;
-	bool pushed, terminates;
-	int err;
-
-	err = loop_stack_pop(env);
-	if (err)
-		return err;
-
-	err = loop_stack_push(env, &pushed);
-	if (err)
-		return err;
-
-	if (frame->loop_stack_cnt == 0)
-		return 0;
-
-	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
-	if (env->insn_idx != entry->loop_id)
-		return 0;
-
-	/* If nothing was pushed and env->insn_idx is a loop header, we've taken a backedge. */
-	if (!pushed)
-		return maybe_clamp_scev_regs(env);
-
-	err = bpf_compute_loop_iters(env, env->cur_state, &iters);
-	if (err < 0)
-		return err;
-
-	terminates = err == 1;
-	if (!terminates)
-		return 0;
-
-	/* Create a loop_entry checkpoint before widening */
-	err = bpf_split_cur_state(env);
-	if (err)
-		return err;
-
-	entry->min_header_count = iters.min_header_count;
-	entry->max_header_count = iters.max_header_count;
-	entry->iv_reg = iters.iv_reg;
-	entry->iv_range = iters.iv_range;
-	entry->entry_state = cur->parent;
-	entry->terminates = true;
-	err = bpf_widen_scev_regs(env, cur, cur->parent, &iters);
-	if (err < 0)
-		return err;
-	return 0;
-}
-
 /*
  * The current state came back to the head of a loop. If it has a reference more than the states
  * that are there it is neither within one of them nor joined with one. It goes around again:
@@ -21564,6 +21646,7 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 		trip = entry->trip;
 		decided = !entry->probe || entry->decided || !trip;
 		entry->decided = false;
+		entry->flag_apart = false;
 		if (decided && ++entry->trip < entry->max_header_count &&
 		    (entry->counted ||
 		     env->walk_insns[entry->walk_id] <= entry->walk_patience))
@@ -21630,14 +21713,204 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	return err < 0 ? err : 0;
 }
 
-static int do_check(struct bpf_verifier_env *env)
+/* Can the main program return 0? */
+static bool zero_is_valid_retval(struct bpf_verifier_env *env)
+{
+	struct bpf_retval_range range;
+
+	if (program_returns_void(env) || !return_retval_range(env, &range))
+		return true;
+	return range.minval <= 0 && range.maxval >= 0;
+}
+
+/*
+ * A loop or a recursion is out of budget in the current frame of @st. What the
+ * program computed so far is not what it was meant to compute, so it's better
+ * not to go on. The way to stop is to unwind: 'r0 = 0; unwind_exit'. The landing
+ * pads of the callers are code like any other. They release what the callers
+ * hold, or the program is rejected.
+ *
+ * False when it cannot:
+ * - There is no landing pad for what this frame took itself.
+ * - There is a call of the subprog that unwind_goto doesn't follow. That is every
+ *   program that is compiled without unwinding.
+ * - A helper that calls a callback doesn't expect it to unwind.
+ * Then the program goes on along a path that it has, like after may_goto of
+ * its own. It releases what it holds there.
+ *
+ * @or_exit: there is no such path. 'r0 = 0; exit' has to do where the subprog
+ * cannot unwind, and the caller goes on with 0 in place of the result.
+ */
+static bool budget_can_unwind(struct bpf_verifier_env *env, const struct bpf_verifier_state *st,
+			      bool or_exit)
+{
+	const struct bpf_func_state *frame = st->frame[st->curframe];
+	const struct bpf_subprog_info *sub = &env->subprog_info[frame->subprogno];
+	bool top;
+
+	if (sub->bail_insn < 0)
+		return false;
+	if (st->acquired_refs != frame->entry_refs ||
+	    st->active_rcu_locks != frame->entry_rcu_locks ||
+	    st->active_preempt_locks != frame->entry_preempt_locks)
+		return false;
+	if (frame->in_callback_fn || frame->in_exception_callback_fn)
+		return false;
+	/* the kernel called it: either exit ends it */
+	top = !st->curframe && !st->callers_unknown &&
+	      (!frame->subprogno || frame->in_async_callback_fn);
+	if (top && !frame->subprogno && !zero_is_valid_retval(env))
+		return false;
+	return top || sub->bail_unwinds || or_exit;
+}
+
+/* May_goto at the head of a loop skips the body. Does it go to code that reads what is not set? */
+static bool guard_reads_unset_reg(struct bpf_verifier_env *env,
+				  const struct bpf_insn_aux_data *aux)
+{
+	u16 live = env->insn_aux_data[aux->guard_to].live_regs_before;
+	int r;
+
+	for (r = BPF_REG_0; r < BPF_REG_FP; r++)
+		if ((live & BIT(r)) && cur_regs(env)[r].type == NOT_INIT)
+			return true;
+	return false;
+}
+
+/*
+ * The current frame leaves with 0 in place of the result: with unwind_exit if the subprog
+ * can do that, with exit otherwise. The insns are added when the main pass is over. Until
+ * then an exit that the subprog has stands for them.
+ */
+static int push_budget_exit(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
+	struct bpf_verifier_state *branch;
+	struct bpf_reg_state *regs;
+
+	branch = push_stack(env, sub->bail_insn, env->insn_idx, false);
+	if (IS_ERR(branch))
+		return PTR_ERR(branch);
+	branch->unwound = sub->bail_unwinds;
+	regs = branch->frame[branch->curframe]->regs;
+	__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	if (bpf_ret_reg_pair(env, cur_func(env)->subprogno))
+		__mark_reg_const_zero(env, &regs[BPF_REG_2]);
+	return 0;
+}
+
+/*
+ * The loop that is not counted and has nothing that stops it at run time gets
+ * may_goto in front of this insn. When it is out of budget the program goes
+ * where may_goto goes with the registers and the stack as they are here.
+ * Every state that goes through the loop does, see bpf_flow_loop_gets_guards().
+ */
+static int check_loop_guard(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = &env->insn_aux_data[env->insn_idx];
+	struct bpf_verifier_state *branch;
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *entry;
+	struct bpf_loop *loop;
+	int h, err;
+
+	h = bpf_loop_at_index(env, env->insn_idx);
+	/* The callback came back to the helper that calls it: the program is not in front of it. */
+	if (h < 0 || !frame->loop_stack_cnt || frame->callback_depth)
+		return 0;
+	loop = env->insn_aux_data[h].loop;
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	if (loop->rt_bounded || !env->insn_aux_data[env->insn_idx].guard_place ||
+	    entry->loop_id != h)
+		return 0;
+	if (verifier_bug_if(!entry->terminates && !entry->guarded, env,
+			    "nothing stops the loop at %d", h))
+		return -EFAULT;
+	if (!entry->guarded && !bpf_loop_walk_may_give_up(entry))
+		return 0;
+
+	/* the device runs the insns that the driver was given, without may_goto */
+	if (bpf_prog_is_offloaded(env->prog->aux)) {
+		verbose(env, "loop at insn %d has no known bound in an offloaded program\n", h);
+		return -EINVAL;
+	}
+
+	/*
+	 * A guard has one way out, and only that one is looked at.
+	 *
+	 * Leave the subprog, if that is possible every time the program gets here:
+	 *	do { p = lookup(); } while (!p); *p
+	 * The code that follows the loop counts on what the jump that leaves it has found out.
+	 *
+	 * Otherwise leave the loop:
+	 *	lock(); do { ... } while (...); unlock();
+	 * The code that follows the loop releases what is held.
+	 *
+	 * From the head of a loop to where the loop is left is not an edge of the program:
+	 *	for (;;) { c = *s++; if (!c) return 0; if (c == '*') return 1; }
+	 * The loop is left to 'exit', and only its body sets r0. Leave the subprog then, with
+	 * a plain exit if it cannot unwind, like check_recursion_guard().
+	 *
+	 * It's the same way for all states that get here. As long as they leave the subprog
+	 * what would leave the loop is dropped. Both are dropped as long as the loop has
+	 * no may_goto: the state is in a walk that may get to the end of the loop.
+	 */
+	if (aux->guard_at_header && !aux->guard_unset_reg && guard_reads_unset_reg(env, aux)) {
+		aux->guard_unset_reg = true;
+		/* The states that were here left the loop. All again. */
+		if (aux->guard_no_unwind)
+			env->guards_late = true;
+		aux->guard_no_unwind = false;
+	}
+	if (!aux->guard_no_unwind &&
+	    !budget_can_unwind(env, env->cur_state, aux->guard_unset_reg)) {
+		aux->guard_no_unwind = true;
+		/* What would leave the loop from the states that were here is gone. All again. */
+		if (aux->guard_edge_dropped && loop->guarded)
+			env->guards_late = true;
+	}
+	if (!aux->guard_no_unwind) {
+		err = push_budget_exit(env);
+		if (err)
+			return err;
+		env->head->guard_edge = BPF_GUARD_LEAVES_SUBPROG;
+	}
+
+	branch = push_stack(env, env->insn_aux_data[env->insn_idx].guard_to, env->insn_idx, false);
+	if (IS_ERR(branch))
+		return PTR_ERR(branch);
+	env->head->guard_edge = BPF_GUARD_LEAVES_LOOP;
+	return 0;
+}
+
+/* The states that the insn pushed go where they were pushed to. */
+static int flow_drain(struct bpf_verifier_env *env)
+{
+	struct bpf_verifier_stack_elem *elem;
+	int err = 0;
+
+	while ((elem = env->head)) {
+		env->head = elem->next;
+		env->stack_size--;
+		if (!err && elem->guard_edge)
+			err = bpf_flow_guard_edge(env, &elem->st, elem->insn_idx,
+						  elem->prev_insn_idx, elem->guard_edge);
+		else if (!err)
+			err = bpf_flow_deliver(env, &elem->st, elem->insn_idx,
+					       elem->prev_insn_idx);
+		bpf_free_verifier_state(&elem->st, false);
+		kfree(elem);
+	}
+	return err;
+}
+
+/* Process the current state until it gets to the start of a basic block. */
+static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 {
-	bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
 	struct bpf_verifier_state *state = env->cur_state;
 	struct bpf_insn *insns = env->prog->insnsi;
 	int insn_cnt = env->prog->len;
-	bool do_print_state = false;
-	int prev_insn_idx = -1;
+	bool do_print_state = true;
 
 	for (;;) {
 		struct bpf_insn *insn;
@@ -21650,9 +21923,6 @@ static int do_check(struct bpf_verifier_env *env)
 		if (need_resched())
 			cond_resched();
 
-		/* reset current history entry on each new instruction */
-		env->cur_hist_ent = NULL;
-
 		env->prev_insn_idx = prev_insn_idx;
 		if (env->insn_idx >= insn_cnt) {
 			verbose(env, "invalid insn idx %d insn_cnt %d\n",
@@ -21665,18 +21935,10 @@ static int do_check(struct bpf_verifier_env *env)
 
 		account_processed_insn(env);
 
-		if (env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) {
-			verbose(env,
-				"BPF program is too large. Processed %d insn\n",
-				env->insn_processed);
-			return -E2BIG;
-		}
-
 		state->last_insn_idx = env->prev_insn_idx;
 		state->insn_idx = env->insn_idx;
 		/*
-		 * Record the incoming edge so active and queued paths use the same
-		 * branch-recording path. A zero-offset conditional has identical
+		 * Record the incoming edge. A zero-offset conditional has identical
 		 * successors, so its outcome cannot be reconstructed from the edge.
 		 */
 		if (!state->speculative && prev_insn_idx >= 0 && prev_insn_idx < insn_cnt) {
@@ -21696,40 +21958,6 @@ static int do_check(struct bpf_verifier_env *env)
 			}
 		}
 
-		/*
-		 * Possibly widen the registers before creating a checkpoint
-		 * in bpf_is_state_visited(). The next loop iteration will
-		 * have a chance to hit this checkpoint and converge.
-		 */
-		err = handle_loop_entry_exit(env);
-		if (err)
-			return err;
-
-		if (bpf_is_prune_point(env, env->insn_idx)) {
-			err = bpf_is_state_visited(env, env->insn_idx);
-			if (err < 0)
-				return err;
-			if (err == 1) {
-				/* found equivalent state, can prune the search */
-				if (env->log.level & BPF_LOG_LEVEL) {
-					if (do_print_state)
-						verbose(env, "\nfrom %d to %d%s: safe\n",
-							env->prev_insn_idx, env->insn_idx,
-							env->cur_state->speculative ?
-							" (speculative execution)" : "");
-					else
-						verbose(env, "%d: safe\n", env->insn_idx);
-				}
-				goto process_bpf_exit;
-			}
-		}
-
-		if (bpf_is_jmp_point(env, env->insn_idx)) {
-			err = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);
-			if (err)
-				return err;
-		}
-
 		if (env->log.level & BPF_LOG_LEVEL2 && do_print_state) {
 			verbose(env, "\nfrom %d to %d%s:",
 				env->prev_insn_idx, env->insn_idx,
@@ -21739,13 +21967,6 @@ static int do_check(struct bpf_verifier_env *env)
 			do_print_state = false;
 		}
 
-		if (bpf_loop_at_index(env, env->insn_idx) >= 0 &&
-		    cur_func(env)->loop_stack_cnt == 0) {
-			verifier_bug(env, "loop stack empty at %d, while inside the loop %d\n",
-				     env->insn_idx, bpf_loop_at_index(env, env->insn_idx));
-			return -EFAULT;
-		}
-
 		if (env->log.level & BPF_LOG_LEVEL) {
 			if (verifier_state_scratched(env))
 				print_insn_state(env, state, state->curframe);
@@ -21796,6 +22017,10 @@ static int do_check(struct bpf_verifier_env *env)
 		if (state->speculative && insn_aux->nospec)
 			goto process_bpf_exit;
 
+		err = check_loop_guard(env);
+		if (err)
+			return err;
+
 		err = do_check_insn(env, &do_print_state);
 		if (error_recoverable_with_nospec(err) && state->speculative) {
 			/* Prevent this speculative path from ever reaching the
@@ -21817,19 +22042,9 @@ static int do_check(struct bpf_verifier_env *env)
 		}
 
 		if (state->speculative && insn_aux->nospec_result) {
-			/* If we are on a path that performed a jump-op, this
-			 * may skip a nospec patched-in after the jump. This can
-			 * currently never happen because nospec_result is only
-			 * used for the write-ops
-			 * `*(size*)(dst_reg+off)=src_reg|imm32` and helper
-			 * calls. These must never skip the following insn
-			 * (i.e., bpf_insn_successors()'s opcode_info.can_jump
-			 * is false). Still, add a warning to document this in
-			 * case nospec_result is used elsewhere in the future.
-			 *
-			 * All non-branch instructions have a single
-			 * fall-through edge. For these, nospec_result should
-			 * already work.
+			/*
+			 * All non-branch instructions have a single fall-through
+			 * edge. For these, nospec_result should already work.
 			 */
 			if (verifier_bug_if((BPF_CLASS(insn->code) == BPF_JMP ||
 					     BPF_CLASS(insn->code) == BPF_JMP32) &&
@@ -21839,23 +22054,40 @@ static int do_check(struct bpf_verifier_env *env)
 process_bpf_exit:
 			account_current_path(env);
 			mark_verifier_state_scratched(env);
-			err = bpf_update_branch_counts(env, env->cur_state);
-			if (err)
-				return err;
-			err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
-					pop_log);
-			if (err < 0) {
-				if (err != -ENOENT)
-					return err;
-				break;
-			} else {
-				do_print_state = true;
-				continue;
-			}
+			bpf_free_verifier_state(state, false);
+			memset(state, 0, sizeof(*state));
+			return flow_drain(env);
+		}
+
+		err = flow_drain(env);
+		if (err)
+			return err;
+
+		if (env->insn_idx < insn_cnt && bpf_flow_is_point(env, env->insn_idx)) {
+			account_current_path(env);
+			mark_verifier_state_scratched(env);
+			return bpf_flow_deliver(env, state, env->insn_idx, prev_insn_idx);
 		}
 	}
+}
 
-	return 0;
+static int do_check(struct bpf_verifier_env *env)
+{
+	int prev_insn_idx = -1;
+	int err;
+
+	err = bpf_flow_deliver(env, env->cur_state, env->insn_idx, -1);
+	while (!err) {
+		err = bpf_flow_next(env, &prev_insn_idx);
+		if (err == -ENOENT)
+			return 0;
+		if (err)
+			break;
+		/* What is known about how the state came to be is not kept with it. */
+		bpf_diag_event_log_restore(env, 0);
+		err = do_check_block(env, prev_insn_idx);
+	}
+	return err;
 }
 
 static int find_btf_percpu_datasec(struct btf *btf)
@@ -22951,6 +23183,7 @@ static void free_states(struct bpf_verifier_env *env)
 	bpf_free_verifier_state(env->cur_state, true);
 	env->cur_state = NULL;
 	while (!pop_stack(env, NULL, NULL, false));
+	bpf_flow_reset(env);
 
 	list_for_each_safe(pos, tmp, &env->free_list) {
 		sl = container_of(pos, struct bpf_verifier_state_list, node);
@@ -23006,6 +23239,8 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 	state->speculative = false;
 	state->branches = 1;
 	state->in_sleepable = context == BPF_FUNC_CTX_SLEEPABLE;
+	/* nobody knows what the callers of a global function or of an extension hold */
+	state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
 	env->no_rcu_read_lock = context == BPF_FUNC_CTX_NO_RCU_LOCK;
 	state->frame[0] = kzalloc_obj(struct bpf_func_state, GFP_KERNEL_ACCOUNT);
 	if (!state->frame[0]) {
@@ -23112,7 +23347,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 		 * it's matching expected signature, and otherwise mark BTF
 		 * info for main program as unreliable
 		 */
-		if (env->prog->aux->func_info_aux) {
+		if (env->prog->aux->func_info_aux && !env->prog->aux->func_info_aux[0].unreliable) {
 			ret = btf_prepare_func_args(env, 0);
 			if (ret || sub->arg_slot_cnt != 1 ||
 			    sub->args[0].arg_type != ARG_PTR_TO_CTX) {
@@ -23245,6 +23480,44 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
+/*
+ * What stops a loop is assumed before the main pass and checked in it, see flow_is_bound(). The
+ * main prog and global functions may get to the same loop. It has may_goto for all of them or
+ * for none, so all are done again when the assumption doesn't hold, whatever came out.
+ * The same when a loop gets may_goto, or its may_goto gets another way out, and what went through
+ * the loop before is not there any more, see bpf_flow_loop_gets_guards().
+ *
+ * A round that is dropped counts as paths that never run. What it found out about the insns, the
+ * subprogs and the prog stays. Each of these is made for more than one state that gets to an insn:
+ * it grows to the safe side, or another value is refused, or the last one that gets there sets it.
+ * Where the round got to is dropped with its verdict: the passes that follow take an insn that was
+ * seen for one that passed its checks, and a function that is called has to be safe.
+ */
+static int do_check_all(struct bpf_verifier_env *env)
+{
+	struct bpf_func_info_aux *sub_aux;
+	u64 log_pos = env->log.end_pos;
+	int i, ret;
+
+	for (;;) {
+		bpf_flow_new_round(env);
+		ret = do_check_main(env);
+		ret = ret ?: do_check_subprogs(env);
+		if (!(env->restart && bpf_flow_loops_lost_bound(env)) && !env->guards_late)
+			return ret;
+		/* the log is dropped when a function is found to be safe */
+		bpf_vlog_reset(&env->log, min(log_pos, env->log.end_pos));
+		for (i = 0; i < env->prog->len; i++)
+			env->insn_aux_data[i].seen = 0;
+		for (i = 1; env->prog->aux->func_info && i < env->subprog_cnt; i++) {
+			sub_aux = subprog_aux(env, i);
+			memset(sub_aux->called, 0, sizeof(sub_aux->called));
+			memset(sub_aux->verified, 0, sizeof(sub_aux->verified));
+			sub_aux->called_with_pkt_ptrs = false;
+		}
+	}
+}
+
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
@@ -25245,6 +25518,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = sort_subprogs_topo(env);
 	if (ret < 0)
 		goto skip_full_check;
@@ -25287,6 +25563,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_flow_compute_kinds(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	if (env->log.level & BPF_LOG_LEVEL2)
 		log_program(env);
 
@@ -25298,14 +25578,17 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_flow_prepare(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	bpf_free_subprog_jts(env);
 
 	ret = mark_fastcall_patterns(env);
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = do_check_main(env);
-	ret = ret ?: do_check_subprogs(env);
+	ret = do_check_all(env);
 
 	/* reject recursion through the callx edges found by the main pass */
 	if (ret == 0 && env->callx_edges)
@@ -25467,6 +25750,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	bpf_patch_list_free(env);
 	kvfree(env->fd_array);
 	bpf_free_scev(env);
+	kvfree(env->walk_insns);
+	bpf_flow_free(env);
+	bpf_join_free(env);
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.postorder_nums);
 	kvfree(env->cfg.insn_postorder);
-- 
2.55.0

Re: [PATCH bpf-next v4 19/46] bpf: Verify programs as a fixpoint over the control flow graph

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 9930 bytes --]

> bpf: Verify programs as a fixpoint over the control flow graph

TL;DR:
- Unwind-exit as bail_insn may let verifier accept programs where caller sees
  r0=0 after may_goto bail (possible NULL deref/OOB)
- Callback reference-release rule may reject previously loadable programs
- Loop-exit check may wrongly reject programs; offloaded loops possibly
  rejected

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -3456,13 +3479,23 @@ static int next_callx_callee(struct bpf_verifier_env *env, int caller, int from)
>  static int check_unwind_insns(struct bpf_verifier_env *env)
>  {
>  	struct bpf_insn *insn = env->prog->insnsi;
>  	bool seen = false;
>  	int i;
>  
> +	for (i = 0; i < env->subprog_cnt; i++)
> +		env->subprog_info[i].bail_insn = -1;
> +
>  	for (i = 0; i < env->prog->len; i++, insn++) {
> +		if (bpf_is_ldimm64(insn)) {
> +			i++;
> +			insn++;
> +			continue;
> +		}
> +		if (insn->code == (BPF_JMP | BPF_EXIT))
> +			bpf_find_containing_subprog(env, i)->bail_insn = i;
>  		if (bpf_is_unwind_exit(insn)) {
>  			bpf_find_containing_subprog(env, i)->can_unwind = true;
>  			seen = true;
>  		}
>  		if (!bpf_is_unwind_goto(insn))
>  			continue;

Can bail_insn end up pointing at an unwind_exit here?  The test
insn->code == (BPF_JMP | BPF_EXIT) also matches an unwind_exit, since
bpf_is_unwind_exit() is the same code with src_reg == BPF_UNWIND_EXIT.

push_budget_exit() later pushes a state to sub->bail_insn and sets
branch->unwound = sub->bail_unwinds.  Nothing in this commit assigns
bail_unwinds, so it is always false.  But when bail_insn is an unwind_exit,
prepare_func_exit() unwinds anyway:

    if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx]) || state->unwound) {
        ...
        state->unwound = true;

so the caller is verified only at the unwind_goto landing pad.

The insns that are inserted after the main pass do something else.
bpf_insert_loop_guards() -> append_budget_exit() -> budget_exit_insn()
emits:

    if (!sub->bail_unwinds)
        return BPF_EXIT_INSN();

That is a plain 'r0 = 0; exit'.  The subprog returns normally, unwind_goto
emits no branch in the JIT, and the caller continues after the call with
r0 = 0.  The verifier never checked that path with r0 = 0.

Consider a static subprog whose last exit is an unwind_exit, with a loop
whose header guard reaches code that reads a register the head does not
set (the glob_match() case from the commit message).  check_loop_guard() ->
guard_reads_unset_reg() sets guard_unset_reg, budget_can_unwind() returns
true through or_exit, and push_budget_exit() is called.  If the normal
exits return a non-NULL PTR_TO_MAP_VALUE and the caller dereferences r0
right after 'call; unwind_goto', or return [1, N] and the caller uses
r0 - 1 as an index, the program loads.  When may_goto runs out of budget
the caller then dereferences NULL or accesses memory out of bounds.  This
needs CAP_BPF and a JIT with unwind support.  The fall-through after
unwind_goto is only a speculative path, and not followed at all when
bypass_spec_v1 is set.

The commit message says "A subprog that holds nothing is left with
'r0 = 0; exit' then. Its caller is verified with 0."  Is that true for
this case?

If the unwind_exit is dead code for a caller that has no unwind_goto, the
same state would instead get a spurious "unwind_exit, but no unwind_goto
after the call" rejection.

Would skipping unwind exits when choosing bail_insn, for example
'insn->code == (BPF_JMP | BPF_EXIT) && !bpf_is_unwind_exit(insn)', keep the
stand-in neutral whatever bail_unwinds says?  b9dc59a51bf3 ("bpf: Allow
recursive subprogs") later sets bail_unwinds = can_unwind || ..., which
makes the two agree again, but this commit and the next two
(bc7d16f2f77e, 6a00b2a0a6a9) have the mismatch, which matters for
bisection.  The same mismatch shows up in push_budget_exit() and in
check_loop_guard() below, where a LEAVES_SUBPROG guard edge is verified as
an unwind but emitted as a plain return.

[ ... ]

> @@ -12558,12 +12642,22 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>  	if (callee->in_callback_fn) {
>  		if (r0->type != SCALAR_VALUE) {
>  			verbose(env, "R0 not a scalar value\n");
>  			return -EACCES;
>  		}
>  
> +		/* It's called any number of times: what it takes it has to release. */
> +		for (i = callee->entry_refs; i < state->acquired_refs; i++) {
> +			if (state->refs[i].type != REF_TYPE_PTR)
> +				continue;
> +			verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
> +				state->refs[i].id, state->refs[i].insn_idx);
> +			bpf_diag_leak(env, state->refs[i].id, state->refs[i].insn_idx, *insn_idx);
> +			return -EINVAL;
> +		}
> +
>  		/* we are going to rely on register's precise value */
>  		err = mark_chain_precision(env, BPF_REG_0);
>  		if (err)
>  			return err;

Is it intended to bring back the rule that a synchronous callback must
release every REF_TYPE_PTR reference it acquires?  ae6e3a273f59 ("bpf:
Drop special callback reference handling") removed that rule on purpose,
because callbacks are verified as called an unknown number of times.

A program like this used to load:

  static int cb(u32 i, struct ctx *c)
  {
      if (!c->t)
          c->t = bpf_task_acquire(task);
      return 0;
  }
  ...
  bpf_loop(n, cb, &ctx, 0);
  if (ctx.t)
      bpf_task_release(ctx.t);

On the iteration that acquires, refs[entry_refs] is the task reference, so
the new loop rejects it with "Unreleased reference".

The fixpoint does not seem to need this rule for soundness.  States with
different refs[] are never joined (refsafe() and join.c compare them), so
a callback that leaks on every call still hits the state limits or the
leak check at the main exit.

The "What user space sees" list in the commit message does not mention
this change in which programs are accepted.  Should the commit message say
so, or should the check be narrowed?

[ ... ]

> @@ -21243,13 +21359,32 @@ static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
>  	if (!frame->loop_stack)
>  		return -ENOMEM;
>  	loop_stack = frame->loop_stack;
>  
>  	for (; num_missing; num_missing--, cnt++) {
>  		h = missing_headers[num_missing - 1];
> -		loop_stack[cnt] = (struct loop_stack_entry){ .loop_id = h };
> +		/* wherever it is entered: a loop can have more than one entry */
> +		if (!aux[h].loop->exits_cnt) {
> +			verbose(env, "loop at insn %d never exits\n", h);
> +			return -EINVAL;
> +		}

Can exits_cnt be zero for a loop that does have an exit?  struct bpf_loop
documents that @exits can be incomplete:

    /* @exits of a loop in this one is cut. Then @exits of this loop can
       lack exits too. */
    bool nested_exits_overflow;

In the exit computation in kernel/bpf/loops.c, once the innermost loop of
an insn has overflowed (more than MAX_LOOP_EXITS == 256 exits), later
edges from that insn are not recorded for any loop:

    iarray_for_each(s, succ) {
        if (aux[iloop].loop->exits_overflow)
            break;
        ...
        for (t = iloop; t >= 0 && t != sloop; t = aux[t].loop_header) {
            ...
            err = add_exit(aux[t].loop, i, s);

So an outer 'for (;;)' loop that is left only from the body of a nested
loop with more than 256 exits (a large switch, or a gotox whose shared
jump table leaves the inner loop), where those jumps come after the
overflow point, ends up with exits_cnt == 0 and nested_exits_overflow set.

Such a loop can still be walked to its end by flow_try_walk(), or guarded
at a dominating jump that leaves it, since flow_set_guard() uses loop
membership and not @exits.  Before this check the program could load.  Now
it is rejected the first time a state enters the loop, with a message that
is not true.

Should the check be skipped when @exits may be cut, for example
'!aux[h].loop->exits_cnt && !aux[h].loop->nested_exits_overflow'?  The
code at the end of the series (d39416384f5a) is the same, so no later
commit changes this.

[ ... ]

> @@ -21627,59 +21710,238 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop

[ ... ]

> +	if (!entry->guarded && !bpf_loop_walk_may_give_up(entry))
> +		return 0;
> +
> +	/* the device runs the insns that the driver was given, without may_goto */
> +	if (bpf_prog_is_offloaded(env->prog->aux)) {
> +		verbose(env, "loop at insn %d has no known bound in an offloaded program\n", h);
> +		return -EINVAL;
> +	}

Does this reject offloaded programs with a loop that the walk would have
accepted?

For a loop that SCEV does not count, flow_try_walk() sets entry->apart and
entry->terminates and leaves entry->guarded at loop->guarded, which is
false at first.  At this commit entry->counted is never set, so
bpf_loop_walk_may_give_up() (apart && terminates && !counted) is true.
The early return is not taken, and the offload check returns -EINVAL on
the first trip, before the walk has decided anything.

That contradicts the commit message ("counted, or walked to its end:
nothing is added").  It is also a regression against the parent commit,
where do_check() followed the paths of such a bounded loop and accepted
the program, and NFP supports backward jumps.

When the program is not offloaded, the edges pushed here are dropped by
bpf_flow_guard_edge() while the loop has no may_goto.  For offloaded
programs, would it be enough to reject once entry->guarded is set?
flow_end_walk() copies loop->guarded into entry->guarded when a walk gives
up, so a loop that really needs may_goto would still be rejected on the
next pass through the guard place.  The check is the same at the end of
the series (d39416384f5a).


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

Re: [PATCH bpf-next v4 19/46] bpf: Verify programs as a fixpoint over the control flow graph

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 6 potential
issues to consider:
- [High] bpf: missing loop_stack refs update in
  release_reference_state()
- [High] bpf: missing bpf_head_guard_target() edge in update_insn() and
  need_const
- [Medium] bpf: check_loop_guard() rejects offloaded progs during
  patience walk
- [Medium] bpf: unreset loop_guard flag and ignored fatal errors in
  do_check_all()
- [Medium] bpf: do_check_block() mishandles nospec ordering with
  check_loop_guard()
- [Medium] bpf: do_check_all() fails to reset sub_aux->unreliable on
  round restart
--

commit 493d8e622b4ec09b90d62f364aafffd0843d557b
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Verify programs as a fixpoint over the control flow graph

This commit replaces path-by-path symbolic execution in do_check() with a
fixpoint iteration over basic blocks that joins compatible states and
inserts may_goto guards into loops without proven bounds.

> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index 6a2ee407412b8..60296eb880dbe 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c
> @@ -2837,6 +2837,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>  		 */
>  		insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
>  	}
> +	/* the exit that stands for 'r0 = 0; exit' may be dead code, see push_budget_exit() */
> +	for (i = 0; i < env->subprog_cnt; i++)
> +		if (env->subprog_info[i].bail_insn >= 0)
> +			insn_aux[env->subprog_info[i].bail_insn].live_regs_before |=
> +				BIT(BPF_REG_0) | (bpf_ret_reg_pair(env, i) ? BIT(BPF_REG_2) : 0);
>  
>  out:
>  	kvfree(state);

[Severity: High]
While bpf_compute_live_registers() accounts for the synthetic edge from a
loop header to bpf_head_guard_target(env, insn_idx), do stack-slot liveness
in update_insn() and constant propagation in flow_compute_need_const() miss
that edge?

kernel/bpf/liveness.c:update_insn() {
    ...
	succ = bpf_insn_successors(env, insn_idx);
	if (succ->cnt == 0)
		return false;

	/* All instructions of one frame array share the same mask width */
	for (s = 0; s < succ->cnt; ++s) {
		unsigned long *succ_live;

		succ_live = rel_mask(fm, relative_idx(instance, succ->items[s]), FM_LIVE_BEFORE);
		for (w = 0; w < fm->words; w++)
			new_after[w] |= succ_live[w];
	}
    ...
}

When a loop has guard_at_header set and cannot unwind (for example, while a
lock or reference is held across the loop), check_loop_guard() pushes the
state at the loop header to guard_to before do_check_insn() runs at the
header.  If a stack slot is written before the loop, overwritten at the
header or on all in-loop paths to guard_to, and then read at guard_to,
update_insn() marks the slot dead at the loop header.
bpf_clean_verifier_state() in flow_arrive() then clears its spilled_ptr and
marks it STACK_POISON at the header before check_loop_guard() pushes that
state to guard_to.

At guard_to, reading the slot fails with "slot poisoned by dead code
elimination" on a valid program.  Moreover, when env->allow_uninit_stack is
set and the slot held a spilled pointer before the loop that in-loop paths
overwrite with a scalar, join_slot() joins STACK_POISON from the header
guard with the scalar slot into STACK_INVALID, allowing the pointer left on
the stack when may_goto fires at the header to be read as a scalar at
guard_to.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 62a58f38f4e22..0e35f4454c805 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -1605,9 +1617,14 @@ static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx)
>  
>  static void release_reference_state(struct bpf_verifier_state *state, int idx)
>  {
> -	int last_idx;
> +	int last_idx, fr;
>  	size_t rem;
>  
> +	/* the frames that were entered with it don't hold it ever since */
> +	for (fr = 0; fr <= state->curframe; fr++)
> +		if (idx < state->frame[fr]->entry_refs)
> +			state->frame[fr]->entry_refs--;
> +
>  	/* IRQ state requires the relative ordering of elements remaining the
>  	 * same, since it relies on the refs array to behave as a stack, so that
>  	 * it can detect out-of-order IRQ restore. Hence use memmove to shift

[Severity: High]
Does release_reference_state() also need to decrement
state->frame[fr]->loop_stack[l].refs when idx <
state->frame[fr]->loop_stack[l].refs?

loop_stack_push() records env->cur_state->acquired_refs in
loop_stack[cnt].refs, and check_refs_taken_in_loop() scans from
entry->refs to st->acquired_refs - 1 at loop backedges:

kernel/bpf/verifier.c:check_refs_taken_in_loop() {
    ...
	for (i = entry->refs; i < st->acquired_refs; i++) {
		if (st->refs[i].type != REF_TYPE_PTR)
			continue;
    ...
}

If a reference acquired before the loop (idx < entry->refs) is released
inside the loop and a new reference is acquired and leaked in the same
iteration, memmove() shifts the new reference down to entry->refs - 1 while
entry->refs stays unchanged, so check_refs_taken_in_loop() skips it at the
backedge.  In addition, loop_stack_safe() in kernel/bpf/states.c does not
check old_stack[i].refs == cur_stack[i].refs (unlike join_loop_stack()).

[ ... ]

> @@ -21630,14 +21713,204 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop

[ ... ]

> +static int check_loop_guard(struct bpf_verifier_env *env)
> +{
> +	struct bpf_insn_aux_data *aux = &env->insn_aux_data[env->insn_idx];
> +	struct bpf_verifier_state *branch;
> +	struct bpf_func_state *frame = cur_func(env);
> +	struct loop_stack_entry *entry;
> +	struct bpf_loop *loop;
> +	int h, err;
> +
> +	h = bpf_loop_at_index(env, env->insn_idx);
> +	/* The callback came back to the helper that calls it: the program is not in front of it. */
> +	if (h < 0 || !frame->loop_stack_cnt || frame->callback_depth)
> +		return 0;
> +	loop = env->insn_aux_data[h].loop;
> +	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
> +	if (loop->rt_bounded || !env->insn_aux_data[env->insn_idx].guard_place ||
> +	    entry->loop_id != h)
> +		return 0;
> +	if (verifier_bug_if(!entry->terminates && !entry->guarded, env,
> +			    "nothing stops the loop at %d", h))
> +		return -EFAULT;
> +	if (!entry->guarded && !bpf_loop_walk_may_give_up(entry))
> +		return 0;
> +
> +	/* the device runs the insns that the driver was given, without may_goto */
> +	if (bpf_prog_is_offloaded(env->prog->aux)) {
> +		verbose(env, "loop at insn %d has no known bound in an offloaded program\n", h);
> +		return -EINVAL;
> +	}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], does check_loop_guard()
reject offloaded programs during a patience walk even when the walk would
finish without needing a may_goto?

When flow_try_walk() starts walking a loop that bpf_compute_loop_iters()
could not count, entry->guarded is false, but bpf_loop_walk_may_give_up()
is true so that check_loop_guard() can track guard_no_unwind and
guard_edge_dropped in case the walk later gives up.  Testing
bpf_prog_is_offloaded() before checking entry->guarded rejects the
offloaded program on the first trip of the walk, even if the walk would
reach the end of the loop and never add a may_goto.

[ ... ]

> @@ -21796,6 +22017,10 @@ static int do_check(struct bpf_verifier_env *env)
>  		if (state->speculative && insn_aux->nospec)
>  			goto process_bpf_exit;
>  
> +		err = check_loop_guard(env);
> +		if (err)
> +			return err;
> +
>  		err = do_check_insn(env, &do_print_state);
>  		if (error_recoverable_with_nospec(err) && state->speculative) {
>  			/* Prevent this speculative path from ever reaching the

[ ... ]

> @@ -21839,23 +22054,40 @@ static int do_check(struct bpf_verifier_env *env)
>  process_bpf_exit:
>  			account_current_path(env);
>  			mark_verifier_state_scratched(env);
> -			err = bpf_update_branch_counts(env, env->cur_state);
> -			if (err)
> -				return err;
> -			err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
> -					pop_log);
> -			if (err < 0) {
> -				if (err != -ENOENT)
> -					return err;
> -				break;
> -			} else {
> -				do_print_state = true;
> -				continue;
> -			}
> +			bpf_free_verifier_state(state, false);
> +			memset(state, 0, sizeof(*state));
> +			return flow_drain(env);
> +		}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [2], should check_loop_guard()
run before the state->speculative && insn_aux->nospec exit, and should a
speculative instruction that aborts with insn_aux->nospec = true discard
states it pushed onto env->head instead of draining them?

When bpf_insert_loop_guards() prepends a may_goto sequence to a guarded
instruction, patch_list_adj_aux() keeps insn_aux->nospec on the original
instruction after the prepended may_goto jump.  At runtime, speculative
execution reaches the may_goto jump before the nospec barrier, but in
do_check_block() any speculative state reaching an instruction with
insn_aux->nospec already set (from an earlier path or a previous round in
do_check_all()) jumps to process_bpf_exit before check_loop_guard() can
verify the guard edge.

Conversely, when insn_aux->nospec is not yet set and do_check_insn() fails
with error_recoverable_with_nospec(err) on a speculative path, jumping to
process_bpf_exit calls flow_drain(env), which delivers both the guard
branch and any branch state pushed onto env->head by the aborted
do_check_insn() call before it failed.

[ ... ]

> @@ -23245,6 +23480,44 @@ static int do_check_main(struct bpf_verifier_env *env)
>  	return ret;
>  }
>  
> +/*
> + * What stops a loop is assumed before the main pass and checked in it, see flow_is_bound(). The
> + * main prog and global functions may get to the same loop. It has may_goto for all of them or
> + * for none, so all are done again when the assumption doesn't hold, whatever came out.
> + * The same when a loop gets may_goto, or its may_goto gets another way out, and what went through
> + * the loop before is not there any more, see bpf_flow_loop_gets_guards().
> + *
> + * A round that is dropped counts as paths that never run. What it found out about the insns, the
> + * subprogs and the prog stays. Each of these is made for more than one state that gets to an insn:
> + * it grows to the safe side, or another value is refused, or the last one that gets there sets it.
> + * Where the round got to is dropped with its verdict: the passes that follow take an insn that was
> + * seen for one that passed its checks, and a function that is called has to be safe.
> + */
> +static int do_check_all(struct bpf_verifier_env *env)
> +{
> +	struct bpf_func_info_aux *sub_aux;
> +	u64 log_pos = env->log.end_pos;
> +	int i, ret;
> +
> +	for (;;) {
> +		bpf_flow_new_round(env);
> +		ret = do_check_main(env);
> +		ret = ret ?: do_check_subprogs(env);
> +		if (!(env->restart && bpf_flow_loops_lost_bound(env)) && !env->guards_late)
> +			return ret;
> +		/* the log is dropped when a function is found to be safe */
> +		bpf_vlog_reset(&env->log, min(log_pos, env->log.end_pos));
> +		for (i = 0; i < env->prog->len; i++)
> +			env->insn_aux_data[i].seen = 0;

[Severity: Medium]
When do_check_all() restarts a verification round, should it also clear
env->insn_aux_data[i].loop_guard and stop immediately on fatal errors such
as -EAGAIN, -ENOMEM, -EFAULT, or -E2BIG?

bpf_flow_guard_edge() sets aux[prev_insn_idx].loop_guard = true during a
round, and neither bpf_flow_new_round() nor do_check_all() clears it.  If a
guard edge taken in round 1 is abandoned in round 2 renders an inner loop or
called static subprogram unreachable (seen == 0), bpf_insert_loop_guards()
still sees stale loop_guard == true on those dead instructions, patches in
may_goto, and appends 'r0 = 0; exit' with aux[i].seen = 1 forced on the
appended exit instructions.

Also, if env->guards_late or env->restart was set before do_check_main() or
do_check_subprogs() aborted with -EAGAIN (pending signal), -ENOMEM, -EFAULT
(verifier_bug), or -E2BIG (total_states limit), do_check_all() ignores ret,
wipes the error from env->log, and starts another round without resetting
env->total_states.

> +		for (i = 1; env->prog->aux->func_info && i < env->subprog_cnt; i++) {
> +			sub_aux = subprog_aux(env, i);
> +			memset(sub_aux->called, 0, sizeof(sub_aux->called));
> +			memset(sub_aux->verified, 0, sizeof(sub_aux->verified));
> +			sub_aux->called_with_pkt_ptrs = false;
> +		}
> +	}
> +}

[Severity: Medium]
Should this loop also reset sub_aux->unreliable = false when discarding a
round?

In a round that gets restarted (for example when a loop header guard first
pushes a state to guard_to and a later state at the header sets
aux->guard_unset_reg = true and env->guards_late = true), the stale state
pushed to guard_to can reach a subprogram call with mismatched registers
and cause btf_check_subprog_call() to set
prog->aux->func_info_aux[subprog].unreliable = true:

kernel/bpf/verifier.c:btf_check_subprog_call() {
    ...
	if (prog->aux->func_info_aux[subprog].unreliable)
		return -EINVAL;

	err = btf_check_func_arg_match(env, subprog, btf);
	if (err)
		prog->aux->func_info_aux[subprog].unreliable = true;
    ...
}

In the restarted round, btf_check_subprog_call() sees unreliable == true
and immediately returns -EINVAL for a global function (or leaves a static
subprogram marked unreliable for fentry/fexit attachment), rejecting a
valid program after its diagnostic log was already wiped by
bpf_vlog_reset().

[1] https://lore.kernel.org/r/30511ed9caaa7fab02343ad6adbd72f0c7d689b7ae89d4b0c37bdf2beb45a414@mail.kernel.org
[2] https://lore.kernel.org/r/30511ed9caaa7fab02343ad6adbd72f0c7d689b7ae89d4b0c37bdf2beb45a414@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=19

[PATCH bpf-next v4 20/46] bpf: Keep the count of a loop when a register has no closed form

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest pyperf600_nounroll, a profiler that reads the stack of Python:
  for (int i = 0; i < STACK_MAX_LEN; ++i) {                  /* 600 */
      if (!frame_ptr || !get_frame_data(..., &frame, &sym))
          break;
      ...
      event->stack[i] = *symbol_id;
      frame_ptr = frame.f_back;
  }
i is an induction variable: 600 iterations, i = [0, 599]. get_frame_data()
fills frame and sym, which are on the stack, in each iteration. These slots
have no closed form, "can't widen *fp-64, expr is ?", and scalar evolution
gives no count for the full loop. Then i is not known either. The main pass
goes through the 600 iterations, 153987 insns, and adds a may_goto without
need. pyperf600, the same loop unrolled 150 times, and strobemeta are
rejected: "math between map_value pointer and register with unbounded .."
So are res_spin_lock_test_held_lock_max and scx_lavd/lavd_dispatch.

Let the count decide what it knows about: induction variables get their
range from it, and the loop needs no may_goto. bpf_compute_loop_iters()
reports the registers and slots without a closed form in forget_regs, and
bpf_flow_loop_enter() selects a method:

1. No check depends on the value, bpf_flow_is_needed(): the bytes of sym,
   a hash, a checksum. It stays what it is when the loop is entered. What
   comes back to the head is joined with it and widened, as in a loop
   without a count. pyperf600_nounroll: 1281 insns, no may_goto.

2. A check depends on it. Selftest strobemeta_nounroll1:
     for (int i = 0; i < STROBE_MAX_STRS; ++i)               /* 25 */
         payload_off = read_str_var(cfg, i, .., data, payload_off);
   read_str_var() reads a string to &data->payload[off] and returns off or
   off + len. payload_off is loop-carried, but the step is not a constant:
   "(any r8 (+ r8 ?))". Walk the loop: each iteration has its own state,
   with [0, i] in it. The count tells where the walk ends. The limit is
   100000 insns. 17 progs of the selftests (strobemeta, profiler2,
   linked_list, xdp_synproxy) and scx_lavd/lavd_dispatch need it. A pointer
   and a slot that does not hold one full register always count as needed.

3. The walk is too long: as in 1. Widening moves only the bound that grows:
     for (i = 0; i < 100; i++) for (j = 0; j < 100; j++) {
         if (*(volatile __u64 *)v == 1) continue;
         if (n < 8) v[n++] = 0; }                            /* long n */
   n is [0, S32_MAX] at the head. As an unknown scalar: "math between ..".

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  4 ++
 kernel/bpf/scev.c            | 73 ++++++++++++++++++++++++++++--------
 kernel/bpf/verifier.c        | 16 +++++++-
 3 files changed, 76 insertions(+), 17 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 83cd9983a0b2..485c2abee665 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -353,6 +353,8 @@ struct bpf_loop_iters {
 	u32 min_header_count;   /* min number of times header is executed */
 	u32 max_header_count;   /* max number of times header is executed */
 	unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/* live, changed by the loop, no summary of how: left to the join at the head */
+	unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* [0, max] at the entry, and the loop only clears bits */
 	unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/*
@@ -362,6 +364,8 @@ struct bpf_loop_iters {
 	 */
 	int iv_reg;
 	struct cnum64 iv_range;
+	/* a pointer to the stack that stores go through moves in the loop */
+	bool spill_base;
 };
 
 /* Where may_goto that the verifier adds goes, see check_loop_guard() */
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index d62e087f811d..677b64a63be4 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2025,8 +2025,10 @@ scev_reg_for_read(struct bpf_verifier_env *env, const struct bpf_func_state *st,
 
 /*
  * Materialize a writable value in the current state, growing its stack if needed.
- * This does not go through check_stack_write(). The callers skip is_unbound_scev_reg() slots,
- * thus the buffer of a live dynptr slice, see bpf_reg_state::buf, does not become a spill here.
+ * This does not go through check_stack_write(). The buffer of a live dynptr slice, see
+ * bpf_reg_state::buf, does not become a spill here. The callers skip the slots that are
+ * is_unbound_scev_reg() when the loop is entered. A slot that a call in the loop may write
+ * has no description: it is in forget_regs, and they skip these too.
  */
 static int scev_reg_for_write(struct bpf_verifier_env *env, struct bpf_func_state *st,
 			      u32 r, struct bpf_reg_state **out)
@@ -3403,7 +3405,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	struct bpf_loop *loop;
 	int insn_idx = st->insn_idx;
 	u64 until, false_until;
-	bool same = true;
+	bool same = true, walk = false;
 	int err, i;
 	u32 r;
 
@@ -3496,10 +3498,12 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	if (err)
 		return err;
 
+	bitmap_zero(iters->forget_regs, REGS_NUM);
 	bitmap_zero(iters->mask_regs, REGS_NUM);
+	iters->spill_base = false;
 	header_env = find_header_env(scev, insn_idx);
 	for (r = 0; r < REGS_NUM; r++) {
-		struct bpf_reg_state *reg;
+		struct bpf_reg_state *reg = NULL;
 		u32 ra, r_scev, r_expr;
 		bool spill_base = false;
 
@@ -3546,13 +3550,14 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 
 		reg = scev_reg_for_linear(cur_func, r);
 		/*
-		 * The offset of a pointer to arena is not tracked: what enters the loop stands
-		 * for all iterations. If the loop makes something else of it the state that
-		 * returns to the head is not covered by the state at the head, and the body is
-		 * verified again from that state.
+		 * The offset of a pointer to arena is not tracked, so there is nothing to lose.
+		 * If the loop makes something else of it the state that returns to the head
+		 * is not covered by the state at the head.
 		 */
-		if (reg && reg->type == PTR_TO_ARENA)
+		if (reg && reg->type == PTR_TO_ARENA) {
+			__set_bit(r, iters->forget_regs);
 			continue;
+		}
 		/* If SCEV for r is (linear <reg> <slope>) */
 		if (reg && is_simple_linear(scev, r_scev, &coefs) && is_widenable_reg_type(reg)) {
 			/* 32-bit ALU operations on pointers convert them to scalars */
@@ -3601,10 +3606,30 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 				bpf_log(log, ", requires exact stack-offset tracking");
 			bpf_log(log, "\n");
 		}
-		return 0;
+		/* the count stands, it is this one that is not known in the loop */
+		if (spill_base) {
+			iters->spill_base = true;
+			walk = true;
+			continue;
+		}
+		__set_bit(r, iters->forget_regs);
+		/*
+		 * Only a walk through the iterations tells what it is in each of them. That is not
+		 * worth it for a number that nothing looks at: a sum, a hash, what a call leaves in
+		 * r1 - r5. What comes back to the head is joined with what entered then. A pointer,
+		 * or a slot that holds something else than a register, has no such join.
+		 */
+		if (!reg || (reg->type != SCALAR_VALUE && reg->type != NOT_INIT) ||
+		    bpf_flow_is_needed(env, insn_idx, r)) {
+			walk = true;
+			continue;
+		}
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, nothing looks at it\n", insn_idx);
 	}
 
-	return 1;
+	/* 2: the count is known, what some of the live registers are is not and it matters */
+	return walk ? 2 : 1;
 }
 
 /* bpf_compute_loop_iters() checked that every leaf can be unioned into acc. */
@@ -3649,6 +3674,11 @@ static int union_any_reg(struct bpf_verifier_env *env, struct bpf_func_state *lo
 	return 0;
 }
 
+/*
+ * Make @st, which is at the head of a counted loop, stand for all iterations. @loop_entry is the
+ * state that entered the loop and @iters is what bpf_compute_loop_iters() says about it.
+ * @st is a copy of @loop_entry.
+ */
 int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
 			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters)
 {
@@ -3679,16 +3709,27 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 		if (!scev_reg_alive(env, st, r))
 			continue;
 
+		/*
+		 * It stays what it is in @st. What comes back to the head is joined with it and
+		 * widened to where the jumps of the loop stop it, like in a loop without a count.
+		 * NULL or not is all that is known about a pointer to arena.
+		 */
+		if (test_bit(r, iters->forget_regs)) {
+			reg = scev_reg_for_linear(cur_func, r);
+			if (reg && reg->type == PTR_TO_ARENA) {
+				bpf_reg_forget_range(reg);
+				mark_scev_reg_scratched(env, r);
+			}
+			continue;
+		}
+
 		r_expr = header_env->reg2scev[r];
 		spill32 = spill32_counter(scev, cur_func, r_expr, r, &coefs);
-		if (!spill32 && (is_unbound_scev_reg(env, cur_func, r) ||
+		/* @iters says nothing about a register that is not known in @loop_entry */
+		if (!spill32 && (is_unbound_scev_reg(env, entry_func, r) ||
 				 is_const_scev_reg(env, insn_idx, r)))
 			continue;
 
-		reg = scev_reg_for_linear(cur_func, r);
-		if (reg && reg->type == PTR_TO_ARENA)
-			continue;
-
 		if (test_bit(r, iters->mask_regs)) {
 			/* The range stays. It's no longer equal to the registers with its id. */
 			err = scev_reg_for_write(env, cur_func, r, &reg);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0e35f4454c80..0a315a88e355 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21680,8 +21680,22 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	if (err < 0)
 		return err;
 
-	/* @per_trip: the insns of one iteration, times the walks around the loop. */
+	/*
+	 * There is no summary of what the loop does to something that is live.
+	 * Go through the iterations one by one if that is not too long.
+	 * @per_trip: the insns of one iteration, times the walks around the loop.
+	 */
 	per_trip = flow_walk_factor(cur) * loop->nr_insns;
+	if (err == 2 && per_trip <= BPF_FLOW_MAX_WALK &&
+	    iters.max_header_count <= BPF_FLOW_MAX_WALK / (u32)per_trip) {
+		entry->max_header_count = iters.max_header_count;
+		entry->terminates = true;
+		entry->counted = true;
+		entry->apart = true;
+		return 0;
+	}
+	if (err == 2 && iters.spill_base)
+		err = 0;
 	/* The walk has to get to the end of the loop, so it is known to have one. Or not: */
 	if (!err) {
 		err = flow_try_walk(env, entry);
-- 
2.55.0

[PATCH bpf-next v4 21/46] bpf: Verify static subprogs on their own

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Selftest test_verif_scale1, 90 times in a row, jhash() is static noinline:
  ptr = data + i;
  if (ptr + nh_off > data_end)
      break;
  ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++);
The verifier verifies a static subprog in the frames of its callers. States
with different frames have no join: each insn of jhash() has a state for
each way to it: "more than 1024 states that cannot be joined at insn 903".
In Rust each function of core and alloc is a static subprog: 30 tests of
rust-bpf are rejected too.

Verify a static subprog for the states at its first insn. The entry state
has one frame, with the arguments and nothing of the callers. It arrives at
the first insn like each state at an insn: dropped, joined or added. The
call waits. Each exit state goes to all calls that wait for this entry.

  2: call f  --- { r1=scalar } --->  +---------------------+
  7: call f  --- { r1=scalar } --->  | f: verified 1 time  |
  3: ...     <-- { r0=[0,7] } -----  |                     |
  8: ...     <-- { r0=[0,7] } -----  +---------------------+

test_verif_scale1: 50765 insns, today 530362.

The entry state has what a check asks the frames of the callers for: the
references and locks that they hold, sleepable, rcu callback.
inherited_id() gives what is held a name, an id that depends only on its
position, from 2^29: calls that differ only in ids share an entry. 27 progs
of the schedulers of rust-bpf call core::fmt while they hold a reference.
All other ids stay below 2^28: "too many ids". A scalar argument loses its
id and its delta. The frames of the callers stay, func_is_on_its_own(), if
 - an argument is a pointer to the stack or a dynptr: it is their memory,
 - an argument is a pointer with the id of something that is not held: the
   id of its NULL check, or of a variable offset into the packet,
 - an argument can be NULL or is the end of the packet: the subprog does
   a check for the caller. So does a subprog that loads data_end itself:
   'bool has(ctx, p, n) { return p + n <= (void *)(long)ctx->data_end; }'.
   On its own it leaves 'if (has(ctx, p, 14)) x = p[13];' without a range,
 - it gets the context of a struct_ops prog that holds a reference: an
   argument with __ref is found by its id, not by a name. Otherwise:
   "invalid bpf_context access off=8. Reference may already be released",
 - the subprog is a callback, changes packet data, has a tail call or
   LD_ABS, or releases what the caller holds.
BPF_LOG_STATS adds its insns to each caller: verifier_subprog_insn_stats.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |   3 +
 kernel/bpf/verifier.c        | 229 ++++++++++++++++++++++++++++++++++-
 2 files changed, 231 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 485c2abee665..993d8241f6f3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1213,6 +1213,9 @@ struct bpf_id_pair {
  *   check_ids() and jid_note() do not rename it. So two calls share a state at the entry, and what
  *   comes out for it, only if the same places have the same names.
  * - bpf_flow_return() puts the ids of the call that waits back. All other ids get new values.
+ * - What comes out has the ranges of all calls that share the state at the entry. So a name is good
+ *   for what holds whatever the ranges are: the same object. Other ids have no name: a number
+ *   loses its id, any other argument keeps the frames of the caller, see arg_needs_caller().
  * - env->id_gen stays below BPF_MAX_ID. So no id is a name or has a bit of BPF_ADD_CONST.
  */
 #define BPF_INHERITED_ID	0x20000000
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0a315a88e355..3c3d1c8ea452 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1822,6 +1822,7 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	dst_state->in_sleepable = src->in_sleepable;
 	dst_state->callers_unknown = src->callers_unknown;
 	dst_state->unwound = src->unwound;
+	dst_state->changed_pkt_data = src->changed_pkt_data;
 	dst_state->flow_entry = src->flow_entry;
 	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
@@ -11301,6 +11302,8 @@ static void clear_all_pkt_pointers(struct bpf_verifier_env *env)
 			mark_reg_invalid(env, reg);
 		}
 	}));
+	if (env->cur_state->flow_entry)
+		env->cur_state->changed_pkt_data = true;
 }
 
 static bool holds_pkt_pointers(struct bpf_verifier_env *env)
@@ -11849,6 +11852,185 @@ static void account_current_path(struct bpf_verifier_env *env);
  * BPF_INHERITED_ID + the position for the function, and put back at the return.
  */
 
+/* 0 if it is not held */
+static u32 inherited_id(const struct bpf_verifier_state *state, u32 id)
+{
+	int i;
+
+	/* the first one: a lock in an object that the caller owns has the id of the reference */
+	for (i = 0; id && i < state->acquired_refs; i++)
+		if (state->refs[i].id == id)
+			return BPF_INHERITED_ID + i;
+	return 0;
+}
+
+/* What func_is_on_its_own() asks of an argument, be it in a register or on the stack */
+static bool arg_needs_caller(struct bpf_verifier_env *env, const struct bpf_verifier_state *state,
+			     const struct bpf_subprog_info *sub, const struct bpf_reg_state *reg)
+{
+	/* the id of a __ref argument is kept in ctx_arg_info[], see __check_ctx_access() */
+	if (reg->type == PTR_TO_CTX && state->acquired_refs &&
+	    env->prog->type == BPF_PROG_TYPE_STRUCT_OPS)
+		return true;
+	/* only the ids of what is held have names */
+	if (reg->type != SCALAR_VALUE &&
+	    ((reg->id && !inherited_id(state, reg->id)) ||
+	     (reg->parent_id && !inherited_id(state, reg->parent_id)) ||
+	     (reg->map_uid && !inherited_id(state, reg->map_uid))))
+		return true;
+	/*
+	 * Pointers to the packet that share an id have ranges of the same size, see
+	 * find_good_pkt_pointers(). What comes out of the function has the range of all calls.
+	 */
+	if (reg_is_pkt_pointer(reg) && reg->id)
+		return true;
+	/* it checks how much of the packet is there, for the caller */
+	if ((reg->type == PTR_TO_PACKET_END ||
+	     (reg->type == PTR_TO_PACKET && sub->loads_pkt_end)) && !sub->recursive)
+		return true;
+	/* or whether there is something to point to */
+	return type_may_be_null(reg->type) && !sub->recursive;
+}
+
+static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier_state *state,
+			       int subprog)
+{
+	struct bpf_func_state *caller = state->frame[state->curframe];
+	struct bpf_subprog_info *sub = &env->subprog_info[subprog];
+	u16 live = env->insn_aux_data[sub->start].live_regs_before;
+	u32 i;
+
+	if (sub->changes_pkt_data || sub->has_tail_call || sub->has_ld_abs ||
+	    sub->is_cb || sub->is_async_cb || sub->is_exception_cb)
+		return false;
+	if (state->speculative)
+		return false;
+	if (sub->releases_inherited &&
+	    (state->acquired_refs || state->active_rcu_locks || state->active_preempt_locks))
+		return false;
+	/* what must not be done in that callback is refused by looking at the frames */
+	if (in_rbtree_lock_required_cb(env))
+		return false;
+	/* there are that many names for what is held, and bpf_flow_return() has room for them */
+	if (state->acquired_refs > BPF_ID_MAP_SIZE / 2)
+		return false;
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+		if (!(live & BIT(i)))
+			continue;
+		if (caller->regs[i].type == PTR_TO_STACK ||
+		    base_type(caller->regs[i].type) == CONST_PTR_TO_DYNPTR)
+			return false;
+		if (arg_needs_caller(env, state, sub, &caller->regs[i]))
+			return false;
+	}
+	for (i = 0; i < bpf_in_stack_arg_cnt(sub); i++) {
+		if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
+		    base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
+			return false;
+		if (arg_needs_caller(env, state, sub, &caller->stack_arg_regs[i]))
+			return false;
+	}
+	return true;
+}
+
+static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int insn_idx)
+{
+	u16 live = env->insn_aux_data[env->subprog_info[subprog].start].live_regs_before;
+	struct bpf_verifier_state *state = env->cur_state;
+	struct bpf_func_state *caller = state->frame[state->curframe], *callee;
+	struct bpf_verifier_state entry = {};
+	int i, err;
+
+	callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
+	if (!callee)
+		return -ENOMEM;
+	entry.frame[0] = callee;
+	entry.in_sleepable = state->in_sleepable;
+	entry.callers_unknown = state->callers_unknown;
+	init_func_state(env, callee, BPF_MAIN_FUNC, 0, subprog);
+	/* the frame that says so is not there when the function is verified */
+	callee->in_rcu_callback_fn = in_rcu_callback(env);
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+		if (live & BIT(i))
+			callee->regs[i] = caller->regs[i];
+	err = grow_stack_arg_slots(env, callee, bpf_in_stack_arg_cnt(&env->subprog_info[subprog]));
+	if (err) {
+		bpf_free_verifier_state(&entry, false);
+		return err;
+	}
+	for (i = 0; i < callee->out_stack_arg_cnt; i++)
+		callee->stack_arg_regs[i] = caller->stack_arg_regs[i];
+
+	err = copy_reference_state(&entry, state);
+	if (err) {
+		bpf_free_verifier_state(&entry, false);
+		return err;
+	}
+	for (i = 0; i < entry.acquired_refs; i++) {
+		/* the id of a lock is the id of the pointer to it, which may have none */
+		entry.refs[i].id = inherited_id(state, entry.refs[i].id);
+		if (entry.refs[i].type == REF_TYPE_PTR)
+			entry.refs[i].parent_id = inherited_id(state, entry.refs[i].parent_id);
+	}
+	entry.active_irq_id = inherited_id(state, entry.active_irq_id);
+	entry.active_lock_id = inherited_id(state, entry.active_lock_id);
+	callee->entry_refs = entry.acquired_refs;
+	callee->entry_rcu_locks = entry.active_rcu_locks;
+	callee->entry_preempt_locks = entry.active_preempt_locks;
+	for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+		/* what a scalar is linked to stays with the caller, so does the delta to it */
+		if (callee->regs[i].type == SCALAR_VALUE)
+			clear_scalar_id(&callee->regs[i]);
+		callee->regs[i].id = inherited_id(state, callee->regs[i].id);
+		callee->regs[i].parent_id = inherited_id(state, callee->regs[i].parent_id);
+		callee->regs[i].map_uid = inherited_id(state, callee->regs[i].map_uid);
+	}
+	for (i = 0; i < callee->out_stack_arg_cnt; i++) {
+		if (callee->stack_arg_regs[i].type == SCALAR_VALUE)
+			clear_scalar_id(&callee->stack_arg_regs[i]);
+		callee->stack_arg_regs[i].id = inherited_id(state, callee->stack_arg_regs[i].id);
+		callee->stack_arg_regs[i].parent_id =
+			inherited_id(state, callee->stack_arg_regs[i].parent_id);
+		callee->stack_arg_regs[i].map_uid =
+			inherited_id(state, callee->stack_arg_regs[i].map_uid);
+	}
+
+	account_current_path(env);
+	env->subprog_info[subprog].on_its_own = true;
+	err = bpf_flow_call(env, state, &entry, subprog, insn_idx);
+	bpf_free_verifier_state(&entry, false);
+	/* the call waits, there is nothing to go on with */
+	return err ?: PROCESS_BPF_EXIT;
+}
+
+/* The current state is at exit of a function that is verified on its own. */
+static int exit_func_on_its_own(struct bpf_verifier_env *env)
+{
+	struct bpf_verifier_state *state = env->cur_state;
+	struct bpf_func_state *callee = state->frame[0];
+	u32 i, nregs;
+	int err;
+
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+			continue;
+		verbose(env, "cannot return stack pointer to the caller\n");
+		return -EINVAL;
+	}
+	state->unwound |= bpf_is_unwind_exit(&env->prog->insnsi[env->insn_idx]);
+
+	/* Invalidate callee-local dynptrs and their slices before the frame goes away. */
+	err = destroy_dynptrs_in_stack_slots(env, callee, 0,
+					     callee->allocated_stack / BPF_REG_SIZE - 1);
+	if (err)
+		return err;
+
+	account_current_path(env);
+	err = bpf_flow_exit(env, state, env->insn_idx);
+	return err ?: PROCESS_BPF_EXIT;
+}
+
 static u32 fresh_id(struct bpf_verifier_env *env, struct bpf_idmap *idmap, u32 id)
 {
 	u32 flags = id & BPF_ADD_CONST, i;
@@ -12012,6 +12194,9 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
 	if (err)
 		return err;
 
+	if (func_is_on_its_own(env, state, subprog))
+		return call_func_on_its_own(env, subprog, *insn_idx);
+
 	/*
 	 * For regular function entry setup new frame and continue
 	 * from that frame.
@@ -20957,6 +21142,9 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 {
 	struct bpf_func_state *cur_frame = cur_func(env);
 
+	if (!exception_exit && !env->cur_state->curframe && env->cur_state->flow_entry)
+		return exit_func_on_its_own(env);
+
 	/* We must do check_reference_leak here before
 	 * prepare_func_exit to handle the case when
 	 * state->curframe > 0, it may be a callback function,
@@ -21937,6 +22125,12 @@ static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 		if (need_resched())
 			cond_resched();
 
+		/* no insn makes as many ids as there are from here to BPF_INHERITED_ID */
+		if (env->id_gen >= BPF_MAX_ID) {
+			verbose(env, "too many ids\n");
+			return -E2BIG;
+		}
+
 		env->prev_insn_idx = prev_insn_idx;
 		if (env->insn_idx >= insn_cnt) {
 			verbose(env, "invalid insn idx %d insn_cnt %d\n",
@@ -23532,6 +23726,37 @@ static int do_check_all(struct bpf_verifier_env *env)
 	}
 }
 
+/*
+ * What it took to verify a static subprog on its own is not in insns_total of its callers.
+ * Add it, once per caller. The main prog and global subprogs have all of it already.
+ */
+static void add_insns_of_callees(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	struct bpf_insn *insn = env->prog->insnsi;
+	int *added, i, j, s, c;
+
+	added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
+	if (!added)
+		return;
+	/* callees first */
+	for (i = 0; i < env->subprog_cnt; i++) {
+		s = env->subprog_topo_order[i];
+		if (!s || bpf_subprog_is_global(env, s))
+			continue;
+		for (j = si[s].start; j < si[s + 1].start; j++) {
+			if (!bpf_pseudo_call(&insn[j]))
+				continue;
+			c = bpf_find_subprog(env, j + insn[j].imm + 1);
+			if (c <= 0 || c == s || !si[c].on_its_own || added[c] == s)
+				continue;
+			added[c] = s;
+			si[s].insns_total += si[c].insns_total;
+		}
+	}
+	kvfree(added);
+}
+
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
@@ -25603,10 +25828,12 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		goto skip_full_check;
 
 	ret = do_check_all(env);
-
 	/* reject recursion through the callx edges found by the main pass */
 	if (ret == 0 && env->callx_edges)
 		ret = sort_subprogs_topo(env);
+	/* while the calls are what they are in the program */
+	if (env->log.level & BPF_LOG_STATS)
+		add_insns_of_callees(env);
 
 	/* the driver numbers the insns like the program that it was given */
 	if (ret == 0 && bpf_prog_is_offloaded(env->prog->aux))
-- 
2.55.0

Re: [PATCH bpf-next v4 21/46] bpf: Verify static subprogs on their own

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [High] bpf: bpf_flow_return() drops speculative state flag on callee
  exit
- [Medium] bpf: call_func_on_its_own() loses async callback context in
  frame[0]
- [Medium] bpf: add_insns_of_callees() double-counts insns and misses
  callx edges
- [Medium] bpf: bpf_flow_return() lazy fallback leaves stale post-call
  states
--

commit 9af28e54478c87831c1e29826a9fb6f36a5e5256
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Verify static subprogs on their own

This commit verifies eligible static subprograms independently of their
callers' stack frames using a single-frame entry state and remapped
inherited reference IDs.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 0a315a88e355a..3c3d1c8ea4529 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -11849,6 +11852,185 @@ static void account_current_path(struct bpf_verifier_env *env);

[ ... ]

> +static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier_state *state,
> +			       int subprog)
> +{
> +	struct bpf_func_state *caller = state->frame[state->curframe];
> +	struct bpf_subprog_info *sub = &env->subprog_info[subprog];
> +	u16 live = env->insn_aux_data[sub->start].live_regs_before;
> +	u32 i;
> +
> +	if (sub->changes_pkt_data || sub->has_tail_call || sub->has_ld_abs ||
> +	    sub->is_cb || sub->is_async_cb || sub->is_exception_cb)
> +		return false;
> +	if (state->speculative)
> +		return false;
> +	if (sub->releases_inherited &&
> +	    (state->acquired_refs || state->active_rcu_locks || state->active_preempt_locks))
> +		return false;

[Severity: Medium]
Can setting sub->releases_inherited (or sub->changes_pkt_data) lazily in
bpf_flow_return() leave stale post-call states at call_idx + 1 when a
subprogram only releases a caller-held resource on one branch?

When a non-releasing exit of the callee is processed first,
flow_apply_exit() -> bpf_flow_return() sees kept == nheld and delivers a
post-call state with the resource still held to call_idx + 1.  When the
releasing exit later reaches bpf_flow_return(), it sets
sub->releases_inherited = true and returns 1 to re-verify the call in the
caller's context, but the state already delivered to call_idx + 1 is not
retracted and can cause check_resource_leak() to reject a valid program.

> +	/* what must not be done in that callback is refused by looking at the frames */
> +	if (in_rbtree_lock_required_cb(env))
> +		return false;
> +	/* there are that many names for what is held, and bpf_flow_return() has room for them */
> +	if (state->acquired_refs > BPF_ID_MAP_SIZE / 2)
> +		return false;
> +	for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
> +		if (!(live & BIT(i)))
> +			continue;
> +		if (caller->regs[i].type == PTR_TO_STACK ||
> +		    base_type(caller->regs[i].type) == CONST_PTR_TO_DYNPTR)
> +			return false;
> +		if (arg_needs_caller(env, state, sub, &caller->regs[i]))
> +			return false;
> +	}
> +	for (i = 0; i < bpf_in_stack_arg_cnt(sub); i++) {
> +		if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
> +		    base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
> +			return false;
> +		if (arg_needs_caller(env, state, sub, &caller->stack_arg_regs[i]))
> +			return false;
> +	}
> +	return true;
> +}
> +
> +static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int insn_idx)
> +{
> +	u16 live = env->insn_aux_data[env->subprog_info[subprog].start].live_regs_before;
> +	struct bpf_verifier_state *state = env->cur_state;
> +	struct bpf_func_state *caller = state->frame[state->curframe], *callee;
> +	struct bpf_verifier_state entry = {};
> +	int i, err;
> +
> +	callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
> +	if (!callee)
> +		return -ENOMEM;
> +	entry.frame[0] = callee;
> +	entry.in_sleepable = state->in_sleepable;
> +	entry.callers_unknown = state->callers_unknown;
> +	init_func_state(env, callee, BPF_MAIN_FUNC, 0, subprog);
> +	/* the frame that says so is not there when the function is verified */
> +	callee->in_rcu_callback_fn = in_rcu_callback(env);

[Severity: Medium]
Does replacing entry.frame[0] with callee lose
state->frame[0]->in_async_callback_fn and callback_ret_range when a static
subprogram is called from an async callback?

kernel/bpf/verifier.c:has_rcu_read_lock() {
	if (state->active_rcu_locks || in_rcu_callback(env))
		return true;
	if (in_sleepable(env))
		return false;
	return !env->no_rcu_read_lock || state->frame[0]->in_async_callback_fn;
}

When a global subprogram is verified for BPF_FUNC_CTX_NO_RCU_LOCK
(env->no_rcu_read_lock == true) and schedules a non-sleepable async
callback such as a bpf_timer callback, has_rcu_read_lock() relies on
state->frame[0]->in_async_callback_fn.  Without copying
in_async_callback_fn and callback_ret_range to callee, has_rcu_read_lock()
returns false inside the static subprogram (rejecting RCU-protected map
helpers and misclassifying nested global calls), and check_return_code()
on a default bpf_throw() checks the main program's return range instead of
the async callback's.

> +	for (i = BPF_REG_1; i <= BPF_REG_5; i++)
> +		if (live & BIT(i))
> +			callee->regs[i] = caller->regs[i];

[ ... ]

> +/* The current state is at exit of a function that is verified on its own. */
> +static int exit_func_on_its_own(struct bpf_verifier_env *env)
> +{
> +	struct bpf_verifier_state *state = env->cur_state;
> +	struct bpf_func_state *callee = state->frame[0];
> +	u32 i, nregs;
> +	int err;
> +
> +	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
> +	for (i = 0; i < nregs; i++) {
> +		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
> +			continue;
> +		verbose(env, "cannot return stack pointer to the caller\n");
> +		return -EINVAL;
> +	}
> +	state->unwound |= bpf_is_unwind_exit(&env->prog->insnsi[env->insn_idx]);
> +
> +	/* Invalidate callee-local dynptrs and their slices before the frame goes away. */
> +	err = destroy_dynptrs_in_stack_slots(env, callee, 0,
> +					     callee->allocated_stack / BPF_REG_SIZE - 1);
> +	if (err)
> +		return err;
> +
> +	account_current_path(env);
> +	err = bpf_flow_exit(env, state, env->insn_idx);
> +	return err ?: PROCESS_BPF_EXIT;
> +}

[Severity: High]
As reported by bot+bpf-ci@kernel.org on v2 [1], does returning from a
speculative path via exit_func_on_its_own() and bpf_flow_return() drop
exit->speculative?

Although func_is_on_its_own() rejects speculative caller states at entry,
a conditional branch or pointer ALU operation inside the callee can spawn
a speculative path with state->speculative = true.  When that path reaches
BPF_EXIT, flow_apply_exit() copies the non-speculative waiting caller
state and calls bpf_flow_return(), which does not propagate
exit->speculative to caller->speculative.  The speculative return state is
then delivered to the caller as a normal architectural state, marking
architecturally dead instructions as seen and turning recoverable
speculative faults into fatal verification errors.  In addition,
flow_merge() -> flow_within() allows a non-speculative state from another
call entry to subsume a speculative state in the callee and cross-unite
their flow_entry sets via flow_unite().

[ ... ]

> @@ -23532,6 +23726,37 @@ static int do_check_all(struct bpf_verifier_env *env)
>  	}
>  }
>  
> +/*
> + * What it took to verify a static subprog on its own is not in insns_total of its callers.
> + * Add it, once per caller. The main prog and global subprogs have all of it already.
> + */
> +static void add_insns_of_callees(struct bpf_verifier_env *env)
> +{
> +	struct bpf_subprog_info *si = env->subprog_info;
> +	struct bpf_insn *insn = env->prog->insnsi;
> +	int *added, i, j, s, c;
> +
> +	added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
> +	if (!added)
> +		return;
> +	/* callees first */
> +	for (i = 0; i < env->subprog_cnt; i++) {
> +		s = env->subprog_topo_order[i];
> +		if (!s || bpf_subprog_is_global(env, s))
> +			continue;
> +		for (j = si[s].start; j < si[s + 1].start; j++) {
> +			if (!bpf_pseudo_call(&insn[j]))
> +				continue;
> +			c = bpf_find_subprog(env, j + insn[j].imm + 1);
> +			if (c <= 0 || c == s || !si[c].on_its_own || added[c] == s)
> +				continue;
> +			added[c] = s;
> +			si[s].insns_total += si[c].insns_total;
> +		}
> +	}
> +	kvfree(added);
> +}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [2], can add_insns_of_callees()
double-count si[c].insns_total and miss indirect callx callees?

When a static callee c is verified on its own at one call site (or sets
on_its_own before bpf_flow_return() falls back to in-frame verification,
or in an earlier do_check_all() round) and in-frame at another call from
static subprogram s, prepare_func_exit() -> account_processed_insns() has
already added the in-frame instructions to both si[c].insns_total and
si[s].insns_total, so adding si[c].insns_total here counts them a second
time.  Conversely, static subprograms called via callx and verified on
their own via check_func_callx() -> check_static_func_call() are skipped
because the loop only checks bpf_pseudo_call().

[1] https://lore.kernel.org/r/fbc6a76bd8cb0b115a602fe1f8064612f09afac8924a23ae5d64cf4e7fe68b43@mail.kernel.org
[2] https://lore.kernel.org/r/fbc6a76bd8cb0b115a602fe1f8064612f09afac8924a23ae5d64cf4e7fe68b43@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=21

[PATCH bpf-next v4 22/46] bpf: Allow recursive subprogs

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  candidates.sort_unstable_by(|a, b| b.0.cmp(&a.0));    scx_rusty.rs

quicksort() of the core library calls itself, and rusty_tick is rejected:
"recursive call from quicksort<..>() to quicksort<..>()". The same for
progs that drop a list, a tree or an Arc of a parent, that use BTreeMap,
serde_json or regex-automata: 100 of the 225 progs of the schedulers of
rust-bpf and 100 of its 437 tests have a subprog that gets to itself.
A verifier that follows each chain of calls has no choice: there is no
last chain.

  depth(n) { return n ? (depth(n - 1) + 1) & 255 : 0; }

The main pass verifies a static subprog for its entry states, and a call
waits for the exit states. A call of a subprog that is in progress waits
like each other call. The first insn of a recursive subprog is like a loop
head: the entry states are joined with widening. So are the exit states.

    main prog: call depth          entry state { r1=5 }
                  v
  +-->  first insn of depth()      join with widening: { r1=[0,5] }
  |        ...
  +---  call depth(n - 1)          entry state { r1=[0,4] }, the call waits
           ^
        exit of depth()            goes to all calls that wait

How deep the calls go and how many there are is not known. Thus each
subprog on a cycle of the call graph gets the checks of
"bpf: Limit depth and time of recursive subprogs at run time". When a check
fails the subprog returns 0. This is a new edge, and
check_recursion_guard() verifies it.
linux-notes.rst says 'recursive calls are limited at run time' now.

mark_recursive_subprogs() finds the cycles before the main pass: callx can
call each subprog whose address is taken, a helper calls its callback. The
main pass finds what callx calls. A mark that it does not confirm is
dropped, and the subprog gets no check.

Still rejected: a call of the main prog; recursion in an extension, in an
offloaded prog, with a tail call, LD_ABS or bpf_clone_redirect(), in
a kprobe prog, or in a prog that gets a private stack without a request,
like raw_tp and fentry progs; an extension of a subprog that is called in
a recursion; a pointer to the stack as an argument of a call between
recursive subprogs. In the static limit on the stack of a chain of calls
a recursion counts as one frame, the last one.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 Documentation/bpf/linux-notes.rst |   4 +-
 kernel/bpf/cfg.c                  |  43 ++++-
 kernel/bpf/liveness.c             |  40 ++++-
 kernel/bpf/verifier.c             | 283 ++++++++++++++++++++++++++----
 4 files changed, 320 insertions(+), 50 deletions(-)

diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
index 6c036b54a29f..12f0dc43e154 100644
--- a/Documentation/bpf/linux-notes.rst
+++ b/Documentation/bpf/linux-notes.rst
@@ -36,8 +36,8 @@ In both cases only static functions can be referenced. Therefore all functions
 that can be called indirectly are known to the verifier before it starts to
 analyze the program, and ``callx`` is verified as a direct call of every
 function that ``dst`` may point to at that instruction. The same rules apply:
-the calls can not be recursive, and the depth of the call chain and its
-combined stack size are limited.
+the depth of the call chain and its combined stack size are limited, and
+recursive calls are limited at run time.
 
 Calling helper or kernel functions through a register, indirect calls of global
 functions, and tail calls in functions that are called via ``callx`` are not
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 2ee950e9ac82..04c1c98ba248 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -75,21 +75,28 @@ static void mark_subprog_might_throw(struct bpf_verifier_env *env, int off)
 	subprog->might_throw = true;
 }
 
-/* 't' is an index of a call-site.
+/*
+ * 't' is an index of a call-site.
  * 'w' is a callee entry point.
  * Eventually this function would be called when env->cfg.insn_state[w] == EXPLORED.
- * Rely on DFS traversal order and absence of recursive calls to guarantee that
- * callee's effect marks would be correct at that moment.
+ * DFS traversal order guarantees that callee's effect marks are correct at that
+ * moment, unless the callee calls one of its callers: see close_callee_effects().
+ * Returns true if the marks of the caller changed.
  */
-static void merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
+static bool merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
 {
 	struct bpf_subprog_info *caller, *callee;
 
 	caller = bpf_find_containing_subprog(env, t);
 	callee = bpf_find_containing_subprog(env, w);
+	if (caller->changes_pkt_data >= callee->changes_pkt_data &&
+	    caller->might_sleep >= callee->might_sleep &&
+	    caller->might_throw >= callee->might_throw)
+		return false;
 	caller->changes_pkt_data |= callee->changes_pkt_data;
 	caller->might_sleep |= callee->might_sleep;
 	caller->might_throw |= callee->might_throw;
+	return true;
 }
 
 enum {
@@ -534,6 +541,33 @@ static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
 	return DONE_EXPLORING;
 }
 
+/*
+ * A subprog that calls one of its callers is EXPLORED before that caller is: it
+ * took the marks that the caller had so far, and so did everybody who called it
+ * since. Merge again at every call until nothing changes.
+ */
+static void close_callee_effects(struct bpf_verifier_env *env)
+{
+	struct bpf_insn *insn = env->prog->insnsi;
+	struct bpf_func_ptr *ptrs;
+	bool changed;
+	u32 cnt, j;
+	int t;
+
+	do {
+		changed = false;
+		for (t = 0; t < env->prog->len; t++) {
+			if (bpf_pseudo_call(&insn[t]) || bpf_pseudo_func(&insn[t]))
+				changed |= merge_callee_effects(env, t, t + insn[t].imm + 1);
+			ptrs = insn_func_ptrs(env, t, &cnt);
+			for (j = 0; j < cnt; j++)
+				changed |= merge_callee_effects(env, t, ptrs[j].xlated_off);
+			if (bpf_is_ldimm64(&insn[t]))
+				t++;
+		}
+	} while (changed);
+}
+
 /* Visits the instruction at index t and returns one of the following:
  *  < 0 - an error occurred
  *  DONE_EXPLORING - the instruction was fully explored
@@ -759,6 +793,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 		}
 	}
 	ret = 0; /* cfg looks good */
+	close_callee_effects(env);
 	env->prog->aux->changes_pkt_data = env->subprog_info[0].changes_pkt_data;
 	env->prog->aux->might_sleep = env->subprog_info[0].might_sleep;
 
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 60296eb880db..318db60ae230 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2354,6 +2354,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 		struct func_instance *callee_instance;
 		int callee, target;
 		int caller_reg, cb_callee_reg;
+		u16 live;
 
 		j = idx - start; /* relative index within this subprog */
 		memset(callee_args, 0, MAX_AT_TRACK_REGS * sizeof(*callee_args));
@@ -2369,11 +2370,17 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 			if (callee < 0 || bpf_subprog_is_global(env, callee))
 				continue;
 
-			/* Build entry args: R1-R5 and stack args from at_in at call site */
+			/*
+			 * Build entry args: R1-R5 and stack args from at_in at call site.
+			 * What is left over in a register that the callee doesn't read is no arg.
+			 */
+			live = env->insn_aux_data[target].live_regs_before;
 			for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
-				callee_args[r] = info[subprog].at_in[j][r];
+				callee_args[r] = live & BIT(r) ? info[subprog].at_in[j][r] : none;
 			for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
-				callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r];
+				callee_args[MAX_BPF_REG + r] =
+					r < bpf_in_stack_arg_cnt(&env->subprog_info[callee]) ?
+					info[subprog].at_in[j][MAX_BPF_REG + r] : none;
 		} else if (bpf_calls_callback(env, idx)) {
 			callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg);
 			if (callee == -2) {
@@ -2408,6 +2415,17 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 		if (!has_fp_args(callee_args))
 			continue;
 
+		/*
+		 * at_in[] is kept per subprog and the snapshot of the stack per call insn:
+		 * the walk must not enter a subprog that it has not left.
+		 */
+		if (env->subprog_info[subprog].recursive && env->subprog_info[callee].recursive) {
+			verbose(env, "recursive %s() passes a pointer to the stack to %s()\n",
+				bpf_subprog_name(env, subprog), bpf_subprog_name(env, callee));
+			err = -EINVAL;
+			goto out_free;
+		}
+
 		if (depth == MAX_CALL_FRAMES - 1) {
 			verbose(env, "%s() passes a pointer to the stack to %s()\n",
 				bpf_subprog_name(env, subprog), bpf_subprog_name(env, callee));
@@ -2781,14 +2799,15 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
 	}
 
-	/* Forward pass: resolve stack access through FP-derived pointers */
-	err = bpf_compute_subprog_arg_access(env);
-	if (err)
-		goto out;
-
 	changed = true;
 	while (changed) {
 		changed = false;
+		/* a cycle of calls takes a round for each link */
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			goto out;
+		}
+		cond_resched();
 		for (i = 0; i < env->cfg.cur_postorder; ++i) {
 			int insn_idx = env->cfg.insn_postorder[i];
 			struct insn_live_regs *live = &state[insn_idx];
@@ -2843,6 +2862,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			insn_aux[env->subprog_info[i].bail_insn].live_regs_before |=
 				BIT(BPF_REG_0) | (bpf_ret_reg_pair(env, i) ? BIT(BPF_REG_2) : 0);
 
+	/*
+	 * Forward pass: resolve stack access through FP-derived pointers.
+	 * It looks at live_regs_before of the callees.
+	 */
+	err = bpf_compute_subprog_arg_access(env);
 out:
 	kvfree(state);
 	return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 3c3d1c8ea452..ef68796c04dd 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3441,7 +3441,7 @@ static int check_subprogs(struct bpf_verifier_env *env)
 /*
  * The callee of callx is known to the main verification pass only, which
  * records the 'caller' -> 'callee' edge of the call graph for the checks
- * that follow it: absence of recursion and the maximum stack depth.
+ * that follow it: recursion that was not seen coming and the maximum stack depth.
  */
 static int record_callx_edge(struct bpf_verifier_env *env, int caller, int callee)
 {
@@ -3529,9 +3529,9 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
  * the current subprog to discover callees (direct subprogs and sync
  * callbacks).
  *
- * The callees of callx are not known before the main verification pass.
- * When callx is used the sort is repeated after it with the recorded callx
- * edges added to the call graph to reject recursion through indirect calls.
+ * The callees of callx are not known before the main verification pass: callx
+ * is no edge here. The order within a cycle is arbitrary. For recursion see
+ * mark_recursive_subprogs() and check_recursion().
  */
 static int sort_subprogs_topo(struct bpf_verifier_env *env)
 {
@@ -3591,20 +3591,19 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env)
 				}
 				if (color[callee] == 2)
 					continue;
-				if (color[callee] == 1) {
-					if (bpf_pseudo_func(&insn[idx]))
-						continue;
-					verbose(env, "recursive call from %s() to %s()\n",
-						bpf_subprog_name(env, cur),
-						bpf_subprog_name(env, callee));
-					bpf_diag_program_structure(
-						env, idx, "recursive subprogram call",
-						"Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
-						"This bpf2bpf call would make the subprogram call graph recursive. "
-						"The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
+				/*
+				 * The call that is its own target never returns. It is
+				 * also what libbpf leaves in place of a call that it
+				 * could not resolve.
+				 */
+				if (bpf_pseudo_call(&insn[idx]) && insn[idx].imm == -1) {
+					verbose(env, "call at insn %d calls itself\n", idx);
 					ret = -EINVAL;
 					goto out;
 				}
+				/* recursion: the order within the cycle is arbitrary */
+				if (color[callee] == 1)
+					continue;
 				color[callee] = 1;
 				dfs_stack[top++] = callee;
 				pushed = true;
@@ -5860,7 +5859,8 @@ static void close_call_graph(unsigned long *calls, int cnt)
  *
  * What follows for a subprog from its callers, or from what it calls, is computed
  * over this graph and not by a walk of one chain of calls after another: the
- * number of chains grows exponentially.
+ * number of chains grows exponentially, and there is no end to them when subprogs
+ * call themselves.
  */
 struct bpf_call_graph {
 	int cnt; /* subprogs */
@@ -5870,6 +5870,7 @@ struct bpf_call_graph {
 	unsigned long *live; /* what the kernel calls and what that gets to */
 	unsigned long *in_cb; /* callbacks and what they get to */
 	unsigned long *in_async_cb; /* async callbacks and what they get to */
+	unsigned long *in_recursion; /* recursive subprogs and what they get to */
 	unsigned long *tail_calls; /* subprogs that have a tail call */
 	unsigned long *arena_sp; /* subprogs that count on r12 of the caller */
 };
@@ -5900,14 +5901,15 @@ static int build_call_graph(struct bpf_verifier_env *env, struct bpf_call_graph
 
 	cg->cnt = cnt;
 	cg->words = words;
-	cg->calls = kvcalloc((2 * cnt + 5) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+	cg->calls = kvcalloc((2 * cnt + 6) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
 	if (!cg->calls)
 		return -ENOMEM;
 	cg->reach = cg->calls + cnt * words;
 	cg->live = cg->reach + cnt * words;
 	cg->in_cb = cg->live + words;
 	cg->in_async_cb = cg->in_cb + words;
-	cg->tail_calls = cg->in_async_cb + words;
+	cg->in_recursion = cg->in_async_cb + words;
+	cg->tail_calls = cg->in_recursion + words;
 	cg->arena_sp = cg->tail_calls + words;
 
 	for (s = 0; s < cnt; s++) {
@@ -5942,6 +5944,8 @@ static int build_call_graph(struct bpf_verifier_env *env, struct bpf_call_graph
 			add_reach(cg, cg->in_cb, s);
 		if (si[s].is_async_cb)
 			add_reach(cg, cg->in_async_cb, s);
+		if (test_bit(s, subprog_reach(cg, s)))
+			add_reach(cg, cg->in_recursion, s);
 		if (si[s].has_tail_call)
 			__set_bit(s, cg->tail_calls);
 	}
@@ -5984,20 +5988,47 @@ static int check_exception_cb(struct bpf_verifier_env *env, struct bpf_call_grap
 	return 0;
 }
 
+static bool bpf_prog_reenters_datapath(const struct bpf_prog *prog);
+
 /*
- * The DFS of sort_subprogs_topo() follows the address of a function like a call, and
- * tolerates a cycle only when such address closes it, since it doesn't know what it is
- * taken for. Such cycle counts as a recursion, unless the function is an async callback:
- * its address is not an edge of the graph. The main verification pass limits the
- * depth of the recursion, but it doesn't follow calls of global functions.
+ * Every subprog on a cycle has to check the stack at run time. mark_recursive_subprogs()
+ * didn't see the cycle coming when a helper got the callback from a table of
+ * functions, or from another subprog. A subprog that it marked for callx, which turned out
+ * to call something else, needs no check.
  */
 static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *cg)
 {
 	int j, s;
 
 	for_each_set_bit(s, cg->live, cg->cnt) {
-		if (!test_bit(s, subprog_reach(cg, s)))
+		if (!test_bit(s, subprog_reach(cg, s))) {
+			env->subprog_info[s].recursive = false;
 			continue;
+		}
+		if (env->subprog_info[s].recursive) {
+			/*
+			 * These progs run on top of other progs. Their frames belong in the
+			 * private stack, which has room for one frame of a subprog. A kprobe prog
+			 * that can be attached to a uprobe has no private stack, and is no better.
+			 */
+			if ((bpf_enable_priv_stack(env->prog) != NO_PRIV_STACK ||
+			     env->prog->type == BPF_PROG_TYPE_KPROBE) &&
+			    !env->prog->aux->priv_stack_requested) {
+				verbose(env, "recursive %s() in a tracing program\n",
+					bpf_subprog_name(env, s));
+				return -EINVAL;
+			}
+			/* It runs on top of its own frames. The half of the stack is not enough. */
+			if (bpf_prog_reenters_datapath(env->prog)) {
+				verbose(env, "recursive %s() in a program that calls bpf_clone_redirect()\n",
+					bpf_subprog_name(env, s));
+				return -EINVAL;
+			}
+			if (bpf_prog_has_invocation_budget(env->prog))
+				continue;
+			verbose(env, "JIT does not support recursive calls\n");
+			return -EOPNOTSUPP;
+		}
 		/* the last call of the cycle */
 		for_each_set_bit(j, subprog_reach(cg, s), cg->cnt)
 			if (test_bit(s, subprog_calls(cg, j)))
@@ -6006,9 +6037,9 @@ static int check_recursion(struct bpf_verifier_env *env, struct bpf_call_graph *
 			bpf_subprog_name(env, j), bpf_subprog_name(env, s));
 		bpf_diag_program_structure(
 			env, find_call_insn(env, j, s), "recursive subprogram call",
-			"Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
-			"This call or callback would make the subprogram call graph recursive. "
-			"The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
+			"Load the address of the callback in the subprogram that passes it to the helper, or rewrite the recursion as a loop.",
+			"This call or callback closes a cycle of the call graph that was not seen before the main pass: "
+			"the callback comes from a table of functions or from another subprogram. Nothing checks the stack.");
 		return -EINVAL;
 	}
 	return 0;
@@ -6123,7 +6154,8 @@ static int subprog_stack_size(struct bpf_verifier_env *env, int s)
 
 /*
  * A subprog is entered with no more than what one of its callers is entered with plus
- * the frame of that caller. The graph has no cycles, so this comes to an end.
+ * the frame of that caller. Not when both are recursive: the callee checks the stack at run
+ * time. A cycle adds nothing, so this comes to an end.
  */
 static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph *cg,
 			       struct bpf_subprog_call_depth_info *dinfo)
@@ -6143,11 +6175,17 @@ static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_gra
 		for_each_set_bit(s, cg->live, cg->cnt) {
 			if (dinfo[s].depth < 0)
 				continue;
-			depth = dinfo[s].depth + subprog_stack_size(env, s);
-			arena_depth = dinfo[s].arena_depth + si[s].arena_stack_depth;
 			for_each_set_bit(j, subprog_calls(cg, s), cg->cnt) {
 				callee = &dinfo[j];
-				frame = bpf_subprog_is_global(env, j) ? 0 : dinfo[s].frame + 1;
+				depth = dinfo[s].depth;
+				arena_depth = dinfo[s].arena_depth;
+				if (!si[s].recursive || !si[j].recursive) {
+					depth += subprog_stack_size(env, s);
+					arena_depth += si[s].arena_stack_depth;
+				}
+				frame = dinfo[s].frame + !si[j].recursive;
+				if (bpf_subprog_is_global(env, j))
+					frame = 0;
 				if (depth <= callee->depth && arena_depth <= callee->arena_depth &&
 				    frame <= callee->frame)
 					continue;
@@ -6274,6 +6312,10 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
 		 * of caller's stack as shown on the example above.
 		 */
 		depth = dinfo[s].depth;
+		/* nor of the frames of recursive subprogs, see bpf_insert_recursion_checks() */
+		si[s].in_recursion = test_bit(s, cg->in_recursion);
+		if (si[s].in_recursion)
+			depth += THREAD_SIZE / 2;
 		if (s && si[s].has_tail_call && depth >= 256) {
 			verbose(env,
 				"tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\n",
@@ -6311,6 +6353,151 @@ static int check_call_depth(struct bpf_verifier_env *env, struct bpf_call_graph
 	return err;
 }
 
+/*
+ * The address of the callback at @idx goes nowhere but to a helper that keeps it for
+ * later: a timer, a work queue, call_rcu.
+ */
+static bool func_is_kept_for_later(struct bpf_verifier_env *env, int idx, int end)
+{
+	struct bpf_insn *insn = env->prog->insnsi;
+	u8 class, reg = insn[idx].dst_reg;
+	int i;
+
+	if (reg < BPF_REG_1 || reg > BPF_REG_5)
+		return false;
+	for (i = idx + 2; i < end; i++) {
+		class = BPF_CLASS(insn[i].code);
+		if (class == BPF_JMP || class == BPF_JMP32)
+			return bpf_is_async_callback_calling_insn(&insn[i]);
+		if (insn[i].dst_reg == reg)
+			return false;
+		if (bpf_is_ldimm64(&insn[i]))
+			i++;
+		else if (insn[i].src_reg == reg)
+			return false;
+	}
+	return false;
+}
+
+/*
+ * Find the subprogs on cycles of the call graph. It is not known yet what callx
+ * calls: it may be any subprog whose address is taken. A subprog that is
+ * marked for no reason is treated like a recursive one all the same.
+ */
+static int mark_recursive_subprogs(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *si = env->subprog_info;
+	struct bpf_insn *insn = env->prog->insnsi;
+	int cnt = env->subprog_cnt, words = BITS_TO_LONGS(cnt);
+	unsigned long *calls, *taken, *callx;
+	bool plain_callx = false;
+	int i, j, s, err = 0;
+
+	/* calls + s * words: the subprogs that s calls */
+	calls = kvcalloc((cnt + 2) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+	if (!calls)
+		return -ENOMEM;
+	taken = calls + cnt * words;
+	callx = taken + words;
+
+	for (s = 0; s < cnt; s++) {
+		for (i = si[s].start; i < si[s + 1].start; i++) {
+			if (bpf_is_callx(&insn[i])) {
+				__set_bit(s, callx);
+				if (!bpf_is_unwind_goto(&insn[i + 1]))
+					plain_callx = true;
+			}
+			if (bpf_pseudo_call(&insn[i]) || bpf_pseudo_func(&insn[i])) {
+				j = bpf_find_subprog(env, i + insn[i].imm + 1);
+				if (j < 0)
+					continue;
+				/*
+				 * The first insns of the main prog are for the kernel. A call from
+				 * an async callback is on no cycle.
+				 */
+				if (!j && bpf_pseudo_call(&insn[i])) {
+					verbose(env, "recursive call of the main program\n");
+					kvfree(calls);
+					return -EINVAL;
+				}
+				/* the helper that calls a callback doesn't expect it to unwind */
+				if (bpf_pseudo_func(&insn[i]) || !bpf_is_unwind_goto(&insn[i + 1]))
+					si[j].plain_call = true;
+				/*
+				 * The helper that takes a callback calls it, or keeps it: the
+				 * callback that arms itself again doesn't call itself.
+				 */
+				if (bpf_pseudo_call(&insn[i]) ||
+				    !func_is_kept_for_later(env, i, si[s + 1].start))
+					__set_bit(j, calls + s * words);
+				if (bpf_pseudo_func(&insn[i]))
+					__set_bit(j, taken);
+			}
+			if (bpf_is_ldimm64(&insn[i]))
+				i++;
+		}
+	}
+	for (i = 0; i < env->func_ptr_cnt; i++) {
+		j = bpf_find_subprog(env, env->func_ptrs[i].xlated_off);
+		if (j >= 0)
+			__set_bit(j, taken);
+	}
+	for_each_set_bit(s, callx, cnt)
+		bitmap_or(calls + s * words, calls + s * words, taken, cnt);
+
+	/*
+	 * Callers of a global subprog look at can_unwind before the subprog is verified.
+	 * The main prog returns to the kernel.
+	 */
+	for (s = 0; s < cnt; s++) {
+		if (plain_callx && test_bit(s, taken))
+			si[s].plain_call = true;
+		si[s].bail_unwinds = si[s].can_unwind ||
+				     (s && !si[s].plain_call && !bpf_subprog_is_global(env, s) &&
+				      bpf_jit_supports_unwind() &&
+				      !bpf_prog_is_offloaded(env->prog->aux));
+	}
+
+	close_call_graph(calls, cnt);
+
+	for (s = 0; s < cnt; s++) {
+		if (!test_bit(s, calls + s * words))
+			continue;
+		err = -EINVAL;
+		if (!s) {
+			verbose(env, "recursive call of the main program\n");
+			break;
+		}
+		/*
+		 * Each call of the function that it replaces is an invocation of its own: a
+		 * loop in the prog that it extends gets a new time slice each time.
+		 */
+		if (env->prog->type == BPF_PROG_TYPE_EXT) {
+			verbose(env, "recursive %s() in an extension\n", bpf_subprog_name(env, s));
+			break;
+		}
+		/* the device runs the insns that the driver was given, without the checks */
+		if (bpf_prog_is_offloaded(env->prog->aux)) {
+			verbose(env, "recursive %s() in an offloaded program\n",
+				bpf_subprog_name(env, s));
+			break;
+		}
+		if (si[s].has_tail_call || si[s].has_ld_abs) {
+			verbose(env, "recursive %s() cannot have %s\n", bpf_subprog_name(env, s),
+				si[s].has_tail_call ? "tail calls" : "LD_ABS");
+			break;
+		}
+		if (si[s].bail_insn < 0) {
+			verbose(env, "recursive %s() has no exit\n", bpf_subprog_name(env, s));
+			break;
+		}
+		si[s].recursive = true;
+		err = 0;
+	}
+	kvfree(calls);
+	return err;
+}
+
 static int check_max_stack_depth(struct bpf_verifier_env *env)
 {
 	enum priv_stack_mode priv_stack_mode = PRIV_STACK_UNKNOWN;
@@ -6351,10 +6538,11 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
 		 * All async_cb subprogs use normal kernel stack. If a particular
 		 * subprog appears in both main prog and async_cb subtree, that
 		 * subprog will use normal kernel stack to avoid potential nesting.
+		 * The private stack has room for one frame of a subprog.
 		 * x86-64 uses R9 for both private stack frame pointer and arg6.
 		 */
 		if (priv_stack_mode != PRIV_STACK_ADAPTIVE || test_bit(s, cg.in_async_cb) ||
-		    (IS_ENABLED(CONFIG_X86_64) && si[s].stack_arg_cnt)) {
+		    si[s].recursive || (IS_ENABLED(CONFIG_X86_64) && si[s].stack_arg_cnt)) {
 			si[s].priv_stack_mode = NO_PRIV_STACK;
 			continue;
 		}
@@ -22106,6 +22294,21 @@ static int flow_drain(struct bpf_verifier_env *env)
 	return err;
 }
 
+/*
+ * A subprog on a cycle of the call graph checks the stack at run time, see
+ * bpf_insert_recursion_checks(). When there is no room, or the time slice is over, it leaves
+ * at once with 0 in place of the result: it unwinds if it can, see bail_unwinds, otherwise
+ * it returns and callers have to be fine with that.
+ */
+static int check_recursion_guard(struct bpf_verifier_env *env)
+{
+	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
+
+	if (!sub->recursive || env->insn_idx != sub->start)
+		return 0;
+	return push_budget_exit(env);
+}
+
 /* Process the current state until it gets to the start of a basic block. */
 static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 {
@@ -22229,6 +22432,10 @@ static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
 		if (err)
 			return err;
 
+		err = check_recursion_guard(env);
+		if (err)
+			return err;
+
 		err = do_check_insn(env, &do_print_state);
 		if (error_recoverable_with_nospec(err) && state->speculative) {
 			/* Prevent this speculative path from ever reaching the
@@ -24325,6 +24532,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			bpf_log(log, "Only fentry can attach to %s, which can unwind\n", tname);
 			return -EINVAL;
 		}
+		/* It can have tail calls, which are not allowed there, see check_call_depth() */
+		if (aux->func && aux->func[subprog]->aux->in_recursion && prog_extension) {
+			bpf_log(log, "Cannot replace %s, which is called in a recursion\n", tname);
+			return -EINVAL;
+		}
 		conservative = aux->func_info_aux[subprog].unreliable;
 		if (prog_extension) {
 			if (aux->func_info_aux[subprog].linkage != BTF_FUNC_GLOBAL ||
@@ -25757,10 +25969,12 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = sort_subprogs_topo(env);
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = sort_subprogs_topo(env);
+	/* what no JIT can run is reported first */
+	ret = mark_recursive_subprogs(env);
 	if (ret < 0)
 		goto skip_full_check;
 
@@ -25828,9 +26042,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		goto skip_full_check;
 
 	ret = do_check_all(env);
-	/* reject recursion through the callx edges found by the main pass */
-	if (ret == 0 && env->callx_edges)
-		ret = sort_subprogs_topo(env);
 	/* while the calls are what they are in the program */
 	if (env->log.level & BPF_LOG_STATS)
 		add_insns_of_callees(env);
-- 
2.55.0

Re: [PATCH bpf-next v4 22/46] bpf: Allow recursive subprogs

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3178 bytes --]

> bpf: Allow recursive subprogs

TL;DR: The new liveness check for stack pointers passed between recursive
subprogs may wrongly reject programs that loaded before, because provisional
recursive marks from callx edges can be false cycles.

> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c

[ ... ]

> @@ -2408,6 +2415,17 @@ static int analyze_subprog(struct bpf_verifier_env *env,
>  		if (!has_fp_args(callee_args))
>  			continue;
>  
> +		/*
> +		 * at_in[] is kept per subprog and the snapshot of the stack per call insn:
> +		 * the walk must not enter a subprog that it has not left.
> +		 */
> +		if (env->subprog_info[subprog].recursive && env->subprog_info[callee].recursive) {
> +			verbose(env, "recursive %s() passes a pointer to the stack to %s()\n",
> +				bpf_subprog_name(env, subprog), bpf_subprog_name(env, callee));
> +			err = -EINVAL;
> +			goto out_free;
> +		}
> +

Can this reject programs that loaded before this commit?

bpf_compute_live_registers() runs after mark_recursive_subprogs(), but the
recursive marks are only provisional at that point.  Since callx targets are
not known before the main pass, mark_recursive_subprogs() treats every
subprog that contains callx as calling every subprog whose address is taken:

    for_each_set_bit(s, callx, cnt)
        bitmap_or(calls + s * words, calls + s * words, taken, cnt);

The taken set includes plain bpf_loop() callbacks and every entry of a
read-only function pointer table.  check_recursion() only drops the wrong
marks after do_check_all(), which is too late for this check.

For example:

    static int bar(void *p);                         /* only entry of a RO func table */
    static int helper(int *p) { return tbl[0](p); }  /* callx, always calls bar */
    static int cb(u64 i, void *ctx) { int x = 0; return helper(&x); }
    prog: bpf_loop(1, cb, NULL, 0);

cb is in taken (pseudo_func) and helper contains callx, so the provisional
graph has the cycle cb -> helper -> cb and both get recursive = true.
analyze_subprog(cb) then sees a pseudo_call to helper with an FP-derived r1,
which is live at helper's entry because callx reads r1-r5.  The load fails
with "recursive cb() passes a pointer to the stack to helper()".

The parent commit accepts this program, because sort_subprogs_topo() knew no
callx edges before the main pass and the re-sort after it found no cycle.

The check does not seem to be needed for the reason its comment gives.
analyze_subprog() only follows bpf_pseudo_call edges and the constant
callbacks of find_callback_subprog(), never callx, so a cycle that depends on
a callx edge cannot make the walk re-enter a subprog and free
info[subprog].at_in while it is in use.

Would it work to test for a cycle in the graph that analyze_subprog()
actually walks (pseudo_call plus sync callback edges) instead of using the
provisional recursive marks?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

Re: [PATCH bpf-next v4 22/46] bpf: Allow recursive subprogs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] bpf: off-by-one frame count for recursion in
  compute_call_depth()
- [Medium] bpf: spurious budget exit on loop back-edge in
  check_recursion_guard()
--

commit 7c6074f7480e70a1121bc45054dfaf6385da6057
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Allow recursive subprogs

This commit allows recursive static subprograms by detecting cycles in the
call graph, bounding their static stack accounting, and verifying the
runtime recursion guard exit path.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 3c3d1c8ea4529..ef68796c04ddf 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -6143,11 +6175,17 @@ static void compute_call_depth(struct bpf_verifier_env *env, struct bpf_call_gra
>  		for_each_set_bit(s, cg->live, cg->cnt) {
>  			if (dinfo[s].depth < 0)
>  				continue;
> -			depth = dinfo[s].depth + subprog_stack_size(env, s);
> -			arena_depth = dinfo[s].arena_depth + si[s].arena_stack_depth;
>  			for_each_set_bit(j, subprog_calls(cg, s), cg->cnt) {
>  				callee = &dinfo[j];
> -				frame = bpf_subprog_is_global(env, j) ? 0 : dinfo[s].frame + 1;
> +				depth = dinfo[s].depth;
> +				arena_depth = dinfo[s].arena_depth;
> +				if (!si[s].recursive || !si[j].recursive) {
> +					depth += subprog_stack_size(env, s);
> +					arena_depth += si[s].arena_stack_depth;
> +				}
> +				frame = dinfo[s].frame + !si[j].recursive;
> +				if (bpf_subprog_is_global(env, j))
> +					frame = 0;

[Severity: Medium]
Should this increment frame by (!si[s].recursive || !si[j].recursive),
matching the depth and arena_depth calculation above?

When a non-recursive subprogram s calls a recursive subprogram j,
!si[j].recursive is 0, so entering the recursion does not increment
dinfo[j].frame and edges within the recursion also add 0.  A recursion
therefore counts as 0 frames instead of 1 in dinfo[].frame, allowing a
static call chain with recursive subprograms to bypass the MAX_CALL_FRAMES
and MAX_CALL_FRAMES_INTERP checks in check_call_depth().

[ ... ]

> @@ -22106,6 +22294,21 @@ static int flow_drain(struct bpf_verifier_env *env)
>  	return err;
>  }
>  
> +/*
> + * A subprog on a cycle of the call graph checks the stack at run time, see
> + * bpf_insert_recursion_checks(). When there is no room, or the time slice is over, it leaves
> + * at once with 0 in place of the result: it unwinds if it can, see bail_unwinds, otherwise
> + * it returns and callers have to be fine with that.
> + */
> +static int check_recursion_guard(struct bpf_verifier_env *env)
> +{
> +	struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
> +
> +	if (!sub->recursive || env->insn_idx != sub->start)
> +		return 0;
> +	return push_budget_exit(env);
> +}

[Severity: Medium]
Does check_recursion_guard() push a spurious exit when sub->start is also a
loop header reached via an intra-subprogram back-edge?

In bpf_insert_recursion_checks(), bpf_patch_list_insert() redirects only
calls to the runtime recursion check, while intra-subprogram jumps to
sub->start land after the check.  Here, check_recursion_guard() calls
push_budget_exit() every time env->insn_idx == sub->start without checking
whether the state just entered the subprogram or looped back from inside
it.  If a loop headed at sub->start holds a reference, lock, or RCU read
lock across the back-edge, that branch reaches sub->bail_insn with the
resource still held and causes the verifier to reject the program.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=22

[PATCH bpf-next v4 23/46] bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

is_state_visited() is the pruning of a verifier that follows each path of
a prog. It saves states at selected insns and stops a path when a saved
state covers the current state. The main pass is in flow.c now. Each insn
that starts a basic block has a set of states, and a state that arrives is
dropped, joined or added. Nothing calls is_state_visited().

Remove it, and all that exists only for it: 1550 lines.

  removed                     task in the old pass     what does it now
  --------------------------  -----------------------  --------------------
  explored_states, free_list  Hold the saved states.   The sets of states
                                                       in flow.c.
  branches, dfs_depth,        Tell when all paths      No paths. The pass
  equal_state                 from a state are         ends when no set has
                              explored.                a new state.
  bpf_compute_scc(), SCC      Bring precision marks    The marks are not
  visits, backedges           over the back edges of   used. loop_header
                              a loop.                  tells the loops.
  states_maybe_looping(),     Reject a path that       The verifier adds
  iter.depth                  comes back in the same   may_goto to a loop
                              state.                   that it cannot
                                                       bound.
  widen_imprecise_scalars(),  Make a loop with an      A loop that is not
  callback_unroll_depth,      iterator, a callback or  counted is walked,
  may_goto_depth              may_goto converge.       or widened at its
                                                       head.
  bpf_split_cur_state()       Save the state that      flow.c keeps the
                              enters a counted loop.   states that enter.

What user space sees:
 - BPF_F_TEST_STATE_FREQ told is_state_visited() to save a state each time.
   The flag is UAPI and selftests set it. The kernel accepts it as before,
   and it has no effect.
 - The "Program dump" of log level 2 has no scc column.
 - The log does not print "depth=" for an iterator.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 123 +----
 kernel/bpf/cfg.c             | 198 -------
 kernel/bpf/log.c             |   5 +-
 kernel/bpf/states.c          | 998 -----------------------------------
 kernel/bpf/verifier.c        | 240 +--------
 5 files changed, 8 insertions(+), 1556 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 993d8241f6f3..902a339b0caf 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -119,7 +119,6 @@ struct bpf_reg_state {
 			u32 btf_id;
 			/* packing following two fields to fit iter state into 16 bytes */
 			enum bpf_iter_state state:2;
-			int depth:30;
 			/*
 			 * The number of loops, in all frames, that the program was in
 			 * when it made the iterator, see iter_stops_loops().
@@ -568,52 +567,6 @@ struct bpf_verifier_state {
 	struct bpf_verifier_state *parent;
 	/* Acquired reference states */
 	struct bpf_reference_state *refs;
-	/*
-	 * 'branches' field is the number of branches left to explore:
-	 * 0 - all possible paths from this state reached bpf_exit or
-	 * were safely pruned
-	 * 1 - at least one path is being explored.
-	 * This state hasn't reached bpf_exit
-	 * 2 - at least two paths are being explored.
-	 * This state is an immediate parent of two children.
-	 * One is fallthrough branch with branches==1 and another
-	 * state is pushed into stack (to be explored later) also with
-	 * branches==1. The parent of this state has branches==1.
-	 * The verifier state tree connected via 'parent' pointer looks like:
-	 * 1
-	 * 1
-	 * 2 -> 1 (first 'if' pushed into stack)
-	 * 1
-	 * 2 -> 1 (second 'if' pushed into stack)
-	 * 1
-	 * 1
-	 * 1 bpf_exit.
-	 *
-	 * Once do_check() reaches bpf_exit, it calls update_branch_counts()
-	 * and the verifier state tree will look:
-	 * 1
-	 * 1
-	 * 2 -> 1 (first 'if' pushed into stack)
-	 * 1
-	 * 1 -> 1 (second 'if' pushed into stack)
-	 * 0
-	 * 0
-	 * 0 bpf_exit.
-	 * After pop_stack() the do_check() will resume at second 'if'.
-	 *
-	 * If is_state_visited() sees a state with branches > 0 it means
-	 * there is a loop. If such state is exactly equal to the current state
-	 * it's an infinite loop. Note states_equal() checks for states
-	 * equivalency, so two states being 'states_equal' does not mean
-	 * infinite loop. The exact comparison is provided by
-	 * states_maybe_looping() function. It's a stronger pre-check and
-	 * much faster than states_equal().
-	 *
-	 * This algorithm may not find all possible infinite loops or
-	 * loop iteration count may be too high.
-	 * In such cases BPF_COMPLEXITY_LIMIT_INSNS limit kicks in.
-	 */
-	u32 branches;
 	u32 insn_idx;
 	u32 curframe;
 
@@ -648,10 +601,6 @@ struct bpf_verifier_state {
 	 */
 	u32 first_insn_idx;
 	u32 last_insn_idx;
-	/* if this state is a backedge state then equal_state
-	 * records cached state to which this state is equal.
-	 */
-	struct bpf_verifier_state *equal_state;
 	/* jmp history recorded from first to last.
 	 * backtracking is using it to go from last to first.
 	 * For most states jmp_history_cnt is [0-3].
@@ -659,9 +608,6 @@ struct bpf_verifier_state {
 	 */
 	struct bpf_jmp_history_entry *jmp_history;
 	u32 jmp_history_cnt;
-	u32 dfs_depth;
-	u32 callback_unroll_depth;
-	u32 may_goto_depth;
 };
 
 /* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
@@ -761,15 +707,6 @@ static inline bool bpf_stack_args_are_incoming(const struct bpf_func_state *fram
 						1 << STACK_SPILL, __expr);	\
 	})
 
-/* linked list of verifier states used to prune search */
-struct bpf_verifier_state_list {
-	struct bpf_verifier_state state;
-	struct list_head node;
-	u32 miss_cnt;
-	u32 hit_cnt:31;
-	u32 in_free_list:1;
-};
-
 struct bpf_loop_inline_state {
 	unsigned int initialized:1; /* set to true upon first entry */
 	unsigned int fit_for_inline:1; /* true if callback function is the same
@@ -984,11 +921,6 @@ struct bpf_insn_aux_data {
 	u64 jump_target:1;
 
 	unsigned int orig_idx; /* original instruction index, initialized once */
-	/*
-	 * CFG strongly connected component this instruction belongs to,
-	 * zero if it is a singleton SCC.
-	 */
-	u32 scc;
 	/* registers alive before this instruction. */
 	u16 live_regs_before;
 	/*
@@ -1235,38 +1167,6 @@ struct bpf_idset {
 	} entries[BPF_ID_MAP_SIZE];
 };
 
-/* see verifier.c:compute_scc_callchain() */
-struct bpf_scc_callchain {
-	/* call sites from bpf_verifier_state->frame[*]->callsite leading to this SCC */
-	u32 callsites[MAX_CALL_FRAMES - 1];
-	/* last frame in a chain is identified by SCC id */
-	u32 scc;
-};
-
-/* verifier state waiting for propagate_backedges() */
-struct bpf_scc_backedge {
-	struct bpf_scc_backedge *next;
-	struct bpf_verifier_state state;
-};
-
-struct bpf_scc_visit {
-	struct bpf_scc_callchain callchain;
-	/* first state in current verification path that entered SCC
-	 * identified by the callchain
-	 */
-	struct bpf_verifier_state *entry_state;
-	struct bpf_scc_backedge *backedges; /* list of backedges */
-	u32 num_backedges;
-};
-
-/* An array of bpf_scc_visit structs sharing tht same bpf_scc_callchain->scc
- * but having different bpf_scc_callchain->callsites.
- */
-struct bpf_scc_info {
-	u32 num_visits;
-	struct bpf_scc_visit visits[];
-};
-
 struct bpf_liveness;
 struct scev;
 
@@ -1351,7 +1251,6 @@ struct bpf_verifier_env {
 	struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
 	int stack_size;			/* number of states to be processed */
 	bool strict_alignment;		/* perform strict pointer alignment checks */
-	bool test_state_freq;		/* test verifier with different pruning frequency */
 	bool test_reg_invariants;	/* fail verification on register invariants violations */
 	bool uses_arena_sp;		/* the program has insns with BPF_REG_ARENA_SP */
 	bool restart;			/* see bpf_flow_next_is_no_bound() */
@@ -1361,11 +1260,6 @@ struct bpf_verifier_env {
 	u32 *walk_insns;		/* the insns that were looked at in each of them */
 	u32 max_arena_stack_depth;	/* of all call chains */
 	struct bpf_verifier_state *cur_state; /* current verifier state */
-	/* Search pruning optimization, array of list_heads for
-	 * lists of struct bpf_verifier_state_list.
-	 */
-	struct list_head *explored_states;
-	struct list_head free_list;	/* list of struct bpf_verifier_state_list */
 	struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
 	struct btf_mod_pair used_btfs[MAX_USED_BTFS]; /* array of BTF's used by BPF program */
 	struct bpf_map *insn_array_maps[MAX_USED_MAPS]; /* array of INSN_ARRAY map's to be relocated */
@@ -1442,9 +1336,7 @@ struct bpf_verifier_env {
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
 	/* number of instructions analyzed by the verifier */
-	u32 prev_insn_processed, insn_processed;
-	/* number of jmps, calls, exits analyzed so far */
-	u32 prev_jmps_processed, jmps_processed;
+	u32 insn_processed;
 	/* maximum combined stack depth */
 	u32 max_stack_depth;
 	/* stack budget of the program, see bpf_prog_stack_limit() */
@@ -1462,9 +1354,6 @@ struct bpf_verifier_env {
 	u32 peak_states;
 	/* longest register parentage chain walked for liveness marking */
 	u32 longest_mark_read_walk;
-	u32 free_list_size;
-	u32 explored_states_size;
-	u32 num_backedges;
 	/*
 	 * The program's fd_array comes in two shapes, told apart by whether
 	 * the caller passed fd_array_cnt. They are mutually exclusive:
@@ -1497,11 +1386,7 @@ struct bpf_verifier_env {
 	char tmp_arg_name[32];
 	struct bpf_insn insn_buf[INSN_BUF_SIZE];
 	struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
-	struct bpf_scc_callchain callchain_buf;
 	struct bpf_liveness *liveness;
-	/* array of pointers to bpf_scc_info indexed by SCC id */
-	struct bpf_scc_info **scc_info;
-	u32 scc_cnt;
 	struct bpf_iarray *succ;
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
@@ -1699,7 +1584,6 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
 int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
 			struct bpf_verifier_state *cur);
@@ -1748,15 +1632,11 @@ bool bpf_edge_leaves_loop(struct bpf_verifier_env *env, int h, int n, int t);
 int bpf_find_latches(struct bpf_verifier_env *env, bool flags);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
-int bpf_split_cur_state(struct bpf_verifier_env *env);
-int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 
 void bpf_clear_jmp_history(struct bpf_verifier_state *state);
 int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 			    const struct bpf_verifier_state *src);
-struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
 void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
-void bpf_free_backedges(struct bpf_scc_visit *visit);
 int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
 			 int insn_flags, int spi, int frame, const u16 *linked_regs,
 			 u8 linked_regs_cnt);
@@ -2114,7 +1994,6 @@ int bpf_prune_dead_branches(struct bpf_verifier_env *env);
 int bpf_check_cfg(struct bpf_verifier_env *env);
 void bpf_free_subprog_jts(struct bpf_verifier_env *env);
 int bpf_compute_postorder(struct bpf_verifier_env *env);
-int bpf_compute_scc(struct bpf_verifier_env *env);
 
 struct bpf_map_desc {
 	struct bpf_map *ptr;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 04c1c98ba248..94140d1fe1c3 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -868,201 +868,3 @@ int bpf_compute_postorder(struct bpf_verifier_env *env)
 	kvfree(state);
 	return 0;
 }
-
-/*
- * Compute strongly connected components (SCCs) on the CFG.
- * Assign an SCC number to each instruction, recorded in env->insn_aux[*].scc.
- * If instruction is a sole member of its SCC and there are no self edges,
- * assign it SCC number of zero.
- * Uses a non-recursive adaptation of Tarjan's algorithm for SCC computation.
- */
-int bpf_compute_scc(struct bpf_verifier_env *env)
-{
-	const u32 NOT_ON_STACK = U32_MAX;
-
-	struct bpf_insn_aux_data *aux = env->insn_aux_data;
-	const u32 insn_cnt = env->prog->len;
-	int stack_sz, dfs_sz, err = 0;
-	u32 *stack, *pre, *low, *dfs, *dfs_pos;
-	u32 i, j, t, w;
-	u32 next_preorder_num;
-	u32 next_scc_id;
-	bool assign_scc;
-	struct bpf_iarray *succ;
-
-	next_preorder_num = 1;
-	next_scc_id = 1;
-	/*
-	 * - 'stack' accumulates vertices in DFS order, see invariant comment below;
-	 * - 'pre[t] == p' => preorder number of vertex 't' is 'p';
-	 * - 'low[t] == n' => smallest preorder number of the vertex reachable from 't' is 'n';
-	 * - 'dfs' DFS traversal stack, used to emulate explicit recursion;
-	 * - 'dfs_pos[k] == j' => the frame 'dfs[k]' resumes visiting its
-	 *   successors at index 'j'.
-	 */
-	stack = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
-	pre = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
-	low = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
-	dfs = kvcalloc(insn_cnt, sizeof(*dfs), GFP_KERNEL_ACCOUNT);
-	dfs_pos = kvcalloc(insn_cnt, sizeof(*dfs_pos), GFP_KERNEL_ACCOUNT);
-	if (!stack || !pre || !low || !dfs || !dfs_pos) {
-		err = -ENOMEM;
-		goto exit;
-	}
-	/*
-	 * References:
-	 * [1] R. Tarjan "Depth-First Search and Linear Graph Algorithms"
-	 * [2] D. J. Pearce "A Space-Efficient Algorithm for Finding Strongly Connected Components"
-	 *
-	 * The algorithm maintains the following invariant:
-	 * - suppose there is a path 'u' ~> 'v', such that 'pre[v] < pre[u]';
-	 * - then, vertex 'u' remains on stack while vertex 'v' is on stack.
-	 *
-	 * Consequently:
-	 * - If 'low[v] < pre[v]', there is a path from 'v' to some vertex 'u',
-	 *   such that 'pre[u] == low[v]'; vertex 'u' is currently on the stack,
-	 *   and thus there is an SCC (loop) containing both 'u' and 'v'.
-	 * - If 'low[v] == pre[v]', loops containing 'v' have been explored,
-	 *   and 'v' can be considered the root of some SCC.
-	 *
-	 * Here is a pseudo-code for an explicitly recursive version of the algorithm:
-	 *
-	 *    NOT_ON_STACK = insn_cnt + 1
-	 *    pre = [0] * insn_cnt
-	 *    low = [0] * insn_cnt
-	 *    scc = [0] * insn_cnt
-	 *    stack = []
-	 *
-	 *    next_preorder_num = 1
-	 *    next_scc_id = 1
-	 *
-	 *    def recur(w):
-	 *        nonlocal next_preorder_num
-	 *        nonlocal next_scc_id
-	 *
-	 *        pre[w] = next_preorder_num
-	 *        low[w] = next_preorder_num
-	 *        next_preorder_num += 1
-	 *        stack.append(w)
-	 *        for s in successors(w):
-	 *            # Note: for classic algorithm the block below should look as:
-	 *            #
-	 *            # if pre[s] == 0:
-	 *            #     recur(s)
-	 *            #     low[w] = min(low[w], low[s])
-	 *            # elif low[s] != NOT_ON_STACK:
-	 *            #     low[w] = min(low[w], pre[s])
-	 *            #
-	 *            # But replacing both 'min' instructions with 'low[w] = min(low[w], low[s])'
-	 *            # does not break the invariant and makes iterative version of the algorithm
-	 *            # simpler. See 'Algorithm #3' from [2].
-	 *
-	 *            # 's' not yet visited
-	 *            if pre[s] == 0:
-	 *                recur(s)
-	 *            # if 's' is on stack, pick lowest reachable preorder number from it;
-	 *            # if 's' is not on stack 'low[s] == NOT_ON_STACK > low[w]',
-	 *            # so 'min' would be a noop.
-	 *            low[w] = min(low[w], low[s])
-	 *
-	 *        if low[w] == pre[w]:
-	 *            # 'w' is the root of an SCC, pop all vertices
-	 *            # below 'w' on stack and assign same SCC to them.
-	 *            while True:
-	 *                t = stack.pop()
-	 *                low[t] = NOT_ON_STACK
-	 *                scc[t] = next_scc_id
-	 *                if t == w:
-	 *                    break
-	 *            next_scc_id += 1
-	 *
-	 *    for i in range(0, insn_cnt):
-	 *        if pre[i] == 0:
-	 *            recur(i)
-	 *
-	 * Below implementation replaces explicit recursion with array 'dfs'.
-	 */
-	for (i = 0; i < insn_cnt; i++) {
-		if (pre[i])
-			continue;
-		stack_sz = 0;
-		dfs_sz = 1;
-		dfs[0] = i;
-		dfs_pos[0] = 0;
-dfs_continue:
-		while (dfs_sz) {
-			w = dfs[dfs_sz - 1];
-			if (pre[w] == 0) {
-				low[w] = next_preorder_num;
-				pre[w] = next_preorder_num;
-				next_preorder_num++;
-				stack[stack_sz++] = w;
-			}
-			/* Visit remaining 'w' successors */
-			succ = bpf_insn_successors(env, w);
-			for (j = dfs_pos[dfs_sz - 1]; j < succ->cnt; ++j) {
-				if (pre[succ->items[j]]) {
-					low[w] = min(low[w], low[succ->items[j]]);
-				} else {
-					/*
-					 * Once DFS for succ->items[j] is complete,
-					 * pre[succ->items[j]] is non-zero, hence
-					 * resuming at 'j' allows to follow the
-					 * low[w] = min(...) update branch above.
-					 */
-					dfs_pos[dfs_sz - 1] = j;
-					dfs_pos[dfs_sz] = 0;
-					dfs[dfs_sz] = succ->items[j];
-					dfs_sz++;
-					goto dfs_continue;
-				}
-			}
-			/*
-			 * Preserve the invariant: if some vertex above in the stack
-			 * is reachable from 'w', keep 'w' on the stack.
-			 */
-			if (low[w] < pre[w]) {
-				dfs_sz--;
-				goto dfs_continue;
-			}
-			/*
-			 * Assign SCC number only if component has two or more elements,
-			 * or if component has a self reference, or if instruction is a
-			 * callback calling function (implicit loop).
-			 */
-			assign_scc = stack[stack_sz - 1] != w;	/* two or more elements? */
-			for (j = 0; j < succ->cnt; ++j) {	/* self reference? */
-				if (succ->items[j] == w) {
-					assign_scc = true;
-					break;
-				}
-			}
-			if (bpf_calls_callback(env, w)) /* implicit loop? */
-				assign_scc = true;
-			/* Pop component elements from stack */
-			do {
-				t = stack[--stack_sz];
-				low[t] = NOT_ON_STACK;
-				if (assign_scc)
-					aux[t].scc = next_scc_id;
-			} while (t != w);
-			if (assign_scc)
-				next_scc_id++;
-			dfs_sz--;
-		}
-	}
-	env->scc_info = kvzalloc_objs(*env->scc_info, next_scc_id,
-				      GFP_KERNEL_ACCOUNT);
-	if (!env->scc_info) {
-		err = -ENOMEM;
-		goto exit;
-	}
-	env->scc_cnt = next_scc_id;
-exit:
-	kvfree(stack);
-	kvfree(pre);
-	kvfree(low);
-	kvfree(dfs);
-	kvfree(dfs_pos);
-	return err;
-}
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index e1c9b56cd13f..00cb219e24cd 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -779,11 +779,10 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 			if (!reg->id)
 				continue;
 
-			verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)",
+			verbose(env, " fp%d=iter_%s(id=%d,state=%s)",
 				(-i - 1) * BPF_REG_SIZE,
 				iter_type_str(reg->iter.btf, reg->iter.btf_id),
-				reg->id, iter_state_str(reg->iter.state),
-				reg->iter.depth);
+				reg->id, iter_state_str(reg->iter.state));
 			break;
 		case STACK_MISC:
 		case STACK_ZERO:
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 4b2506f3e3e2..343f29b52d79 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -7,297 +7,6 @@
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
-#define BPF_COMPLEXITY_LIMIT_STATES	64
-
-static bool is_may_goto_insn_at(struct bpf_verifier_env *env, int insn_idx)
-{
-	return bpf_is_may_goto_insn(&env->prog->insnsi[insn_idx]);
-}
-
-static bool is_iter_next_insn(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].is_iter_next;
-}
-
-static void update_peak_states(struct bpf_verifier_env *env)
-{
-	u32 cur_states;
-
-	cur_states = env->explored_states_size + env->free_list_size + env->num_backedges;
-	env->peak_states = max(env->peak_states, cur_states);
-}
-
-/* struct bpf_verifier_state->parent refers to states
- * that are in either of env->{expored_states,free_list}.
- * In both cases the state is contained in struct bpf_verifier_state_list.
- */
-static struct bpf_verifier_state_list *state_parent_as_list(struct bpf_verifier_state *st)
-{
-	if (st->parent)
-		return container_of(st->parent, struct bpf_verifier_state_list, state);
-	return NULL;
-}
-
-static bool incomplete_read_marks(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st);
-
-/* A state can be freed if it is no longer referenced:
- * - is in the env->free_list;
- * - has no children states;
- */
-static void maybe_free_verifier_state(struct bpf_verifier_env *env,
-				      struct bpf_verifier_state_list *sl)
-{
-	if (!sl->in_free_list
-	    || sl->state.branches != 0
-	    || incomplete_read_marks(env, &sl->state))
-		return;
-	list_del(&sl->node);
-	bpf_free_verifier_state(&sl->state, false);
-	kfree(sl);
-	env->free_list_size--;
-}
-
-/* For state @st look for a topmost frame with frame_insn_idx() in some SCC,
- * if such frame exists form a corresponding @callchain as an array of
- * call sites leading to this frame and SCC id.
- * E.g.:
- *
- *    void foo()  { A: loop {... SCC#1 ...}; }
- *    void bar()  { B: loop { C: foo(); ... SCC#2 ... }
- *                  D: loop { E: foo(); ... SCC#3 ... } }
- *    void main() { F: bar(); }
- *
- * @callchain at (A) would be either (F,SCC#2) or (F,SCC#3) depending
- * on @st frame call sites being (F,C,A) or (F,E,A).
- */
-static bool compute_scc_callchain(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st,
-				  struct bpf_scc_callchain *callchain)
-{
-	u32 i, scc, insn_idx;
-
-	memset(callchain, 0, sizeof(*callchain));
-	for (i = 0; i <= st->curframe; i++) {
-		insn_idx = bpf_frame_insn_idx(st, i);
-		scc = env->insn_aux_data[insn_idx].scc;
-		if (scc) {
-			callchain->scc = scc;
-			break;
-		} else if (i < st->curframe) {
-			callchain->callsites[i] = insn_idx;
-		} else {
-			return false;
-		}
-	}
-	return true;
-}
-
-/* Check if bpf_scc_visit instance for @callchain exists. */
-static struct bpf_scc_visit *scc_visit_lookup(struct bpf_verifier_env *env,
-					      struct bpf_scc_callchain *callchain)
-{
-	struct bpf_scc_info *info = env->scc_info[callchain->scc];
-	struct bpf_scc_visit *visits = info->visits;
-	u32 i;
-
-	if (!info)
-		return NULL;
-	for (i = 0; i < info->num_visits; i++)
-		if (memcmp(callchain, &visits[i].callchain, sizeof(*callchain)) == 0)
-			return &visits[i];
-	return NULL;
-}
-
-/* Allocate a new bpf_scc_visit instance corresponding to @callchain.
- * Allocated instances are alive for a duration of the do_check_common()
- * call and are freed by free_states().
- */
-static struct bpf_scc_visit *scc_visit_alloc(struct bpf_verifier_env *env,
-					     struct bpf_scc_callchain *callchain)
-{
-	struct bpf_scc_visit *visit;
-	struct bpf_scc_info *info;
-	u32 scc, num_visits;
-	u64 new_sz;
-
-	scc = callchain->scc;
-	info = env->scc_info[scc];
-	num_visits = info ? info->num_visits : 0;
-	new_sz = sizeof(*info) + sizeof(struct bpf_scc_visit) * (num_visits + 1);
-	info = kvrealloc(env->scc_info[scc], new_sz, GFP_KERNEL_ACCOUNT);
-	if (!info)
-		return NULL;
-	env->scc_info[scc] = info;
-	info->num_visits = num_visits + 1;
-	visit = &info->visits[num_visits];
-	memset(visit, 0, sizeof(*visit));
-	memcpy(&visit->callchain, callchain, sizeof(*callchain));
-	return visit;
-}
-
-/* Form a string '(callsite#1,callsite#2,...,scc)' in env->tmp_str_buf */
-static char *format_callchain(struct bpf_verifier_env *env, struct bpf_scc_callchain *callchain)
-{
-	char *buf = env->tmp_str_buf;
-	int i, delta = 0;
-
-	delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "(");
-	for (i = 0; i < ARRAY_SIZE(callchain->callsites); i++) {
-		if (!callchain->callsites[i])
-			break;
-		delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u,",
-				  callchain->callsites[i]);
-	}
-	delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u)", callchain->scc);
-	return env->tmp_str_buf;
-}
-
-/* If callchain for @st exists (@st is in some SCC), ensure that
- * bpf_scc_visit instance for this callchain exists.
- * If instance does not exist or is empty, assign visit->entry_state to @st.
- */
-static int maybe_enter_scc(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain))
-		return 0;
-	visit = scc_visit_lookup(env, callchain);
-	visit = visit ?: scc_visit_alloc(env, callchain);
-	if (!visit)
-		return -ENOMEM;
-	if (!visit->entry_state) {
-		visit->entry_state = st;
-		if (env->log.level & BPF_LOG_LEVEL2)
-			verbose(env, "SCC enter %s\n", format_callchain(env, callchain));
-	}
-	return 0;
-}
-
-static int propagate_backedges(struct bpf_verifier_env *env, struct bpf_scc_visit *visit);
-
-/* If callchain for @st exists (@st is in some SCC), make it empty:
- * - set visit->entry_state to NULL;
- * - flush accumulated backedges.
- */
-static int maybe_exit_scc(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain))
-		return 0;
-	visit = scc_visit_lookup(env, callchain);
-	if (!visit) {
-		/*
-		 * If path traversal stops inside an SCC, corresponding bpf_scc_visit
-		 * must exist for non-speculative paths. For non-speculative paths
-		 * traversal stops when:
-		 * a. Verification error is found, maybe_exit_scc() is not called.
-		 * b. Top level BPF_EXIT is reached. Top level BPF_EXIT is not a member
-		 *    of any SCC.
-		 * c. A checkpoint is reached and matched. Checkpoints are created by
-		 *    is_state_visited(), which calls maybe_enter_scc(), which allocates
-		 *    bpf_scc_visit instances for checkpoints within SCCs.
-		 * (c) is the only case that can reach this point.
-		 */
-		if (!st->speculative) {
-			verifier_bug(env, "scc exit: no visit info for call chain %s",
-				     format_callchain(env, callchain));
-			return -EFAULT;
-		}
-		return 0;
-	}
-	if (visit->entry_state != st)
-		return 0;
-	if (env->log.level & BPF_LOG_LEVEL2)
-		verbose(env, "SCC exit %s\n", format_callchain(env, callchain));
-	visit->entry_state = NULL;
-	env->num_backedges -= visit->num_backedges;
-	visit->num_backedges = 0;
-	update_peak_states(env);
-	return propagate_backedges(env, visit);
-}
-
-/* Lookup an bpf_scc_visit instance corresponding to @st callchain
- * and add @backedge to visit->backedges. @st callchain must exist.
- */
-static int add_scc_backedge(struct bpf_verifier_env *env,
-			    struct bpf_verifier_state *st,
-			    struct bpf_scc_backedge *backedge)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain)) {
-		verifier_bug(env, "add backedge: no SCC in verification path, insn_idx %d",
-			     st->insn_idx);
-		return -EFAULT;
-	}
-	visit = scc_visit_lookup(env, callchain);
-	if (!visit) {
-		verifier_bug(env, "add backedge: no visit info for call chain %s",
-			     format_callchain(env, callchain));
-		return -EFAULT;
-	}
-	if (env->log.level & BPF_LOG_LEVEL2)
-		verbose(env, "SCC backedge %s\n", format_callchain(env, callchain));
-	backedge->next = visit->backedges;
-	visit->backedges = backedge;
-	visit->num_backedges++;
-	env->num_backedges++;
-	update_peak_states(env);
-	return 0;
-}
-
-/* bpf_reg_state->live marks for registers in a state @st are incomplete,
- * if state @st is in some SCC and not all execution paths starting at this
- * SCC are fully explored.
- */
-static bool incomplete_read_marks(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st)
-{
-	struct bpf_scc_callchain *callchain = &env->callchain_buf;
-	struct bpf_scc_visit *visit;
-
-	if (!compute_scc_callchain(env, st, callchain))
-		return false;
-	visit = scc_visit_lookup(env, callchain);
-	if (!visit)
-		return false;
-	return !!visit->backedges;
-}
-
-int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_verifier_state_list *sl = NULL, *parent_sl;
-	struct bpf_verifier_state *parent;
-	int err;
-
-	while (st) {
-		u32 br = --st->branches;
-
-		/* verifier_bug_if(br > 1, ...) technically makes sense here,
-		 * but see comment in push_stack(), hence:
-		 */
-		verifier_bug_if((int)br < 0, env, "%s:branches_to_explore=%d", __func__, br);
-		if (br)
-			break;
-		err = maybe_exit_scc(env, st);
-		if (err)
-			return err;
-		parent = st->parent;
-		parent_sl = state_parent_as_list(st);
-		if (sl)
-			maybe_free_verifier_state(env, sl);
-		st = parent;
-		sl = parent_sl;
-	}
-	return 0;
-}
-
 /* check %cur's range satisfies %old's */
 static bool range_within(const struct bpf_reg_state *old,
 			 const struct bpf_reg_state *cur)
@@ -830,18 +539,11 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		case STACK_ITER:
 			old_reg = &old_slot->spilled_ptr;
 			cur_reg = &cur_slot->spilled_ptr;
-			/* iter.depth is not compared between states as it
-			 * doesn't matter for correctness and would otherwise
-			 * prevent convergence; we maintain it only to prevent
-			 * infinite loop check triggering, see
-			 * iter_active_depths_differ()
-			 */
 			if (old_reg->type != cur_reg->type ||
 			    old_reg->iter.btf != cur_reg->iter.btf ||
 			    old_reg->iter.btf_id != cur_reg->iter.btf_id ||
 			    old_reg->iter.state != cur_reg->iter.state ||
 			    old_reg->iter.loops != cur_reg->iter.loops ||
-			    /* ignore {old_reg,cur_reg}->iter.depth, see above */
 			    !check_ids(old_reg->id, cur_reg->id, idmap))
 				return false;
 			break;
@@ -1110,706 +812,6 @@ static bool states_equal(struct bpf_verifier_env *env,
 	return true;
 }
 
-/* find precise scalars in the previous equivalent state and
- * propagate them into the current state
- */
-static int propagate_precision(struct bpf_verifier_env *env,
-			       const struct bpf_verifier_state *old,
-			       struct bpf_verifier_state *cur,
-			       bool *changed)
-{
-	struct bpf_reg_state *state_reg;
-	struct bpf_func_state *state;
-	int i, err = 0, fr;
-	bool first;
-
-	for (fr = old->curframe; fr >= 0; fr--) {
-		state = old->frame[fr];
-		state_reg = state->regs;
-		first = true;
-		for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
-			if (state_reg->type != SCALAR_VALUE ||
-			    !state_reg->precise)
-				continue;
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				if (first)
-					verbose(env, "frame %d: propagating r%d", fr, i);
-				else
-					verbose(env, ",r%d", i);
-			}
-			bpf_bt_set_frame_reg(&env->bt, fr, i);
-			first = false;
-		}
-
-		for (i = 0; i < bpf_stack_nr_slots(state); i++) {
-			struct bpf_stack_state *ss = bpf_stack_slot(state, i);
-
-			if (!bpf_is_spilled_reg(ss))
-				continue;
-			state_reg = &ss->spilled_ptr;
-			if (state_reg->type != SCALAR_VALUE ||
-			    !state_reg->precise)
-				continue;
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				if (first)
-					verbose(env, "frame %d: propagating fp%d",
-						fr, (-i - 1) * BPF_REG_SIZE);
-				else
-					verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
-			}
-			bpf_bt_set_frame_slot(&env->bt, fr, i);
-			first = false;
-		}
-		for (i = 0; i < state->out_stack_arg_cnt; i++) {
-			state_reg = &state->stack_arg_regs[i];
-			if (state_reg->type != SCALAR_VALUE || !state_reg->precise)
-				continue;
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				if (first)
-					verbose(env, "frame %d: propagating r11%d",
-						fr, -(i + 1) * BPF_REG_SIZE);
-				else
-					verbose(env, ",r11%d", -(i + 1) * BPF_REG_SIZE);
-			}
-			bt_set_frame_stack_arg_slot(&env->bt, fr, i);
-			first = false;
-		}
-		if (!first && (env->log.level & BPF_LOG_LEVEL2))
-			verbose(env, "\n");
-	}
-
-	err = bpf_mark_chain_precision(env, cur, -1, changed);
-	if (err < 0)
-		return err;
-
-	return 0;
-}
-
-#define MAX_BACKEDGE_ITERS 64
-
-/* Propagate read and precision marks from visit->backedges[*].state->equal_state
- * to corresponding parent states of visit->backedges[*].state until fixed point is reached,
- * then free visit->backedges.
- * After execution of this function incomplete_read_marks() will return false
- * for all states corresponding to @visit->callchain.
- */
-static int propagate_backedges(struct bpf_verifier_env *env, struct bpf_scc_visit *visit)
-{
-	struct bpf_scc_backedge *backedge;
-	struct bpf_verifier_state *st;
-	bool changed;
-	int i, err;
-
-	i = 0;
-	do {
-		if (i++ > MAX_BACKEDGE_ITERS) {
-			if (env->log.level & BPF_LOG_LEVEL2)
-				verbose(env, "%s: too many iterations\n", __func__);
-			for (backedge = visit->backedges; backedge; backedge = backedge->next)
-				bpf_mark_all_scalars_precise(env, &backedge->state);
-			break;
-		}
-		changed = false;
-		for (backedge = visit->backedges; backedge; backedge = backedge->next) {
-			st = &backedge->state;
-			err = propagate_precision(env, st->equal_state, st, &changed);
-			if (err)
-				return err;
-		}
-	} while (changed);
-
-	bpf_free_backedges(visit);
-	return 0;
-}
-
-static bool states_maybe_looping(struct bpf_verifier_state *old,
-				 struct bpf_verifier_state *cur)
-{
-	struct bpf_func_state *fold, *fcur;
-	int i, fr = cur->curframe;
-
-	if (old->curframe != fr)
-		return false;
-
-	fold = old->frame[fr];
-	fcur = cur->frame[fr];
-	for (i = 0; i < MAX_BPF_REG; i++)
-		if (memcmp(&fold->regs[i], &fcur->regs[i],
-			   offsetofend(struct bpf_reg_state, step)))
-			return false;
-	return true;
-}
-
-/* is_state_visited() handles iter_next() (see process_iter_next_call() for
- * terminology) calls specially: as opposed to bounded BPF loops, it *expects*
- * states to match, which otherwise would look like an infinite loop. So while
- * iter_next() calls are taken care of, we still need to be careful and
- * prevent erroneous and too eager declaration of "infinite loop", when
- * iterators are involved.
- *
- * Here's a situation in pseudo-BPF assembly form:
- *
- *   0: again:                          ; set up iter_next() call args
- *   1:   r1 = &it                      ; <CHECKPOINT HERE>
- *   2:   call bpf_iter_num_next        ; this is iter_next() call
- *   3:   if r0 == 0 goto done
- *   4:   ... something useful here ...
- *   5:   goto again                    ; another iteration
- *   6: done:
- *   7:   r1 = &it
- *   8:   call bpf_iter_num_destroy     ; clean up iter state
- *   9:   exit
- *
- * This is a typical loop. Let's assume that we have a prune point at 1:,
- * before we get to `call bpf_iter_num_next` (e.g., because of that `goto
- * again`, assuming other heuristics don't get in a way).
- *
- * When we first time come to 1:, let's say we have some state X. We proceed
- * to 2:, fork states, enqueue ACTIVE, validate NULL case successfully, exit.
- * Now we come back to validate that forked ACTIVE state. We proceed through
- * 3-5, come to goto, jump to 1:. Let's assume our state didn't change, so we
- * are converging. But the problem is that we don't know that yet, as this
- * convergence has to happen at iter_next() call site only. So if nothing is
- * done, at 1: verifier will use bounded loop logic and declare infinite
- * looping (and would be *technically* correct, if not for iterator's
- * "eventual sticky NULL" contract, see process_iter_next_call()). But we
- * don't want that. So what we do in process_iter_next_call() when we go on
- * another ACTIVE iteration, we bump slot->iter.depth, to mark that it's
- * a different iteration. So when we suspect an infinite loop, we additionally
- * check if any of the *ACTIVE* iterator states depths differ. If yes, we
- * pretend we are not looping and wait for next iter_next() call.
- *
- * This only applies to ACTIVE state. In DRAINED state we don't expect to
- * loop, because that would actually mean infinite loop, as DRAINED state is
- * "sticky", and so we'll keep returning into the same instruction with the
- * same state (at least in one of possible code paths).
- *
- * This approach allows to keep infinite loop heuristic even in the face of
- * active iterator. E.g., C snippet below is and will be detected as
- * infinitely looping:
- *
- *   struct bpf_iter_num it;
- *   int *p, x;
- *
- *   bpf_iter_num_new(&it, 0, 10);
- *   while ((p = bpf_iter_num_next(&t))) {
- *       x = p;
- *       while (x--) {} // <<-- infinite loop here
- *   }
- *
- */
-static bool iter_active_depths_differ(struct bpf_verifier_state *old, struct bpf_verifier_state *cur)
-{
-	struct bpf_reg_state *slot, *cur_slot;
-	struct bpf_func_state *state;
-	int i, fr;
-
-	for (fr = old->curframe; fr >= 0; fr--) {
-		state = old->frame[fr];
-		for (i = 0; i < bpf_stack_nr_slots(state); i++) {
-			struct bpf_stack_state *ss = bpf_stack_slot(state, i);
-
-			if (ss->slot_type[0] != STACK_ITER)
-				continue;
-
-			slot = &ss->spilled_ptr;
-			if (slot->iter.state != BPF_ITER_STATE_ACTIVE)
-				continue;
-
-			cur_slot = &bpf_stack_slot(cur->frame[fr], i)->spilled_ptr;
-			if (cur_slot->iter.depth != slot->iter.depth)
-				return true;
-		}
-	}
-	return false;
-}
-
-static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *func;
-	struct bpf_reg_state *reg;
-	int i, j;
-
-	for (i = 0; i <= st->curframe; i++) {
-		func = st->frame[i];
-		for (j = 0; j < BPF_REG_FP; j++) {
-			reg = &func->regs[j];
-			if (reg->type != SCALAR_VALUE)
-				continue;
-			reg->precise = false;
-		}
-		for (j = 0; j < bpf_stack_nr_slots(func); j++) {
-			struct bpf_stack_state *ss = bpf_stack_slot(func, j);
-
-			if (!bpf_is_spilled_reg(ss))
-				continue;
-			reg = &ss->spilled_ptr;
-			if (reg->type != SCALAR_VALUE)
-				continue;
-			reg->precise = false;
-		}
-	}
-}
-
-int bpf_split_cur_state(struct bpf_verifier_env *env)
-{
-	struct bpf_verifier_state *cur = env->cur_state, *new;
-	struct bpf_verifier_state_list *new_sl;
-	struct list_head *head;
-	int insn_idx = cur->insn_idx;
-	int err;
-
-	head = bpf_explored_state(env, insn_idx);
-	new_sl = kzalloc_obj(struct bpf_verifier_state_list, GFP_KERNEL_ACCOUNT);
-	if (!new_sl)
-		return -ENOMEM;
-	env->total_states++;
-	env->explored_states_size++;
-	update_peak_states(env);
-	env->prev_jmps_processed = env->jmps_processed;
-	env->prev_insn_processed = env->insn_processed;
-
-	/* forget precise markings we inherited, see __mark_chain_precision */
-	if (env->bpf_capable)
-		mark_all_scalars_imprecise(env, cur);
-
-	bpf_clear_singular_ids(env, cur);
-
-	/* add new state to the head of linked list */
-	new = &new_sl->state;
-	err = bpf_copy_verifier_state(new, cur);
-	if (err) {
-		bpf_free_verifier_state(new, false);
-		kfree(new_sl);
-		return err;
-	}
-	new->insn_idx = insn_idx;
-	verifier_bug_if(new->branches != 1, env,
-			"%s:branches_to_explore=%d insn %d",
-			__func__, new->branches, insn_idx);
-	err = maybe_enter_scc(env, new);
-	if (err) {
-		bpf_free_verifier_state(new, false);
-		kfree(new_sl);
-		return err;
-	}
-
-	cur->parent = new;
-	cur->last_insn_idx = -1;
-	cur->first_insn_idx = insn_idx;
-	cur->dfs_depth = new->dfs_depth + 1;
-	bpf_clear_jmp_history(cur);
-	list_add(&new_sl->node, head);
-	return 0;
-}
-
-/* Force a checkpoint upon reaching a loop header for a terminating loop. */
-static bool need_loop_checkpoint(struct bpf_verifier_env *env, int insn_idx)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_func_state *frame = cur->frame[cur->curframe];
-	struct loop_stack_entry *top;
-
-	if (bpf_loop_at_index(env, insn_idx) != insn_idx || !frame->loop_stack_cnt)
-		return false;
-	top = &frame->loop_stack[frame->loop_stack_cnt - 1];
-	return top->loop_id == insn_idx && top->terminates;
-}
-
-/* Searches for an innermost loop entry starting from an innermost frame. */
-static struct loop_stack_entry *current_loop(struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *frame;
-	int fr;
-
-	for (fr = st->curframe; fr >= 0; fr--) {
-		frame = st->frame[fr];
-		if (frame->loop_stack_cnt)
-			return &frame->loop_stack[frame->loop_stack_cnt - 1];
-	}
-	return NULL;
-}
-
-/* True if any of the loops call chain is currently in terminates. */
-static bool in_a_terminating_loop(struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *frame;
-	int l, fr;
-
-	for (fr = st->curframe; fr >= 0; fr--) {
-		frame = st->frame[fr];
-		for (l = frame->loop_stack_cnt - 1; l >= 0; l--)
-			if (frame->loop_stack[l].terminates)
-				return true;
-	}
-	return false;
-}
-
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
-{
-	struct loop_stack_entry *old_loop, *cur_loop;
-	struct bpf_verifier_state_list *sl;
-	struct bpf_verifier_state *cur = env->cur_state;
-	bool force_new_state, add_new_state, loop, loop_checkpoint;
-	int n, err, states_cnt = 0;
-	struct list_head *pos, *tmp, *head;
-
-	loop_checkpoint = need_loop_checkpoint(env, insn_idx);
-	force_new_state = env->test_state_freq || bpf_is_force_checkpoint(env, insn_idx) ||
-			  loop_checkpoint ||
-			  /* Avoid accumulating infinitely long jmp history */
-			  cur->jmp_history_cnt > 40;
-
-	/* bpf progs typically have pruning point every 4 instructions
-	 * http://vger.kernel.org/bpfconf2019.html#session-1
-	 * Do not add new state for future pruning if the verifier hasn't seen
-	 * at least 2 jumps and at least 8 instructions.
-	 * This heuristics helps decrease 'total_states' and 'peak_states' metric.
-	 * In tests that amounts to up to 50% reduction into total verifier
-	 * memory consumption and 20% verifier time speedup.
-	 */
-	add_new_state = force_new_state;
-	if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
-	    env->insn_processed - env->prev_insn_processed >= 8)
-		add_new_state = true;
-
-	/* keep cleaning the current state as registers/stack become dead */
-	err = clean_verifier_state(env, cur);
-	if (err)
-		return err;
-
-	loop = false;
-	head = bpf_explored_state(env, insn_idx);
-	list_for_each_safe(pos, tmp, head) {
-		sl = container_of(pos, struct bpf_verifier_state_list, node);
-		states_cnt++;
-		if (sl->state.insn_idx != insn_idx)
-			continue;
-
-		if (sl->state.branches) {
-			struct bpf_func_state *old_top_frame = sl->state.frame[0];
-
-			if (old_top_frame->in_async_callback_fn &&
-			    old_top_frame->async_entry_cnt != cur->frame[0]->async_entry_cnt) {
-				/* Different async_entry_cnt means that the verifier is
-				 * processing another entry into async callback.
-				 * Seeing the same state is not an indication of infinite
-				 * loop or infinite recursion.
-				 * But finding the same state doesn't mean that it's safe
-				 * to stop processing the current state. The previous state
-				 * hasn't yet reached bpf_exit, since state.branches > 0.
-				 * Checking in_async_callback_fn alone is not enough either.
-				 * Since the verifier still needs to catch infinite loops
-				 * inside async callbacks.
-				 */
-				goto skip_inf_loop_check;
-			}
-			/* BPF open-coded iterators loop detection is special.
-			 * states_maybe_looping() logic is too simplistic in detecting
-			 * states that *might* be equivalent, because it doesn't know
-			 * about ID remapping, so don't even perform it.
-			 * See process_iter_next_call() and iter_active_depths_differ()
-			 * for overview of the logic. When current and one of parent
-			 * states are detected as equivalent, it's a good thing: we prove
-			 * convergence and can stop simulating further iterations.
-			 * It's safe to assume that iterator loop will finish, taking into
-			 * account iter_next() contract of eventually returning
-			 * sticky NULL result.
-			 *
-			 * Note, that states have to be compared exactly in this case because
-			 * read and precision marks might not be finalized inside the loop.
-			 * E.g. as in the program below:
-			 *
-			 *     1. r7 = -16
-			 *     2. r6 = bpf_get_prandom_u32()
-			 *     3. while (bpf_iter_num_next(&fp[-8])) {
-			 *     4.   if (r6 != 42) {
-			 *     5.     r7 = -32
-			 *     6.     r6 = bpf_get_prandom_u32()
-			 *     7.     continue
-			 *     8.   }
-			 *     9.   r0 = r10
-			 *    10.   r0 += r7
-			 *    11.   r8 = *(u64 *)(r0 + 0)
-			 *    12.   r6 = bpf_get_prandom_u32()
-			 *    13. }
-			 *
-			 * Here verifier would first visit path 1-3, create a checkpoint at 3
-			 * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
-			 * not have read or precision mark for r7 yet, thus inexact states
-			 * comparison would discard current state with r7=-32
-			 * => unsafe memory access at 11 would not be caught.
-			 */
-			if (is_iter_next_insn(env, insn_idx)) {
-				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					struct bpf_func_state *cur_frame, *iter_frame, *old_frame;
-					struct bpf_reg_state *iter_state, *iter_reg;
-					struct bpf_reg_state *old_iter;
-					int spi;
-
-					cur_frame = cur->frame[cur->curframe];
-					/* btf_check_iter_kfuncs() enforces that
-					 * iter state pointer is always the first arg
-					 */
-					iter_reg = &cur_frame->regs[BPF_REG_1];
-					/* current state is valid due to states_equal(),
-					 * so we can assume valid iter and reg state,
-					 * no need for extra (re-)validations
-					 */
-					spi = bpf_get_spi(iter_reg->var_off.value);
-					iter_frame = bpf_func(env, iter_reg);
-					iter_state = &bpf_stack_slot(iter_frame, spi)->spilled_ptr;
-					old_frame = sl->state.frame[iter_reg->frameno];
-					old_iter = &bpf_stack_slot(old_frame, spi)->spilled_ptr;
-					/*
-					 * states_equal() matches ids through idmap.
-					 * The loop converged only if it's the same
-					 * iterator. The one that was destroyed and
-					 * created again starts from the beginning.
-					 */
-					if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE &&
-					    iter_state->id == old_iter->id) {
-						loop = true;
-						goto hit;
-					}
-				}
-				goto skip_inf_loop_check;
-			}
-			if (is_may_goto_insn_at(env, insn_idx)) {
-				if (sl->state.may_goto_depth != cur->may_goto_depth &&
-				    states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					loop = true;
-					goto hit;
-				}
-			}
-			if (bpf_calls_callback(env, insn_idx)) {
-				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					loop = true;
-					goto hit;
-				}
-				goto skip_inf_loop_check;
-			}
-			/*
-			 * If old state belongs to a control flow loop that we know terminates,
-			 * it should be safe to prune current state. However, account for the
-			 * following situation:
-			 *
-			 *   for (;;) {
-			 *     for (i = 0; i < 10; i++)
-			 *       ...
-			 *   }
-			 *
-			 * Here a backedge to a non-terminating outer loop is dominated by an exit
-			 * from the inner loop. Pruning at a re-entry to the inner loop therefore
-			 * would hide the fact that outer loop is non-terminating.
-			 * Hence, only allow pruning states within the same entry state.
-			 */
-			old_loop = current_loop(&sl->state);
-			cur_loop = current_loop(cur);
-			if (old_loop && cur_loop && in_a_terminating_loop(cur)) {
-				if (old_loop->terminates && cur_loop->terminates &&
-				    old_loop->entry_state == cur_loop->entry_state &&
-				    states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
-					loop = true;
-					goto hit;
-				}
-				goto skip_inf_loop_check;
-			}
-
-			/* attempt to detect infinite loop to avoid unnecessary doomed work */
-			if (states_maybe_looping(&sl->state, cur) &&
-			    states_equal(env, &sl->state, cur, EXACT) &&
-			    !iter_active_depths_differ(&sl->state, cur) &&
-			    sl->state.may_goto_depth == cur->may_goto_depth &&
-			    sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
-				verbose_linfo(env, insn_idx, "; ");
-				verbose(env, "infinite loop detected at insn %d\n", insn_idx);
-				verbose(env, "cur state:");
-				print_verifier_state(env, cur, cur->curframe, true);
-				verbose(env, "old state:");
-				print_verifier_state(env, &sl->state, cur->curframe, true);
-				return -EINVAL;
-			}
-			/* if the verifier is processing a loop, avoid adding new state
-			 * too often, since different loop iterations have distinct
-			 * states and may not help future pruning.
-			 * This threshold shouldn't be too low to make sure that
-			 * a loop with large bound will be rejected quickly.
-			 * The most abusive loop will be:
-			 * r1 += 1
-			 * if r1 < 1000000 goto pc-2
-			 * 1M insn_procssed limit / 100 == 10k peak states.
-			 * This threshold shouldn't be too high either, since states
-			 * at the end of the loop are likely to be useful in pruning.
-			 */
-skip_inf_loop_check:
-			if (!force_new_state &&
-			    env->jmps_processed - env->prev_jmps_processed < 20 &&
-			    env->insn_processed - env->prev_insn_processed < 100)
-				add_new_state = false;
-			goto miss;
-		}
-		/* See comments for mark_all_regs_read_and_precise() */
-		loop = incomplete_read_marks(env, &sl->state);
-		if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
-hit:
-			sl->hit_cnt++;
-
-			/* if previous state reached the exit with precision and
-			 * current state is equivalent to it (except precision marks)
-			 * the precision needs to be propagated back in
-			 * the current state.
-			 */
-			err = 0;
-			if (bpf_is_jmp_point(env, env->insn_idx))
-				err = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);
-			err = err ? : propagate_precision(env, &sl->state, cur, NULL);
-			if (err)
-				return err;
-			/* When processing iterator based loops above propagate_liveness and
-			 * propagate_precision calls are not sufficient to transfer all relevant
-			 * read and precision marks. E.g. consider the following case:
-			 *
-			 *  .-> A --.  Assume the states are visited in the order A, B, C.
-			 *  |   |   |  Assume that state B reaches a state equivalent to state A.
-			 *  |   v   v  At this point, state C is not processed yet, so state A
-			 *  '-- B   C  has not received any read or precision marks from C.
-			 *             Thus, marks propagated from A to B are incomplete.
-			 *
-			 * The verifier mitigates this by performing the following steps:
-			 *
-			 * - Prior to the main verification pass, strongly connected components
-			 *   (SCCs) are computed over the program's control flow graph,
-			 *   intraprocedurally.
-			 *
-			 * - During the main verification pass, `maybe_enter_scc()` checks
-			 *   whether the current verifier state is entering an SCC. If so, an
-			 *   instance of a `bpf_scc_visit` object is created, and the state
-			 *   entering the SCC is recorded as the entry state.
-			 *
-			 * - This instance is associated not with the SCC itself, but with a
-			 *   `bpf_scc_callchain`: a tuple consisting of the call sites leading to
-			 *   the SCC and the SCC id. See `compute_scc_callchain()`.
-			 *
-			 * - When a verification path encounters a `states_equal(...,
-			 *   RANGE_WITHIN)` condition, there exists a call chain describing the
-			 *   current state and a corresponding `bpf_scc_visit` instance. A copy
-			 *   of the current state is created and added to
-			 *   `bpf_scc_visit->backedges`.
-			 *
-			 * - When a verification path terminates, `maybe_exit_scc()` is called
-			 *   from `bpf_update_branch_counts()`. For states with `branches == 0`, it
-			 *   checks whether the state is the entry state of any `bpf_scc_visit`
-			 *   instance. If it is, this indicates that all paths originating from
-			 *   this SCC visit have been explored. `propagate_backedges()` is then
-			 *   called, which propagates read and precision marks through the
-			 *   backedges until a fixed point is reached.
-			 *   (In the earlier example, this would propagate marks from A to B,
-			 *    from C to A, and then again from A to B.)
-			 *
-			 * A note on callchains
-			 * --------------------
-			 *
-			 * Consider the following example:
-			 *
-			 *     void foo() { loop { ... SCC#1 ... } }
-			 *     void main() {
-			 *       A: foo();
-			 *       B: ...
-			 *       C: foo();
-			 *     }
-			 *
-			 * Here, there are two distinct callchains leading to SCC#1:
-			 * - (A, SCC#1)
-			 * - (C, SCC#1)
-			 *
-			 * Each callchain identifies a separate `bpf_scc_visit` instance that
-			 * accumulates backedge states. The `propagate_{liveness,precision}()`
-			 * functions traverse the parent state of each backedge state, which
-			 * means these parent states must remain valid (i.e., not freed) while
-			 * the corresponding `bpf_scc_visit` instance exists.
-			 *
-			 * Associating `bpf_scc_visit` instances directly with SCCs instead of
-			 * callchains would break this invariant:
-			 * - States explored during `C: foo()` would contribute backedges to
-			 *   SCC#1, but SCC#1 would only be exited once the exploration of
-			 *   `A: foo()` completes.
-			 * - By that time, the states explored between `A: foo()` and `C: foo()`
-			 *   (i.e., `B: ...`) may have already been freed, causing the parent
-			 *   links for states from `C: foo()` to become invalid.
-			 */
-			if (loop) {
-				struct bpf_scc_backedge *backedge;
-
-				backedge = kzalloc_obj(*backedge,
-						       GFP_KERNEL_ACCOUNT);
-				if (!backedge)
-					return -ENOMEM;
-				err = bpf_copy_verifier_state(&backedge->state, cur);
-				backedge->state.equal_state = &sl->state;
-				backedge->state.insn_idx = insn_idx;
-				err = err ?: add_scc_backedge(env, &sl->state, backedge);
-				if (err) {
-					bpf_free_verifier_state(&backedge->state, false);
-					kfree(backedge);
-					return err;
-				}
-			}
-			return 1;
-		}
-miss:
-		/* when new state is not going to be added do not increase miss count.
-		 * Otherwise several loop iterations will remove the state
-		 * recorded earlier. The goal of these heuristics is to have
-		 * states from some iterations of the loop (some in the beginning
-		 * and some at the end) to help pruning.
-		 */
-		if (add_new_state)
-			sl->miss_cnt++;
-		/* heuristic to determine whether this state is beneficial
-		 * to keep checking from state equivalence point of view.
-		 * Higher numbers increase max_states_per_insn and verification time,
-		 * but do not meaningfully decrease insn_processed.
-		 * 'n' controls how many times state could miss before eviction.
-		 * Use bigger 'n' for checkpoints because evicting checkpoint states
-		 * too early would hinder iterator convergence.
-		 */
-		n = (bpf_is_force_checkpoint(env, insn_idx) || loop_checkpoint) &&
-		    sl->state.branches > 0 ? 64 : 3;
-		if (sl->miss_cnt > sl->hit_cnt * n + n) {
-			/* the state is unlikely to be useful. Remove it to
-			 * speed up verification
-			 */
-			sl->in_free_list = true;
-			list_del(&sl->node);
-			list_add(&sl->node, &env->free_list);
-			env->free_list_size++;
-			env->explored_states_size--;
-			maybe_free_verifier_state(env, sl);
-		}
-	}
-
-	if (env->max_states_per_insn < states_cnt)
-		env->max_states_per_insn = states_cnt;
-
-	if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
-		return 0;
-
-	if (!add_new_state)
-		return 0;
-
-	/* There were no equivalent states, remember the current one.
-	 * Technically the current state is not proven to be safe yet,
-	 * but it will either reach outer most bpf_exit (which means it's safe)
-	 * or it will be rejected. When there are no loops the verifier won't be
-	 * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
-	 * again on the way to bpf_exit.
-	 * When looping the sl->state.branches will be > 0 and this state
-	 * will not be considered for equivalence until branches == 0.
-	 */
-	return bpf_split_cur_state(env);
-}
-
 int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
 {
 	return clean_verifier_state(env, st);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ef68796c04dd..616a69b4cfb5 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -202,8 +202,6 @@ struct bpf_verifier_stack_elem {
 	u64 diag_log_pos;
 };
 
-#define BPF_COMPLEXITY_LIMIT_STATES	64
-
 #define BPF_GLOBAL_PERCPU_MA_MAX_SIZE  512
 
 #define BPF_PRIV_STACK_MIN_SIZE		64
@@ -1082,7 +1080,6 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 		st->iter.btf_id = btf_id;
 		st->iter.state = BPF_ITER_STATE_ACTIVE;
 		st->iter.loops = nr_loops_below(env->cur_state, env->cur_state->curframe + 1);
-		st->iter.depth = 0;
 
 		for (j = 0; j < BPF_REG_SIZE; j++)
 			slot->slot_type[j] = STACK_ITER;
@@ -1826,14 +1823,9 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	dst_state->flow_entry = src->flow_entry;
 	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
-	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
 	dst_state->first_insn_idx = src->first_insn_idx;
 	dst_state->last_insn_idx = src->last_insn_idx;
-	dst_state->dfs_depth = src->dfs_depth;
-	dst_state->callback_unroll_depth = src->callback_unroll_depth;
-	dst_state->may_goto_depth = src->may_goto_depth;
-	dst_state->equal_state = src->equal_state;
 	for (i = 0; i <= src->curframe; i++) {
 		dst = dst_state->frame[i];
 		if (!dst) {
@@ -1849,45 +1841,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	return 0;
 }
 
-static u32 state_htab_size(struct bpf_verifier_env *env)
-{
-	return env->prog->len;
-}
-
-struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_func_state *state = cur->frame[cur->curframe];
-
-	return &env->explored_states[(idx ^ state->callsite) % state_htab_size(env)];
-}
-
-static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_state *b)
-{
-	int fr;
-
-	if (a->curframe != b->curframe)
-		return false;
-
-	for (fr = a->curframe; fr >= 0; fr--)
-		if (a->frame[fr]->callsite != b->frame[fr]->callsite)
-			return false;
-
-	return true;
-}
-
-void bpf_free_backedges(struct bpf_scc_visit *visit)
-{
-	struct bpf_scc_backedge *backedge, *next;
-
-	for (backedge = visit->backedges; backedge; backedge = next) {
-		bpf_free_verifier_state(&backedge->state, false);
-		next = backedge->next;
-		kfree(backedge);
-	}
-	visit->backedges = NULL;
-}
-
 static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
 		     int *insn_idx, bool pop_log)
 {
@@ -1953,18 +1906,6 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	if (err)
 		return ERR_PTR(-ENOMEM);
 	elem->st.speculative |= speculative;
-	if (elem->st.parent) {
-		++elem->st.parent->branches;
-		/* WARN_ON(branches > 2) technically makes sense here,
-		 * but
-		 * 1. speculative states will bump 'branches' for non-branch
-		 * instructions
-		 * 2. is_state_visited() heuristics may decide not to create
-		 * a new state for a sequence of branches and all such current
-		 * and cloned states will be pointing to a single parent state
-		 * which might have large 'branches' count.
-		 */
-	}
 	return &elem->st;
 }
 
@@ -2636,7 +2577,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	 * This is async callback. It starts in a fresh stack.
 	 * Initialize it similar to do_check_common().
 	 */
-	elem->st.branches = 1;
 	elem->st.in_sleepable = is_sleepable;
 	frame = kzalloc_obj(*frame, GFP_KERNEL_ACCOUNT);
 	if (!frame)
@@ -9228,85 +9168,6 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-/* Look for a previous loop entry at insn_idx: nearest parent state
- * stopped at insn_idx with callsites matching those in cur->frame.
- */
-static struct bpf_verifier_state *find_prev_entry(struct bpf_verifier_env *env,
-						  struct bpf_verifier_state *cur,
-						  int insn_idx)
-{
-	struct bpf_verifier_state_list *sl;
-	struct bpf_verifier_state *st;
-	struct list_head *pos, *head;
-
-	/* Explored states are pushed in stack order, most recent states come first */
-	head = bpf_explored_state(env, insn_idx);
-	list_for_each(pos, head) {
-		sl = container_of(pos, struct bpf_verifier_state_list, node);
-		/* If st->branches != 0 state is a part of current DFS verification path,
-		 * hence cur & st for a loop.
-		 */
-		st = &sl->state;
-		if (st->insn_idx == insn_idx && st->branches && same_callsites(st, cur) &&
-		    st->dfs_depth < cur->dfs_depth)
-			return st;
-	}
-
-	return NULL;
-}
-
-/*
- * Check if scalar registers are exact for the purpose of not widening.
- * More lenient than regs_exact()
- */
-static bool scalars_exact_for_widen(const struct bpf_reg_state *rold,
-				    const struct bpf_reg_state *rcur)
-{
-	return !memcmp(rold, rcur, offsetof(struct bpf_reg_state, id));
-}
-
-static void maybe_widen_reg(struct bpf_verifier_env *env,
-			    struct bpf_reg_state *rold, struct bpf_reg_state *rcur)
-{
-	if (rold->type != SCALAR_VALUE)
-		return;
-	if (rold->type != rcur->type)
-		return;
-	if (rold->precise || rcur->precise || scalars_exact_for_widen(rold, rcur))
-		return;
-	__mark_reg_unknown(env, rcur);
-}
-
-static int widen_imprecise_scalars(struct bpf_verifier_env *env,
-				   struct bpf_verifier_state *old,
-				   struct bpf_verifier_state *cur)
-{
-	struct bpf_func_state *fold, *fcur;
-	int i, fr, num_slots;
-
-	for (fr = old->curframe; fr >= 0; fr--) {
-		fold = old->frame[fr];
-		fcur = cur->frame[fr];
-
-		for (i = 0; i < MAX_BPF_REG; i++)
-			maybe_widen_reg(env,
-					&fold->regs[i],
-					&fcur->regs[i]);
-
-		num_slots = min(bpf_stack_nr_slots(fold), bpf_stack_nr_slots(fcur));
-		for (i = 0; i < num_slots; i++) {
-			struct bpf_stack_state *old_ss = bpf_stack_slot(fold, i);
-			struct bpf_stack_state *cur_ss = bpf_stack_slot(fcur, i);
-
-			if (!bpf_is_spilled_reg(old_ss) || !bpf_is_spilled_reg(cur_ss))
-				continue;
-
-			maybe_widen_reg(env, &old_ss->spilled_ptr, &cur_ss->spilled_ptr);
-		}
-	}
-	return 0;
-}
-
 static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_st,
 						 struct bpf_call_arg_meta *meta)
 {
@@ -9454,13 +9315,11 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 
 		queued_iter = get_iter_from_state(queued_st, meta);
 		queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
-		queued_iter->iter.depth++;
 
 		queued_fr = queued_st->frame[queued_st->curframe];
 		mark_ptr_not_null_reg(&queued_fr->regs[BPF_REG_0]);
 	}
 
-	/* switch to DRAINED state, but keep the depth unchanged */
 	/* mark current iter state as drained and assume returned NULL */
 	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
 	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
@@ -12014,7 +11873,6 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	if (err)
 		return err;
 
-	callback_state->callback_unroll_depth++;
 	callback_state->frame[callback_state->curframe - 1]->callback_depth++;
 	caller->callback_depth = 0;
 	return 0;
@@ -12975,10 +12833,9 @@ static bool retval_range_within(struct bpf_retval_range range, const struct bpf_
 
 static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 {
-	struct bpf_verifier_state *state = env->cur_state, *prev_st;
+	struct bpf_verifier_state *state = env->cur_state;
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
-	bool in_callback_fn;
 	u32 i, nregs;
 	int err;
 
@@ -13067,11 +12924,9 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	invalidate_stack_slices(env, callee->frameno, INT_MIN, 0);
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
-	 * there function call logic would reschedule callback visit. If iteration
-	 * converges is_state_visited() would prune that visit eventually.
+	 * there function call logic would reschedule callback visit.
 	 */
-	in_callback_fn = callee->in_callback_fn;
-	if (in_callback_fn)
+	if (callee->in_callback_fn)
 		*insn_idx = callee->callsite;
 	else
 		*insn_idx = callee->callsite + 1;
@@ -13088,24 +12943,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	free_func_state(callee);
 	state->frame[state->curframe--] = NULL;
 	invalidate_outgoing_stack_args(env, caller);
-
-	/* for callbacks widen imprecise scalars to make programs like below verify:
-	 *
-	 *   struct ctx { int i; }
-	 *   void cb(int idx, struct ctx *ctx) { ctx->i++; ... }
-	 *   ...
-	 *   struct ctx = { .i = 0; }
-	 *   bpf_loop(100, cb, &ctx, 0);
-	 *
-	 * This is similar to what is done in process_iter_next_call() for open
-	 * coded iterators.
-	 */
-	prev_st = in_callback_fn ? find_prev_entry(env, state, *insn_idx) : NULL;
-	if (prev_st) {
-		err = widen_imprecise_scalars(env, prev_st, state);
-		if (err)
-			return err;
-	}
 	return 0;
 }
 
@@ -20140,19 +19977,14 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	}
 
 	if (opcode == BPF_JCOND) {
-		struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
+		struct bpf_verifier_state *queued_st;
 		int idx = *insn_idx;
 
-		prev_st = find_prev_entry(env, cur_st->parent, idx);
-
 		/* branch out 'fallthrough' insn as a new state to explore */
 		queued_st = push_stack(env, idx + 1, idx, false);
 		if (IS_ERR(queued_st))
 			return PTR_ERR(queued_st);
 
-		queued_st->may_goto_depth++;
-		if (prev_st)
-			widen_imprecise_scalars(env, prev_st, queued_st);
 		*insn_idx += insn->off;
 		return 0;
 	}
@@ -21620,7 +21452,6 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 	case BPF_JMP32: {
 		u8 opcode = BPF_OP(insn->code);
 
-		env->jmps_processed++;
 		if (opcode == BPF_CALL) {
 			if (env->cur_state->active_locks) {
 				/* similar to static subprog calls callx is allowed under a lock */
@@ -23590,46 +23421,10 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 
 static void free_states(struct bpf_verifier_env *env)
 {
-	struct bpf_verifier_state_list *sl;
-	struct list_head *head, *pos, *tmp;
-	struct bpf_scc_info *info;
-	int i, j;
-
 	bpf_free_verifier_state(env->cur_state, true);
 	env->cur_state = NULL;
 	while (!pop_stack(env, NULL, NULL, false));
 	bpf_flow_reset(env);
-
-	list_for_each_safe(pos, tmp, &env->free_list) {
-		sl = container_of(pos, struct bpf_verifier_state_list, node);
-		bpf_free_verifier_state(&sl->state, false);
-		kfree(sl);
-	}
-	INIT_LIST_HEAD(&env->free_list);
-
-	for (i = 0; i < env->scc_cnt; ++i) {
-		info = env->scc_info[i];
-		if (!info)
-			continue;
-		for (j = 0; j < info->num_visits; j++)
-			bpf_free_backedges(&info->visits[j]);
-		kvfree(info);
-		env->scc_info[i] = NULL;
-	}
-
-	if (!env->explored_states)
-		return;
-
-	for (i = 0; i < state_htab_size(env); i++) {
-		head = &env->explored_states[i];
-
-		list_for_each_safe(pos, tmp, head) {
-			sl = container_of(pos, struct bpf_verifier_state_list, node);
-			bpf_free_verifier_state(&sl->state, false);
-			kfree(sl);
-		}
-		INIT_LIST_HEAD(&env->explored_states[i]);
-	}
 }
 
 static int do_check_common(struct bpf_verifier_env *env, int subprog,
@@ -23652,7 +23447,6 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 		return -ENOMEM;
 	state->curframe = 0;
 	state->speculative = false;
-	state->branches = 1;
 	state->in_sleepable = context == BPF_FUNC_CTX_SLEEPABLE;
 	/* nobody knows what the callers of a global function or of an extension hold */
 	state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
@@ -25746,13 +25540,9 @@ static void log_program(struct bpf_verifier_env *env)
 	u64 pos, insn_pos;
 	u32 i, j;
 
-	verbose(env, "Program dump (scc? loop_header? idom insn#: live_regs_before):\n");
+	verbose(env, "Program dump (loop_header? idom insn#: live_regs_before):\n");
 	for (i = 0; i < insn_cnt; ++i) {
 		verbose_linfo(env, i, "    ; ");
-		if (env->insn_aux_data[i].scc)
-			verbose(env, "%3d ", env->insn_aux_data[i].scc);
-		else
-			verbose(env, "    ");
 		if (env->insn_aux_data[i].loop_header >= 0)
 			verbose(env, "%3d ", env->insn_aux_data[i].loop_header);
 		else
@@ -25875,21 +25665,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
 		env->strict_alignment = false;
 
-	if (is_priv)
-		env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
 	env->test_reg_invariants = attr->prog_flags & BPF_F_TEST_REG_INVARIANTS;
 
-	env->explored_states = kvzalloc_objs(struct list_head,
-					     state_htab_size(env),
-					     GFP_KERNEL_ACCOUNT);
-	ret = -ENOMEM;
-	if (!env->explored_states)
-		goto skip_full_check;
-
-	for (i = 0; i < state_htab_size(env); i++)
-		INIT_LIST_HEAD(&env->explored_states[i]);
-	INIT_LIST_HEAD(&env->free_list);
-
 	/* Prepare BTF and func_info needed to discover all subprograms. */
 	ret = bpf_prepare_btf_info(env, attr, uattr);
 	if (ret < 0)
@@ -25978,10 +25755,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = bpf_compute_scc(env);
-	if (ret < 0)
-		goto skip_full_check;
-
 	/* must precede the first bpf_ret_reg_pair() user below */
 	ret = bpf_compute_subprog_ret_regs(env);
 	if (ret < 0)
@@ -26054,8 +25827,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		ret = bpf_insert_loop_guards(env);
 
 skip_full_check:
-	kvfree(env->explored_states);
-
 	/* might decrease stack depth, keep it before passes that
 	 * allocate additional slots.
 	 */
@@ -26208,7 +25979,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.postorder_nums);
 	kvfree(env->cfg.insn_postorder);
-	kvfree(env->scc_info);
 	kvfree(env->succ);
 	kvfree(env->gotox_tmp_buf);
 	kvfree(env->callx_edges);
-- 
2.55.0

[PATCH bpf-next v4 24/46] bpf: Remove precision tracking

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Precision tracking is a part of pruning. The verifier followed each path
and stopped a path when a state that it saw before at the same insn covered
the current state. To stop more paths, the comparison ignored a scalar
unless a later check depended on its value. To find these scalars, the
verifier went back from each check through the history of the path and
marked them as precise.

The verifier does not prune paths any more. It joins the states that meet
at the start of a basic block, and the join covers both:

  r1=0 and r1=1 meet      pruning                      join
  ----------------------  ---------------------------  ---------------
  a check depends on r1   r1 is precise,               verify r1=[0,1]
                          verify the two states
  no check depends on r1  drop the second state        verify r1=[0,1]

Each later check sees the full range of r1. Thus the verifier does not have
to find out afterwards which scalars a check depends on, and nothing reads
the marks.

One task of the marks is left. Some checks need a constant, for example the
size for bpf_ringbuf_reserve(), and the join of two constants is a range.
flow_compute_need_const() finds the registers and stack slots that go into
such checks, one time, before the main pass. States that differ in them are
not joined.

Remove the 'precise' bit, backtrack.c, jmp_history, the parent link of
a state and the marks that scalar evolution asks for: 1650 lines. With the
bit gone, __mark_reg_unknown() and bpf_mark_reg_unknown_imprecise() are one
function, bpf_mark_reg_unknown(). __mark_reg_const_zero() does not need
env, and BPF_REGMASK_ARGS has no user.

What user space sees:
 - The log has no "mark_precise:" lines and no 'P' in "R0=Pscalar()".
 - Nothing sets longest_mark_read_walk. veristat and other tools parse the
   statistics line, thus it continues to show "mark_read 0".

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  120 +---
 kernel/bpf/Makefile          |    2 +-
 kernel/bpf/backtrack.c       | 1059 ----------------------------------
 kernel/bpf/log.c             |    2 -
 kernel/bpf/scev.c            |   42 +-
 kernel/bpf/states.c          |   11 +-
 kernel/bpf/verifier.c        |  475 ++-------------
 7 files changed, 63 insertions(+), 1648 deletions(-)
 delete mode 100644 kernel/bpf/backtrack.c

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 902a339b0caf..7ee3e40214b5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -35,8 +35,8 @@ static inline u16 imod(s64 v, u16 step)
 #define BPF_MAX_VAR_SIZ	(1 << 29)
 /*
  * size of tmp_str_buf in bpf_verifier.
- * we need at least 1399 bytes to fit full stack mask representation
- * (in the "-8,-16,...,-2048" form)
+ * The longest text is the list of stack slots of fmt_spis_mask(),
+ * which cuts what does not fit.
  */
 #define TMP_STR_BUF_LEN 1408
 /* Patch buffer size */
@@ -201,9 +201,7 @@ struct bpf_reg_state {
 	 * other registers. Kept outside the metadata union for ID remapping
 	 * during state comparisons.
 	 */
-	u32 map_uid:31;
-	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
-	u32 precise:1;
+	u32 map_uid;
 };
 
 static inline s64 reg_smin(const struct bpf_reg_state *reg)
@@ -295,10 +293,6 @@ enum bpf_stack_slot_type {
 #define BPF_HALF_REG_SIZE	4
 #define STACK_SLOT_SZ		4
 
-#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
-			  (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
-			  (1 << BPF_REG_5))
-
 #define BPF_MAIN_FUNC (-1)
 
 #define BPF_DYNPTR_SIZE		sizeof(struct bpf_dynptr_kern)
@@ -351,7 +345,6 @@ struct bpf_retval_range {
 struct bpf_loop_iters {
 	u32 min_header_count;   /* min number of times header is executed */
 	u32 max_header_count;   /* max number of times header is executed */
-	unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* live, changed by the loop, no summary of how: left to the join at the head */
 	unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* [0, max] at the entry, and the loop only clears bits */
@@ -518,44 +511,6 @@ struct bpf_func_state {
 /* What the interpreter has stack for. A longer chain of calls requires the JIT. */
 #define MAX_CALL_FRAMES_INTERP 16
 
-/* instruction history flags, used in bpf_jmp_history_entry.flags field.
- * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
- */
-enum {
-	INSN_F_STACK_ACCESS = BIT(0),
-
-	INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
-	INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
-
-	INSN_F_STACK_ARG_ACCESS = BIT(3),
-};
-
-/* Registers linked to one jump condition that a history entry can record */
-#define BPF_LINKED_REGS_MAX	5
-
-struct bpf_jmp_history_entry {
-	/* insn idx can't be bigger than 1 million */
-	u32 idx : 20;
-	u32 frame : 5;	/* stack access frame number */
-	/* special INSN_F_xxx flags */
-	u32 flags : 4;
-	u32 : 3;
-	u32 prev_idx : 20;
-	u32 spi : 12;	/* stack slot index */
-	/*
-	 * Scalar registers and spilled scalars linked to the condition of
-	 * this jump, which need precision tracking together when the jump is
-	 * backtracked. Each is packed as 5 bits of frame number, one bit
-	 * telling a register from a stack slot and 10 bits of register or
-	 * slot index, see linked_regs_pack().
-	 */
-	u16 linked_regs[BPF_LINKED_REGS_MAX];
-	u8 linked_regs_cnt;
-};
-
-static_assert(MAX_CALL_FRAMES <= (1 << 5));
-static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
-
 /* Maximum number of bpf_reg_state objects that can exist at once */
 #define MAX_STACK_ARG_SLOTS (MAX_BPF_STATIC_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
 #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
@@ -564,7 +519,6 @@ static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
 struct bpf_verifier_state {
 	/* call stack tracking */
 	struct bpf_func_state *frame[MAX_CALL_FRAMES];
-	struct bpf_verifier_state *parent;
 	/* Acquired reference states */
 	struct bpf_reference_state *refs;
 	u32 insn_idx;
@@ -595,19 +549,8 @@ struct bpf_verifier_state {
 	 */
 	u32 flow_entry;
 
-	/*
-	 * First and last insn idx of this verifier state.
-	 * last_insn_idx is -1 if no instructions have been executed yet.
-	 */
-	u32 first_insn_idx;
+	/* The insn that the state comes from, -1 if no instructions have been executed yet. */
 	u32 last_insn_idx;
-	/* jmp history recorded from first to last.
-	 * backtracking is using it to go from last to first.
-	 * For most states jmp_history_cnt is [0-3].
-	 * For loops can go up to ~40.
-	 */
-	struct bpf_jmp_history_entry *jmp_history;
-	u32 jmp_history_cnt;
 };
 
 /* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
@@ -957,8 +900,8 @@ struct bpf_insn_aux_data {
 #define MAX_USED_BTFS 64 /* max number of BTFs accessed by one BPF program */
 
 /*
- * Longest line the verifier log can carry: a full stack mask of
- * MAX_BPF_STACK_SLOTS slots, see TMP_STR_BUF_LEN, plus its prefix.
+ * Longest line the verifier log can carry: tmp_str_buf, see
+ * TMP_STR_BUF_LEN, plus its prefix.
  */
 #define BPF_VERIFIER_TMP_LOG_SIZE	2048
 
@@ -1123,14 +1066,6 @@ static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
 struct bpf_diag;
 struct bpf_verifier_env;
 
-struct backtrack_state {
-	struct bpf_verifier_env *env;
-	u32 frame;
-	u32 reg_masks[MAX_CALL_FRAMES];
-	unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
-	u32 stack_arg_masks[MAX_CALL_FRAMES];
-};
-
 struct bpf_id_pair {
 	u32 old;
 	u32 cur;
@@ -1329,8 +1264,6 @@ struct bpf_verifier_env {
 		u32 gotox_edges;
 		bool subprog_jts_ready;
 	} cfg;
-	struct backtrack_state bt;
-	struct bpf_jmp_history_entry *cur_hist_ent;
 	/* Per-callsite snapshot of the parent's tracked spill slots for cross-frame fills. */
 	struct spill_snapshot **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
@@ -1352,8 +1285,6 @@ struct bpf_verifier_env {
 	 * memory consumption during verification
 	 */
 	u32 peak_states;
-	/* longest register parentage chain walked for liveness marking */
-	u32 longest_mark_read_walk;
 	/*
 	 * The program's fd_array comes in two shapes, told apart by whether
 	 * the caller passed fd_array_cnt. They are mutually exclusive:
@@ -1582,7 +1513,6 @@ int bpf_check_ext_prologue(struct bpf_verifier_log *log,
 			   const struct bpf_prog *tgt_prog, u32 btf_id);
 void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 
-int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
@@ -1633,25 +1563,15 @@ int bpf_find_latches(struct bpf_verifier_env *env, bool flags);
 bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
 int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
 
-void bpf_clear_jmp_history(struct bpf_verifier_state *state);
 int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 			    const struct bpf_verifier_state *src);
 void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
-			 int insn_flags, int spi, int frame, const u16 *linked_regs,
-			 u8 linked_regs_cnt);
-void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg);
 void bpf_mark_reg_known_scalar(struct bpf_reg_state *reg, u64 imm);
-void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg);
+void bpf_mark_reg_unknown(struct bpf_reg_state *reg);
 int bpf_grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size);
-void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
-				  struct bpf_verifier_state *st);
 void bpf_clear_singular_ids(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
-int bpf_mark_chain_precision(struct bpf_verifier_env *env,
-			     struct bpf_verifier_state *starting_state,
-			     int regno, bool *changed);
 
 static inline int bpf_get_spi(s32 off)
 {
@@ -1728,31 +1648,6 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
 	return reg->type == SCALAR_VALUE && tnum_equals_const(reg->var_off, 0);
 }
 
-static inline void bpf_bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
-{
-	bt->reg_masks[frame] |= 1 << reg;
-}
-
-static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	__set_bit(slot, bt->stack_masks[frame]);
-}
-
-static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	bt->stack_arg_masks[frame] |= 1 << slot;
-}
-
-static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
-{
-	return bt->reg_masks[frame] & (1 << reg);
-}
-
-static inline bool bt_is_frame_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	return test_bit(slot, bt->stack_masks[frame]);
-}
-
 bool bpf_map_is_rdonly(const struct bpf_map *map);
 struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
 				       const struct bpf_map *map, u32 *cnt);
@@ -1974,7 +1869,6 @@ static inline int bpf_head_guard_target(const struct bpf_verifier_env *env, int
 
 	return aux->guard_at_header ? aux->guard_to : -1;
 }
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 
 /* Kinds of member a by-value struct or union may be composed of. */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index ab3ba42596ec..d26c62326e58 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o join.o flow.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
deleted file mode 100644
index a6e69c1f5ff6..000000000000
--- a/kernel/bpf/backtrack.c
+++ /dev/null
@@ -1,1059 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
-#include <linux/bpf.h>
-#include <linux/bpf_verifier.h>
-#include <linux/filter.h>
-#include <linux/bitmap.h>
-
-#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-
-/* for any branch, call, exit record the history of jmps in the given state */
-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
-			 int insn_flags, int spi, int frame, const u16 *linked_regs,
-			 u8 linked_regs_cnt)
-{
-	u32 cnt = cur->jmp_history_cnt;
-	struct bpf_jmp_history_entry *p;
-	size_t alloc_size;
-
-	/* FLOW: nothing looks at the history */
-	return 0;
-
-	/* combine instruction flags if we already recorded this instruction */
-	if (env->cur_hist_ent) {
-		/* atomic instructions push insn_flags twice, for READ and
-		 * WRITE sides, but they should agree on stack slot
-		 */
-		verifier_bug_if((env->cur_hist_ent->flags & insn_flags) &&
-				(env->cur_hist_ent->flags & insn_flags) != insn_flags,
-				env, "insn history: insn_idx %d cur flags %x new flags %x",
-				env->insn_idx, env->cur_hist_ent->flags, insn_flags);
-		env->cur_hist_ent->flags |= insn_flags;
-		env->cur_hist_ent->spi = spi;
-		env->cur_hist_ent->frame = frame;
-		verifier_bug_if(env->cur_hist_ent->linked_regs_cnt != 0, env,
-				"insn history: insn_idx %d has %u linked regs",
-				env->insn_idx, env->cur_hist_ent->linked_regs_cnt);
-		if (linked_regs_cnt)
-			memcpy(env->cur_hist_ent->linked_regs, linked_regs,
-			       linked_regs_cnt * sizeof(*linked_regs));
-		env->cur_hist_ent->linked_regs_cnt = linked_regs_cnt;
-		return 0;
-	}
-
-	cnt++;
-	alloc_size = kmalloc_size_roundup(size_mul(cnt, sizeof(*p)));
-	p = krealloc(cur->jmp_history, alloc_size, GFP_KERNEL_ACCOUNT);
-	if (!p)
-		return -ENOMEM;
-	cur->jmp_history = p;
-
-	p = &cur->jmp_history[cnt - 1];
-	p->idx = env->insn_idx;
-	p->prev_idx = env->prev_insn_idx;
-	p->flags = insn_flags;
-	p->spi = spi;
-	p->frame = frame;
-	if (linked_regs_cnt)
-		memcpy(p->linked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));
-	p->linked_regs_cnt = linked_regs_cnt;
-	cur->jmp_history_cnt = cnt;
-	env->cur_hist_ent = p;
-
-	return 0;
-}
-
-static bool is_atomic_load_insn(const struct bpf_insn *insn)
-{
-	return BPF_CLASS(insn->code) == BPF_STX &&
-	       BPF_MODE(insn->code) == BPF_ATOMIC &&
-	       insn->imm == BPF_LOAD_ACQ;
-}
-
-static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
-{
-	return BPF_CLASS(insn->code) == BPF_STX &&
-	       BPF_MODE(insn->code) == BPF_ATOMIC &&
-	       (insn->imm & BPF_FETCH);
-}
-
-/* Backtrack one insn at a time. If idx is not at the top of recorded
- * history then previous instruction came from straight line execution.
- * Return -ENOENT if we exhausted all instructions within given state.
- *
- * It's legal to have a bit of a looping with the same starting and ending
- * insn index within the same state, e.g.: 3->4->5->3, so just because current
- * instruction index is the same as state's first_idx doesn't mean we are
- * done. If there is still some jump history left, we should keep going. We
- * need to take into account that we might have a jump history between given
- * state's parent and itself, due to checkpointing. In this case, we'll have
- * history entry recording a jump from last instruction of parent state and
- * first instruction of given state.
- */
-static int get_prev_insn_idx(struct bpf_verifier_state *st, int i,
-			     u32 *history)
-{
-	u32 cnt = *history;
-
-	if (i == st->first_insn_idx) {
-		if (cnt == 0)
-			return -ENOENT;
-		if (cnt == 1 && st->jmp_history[0].idx == i)
-			return -ENOENT;
-	}
-
-	if (cnt && st->jmp_history[cnt - 1].idx == i) {
-		i = st->jmp_history[cnt - 1].prev_idx;
-		(*history)--;
-	} else {
-		i--;
-	}
-	return i;
-}
-
-static struct bpf_jmp_history_entry *get_jmp_hist_entry(struct bpf_verifier_state *st,
-						        u32 hist_end, int insn_idx)
-{
-	if (hist_end > 0 && st->jmp_history[hist_end - 1].idx == insn_idx)
-		return &st->jmp_history[hist_end - 1];
-	return NULL;
-}
-
-static inline void bt_init(struct backtrack_state *bt, u32 frame)
-{
-	bt->frame = frame;
-}
-
-static inline void bt_reset(struct backtrack_state *bt)
-{
-	struct bpf_verifier_env *env = bt->env;
-
-	memset(bt, 0, sizeof(*bt));
-	bt->env = env;
-}
-
-static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)
-{
-	return bitmap_empty(bt->stack_masks[frame], MAX_BPF_STACK_SLOTS);
-}
-
-static inline bool bt_stack_empty(struct backtrack_state *bt)
-{
-	return bt_frame_stack_empty(bt, bt->frame);
-}
-
-static inline bool bt_empty(struct backtrack_state *bt)
-{
-	u32 mask = 0;
-	int i;
-
-	for (i = 0; i <= bt->frame; i++) {
-		mask |= bt->reg_masks[i] | bt->stack_arg_masks[i];
-		if (!bt_frame_stack_empty(bt, i))
-			return false;
-	}
-
-	return mask == 0;
-}
-
-static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	bt->stack_arg_masks[frame] &= ~(1 << slot);
-}
-
-static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	return bt->stack_arg_masks[frame] & (1 << slot);
-}
-
-static inline int bt_subprog_enter(struct backtrack_state *bt)
-{
-	if (bt->frame == MAX_CALL_FRAMES - 1) {
-		verifier_bug(bt->env, "subprog enter from frame %d", bt->frame);
-		return -EFAULT;
-	}
-	bt->frame++;
-	return 0;
-}
-
-static inline int bt_subprog_exit(struct backtrack_state *bt)
-{
-	if (bt->frame == 0) {
-		verifier_bug(bt->env, "subprog exit from frame 0");
-		return -EFAULT;
-	}
-	bt->frame--;
-	return 0;
-}
-
-static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
-{
-	bt->reg_masks[frame] &= ~(1 << reg);
-}
-
-static inline void bt_set_reg(struct backtrack_state *bt, u32 reg)
-{
-	bpf_bt_set_frame_reg(bt, bt->frame, reg);
-}
-
-static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)
-{
-	bt_clear_frame_reg(bt, bt->frame, reg);
-}
-
-static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
-	__clear_bit(slot, bt->stack_masks[frame]);
-}
-
-static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)
-{
-	return bt->reg_masks[frame];
-}
-
-static inline u32 bt_reg_mask(struct backtrack_state *bt)
-{
-	return bt->reg_masks[bt->frame];
-}
-
-static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
-{
-	return bt->stack_masks[frame];
-}
-
-static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
-{
-	return bt_frame_stack_mask(bt, bt->frame);
-}
-
-static inline u32 bt_stack_arg_mask(struct backtrack_state *bt)
-{
-	return bt->stack_arg_masks[bt->frame];
-}
-
-static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
-{
-	return bt->reg_masks[bt->frame] & (1 << reg);
-}
-
-/* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
-static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
-{
-	DECLARE_BITMAP(mask, 64);
-	bool first = true;
-	int i, n;
-
-	buf[0] = '\0';
-
-	bitmap_from_u64(mask, reg_mask);
-	for_each_set_bit(i, mask, 32) {
-		n = snprintf(buf, buf_sz, "%sr%d", first ? "" : ",", i);
-		first = false;
-		buf += n;
-		buf_sz -= n;
-		if (buf_sz < 0)
-			break;
-	}
-}
-
-/* format stack slots bitmask, e.g., "-8,-24,-40" for slots 0, 2 and 4 */
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)
-{
-	bool first = true;
-	int i, n;
-
-	buf[0] = '\0';
-
-	for_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {
-		n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
-		first = false;
-		buf += n;
-		buf_sz -= n;
-		if (buf_sz < 0)
-			break;
-	}
-}
-
-/* For given verifier state backtrack_insn() is called from the last insn to
- * the first insn. Its purpose is to compute a bitmask of registers and
- * stack slots that needs precision in the parent verifier state.
- *
- * @idx is an index of the instruction we are currently processing;
- * @subseq_idx is an index of the subsequent instruction that:
- *   - *would be* executed next, if jump history is viewed in forward order;
- *   - *was* processed previously during backtracking.
- */
-static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
-			  struct bpf_jmp_history_entry *hist, struct backtrack_state *bt)
-{
-	struct bpf_insn *insn = env->prog->insnsi + idx;
-	u8 class = BPF_CLASS(insn->code);
-	u8 opcode = BPF_OP(insn->code);
-	u8 mode = BPF_MODE(insn->code);
-	u32 dreg = insn->dst_reg;
-	u32 sreg = insn->src_reg;
-	u32 spi, i, fr;
-
-	if (insn->code == 0)
-		return 0;
-	if (env->log.level & BPF_LOG_LEVEL2) {
-		fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_reg_mask(bt));
-		verbose(env, "mark_precise: frame%d: regs=%s ",
-			bt->frame, env->tmp_str_buf);
-		bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
-		verbose(env, "stack=%s before ", env->tmp_str_buf);
-		verbose(env, "%d: ", idx);
-		bpf_verbose_insn(env, insn);
-		verbose(env, "\n");
-	}
-
-	/* If there is a history record that some registers gained range at this insn,
-	 * propagate precision marks to those registers, so that bt_is_reg_set()
-	 * accounts for these registers.
-	 */
-	bpf_bt_sync_linked_regs(bt, hist);
-
-	if (class == BPF_ALU || class == BPF_ALU64) {
-		if (!bt_is_reg_set(bt, dreg))
-			return 0;
-		if (opcode == BPF_END || opcode == BPF_NEG) {
-			/* sreg is reserved and unused
-			 * dreg still need precision before this insn
-			 */
-			return 0;
-		} else if (opcode == BPF_MOV) {
-			if (BPF_SRC(insn->code) == BPF_X) {
-				/* dreg = sreg or dreg = (s8, s16, s32)sreg
-				 * dreg needs precision after this insn
-				 * sreg needs precision before this insn
-				 */
-				bt_clear_reg(bt, dreg);
-				if (sreg != BPF_REG_FP && sreg != BPF_REG_ARENA_SP)
-					bt_set_reg(bt, sreg);
-			} else {
-				/* dreg = K
-				 * dreg needs precision after this insn.
-				 * Corresponding register is already marked
-				 * as precise=true in this verifier state.
-				 * No further markings in parent are necessary
-				 */
-				bt_clear_reg(bt, dreg);
-			}
-		} else {
-			if (BPF_SRC(insn->code) == BPF_X) {
-				/* dreg += sreg
-				 * both dreg and sreg need precision
-				 * before this insn
-				 */
-				if (sreg != BPF_REG_FP)
-					bt_set_reg(bt, sreg);
-			} /* else dreg += K
-			   * dreg still needs precision before this insn
-			   */
-		}
-	} else if (class == BPF_LDX ||
-		   is_atomic_load_insn(insn) ||
-		   is_atomic_fetch_insn(insn)) {
-		u32 load_reg = dreg;
-
-		/*
-		 * Atomic fetch operation writes the old value into
-		 * a register (sreg or r0) and if it was tracked for
-		 * precision, propagate to the stack slot like we do
-		 * in regular ldx.
-		 */
-		if (is_atomic_fetch_insn(insn))
-			load_reg = insn->imm == BPF_CMPXCHG ?
-				   BPF_REG_0 : sreg;
-
-		if (!bt_is_reg_set(bt, load_reg))
-			return 0;
-		bt_clear_reg(bt, load_reg);
-
-		if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
-			spi = hist->spi;
-			/*
-			 * Stack arg read: callee reads from r11+off, but
-			 * the data lives in the caller's stack_arg_regs.
-			 * Set the mask in the caller frame so precision
-			 * is marked in the caller's slot at the callee
-			 * entry checkpoint.
-			 */
-			bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi);
-			return 0;
-		}
-
-		/* scalars can only be spilled into stack w/o losing precision.
-		 * Load from any other memory can be zero extended.
-		 * The desire to keep that precision is already indicated
-		 * by 'precise' mark in corresponding register of this state.
-		 * No further tracking necessary.
-		 */
-		if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
-			return 0;
-		/* dreg = *(u64 *)[fp - off] was a fill from the stack.
-		 * that [fp - off] slot contains scalar that needs to be
-		 * tracked with precision
-		 */
-		spi = hist->spi;
-		fr = hist->frame;
-		bpf_bt_set_frame_slot(bt, fr, spi);
-	} else if (class == BPF_STX || class == BPF_ST) {
-		if (bt_is_reg_set(bt, dreg))
-			/* stx & st shouldn't be using _scalar_ dst_reg
-			 * to access memory. It means backtracking
-			 * encountered a case of pointer subtraction.
-			 */
-			return -ENOTSUPP;
-
-		if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
-			spi = hist->spi;
-			if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi))
-				return 0;
-			bt_clear_frame_stack_arg_slot(bt, bt->frame, spi);
-			if (class == BPF_STX)
-				bt_set_reg(bt, sreg);
-			return 0;
-		}
-
-		/* scalars can only be spilled into stack */
-		if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
-			return 0;
-		spi = hist->spi;
-		fr = hist->frame;
-		if (!bt_is_frame_slot_set(bt, fr, spi))
-			return 0;
-		bt_clear_frame_slot(bt, fr, spi);
-		if (class == BPF_STX)
-			bt_set_reg(bt, sreg);
-	} else if (class == BPF_JMP || class == BPF_JMP32) {
-		if (bpf_pseudo_call(insn) || bpf_is_callx(insn)) {
-			int subprog_insn_idx, subprog = -1;
-
-			if (bpf_pseudo_call(insn)) {
-				subprog_insn_idx = idx + insn->imm + 1;
-				subprog = bpf_find_subprog(env, subprog_insn_idx);
-				if (subprog < 0)
-					return -EFAULT;
-			}
-
-			/* callx calls static subprogs only */
-			if (subprog >= 0 && bpf_subprog_is_global(env, subprog)) {
-				/* check that jump history doesn't have any
-				 * extra instructions from subprog; the next
-				 * instruction after call to global subprog
-				 * should be literally next instruction in
-				 * caller program
-				 */
-				verifier_bug_if(idx + 1 != subseq_idx, env,
-						"extra insn from subprog");
-				/* global subprog always sets R0 */
-				bt_clear_reg(bt, BPF_REG_0);
-				/* and if it does not set R2, main pass would catch it */
-				bt_clear_reg(bt, BPF_REG_2);
-				/* r1-r5 are invalidated after subprog call,
-				 * so for global func call it shouldn't be set
-				 * anymore
-				 */
-				if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
-					verifier_bug(env, "global subprog unexpected regs %x",
-						     bt_reg_mask(bt));
-					return -EFAULT;
-				}
-				return 0;
-			} else {
-				/* static subprog call instruction, which
-				 * means that we are exiting current subprog,
-				 * so only r1-r5 could be still requested as
-				 * precise, r0 and r6-r10 or any stack slot in
-				 * the current frame should be zero by now
-				 */
-				if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) {
-					verifier_bug(env, "static subprog unexpected regs %x",
-						     bt_reg_mask(bt));
-					return -EFAULT;
-				}
-				/* we are now tracking register spills correctly,
-				 * so any instance of leftover slots is a bug
-				 */
-				if (!bt_stack_empty(bt)) {
-					bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-							   bt_stack_mask(bt));
-					verifier_bug(env, "static subprog leftover stack slots %s",
-						     env->tmp_str_buf);
-					return -EFAULT;
-				}
-				/* propagate r1-r5 to the caller */
-				for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
-					if (bt_is_reg_set(bt, i)) {
-						bt_clear_reg(bt, i);
-						bpf_bt_set_frame_reg(bt, bt->frame - 1, i);
-					}
-				}
-				if (bt_stack_arg_mask(bt)) {
-					verifier_bug(env,
-						     "static subprog leftover stack arg slots %x",
-						     bt_stack_arg_mask(bt));
-					return -EFAULT;
-				}
-				if (bt_subprog_exit(bt))
-					return -EFAULT;
-				return 0;
-			}
-		} else if (bpf_is_sync_callback_calling_insn(insn) && idx != subseq_idx - 1) {
-			/* exit from callback subprog to callback-calling helper or
-			 * kfunc call. Use idx/subseq_idx check to discern it from
-			 * straight line code backtracking.
-			 * Unlike the subprog call handling above, we shouldn't
-			 * propagate precision of r1-r5 (if any requested), as they are
-			 * not actually arguments passed directly to callback subprogs
-			 */
-			if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) {
-				verifier_bug(env, "callback unexpected regs %x",
-					     bt_reg_mask(bt));
-				return -EFAULT;
-			}
-			if (!bt_stack_empty(bt)) {
-				bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-						   bt_stack_mask(bt));
-				verifier_bug(env, "callback leftover stack slots %s",
-					     env->tmp_str_buf);
-				return -EFAULT;
-			}
-			/* clear r1-r5 in callback subprog's mask */
-			for (i = BPF_REG_1; i <= BPF_REG_5; i++)
-				bt_clear_reg(bt, i);
-			if (bt_subprog_exit(bt))
-				return -EFAULT;
-			return 0;
-		} else if (opcode == BPF_CALL) {
-			/* kfunc with imm==0 is invalid and fixup_kfunc_call will
-			 * catch this error later. Make backtracking conservative
-			 * with ENOTSUPP.
-			 */
-			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && insn->imm == 0)
-				return -ENOTSUPP;
-			/* regular helper call sets R0 */
-			bt_clear_reg(bt, BPF_REG_0);
-			/* kfunc might also set R2 */
-			bt_clear_reg(bt, BPF_REG_2);
-			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
-				/* if backtracking was looking for registers R1-R5
-				 * they should have been found already.
-				 */
-				verifier_bug(env, "backtracking call unexpected regs %x",
-					     bt_reg_mask(bt));
-				return -EFAULT;
-			}
-			if (insn->src_reg == BPF_REG_0 && insn->imm == BPF_FUNC_tail_call
-			    && subseq_idx - idx != 1) {
-				if (bt_subprog_enter(bt))
-					return -EFAULT;
-			}
-		} else if (opcode == BPF_EXIT) {
-			bool from_subprog_call, r0_precise, r2_precise;
-
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate the precision of the
-			 * return registers, if necessary. In the latter we
-			 * never do that.
-			 */
-			from_subprog_call = subseq_idx - 1 >= 0 &&
-					    (bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) ||
-					     bpf_is_callx(&env->prog->insnsi[subseq_idx - 1]));
-
-			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
-			r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
-
-			/* Backtracking to a nested function call, 'idx' is a part of
-			 * the inner frame 'subseq_idx' is a part of the outer frame.
-			 * In case of a regular function call, the callee defines the
-			 * return registers R0 and R2, so clear them before checking
-			 * that instructions giving precision to registers R1-R5 have
-			 * been found already. R2 is also an argument register, hence
-			 * it has to be cleared before that check.
-			 * In case of a callback from bpf_loop(), R{1,4} in the calling
-			 * frame would be set as precise and that is correct, and R2
-			 * might be a precise helper argument as well, so leave it set.
-			 */
-			bt_clear_reg(bt, BPF_REG_0);
-			if (from_subprog_call)
-				bt_clear_reg(bt, BPF_REG_2);
-			if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
-				verifier_bug(env, "backtracking exit unexpected regs %x",
-					     bt_reg_mask(bt));
-				return -EFAULT;
-			}
-
-			if (bt_subprog_enter(bt))
-				return -EFAULT;
-
-			if (r0_precise)
-				bt_set_reg(bt, BPF_REG_0);
-			if (r2_precise)
-				bt_set_reg(bt, BPF_REG_2);
-			/* r6-r9 and stack slots will stay set in caller frame
-			 * bitmasks until we return back from callee(s)
-			 */
-			return 0;
-		} else if (BPF_SRC(insn->code) == BPF_X) {
-			if (!bt_is_reg_set(bt, dreg) && !bt_is_reg_set(bt, sreg))
-				return 0;
-			/* dreg <cond> sreg
-			 * Both dreg and sreg need precision before
-			 * this insn. If only sreg was marked precise
-			 * before it would be equally necessary to
-			 * propagate it to dreg.
-			 */
-			if (!hist || !(hist->flags & INSN_F_SRC_REG_STACK))
-				bt_set_reg(bt, sreg);
-			if (!hist || !(hist->flags & INSN_F_DST_REG_STACK))
-				bt_set_reg(bt, dreg);
-		} else if (BPF_SRC(insn->code) == BPF_K) {
-			 /* dreg <cond> K
-			  * Only dreg still needs precision before
-			  * this insn, so for the K-based conditional
-			  * there is nothing new to be marked.
-			  */
-		}
-	} else if (class == BPF_LD) {
-		/* It's ld_imm64 or ld_abs or ld_ind.
-		 * For ld_imm64 no further tracking of precision
-		 * into parent is necessary
-		 */
-		if (mode == BPF_IMM) {
-			bt_clear_reg(bt, dreg);
-			return 0;
-		}
-		/*
-		 * BPF_{IND,ABS} are modelled as two branches:
-		 * - fallthrough;
-		 * - implicit subprogram exit.
-		 * It is necessary to switch current frame if
-		 * implicit subprogram exit branch is backtracked.
-		 */
-		if (mode == BPF_IND || mode == BPF_ABS) {
-			if (bt_is_reg_set(bt, dreg))
-				return -ENOTSUPP;
-			if (subseq_idx != idx + 1)
-				if (bt_subprog_enter(bt))
-					return -EFAULT;
-			return 0;
-		}
-	}
-	/* Propagate precision marks to linked registers, to account for
-	 * registers marked as precise in this function.
-	 */
-	bpf_bt_sync_linked_regs(bt, hist);
-	return 0;
-}
-
-/* the scalar precision tracking algorithm:
- * . at the start all registers have precise=false.
- * . scalar ranges are tracked as normal through alu and jmp insns.
- * . once precise value of the scalar register is used in:
- *   .  ptr + scalar alu
- *   . if (scalar cond K|scalar)
- *   .  helper_call(.., scalar, ...) where ARG_CONST is expected
- *   backtrack through the verifier states and mark all registers and
- *   stack slots with spilled constants that these scalar registers
- *   should be precise.
- * . during state pruning two registers (or spilled stack slots)
- *   are equivalent if both are not precise.
- *
- * Note the verifier cannot simply walk register parentage chain,
- * since many different registers and stack slots could have been
- * used to compute single precise scalar.
- *
- * The approach of starting with precise=true for all registers and then
- * backtrack to mark a register as not precise when the verifier detects
- * that program doesn't care about specific value (e.g., when helper
- * takes register as ARG_ANYTHING parameter) is not safe.
- *
- * It's ok to walk single parentage chain of the verifier states.
- * It's possible that this backtracking will go all the way till 1st insn.
- * All other branches will be explored for needing precision later.
- *
- * The backtracking needs to deal with cases like:
- *   R8=map_value(id=0,off=0,ks=4,vs=1952,imm=0) R9_w=map_value(id=0,off=40,ks=4,vs=1952,imm=0)
- * r9 -= r8
- * r5 = r9
- * if r5 > 0x79f goto pc+7
- *    R5_w=inv(id=0,umax_value=1951,var_off=(0x0; 0x7ff))
- * r5 += 1
- * ...
- * call bpf_perf_event_output#25
- *   where .arg5_type = ARG_MEM_SIZE_OR_ZERO
- *
- * and this case:
- * r6 = 1
- * call foo // uses callee's r6 inside to compute r0
- * r0 += r6
- * if r0 == 0 goto
- *
- * to track above reg_mask/stack_mask needs to be independent for each frame.
- *
- * Also if parent's curframe > frame where backtracking started,
- * the verifier need to mark registers in both frames, otherwise callees
- * may incorrectly prune callers. This is similar to
- * commit 7640ead93924 ("bpf: verifier: make sure callees don't prune with caller differences")
- *
- * For now backtracking falls back into conservative marking.
- */
-void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
-				 struct bpf_verifier_state *st)
-{
-	struct bpf_func_state *func;
-	struct bpf_reg_state *reg;
-	int i, j;
-
-	if (env->log.level & BPF_LOG_LEVEL2) {
-		verbose(env, "mark_precise: frame%d: falling back to forcing all scalars precise\n",
-			st->curframe);
-	}
-
-	/* big hammer: mark all scalars precise in this path.
-	 * pop_stack may still get !precise scalars.
-	 * We also skip current state and go straight to first parent state,
-	 * because precision markings in current non-checkpointed state are
-	 * not needed. See why in the comment in __mark_chain_precision below.
-	 */
-	for (st = st->parent; st; st = st->parent) {
-		for (i = 0; i <= st->curframe; i++) {
-			func = st->frame[i];
-			for (j = 0; j < BPF_REG_FP; j++) {
-				reg = &func->regs[j];
-				if (reg->type != SCALAR_VALUE || reg->precise)
-					continue;
-				reg->precise = true;
-				if (env->log.level & BPF_LOG_LEVEL2) {
-					verbose(env, "force_precise: frame%d: forcing r%d to be precise\n",
-						i, j);
-				}
-			}
-			for (j = 0; j < bpf_stack_nr_slots(func); j++) {
-				struct bpf_stack_state *ss = bpf_stack_slot(func, j);
-
-				if (!bpf_is_spilled_reg(ss))
-					continue;
-				reg = &ss->spilled_ptr;
-				if (reg->type != SCALAR_VALUE || reg->precise)
-					continue;
-				reg->precise = true;
-				if (env->log.level & BPF_LOG_LEVEL2) {
-					verbose(env, "force_precise: frame%d: forcing fp%d to be precise\n",
-						i, -(j + 1) * 8);
-				}
-			}
-			for (j = 0; j < func->out_stack_arg_cnt; j++) {
-				reg = &func->stack_arg_regs[j];
-				if (reg->type != SCALAR_VALUE || reg->precise)
-					continue;
-				reg->precise = true;
-				if (env->log.level & BPF_LOG_LEVEL2) {
-					verbose(env,
-						"force_precise: frame%d: forcing r11%d to be precise\n",
-						i, -(j + 1) * BPF_REG_SIZE);
-				}
-			}
-		}
-	}
-}
-
-/*
- * bpf_mark_chain_precision() backtracks BPF program instruction sequence and
- * chain of verifier states making sure that register *regno* (if regno >= 0)
- * and/or stack slot *spi* (if spi >= 0) are marked as precisely tracked
- * SCALARS, as well as any other registers and slots that contribute to
- * a tracked state of given registers/stack slots, depending on specific BPF
- * assembly instructions (see backtrack_insns() for exact instruction handling
- * logic). This backtracking relies on recorded jmp_history and is able to
- * traverse entire chain of parent states. This process ends only when all the
- * necessary registers/slots and their transitive dependencies are marked as
- * precise.
- *
- * One important and subtle aspect is that precise marks *do not matter* in
- * the currently verified state (current state). It is important to understand
- * why this is the case.
- *
- * First, note that current state is the state that is not yet "checkpointed",
- * i.e., it is not yet put into env->explored_states, and it has no children
- * states as well. It's ephemeral, and can end up either a) being discarded if
- * compatible explored state is found at some point or BPF_EXIT instruction is
- * reached or b) checkpointed and put into env->explored_states, branching out
- * into one or more children states.
- *
- * In the former case, precise markings in current state are completely
- * ignored by state comparison code (see regsafe() for details). Only
- * checkpointed ("old") state precise markings are important, and if old
- * state's register/slot is precise, regsafe() assumes current state's
- * register/slot as precise and checks value ranges exactly and precisely. If
- * states turn out to be compatible, current state's necessary precise
- * markings and any required parent states' precise markings are enforced
- * after the fact with propagate_precision() logic, after the fact. But it's
- * important to realize that in this case, even after marking current state
- * registers/slots as precise, we immediately discard current state. So what
- * actually matters is any of the precise markings propagated into current
- * state's parent states, which are always checkpointed (due to b) case above).
- * As such, for scenario a) it doesn't matter if current state has precise
- * markings set or not.
- *
- * Now, for the scenario b), checkpointing and forking into child(ren)
- * state(s). Note that before current state gets to checkpointing step, any
- * processed instruction always assumes precise SCALAR register/slot
- * knowledge: if precise value or range is useful to prune jump branch, BPF
- * verifier takes this opportunity enthusiastically. Similarly, when
- * register's value is used to calculate offset or memory address, exact
- * knowledge of SCALAR range is assumed, checked, and enforced. So, similar to
- * what we mentioned above about state comparison ignoring precise markings
- * during state comparison, BPF verifier ignores and also assumes precise
- * markings *at will* during instruction verification process. But as verifier
- * assumes precision, it also propagates any precision dependencies across
- * parent states, which are not yet finalized, so can be further restricted
- * based on new knowledge gained from restrictions enforced by their children
- * states. This is so that once those parent states are finalized, i.e., when
- * they have no more active children state, state comparison logic in
- * is_state_visited() would enforce strict and precise SCALAR ranges, if
- * required for correctness.
- *
- * To build a bit more intuition, note also that once a state is checkpointed,
- * the path we took to get to that state is not important. This is crucial
- * property for state pruning. When state is checkpointed and finalized at
- * some instruction index, it can be correctly and safely used to "short
- * circuit" any *compatible* state that reaches exactly the same instruction
- * index. I.e., if we jumped to that instruction from a completely different
- * code path than original finalized state was derived from, it doesn't
- * matter, current state can be discarded because from that instruction
- * forward having a compatible state will ensure we will safely reach the
- * exit. States describe preconditions for further exploration, but completely
- * forget the history of how we got here.
- *
- * This also means that even if we needed precise SCALAR range to get to
- * finalized state, but from that point forward *that same* SCALAR register is
- * never used in a precise context (i.e., it's precise value is not needed for
- * correctness), it's correct and safe to mark such register as "imprecise"
- * (i.e., precise marking set to false). This is what we rely on when we do
- * not set precise marking in current state. If no child state requires
- * precision for any given SCALAR register, it's safe to dictate that it can
- * be imprecise. If any child state does require this register to be precise,
- * we'll mark it precise later retroactively during precise markings
- * propagation from child state to parent states.
- *
- * Skipping precise marking setting in current state is a mild version of
- * relying on the above observation. But we can utilize this property even
- * more aggressively by proactively forgetting any precise marking in the
- * current state (which we inherited from the parent state), right before we
- * checkpoint it and branch off into new child state. This is done by
- * mark_all_scalars_imprecise() to hopefully get more permissive and generic
- * finalized states which help in short circuiting more future states.
- */
-int bpf_mark_chain_precision(struct bpf_verifier_env *env,
-			    struct bpf_verifier_state *starting_state,
-			    int regno,
-			    bool *changed)
-{
-	struct bpf_verifier_state *st = starting_state;
-	struct backtrack_state *bt = &env->bt;
-	int first_idx = st->first_insn_idx;
-	int last_idx = starting_state->insn_idx;
-	int subseq_idx = -1;
-	struct bpf_func_state *func;
-	bool tmp, skip_first = true;
-	struct bpf_stack_state *ss;
-
-	/* FLOW: there are no precision marks */
-	bt_reset(bt);
-	return 0;
-	struct bpf_reg_state *reg;
-	int i, fr, err;
-
-	if (!env->bpf_capable)
-		return 0;
-
-	changed = changed ?: &tmp;
-	/* set frame number from which we are starting to backtrack */
-	bt_init(bt, starting_state->curframe);
-
-	/* Do sanity checks against current state of register and/or stack
-	 * slot, but don't set precise flag in current state, as precision
-	 * tracking in the current state is unnecessary.
-	 */
-	func = st->frame[bt->frame];
-	if (regno >= 0) {
-		reg = &func->regs[regno];
-		if (reg->type != SCALAR_VALUE) {
-			verifier_bug(env, "backtracking misuse");
-			return -EFAULT;
-		}
-		bt_set_reg(bt, regno);
-	}
-
-	if (bt_empty(bt))
-		return 0;
-
-	for (;;) {
-		DECLARE_BITMAP(mask, 64);
-		u32 history = st->jmp_history_cnt;
-		struct bpf_jmp_history_entry *hist;
-
-		if (env->log.level & BPF_LOG_LEVEL2) {
-			verbose(env, "mark_precise: frame%d: last_idx %d first_idx %d subseq_idx %d \n",
-				bt->frame, last_idx, first_idx, subseq_idx);
-		}
-
-		if (last_idx < 0) {
-			/* Consecutive checkpoints can have no instructions between them. */
-			if (st->parent)
-				goto parent;
-
-			/* we are at the entry into subprog, which
-			 * is expected for global funcs, but only if
-			 * requested precise registers are R1-R5
-			 * (which are global func's input arguments)
-			 */
-			if (st->curframe == 0 &&
-			    st->frame[0]->subprogno > 0 &&
-			    st->frame[0]->callsite == BPF_MAIN_FUNC &&
-			    bt_stack_empty(bt) &&
-			    (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) == 0) {
-				bitmap_from_u64(mask, bt_reg_mask(bt));
-				for_each_set_bit(i, mask, 32) {
-					reg = &st->frame[0]->regs[i];
-					bt_clear_reg(bt, i);
-					if (reg->type == SCALAR_VALUE) {
-						reg->precise = true;
-						*changed = true;
-					}
-				}
-				return 0;
-			}
-
-			bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
-			verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %s",
-				     st->frame[0]->subprogno, bt_reg_mask(bt), env->tmp_str_buf);
-			return -EFAULT;
-		}
-
-		for (i = last_idx;;) {
-			if (skip_first) {
-				err = 0;
-				skip_first = false;
-			} else {
-				hist = get_jmp_hist_entry(st, history, i);
-				err = backtrack_insn(env, i, subseq_idx, hist, bt);
-			}
-			if (err == -ENOTSUPP) {
-				bpf_mark_all_scalars_precise(env, starting_state);
-				bt_reset(bt);
-				return 0;
-			} else if (err) {
-				return err;
-			}
-			if (bt_empty(bt))
-				/* Found assignment(s) into tracked register in this state.
-				 * Since this state is already marked, just return.
-				 * Nothing to be tracked further in the parent state.
-				 */
-				return 0;
-			subseq_idx = i;
-			i = get_prev_insn_idx(st, i, &history);
-			if (i == -ENOENT)
-				break;
-			if (i >= env->prog->len) {
-				/* This can happen if backtracking reached insn 0
-				 * and there are still reg_mask or stack_mask
-				 * to backtrack.
-				 * It means the backtracking missed the spot where
-				 * particular register was initialized with a constant.
-				 */
-				verifier_bug(env, "backtracking idx %d", i);
-				return -EFAULT;
-			}
-		}
-parent:
-		st = st->parent;
-		if (!st)
-			break;
-
-		for (fr = bt->frame; fr >= 0; fr--) {
-			func = st->frame[fr];
-			bitmap_from_u64(mask, bt_frame_reg_mask(bt, fr));
-			for_each_set_bit(i, mask, 32) {
-				reg = &func->regs[i];
-				if (reg->type != SCALAR_VALUE) {
-					bt_clear_frame_reg(bt, fr, i);
-					continue;
-				}
-				if (reg->precise) {
-					bt_clear_frame_reg(bt, fr, i);
-				} else {
-					reg->precise = true;
-					*changed = true;
-				}
-			}
-
-			for_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {
-				if (verifier_bug_if(i >= bpf_stack_nr_slots(func),
-						    env, "stack slot %d, total slots %d",
-						    i, bpf_stack_nr_slots(func)))
-					return -EFAULT;
-
-				ss = bpf_stack_slot(func, i);
-				if (!bpf_is_spilled_scalar_reg(ss)) {
-					bt_clear_frame_slot(bt, fr, i);
-					continue;
-				}
-				reg = &ss->spilled_ptr;
-				if (reg->precise) {
-					bt_clear_frame_slot(bt, fr, i);
-				} else {
-					reg->precise = true;
-					*changed = true;
-				}
-			}
-			for (i = 0; i < func->out_stack_arg_cnt; i++) {
-				if (!bt_is_frame_stack_arg_slot_set(bt, fr, i))
-					continue;
-				reg = &func->stack_arg_regs[i];
-				if (reg->type != SCALAR_VALUE || reg->precise) {
-					bt_clear_frame_stack_arg_slot(bt, fr, i);
-				} else {
-					reg->precise = true;
-					*changed = true;
-				}
-			}
-			if (env->log.level & BPF_LOG_LEVEL2) {
-				fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-					     bt_frame_reg_mask(bt, fr));
-				verbose(env, "mark_precise: frame%d: parent state regs=%s ",
-					fr, env->tmp_str_buf);
-				bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
-					       bt_frame_stack_mask(bt, fr));
-				verbose(env, "stack=%s: ", env->tmp_str_buf);
-				print_verifier_state(env, st, fr, true);
-			}
-		}
-
-		if (bt_empty(bt))
-			return 0;
-
-		subseq_idx = first_idx;
-		last_idx = st->last_insn_idx;
-		first_idx = st->first_insn_idx;
-	}
-
-	/* if we still have requested precise regs or slots, we missed
-	 * something (e.g., stack access through non-r10 register), so
-	 * fallback to marking all precise
-	 */
-	if (!bt_empty(bt)) {
-		bpf_mark_all_scalars_precise(env, starting_state);
-		bt_reset(bt);
-	}
-
-	return 0;
-}
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 00cb219e24cd..db496253bf62 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -637,8 +637,6 @@ static void print_reg_state(struct bpf_verifier_env *env,
 	const char *sep = "";
 
 	t = reg->type;
-	if (t == SCALAR_VALUE && reg->precise)
-		verbose(env, "P");
 	if (t == SCALAR_VALUE && tnum_is_const(reg->var_off)) {
 		bpf_verbose_snum(env, reg->var_off.value);
 		return;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 677b64a63be4..de7592424056 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1912,7 +1912,7 @@ int bpf_init_scev(struct bpf_verifier_env *env)
 		return -ENOMEM;
 	env->scev = scev;
 	bpf_mark_reg_known_scalar(&scev->zero_reg, 0);
-	bpf_mark_reg_unknown_imprecise(&scev->unknown_reg);
+	bpf_mark_reg_unknown(&scev->unknown_reg);
 	/* A power of two for hash & (exprs_ht_cnt - 1)` indexing. */
 	scev->exprs_ht_cnt = roundup_pow_of_two(max(256U, DIV_ROUND_UP(env->prog->len, 4)));
 	scev->exprs_ht = kvcalloc(scev->exprs_ht_cnt, sizeof(*scev->exprs_ht), GFP_KERNEL_ACCOUNT);
@@ -3256,30 +3256,6 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
 		bounds->r64 = r64;
 }
 
-static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 id,
-			     unsigned long *precise_regs)
-{
-	struct scev *scev = env->scev;
-	struct bpf_reg_state *reg;
-	u32 r, order;
-
-	scev->stack_sz = 0;
-	expr_stack_push(scev, id);
-	while (expr_next(scev, &id, &order)) {
-		if (order & DEPTH_LIMIT) {
-			verifier_bug(env, "scev precision marking exceeds expression depth limit");
-			return -EFAULT;
-		}
-		if (!(order & PRE) || !is_reg(scev, id, &r))
-			continue;
-		/* Synthetic values have no spilled register to mark precise. */
-		reg = scev_reg_for_linear(st, r) ?: scev_reg_for_spill32(st, r);
-		if (reg && reg->type == SCALAR_VALUE)
-			__set_bit(r, precise_regs);
-	}
-	return 0;
-}
-
 /*
  * Does @id have no bit that register @r doesn't have at the head of the loop? 'x & e' and the
  * low bits of x lose bits only. 'x >> e', 'x / e' and 'x % e' are not greater than x as well, but
@@ -3434,17 +3410,11 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	 * which iterations it can be taken. The loop ends, at the latest, in an iteration
 	 * in which no back edge can be taken.
 	 */
-	bitmap_zero(iters->precise_regs, REGS_NUM);
 	for (i = 0; i < loop->backedges_cnt; i++) {
 		err = count_backedge(env, cur_func, insn_idx, &loop->backedges[i],
 				     i ? &tmp : iters, &latch, &until);
 		if (err <= 0)
 			return err;
-		err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs) ?:
-		      mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs) ?:
-		      mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
-		if (err)
-			return err;
 		if (!i) {
 			false_until = until;
 			first = latch;
@@ -3694,16 +3664,6 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 	int insn_idx = st->insn_idx;
 	int err;
 
-	for_each_set_bit(r, iters->precise_regs, REGS_NUM) {
-		if (r < MAX_BPF_REG)
-			bpf_bt_set_frame_reg(&env->bt, st->curframe, r);
-		else
-			bpf_bt_set_frame_slot(&env->bt, st->curframe, r - MAX_BPF_REG);
-	}
-	err = bpf_mark_chain_precision(env, st, -1, NULL);
-	if (err)
-		return err;
-
 	header_env = find_header_env(scev, insn_idx);
 	for (r = 0; r < REGS_NUM; r++) {
 		if (!scev_reg_alive(env, st, r))
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 343f29b52d79..5935e16730fd 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -5,8 +5,6 @@
 #include <linux/cnum.h>
 #include <linux/filter.h>
 
-#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-
 /* check %cur's range satisfies %old's */
 static bool range_within(const struct bpf_reg_state *old,
 			 const struct bpf_reg_state *cur)
@@ -182,10 +180,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
 				if (lo_live && stype == STACK_SPILL) {
 					if (spill->type != SCALAR_VALUE)
 						continue;
-					/*
-					 * Can't replace with STACK_ZERO, because
-					 * that requires bpf_mark_chain_precision().
-					 */
+					/* keep what is known about the low half */
 					if (bpf_register_is_null(spill))
 						continue;
 					for (j = 0; j < 4; j++) {
@@ -281,8 +276,6 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 			return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
 			       check_scalar_ids(rold->id, rcur->id, idmap);
 		}
-		if (!rold->precise && exact == NOT_EXACT)
-			return true;
 		/*
 		 * Linked register tracking uses rold->id to detect relationships.
 		 * When rold->id == 0, the register is independent and any linking
@@ -394,7 +387,7 @@ static struct bpf_reg_state unbound_reg;
 
 static __init int unbound_reg_init(void)
 {
-	bpf_mark_reg_unknown_imprecise(&unbound_reg);
+	bpf_mark_reg_unknown(&unbound_reg);
 	return 0;
 }
 late_initcall(unbound_reg_init);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 616a69b4cfb5..85dae0feeb1a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -302,11 +302,6 @@ static int arg_from_argno(argno_t a)
 	return -1;
 }
 
-static int arg_idx_from_argno(argno_t a)
-{
-	return arg_from_argno(a) - 1;
-}
-
 /* Normalize helper register numbers and kfunc argument numbers to ABI slots. */
 static u32 arg_slot_from_argno(argno_t a)
 {
@@ -856,15 +851,12 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
 				      : reg->id);
 }
 
-static void __mark_reg_unknown(const struct bpf_verifier_env *env,
-			       struct bpf_reg_state *reg);
-
 static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	if (!env->allow_ptr_leaks)
 		bpf_mark_reg_not_init(env, reg);
 	else
-		__mark_reg_unknown(env, reg);
+		bpf_mark_reg_unknown(reg);
 }
 
 static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
@@ -1754,13 +1746,6 @@ static void free_func_state(struct bpf_func_state *state)
 	kfree(state);
 }
 
-void bpf_clear_jmp_history(struct bpf_verifier_state *state)
-{
-	kfree(state->jmp_history);
-	state->jmp_history = NULL;
-	state->jmp_history_cnt = 0;
-}
-
 void bpf_free_verifier_state(struct bpf_verifier_state *state,
 			    bool free_self)
 {
@@ -1771,7 +1756,6 @@ void bpf_free_verifier_state(struct bpf_verifier_state *state,
 		state->frame[i] = NULL;
 	}
 	kfree(state->refs);
-	bpf_clear_jmp_history(state);
 	if (free_self)
 		kfree(state);
 }
@@ -1798,13 +1782,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	struct bpf_func_state *dst;
 	int i, err;
 
-	dst_state->jmp_history = copy_array(dst_state->jmp_history, src->jmp_history,
-					  src->jmp_history_cnt, sizeof(*dst_state->jmp_history),
-					  GFP_KERNEL_ACCOUNT);
-	if (!dst_state->jmp_history)
-		return -ENOMEM;
-	dst_state->jmp_history_cnt = src->jmp_history_cnt;
-
 	/* if dst has more stack frames then src frame, free them, this is also
 	 * necessary in case of exceptional exits using bpf_throw.
 	 */
@@ -1823,8 +1800,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	dst_state->flow_entry = src->flow_entry;
 	dst_state->insn_idx = src->insn_idx;
 	dst_state->curframe = src->curframe;
-	dst_state->parent = src->parent;
-	dst_state->first_insn_idx = src->first_insn_idx;
 	dst_state->last_insn_idx = src->last_insn_idx;
 	for (i = 0; i <= src->curframe; i++) {
 		dst = dst_state->frame[i];
@@ -2094,14 +2069,10 @@ static void __mark_reg_known_zero(struct bpf_reg_state *reg)
 	__mark_reg_known(reg, 0);
 }
 
-static void __mark_reg_const_zero(const struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static void __mark_reg_const_zero(struct bpf_reg_state *reg)
 {
 	__mark_reg_known(reg, 0);
 	reg->type = SCALAR_VALUE;
-	/* all scalars are assumed imprecise initially (unless unprivileged,
-	 * in which case everything is forced to be precise)
-	 */
-	reg->precise = !env->bpf_capable;
 }
 
 static void mark_reg_known_zero(struct bpf_verifier_env *env,
@@ -2434,7 +2405,7 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env,
 }
 
 /* Mark a register as having a completely unknown (scalar) value. */
-void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
+void bpf_mark_reg_unknown(struct bpf_reg_state *reg)
 {
 	memset(reg, 0, sizeof(*reg));
 	reg->type = SCALAR_VALUE;
@@ -2443,20 +2414,10 @@ void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
 	reg_step_reset(reg);
 }
 
-/* Mark a register as having a completely unknown (scalar) value,
- * initialize .precise as true when not bpf capable.
- */
-static void __mark_reg_unknown(const struct bpf_verifier_env *env,
-			       struct bpf_reg_state *reg)
-{
-	bpf_mark_reg_unknown_imprecise(reg);
-	reg->precise = !env->bpf_capable;
-}
-
 static void mark_reg_unknown(struct bpf_verifier_env *env,
 			     struct bpf_reg_state *regs, u32 regno)
 {
-	__mark_reg_unknown(env, regs + regno);
+	bpf_mark_reg_unknown(regs + regno);
 }
 
 static int __mark_reg_s32_range(struct bpf_verifier_env *env,
@@ -2482,7 +2443,7 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env,
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg)
 {
-	__mark_reg_unknown(env, reg);
+	bpf_mark_reg_unknown(reg);
 	reg->type = NOT_INIT;
 }
 
@@ -3628,19 +3589,7 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
 	env->insn_aux_data[idx].non_stack_access = true;
 }
 
-/* Layout of one packed linked register in the jump history, see linked_regs_pack() */
-#define LR_FRAMENO_BITS	5
-#define LR_INDEX_BITS	10
-#define LR_FRAMENO_MASK	((1u << LR_FRAMENO_BITS) - 1)
-#define LR_IS_REG	BIT(LR_FRAMENO_BITS)
-#define LR_INDEX_OFF	(LR_FRAMENO_BITS + 1)
-#define LR_INDEX_MASK	((1u << LR_INDEX_BITS) - 1)
-#define LINKED_REGS_MAX	BPF_LINKED_REGS_MAX
-
-static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
-static_assert(MAX_BPF_REG <= (1 << LR_INDEX_BITS));
-static_assert(MAX_BPF_STACK_SLOTS <= (1 << LR_INDEX_BITS));
-static_assert(LR_INDEX_OFF + LR_INDEX_BITS <= 16);
+#define LINKED_REGS_MAX	5
 
 struct linked_reg {
 	u8 frameno;
@@ -3664,38 +3613,6 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s)
 	return NULL;
 }
 
-/*
- * Pack linked registers for a jump history entry, one u16 each:
- * - 5 bits frameno
- * - 1 bit  is_reg
- * - 10 bits register or stack slot index
- */
-static void linked_regs_pack(const struct linked_regs *s, u16 *packed)
-{
-	int i;
-
-	for (i = 0; i < s->cnt; ++i) {
-		const struct linked_reg *e = &s->entries[i];
-
-		packed[i] = e->frameno | (e->is_reg ? LR_IS_REG : 0) | (e->spi << LR_INDEX_OFF);
-	}
-}
-
-static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)
-{
-	int i;
-
-	s->cnt = hist->linked_regs_cnt;
-	for (i = 0; i < s->cnt; ++i) {
-		struct linked_reg *e = &s->entries[i];
-		u16 packed = hist->linked_regs[i];
-
-		e->frameno = packed & LR_FRAMENO_MASK;
-		e->is_reg  = packed & LR_IS_REG;
-		e->spi     = (packed >> LR_INDEX_OFF) & LR_INDEX_MASK;
-	}
-}
-
 const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn)
 {
 	const struct btf_type *func;
@@ -3725,56 +3642,6 @@ void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	print_bpf_insn(&cbs, insn, env->allow_ptr_leaks);
 }
 
-/* If any register R in hist->linked_regs is marked as precise in bt,
- * do bt_set_frame_{reg,slot}(bt, R) for all registers in hist->linked_regs.
- */
-void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist)
-{
-	struct linked_regs linked_regs;
-	bool some_precise = false;
-	int i;
-
-	if (!hist || !hist->linked_regs_cnt)
-		return;
-
-	linked_regs_unpack(hist, &linked_regs);
-	for (i = 0; i < linked_regs.cnt; ++i) {
-		struct linked_reg *e = &linked_regs.entries[i];
-
-		if ((e->is_reg && bt_is_frame_reg_set(bt, e->frameno, e->regno)) ||
-		    (!e->is_reg && bt_is_frame_slot_set(bt, e->frameno, e->spi))) {
-			some_precise = true;
-			break;
-		}
-	}
-
-	if (!some_precise)
-		return;
-
-	for (i = 0; i < linked_regs.cnt; ++i) {
-		struct linked_reg *e = &linked_regs.entries[i];
-
-		if (e->is_reg)
-			bpf_bt_set_frame_reg(bt, e->frameno, e->regno);
-		else
-			bpf_bt_set_frame_slot(bt, e->frameno, e->spi);
-	}
-}
-
-int mark_chain_precision(struct bpf_verifier_env *env, int regno)
-{
-	return bpf_mark_chain_precision(env, env->cur_state, regno, NULL);
-}
-
-/* mark_chain_precision_batch() assumes that env->bt is set in the caller to
- * desired reg and stack masks across all relevant frames
- */
-static int mark_chain_precision_batch(struct bpf_verifier_env *env,
-				      struct bpf_verifier_state *starting_state)
-{
-	return bpf_mark_chain_precision(env, starting_state, -1, NULL);
-}
-
 /* check if register is a constant scalar value */
 static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32)
 {
@@ -3905,8 +3772,6 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 	int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	struct bpf_reg_state *reg = NULL;
-	int insn_flags = INSN_F_STACK_ACCESS;
-	int hist_spi = spi, hist_frame = state->frameno;
 	struct bpf_stack_state *ss = bpf_stack_slot(state, spi);
 
 	/*
@@ -3996,28 +3861,14 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 
 		/* when we zero initialize stack slots mark them as such */
 		if ((reg && bpf_register_is_null(reg)) ||
-		    (!reg && is_bpf_st_mem(insn) && insn->imm == 0)) {
-			/* STACK_ZERO case happened because register spill
-			 * wasn't properly aligned at the stack slot boundary,
-			 * so it's not a register spill anymore; force
-			 * originating register to be precise to make
-			 * STACK_ZERO correct for subsequent states
-			 */
-			err = mark_chain_precision(env, value_regno);
-			if (err)
-				return err;
+		    (!reg && is_bpf_st_mem(insn) && insn->imm == 0))
 			type = STACK_ZERO;
-		}
 
 		/* Mark slots affected by this stack write. */
 		for (i = 0; i < size; i++)
 			ss->slot_type[(slot - i) % BPF_REG_SIZE] = type;
-		insn_flags = 0; /* not a register spill */
 	}
 
-	if (insn_flags)
-		return bpf_push_jmp_history(env, env->cur_state, insn_flags,
-					    hist_spi, hist_frame, NULL, 0);
 	return 0;
 }
 
@@ -4051,10 +3902,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 	struct bpf_reg_state *value_reg = NULL;
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	bool writing_zero = false;
-	/* set if the fact that we're writing a zero is used to let any
-	 * stack slots remain STACK_ZERO
-	 */
-	bool zero_used = false;
 
 	cur = env->cur_state->frame[env->cur_state->curframe];
 	min_off = reg_smin(ptr_reg) + off;
@@ -4105,10 +3952,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 		if (writing_zero && *stype == STACK_SPILL && bpf_is_spilled_scalar_reg(ss)) {
 			struct bpf_reg_state *spill_reg = &ss->spilled_ptr;
 
-			if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0) {
-				zero_used = true;
+			if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0)
 				continue;
-			}
 		}
 
 		/*
@@ -4121,10 +3966,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 
 		/* Update the slot type. */
 		new_type = STACK_MISC;
-		if (writing_zero && *stype == STACK_ZERO) {
+		if (writing_zero && *stype == STACK_ZERO)
 			new_type = STACK_ZERO;
-			zero_used = true;
-		}
 		/* If the slot is STACK_INVALID, we check whether it's OK to
 		 * pretend that it will be initialized by this write. The slot
 		 * might not actually be written to, and so if we mark it as
@@ -4142,12 +3985,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
 		}
 		*stype = new_type;
 	}
-	if (zero_used) {
-		/* backtracking doesn't work for STACK_ZERO yet. */
-		err = mark_chain_precision(env, value_regno);
-		if (err)
-			return err;
-	}
 	bpf_diag_record_scrub_stack(env, state, min_off, max_off,
 				    BPF_DIAG_MOD_VAR_WRITE);
 	return 0;
@@ -4162,21 +3999,18 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
  * read.
  *
  * STACK_SPILL bytes backed by spilled scalar const zeroes are also considered
- * zero bytes. In that case, mark the contributing stack slots precise so
- * pruning cannot reuse a zero-spill state for a later non-zero spill state.
- *
- * Returns an error if precision backtracking fails.
+ * zero bytes.
  */
-static int mark_reg_stack_read(struct bpf_verifier_env *env,
-			       /* func where src register points to */
-			       struct bpf_func_state *ptr_state,
-			       int min_off, int max_off, int dst_regno)
+static void mark_reg_stack_read(struct bpf_verifier_env *env,
+				/* func where src register points to */
+				struct bpf_func_state *ptr_state,
+				int min_off, int max_off, int dst_regno)
 {
 	struct bpf_verifier_state *vstate = env->cur_state;
 	struct bpf_func_state *state = vstate->frame[vstate->curframe];
 	int i, slot, spi;
 	u8 *stype;
-	int zeros = 0, zero_spills = 0;
+	int zeros = 0;
 
 	for (i = min_off; i < max_off; i++) {
 		slot = -i - 1;
@@ -4189,7 +4023,6 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
 		}
 		if (stype[slot % BPF_REG_SIZE] == STACK_SPILL &&
 		    bpf_register_is_null(&bpf_stack_slot(ptr_state, spi)->spilled_ptr)) {
-			zero_spills++;
 			zeros++;
 			continue;
 		}
@@ -4199,23 +4032,11 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
 		/* Any access_size read into register is zero extended,
 		 * so the whole register == const_zero.
 		 */
-		__mark_reg_const_zero(env, &state->regs[dst_regno]);
-		if (zero_spills) {
-			for (i = min_off; i < max_off; i++) {
-				slot = -i - 1;
-				spi = slot / BPF_REG_SIZE;
-				stype = bpf_stack_slot(ptr_state, spi)->slot_type;
-				if (stype[slot % BPF_REG_SIZE] == STACK_SPILL)
-					bpf_bt_set_frame_slot(&env->bt, ptr_state->frameno, spi);
-			}
-			return mark_chain_precision_batch(env, env->cur_state);
-		}
+		__mark_reg_const_zero(&state->regs[dst_regno]);
 	} else {
 		/* have read misc data from the stack */
 		mark_reg_unknown(env, state->regs, dst_regno);
 	}
-
-	return 0;
 }
 
 static void bpf_diag_stack_read_invalid(struct bpf_verifier_env *env, int off, int i, int size,
@@ -4279,9 +4100,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 	int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;
 	struct bpf_reg_state *reg;
 	u8 *stype, type;
-	int err;
-	int insn_flags = INSN_F_STACK_ACCESS;
-	int hist_spi = spi, hist_frame = reg_state->frameno;
 	struct bpf_stack_state *ss = bpf_stack_slot(reg_state, spi);
 
 	stype = ss->slot_type;
@@ -4356,18 +4174,13 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 
 				if (spill_cnt == size &&
 				    tnum_is_const(reg->var_off) && reg->var_off.value == 0) {
-					__mark_reg_const_zero(env, &state->regs[dst_regno]);
-					/* this IS register fill, so keep insn_flags */
+					__mark_reg_const_zero(&state->regs[dst_regno]);
 				} else if (zero_cnt == size) {
 					/* similarly to mark_reg_stack_read(), preserve zeroes */
-					__mark_reg_const_zero(env, &state->regs[dst_regno]);
-					insn_flags = 0; /* not restoring original register state */
+					__mark_reg_const_zero(&state->regs[dst_regno]);
 				} else {
-					err = mark_reg_stack_read(env, reg_state, off, off + size,
-								  dst_regno);
-					if (err)
-						return err;
-					insn_flags = 0; /* not restoring original register state */
+					mark_reg_stack_read(env, reg_state, off, off + size,
+							    dst_regno);
 				}
 			}
 		} else if (dst_regno >= 0) {
@@ -4378,10 +4191,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 				 */
 				assign_scalar_id_before_mov(env, reg);
 			state->regs[dst_regno] = *reg;
-			/* mark reg as written since spilled pointer state likely
-			 * has its liveness marks cleared by is_state_visited()
-			 * which resets stack/reg liveness for state transitions
-			 */
 		} else if (__is_pointer_value(env->allow_ptr_leaks, reg)) {
 			/* If dst_regno==-1, the caller is asking us whether
 			 * it is acceptable to use this value as a SCALAR_VALUE
@@ -4412,16 +4221,9 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 			}
 			return -EACCES;
 		}
-		if (dst_regno >= 0) {
-			err = mark_reg_stack_read(env, reg_state, off, off + size, dst_regno);
-			if (err)
-				return err;
-		}
-		insn_flags = 0; /* we are not restoring spilled register */
+		if (dst_regno >= 0)
+			mark_reg_stack_read(env, reg_state, off, off + size, dst_regno);
 	}
-	if (insn_flags)
-		return bpf_push_jmp_history(env, env->cur_state, insn_flags,
-					    hist_spi, hist_frame, NULL, 0);
 	return 0;
 }
 
@@ -4470,10 +4272,7 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg
 
 	min_off = reg_smin(reg) + off;
 	max_off = reg_smax(reg) + off;
-	err = mark_reg_stack_read(env, ptr_state, min_off, max_off + size,
-				  dst_regno);
-	if (err)
-		return err;
+	mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno);
 	return 0;
 }
 
@@ -4661,8 +4460,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
 	}
 	bpf_diag_mod_end(env);
-	return bpf_push_jmp_history(env, env->cur_state,
-				    INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
+	return 0;
 }
 
 /*
@@ -4700,26 +4498,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
 	bpf_diag_mod_begin(env, &cur->regs[dst_regno], arg, BPF_DIAG_MOD_WRITE);
 	cur->regs[dst_regno] = *arg;
 	bpf_diag_mod_end(env);
-	return bpf_push_jmp_history(env, env->cur_state,
-				    INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
-}
-
-static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
-{
-	struct bpf_func_state *caller = cur_func(env);
-	int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS;
-
-	bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi);
-	return mark_chain_precision_batch(env, env->cur_state);
-}
-
-static int mark_arg_precision(struct bpf_verifier_env *env, argno_t argno)
-{
-	int regno = reg_from_argno(argno);
-
-	if (regno >= 0)
-		return mark_chain_precision(env, regno);
-	return mark_stack_arg_precision(env, arg_idx_from_argno(argno));
+	return 0;
 }
 
 static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller,
@@ -5184,14 +4963,8 @@ static int check_map_kptr_access(struct bpf_verifier_env *env,
 			return ret;
 	} else if (class == BPF_STX) {
 		val_reg = reg_state(env, value_regno);
-		if (bpf_register_is_null(val_reg)) {
-			/*
-			 * This store is valid only because the scalar is known to be
-			 * zero. Mark it precise so another scalar cannot be pruned
-			 * against this state.
-			 */
-			return mark_chain_precision(env, value_regno);
-		}
+		if (bpf_register_is_null(val_reg))
+			return 0;
 		if (map_kptr_match_type(env, kptr_field, val_reg, value_regno))
 			return -EACCES;
 	} else if (class == BPF_ST) {
@@ -8275,7 +8048,7 @@ static int check_stack_range_initialized(
 		    (ss->spilled_ptr.type == SCALAR_VALUE ||
 		     env->allow_ptr_leaks)) {
 			if (clobber) {
-				__mark_reg_unknown(env, &ss->spilled_ptr);
+				bpf_mark_reg_unknown(&ss->spilled_ptr);
 				for (j = 0; j < BPF_REG_SIZE; j++)
 					scrub_spilled_slot(&ss->slot_type[j]);
 			}
@@ -8473,9 +8246,6 @@ static int check_mem_size_reg(struct bpf_verifier_env *env,
 	if (err && failure)
 		*failure = BPF_MEM_SIZE_FAIL_MEMORY;
 
-	if (!err)
-		err = mark_arg_precision(env, size_argno);
-
 	return err;
 
 size_error:
@@ -8496,7 +8266,7 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
 	int size, err = 0;
 
 	if (bpf_register_is_null(reg))
-		return mark_arg_precision(env, argno);
+		return 0;
 	if (known_memory)
 		*known_memory = true;
 
@@ -8526,9 +8296,6 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
 static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 					argno_t argno, struct ret_mem_desc *ret_mem)
 {
-	int regno = reg_from_argno(argno);
-	int err;
-
 	if (ret_mem->found) {
 		verifier_bug(env, "only one allocation size argument permitted");
 		return -EFAULT;
@@ -8544,13 +8311,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
 		return -EINVAL;
 	}
 
-	if (regno >= 0)
-		err = mark_chain_precision(env, regno);
-	else
-		err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
-	if (err)
-		return err;
-
 	ret_mem->size = reg->var_off.value;
 	ret_mem->found = true;
 
@@ -8560,9 +8320,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
 static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 			     argno_t argno, struct bpf_call_arg_meta *meta)
 {
-	int regno = reg_from_argno(argno);
-	int err;
-
 	if (meta->arg_constant.found) {
 		verifier_bug(env, "only one constant argument permitted");
 		return -EFAULT;
@@ -8573,13 +8330,6 @@ static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state
 		return -EINVAL;
 	}
 
-	if (regno >= 0)
-		err = mark_chain_precision(env, regno);
-	else
-		err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
-	if (err < 0)
-		return err;
-
 	meta->arg_constant.found = true;
 	meta->arg_constant.value = reg->var_off.value;
 
@@ -9322,7 +9072,7 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 
 	/* mark current iter state as drained and assume returned NULL */
 	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
-	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
+	__mark_reg_const_zero(&cur_fr->regs[BPF_REG_0]);
 
 	return 0;
 }
@@ -10009,7 +9759,7 @@ static int get_max_map_key(struct bpf_verifier_env *env,
 	int spill_size = 0;
 	int zero_size = 0;
 	int stack_off;
-	int i, err;
+	int i;
 	u8 *stype;
 
 	if (!env->bpf_capable)
@@ -10051,15 +9801,6 @@ static int get_max_map_key(struct bpf_verifier_env *env,
 		/* The lookup can return NULL: nothing depends on the key */
 		return -EOPNOTSUPP;
 
-	/*
-	 * We are relying on the range of the key. So mark as precise
-	 * to prevent pruning on it.
-	 */
-	bpf_bt_set_frame_slot(&env->bt, key->frameno, spi);
-	err = mark_chain_precision_batch(env, env->cur_state);
-	if (err < 0)
-		return err;
-
 	*value = reg_umax(reg);
 	return 0;
 }
@@ -10177,12 +9918,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, u32 slot, u32 p
 	if (arg_type_is_raw_mem(arg_type))
 		meta->arg_raw_mem.mask |= BIT(slot);
 
-	if (bpf_register_is_null(reg) && type_may_be_null(arg_type)) {
-		err = mark_arg_precision(env, argno);
-		if (err)
-			return err;
+	if (bpf_register_is_null(reg) && type_may_be_null(arg_type))
 		return 0;
-	}
 
 	err = check_func_arg_nullability(env, reg, arg, argno, arg_type, meta, insn_idx);
 	if (err)
@@ -12885,11 +12622,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EINVAL;
 		}
 
-		/* we are going to rely on register's precise value */
-		err = mark_chain_precision(env, BPF_REG_0);
-		if (err)
-			return err;
-
 		/* enforce R0 return value range, and bpf_callback_t returns 64bit */
 		if (!retval_range_within(callee->callback_ret_range, r0)) {
 			verbose_invalid_scalar(env, r0, callee->callback_ret_range,
@@ -13055,7 +12787,6 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 	struct bpf_reg_state *reg;
 	struct bpf_map *map = meta->map.ptr;
 	u64 val, max;
-	int err;
 
 	if (func_id != BPF_FUNC_tail_call)
 		return 0;
@@ -13073,9 +12804,6 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		return 0;
 	}
 
-	err = mark_chain_precision(env, BPF_REG_3);
-	if (err)
-		return err;
 	if (bpf_map_key_unseen(aux))
 		bpf_map_key_store(aux, val);
 	else if (!bpf_map_key_poisoned(aux) &&
@@ -13218,45 +12946,27 @@ static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *env)
 	return &env->insn_aux_data[env->insn_idx];
 }
 
-/* Returns 1 if R4 is a known zero, 0 if it is not, a negative errno on error. */
-static int loop_flag_is_zero(struct bpf_verifier_env *env)
+static bool loop_flag_is_zero(struct bpf_verifier_env *env)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_4);
-	int err;
-
-	if (!bpf_register_is_null(reg))
-		return 0;
-
-	err = mark_chain_precision(env, BPF_REG_4);
-	if (err)
-		return err;
-	return 1;
+	return bpf_register_is_null(reg_state(env, BPF_REG_4));
 }
 
-static int update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno)
+static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno)
 {
 	struct bpf_loop_inline_state *state = &cur_aux(env)->loop_inline_state;
-	int flag_is_zero;
 
 	if (!state->initialized) {
-		flag_is_zero = loop_flag_is_zero(env);
-		if (flag_is_zero < 0)
-			return flag_is_zero;
 		state->initialized = 1;
-		state->fit_for_inline = flag_is_zero;
+		state->fit_for_inline = loop_flag_is_zero(env);
 		state->callback_subprogno = subprogno;
-		return 0;
+		return;
 	}
 
 	if (!state->fit_for_inline)
-		return 0;
+		return;
 
-	flag_is_zero = loop_flag_is_zero(env);
-	if (flag_is_zero < 0)
-		return flag_is_zero;
-	state->fit_for_inline = (flag_is_zero &&
+	state->fit_for_inline = (loop_flag_is_zero(env) &&
 				 state->callback_subprogno == subprogno);
-	return 0;
 }
 
 /* Returns whether or not the given map can potentially elide
@@ -13475,9 +13185,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 			verbose(env, "get_local_storage() doesn't support non-zero flags\n");
 			return -EINVAL;
 		}
-		err = mark_chain_precision(env, BPF_REG_2);
-		if (err)
-			return err;
 		break;
 	case BPF_FUNC_for_each_map_elem:
 		err = push_callback_call(env, insn, insn_idx, meta.subprogno,
@@ -13495,15 +13202,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		err = check_bpf_snprintf_call(env, regs);
 		break;
 	case BPF_FUNC_loop:
-		err = update_loop_inline_state(env, meta.subprogno);
-		if (err)
-			return err;
-		/* Verifier relies on R1 value to determine if bpf_loop() iteration
-		 * is finished, thus mark it precise.
-		 */
-		err = mark_chain_precision(env, BPF_REG_1);
-		if (err)
-			return err;
+		update_loop_inline_state(env, meta.subprogno);
 		if (cur_func(env)->callback_depth < reg_umax(&regs[BPF_REG_1])) {
 			err = push_callback_call(env, insn, insn_idx, meta.subprogno,
 						 set_loop_callback_state);
@@ -13552,10 +13251,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 			bpf_lsm_get_retval_range(env->prog, &range);
 		}
 
-		err = mark_chain_precision(env, BPF_REG_1);
-		if (err)
-			return err;
-
 		if (!retval_range_within(range, r1)) {
 			verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1");
 			return -EINVAL;
@@ -16563,7 +16258,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		ret_nregs = mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
-			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+			__mark_reg_const_zero(&regs[BPF_REG_0]);
 	} else if (btf_type_is_struct(t)) {
 		struct btf_member_path path = {};
 		const char *member_note = "";
@@ -17194,14 +16889,14 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 		/* Taint dst register if offset had invalid bounds derived from
 		 * e.g. dead branches.
 		 */
-		__mark_reg_unknown(env, dst_reg);
+		bpf_mark_reg_unknown(dst_reg);
 		return 0;
 	}
 
 	if (BPF_CLASS(insn->code) != BPF_ALU64 && !to_scalar) {
 		/* 32-bit ALU ops on pointers produce (meaningless) scalars */
 		if (opcode == BPF_SUB && env->allow_ptr_leaks) {
-			__mark_reg_unknown(env, dst_reg);
+			bpf_mark_reg_unknown(dst_reg);
 			return 0;
 		}
 
@@ -17265,7 +16960,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 	}
 
 	if (to_scalar) {
-		__mark_reg_unknown(env, dst_reg);
+		bpf_mark_reg_unknown(dst_reg);
 		return 0;
 	}
 
@@ -18237,7 +17932,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 	int ret;
 
 	if (!is_safe_to_compute_dst_reg_range(insn, &src_reg)) {
-		__mark_reg_unknown(env, dst_reg);
+		bpf_mark_reg_unknown(dst_reg);
 		return 0;
 	}
 
@@ -18296,7 +17991,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 		if (bpf_insn_is_hmul(insn) &&
 		    (!is_reg_const(dst_reg, false) ||
 		     !is_reg_const(&src_reg, false))) {
-			__mark_reg_unknown(env, dst_reg);
+			bpf_mark_reg_unknown(dst_reg);
 			break;
 		}
 		if (!alu32 && insn->off == BPF_UHMUL) {
@@ -18508,24 +18203,13 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
 				 * This is legal, but we have to reverse our
 				 * src/dest handling in computing the range
 				 */
-				err = mark_chain_precision(env, insn->dst_reg);
-				if (err)
-					return err;
 				off_reg = *dst_reg;
 				return adjust_ptr_min_max_vals(env, insn, insn->src_reg, src_reg,
 							       &off_reg);
 			}
 		} else if (ptr_reg) {
 			/* pointer += scalar */
-			err = mark_chain_precision(env, insn->src_reg);
-			if (err)
-				return err;
 			return adjust_ptr_min_max_vals(env, insn, insn->dst_reg, dst_reg, src_reg);
-		} else if (dst_reg->precise) {
-			/* if dst_reg is precise, src_reg should be precise as well */
-			err = mark_chain_precision(env, insn->src_reg);
-			if (err)
-				return err;
 		}
 	} else {
 		/* Pretend the src is a reg with a known value, since we only
@@ -19948,7 +19632,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	struct bpf_reg_state *eq_branch_regs;
 	struct linked_regs linked_regs = {};
 	u8 opcode = BPF_OP(insn->code);
-	int insn_flags = 0;
 	bool is_jmp32;
 	int pred = -1;
 	int err;
@@ -20009,24 +19692,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 			return -EACCES;
 		}
 
-		if (src_reg->type == PTR_TO_STACK)
-			insn_flags |= INSN_F_SRC_REG_STACK;
-		if (dst_reg->type == PTR_TO_STACK)
-			insn_flags |= INSN_F_DST_REG_STACK;
 	} else {
 		src_reg = &env->fake_reg[0];
 		memset(src_reg, 0, sizeof(*src_reg));
 		src_reg->type = SCALAR_VALUE;
 		__mark_reg_known(src_reg, insn->imm);
-
-		if (dst_reg->type == PTR_TO_STACK)
-			insn_flags |= INSN_F_DST_REG_STACK;
-	}
-
-	if (insn_flags) {
-		err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);
-		if (err)
-			return err;
 	}
 
 	/*
@@ -20047,19 +19717,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	pred = is_branch_taken(env, dst_reg, src_reg, opcode, is_jmp32);
 	if (pred >= 0)
 		mark_loop_exit_decided(env, this_branch);
-	if (pred >= 0) {
-		/* If we get here with a dst_reg pointer type it is because
-		 * above is_branch_taken() special cased the 0 comparison.
-		 */
-		if (!__is_pointer_value(false, dst_reg))
-			err = mark_chain_precision(env, insn->dst_reg);
-		if (BPF_SRC(insn->code) == BPF_X && !err &&
-		    !__is_pointer_value(false, src_reg))
-			err = mark_chain_precision(env, insn->src_reg);
-		if (err)
-			return err;
-	}
-
 	if (pred == 1) {
 		/* Only follow the goto, ignore fall-through. If needed, push
 		 * the fall-through branch for simulation under speculative
@@ -20090,20 +19747,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 		return 0;
 	}
 
-	/* Push scalar registers sharing same ID to jump history,
-	 * do this before creating 'other_branch', so that both
-	 * 'this_branch' and 'other_branch' share this history
-	 * if parent state is created.
-	 */
-	if (linked_regs.cnt > 1) {
-		u16 packed[LINKED_REGS_MAX];
-
-		linked_regs_pack(&linked_regs, packed);
-		err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);
-		if (err)
-			return err;
-	}
-
 	other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx, false);
 	if (IS_ERR(other_branch))
 		return PTR_ERR(other_branch);
@@ -20181,15 +19824,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	    type_may_be_null(dst_reg->type) &&
 	    ((BPF_SRC(insn->code) == BPF_K && insn->imm == 0) ||
 	     (BPF_SRC(insn->code) == BPF_X && bpf_register_is_null(src_reg)))) {
-		/*
-		 * For BPF_X the zero is a property of this execution path,
-		 * hence src_reg has to be precise.
-		 */
-		if (BPF_SRC(insn->code) == BPF_X) {
-			err = mark_chain_precision(env, insn->src_reg);
-			if (err)
-				return err;
-		}
 		/* Mark all identical registers in each branch as either
 		 * safe or unknown depending R == 0 or R != 0 conditional.
 		 */
@@ -20296,7 +19930,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
 	    insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
 		if (map->map_type == BPF_MAP_TYPE_ARENA) {
-			__mark_reg_unknown(env, dst_reg);
+			bpf_mark_reg_unknown(dst_reg);
 			dst_reg->map_ptr = map;
 			bpf_diag_mod_end(env);
 			return 0;
@@ -20645,10 +20279,6 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 		return -EINVAL;
 	}
 
-	err = mark_chain_precision(env, regno);
-	if (err)
-		return err;
-
 	if (!retval_range_within(range, reg)) {
 		verbose_invalid_scalar(env, reg, range, exit_ctx, reg_name);
 		if (prog->expected_attach_type == BPF_LSM_CGROUP &&
@@ -22014,9 +21644,9 @@ static int push_budget_exit(struct bpf_verifier_env *env)
 		return PTR_ERR(branch);
 	branch->unwound = sub->bail_unwinds;
 	regs = branch->frame[branch->curframe]->regs;
-	__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	__mark_reg_const_zero(&regs[BPF_REG_0]);
 	if (bpf_ret_reg_pair(env, cur_func(env)->subprogno))
-		__mark_reg_const_zero(env, &regs[BPF_REG_2]);
+		__mark_reg_const_zero(&regs[BPF_REG_2]);
 	return 0;
 }
 
@@ -23461,7 +23091,6 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
 			BPF_MAIN_FUNC /* callsite */,
 			0 /* frameno */,
 			subprog);
-	state->first_insn_idx = env->subprog_info[subprog].start;
 	state->last_insn_idx = -1;
 
 	regs = state->frame[state->curframe]->regs;
@@ -23781,11 +23410,12 @@ static void print_verification_stats(struct bpf_verifier_env *env)
 				env->subprog_info[i].stack_depth);
 		}
 	}
+	/* user space parses this line. There are no read marks. */
 	verbose(env, "processed %d insns (limit %d) max_states_per_insn %d "
-		"total_states %d peak_states %d mark_read %d\n",
+		"total_states %d peak_states %d mark_read 0\n",
 		env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS,
 		env->max_states_per_insn, env->total_states,
-		env->peak_states, env->longest_mark_read_walk);
+		env->peak_states);
 }
 
 int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
@@ -25587,7 +25217,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (!env)
 		return -ENOMEM;
 
-	env->bt.env = env;
 	env->prog = *prog;
 	env->ops = bpf_verifier_ops[env->prog->type];
 	env->stack_limit = bpf_prog_stack_limit(env->prog);
-- 
2.55.0

[PATCH bpf-next v4 25/46] bpf: Remove prune points, levels of state comparison and pop_stack()

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The old main pass followed each path of a prog depth first and compared
states at selected insns to prune paths. flow.c goes from basic block to
basic block. Three more parts of the old pass have no user.

  removed              task in the old pass       what does it now
  -------------------  -------------------------  -------------------------
  prune_point,         Select the insns at which  States meet at the start
  jmp_point,           the pass saves and         of each basic block.
  force_checkpoint,    compares states.
  is_iter_next
  enum exact_level     Select one of three ways   bpf_state_subsumes() has
                       to compare states.         one way, the way of
                                                  RANGE_WITHIN.
  pop_stack()          Get the next state when    flow_drain() empties the
                       a path ends.               stack after each insn.

NOT_EXACT ignored scalars without a precision mark, and the marks are gone.
EXACT accepted only identical states, for "infinite loop detected", which
is gone too. What RANGE_WITHIN did stays: the range of each live scalar of
the new state is within its range in the old state.

push_stack() stays. The code that verifies an insn uses it to make a second
state, for example for the other branch of a jump. flow_drain() gives these
states to flow.c.

What user space sees:
 - The statistics line shows "processed %d insns (limit 1000000)". The
   verifier limits the number of states, not of processed insns, and
   BPF_COMPLEXITY_LIMIT_INSNS is only the maximum length of a prog.
   veristat and other tools parse the line. Keep the format, print 0.
 - The report of a register that is not readable said
   "R1 has never been initialized on this path". It needs the history of
   the whole path, and the verifier has that of the current basic block.
   The report says that the register may never have been initialized, or
   that an earlier operation may have invalidated it.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf.h          |  4 +++
 include/linux/bpf_verifier.h | 38 --------------------
 kernel/bpf/cfg.c             | 56 ++---------------------------
 kernel/bpf/diagnostics.c     | 13 -------
 kernel/bpf/diagnostics.h     |  1 -
 kernel/bpf/scev.c            |  4 ---
 kernel/bpf/states.c          | 70 ++++++++----------------------------
 kernel/bpf/verifier.c        | 61 ++++++++-----------------------
 8 files changed, 36 insertions(+), 211 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index cdf3c9b90987..520fd72e21fd 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2633,6 +2633,10 @@ struct bpf_array {
  */
 int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
 
+/*
+ * the longest program. The verifier limits the number of states, not of
+ * processed insns, see BPF_FLOW_MAX_STATES.
+ */
 #define BPF_COMPLEXITY_LIMIT_INSNS      1000000 /* yes. 1M insns */
 #define MAX_TAIL_CALL_CNT 33
 
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7ee3e40214b5..762fe83cfd3a 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -814,7 +814,6 @@ struct bpf_insn_aux_data {
 	u64 needs_zext:1; /* alu op needs to clear upper bits */
 	u64 prevent_zext:1; /* alu op cannot be zext (already used with 64-bit scalars) */
 	u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
-	u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
 	u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	u64 cast_arena:1; /* addr_space_cast of an address in arena */
 	u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
@@ -848,12 +847,6 @@ struct bpf_insn_aux_data {
 	u64 non_arena_load:1;
 
 	/* below flags are initialized once */
-	u64 jmp_point:1;
-	u64 prune_point:1;
-	/* ensure we check state equivalence and save state checkpoint and
-	 * this instruction, regardless of any heuristics
-	 */
-	u64 force_checkpoint:1;
 	/* true if instruction is a call to a helper function that
 	 * accepts callback function as a parameter.
 	 */
@@ -1184,7 +1177,6 @@ struct bpf_verifier_env {
 	const struct bpf_verifier_ops *ops;
 	struct module *attach_btf_mod;	/* The owner module of prog->aux->attach_btf */
 	struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
-	int stack_size;			/* number of states to be processed */
 	bool strict_alignment;		/* perform strict pointer alignment checks */
 	bool test_reg_invariants;	/* fail verification on register invariants violations */
 	bool uses_arena_sp;		/* the program has insns with BPF_REG_ARENA_SP */
@@ -1405,26 +1397,6 @@ void bpf_verbose_snum32(struct bpf_verifier_env *env, s32 num);
 		bpf_log(&env->log, "verifier bug: " fmt "\n", ##args);				\
 	})
 
-static inline void mark_prune_point(struct bpf_verifier_env *env, int idx)
-{
-	env->insn_aux_data[idx].prune_point = true;
-}
-
-static inline bool bpf_is_prune_point(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].prune_point;
-}
-
-static inline void mark_force_checkpoint(struct bpf_verifier_env *env, int idx)
-{
-	env->insn_aux_data[idx].force_checkpoint = true;
-}
-
-static inline bool bpf_is_force_checkpoint(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].force_checkpoint;
-}
-
 static inline void clear_scalar_id(struct bpf_reg_state *reg)
 {
 	reg->id = 0;
@@ -1441,11 +1413,6 @@ static inline bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx
 	return env->insn_aux_data[insn_idx].calls_callback;
 }
 
-static inline void mark_jmp_point(struct bpf_verifier_env *env, int idx)
-{
-	env->insn_aux_data[idx].jmp_point = true;
-}
-
 static inline void mark_jump_target(struct bpf_verifier_env *env, int idx)
 {
 	env->insn_aux_data[idx].jump_target = true;
@@ -1619,11 +1586,6 @@ static inline u32 bpf_frame_insn_idx(struct bpf_verifier_state *st, u32 frame)
 	       : st->frame[frame + 1]->callsite;
 }
 
-static inline bool bpf_is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
-{
-	return env->insn_aux_data[insn_idx].jmp_point;
-}
-
 static inline bool bpf_is_spilled_reg(const struct bpf_stack_state *stack)
 {
 	return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 94140d1fe1c3..bbcf42eae9d9 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -130,12 +130,8 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 		return -EINVAL;
 	}
 
-	if (e == BRANCH) {
-		/* mark branch target for state pruning */
-		mark_prune_point(env, w);
-		mark_jmp_point(env, w);
+	if (e == BRANCH)
 		mark_jump_target(env, w);
-	}
 
 	if (insn_state[w] == 0) {
 		/* tree-edge */
@@ -179,13 +175,8 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
 	if (ret)
 		return ret;
 
-	mark_prune_point(env, t + insn_sz);
-	/* when we exit from subprog, we need to record non-linear history */
-	mark_jmp_point(env, t + insn_sz);
-
 	if (visit_callee) {
 		w = t + insns[t].imm + 1;
-		mark_prune_point(env, t);
 		merge_callee_effects(env, t, w);
 		ret = push_insn(t, w, BRANCH, env);
 	}
@@ -446,14 +437,12 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
 		return -E2BIG;
 	}
 
-	mark_prune_point(env, t);
 	for (i = 0; i < jt->cnt; i++) {
 		w = jt->items[i];
 		if (verifier_bug_if(w < 0 || w >= env->prog->len, env,
 				    "indirect jump from insn %d to %d out of range", t, w))
 			return -EFAULT;
 
-		mark_jmp_point(env, w);
 		mark_jump_target(env, w);
 
 		/* EXPLORED || DISCOVERED */
@@ -528,14 +517,11 @@ static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
 	}
 
 	/* all functions are explored, their effects are known */
-	mark_prune_point(env, t);
 	for (i = 0; i < cnt; i++) {
 		w = ptrs[i].xlated_off;
 		merge_callee_effects(env, t, w);
 
-		/* the same marks as push_insn() leaves on a branch target */
-		mark_prune_point(env, w);
-		mark_jmp_point(env, w);
+		/* the same mark as push_insn() leaves on a branch target */
 		mark_jump_target(env, w);
 	}
 	return DONE_EXPLORING;
@@ -599,28 +585,13 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 		return DONE_EXPLORING;
 
 	case BPF_CALL:
-		if (bpf_is_async_callback_calling_insn(insn))
-			/* Mark this call insn as a prune point to trigger
-			 * is_state_visited() check before call itself is
-			 * processed by __check_func_call(). Otherwise new
-			 * async state will be pushed for further exploration.
-			 */
-			mark_prune_point(env, t);
 		/* For functions that invoke callbacks it is not known how many times
 		 * callback would be called. Verifier models callback calling functions
 		 * by repeatedly visiting callback bodies and returning to origin call
 		 * instruction.
-		 * In order to stop such iteration verifier needs to identify when a
-		 * state identical some state from a previous iteration is reached.
-		 * Check below forces creation of checkpoint before callback calling
-		 * instruction to allow search for such identical states.
 		 */
-		if (bpf_is_sync_callback_calling_insn(insn)) {
+		if (bpf_is_sync_callback_calling_insn(insn))
 			mark_calls_callback(env, t);
-			mark_force_checkpoint(env, t);
-			mark_prune_point(env, t);
-			mark_jmp_point(env, t);
-		}
 		if (bpf_helper_call(insn)) {
 			const struct bpf_func_proto *fp;
 
@@ -640,21 +611,6 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 			struct bpf_call_arg_meta meta;
 
 			ret = bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta);
-			if (ret == 0 && bpf_is_iter_next_kfunc(&meta)) {
-				mark_prune_point(env, t);
-				/* Checking and saving state checkpoints at iter_next() call
-				 * is crucial for fast convergence of open-coded iterator loop
-				 * logic, so we need to force it. If we don't do that,
-				 * is_state_visited() might skip saving a checkpoint, causing
-				 * unnecessarily long sequence of not checkpointed
-				 * instructions and jumps, leading to exhaustion of jump
-				 * history buffer, and potentially other undesired outcomes.
-				 * It is expected that with correct open-coded iterators
-				 * convergence will happen quickly, so we don't run a risk of
-				 * exhausting memory.
-				 */
-				mark_force_checkpoint(env, t);
-			}
 			/* Same as helpers, if called in a non-sleepable context
 			 * program will be rejected anyway, so we should end up
 			 * with precise sleepable marks on subprogs, except for
@@ -683,18 +639,12 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 		if (ret)
 			return ret;
 
-		mark_prune_point(env, t + off + 1);
-		mark_jmp_point(env, t + off + 1);
 		mark_jump_target(env, t + off + 1);
 
 		return ret;
 
 	default:
 		/* conditional jump with two edges */
-		mark_prune_point(env, t);
-		if (bpf_is_may_goto_insn(insn))
-			mark_force_checkpoint(env, t);
-
 		ret = push_insn(t, t + 1, FALLTHROUGH, env);
 		if (ret)
 			return ret;
diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
index 08e12714bb95..3de3e1486bbd 100644
--- a/kernel/bpf/diagnostics.c
+++ b/kernel/bpf/diagnostics.c
@@ -392,13 +392,6 @@ static u32 log_pos(const struct bpf_diag_log *log, u32 idx)
 	return pos < log->cap ? pos : pos - log->cap;
 }
 
-u64 bpf_diag_event_log_save(struct bpf_verifier_env *env)
-{
-	struct bpf_diag *diag = env->diag;
-
-	return diag ? log_end(&diag->log) : 0;
-}
-
 void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos)
 {
 	struct bpf_diag *diag = env->diag;
@@ -1357,10 +1350,6 @@ void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int reg
 		diag_reason(
 			env, "R%d is not readable here. A previous operation invalidated this register, so the verifier cannot use it as an input.",
 			regno);
-	else if (log && !log->first_seq)
-		diag_reason(env,
-			    "R%d has never been initialized on this path, so the verifier cannot use it as an input.",
-			    regno);
 	else
 		diag_reason(
 			env, "R%d is not readable here. It may never have been initialized, or an earlier operation may have invalidated it.",
@@ -1375,8 +1364,6 @@ void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int reg
 	if (invalidated)
 		diag_suggestion(
 			env, "Avoid using the register after it is invalidated, or initialize it again before this instruction.");
-	else if (log && !log->first_seq)
-		diag_suggestion(env, "Initialize R%d on every path before this instruction.", regno);
 	else
 		diag_suggestion(
 			env, "Initialize the register on every path, or initialize it again after any operation that invalidates it.");
diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h
index 490decc133f6..e155d2588a51 100644
--- a/kernel/bpf/diagnostics.h
+++ b/kernel/bpf/diagnostics.h
@@ -53,7 +53,6 @@ const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __p
 const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id);
 const char *bpf_diag_reg_type_plain(struct bpf_verifier_env *env, enum bpf_reg_type type);
 const char *bpf_diag_arg_type_plain(enum bpf_arg_type type);
-u64 bpf_diag_event_log_save(struct bpf_verifier_env *env);
 void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos);
 u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state);
 void bpf_diag_free(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index de7592424056..9377b0907f21 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1861,10 +1861,6 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
 						 find_header_env(scev, header));
 	}
 
-	for (i = 0; i < env->prog->len; i++)
-		if (aux[i].loop)
-			aux[i].prune_point = true;
-
 	if (env->log.level & BPF_LOG_LEVEL2) {
 		err = log_scevs(env);
 		if (err)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 5935e16730fd..fb7998654e16 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -198,8 +198,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
 	}
 }
 
-static int clean_verifier_state(struct bpf_verifier_env *env,
-				 struct bpf_verifier_state *st)
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
 {
 	int i, err;
 
@@ -225,20 +224,10 @@ static bool regs_exact(const struct bpf_reg_state *rold,
 	       check_ids(rold->map_uid, rcur->map_uid, idmap);
 }
 
-enum exact_level {
-	NOT_EXACT,
-	EXACT,
-	RANGE_WITHIN
-};
-
 /* Returns true if (rold safe implies rcur safe) */
 static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
-		    struct bpf_reg_state *rcur, struct bpf_idmap *idmap,
-		    enum exact_level exact)
+		    struct bpf_reg_state *rcur, struct bpf_idmap *idmap)
 {
-	if (exact == EXACT)
-		return regs_exact(rold, rcur, idmap);
-
 	if (rold->type == NOT_INIT)
 		/* explored state can't have used this */
 		return true;
@@ -427,8 +416,7 @@ static struct bpf_reg_state *scalar_reg_for_stack(struct bpf_verifier_env *env,
 }
 
 static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
-		      struct bpf_func_state *cur, struct bpf_idmap *idmap,
-		      enum exact_level exact)
+		      struct bpf_func_state *cur, struct bpf_idmap *idmap)
 {
 	int i, spi;
 
@@ -447,18 +435,6 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		old_type = old_slot->slot_type[im];
 		cur_slot = i < cur->allocated_stack ? bpf_stack_slot(cur, spi) : NULL;
 
-		if (exact == EXACT) {
-			u8 cur_type = cur_slot ? cur_slot->slot_type[im] : STACK_INVALID;
-
-			/* STACK_INVALID and STACK_POISON are equivalent for pruning */
-			if (old_type == STACK_POISON)
-				old_type = STACK_INVALID;
-			if (cur_type == STACK_POISON)
-				cur_type = STACK_INVALID;
-			if (!cur_slot || old_type != cur_type)
-				return false;
-		}
-
 		if (old_type == STACK_INVALID || old_type == STACK_POISON)
 			continue;
 
@@ -481,7 +457,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			old_reg = scalar_reg_for_stack(env, old_slot, im);
 			cur_reg = scalar_reg_for_stack(env, cur_slot, im);
 			if (old_reg && cur_reg) {
-				if (!regsafe(env, old_reg, cur_reg, idmap, exact))
+				if (!regsafe(env, old_reg, cur_reg, idmap))
 					return false;
 				i += (im == 0 ? BPF_REG_SIZE - 1 : 3);
 				continue;
@@ -516,8 +492,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			 * such verifier states are not equivalent.
 			 * return false to continue verification of this path
 			 */
-			if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr,
-				     idmap, exact))
+			if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr, idmap))
 				return false;
 			break;
 		case STACK_DYNPTR:
@@ -565,8 +540,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
  * Outgoing stack args are path-local state and must agree for pruning.
  */
 static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
-			   struct bpf_func_state *cur, struct bpf_idmap *idmap,
-			   enum exact_level exact)
+			   struct bpf_func_state *cur, struct bpf_idmap *idmap)
 {
 	int i, nslots, cur_cnt = cur->out_stack_arg_cnt;
 
@@ -582,7 +556,7 @@ static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *
 			  &old->stack_arg_regs[i] : &not_init;
 		cur_arg = i < cur_cnt ?
 			  &cur->stack_arg_regs[i] : &not_init;
-		if (!regsafe(env, old_arg, cur_arg, idmap, exact))
+		if (!regsafe(env, old_arg, cur_arg, idmap))
 			return false;
 	}
 
@@ -710,7 +684,7 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
  * the current state will reach 'bpf_exit' instruction safely
  */
 static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
-			      struct bpf_func_state *cur, u32 insn_idx, enum exact_level exact)
+			      struct bpf_func_state *cur, u32 insn_idx)
 {
 	u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
 	u16 i;
@@ -736,14 +710,13 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (((1 << i) & live_regs) &&
-		    !regsafe(env, &old->regs[i], &cur->regs[i],
-			     &env->idmap_scratch, exact))
+		    !regsafe(env, &old->regs[i], &cur->regs[i], &env->idmap_scratch))
 			return false;
 
-	if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
+	if (!stacksafe(env, old, cur, &env->idmap_scratch))
 		return false;
 
-	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
+	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch))
 		return false;
 
 	if (!loop_stack_safe(env, old, cur))
@@ -760,10 +733,9 @@ static void reset_idmap_scratch(struct bpf_verifier_env *env)
 	idmap->cnt = 0;
 }
 
-static bool states_equal(struct bpf_verifier_env *env,
-			 struct bpf_verifier_state *old,
-			 struct bpf_verifier_state *cur,
-			 enum exact_level exact)
+/* Everything that @cur describes is described by @old. */
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+			struct bpf_verifier_state *cur)
 {
 	u32 insn_idx;
 	int i;
@@ -798,21 +770,9 @@ static bool states_equal(struct bpf_verifier_env *env,
 		insn_idx = bpf_frame_insn_idx(old, i);
 		if (old->frame[i]->callsite != cur->frame[i]->callsite)
 			return false;
-		if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx, exact))
+		if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx))
 			return false;
 	}
 
 	return true;
 }
-
-int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
-	return clean_verifier_state(env, st);
-}
-
-/* Everything that @cur describes is described by @old. */
-bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
-			struct bpf_verifier_state *cur)
-{
-	return states_equal(env, old, cur, RANGE_WITHIN);
-}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 85dae0feeb1a..c786755b2a3c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -185,7 +185,10 @@ static bool bpf_global_percpu_ma_set;
  * reference remains at the end of the program, the verifier rejects it.
  */
 
-/* verifier_state + insn_idx are pushed to stack when branch is encountered */
+/*
+ * verifier_state + insn_idx are pushed to stack when branch is encountered.
+ * It's emptied after every insn, see flow_drain().
+ */
 struct bpf_verifier_stack_elem {
 	/* verifier state is 'st'
 	 * before processing instruction 'insn_idx'
@@ -197,9 +200,6 @@ struct bpf_verifier_stack_elem {
 	/* it goes there only if may_goto in front of prev_insn_idx does, see check_loop_guard() */
 	enum bpf_guard_edge guard_edge;
 	struct bpf_verifier_stack_elem *next;
-	/* length of verifier log at the time this state was pushed on stack */
-	u32 log_pos;
-	u64 diag_log_pos;
 };
 
 #define BPF_GLOBAL_PERCPU_MA_MAX_SIZE  512
@@ -1816,36 +1816,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	return 0;
 }
 
-static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
-		     int *insn_idx, bool pop_log)
-{
-	struct bpf_verifier_state *cur = env->cur_state;
-	struct bpf_verifier_stack_elem *elem, *head = env->head;
-	int err;
-
-	if (env->head == NULL)
-		return -ENOENT;
-
-	if (cur) {
-		err = bpf_copy_verifier_state(cur, &head->st);
-		if (err)
-			return err;
-		bpf_diag_event_log_restore(env, head->diag_log_pos);
-	}
-	if (pop_log)
-		bpf_vlog_reset(&env->log, head->log_pos);
-	if (insn_idx)
-		*insn_idx = head->insn_idx;
-	if (prev_insn_idx)
-		*prev_insn_idx = head->prev_insn_idx;
-	elem = head->next;
-	bpf_free_verifier_state(&head->st, false);
-	kfree(head);
-	env->head = elem;
-	env->stack_size--;
-	return 0;
-}
-
 static bool error_recoverable_with_nospec(int err)
 {
 	/* Should only return true for non-fatal errors that are allowed to
@@ -1873,10 +1843,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	elem->insn_idx = insn_idx;
 	elem->prev_insn_idx = prev_insn_idx;
 	elem->next = env->head;
-	elem->log_pos = env->log.end_pos;
-	elem->diag_log_pos = bpf_diag_event_log_save(env);
 	env->head = elem;
-	env->stack_size++;
 	err = bpf_copy_verifier_state(&elem->st, cur);
 	if (err)
 		return ERR_PTR(-ENOMEM);
@@ -2529,10 +2496,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	elem->insn_idx = insn_idx;
 	elem->prev_insn_idx = prev_insn_idx;
 	elem->next = env->head;
-	elem->log_pos = env->log.end_pos;
-	elem->diag_log_pos = bpf_diag_event_log_save(env);
 	env->head = elem;
-	env->stack_size++;
 	/* Unlike push_stack() do not bpf_copy_verifier_state().
 	 * The caller state doesn't matter.
 	 * This is async callback. It starts in a fresh stack.
@@ -15530,7 +15494,6 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
 	return info;
 }
 
-/* insn_aux_data->is_iter_next is not there before the main pass */
 bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn)
 {
 	struct bpf_call_arg_meta meta;
@@ -15990,7 +15953,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 	meta.fn = &desc->proto;
 
-	insn_aux->is_iter_next = bpf_is_iter_next_kfunc(&meta);
 
 	if (!insn->off &&
 	    (insn->imm == special_kfunc_list[KF_bpf_res_spin_lock] ||
@@ -21742,7 +21704,6 @@ static int flow_drain(struct bpf_verifier_env *env)
 
 	while ((elem = env->head)) {
 		env->head = elem->next;
-		env->stack_size--;
 		if (!err && elem->guard_edge)
 			err = bpf_flow_guard_edge(env, &elem->st, elem->insn_idx,
 						  elem->prev_insn_idx, elem->guard_edge);
@@ -23051,9 +23012,15 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 
 static void free_states(struct bpf_verifier_env *env)
 {
+	struct bpf_verifier_stack_elem *elem;
+
 	bpf_free_verifier_state(env->cur_state, true);
 	env->cur_state = NULL;
-	while (!pop_stack(env, NULL, NULL, false));
+	while ((elem = env->head)) {
+		env->head = elem->next;
+		bpf_free_verifier_state(&elem->st, false);
+		kfree(elem);
+	}
 	bpf_flow_reset(env);
 }
 
@@ -23410,10 +23377,10 @@ static void print_verification_stats(struct bpf_verifier_env *env)
 				env->subprog_info[i].stack_depth);
 		}
 	}
-	/* user space parses this line. There are no read marks. */
-	verbose(env, "processed %d insns (limit %d) max_states_per_insn %d "
+	/* user space parses this line. There is no limit and there are no read marks. */
+	verbose(env, "processed %d insns (limit 0) max_states_per_insn %d "
 		"total_states %d peak_states %d mark_read 0\n",
-		env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS,
+		env->insn_processed,
 		env->max_states_per_insn, env->total_states,
 		env->peak_states);
 }
-- 
2.55.0

[PATCH bpf-next v4 26/46] bpf: Update comments about paths, pruning and explored states

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

About 30 comments in verifier.c, states.c and bpf_verifier.h describe
a verifier that explores all paths of a prog and prunes a path against
explored states. The reader of regsafe() or stacksafe() learns from them
that the old state "already passed validation" and that the function
returns false "to continue verification of this path". Nothing of it is
true since the main pass is a fixpoint over the control flow graph.

Say what the code does now. States meet at the start of a basic block. The
verifier drops a state when a state that is already there covers it: if the
prog is fine with the old state, it is fine with the current one. Otherwise
it joins the two states or keeps both.

The largest change is above process_iter_next_call(). 60 lines explain how
a loop with an open-coded iterator converges: exact comparison, the depth
of the iteration, widening of scalars that are not precise. Such a loop is
a loop like any other now: what comes back to its head is joined with what
is there until nothing changes.

No functional changes.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  11 ++--
 kernel/bpf/states.c          |  59 ++++++++------------
 kernel/bpf/verifier.c        | 102 ++++++++---------------------------
 3 files changed, 49 insertions(+), 123 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 762fe83cfd3a..6e855ef0b6e6 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -58,7 +58,7 @@ struct bpf_stack_range {
 };
 
 struct bpf_reg_state {
-	/* Ordering of fields matters.  See states_equal() */
+	/* Ordering of fields matters.  See regsafe() */
 	enum bpf_reg_type type;
 	/*
 	 * Constant delta between "linked" scalars with the same ID.
@@ -443,10 +443,9 @@ struct bpf_func_state {
 	 * zero == main subprog
 	 */
 	u32 subprogno;
-	/* Every bpf_timer_start will increment async_entry_cnt.
-	 * It's used to distinguish:
-	 * void foo(void) { for(;;); }
-	 * void foo(void) { bpf_timer_set_callback(,foo); }
+	/*
+	 * The frame of an async callback has the count of frame 0 of the state
+	 * that sets the callback plus one. Frames that differ in it are not joined.
 	 */
 	u32 async_entry_cnt;
 	struct bpf_retval_range callback_ret_range;
@@ -1984,7 +1983,7 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn);
 void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 			  struct bpf_call_summary *cs);
-/* Stack effects used for state pruning; must_write implies may_write. */
+/* Stack effects used for liveness; must_write implies may_write. */
 struct arg_access_info {
 	u32 size;		/* Maximum extent; U32_MAX if unknown. */
 	u8 may_read:1;		/* Incoming contents or initialization may be needed. */
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index fb7998654e16..07b1f0becd2e 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -229,7 +229,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		    struct bpf_reg_state *rcur, struct bpf_idmap *idmap)
 {
 	if (rold->type == NOT_INIT)
-		/* explored state can't have used this */
+		/* nothing can be done with it */
 		return true;
 
 	/* Enforce that register types have to match exactly, including their
@@ -308,7 +308,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		 * ADD_CONST flags must match exactly: BPF_ADD_CONST32 and
 		 * BPF_ADD_CONST64 have different linking semantics in
 		 * sync_linked_regs() (alu32 zero-extends, alu64 does not),
-		 * so pruning across different flag types is unsafe.
+		 * so one doesn't stand for the other.
 		 */
 		if (rold->id &&
 		    (rold->id & BPF_ADD_CONST) != (rcur->id & BPF_ADD_CONST))
@@ -420,9 +420,10 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 {
 	int i, spi;
 
-	/* walk slots of the explored stack and ignore any additional
-	 * slots in the current stack, since explored(safe) state
-	 * didn't use them
+	/*
+	 * walk slots of the old stack and ignore any additional
+	 * slots in the current stack, since the old state
+	 * can't use them
 	 */
 	for (i = 0; i < old->allocated_stack; i++) {
 		struct bpf_stack_state *old_slot, *cur_slot;
@@ -441,9 +442,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		if (env->allow_uninit_stack && old_type == STACK_MISC)
 			continue;
 
-		/* explored stack has more populated slots than current stack
-		 * and these slots were used
-		 */
+		/* old stack has more populated slots than current stack */
 		if (!cur_slot)
 			return false;
 
@@ -471,10 +470,10 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		if (old_type == STACK_MISC && cur_slot->slot_type[im] == STACK_ZERO)
 			continue;
 		if (old_type != cur_slot->slot_type[im])
-			/* Ex: old explored (safe) state has STACK_SPILL in
+			/*
+			 * Ex: old state has STACK_SPILL in
 			 * this stack slot, but current has STACK_MISC ->
-			 * this verifier states are not equivalent,
-			 * return false to continue verification of this path
+			 * this verifier states are not equivalent
 			 */
 			return false;
 		if (im != BPF_REG_SIZE - 1)
@@ -482,15 +481,15 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		/* Both old and cur are having same slot_type */
 		switch (old_type) {
 		case STACK_SPILL:
-			/* when explored and current stack slot are both storing
+			/*
+			 * when old and current stack slot are both storing
 			 * spilled registers, check that stored pointers types
 			 * are the same as well.
-			 * Ex: explored safe path could have stored
+			 * Ex: old state could have stored
 			 * (bpf_reg_state) {.type = PTR_TO_STACK, .off = -8}
-			 * but current path has stored:
+			 * but current state has stored:
 			 * (bpf_reg_state) {.type = PTR_TO_STACK, .off = -16}
 			 * such verifier states are not equivalent.
-			 * return false to continue verification of this path
 			 */
 			if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr, idmap))
 				return false;
@@ -537,7 +536,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 
 /*
  * Compare stack arg slots between old and current states.
- * Outgoing stack args are path-local state and must agree for pruning.
+ * Outgoing stack args are a part of the state and must agree.
  */
 static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			   struct bpf_func_state *cur, struct bpf_idmap *idmap)
@@ -659,29 +658,17 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
 
 /* compare two verifier states
  *
- * all states stored in state_list are known to be valid, since
- * verifier reached 'bpf_exit' instruction through them
- *
- * this function is called when verifier exploring different branches of
- * execution popped from the state stack. If it sees an old state that has
- * more strict register state and more strict stack state then this execution
- * branch doesn't need to be explored further, since verifier already
- * concluded that more strict state leads to valid finish.
- *
- * Therefore two states are equivalent if register state is more conservative
- * and explored stack state is more conservative than the current one.
+ * The state that gets to an insn doesn't have to be looked at when there is
+ * a state at that insn that is more conservative in registers and in the
+ * stack: if the program is fine with the old state it's fine with the current
+ * one.
  * Example:
- *       explored                   current
+ *          old                     current
  * (slot1=INV slot2=MISC) == (slot1=MISC slot2=MISC)
  * (slot1=MISC slot2=MISC) != (slot1=INV slot2=MISC)
  *
- * In other words if current stack state (one being explored) has more
- * valid slots than old one that already passed validation, it means
- * the verifier can stop exploring and conclude that current state is valid too
- *
- * Similarly with registers. If explored state has register type as invalid
- * whereas register type in current state is meaningful, it means that
- * the current state will reach 'bpf_exit' instruction safely
+ * Similarly with registers: the type in the old state is invalid whereas
+ * the type in the current state is meaningful.
  */
 static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
 			      struct bpf_func_state *cur, u32 insn_idx)
@@ -746,7 +733,7 @@ bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state
 	reset_idmap_scratch(env);
 
 	/* Verification state from speculative execution simulation
-	 * must never prune a non-speculative execution one.
+	 * must never stand for a non-speculative execution one.
 	 */
 	if (old->speculative && !cur->speculative)
 		return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c786755b2a3c..7769b4bd1467 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -62,19 +62,16 @@ static bool bpf_global_percpu_ma_set;
 
 /* bpf_check() is a static code analyzer that walks eBPF program
  * instruction by instruction and updates register/stack state.
- * All paths of conditional branches are analyzed until 'bpf_exit' insn.
  *
- * The first pass is depth-first-search to check that the program is a DAG.
+ * The first pass is depth-first-search over the control flow graph.
  * It rejects the following programs:
- * - larger than BPF_MAXINSNS insns
- * - if loop is present (detected via back-edge)
+ * - if loop is present (detected via back-edge) and the program is unprivileged
  * - unreachable insns exist (shouldn't be a forest. program = one function)
  * - out of bounds or malformed jumps
- * The second pass is all possible path descent from the 1st insn.
- * Since it's analyzing all paths through the program, the length of the
- * analysis is limited to 64k insn, which may be hit even if total number of
- * insn is less then 4K, but there are too many branches that change stack/regs.
- * Number of 'branches to be analyzed' is limited to 1k
+ * The second pass goes from basic block to basic block, starting at the 1st
+ * insn. States that meet at the start of a block are joined, so the length of
+ * the analysis follows the size of the program and not the number of paths
+ * through it. See flow.c.
  *
  * On entry to each instruction, each register has a type, and the instruction
  * changes the types of the registers depending on instruction semantics.
@@ -1358,7 +1355,7 @@ static bool is_stack_slot_special(const struct bpf_stack_state *stack)
  * - STACK_POISON, which truly forbids access to the slot.
  * Regardless of allow_ptr_leaks setting (i.e., privileged or unprivileged
  * mode), we won't promote STACK_INVALID to STACK_MISC. In privileged case it is
- * unnecessary as both are considered equivalent when loading data and pruning,
+ * unnecessary as both are considered equivalent when loading data and comparing states,
  * in case of unprivileged mode it will be incorrect to allow reads of invalid
  * slots.
  */
@@ -1782,9 +1779,7 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
 	struct bpf_func_state *dst;
 	int i, err;
 
-	/* if dst has more stack frames then src frame, free them, this is also
-	 * necessary in case of exceptional exits using bpf_throw.
-	 */
+	/* if dst has more stack frames then src frame, free them */
 	for (i = src->curframe + 1; i <= dst_state->curframe; i++) {
 		free_func_state(dst_state->frame[i]);
 		dst_state->frame[i] = NULL;
@@ -8937,69 +8932,13 @@ static void iter_stops_loops(struct bpf_verifier_env *env, int insn_idx,
  * without having to simulate infinite amount of iterations.
  *
  * In current state, we first assume that iter_next() returned NULL and
- * iterator state is set to DRAINED (BPF_ITER_STATE_DRAINED). In such
- * conditions we should not form an infinite loop and should eventually reach
- * exit.
- *
- * Besides that, we also fork current state and enqueue it for later
- * verification. In a forked state we keep iterator state as ACTIVE
- * (BPF_ITER_STATE_ACTIVE) and assume non-NULL return from iter_next(). We
- * also bump iteration depth to prevent erroneous infinite loop detection
- * later on (see iter_active_depths_differ() comment for details). In this
- * state we assume that we'll eventually loop back to another iter_next()
- * calls (it could be in exactly same location or in some other instruction,
- * it doesn't matter, we don't make any unnecessary assumptions about this,
- * everything revolves around iterator state in a stack slot, not which
- * instruction is calling iter_next()). When that happens, we either will come
- * to iter_next() with equivalent state and can conclude that next iteration
- * will proceed in exactly the same way as we just verified, so it's safe to
- * assume that loop converges. If not, we'll go on another iteration
- * simulation with a different input state, until all possible starting states
- * are validated or we reach maximum number of instructions limit.
- *
- * This way, we will either exhaustively discover all possible input states
- * that iterator loop can start with and eventually will converge, or we'll
- * effectively regress into bounded loop simulation logic and either reach
- * maximum number of instructions if loop is not provably convergent, or there
- * is some statically known limit on number of iterations (e.g., if there is
- * an explicit `if n > 100 then break;` statement somewhere in the loop).
- *
- * Iteration convergence logic in is_state_visited() relies on exact
- * states comparison, which ignores read and precision marks.
- * This is necessary because read and precision marks are not finalized
- * while in the loop. Exact comparison might preclude convergence for
- * simple programs like below:
- *
- *     i = 0;
- *     while(iter_next(&it))
- *       i++;
+ * iterator state is set to DRAINED (BPF_ITER_STATE_DRAINED).
  *
- * At each iteration step i++ would produce a new distinct state and
- * eventually instruction processing limit would be reached.
- *
- * To avoid such behavior speculatively forget (widen) range for
- * imprecise scalar registers, if those registers were not precise at the
- * end of the previous iteration and do not match exactly.
- *
- * This is a conservative heuristic that allows to verify wide range of programs,
- * however it precludes verification of programs that conjure an
- * imprecise value on the first loop iteration and use it as precise on a second.
- * For example, the following safe program would fail to verify:
- *
- *     struct bpf_num_iter it;
- *     int arr[10];
- *     int i = 0, a = 0;
- *     bpf_iter_num_new(&it, 0, 10);
- *     while (bpf_iter_num_next(&it)) {
- *       if (a == 0) {
- *         a = 1;
- *         i = 7; // Because i changed verifier would forget
- *                // it's range on second loop entry.
- *       } else {
- *         arr[i] = 42; // This would fail to verify.
- *       }
- *     }
- *     bpf_iter_num_destroy(&it);
+ * Besides that, we also fork current state. In a forked state we keep iterator
+ * state as ACTIVE (BPF_ITER_STATE_ACTIVE) and assume non-NULL return from
+ * iter_next(). This state goes around the loop like in any other loop: what
+ * comes back to the head of the loop is joined with what is there until
+ * nothing changes, see flow.c.
  */
 static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 				  struct bpf_call_arg_meta *meta)
@@ -12635,7 +12574,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	}
 	account_processed_insns(env, callee, caller);
 	/* clear everything in the callee. In case of exceptional exits using
-	 * bpf_throw, this will be done by copy_verifier_state for extra frames. */
+	 * bpf_throw, this will be done by do_check_block() for extra frames. */
 	free_func_state(callee);
 	state->frame[state->curframe--] = NULL;
 	invalidate_outgoing_stack_args(env, caller);
@@ -15361,7 +15300,7 @@ int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env,
  * Describe a helper's stack access through argument @arg (0-based, R1-R5).
  * must_write means the verifier destroys prior state throughout size bytes.
  * In other words, must_write follows the verifier's model of the
- * function's behaviour, which is safe to use for is_state_visited() pruning.
+ * function's behaviour, which is safe to use when states are cleaned and compared.
  */
 struct arg_access_info
 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
@@ -16643,7 +16582,8 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
 		alu_state |= ptr_is_dst_reg ?
 			     BPF_ALU_SANITIZE_SRC : BPF_ALU_SANITIZE_DST;
 
-		/* Limit pruning on unknown scalars to enable deep search for
+		/*
+		 * Keep states that differ in unknown scalars apart to find
 		 * potential masking differences from other program paths.
 		 */
 		if (!off_is_imm)
@@ -19308,7 +19248,7 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state,
 		if (is_null) {
 			/* We don't need id from this point
 			 * onwards anymore, thus we should better reset it,
-			 * so that state pruning has chances to take effect.
+			 * so that states have chances to be equal.
 			 */
 			__mark_reg_known_zero(reg);
 			reg->type = SCALAR_VALUE;
@@ -20773,8 +20713,8 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 	/* The side effect of the prepare_func_exit which is
 	 * being skipped is that it frees bpf_func_state.
 	 * Typically, process_bpf_exit will only be hit with
-	 * outermost exit. copy_verifier_state in pop_stack will
-	 * handle freeing of any extra bpf_func_state left over
+	 * outermost exit. do_check_block() frees the state with
+	 * any extra bpf_func_state left over
 	 * from not processing all nested function exits. We
 	 * also skip return code checks as they are not needed
 	 * for exceptional exits.
-- 
2.55.0

Re: [PATCH bpf-next v4 26/46] bpf: Update comments about paths, pruning and explored states

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] bpf: non-standard multi-line comment style in updated verifier
  comments
--

commit a631e21cb29a0618e130f63e9c4f9c8e4eb63371
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Update comments about paths, pruning and explored states

This commit updates verifier comments in include/linux/bpf_verifier.h,
kernel/bpf/states.c, and kernel/bpf/verifier.c to describe the CFG fixpoint
and state joining design instead of path pruning.

[Severity: Low]
This isn't a bug, but several multi-line comments updated by this patch keep
text on the opening /* line (and on the closing */ line in
prepare_func_exit()), whereas BPF coding style places /* and */ on their own
lines for multi-line comments:

- kernel/bpf/states.c: func_states_equal() (line 659) and
  bpf_state_subsumes() (line 735)
- kernel/bpf/verifier.c: the bpf_check() overview comment (line 63),
  process_iter_next_call() (line 8921), prepare_func_exit() (line 12576),
  mark_ptr_or_null_reg() (line 19249), and process_bpf_exit_full()
  (line 20713)

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=26

[PATCH bpf-next v4 27/46] bpf: Track a pointer to one of a set of functions

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Each '{}' of a format string in Rust ends in core::fmt::write(), which
calls the fmt() function of the argument through a pointer:

  ArgumentType::Placeholder { formatter, value, .. } => formatter(value, f)

rust-bpf compiles it to a load from the table of all fmt() functions of the
object, and callx. k2_text.o of its corpus, with serde_json, uuid, semver,
ryu and regex-automata, has 245. At the load the verifier goes on with one
function and makes a state for each of the other 244. 3 states get to the
load: 732 states that differ only in r3 meet at the next insn. The limit is
1024. Each more type with Display or Debug adds 3 states, and 100 more
types give "more than 1024 states that cannot be joined at insn N".

  if r2 < 244 goto +1
  r2 = 244
  r2 <<= 3
  r2 += r3                 ; the table, in a read-only map
  r3 = *(u64 *)(r2 + 8)
  ...
  callx r3

Do not split the state at the load. r3 is a pointer to one of a set of
functions until an insn must know the function.

     r3 = *(u64 *)(r2 + 8)              r3 = *(u64 *)(r2 + 8)
      /        |        \                         |
   r3=f1     r3=f2 ...  r3=fN             r3 = one of f1 .. fN
     |         |          |                       |
  callx r3  callx r3   callx r3               callx r3
     |         |          |                  /    |    \
    f1        f2         fN                f1    f2 ... fN

The register holds the number of the set, and equal sets have the same
number. Thus the code that compares and joins registers does not change.
The log shows R3=func(one_of=245).

callx verifies each function of the set, as before. For a function that is
verified on its own, a copy of the state of the caller waits for its exit
states. The copy is not a state at an insn. Other functions, and a helper
or a kfunc that gets the pointer as a callback, get one state for each
function. uuid_semver of k2_text.o: 732 -> 257 states at an insn and
-19 % insns.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  15 ++-
 kernel/bpf/log.c             |   2 +
 kernel/bpf/verifier.c        | 181 +++++++++++++++++++++++++++++------
 3 files changed, 169 insertions(+), 29 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6e855ef0b6e6..098b3c30c8b2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -140,7 +140,11 @@ struct bpf_reg_state {
 			unsigned long raw2;
 		} raw;
 
-		u32 subprogno; /* for PTR_TO_FUNC */
+		struct { /* for PTR_TO_FUNC */
+			u32 subprogno;
+			/* if not 0: one of env->func_sets[func_set - 1], subprogno is 0 */
+			u32 func_set;
+		};
 	};
 	/* For scalar types (SCALAR_VALUE), this represents our knowledge of
 	 * the actual value.
@@ -1228,6 +1232,9 @@ struct bpf_verifier_env {
 	 */
 	struct bpf_func_ptr *func_ptrs;
 	u32 func_ptr_cnt;
+	/* What a pointer that was read from a table of functions may point to. */
+	struct bpf_func_set **func_sets;
+	u32 func_set_cnt;
 	bool has_callx;
 	/*
 	 * Call graph edges created by callx instructions. A bitmap of
@@ -1610,6 +1617,12 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
 }
 
 bool bpf_map_is_rdonly(const struct bpf_map *map);
+/* Sorted, no function is there twice */
+struct bpf_func_set {
+	u16 cnt;
+	u16 subprog[] __counted_by(cnt);
+};
+
 struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
 				       const struct bpf_map *map, u32 *cnt);
 struct bpf_func_ptr *bpf_map_range_func_ptrs(struct bpf_verifier_env *env,
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index db496253bf62..1253118dfd05 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -664,6 +664,8 @@ static void print_reg_state(struct bpf_verifier_env *env,
 		verbose_a("parent_id=%d", reg->parent_id);
 	if (type_is_non_owning_ref(reg->type))
 		verbose_a("%s", "non_own_ref");
+	if (t == PTR_TO_FUNC && reg->func_set)
+		verbose_a("one_of=%d", env->func_sets[reg->func_set - 1]->cnt);
 	if (type_is_map_ptr(t)) {
 		if (reg->map_ptr->name[0])
 			verbose_a("map=%s", reg->map_ptr->name);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 7769b4bd1467..a5c3e86f3f87 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -7207,7 +7207,60 @@ static void mark_reg_func_ptr(struct bpf_verifier_env *env, struct bpf_reg_state
 	regs[regno].subprogno = subprog;
 }
 
-/* a read from a table of functions branches into that many states at most */
+static int cmp_u16(const void *a, const void *b)
+{
+	return *(const u16 *)a - *(const u16 *)b;
+}
+
+/* that many different sets at most: each load from a table looks through all of them */
+#define BPF_MAX_FUNC_SETS 4096
+
+/*
+ * A load from a table of functions gives one register: a pointer to one of a set of functions.
+ * One state for each function would differ in nothing else up to the call, and there every
+ * function of the set is verified.
+ * Equal sets have the same number, so registers are compared as they are.
+ */
+static int mark_reg_func_set(struct bpf_verifier_env *env, struct bpf_reg_state *regs, int regno,
+			     u16 *subprogs, u32 cnt)
+{
+	struct bpf_func_set **sets, *set;
+	u32 i, n = 0;
+
+	sort(subprogs, cnt, sizeof(*subprogs), cmp_u16, NULL);
+	for (i = 0; i < cnt; i++)
+		if (!n || subprogs[n - 1] != subprogs[i])
+			subprogs[n++] = subprogs[i];
+	mark_reg_func_ptr(env, regs, regno, n == 1 ? subprogs[0] : 0);
+	if (n == 1)
+		return 0;
+
+	for (i = 0; i < env->func_set_cnt; i++) {
+		set = env->func_sets[i];
+		if (set->cnt == n && !memcmp(set->subprog, subprogs, n * sizeof(*subprogs)))
+			break;
+	}
+	if (i == env->func_set_cnt) {
+		if (i == BPF_MAX_FUNC_SETS) {
+			verbose(env, "more than %d sets of functions\n", BPF_MAX_FUNC_SETS);
+			return -E2BIG;
+		}
+		sets = krealloc_array(env->func_sets, i + 1, sizeof(*sets), GFP_KERNEL_ACCOUNT);
+		if (!sets)
+			return -ENOMEM;
+		env->func_sets = sets;
+		set = kzalloc_flex(*set, subprog, n, GFP_KERNEL_ACCOUNT);
+		if (!set)
+			return -ENOMEM;
+		set->cnt = n;
+		memcpy(set->subprog, subprogs, n * sizeof(*subprogs));
+		sets[env->func_set_cnt++] = set;
+	}
+	regs[regno].func_set = i + 1;
+	return 0;
+}
+
+/* a read from a table of functions yields one of that many at most */
 #define BPF_MAX_FUNC_PTR_TARGETS BPF_MAX_SUBPROGS
 /* and that many pointers at most are next to what is read at variable offset */
 #define BPF_MAX_FUNC_PTR_RANGE 4096
@@ -7240,7 +7293,7 @@ static u32 count_read_offsets(struct tnum offs, u64 min_off, u64 max_off, u32 li
  * is possible at, e.g. var_off is the stride of the elements of an array.
  * A read of exactly one pointer yields PTR_TO_FUNC. When the offset is variable
  * and only pointers can be read, which is how an element of a table of
- * functions is loaded, the verification continues with each of them. Other
+ * functions is loaded, it is one of them, see mark_reg_func_set(). Other
  * reads that overlap with a pointer are rejected, because their result is not
  * known until the program is jitted.
  *
@@ -7253,7 +7306,6 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
 	struct tnum offs = tnum_add(reg->var_off, tnum_const(off));
 	struct bpf_reg_state *regs = cur_regs(env);
 	struct bpf_map *map = reg->map_ptr;
-	struct bpf_verifier_state *branch;
 	u16 *targets __free(kfree) = NULL;
 	struct bpf_func_ptr *ptrs;
 	u32 i, cnt, n = 0;
@@ -7304,16 +7356,7 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
 	if (err)
 		return err;
 
-	for (i = 0; i < n - 1; i++) {
-		branch = push_stack(env, env->insn_idx + 1, env->insn_idx,
-				    env->cur_state->speculative);
-		if (IS_ERR(branch))
-			return PTR_ERR(branch);
-		mark_reg_func_ptr(env, branch->frame[branch->curframe]->regs, value_regno,
-				  targets[i]);
-	}
-	mark_reg_func_ptr(env, regs, value_regno, targets[n - 1]);
-	return 0;
+	return mark_reg_func_set(env, regs, value_regno, targets, n);
 
 overlap:
 	verbose(env, "read of %d bytes at offset [%llu,%llu] of map '%s' overlaps with a pointer to a function\n",
@@ -11609,7 +11652,8 @@ static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier
 		if (arg_needs_caller(env, state, sub, &caller->regs[i]))
 			return false;
 	}
-	for (i = 0; i < bpf_in_stack_arg_cnt(sub); i++) {
+	/* callx of one function of a set asks before it is checked that the arguments are there */
+	for (i = 0; i < bpf_in_stack_arg_cnt(sub) && i < caller->out_stack_arg_cnt; i++) {
 		if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
 		    base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
 			return false;
@@ -12044,13 +12088,93 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  * PTR_TO_FUNC register identifies the callee, so from here on callx is verified
  * as a direct call of that static subprog.
  */
+static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int *insn_idx)
+{
+	int err;
+
+	err = btf_check_subprog_call(env, subprog);
+	if (err == -EFAULT)
+		return err;
+
+	err = record_callx_edge(env, cur_func(env)->subprogno, subprog);
+	if (err)
+		return err;
+	env->subprog_info[subprog].via_callx = true;
+
+	return check_static_func_call(env, subprog, insn_idx);
+}
+
+/* The insn needs to know the function: go on with each function of the set. */
+static int split_func_set(struct bpf_verifier_env *env, int regno)
+{
+	struct bpf_reg_state *reg = reg_state(env, regno);
+	struct bpf_verifier_state *branch;
+	struct bpf_func_set *set;
+	u32 i;
+
+	if (reg->type != PTR_TO_FUNC || !reg->func_set)
+		return 0;
+	set = env->func_sets[reg->func_set - 1];
+	reg->func_set = 0;
+	for (i = 0; i < set->cnt; i++) {
+		reg->subprogno = set->subprog[i];
+		if (i == set->cnt - 1)
+			break;
+		branch = push_stack(env, env->insn_idx, env->insn_idx, false);
+		if (IS_ERR(branch))
+			return PTR_ERR(branch);
+	}
+	return 0;
+}
+
+/*
+ * callx of one of the functions of a set. A function that is verified on its own
+ * gets a copy of the state that waits for it. Nothing meets at an insn.
+ */
+static int check_func_callx_set(struct bpf_verifier_env *env, int regno, int *insn_idx)
+{
+	struct bpf_verifier_state *cur = env->cur_state, *copy, *branch;
+	struct bpf_reg_state *reg = reg_state(env, regno);
+	struct bpf_func_set *set = env->func_sets[reg->func_set - 1];
+	int err = 0;
+	u32 i;
+
+	/* once, and not in every copy */
+	account_current_path(env);
+	reg->func_set = 0;
+	for (i = 0; i < set->cnt && !err; i++) {
+		reg->subprogno = set->subprog[i];
+		if (!func_is_on_its_own(env, cur, set->subprog[i])) {
+			branch = push_stack(env, *insn_idx, *insn_idx, false);
+			if (IS_ERR(branch))
+				return PTR_ERR(branch);
+			continue;
+		}
+		copy = kzalloc_obj(*copy, GFP_KERNEL_ACCOUNT);
+		if (!copy)
+			return -ENOMEM;
+		err = bpf_copy_verifier_state(copy, cur);
+		if (!err) {
+			env->cur_state = copy;
+			err = check_func_callx_one(env, set->subprog[i], insn_idx);
+			env->cur_state = cur;
+		}
+		bpf_free_verifier_state(copy, true);
+		if (err == PROCESS_BPF_EXIT)
+			err = 0;
+		else if (verifier_bug_if(!err, env, "callx: func#%d doesn't wait", set->subprog[i]))
+			err = -EFAULT;
+	}
+	/* all that is to be done has a state of its own */
+	return err ?: PROCESS_BPF_EXIT;
+}
+
 static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			    int *insn_idx)
 {
-	struct bpf_func_state *caller = cur_func(env);
 	struct bpf_reg_state *reg;
 	const char *reason;
-	int err, subprog;
+	int err;
 
 	err = check_reg_arg(env, insn->dst_reg, SRC_OP);
 	if (err)
@@ -12083,17 +12207,9 @@ static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		return err;
 
 	/* PTR_TO_FUNC is a pointer to a static subprog */
-	subprog = reg->subprogno;
-	err = btf_check_subprog_call(env, subprog);
-	if (err == -EFAULT)
-		return err;
-
-	err = record_callx_edge(env, caller->subprogno, subprog);
-	if (err)
-		return err;
-	env->subprog_info[subprog].via_callx = true;
-
-	return check_static_func_call(env, subprog, insn_idx);
+	if (reg->func_set)
+		return check_func_callx_set(env, insn->dst_reg, insn_idx);
+	return check_func_callx_one(env, reg->subprogno, insn_idx);
 }
 
 int map_set_for_each_callback_args(struct bpf_verifier_env *env,
@@ -20932,7 +21048,7 @@ static int check_arena_sp_insn(struct bpf_verifier_env *env, struct bpf_insn *in
 
 static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 {
-	int err;
+	int err, i;
 	struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
 	u8 class = BPF_CLASS(insn->code);
 
@@ -21009,6 +21125,12 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 				return check_func_callx(env, insn, &env->insn_idx);
 			if (insn->src_reg == BPF_PSEUDO_CALL)
 				return check_func_call(env, insn, &env->insn_idx);
+			/* a callback is one function */
+			for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+				err = split_func_set(env, i);
+				if (err)
+					return err;
+			}
 			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
 				return check_kfunc_call(env, insn, &env->insn_idx);
 			return check_helper_call(env, insn, &env->insn_idx);
@@ -25519,6 +25641,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->gotox_tmp_buf);
 	kvfree(env->callx_edges);
 	kvfree(env->func_ptrs);
+	for (i = 0; i < env->func_set_cnt; i++)
+		kfree(env->func_sets[i]);
+	kfree(env->func_sets);
 	bpf_diag_free(env);
 	kvfree(env->idoms);
 	kvfree(env);
-- 
2.55.0

Re: [PATCH bpf-next v4 27/46] bpf: Track a pointer to one of a set of functions

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] bpf: unconditional R1-R5 split in do_check_insn() causes
  state explosion
--

commit 5fe8c15ac1cfee1e78fee4f0e12cfb386fffa7e1
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Track a pointer to one of a set of functions

This commit defers splitting verifier states when loading function pointers
from a read-only map table by representing the loaded register as a set of
possible functions until a callx or callback helper/kfunc requires a single
target.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 7769b4bd14673..a5c3e86f3f87f 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -21009,6 +21125,12 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
>  				return check_func_callx(env, insn, &env->insn_idx);
>  			if (insn->src_reg == BPF_PSEUDO_CALL)
>  				return check_func_call(env, insn, &env->insn_idx);
> +			/* a callback is one function */
> +			for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
> +				err = split_func_set(env, i);
> +				if (err)
> +					return err;
> +			}
>  			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
>  				return check_kfunc_call(env, insn, &env->insn_idx);
>  			return check_helper_call(env, insn, &env->insn_idx);

[Severity: Medium]
Can splitting all of R1 through R5 before every helper and kfunc call cause
an N * N state explosion or unnecessary state cloning?

When a function set of size N is loaded into R3 and copied to R2 before
calling bpf_timer_set_callback(), both R2 and R3 are marked live in
live_regs_before because bpf_timer_set_callback_proto declares a third
ARG_PTR_TO_PROG_AUX argument.  Splitting R2 and R3 independently produces
N * N combinations at the call instruction instead of the N states produced
before this patch, exceeding BPF_FLOW_MAX_STATES (1024) for N >= 33 and
rejecting the program with -E2BIG.

In addition, non-sync-callback helper and kfunc calls inside a basic block
are not flow points, so dead caller-saved registers in R1 through R5 have
not been scrubbed by bpf_clean_verifier_state() when do_check_insn() runs.
If a dead register in R1 through R5 still holds a func_set of size N,
split_func_set() pushes N - 1 states onto env->head at env->insn_idx,
inflating env->total_states and re-verifying the call and the rest of the
basic block.

Splitting only when checking an argument that expects a function pointer (such
as in process_kf_arg_ptr_to_func() and the ARG_PTR_TO_FUNC path in
check_func_arg()) would avoid splitting unrelated or dead registers.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=27

[PATCH bpf-next v4 28/46] bpf: callx reads what the subprogs whose address is taken read

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

core::fmt::write() is in each Rust prog that prints. It reads the options
of a '{}' and calls the fmt() function of the argument through a pointer:

  if n & 8 != 0  { arg_index = ...(template.cast_array().read()); }
  if n & 16 != 0 { opt.width = args.add(opt.width)...as_u16(); }
  if n & 32 != 0 { opt.precision = args.add(opt.precision)...as_u16(); }
  args.add(arg_index).as_ref().fmt(&mut Formatter::new(output, opt))?;

The last line is callx with two arguments, r1 and r2. The pass that finds
the live registers does not know the callee: callx counts as a read of
r1-r5. What the branches leave in r4 and r5 stays live:

  w4 &= 16
  if w4 == 0 goto +4          ; r4 = 0
  r3 <<= 4
  r4 = *(u64 *)(r10 - 24)     ; r4 = args, a pointer to arena
  ...
  callx r3

r4 is 0 or a pointer, r5 is a byte or a pointer. A number and a pointer
have no join: 4 states go to callx in the first pass over the loop, 2 in
each later pass, where 1 is enough. A subprog with callx in it also seems
to take 5 arguments: what its callers leave in r3-r5 is a part of its entry
state, and it is verified again for each such leftover.

callx calls only a static subprog whose address the prog takes: with
ld_imm64, or in a table in a read-only map. mark_recursive_subprogs() has
this set. Store it in addr_taken of bpf_subprog_info. The passes in flow.c
that look through callx ahead of the main pass read the bit already.
Nothing set it so far: they took no subprog for a target of callx.

  insn                        the registers of r1-r5 that it reads
  --------------------------  ---------------------------------------------
  call of a static subprog f  live at the first insn of f, as before
  callx, before               all
  callx, this patch           live at the first insn of a subprog that has
                              addr_taken, or declared in its BTF

bpf_compute_live_registers() finds both in one loop, until nothing changes.
zext_dst comes from the same result, thus check_func_callx_one() reports
a verifier bug if the callee has no addr_taken.

bpfland_stopping of scx_bpfland.rs: -13 % insns, -19 % states. 99 of the
225 progs of the schedulers of rust-bpf change: -5 % insns, -9 % states.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/liveness.c | 28 ++++++++++++++++++++++++++--
 kernel/bpf/verifier.c | 11 ++++++++++-
 2 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 318db60ae230..e930d928afa5 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2732,9 +2732,12 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			}
 			def = mask_widen(def);
 			use = mask_widen(use);
-			/* callx reads the address of the callee from dst_reg */
+			/*
+			 * callx reads the address of the callee from dst_reg. What the callee
+			 * reads is added in bpf_compute_live_registers().
+			 */
 			if (bpf_is_callx(insn))
-				use |= dst;
+				use = dst;
 			break;
 		default:
 			def = 0;
@@ -2801,6 +2804,25 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 
 	changed = true;
 	while (changed) {
+		u32 args = mask_widen(GENMASK(BPF_REG_5, BPF_REG_1)), callx_use = 0, declared;
+		bool taken = false;
+
+		/*
+		 * callx calls one of the subprogs whose address is taken, see
+		 * check_func_callx_one(). It reads what they read, as computed so far.
+		 */
+		for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+			if (!env->subprog_info[subprog].addr_taken)
+				continue;
+			callx_use |= state[env->subprog_info[subprog].start].in & args;
+			/* and what its BTF declares: the BTF is unreliable when one is not set */
+			declared = env->subprog_info[subprog].arg_slot_cnt;
+			callx_use |= mask_widen(GENMASK(declared, 1)) & args;
+			taken = true;
+		}
+		if (!taken)
+			callx_use = args;
+
 		changed = false;
 		/* a cycle of calls takes a round for each link */
 		if (signal_pending(current)) {
@@ -2829,6 +2851,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			    !bpf_subprog_is_global(env, callee_subprog(env, &insns[insn_idx])))
 				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
 				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
+			if (bpf_is_callx(&insns[insn_idx]))
+				use |= callx_use;
 			new_in = (new_out & ~live->def) | use;
 			/* may_goto that is added at the head of a loop goes there */
 			t = bpf_head_guard_target(env, insn_idx);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a5c3e86f3f87..a3095efb1ee0 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6116,6 +6116,8 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env)
 	}
 	for_each_set_bit(s, callx, cnt)
 		bitmap_or(calls + s * words, calls + s * words, taken, cnt);
+	for_each_set_bit(s, taken, cnt)
+		si[s].addr_taken = true;
 
 	/*
 	 * Callers of a global subprog look at can_unwind before the subprog is verified.
@@ -11377,7 +11379,8 @@ static void gen_subprog_arg_proto(const struct bpf_subprog_info *sub, const stru
 
 /*
  * The argument slots that the code of a static subprog takes, whatever its BTF says.
- * Liveness errs on the safe side: a callx in the subprog counts as a read of r1-r5.
+ * Liveness errs on the safe side: a callx in the subprog counts as a read of all that the subprogs
+ * whose address is taken read or declare.
  * But it misses r3-r5 of bpf_trace_printk() and the flags of bpf_get_branch_snapshot(), which
  * their protos do not declare. Such a subprog prints or passes a wrong number under a tracer.
  * The helpers are safe with any value there: the verifier never checks these registers.
@@ -12096,6 +12099,12 @@ static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int *
 	if (err == -EFAULT)
 		return err;
 
+	/* bpf_compute_live_registers() counts on it */
+	if (verifier_bug_if(!env->subprog_info[subprog].addr_taken, env,
+			    "callx of %s() whose address is not taken",
+			    bpf_subprog_name(env, subprog)))
+		return -EFAULT;
+
 	err = record_callx_edge(env, cur_func(env)->subprogno, subprog);
 	if (err)
 		return err;
-- 
2.55.0

[PATCH bpf-next v4 29/46] bpf: Keep the first iteration of a counted loop apart

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  idx = 0;
  for (i = 0; i < 100000; i++)
          idx = table[idx];       /* u8 table[256] */

Scalar evolution has the count and no formula for idx. If something looks
at such a register, like the load here, the verifier goes through the
iterations one by one. 100000 iterations are too many for that. Then what
returns to the loop head is joined with the 0 and widened: the bound that
grew goes to a fixed stop, and a prog that loaded before is rejected:

  R0 unbounded memory access, make sure to bounds check any such access

But nothing that idx is made from is what a register had at the loop head.
It is a byte in each iteration. Only the first one differs: it has the 0
from the entry.

Verify the first iteration with the state that enters the loop. The state
that returns to the loop head has idx in [0, 255]. Give its induction
variables the ranges that follow from the count, as at the entry. It stands
for all other iterations. What returns next must be within it, otherwise
the two are joined like at each loop head. The loop takes 12 insns.

expr_is_fresh() finds the registers, bpf_loop_iters.fresh_regs has them,
loop_stack_entry.first marks a state that is in the first iteration.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h | 15 ++++++--
 kernel/bpf/flow.c            |  4 +--
 kernel/bpf/join.c            |  1 +
 kernel/bpf/scev.c            | 68 ++++++++++++++++++++++++++++++++++--
 kernel/bpf/states.c          |  1 +
 kernel/bpf/verifier.c        | 33 ++++++++++++++++-
 6 files changed, 114 insertions(+), 8 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 098b3c30c8b2..ae57853e7cc0 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -351,6 +351,11 @@ struct bpf_loop_iters {
 	u32 max_header_count;   /* max number of times header is executed */
 	/* live, changed by the loop, no summary of how: left to the join at the head */
 	unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/*
+	 * live, made anew in every iteration: the first one tells what it is in the rest.
+	 * Or not set at the entry, and also in forget_regs: the first one sets it.
+	 */
+	unsigned long fresh_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/* [0, max] at the entry, and the loop only clears bits */
 	unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	/*
@@ -395,6 +400,11 @@ struct loop_stack_entry {
 	 * flow_join_makes_up(). The test of the flag is decided in both, and decides nothing.
 	 */
 	u32 flag_apart:1;
+	/*
+	 * The first iteration of a counted loop is kept apart. The state that comes back
+	 * to the head is made to stand for the rest.
+	 */
+	u32 first:1;
 	/*
 	 * How many times the state came back to the head of the loop. With @apart that is
 	 * the iteration that it is in. A state that stands for many iterations is a join, and has
@@ -421,7 +431,7 @@ struct loop_stack_entry {
 /* The state is in a walk on patience, which may not get to the end of the loop */
 static inline bool bpf_loop_walk_may_give_up(const struct loop_stack_entry *entry)
 {
-	return entry->apart && entry->terminates && !entry->counted;
+	return entry->apart && entry->terminates && !entry->counted && !entry->first;
 }
 
 #define LOOP_STACK_SIZE 16
@@ -2181,7 +2191,8 @@ int bpf_compute_scev(struct bpf_verifier_env *env);
 int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
 			   struct bpf_loop_iters *iters);
 int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
-			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters);
+			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters,
+			bool back);
 int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 insn_idx,
 			const struct loop_stack_entry *entry);
 
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index 866835eef0cc..a286f43594cd 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1891,8 +1891,8 @@ static void flow_shape_ctx(const struct bpf_verifier_state *st, struct flow_shap
 		for (i = 0; i < frame->loop_stack_cnt; i++) {
 			entry = &frame->loop_stack[i];
 			chain = jhash_1word(entry->loop_id << 1 | entry->apart, chain);
-			ctx = jhash_3words(entry->counted << 2 | entry->guarded << 1 |
-					   entry->terminates,
+			ctx = jhash_3words(entry->counted << 3 | entry->guarded << 2 |
+					   entry->first << 1 | entry->terminates,
 					   entry->max_header_count,
 					   entry->counted ? entry->trip : 0, ctx);
 		}
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 0110e5f85003..88fe3d32a6de 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -415,6 +415,7 @@ static bool join_loop_stack(const struct bpf_func_state *fa, const struct bpf_fu
 		    fa->loop_stack[i].guarded != fb->loop_stack[i].guarded ||
 		    fa->loop_stack[i].counted != fb->loop_stack[i].counted ||
 		    fa->loop_stack[i].apart != fb->loop_stack[i].apart ||
+		    fa->loop_stack[i].first != fb->loop_stack[i].first ||
 		    (fa->loop_stack[i].apart && fa->loop_stack[i].trip != fb->loop_stack[i].trip) ||
 		    fa->loop_stack[i].refs != fb->loop_stack[i].refs ||
 		    fa->loop_stack[i].max_header_count != fb->loop_stack[i].max_header_count ||
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 9377b0907f21..d8c847efbd7f 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2024,7 +2024,7 @@ scev_reg_for_read(struct bpf_verifier_env *env, const struct bpf_func_state *st,
  * This does not go through check_stack_write(). The buffer of a live dynptr slice, see
  * bpf_reg_state::buf, does not become a spill here. The callers skip the slots that are
  * is_unbound_scev_reg() when the loop is entered. A slot that a call in the loop may write
- * has no description: it is in forget_regs, and they skip these too.
+ * has no description: it is in forget_regs or fresh_regs, and they skip these too.
  */
 static int scev_reg_for_write(struct bpf_verifier_env *env, struct bpf_func_state *st,
 			      u32 r, struct bpf_reg_state **out)
@@ -3252,6 +3252,31 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
 		bounds->r64 = r64;
 }
 
+/*
+ * Nothing that @id is computed from is what a register had at the head of the loop, and
+ * some of it is not known: it's made anew in every iteration.
+ */
+static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
+{
+	struct scev *scev = env->scev;
+	bool opaque = false;
+	u32 order, regno;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if (order & DEPTH_LIMIT)
+			return false;
+		if (!(order & PRE))
+			continue;
+		if (is_reg(scev, id, &regno) || is_op(scev, id, UNKNOWN))
+			return false;
+		if (is_opaque(scev, id))
+			opaque = true;
+	}
+	return opaque;
+}
+
 /*
  * Does @id have no bit that register @r doesn't have at the head of the loop? 'x & e' and the
  * low bits of x lose bits only. 'x >> e', 'x / e' and 'x % e' are not greater than x as well, but
@@ -3465,6 +3490,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 		return err;
 
 	bitmap_zero(iters->forget_regs, REGS_NUM);
+	bitmap_zero(iters->fresh_regs, REGS_NUM);
 	bitmap_zero(iters->mask_regs, REGS_NUM);
 	iters->spill_base = false;
 	header_env = find_header_env(scev, insn_idx);
@@ -3578,7 +3604,23 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 			walk = true;
 			continue;
 		}
+		/*
+		 * What is loaded from memory in every iteration is the same kind of thing in all
+		 * of them. One iteration tells what. There is nothing more to find out when
+		 * iterations are looked at one by one.
+		 */
+		if (expr_is_fresh(env, header_env->reg2expr[r])) {
+			__set_bit(r, iters->fresh_regs);
+			continue;
+		}
 		__set_bit(r, iters->forget_regs);
+		/*
+		 *	for (i = 0; i < 40; i++) { if (i) sum += prev; prev = (x ^ i) * 3; }
+		 * It's not set in the first iteration only. That counts if the loop is not walked:
+		 * nothing looks at prev, or there are too many iterations.
+		 */
+		if (reg && reg->type == NOT_INIT)
+			__set_bit(r, iters->fresh_regs);
 		/*
 		 * Only a walk through the iterations tells what it is in each of them. That is not
 		 * worth it for a number that nothing looks at: a sum, a hash, what a call leaves in
@@ -3643,13 +3685,18 @@ static int union_any_reg(struct bpf_verifier_env *env, struct bpf_func_state *lo
 /*
  * Make @st, which is at the head of a counted loop, stand for all iterations. @loop_entry is the
  * state that entered the loop and @iters is what bpf_compute_loop_iters() says about it.
- * @st is a copy of @loop_entry.
+ *
+ * @st is a copy of @loop_entry, or, if the loop has iters->fresh_regs, the state that comes back to
+ * the head after the first iteration, @back. These registers are left as the first iteration
+ * made them. At the entry they are something else: 0, a pointer, not set.
  */
 int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
-			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters)
+			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters,
+			bool back)
 {
 	struct bpf_func_state *cur_func = st->frame[st->curframe];
 	struct bpf_func_state *entry_func = loop_entry->frame[loop_entry->curframe];
+	const struct bpf_reg_state *val, *entry_val;
 	struct bpf_verifier_log *log = &env->log;
 	struct scev *scev = env->scev;
 	struct bpf_reg_state *reg, *spill32;
@@ -3678,6 +3725,8 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 			}
 			continue;
 		}
+		if (test_bit(r, iters->fresh_regs))
+			continue;
 
 		r_expr = header_env->reg2scev[r];
 		spill32 = spill32_counter(scev, cur_func, r_expr, r, &coefs);
@@ -3686,9 +3735,19 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 				 is_const_scev_reg(env, insn_idx, r)))
 			continue;
 
+		/*
+		 * A call in the loop can change the type, see bpf_clamp_scev_regs(). Then the
+		 * register is left alone. What comes back to the head next is compared with it.
+		 */
+		val = scev_reg_for_read(env, cur_func, r);
+		entry_val = scev_reg_for_read(env, entry_func, r);
+		if (back && (!val || !entry_val || !bpf_reg_union_compatible(val, entry_val)))
+			continue;
 		if (test_bit(r, iters->mask_regs)) {
 			/* The range stays. It's no longer equal to the registers with its id. */
 			err = scev_reg_for_write(env, cur_func, r, &reg);
+			if (!err && back)
+				err = bpf_reg_union(env, reg, entry_val);
 			if (err)
 				return err;
 			if (log->level & BPF_LOG_LEVEL2) {
@@ -3704,6 +3763,9 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 
 		/* If SCEV for r is (linear <reg> <slope>)*/
 		if (spill32 || (is_simple_linear(scev, r_expr, &coefs) && coefs.base_reg == r)) {
+			/* the range is derived from @loop_entry: bpf_clamp_scev_regs() */
+			if (back)
+				continue;
 			/* the other half of the slot stays as it is */
 			reg = spill32;
 			err = reg ? 0 : scev_reg_for_write(env, cur_func, r, &reg);
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 07b1f0becd2e..efc0870b85c2 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -646,6 +646,7 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
 		    old_stack[i].guarded == cur_stack[i].guarded &&
 		    old_stack[i].counted == cur_stack[i].counted &&
 		    old_stack[i].apart == cur_stack[i].apart &&
+		    old_stack[i].first == cur_stack[i].first &&
 		    (!old_stack[i].apart ||
 		     (old_stack[i].probe ? old_stack[i].trip <= cur_stack[i].trip :
 					   old_stack[i].trip == cur_stack[i].trip)))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a3095efb1ee0..1e97444f0279 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21511,6 +21511,26 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 			*backedge = true;
 			return check_refs_taken_in_loop(env, entry) ?: maybe_clamp_scev_regs(env);
 		}
+		/*
+		 * The first iteration is over. What the loop makes anew in every iteration is
+		 * in this state. The rest is what the count says, like at the entry.
+		 */
+		if (entry->first) {
+			entry->first = false;
+			entry->apart = false;
+			entry->trip++;
+			err = bpf_compute_loop_iters(env, entry->entry_state, &iters);
+			if (err < 0)
+				return err;
+			if (verifier_bug_if(!err, env, "loop at %d is not counted any more",
+					    env->insn_idx))
+				return -EFAULT;
+			err = maybe_clamp_scev_regs(env);
+			if (err)
+				return err;
+			err = bpf_widen_scev_regs(env, cur, entry->entry_state, &iters, true);
+			return err < 0 ? err : 0;
+		}
 		loop = env->insn_aux_data[env->insn_idx].loop;
 		/* the next iteration is another kind of state, there is no cycle */
 		trip = entry->trip;
@@ -21593,7 +21613,18 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	entry->iv_range = iters.iv_range;
 	entry->entry_state = entry_state;
 	entry->terminates = true;
-	err = bpf_widen_scev_regs(env, cur, entry_state, &iters);
+	/*
+	 *	idx = 0; for (i = 0; i < 100; i++) idx = table[idx];
+	 * What the loop makes of idx is not known before its body is looked at, and what idx is
+	 * at the entry goes with i == 0 only. So the state goes through the first iteration as
+	 * it is. The state that comes back to the head has idx in 0..255.
+	 */
+	if (!bitmap_empty(iters.fresh_regs, BPF_SCEV_REGS_NUM)) {
+		entry->first = true;
+		entry->apart = true;
+		return 0;
+	}
+	err = bpf_widen_scev_regs(env, cur, entry_state, &iters, false);
 	return err < 0 ? err : 0;
 }
 
-- 
2.55.0

Re: [PATCH bpf-next v4 29/46] bpf: Keep the first iteration of a counted loop apart

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] bpf: false rejection of counted loops due to expr_is_fresh()
  on loads
--

commit 161a1c1a744b39f1b6247cc6557f53d0f033b91f
Author: Alexei Starovoitov <ast@kernel.org>

bpf: Keep the first iteration of a counted loop apart

This commit verifies the first iteration of a counted loop separately when
live registers are computed anew in each iteration, using the state returning
to the loop header after the first iteration to stand for the remaining
iterations.

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 9377b0907f215..d8c847efbd7f3 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
[ ... ]
> @@ -3578,7 +3604,23 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
>  			walk = true;
>  			continue;
>  		}
> +		/*
> +		 * What is loaded from memory in every iteration is the same kind of thing in all
> +		 * of them. One iteration tells what. There is nothing more to find out when
> +		 * iterations are looked at one by one.
> +		 */
> +		if (expr_is_fresh(env, header_env->reg2expr[r])) {
> +			__set_bit(r, iters->fresh_regs);
> +			continue;
> +		}

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v3 [1], can continuing here without
setting walk = true cause a regression where the verifier falsely rejects
valid counted-loop programs?

Every non-FP memory load gets OPAQUE_EXPR_ID in maybe_load_fp(), so
expr_is_fresh() marks any loaded register as fresh and bypasses the walk
check below, returning 1 instead of 2 from bpf_compute_loop_iters().

For a loop that indexes a BPF_F_RDONLY_PROG map (for example, idx =
rodata_table[idx]), the offset in the first iteration (entry->first == true)
is still a known constant, so check_map_mem_read() constant-folds idx to
rodata_table[0] instead of [0, 255].  When the first iteration returns to
the loop header, bpf_flow_loop_enter() clears entry->first and entry->apart
and leaves idx at that constant.  The second iteration then joins against
that state with widening enabled in flow_merge() -> bpf_states_join(), which
widens the upper bound of idx to S32_MAX whenever the new value exceeds
rodata_table[0] and causes subsequent bounds-checked array accesses to be
rejected.

Similarly, for a short counted loop that loads a nullable kptr into t and
tracks it with a scalar flag (have = 0; if (t && t->y > 100) have = 1;),
skipping the per-iteration walk causes bpf_widen_scev_regs() to widen have to
[0, 1] via union_any_reg() on the t == NULL path after the first iteration,
so a subsequent if (have) sum += t->x in the next iteration fails on a
NULL pointer dereference.

[1] https://lore.kernel.org/r/19e3280680aa046388274873789d649eac600b99eff2633376ba2a073fd30acd@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=29

[PATCH bpf-next v4 30/46] bpf: What a call returns in each iteration is like what the loop loads

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  for (i = 0; i < 20000; i++) {
          sum += a[x];
          x = bpf_get_prandom_u32() & 15;
  }

x is loop-carried: an iteration uses what the iteration before made. The
loop is counted, and too long to go through its iterations one by one.
Scalar evolution knows nothing about r0 after a call, so x is widened at
the loop head. The prog loaded before states were joined:

  R0 unbounded memory access, make sure to bounds check any such access

A value that the loop loads from memory in each iteration is handled: the
first iteration is verified on its own, and tells what the value is in the
others. What a call returns is made anew in each iteration in the same way.

Give r0 after a call the expression that a load gets, with a parameter that
tells the two apart. A call can return NULL in one iteration and set a flag
with it, so a loop that is short enough is still walked first.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/scev.c | 31 ++++++++++++++++++++-----------
 1 file changed, 20 insertions(+), 11 deletions(-)

diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index d8c847efbd7f..06f485853df1 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -852,8 +852,11 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
 		break;
 	case BPF_JMP:
 	case BPF_JMP32:
-		if (opcode == BPF_CALL)
+		if (opcode == BPF_CALL) {
 			forget_call_regs(e);
+			/* made anew, like what is loaded. The parameter tells the two apart. */
+			return replace_reg(scev, e, BPF_REG_0, expr1(scev, OPAQUE, 1));
+		}
 		/* for non-CALL there are no changes in register states */
 		break;
 	case BPF_LD:
@@ -3254,14 +3257,15 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
 
 /*
  * Nothing that @id is computed from is what a register had at the head of the loop, and
- * some of it is not known: it's made anew in every iteration.
+ * some of it is not known: it's made anew in every iteration. @call: by a call.
  */
-static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
+static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id, bool *call)
 {
 	struct scev *scev = env->scev;
 	bool opaque = false;
 	u32 order, regno;
 
+	*call = false;
 	scev->stack_sz = 0;
 	expr_stack_push(scev, id);
 	while (expr_next(scev, &id, &order)) {
@@ -3271,8 +3275,10 @@ static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
 			continue;
 		if (is_reg(scev, id, &regno) || is_op(scev, id, UNKNOWN))
 			return false;
-		if (is_opaque(scev, id))
+		if (is_opaque(scev, id)) {
 			opaque = true;
+			*call |= scev->exprs[id].params[0];
+		}
 	}
 	return opaque;
 }
@@ -3402,7 +3408,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	struct bpf_loop *loop;
 	int insn_idx = st->insn_idx;
 	u64 until, false_until;
-	bool same = true, walk = false;
+	bool same = true, walk = false, call;
 	int err, i;
 	u32 r;
 
@@ -3605,15 +3611,18 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 			continue;
 		}
 		/*
-		 * What is loaded from memory in every iteration is the same kind of thing in all
-		 * of them. One iteration tells what. There is nothing more to find out when
-		 * iterations are looked at one by one.
+		 * What is loaded from memory or returned by a call in every iteration is the same
+		 * kind of thing in all of them. One iteration tells what. There is nothing more
+		 * to find out when iterations are looked at one by one. But for what goes with it:
+		 * a call returns NULL and a flag is set. So a walk comes first for a call.
 		 */
-		if (expr_is_fresh(env, header_env->reg2expr[r])) {
+		if (expr_is_fresh(env, header_env->reg2expr[r], &call)) {
 			__set_bit(r, iters->fresh_regs);
-			continue;
+			if (!call)
+				continue;
+		} else {
+			__set_bit(r, iters->forget_regs);
 		}
-		__set_bit(r, iters->forget_regs);
 		/*
 		 *	for (i = 0; i < 40; i++) { if (i) sum += prev; prev = (x ^ i) * 3; }
 		 * It's not set in the first iteration only. That counts if the loop is not walked:
-- 
2.55.0

Re: [PATCH bpf-next v4 30/46] bpf: What a call returns in each iteration is like what the loop loads

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] bpf: missing call OPAQUE handling in
  made_from_partial_spill()
--

commit 17f294cedba72c038cc095f26f37091b91a7fbf0
Author: Alexei Starovoitov <ast@kernel.org>

bpf: What a call returns in each iteration is like what the loop loads

This commit assigns an OPAQUE scalar evolution expression with parameter 1
to r0 after a BPF_CALL, allowing loop registers updated from call return
values to be marked in fresh_regs after walking short loops.

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index d8c847efbd7f3..06f485853df13 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
> @@ -852,8 +852,11 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
>  		break;
>  	case BPF_JMP:
>  	case BPF_JMP32:
> -		if (opcode == BPF_CALL)
> +		if (opcode == BPF_CALL) {
>  			forget_call_regs(e);
> +			/* made anew, like what is loaded. The parameter tells the two apart. */
> +			return replace_reg(scev, e, BPF_REG_0, expr1(scev, OPAQUE, 1));
> +		}

[Severity: Medium]
Does made_from_partial_spill() also need to treat a call's OPAQUE
expression like UNKNOWN?

Before this change, BPF_CALL left r0 as UNKNOWN_EXPR_ID, so a known live
partial stack spill updated from a subprogram or helper return value had
made_from_partial_spill() return true and jumped to cant_widen in
bpf_compute_loop_iters():

kernel/bpf/scev.c:made_from_partial_spill() {
	const struct expr *e = &scev->exprs[id];
	u32 i;

	if (lvl >= EXPR_STACK_DEPTH || e->op == UNKNOWN)
		return true;
    ...
}

kernel/bpf/scev.c:bpf_compute_loop_iters() {
    ...
		if (is_unbound_scev_reg(env, cur_func, r)) {
			if (is_known_partial_spill(cur_func, r) &&
			    bpf_stack_slot_alive(env, st->curframe, (r - MAX_BPF_REG) * 2 + 1) &&
			    made_from_partial_spill(scev, cur_func, header_env,
						    header_env->reg2expr[r], r, 0))
				goto cant_widen;
			continue;
		}
    ...
}

Because op_params_num(OPAQUE) is 0 and e->op is no longer UNKNOWN,
made_from_partial_spill() now returns false when a partial spill is set
from r0 after a call.  bpf_compute_loop_iters() then skips cant_widen,
leaving walk false and the slot out of fresh_regs, which causes join_slot()
to widen the partial spill at the loop head and causes a regression where
valid programs with bounded memory accesses using that spill are rejected.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=30

[PATCH bpf-next v4 31/46] bpf: Find where the jumps of a loop stop a number

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  bpf_for(i, 0, 100)
          bpf_for(j, 0, 100)
                  if (n < 8)
                          v->a[n++] = j;
  return v->a[n];

  while (off < 8000) { l = v->big[off]; if (!l) break; off += l; }

n and off grow by what the data say, so there is no formula for them.
Widening moves the bound that grows to a fixed stop: 0, S32_MAX, U32_MAX or
S64_MAX. n is [0, S32_MAX] at the loop head. The jump limits it in the
loop, nothing limits it after the loop. clang tests off at the end of the
loop, so nothing limits it at the load. Both loaded before states were
joined:

  R6 unbounded memory access, make sure to bounds check any such access

The jumps of the loop tell where the bound stops. Examine one iteration
before the main pass, from the head to the back edges, and write what is in
a register or a stack slot as 'root + constant'. A root is the value at the
head, or a value that the iteration makes. A jump that compares 'root + c'
with a constant limits the root on each edge that stays in the loop. What
returns to the head is 'root + c' again, and the limits give a stop: 8 and
7999 here. bpf_reg_widen_range() takes the nearest of all stops.

A stop is a guess. What returns to the head next is compared with the
widened state as always. A wrong stop costs one more pass through the loop.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  24 +-
 kernel/bpf/fixups.c          |   1 +
 kernel/bpf/flow.c            | 438 ++++++++++++++++++++++++++++++++++-
 kernel/bpf/join.c            |  25 +-
 kernel/bpf/verifier.c        |  36 ++-
 5 files changed, 504 insertions(+), 20 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index ae57853e7cc0..85404ef790ef 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -728,6 +728,23 @@ struct bpf_loop_exit {
 	int to; /* instruction outside the loop */
 };
 
+/* The places that the table of a loop knows: registers, then stack slots, fp-8 first */
+#define BPF_LOOP_SLOT	64
+#define BPF_LOOP_PLACES	128
+
+/*
+ * Where the bounds of the number in @place may come to rest at the head of a loop: the lowest and
+ * the highest upper bound, the highest and the lowest lower bound that the jumps of the loop
+ * let come back to the head, see flow_compute_rungs().
+ */
+struct bpf_loop_rungs {
+	s64 max[2];
+	s64 min[2];
+	bool has_max;
+	bool has_min;
+	u8 place;
+};
+
 struct bpf_loop {
 	struct bpf_backedge backedges[MAX_BACKEDGES];
 	/* edges exiting from this loop, includes edges from nested loops */
@@ -774,6 +791,10 @@ struct bpf_loop {
 	 */
 	unsigned long linear_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 	unsigned long linear32_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+	/* the places that insns of the loop write, as far as the insns tell */
+	unsigned long writes[BITS_TO_LONGS(BPF_LOOP_PLACES)];
+	struct bpf_loop_rungs *rungs;
+	u32 rungs_cnt;
 };
 
 struct bpf_insn_aux_data {
@@ -2032,7 +2053,8 @@ int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
 void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src);
 void bpf_reg_forget_range(struct bpf_reg_state *reg);
-void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear);
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old,
+			 const struct bpf_loop_rungs *rungs, int linear);
 int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
 		  const struct bpf_reg_state *src);
 
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 3413c7addca8..108b2aa924a9 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -928,6 +928,7 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
 	for (i = start; i < end; i++) {
 		if (aux_data[i].loop) {
 			kvfree(aux_data[i].loop->exits);
+			kvfree(aux_data[i].loop->rungs);
 			kvfree(aux_data[i].loop);
 			aux_data[i].loop = NULL;
 		}
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
index a286f43594cd..dc3f2f0afaea 100644
--- a/kernel/bpf/flow.c
+++ b/kernel/bpf/flow.c
@@ -1192,6 +1192,439 @@ bool bpf_flow_is_needed(struct bpf_verifier_env *env, int idx, u32 r)
 	return false;
 }
 
+/*
+ * A number that grows from one iteration of a loop to the next is not followed step by step. The
+ * bound that moved is put where it may come to rest, see bpf_reg_widen_range(). The jumps of
+ * the loop tell where that is:
+ *	do { a[i] = 0; i += 3; } while (i < 30);		29 at most comes back to the head
+ *	while (i < 30) { a[i] = 0; i += 3; }			32
+ *	loop { if (...) { if (n >= 64) break; a[n++] = x; } }	64
+ *
+ * One iteration is looked at, from the head to the jumps back. What a register or a stack slot
+ * holds is written as root + constant. A root is what a place held at the head, or what was made
+ * in a place since. A copy, an extension, a spill and a fill keep the root. += and -= of a constant
+ * change the constant. Everything else makes a new root. Where paths meet two constants become
+ * a range of them, and two roots a new one.
+ * A jump that compares root + c with a constant bounds the root on each of its edges that stays in
+ * the loop. What comes back to the head in a place is root + c again: that, with the bounds of
+ * the root, is where the place may come to rest.
+ *
+ * It's a guess. The state that comes back to the head is compared with it like with any other
+ * state. So it's fine to take a 32-bit add for an add, to miss a store through a pointer, to mix up
+ * two numbers that are made in one place, and to know nothing of the edges that the verifier adds.
+ */
+struct flow_sym {
+	s32 lo, hi;
+	u8 root;
+};
+
+#define FLOW_NO_PLACE	U8_MAX
+/* How many of them the look at one loop may take. A loop that needs more has no table. */
+#define BPF_FLOW_MAX_SYMS	(4 << 20)
+
+static_assert(FLOW_SLOT == BPF_LOOP_SLOT && FLOW_PLACES == BPF_LOOP_PLACES);
+
+/* @insn makes the place *@to a copy of the place *@from, extended or cut to its low bits */
+static bool flow_copies_place(const struct bpf_insn *insn, u8 *to, u8 *from)
+{
+	u8 class = BPF_CLASS(insn->code);
+	int slot;
+
+	if (class == BPF_ALU || class == BPF_ALU64) {
+		*to = insn->dst_reg;
+		*from = insn->src_reg;
+		return flow_is_copy(insn) && *to < BPF_REG_FP && *from < BPF_REG_FP;
+	}
+	if (class != BPF_LDX && class != BPF_STX)
+		return false;
+	slot = flow_fp_slot(insn, class == BPF_LDX ? insn->src_reg : insn->dst_reg);
+	if (slot < 0)
+		return false;
+	*to = class == BPF_LDX ? insn->dst_reg : FLOW_SLOT + slot;
+	*from = class == BPF_LDX ? FLOW_SLOT + slot : insn->src_reg;
+	return *to != BPF_REG_FP && *from != BPF_REG_FP;
+}
+
+/* An ALU insn that leaves a small number as it is: an extension, or half of one */
+static bool flow_extends(const struct bpf_insn *insn)
+{
+	if (BPF_SRC(insn->code) != BPF_K)
+		return false;
+	if (flow_is_shift(insn))
+		return insn->imm == 32 || insn->imm == 48 || insn->imm == 56;
+	return BPF_OP(insn->code) == BPF_AND &&
+	       (insn->imm == 0xff || insn->imm == 0xffff || insn->imm == -1);
+}
+
+static void flow_written_places(const struct bpf_insn *insn, unsigned long *places)
+{
+	u8 class = BPF_CLASS(insn->code);
+	int p;
+
+	bitmap_zero(places, FLOW_PLACES);
+	if (class == BPF_ALU || class == BPF_ALU64 || class == BPF_LDX || bpf_is_ldimm64(insn)) {
+		__set_bit(insn->dst_reg, places);
+	} else if (class == BPF_LD || (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL)) {
+		bitmap_set(places, BPF_REG_0, BPF_REG_5 + 1);
+	} else if (class == BPF_ST || class == BPF_STX) {
+		p = bpf_atomic_load_reg(insn);
+		if (p >= 0)
+			__set_bit(p, places);
+		p = flow_mem_place(insn, insn->dst_reg);
+		if (p >= FLOW_SLOT)
+			__set_bit(p, places);
+	}
+}
+
+/*
+ * The jump at @idx compares the register *@reg with a constant: it's taken if "reg op k".
+ * The constant has 32 bits, so does what is added to it below: nothing wraps.
+ */
+static bool flow_jump_cmp(struct bpf_verifier_env *env, int idx, u8 *reg, u8 *op, s64 *k)
+{
+	const struct bpf_insn_aux_data *aux = &env->insn_aux_data[idx];
+	const struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code);
+
+	*op = BPF_OP(insn->code);
+	if ((class != BPF_JMP && class != BPF_JMP32) || *op == BPF_JA || *op == BPF_CALL ||
+	    *op == BPF_EXIT || *op == BPF_JCOND || *op == BPF_JSET)
+		return false;
+	*reg = insn->dst_reg;
+	if (BPF_SRC(insn->code) == BPF_K) {
+		*k = insn->imm;
+	} else if (aux->const_reg_mask & BIT(insn->src_reg)) {
+		*k = aux->const_reg_vals[insn->src_reg];
+	} else if (aux->const_reg_mask & BIT(insn->dst_reg)) {
+		*k = aux->const_reg_vals[insn->dst_reg];
+		*reg = insn->src_reg;
+		*op = bpf_flip_opcode(*op);
+	} else {
+		return false;
+	}
+	if (class == BPF_JMP32)
+		*k = *op == BPF_JSGT || *op == BPF_JSGE || *op == BPF_JSLT || *op == BPF_JSLE ?
+		     (s64)(s32)*k : (s64)(u32)*k;
+	return *reg < BPF_REG_FP;
+}
+
+static void flow_add_max(struct bpf_loop_rungs *r, s64 max)
+{
+	r->max[0] = r->has_max ? min(r->max[0], max) : max;
+	r->max[1] = r->has_max ? max(r->max[1], max) : max;
+	r->has_max = true;
+}
+
+static void flow_add_min(struct bpf_loop_rungs *r, s64 min)
+{
+	r->min[0] = r->has_min ? max(r->min[0], min) : min;
+	r->min[1] = r->has_min ? min(r->min[1], min) : min;
+	r->has_min = true;
+}
+
+/* "x op k" holds. What does that say about the root of @x? */
+static void flow_note_cmp(struct bpf_loop_rungs *root, const struct flow_sym *x, u8 op, s64 k)
+{
+	bool has_max = true, has_min = true;
+	s64 max = k, min = k;
+
+	switch (op) {
+	case BPF_JEQ:
+		break;
+	case BPF_JNE:
+		/* it cuts off the end of a range */
+		max = k - 1;
+		min = k + 1;
+		break;
+	case BPF_JLT:
+		max = k - 1;
+		fallthrough;
+	case BPF_JLE:
+		/* the ranges are signed: nothing is said by a constant above S64_MAX */
+		min = 0;
+		has_max = max >= 0;
+		has_min = max >= 0;
+		break;
+	case BPF_JGT:
+		min = k + 1;
+		fallthrough;
+	case BPF_JGE:
+		has_max = false;
+		has_min = k >= 0;
+		break;
+	case BPF_JSLT:
+		max = k - 1;
+		fallthrough;
+	case BPF_JSLE:
+		has_min = false;
+		break;
+	case BPF_JSGT:
+		min = k + 1;
+		fallthrough;
+	case BPF_JSGE:
+		has_max = false;
+		break;
+	default:
+		return;
+	}
+	if (has_max)
+		flow_add_max(root, max - x->lo);
+	if (has_min)
+		flow_add_min(root, min - x->hi);
+}
+
+/* @cur is what the @n places hold in front of @insn. Make it what they hold behind it. */
+static void flow_sym_step(const struct bpf_insn *insn, struct flow_sym *cur, const u8 *idx, u32 n)
+{
+	u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code), to, from;
+	DECLARE_BITMAP(written, FLOW_PLACES);
+	struct flow_sym *x;
+	s64 c = insn->imm;
+	int p;
+
+	/* both are among the places or none is, see flow_compute_loop_rungs() */
+	if (flow_copies_place(insn, &to, &from)) {
+		if (idx[to] != FLOW_NO_PLACE)
+			cur[idx[to]] = cur[idx[from]];
+		return;
+	}
+	if ((class == BPF_ALU || class == BPF_ALU64) && idx[insn->dst_reg] != FLOW_NO_PLACE) {
+		x = &cur[idx[insn->dst_reg]];
+		if (flow_extends(insn))
+			return;
+		if (op == BPF_SUB)
+			c = -c;
+		if ((op == BPF_ADD || op == BPF_SUB) && BPF_SRC(insn->code) == BPF_K &&
+		    x->lo + c >= S32_MIN && x->hi + c <= S32_MAX) {
+			x->lo += c;
+			x->hi += c;
+			return;
+		}
+	}
+	flow_written_places(insn, written);
+	for_each_set_bit(p, written, FLOW_PLACES)
+		if (idx[p] != FLOW_NO_PLACE)
+			cur[idx[p]] = (struct flow_sym){ .root = n + idx[p] };
+}
+
+/* One more path leads to where @to holds */
+static void flow_sym_meet(struct flow_sym *to, const struct flow_sym *from, u32 n)
+{
+	u32 j;
+
+	if (to[0].root == FLOW_NO_PLACE) {
+		memcpy(to, from, n * sizeof(*to));
+		return;
+	}
+	for (j = 0; j < n; j++) {
+		if (to[j].root != from[j].root) {
+			to[j] = (struct flow_sym){ .root = n + j };
+			continue;
+		}
+		to[j].lo = min(to[j].lo, from[j].lo);
+		to[j].hi = max(to[j].hi, from[j].hi);
+	}
+}
+
+static u8 flow_place_set(u8 *set, u8 p)
+{
+	while (set[p] != p) {
+		set[p] = set[set[p]];
+		p = set[p];
+	}
+	return p;
+}
+
+/* @at: the insns in the order. The insns of a loop are next to each other, the head first. */
+static int flow_compute_loop_rungs(struct bpf_verifier_env *env, const u32 *at, u32 total, int h)
+{
+	const struct bpf_insn *insns = env->prog->insnsi, *insn;
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop_rungs *roots = NULL, *rungs, *r;
+	u8 set[FLOW_PLACES], idx[FLOW_PLACES], to, from, reg, op;
+	u32 first = env->flow->pos[h], nr, n = 0, back_cnt = 0, cnt = 0, j, k;
+	DECLARE_BITMAP(tested, FLOW_PLACES) = {};
+	DECLARE_BITMAP(moves, FLOW_PLACES) = {};
+	struct flow_sym *syms, *cur, *back, *x;
+	struct bpf_loop *loop = aux[h].loop;
+	const u32 *pos = env->flow->pos;
+	struct bpf_iarray *succ;
+	int i, p, t, err = -ENOMEM;
+	bool carry = false;
+	s64 c;
+
+	nr = min(loop->nr_insns, total - first);
+
+	/*
+	 * The places to look at: what a jump compares with a constant, and what that is copied
+	 * from and to. One of them has to change in the loop.
+	 */
+	for (p = 0; p < FLOW_PLACES; p++)
+		set[p] = p;
+	for (k = 0; k < nr; k++) {
+		i = at[first + k];
+		if (flow_copies_place(&insns[i], &to, &from))
+			set[flow_place_set(set, to)] = flow_place_set(set, from);
+		else if (flow_jump_cmp(env, i, &reg, &op, &c))
+			__set_bit(reg, tested);
+	}
+	for_each_set_bit(p, tested, FLOW_PLACES)
+		__set_bit(flow_place_set(set, p), moves);
+	bitmap_copy(tested, moves, FLOW_PLACES);
+	bitmap_zero(moves, FLOW_PLACES);
+	for_each_set_bit(p, loop->writes, FLOW_PLACES)
+		__set_bit(flow_place_set(set, p), moves);
+	for (p = 0; p < FLOW_PLACES; p++) {
+		i = flow_place_set(set, p);
+		idx[p] = test_bit(i, tested) && test_bit(i, moves) ? n++ : FLOW_NO_PLACE;
+	}
+	if (!n || (u64)n * nr > BPF_FLOW_MAX_SYMS)
+		return 0;
+
+	/* for each insn, for the current insn, for each jump back */
+	syms = kvmalloc_array((size_t)nr + 1 + loop->backedges_cnt, n * sizeof(*syms),
+			      GFP_KERNEL_ACCOUNT);
+	/* for each root, and the result for each place */
+	roots = kvcalloc(3 * n, sizeof(*roots), GFP_KERNEL_ACCOUNT);
+	if (!syms || !roots)
+		goto out;
+	memset(syms, 0xff, (size_t)nr * n * sizeof(*syms));
+	cur = syms + (size_t)nr * n;
+	back = cur + n;
+	rungs = roots + 2 * n;
+	for (j = 0; j < n; j++)
+		syms[j] = (struct flow_sym){ .root = j };
+
+	for (k = 0; k < nr; k++) {
+		i = at[first + k];
+		insn = &insns[i];
+		if (!carry)
+			memcpy(cur, syms + (size_t)k * n, n * sizeof(*cur));
+		/* nothing but a jump back leads here */
+		for (j = 0; !carry && j < n; j++)
+			if (cur[j].root == FLOW_NO_PLACE)
+				cur[j] = (struct flow_sym){ .root = n + j };
+		/* a loop in the loop: nobody knows how many times it does what it does */
+		if (k && aux[i].loop)
+			for_each_set_bit(p, aux[i].loop->writes, FLOW_PLACES)
+				if (idx[p] != FLOW_NO_PLACE)
+					cur[idx[p]] = (struct flow_sym){ .root = n + idx[p] };
+
+		if (flow_jump_cmp(env, i, &reg, &op, &c) && idx[reg] != FLOW_NO_PLACE) {
+			x = &cur[idx[reg]];
+			if (pos[i + insn->off + 1] - first < nr)
+				flow_note_cmp(&roots[x->root], x, op, c);
+			if (pos[i + 1] - first < nr)
+				flow_note_cmp(&roots[x->root], x, bpf_rev_opcode(op), c);
+		}
+		flow_sym_step(insn, cur, idx, n);
+
+		succ = bpf_insn_successors(env, i);
+		carry = succ->cnt == 1 && !test_bit(succ->items[0], env->flow->point) &&
+			pos[succ->items[0]] == first + k + 1;
+		if (carry)
+			continue;
+		iarray_for_each(t, succ) {
+			if (t == h && back_cnt < loop->backedges_cnt)
+				memcpy(back + (size_t)back_cnt++ * n, cur, n * sizeof(*cur));
+			else if (pos[t] > pos[i] && pos[t] - first < nr)
+				flow_sym_meet(syms + (size_t)(pos[t] - first) * n, cur, n);
+			if (t == h || pos[t] > pos[i] || pos[t] - first >= nr)
+				continue;
+			/* jump back of an inner loop: what it adds moves the bounds of the root */
+			for (j = 0; j < n; j++) {
+				r = &roots[cur[j].root];
+				if (r->has_max && cur[j].hi > 0)
+					flow_add_max(r, r->max[1] + cur[j].hi);
+				if (r->has_min && cur[j].lo < 0)
+					flow_add_min(r, r->min[1] + cur[j].lo);
+			}
+		}
+	}
+
+	for (k = 0; k < back_cnt; k++) {
+		for (j = 0; j < n; j++) {
+			x = &back[(size_t)k * n + j];
+			r = &roots[x->root];
+			/* it is what it was */
+			if (x->root == j && !x->lo && !x->hi)
+				continue;
+			if (r->has_max) {
+				flow_add_max(&rungs[j], r->max[0] + x->hi);
+				flow_add_max(&rungs[j], r->max[1] + x->hi);
+			}
+			if (r->has_min) {
+				flow_add_min(&rungs[j], r->min[0] + x->lo);
+				flow_add_min(&rungs[j], r->min[1] + x->lo);
+			}
+		}
+	}
+	for (j = 0; j < n; j++)
+		cnt += rungs[j].has_max || rungs[j].has_min;
+	err = 0;
+	if (!cnt)
+		goto out;
+	loop->rungs = kvcalloc(cnt, sizeof(*loop->rungs), GFP_KERNEL_ACCOUNT);
+	if (!loop->rungs) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (p = 0; p < FLOW_PLACES; p++) {
+		if (idx[p] == FLOW_NO_PLACE || (!rungs[idx[p]].has_max && !rungs[idx[p]].has_min))
+			continue;
+		loop->rungs[loop->rungs_cnt] = rungs[idx[p]];
+		loop->rungs[loop->rungs_cnt++].place = p;
+	}
+out:
+	kvfree(syms);
+	kvfree(roots);
+	return err;
+}
+
+static int flow_compute_rungs(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	u32 total = env->cfg.cur_postorder, depth, k, *at;
+	DECLARE_BITMAP(written, FLOW_PLACES);
+	struct bpf_loop *loop;
+	int h, i, err = 0;
+
+	for (h = 0; h < env->prog->len && !aux[h].loop; h++)
+		;
+	if (h == env->prog->len)
+		return 0;
+	at = kvmalloc_array(total, sizeof(*at), GFP_KERNEL_ACCOUNT);
+	if (!at)
+		return -ENOMEM;
+	for (k = 0; k < total; k++) {
+		i = env->cfg.insn_postorder[k];
+		at[env->flow->pos[i]] = i;
+		flow_written_places(&env->prog->insnsi[i], written);
+		for (h = bpf_loop_at_index(env, i); h >= 0; h = aux[h].loop_header)
+			bitmap_or(aux[h].loop->writes, aux[h].loop->writes, written, FLOW_PLACES);
+	}
+	for (h = 0; h < env->prog->len && !err; h++) {
+		loop = aux[h].loop;
+		/* the order says little about a loop with more than one entry */
+		if (!loop || loop->irreducible || loop->backedges_overflow)
+			continue;
+		/* the main pass doesn't get that deep */
+		depth = 0;
+		for (i = aux[h].loop_header; i >= 0; i = aux[i].loop_header)
+			depth++;
+		if (depth >= LOOP_STACK_SIZE)
+			continue;
+		if (signal_pending(current)) {
+			err = -EAGAIN;
+			break;
+		}
+		cond_resched();
+		err = flow_compute_loop_rungs(env, at, total, h);
+	}
+	kvfree(at);
+	return err;
+}
+
 int bpf_flow_prepare(struct bpf_verifier_env *env)
 {
 	struct bpf_insn_aux_data *aux = env->insn_aux_data;
@@ -1279,7 +1712,10 @@ int bpf_flow_prepare(struct bpf_verifier_env *env)
 		return err;
 	for (i = 0; !err && i < kinds; i++)
 		err = flow_compute_need_const(env, &flow->need[i], i);
-	return err;
+	err = err ?: flow_compute_rungs(env);
+	if (err)
+		return err;
+	return 0;
 }
 
 static void flow_free_state(struct bpf_verifier_env *env, struct flow_state *s)
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index 88fe3d32a6de..cc4494aef187 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -39,7 +39,7 @@ struct bpf_join_state {
 	bool widen;
 	/* the states came around this loop in the frame that is joined, NULL if not */
 	const struct bpf_loop *loop;
-	/* the register or stack slot that is joined, as scalar evolution numbers it, or -1 */
+	/* the register or stack slot that is joined, as in struct bpf_loop_rungs, or -1 */
 	int place;
 };
 
@@ -142,13 +142,26 @@ static bool range_equal(const struct bpf_reg_state *ra, const struct bpf_reg_sta
 	       ra->base == rb->base && ra->step == rb->step;
 }
 
+static const struct bpf_loop_rungs *join_rungs(const struct bpf_join_state *js)
+{
+	u32 i;
+
+	for (i = 0; js->loop && i < js->loop->rungs_cnt; i++)
+		if (js->loop->rungs[i].place == js->place)
+			return &js->loop->rungs[i];
+	return NULL;
+}
+
 /* Does every iteration add a constant to what is in the place? In 64 or in 32 bits, 0 if not */
 static int join_place_is_linear(const struct bpf_join_state *js)
 {
-	if (!js->loop || js->place < 0 || js->place >= BPF_SCEV_REGS_NUM)
+	/* the place as scalar evolution numbers it */
+	u32 r = js->place < BPF_LOOP_SLOT ? js->place : js->place - BPF_LOOP_SLOT + MAX_BPF_REG;
+
+	if (!js->loop || js->place < 0 || r >= BPF_SCEV_REGS_NUM)
 		return 0;
-	return test_bit(js->place, js->loop->linear_regs) ? 64 :
-	       test_bit(js->place, js->loop->linear32_regs) ? 32 : 0;
+	return test_bit(r, js->loop->linear_regs) ? 64 :
+	       test_bit(r, js->loop->linear32_regs) ? 32 : 0;
 }
 
 /*
@@ -172,7 +185,7 @@ static bool join_range(struct bpf_join_state *js, struct bpf_reg_state *dst,
 	 */
 	if (dst->type != SCALAR_VALUE)
 		return false;
-	bpf_reg_widen_range(dst, ra, join_place_is_linear(js));
+	bpf_reg_widen_range(dst, ra, join_rungs(js), join_place_is_linear(js));
 	return true;
 }
 
@@ -468,7 +481,7 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
 	for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
 		const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
 
-		js->place = MAX_BPF_REG + i;
+		js->place = BPF_LOOP_SLOT + i;
 		if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
 			return false;
 	}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1e97444f0279..fa7aa8218218 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -18425,23 +18425,27 @@ void bpf_reg_forget_range(struct bpf_reg_state *reg)
  * @reg is what @old became after a trip around a cycle, @old included. The end
  * of the range that moved is going to move again. Put it where it may come
  * to rest. There are few such places, so this is done a few times at most.
- * It's put at the nearest of these places: every value between the range and the place is
+ * @rungs: where the jumps of the loop stop it, if they do.
+ * It's put at the nearest of all these places: every value between the range and the place is
  * made up, and the program is checked with it. Wherever it's put, the range only gets larger.
  * @linear: every iteration adds a constant to it, in that many bits. Then nothing but a jump
- * stops it. It goes on until it wraps, and is any number of that many bits.
+ * stops it. If there is none ahead it goes on until it wraps, and is any number of that many bits.
  */
-void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old, int linear)
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old,
+			 const struct bpf_loop_rungs *rungs, int linear)
 {
 	bool moved = reg->r64.base != old->r64.base || reg->r64.size != old->r64.size ||
 		     reg->r32.base != old->r32.base || reg->r32.size != old->r32.size;
-	s64 min = reg_smin(reg), max = reg_smax(reg);
+	s64 min = reg_smin(reg), max = reg_smax(reg), stop;
 
 	/* Only the step got weaker. It comes to rest without help, see below. */
 	if (!moved && reg->var_off.value == old->var_off.value &&
 	    reg->var_off.mask == old->var_off.mask)
 		return;
 
-	if (linear && (max > reg_smax(old) || min < reg_smin(old))) {
+	if (linear &&
+	    ((max > reg_smax(old) && !(rungs && rungs->has_max && max <= rungs->max[1])) ||
+	     (min < reg_smin(old) && !(rungs && rungs->has_min && min >= rungs->min[1])))) {
 		if (linear == 64 || min < 0 || max > U32_MAX) {
 			bpf_reg_forget_range(reg);
 			return;
@@ -18449,13 +18453,21 @@ void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *
 		min = 0;
 		max = U32_MAX;
 	}
-	if (max > reg_smax(old))
-		max = max <= 0 ? 0 :
-		      max <= S32_MAX ? S32_MAX :
-		      max <= U32_MAX ? U32_MAX : S64_MAX;
-	if (min < reg_smin(old))
-		min = min >= 0 ? 0 :
-		      min >= S32_MIN ? S32_MIN : S64_MIN;
+	if (max > reg_smax(old)) {
+		stop = max <= 0 ? 0 :
+		       max <= S32_MAX ? S32_MAX :
+		       max <= U32_MAX ? U32_MAX : S64_MAX;
+		if (rungs && rungs->has_max && max <= rungs->max[1])
+			stop = min(stop, max <= rungs->max[0] ? rungs->max[0] : rungs->max[1]);
+		max = stop;
+	}
+	if (min < reg_smin(old)) {
+		stop = min >= 0 ? 0 :
+		       min >= S32_MIN ? S32_MIN : S64_MIN;
+		if (rungs && rungs->has_min && min >= rungs->min[1])
+			stop = max(stop, min >= rungs->min[0] ? rungs->min[0] : rungs->min[1]);
+		min = stop;
+	}
 	if (min == S64_MIN && max == S64_MAX) {
 		bpf_reg_forget_range(reg);
 		return;
-- 
2.55.0

[PATCH bpf-next v4 32/46] bpf: A loop is one iteration shorter behind the jump that counts it

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

	#pragma clang loop unroll_count(3)
	for (i = 0; i < 250; i++) {
		if (!bpf_get_prandom_u32())
			break;
		out[i] = bpf_get_prandom_u32();
	}

is rejected with
'invalid access to map value, value_size=1000 off=1000 size=4'.
250 is 83 * 3 + 1. The compiler makes a loop of three copies of the body
and leaves it behind the first copy in iteration 84:

	l: ...; *(u32 *)(r6 - 4) = w0;
	   if r7 == 251 goto out;
	   ...; *(u32 *)(r6 + 0) = w0;
	   ...; *(u32 *)(r6 + 4) = w0;
	   r6 += 12; r7 += 3; goto l;

The loop is counted. At its head r6 gets the offsets of all 84 iterations:
4 to 1000. The jump narrows r7, which it tests. It says nothing about r6:
the two stores behind it are checked with an offset that they never see.

A state that goes on in the loop behind the jump that counts is not in the
last iteration. Give the range of one iteration less to what is linear at
the jump: r6 is 4 to 988. Only for a jump that leaves the loop itself. An
edge that leaves by way of a flag stays in the loop for a while, with the
last iteration: no shorter range behind a jump that has such an edge.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  5 ++-
 kernel/bpf/scev.c            | 59 ++++++++++++++++++++++++++++++++----
 kernel/bpf/verifier.c        |  9 ++++--
 3 files changed, 64 insertions(+), 9 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 85404ef790ef..61f133e4d72f 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -365,6 +365,8 @@ struct bpf_loop_iters {
 	 */
 	int iv_reg;
 	struct cnum64 iv_range;
+	/* a jump that counts: it leaves the loop in the last iteration, if not before. Or -1 */
+	int count_insn;
 	/* a pointer to the stack that stores go through moves in the loop */
 	bool spill_base;
 };
@@ -426,6 +428,7 @@ struct loop_stack_entry {
 	u32 walk_patience;
 	int iv_reg;
 	struct cnum64 iv_range;
+	int count_insn;
 };
 
 /* The state is in a walk on patience, which may not get to the end of the loop */
@@ -2216,6 +2219,6 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters,
 			bool back);
 int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 insn_idx,
-			const struct loop_stack_entry *entry);
+			const struct loop_stack_entry *entry, bool behind);
 
 #endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 06f485853df1..4ce8ad5d5a27 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -3470,6 +3470,13 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 		return 0;
 	}
 
+	/*
+	 * It leaves the loop itself: a flag does that later. If one edge does the one and the
+	 * other edge the other, the last iteration goes on behind the jump as well.
+	 */
+	iters->count_insn = bpf_jump_leaves_loop(env, insn_idx, first.insn_idx) &&
+			    aux[first.insn_idx].edge_leaves != 3 ? first.insn_idx : -1;
+
 	if (iters->max_header_count == 0) {
 		if (log->level & BPF_LOG_LEVEL2)
 			bpf_log(log, "loop header at %d, 0 iterations count\n", insn_idx);
@@ -3822,12 +3829,31 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 	return 1;
 }
 
+/* (linear (+ rX imm) slope): the iteration has added a constant to rX so far */
+static bool is_moved_linear(struct scev *scev, u32 id, struct linear_coefs *out, s64 *moved)
+{
+	u32 base, slope, l, r;
+
+	out->spill32 = false;
+	return is_linear_or_linear32(scev, id, &base, &slope, &out->zext32) && !out->zext32 &&
+	       is_add(scev, base, &l, &r) && is_reg(scev, l, &out->base_reg) &&
+	       is_imm(scev, r, moved) && is_simple_slope(scev, slope, &out->slope);
+}
+
+/*
+ * The state came back to the head of a counted loop. Or, with @behind, it goes on in the loop
+ * behind the jump that counts. That jump leaves the loop in the last iteration, so this is
+ * not the last one:
+ *	for (i = 0; ; i++, p += 8) { if (i == 4) break; *p = 0; }
+ * At the head p is up to 32 bytes from where it was, at the store up to 24. The jump tells that
+ * about i only. It holds for all that is linear at the jump.
+ */
 int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur_func_state, u32 insn_idx,
-			const struct loop_stack_entry *entry)
+			const struct loop_stack_entry *entry, bool behind)
 {
 	struct bpf_verifier_state *entry_state = entry->entry_state;
 	struct bpf_func_state *entry_st = entry_state->frame[entry_state->curframe];
-	u32 max_header_count = entry->max_header_count;
+	u32 max_header_count = entry->max_header_count - behind;
 	struct bpf_verifier_log *log = &env->log;
 	struct bpf_reg_state *reg, *entry_reg;
 	struct scev *scev = env->scev;
@@ -3835,6 +3861,7 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 	struct env *header_env;
 	struct bounds bounds;
 	u32 r, r_scev;
+	s64 moved;
 	int err;
 
 	if (entry_state->curframe != cur_func_state->frameno) {
@@ -3843,16 +3870,21 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 		return -EFAULT;
 	}
 
-	header_env = find_header_env(scev, insn_idx);
+	header_env = find_loop_env(scev, insn_idx, behind ? entry->count_insn : insn_idx);
+	if (!max_header_count)
+		return 0;
 	for (r = 0; r < REGS_NUM; r++) {
 		r_scev = header_env->reg2scev[r];
+		moved = 0;
 		entry_reg = spill32_counter(scev, entry_st, r_scev, r, &coefs);
 		reg = spill32_counter(scev, cur_func_state, r_scev, r, &coefs);
 		if (entry_reg && reg)
 			goto clamp;
 
 		/* If SCEV for r is (linear <reg> <slope>)*/
-		if (!is_simple_linear(scev, r_scev, &coefs) || coefs.base_reg != r)
+		if ((!is_simple_linear(scev, r_scev, &coefs) &&
+		     !(behind && is_moved_linear(scev, r_scev, &coefs, &moved))) ||
+		    coefs.base_reg != r)
 			continue;
 
 		entry_reg = scev_reg_for_linear(entry_st, r);
@@ -3878,8 +3910,22 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 			continue;
 clamp:
 		linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
-		if (r == entry->iv_reg)
+		if (r == entry->iv_reg && !moved)
 			iv_bounds(&bounds, entry->iv_range);
+		if (moved) {
+			bounds.r64 = cnum64_add(bounds.r64, cnum64_from_srange(moved, moved));
+			bounds.base = imod(bounds.base + moved, bounds.step);
+		}
+		if (behind) {
+			/* jumps on the way may have told more */
+			bounds.r64 = cnum64_intersect(bounds.r64, reg->r64);
+			bounds.r32 = cnum32_intersect(bounds.r32, reg->r32);
+			if (type_is_pkt_pointer(reg->type) || cnum64_is_empty(bounds.r64) ||
+			    cnum32_is_empty(bounds.r32) ||
+			    (cnum64_is_subset(bounds.r64, reg->r64) &&
+			     cnum32_is_subset(bounds.r32, reg->r32)))
+				continue;
+		}
 		if (log->level & BPF_LOG_LEVEL2) {
 			bpf_log(log, "loop header at %d, clamping ", insn_idx);
 			log_reg(env, r);
@@ -3887,7 +3933,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 			log_bounds(env, &bounds);
 			bpf_log(log, "\n");
 		}
-		scratch_widened_reg_id(env, reg);
+		if (!behind)
+			scratch_widened_reg_id(env, reg);
 		/*
 		 * The range of a packet pointer goes together with its id and with the offsets
 		 * that it had in the compare with pkt_end. Both are replaced here. What holds for
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fa7aa8218218..afb5b9dd18b4 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21329,7 +21329,7 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
 	if (!entry->terminates)
 		return 0;
 
-	return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry);
+	return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry, false);
 }
 
 /*
@@ -21501,8 +21501,12 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 		return 0;
 
 	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
-	if (env->insn_idx != entry->loop_id)
+	if (env->insn_idx != entry->loop_id) {
+		/* the jump that counts didn't leave the loop */
+		if (entry->entry_state && !entry->apart && cur->last_insn_idx == entry->count_insn)
+			return bpf_clamp_scev_regs(env, frame, entry->loop_id, entry, true);
 		return 0;
+	}
 
 	/*
 	 * No edge goes from a call to itself: the call goes on with one function of a set, see
@@ -21623,6 +21627,7 @@ int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop
 	entry->max_header_count = iters.max_header_count;
 	entry->iv_reg = iters.iv_reg;
 	entry->iv_range = iters.iv_range;
+	entry->count_insn = iters.count_insn;
 	entry->entry_state = entry_state;
 	entry->terminates = true;
 	/*
-- 
2.55.0

Re: [PATCH bpf-next v4 32/46] bpf: A loop is one iteration shorter behind the jump that counts it

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2319 bytes --]

> bpf: A loop is one iteration shorter behind the jump that counts it

TL;DR: Possible s64 wrap when adding "moved" in bpf_clamp_scev_regs() may
leave a wrong base/step, so the verifier could drop values a register can
really hold; scalar_step_add() resets the step in this case.

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c

[ ... ]

> @@ -3887,7 +3933,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
>  clamp:
>  		linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
> -		if (r == entry->iv_reg)
> +		if (r == entry->iv_reg && !moved)
>  			iv_bounds(&bounds, entry->iv_range);
> +		if (moved) {
> +			bounds.r64 = cnum64_add(bounds.r64, cnum64_from_srange(moved, moved));
> +			bounds.base = imod(bounds.base + moved, bounds.step);
> +		}

Can this wrap in a way that leaves bounds.base wrong?  linear_bounds() only
rules out signed overflow of entry + slope * (n - 1), not of the extra
"+ moved".  cnum64_add() wraps modulo 2**64, so if entry + moved + slope * k
crosses S64_MAX or S64_MIN, the signed value is off by 2**64.

When the entry is a constant, step is |slope|, which need not be a power of
two.  Then 2**64 mod step is not 0, and the residue computed by

    bounds.base = imod(bounds.base + moved, bounds.step);

no longer matches the values in bounds.r64.

scalar_step_add() in kernel/bpf/verifier.c handles this case: it calls
check_add_overflow() on reg_smin()/reg_smax() and resets the step if the add
can wrap, noting that keeping the step "requires 2**64 mod s = 0".

bpf_set_reg_range() installs the new base/step as is, replacing what the
register had, and reg_bounds_sync() then uses base/step to round smin/smax to
values congruent to base.  The intersection with reg->r64 does not help,
because base/step are replaced rather than intersected.  Could this drop
values the register can really hold?

Should this fall back to step = 1, base = 0 (or skip the register) when adding
moved to the bounds can overflow s64, as scalar_step_add() does?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

Re: [PATCH bpf-next v4 32/46] bpf: A loop is one iteration shorter behind the jump that counts it

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] bpf: signed overflow and stride corruption on moved in
  bpf_clamp_scev_regs()
--

commit bb719964b0079686d30154660558b2b4d7792c19
Author: Alexei Starovoitov <ast@kernel.org>

bpf: A loop is one iteration shorter behind the jump that counts it

This commit narrows the SCEV-derived register ranges by one iteration on the
fall-through path behind a loop-counting exit jump, including registers to
which the current iteration has already added a constant offset.

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 06f485853df13..4ce8ad5d5a279 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c

[ ... ]

> @@ -3878,8 +3910,22 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
>  			continue;
>  clamp:
>  		linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
> -		if (r == entry->iv_reg)
> +		if (r == entry->iv_reg && !moved)
>  			iv_bounds(&bounds, entry->iv_range);
> +		if (moved) {
> +			bounds.r64 = cnum64_add(bounds.r64, cnum64_from_srange(moved, moved));
> +			bounds.base = imod(bounds.base + moved, bounds.step);
> +		}

[Severity: High]
Can shifting bounds by moved here corrupt bounds.base and bounds.step when
bounds.step is not a power of two?

When entry_reg is a constant scalar, linear_bounds() sets bounds.step to
|slope| (up to S16_MAX) and tracks the signed congruence x mod step == base.
If adding moved to cnum64_smin(bounds.r64) or cnum64_smax(bounds.r64) wraps
s64, 64-bit addition shifts the signed values by +/- 2^64, which is not a
multiple of a non-power-of-two step, so the wrapped range in bounds.r64 no
longer has remainder imod(bounds.base + moved, bounds.step).  Passing the
mismatched bounds.base and bounds.step to bpf_set_reg_range() ->
cnum64_intersect_linear() can then narrow reg to a bogus constant that the
register never holds at runtime, or collapse it to CNUM64_EMPTY and hit
verifier_bug().

In addition, bounds.base + moved is evaluated in signed 64-bit arithmetic and
can overflow s64 when moved is near S64_MAX (with bounds.base > 0), unlike
scalar_step_add():

kernel/bpf/verifier.c:scalar_step_add() {
	...
	if (check_add_overflow(reg_smin(reg), amount, &tmp) ||
	    check_add_overflow(reg_smax(reg), amount, &tmp)) {
		reg_step_reset(dst_reg);
		return;
	}

	dst_reg->base = (reg->base + imod(amount, reg->step)) % reg->step;
	dst_reg->step = reg->step;
}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=32

[PATCH bpf-next v4 33/46] bpf: Give the places of a back edge to the jumps that count

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

A stack walker reads a few words per frame, leaves the loop when a read
fails, and appends up to 32 words per frame:

	bpf_for(i, 0, 100) {
		if (frames == 8)
			break;
		if (READ(a) || READ(b) || READ(c) || READ(d) || READ(e))
			break;
		n = 32 - (a & 1);
		for (j = 0; j < n; j++)
			out[idx + j] = b + c + d + e;
		idx += n;
		frames++;
	}

'R1 unbounded memory access': only 'frames == 8' limits idx, and nothing
looks at that jump. A back edge has four places for the jumps that leave
the loop and dominate it. The nearest ones take them: the NULL test of
bpf_iter_num_next() and three statuses. None of them counts.

Scalar evolution knows which jumps test a counter. Collect the jumps again
when it is done and pass over the others. The latch keeps its place. The
search ends at the head of the loop, as for a loop with less than four such
jumps. Every jump that is kept tells iterations in which the back edge is
not taken, so any choice is sound.

The loop has 9 iterations at most. idx has no closed form, so they are
looked at one by one: idx is 0, [31, 32], [62, 64] .. [217, 224].
pyperf.h with STACK_MAX_LEN 250 and UNROLL_COUNT 8 leaves its loops from
the second of 8 copies of the body: 59713 -> 2877 insns.

A jump can leave the loop by way of a flag. These jumps are known before
scalar evolution starts, so mark_latches() asks for the SCEV of each.
  do { ..; more = ++i < 256 && v->in[i & 63];
       if (READ(a) || READ(b) || READ(c)) break; } while (more);
The four places went to jumps that are nearer to the back edge:
"R1 unbounded memory access", 12562 insns on bpf-next. Now 63.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 include/linux/bpf_verifier.h |  2 ++
 kernel/bpf/loops.c           | 27 ++++++++++++++++++++-------
 kernel/bpf/scev.c            | 24 ++++++++++++++++++++++++
 3 files changed, 46 insertions(+), 7 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 61f133e4d72f..656ed55b96ec 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -716,6 +716,7 @@ struct bpf_backedge {
 	 * Conditional jumps within the loop that dominate 'from' and have
 	 * a successor outside the loop, nearest first. latches[0] is the
 	 * latch; the remaining entries are additional exit conditions.
+	 * After scalar evolution: those that test a counter, see bpf_scev_jump_counts().
 	 * One of them may leave the loop by way of a flag, see bpf_edge_leaves_loop().
 	 * Each usable condition tells iterations in which the back edge is
 	 * not taken, so keeping only a subset is sound.
@@ -2212,6 +2213,7 @@ bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int oute
 int bpf_init_scev(struct bpf_verifier_env *env);
 void bpf_free_scev(struct bpf_verifier_env *env);
 int bpf_compute_scev(struct bpf_verifier_env *env);
+int bpf_scev_jump_counts(struct bpf_verifier_env *env, int header, int n, bool taken_leaves);
 
 int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
 			   struct bpf_loop_iters *iters);
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 0f3b0411003b..ce6cef5e1c15 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -307,18 +307,19 @@ int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx)
 /*
  * Collect conditional jumps within n's loop that dominate 'n' and leave
  * the loop on one edge, see bpf_edge_leaves_loop(), nearest first.
+ * A jump that cannot count the iterations takes no place, see bpf_scev_jump_counts().
  */
-static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
-				       struct bpf_backedge *backedge)
+static int find_dominating_conditions(struct bpf_verifier_env *env, int n,
+				      struct bpf_backedge *backedge)
 {
 	struct bpf_insn *insns = env->prog->insnsi;
 	bool t_exits_loop, f_exits_loop;
 	int *idoms = env->idoms;
-	int n_loop;
+	int n_loop, counts;
 
 	n_loop = bpf_loop_at_index(env, n);
 	if (idoms_intersect(env, n, n_loop) != n_loop)
-		return;
+		return 0;
 	while (n >= 0) {
 		if (is_cond_jmp_insn(&insns[n]) && bpf_loop_at_index(env, n) == n_loop) {
 			t_exits_loop = bpf_edge_leaves_loop(env, n_loop, n, n + insns[n].off + 1);
@@ -326,22 +327,32 @@ static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
 			/* latches[0] is the latch: it has to leave the loop itself */
 			if ((t_exits_loop || f_exits_loop) &&
 			    (backedge->latches_cnt || bpf_jump_leaves_loop(env, n_loop, n))) {
+				counts = !backedge->latches_cnt ? 1 :
+					 bpf_scev_jump_counts(env, n_loop, n, t_exits_loop);
+				if (counts < 0)
+					return counts;
+				if (!counts)
+					goto next;
 				if (backedge->latches_cnt == MAX_LATCHES)
-					return;
+					return 0;
 				if (t_exits_loop)
 					backedge->taken_leaves |= BIT(backedge->latches_cnt);
 				backedge->latches[backedge->latches_cnt++] = n;
 			}
 		}
+next:
 		if (n == n_loop)
 			break;
 		n = idoms[n];
 	}
+	return 0;
 }
 
 /*
  * Without @flags: the jumps that leave a loop themselves. That tells where the verifier may add
  * may_goto. It takes the constants that hold with these edges to follow a flag.
+ * Once more after scalar evolution, which knows the jumps that count. The flags that were found
+ * are remembered.
  */
 int bpf_find_latches(struct bpf_verifier_env *env, bool flags)
 {
@@ -377,8 +388,10 @@ int bpf_find_latches(struct bpf_verifier_env *env, bool flags)
 			 */
 			if (bpf_loop_at_index(env, backedge->from) != i)
 				continue;
-			find_dominating_conditions(env, backedge->from, backedge);
-			for (k = 1; k < backedge->latches_cnt; k++)
+			err = find_dominating_conditions(env, backedge->from, backedge);
+			if (err)
+				return err;
+			for (k = 1; flags && k < backedge->latches_cnt; k++)
 				if ((env->log.level & BPF_LOG_LEVEL2) &&
 				    !bpf_jump_leaves_loop(env, i, backedge->latches[k]))
 					bpf_log(&env->log, "loop at %d, exit by way of a flag at %d\n",
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 4ce8ad5d5a27..64903ab738a0 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1452,6 +1452,10 @@ static void mark_latches(struct bpf_verifier_env *env)
 	int i, j, k, from;
 
 	for (i = 0; i < len; i++) {
+		/* An exit by way of a flag. It may have no place yet, see bpf_scev_jump_counts() */
+		if (aux[i].edge_leaves &&
+		    can_compute_loop_scev(aux[bpf_loop_at_index(env, i)].loop))
+			aux[i].need_scev = true;
 		loop = aux[i].loop;
 		if (!loop)
 			continue;
@@ -1864,6 +1868,11 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
 						 find_header_env(scev, header));
 	}
 
+	/* Now it's known which jumps test a counter: the places of a back edge are for them */
+	err = bpf_find_latches(env, false);
+	if (err)
+		return err;
+
 	if (env->log.level & BPF_LOG_LEVEL2) {
 		err = log_scevs(env);
 		if (err)
@@ -2190,6 +2199,21 @@ static int match_linear_latch(struct bpf_verifier_env *env,
 	return false;
 }
 
+/*
+ * Can the jump at @n tell the number of iterations of the loop at @header? The status of a call
+ * and a NULL test cannot. 1 as long as scalar evolution has not looked at the program.
+ */
+int bpf_scev_jump_counts(struct bpf_verifier_env *env, int header, int n, bool taken_leaves)
+{
+	struct linear_latch latch;
+
+	if (!env->scev)
+		return 1;
+	if (!find_loop_env(env->scev, header, n))
+		return 0;
+	return match_linear_latch(env, header, n, taken_leaves, &latch);
+}
+
 /* What is known about a scalar or about the offset of a pointer. */
 struct scev_value {
 	u64 umin, umax;
-- 
2.55.0

[PATCH bpf-next v4 34/46] bpf: Count a loop whose exit tests a copy of the counter

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

	u32 i = 0;
	while (buf[i++] && i < 1023)
		;
	return buf[i];

clang keeps what i was before the increment in a second register. The
jump and the load use that one:

	l:   r1 += 1
	     if r2 > 1021 goto out
	     r2 += r0
	     w3 = *(u8 *)(r2 + 0)
	     r2 = r1
	     if w3 != 0 goto l
	out: r0 += r1

Scalar evolution has (linear r1 1) for r1 and nothing for r2, which is
made from another register: the loop has no count. No jump tests r1, so
widening has no stop for it:
  math between map_value pointer and register with unbounded min value

An iteration leaves the same number in r1 and r2. If they are equal when
the loop is entered as well, one constant or one id, r2 is a copy of r1
at the header in every iteration, and what is known about r1 is known
about r2: (copy r2 (linear r1 1)). The jump counts the loop, 1023
iterations, and both registers get their range from the count. They also
get one id, like after 'r2 = r1': what the jump says about r2 bounds r1.

That is true at the header. Behind the jump that counts, where ranges are
one iteration shorter, r1 has moved on: r2 gets its range, and no id. A
register that took the value of r2 on the way, 'r4 = r2', gets nothing:
what r4 was when the loop was entered has nothing to do with it.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/scev.c | 128 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 119 insertions(+), 9 deletions(-)

diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 64903ab738a0..5415fcf9df2f 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -43,6 +43,11 @@ enum expr_op {
 	 * bpf_loop_at_index().
 	 */
 	LINEAR_SCEV,
+	/*
+	 * param[1], given that register param[0] and the register that param[1] starts from are
+	 * equal when the loop is entered. Not known otherwise.
+	 */
+	COPY_SCEV,
 };
 
 struct expr {
@@ -259,6 +264,13 @@ static bool is_linear(struct scev *scev, int id, u32 *base, u32 *slope)
 	return is_binop(scev, LINEAR_SCEV, id, base, slope);
 }
 
+static bool is_copy(struct scev *scev, int id, u32 *reg, u32 *orig)
+{
+	u32 t;
+
+	return is_binop(scev, COPY_SCEV, id, &t, orig) && is_reg(scev, t, reg);
+}
+
 static bool is_linear_or_linear32(struct scev *scev, int id, u32 *base, u32 *slope, bool *zext32)
 {
 	u32 t;
@@ -319,6 +331,7 @@ static const char *op_str(u32 op)
 	case SPILL32:  return "spill32";
 	case ANY:      return "any";
 	case LINEAR_SCEV:  return "linear";
+	case COPY_SCEV:    return "copy";
 	}
 	return NULL;
 }
@@ -341,6 +354,7 @@ static u32 op_params_num(u32 op)
 	case SMOD:
 	case ANY:
 	case LINEAR_SCEV:
+	case COPY_SCEV:
 		return 2;
 	case BPF_NEG:
 	case SEXT8:
@@ -1626,6 +1640,7 @@ static int compute_header_scevs(struct bpf_verifier_env *env, struct env *header
 {
 	struct scev *scev = env->scev;
 	int l, r, t, id, ra, rb, ra_expr;
+	bool zext32;
 	s64 imm;
 
 	/*
@@ -1702,6 +1717,28 @@ static int compute_header_scevs(struct bpf_verifier_env *env, struct env *header
 		}
 
 	}
+	/*
+	 *	do { i++; if (old > 1021) break; c = buf[old]; old = i; } while (c);
+	 * rA = (+ rB IMM) and rB = (+ rB IMM): an iteration leaves the same number in both. If
+	 * they are equal when the loop is entered rA is at the header what rB is.
+	 */
+	for (ra = 0; ra < REGS_NUM; ra++) {
+		if (header_env->reg2scev[ra] != UNKNOWN_EXPR_ID ||
+		    header_env->reg2expr[ra] == UNKNOWN_EXPR_ID)
+			continue;
+		for (rb = 0; rb < REGS_NUM; rb++) {
+			id = header_env->reg2scev[rb];
+			if (header_env->reg2expr[rb] != header_env->reg2expr[ra] ||
+			    !is_linear_or_linear32(scev, id, &l, &r, &zext32) ||
+			    !is_reg(scev, l, &t) || t != rb)
+				continue;
+			id = expr2(scev, COPY_SCEV, expr1(scev, REG, ra), id);
+			if (id < 0)
+				return id;
+			header_env->reg2scev[ra] = id;
+			break;
+		}
+	}
 	return 0;
 }
 
@@ -1977,6 +2014,19 @@ static struct bpf_reg_state *scev_reg_for_linear(struct bpf_func_state *st, u32
 	return &stack->spilled_ptr;
 }
 
+/* Do the registers or stack slots @a and @b hold the same number in @st? */
+static bool scev_regs_equal(struct bpf_func_state *st, u32 a, u32 b)
+{
+	const struct bpf_reg_state *ra = scev_reg_for_linear(st, a);
+	const struct bpf_reg_state *rb = scev_reg_for_linear(st, b);
+
+	if (!ra || !rb || ra->type != SCALAR_VALUE || rb->type != SCALAR_VALUE)
+		return false;
+	if (tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off))
+		return ra->var_off.value == rb->var_off.value;
+	return ra->id && ra->id == rb->id && ra->delta == rb->delta;
+}
+
 /* '*(u32 *)(r10 - 8) = rX' left a number in the low half of the slot. */
 static struct bpf_reg_state *scev_reg_for_spill32(struct bpf_func_state *st, u32 r)
 {
@@ -2090,12 +2140,16 @@ static bool scev_reg_alive(struct bpf_verifier_env *env, struct bpf_verifier_sta
  * 'i' is the loop iteration number, starting from 0.
  * 'zext32' indicates the latch is for (zext32 (linear ...)) SCEV.
  * 'jmp32' indicates that the latch is a JMP32 instruction.
+ * 'base_reg' is the register in <base>. 'copy_reg' is -1, or <reg> is made from that register,
+ * which has to be equal to 'base_reg' at the entry of the loop, see COPY_SCEV.
  */
 struct linear_latch {
 	u32 insn_idx;
 	u32 base_expr;
 	u32 step_expr;
 	u32 bound_expr;
+	u32 base_reg;
+	int copy_reg;
 	u8 op;
 	bool zext32;
 	bool jmp32;
@@ -2149,6 +2203,11 @@ static int match_linear_latch(struct bpf_verifier_env *env,
 		return -EFAULT;
 	dst_reg_scev = latch_env->reg2scev[insn->dst_reg];
 	src_reg_scev = latch_env->reg2scev[insn->src_reg];
+	latch->copy_reg = -1;
+	if (is_copy(scev, dst_reg_scev, &latch->copy_reg, &t))
+		dst_reg_scev = t;
+	else if (BPF_SRC(insn->code) == BPF_X && is_copy(scev, src_reg_scev, &latch->copy_reg, &t))
+		src_reg_scev = t;
 	/*
 	 * Make sure (linear ...) is in dst_reg_scev:
 	 *   'if <invariant> <op> <linear>' ⇔
@@ -2187,12 +2246,12 @@ static int match_linear_latch(struct bpf_verifier_env *env,
 		return false;
 
 	/* (linear register step) */
-	if (is_reg_or_low32(scev, latch->base_expr, &t))
+	if (is_reg_or_low32(scev, latch->base_expr, &latch->base_reg))
 		return true;
 
 	/* (linear (+ register imm) step) */
 	if (is_add(scev, latch->base_expr, &l, &r) &&
-	    is_reg_or_low32(scev, l, &t) &&
+	    is_reg_or_low32(scev, l, &latch->base_reg) &&
 	    loop_invariant(scev, r))
 		return true;
 
@@ -2846,6 +2905,8 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
 
 	iters->iv_reg = -1;
 
+	if (latch->copy_reg >= 0 && !scev_regs_equal(st, latch->copy_reg, latch->base_reg))
+		return false;
 	if (!eval_expr(env, scev, st, latch->base_expr, &initial_val) ||
 	    !eval_expr(env, scev, st, latch->step_expr, &step_val) ||
 	    !eval_expr(env, scev, st, latch->bound_expr, &bound_val))
@@ -3180,6 +3241,50 @@ static bool is_simple_linear(struct scev *scev, u32 id, struct linear_coefs *out
 	       is_simple_slope(scev, slope, &out->slope);
 }
 
+/*
+ * Is @r 'what it is in @st + slope * k' at the header in iteration k? @id is its SCEV there, @st is
+ * the state that entered the loop. At a jump in the loop @id can be what another register was at
+ * the header. @st says nothing about @r then.
+ */
+static bool is_linear_at_header(struct scev *scev, struct bpf_func_state *st, u32 id, u32 r,
+				struct linear_coefs *out)
+{
+	u32 copy, orig;
+
+	if (is_copy(scev, id, &copy, &orig))
+		return copy == r && is_simple_linear(scev, orig, out) &&
+		       scev_regs_equal(st, r, out->base_reg);
+	return is_simple_linear(scev, id, out) && out->base_reg == r;
+}
+
+/*
+ * @st is at the header of a loop that @entry entered. A copy of an induction variable is equal to
+ * it there in every iteration. They got their ranges one by one: give them one id, like 'r2 = r1'
+ * does, so that a test of one of them is a test of both.
+ */
+static void link_copies(struct bpf_verifier_env *env, struct env *header_env,
+			struct bpf_func_state *entry, struct bpf_func_state *st)
+{
+	struct bpf_reg_state *reg, *orig;
+	struct linear_coefs coefs;
+	u32 r;
+
+	for (r = 0; r < REGS_NUM; r++) {
+		if (!is_linear_at_header(env->scev, entry, header_env->reg2scev[r], r, &coefs) ||
+		    coefs.base_reg == r)
+			continue;
+		reg = scev_reg_for_linear(st, r);
+		orig = scev_reg_for_linear(st, coefs.base_reg);
+		if (!reg || !orig || reg->type != SCALAR_VALUE || orig->type != SCALAR_VALUE ||
+		    tnum_is_const(orig->var_off))
+			continue;
+		if (!orig->id)
+			orig->id = ++env->id_gen;
+		reg->id = orig->id;
+		reg->delta = orig->delta;
+	}
+}
+
 /*
  * A 32-bit counter that lives on the stack: the SCEV of slot @r is
  * (spill32 (linear (zext32 @r) slope)) and there is a 32-bit number in the slot in @st.
@@ -3588,7 +3693,8 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 			continue;
 		}
 		/* If SCEV for r is (linear <reg> <slope>) */
-		if (reg && is_simple_linear(scev, r_scev, &coefs) && is_widenable_reg_type(reg)) {
+		if (reg && is_linear_at_header(scev, cur_func, r_scev, r, &coefs) &&
+		    is_widenable_reg_type(reg)) {
 			/* 32-bit ALU operations on pointers convert them to scalars */
 			if (coefs.zext32 && reg->type != SCALAR_VALUE)
 				goto cant_widen;
@@ -3802,7 +3908,7 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 		}
 
 		/* If SCEV for r is (linear <reg> <slope>)*/
-		if (spill32 || (is_simple_linear(scev, r_expr, &coefs) && coefs.base_reg == r)) {
+		if (spill32 || is_linear_at_header(scev, entry_func, r_expr, r, &coefs)) {
 			/* the range is derived from @loop_entry: bpf_clamp_scev_regs() */
 			if (back)
 				continue;
@@ -3812,7 +3918,7 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 			if (err)
 				return err;
 			linear_bounds(reg, iters->max_header_count, &coefs, &bounds);
-			if (r == iters->iv_reg)
+			if (coefs.base_reg == iters->iv_reg)
 				iv_bounds(&bounds, iters->iv_range);
 			if (log->level & BPF_LOG_LEVEL2) {
 				bpf_log(log, "loop header at %d, widening ", insn_idx);
@@ -3850,6 +3956,7 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
 			mark_scev_reg_scratched(env, r);
 		}
 	}
+	link_copies(env, header_env, entry_func, cur_func);
 	return 1;
 }
 
@@ -3906,9 +4013,9 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 			goto clamp;
 
 		/* If SCEV for r is (linear <reg> <slope>)*/
-		if ((!is_simple_linear(scev, r_scev, &coefs) &&
-		     !(behind && is_moved_linear(scev, r_scev, &coefs, &moved))) ||
-		    coefs.base_reg != r)
+		if (!is_linear_at_header(scev, entry_st, r_scev, r, &coefs) &&
+		    !(behind && is_moved_linear(scev, r_scev, &coefs, &moved) &&
+		      coefs.base_reg == r))
 			continue;
 
 		entry_reg = scev_reg_for_linear(entry_st, r);
@@ -3934,7 +4041,7 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 			continue;
 clamp:
 		linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
-		if (r == entry->iv_reg && !moved)
+		if (coefs.base_reg == entry->iv_reg && !moved)
 			iv_bounds(&bounds, entry->iv_range);
 		if (moved) {
 			bounds.r64 = cnum64_add(bounds.r64, cnum64_from_srange(moved, moved));
@@ -3972,5 +4079,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
 			return err;
 		mark_scev_reg_scratched(env, r);
 	}
+	/* behind the jump the variable may have moved on: its copy is what it was at the head */
+	if (!behind)
+		link_copies(env, header_env, entry_st, cur_func_state);
 	return 0;
 }
-- 
2.55.0

[PATCH bpf-next v4 35/46] bpf: Allow tail calls and LD_ABS in subprogs that return void

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  static __noinline int subprog_tail(struct __sk_buff *skb)
  {
          bpf_tail_call_static(skb, &jmp_table, 0);
          return 0;
  }

LLVM 23 writes the signature that a function has after optimization
into BTF. The callers know the constant, nobody reads r0, the function
returns 'void' there, and the prog is rejected before the first insn is
looked at:

  tail_call is only allowed in functions that return 'int'.

Commit be1d838b88e4 ("selftests/bpf: Keep int return type for tailcall
subprogs") hides the constant from the compiler with barrier_var() and
__sink(). Without them 6 progs of tailcall_bpf2bpf2.c,
tailcall_bpf2bpf_hierarchy[123].c, tailcall_callback.c and
tracing_stack_arg_tailcall_tgt.c are rejected. Everybody else who has
a tail call in a subprog has to find the same trick.

The rule is from commit 09b28d76eac4 ("bpf: Add abnormal return
checks."). A tail call that is taken and LD_ABS that fails return to
the caller with a number in r0. The verifier did not look at that
return, so the type of the function had to promise a number.
Now it does look: check_helper_call() goes back to the caller with an
unknown r0, check_ld_abs() with 0, and r0 is not set behind the call of
a global function that returns void. A caller that reads r0 is checked
with what is there:

  call void_subprog_tail_call      ; r0 = 1 if the tail call is not taken
  if r0 == 1 goto +1
  r0 = r9                          ; R9 !read_ok

So let 'void' through. A function that returns a pointer or a struct is
refused as before.

Two tests of verifier_ld_ind pin the rule for void. They load now.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/check_btf.c                           | 10 +++++++---
 .../selftests/bpf/progs/verifier_ld_ind.c        | 16 ++++++++--------
 2 files changed, 15 insertions(+), 11 deletions(-)

diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c
index 4c1ed842f661..8a3f2421331e 100644
--- a/kernel/bpf/check_btf.c
+++ b/kernel/bpf/check_btf.c
@@ -183,15 +183,19 @@ static int check_btf_func(struct bpf_verifier_env *env,
 		/* Already checked func_proto */
 		func_proto = btf_type_by_id(btf, type->type);
 
+		/*
+		 * LD_ABS that fails and a tail call that is taken return to the caller
+		 * with a number in r0, whatever the type is. 'void' says nothing about r0.
+		 */
 		ret_type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
-		scalar_return =
+		scalar_return = btf_type_is_void(ret_type) ||
 			btf_type_is_small_int(ret_type) || btf_is_any_enum(ret_type);
 		if (i && !scalar_return && env->subprog_info[i].has_ld_abs) {
-			verbose(env, "LD_ABS is only allowed in functions that return 'int'.\n");
+			verbose(env, "LD_ABS is only allowed in functions that return 'int' or 'void'.\n");
 			goto err_free;
 		}
 		if (i && !scalar_return && env->subprog_info[i].has_tail_call) {
-			verbose(env, "tail_call is only allowed in functions that return 'int'.\n");
+			verbose(env, "tail_call is only allowed in functions that return 'int' or 'void'.\n");
 			goto err_free;
 		}
 
diff --git a/tools/testing/selftests/bpf/progs/verifier_ld_ind.c b/tools/testing/selftests/bpf/progs/verifier_ld_ind.c
index 32989f981fb6..b31a5e558409 100644
--- a/tools/testing/selftests/bpf/progs/verifier_ld_ind.c
+++ b/tools/testing/selftests/bpf/progs/verifier_ld_ind.c
@@ -291,8 +291,8 @@ int ld_ind_global_subprog_ok(struct __sk_buff *skb)
 }
 
 /*
- * ld_{abs,ind} in subprogs require scalar (int) return type in BTF.
- * A test with void return must be rejected.
+ * ld_{abs,ind} in subprogs require scalar (int) or void return type in BTF.
+ * The failure return is modeled for a void subprog too.
  */
 __naked __noinline __used
 static void ldabs_void_subprog(void)
@@ -308,9 +308,9 @@ static void ldabs_void_subprog(void)
 }
 
 SEC("socket")
-__description("ld_abs: reject void return subprog")
-__failure __msg("LD_ABS is only allowed in functions that return 'int'")
-__naked void ld_abs_void_subprog_reject(void)
+__description("ld_abs: void return subprog")
+__success
+__naked void ld_abs_void_subprog(void)
 {
 	asm volatile (
 	"call ldabs_void_subprog;"
@@ -334,9 +334,9 @@ static void ldind_void_subprog(void)
 }
 
 SEC("socket")
-__description("ld_ind: reject void return subprog")
-__failure __msg("LD_ABS is only allowed in functions that return 'int'")
-__naked void ld_ind_void_subprog_reject(void)
+__description("ld_ind: void return subprog")
+__success
+__naked void ld_ind_void_subprog(void)
 {
 	asm volatile (
 	"call ldind_void_subprog;"
-- 
2.55.0

[PATCH bpf-next v4 36/46] bpf: A part of a pointer that is spilled is a number

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

xdp->data is a u32 in C and a pointer to the packet for the verifier.
test_dynptr_copy_xdp_overlap() of dynptr_success.c makes a length of it:

  void *data_end = (void *)(long)xdp->data_end;
  void *data = (void *)(long)xdp->data;
  __u32 off = data_end - data - 1000;

  if (fill_xdp(&ptr, off, 1300)) ...
  err = bpf_dynptr_copy(&ptr, off + 100, &ptr, off, 1200);
  if (check_xdp(&ptr, off + 100, 1200, off)) ...

fill_xdp() and check_xdp() are inlined loops. With plain
'for (i = 0; i < n; i++)' in place of their bpf_for() no register is left
for off. The compiler keeps the u32 that it loaded on the stack and
subtracts once more behind the first loop:

  0: r7 = *(u32 *)(r1 +0)          ; R7=pkt(r=0)
  8: *(u32 *)(r10 -48) = r7
  invalid size of register spill

No loop is looked at. The three test_dynptr_copy_xdp_overlap*() progs are
rejected like that, with clang 18 and with today's clang.

With CAP_PERFMON the prog may see the value of a pointer. It can store
4 bytes of it into a map value, or shift it, and has a number. The stack
is the only place that refuses. Treat the store like a store of any
other data: the bytes are STACK_MISC, what was spilled in the slot is
gone. Whatever is filled from the slot is a number, also when it has the
value of the pointer:

  *(u32 *)(r10 - 8) = r1
  r1 >>= 32
  *(u32 *)(r10 - 4) = r1
  r0 = *(u64 *)(r10 - 8)
  r0 = *(u32 *)(r0 + 0)            ; R0 invalid mem access 'scalar'

Nothing changes without CAP_PERFMON.

dynptr/slice_buf_narrow_spill stores 4 bytes of a slice into the buffer
of the slice. It is rejected by the next insn, which reads through the
slice. Update the expected message.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/verifier.c                           | 9 +++++++--
 tools/testing/selftests/bpf/progs/dynptr_fail.c | 5 ++---
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index afb5b9dd18b4..8dff321ea723 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3798,8 +3798,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 		__mark_reg_known(tmp_reg, insn->imm);
 		tmp_reg->type = SCALAR_VALUE;
 		save_register_state(env, state, spi, tmp_reg, size);
-	} else if (reg && is_pointer_regtype(reg->type)) {
-		/* register containing pointer is being spilled into stack */
+	} else if (reg && is_pointer_regtype(reg->type) &&
+		   (size == BPF_REG_SIZE || !env->allow_ptr_leaks)) {
+		/*
+		 * register containing pointer is being spilled into stack.
+		 * A part of it is a number, which is fine when the program
+		 * may see values of pointers anyway.
+		 */
 		if (size != BPF_REG_SIZE) {
 			verbose_linfo(env, insn_idx, "; ");
 			verbose(env, "invalid size of register spill\n");
diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c
index 3770ffdeaaaa..7f8a38c37d9e 100644
--- a/tools/testing/selftests/bpf/progs/dynptr_fail.c
+++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c
@@ -1343,7 +1343,6 @@ __naked int name(void)					\
 
 /* Not in the line of SLICE_BUF_TEST(): the log of the verifier has a copy of that line */
 #define SLICE_BUF_INVALID __failure __msg("R7 invalid mem access 'scalar'")
-#define SLICE_BUF_POINTER __failure __msg("invalid size of register spill")
 
 SLICE_BUF_TEST(slice_buf_first_byte, SLICE_BUF_INVALID,
 	SLICE_BUF(12, 4)
@@ -1375,8 +1374,8 @@ SLICE_BUF_TEST(slice_buf_var_off, SLICE_BUF_INVALID,
 	SLICE_BUF_CHECK
 	"*(u64 *)(r10 - 8) = 0;")
 
-/* the slice is a pointer when it is stored to its buffer, and invalid after that */
-SLICE_BUF_TEST(slice_buf_narrow_spill, SLICE_BUF_POINTER,
+/* the slice is a pointer when a part of it is stored to its buffer, and invalid after that */
+SLICE_BUF_TEST(slice_buf_narrow_spill, SLICE_BUF_INVALID,
 	SLICE_BUF(8, 8)
 	"*(u32 *)(r10 - 8) = r7;")
 
-- 
2.55.0

[PATCH bpf-next v4 37/46] bpf: 32-bit add to a pointer and pointer op pointer make a number

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

xdp_synproxy_kern.c:

  /* Without volatile the verifier throws this error:
   * R9 32-bit pointer arithmetic prohibited
   */
  volatile u64 old_len = data_end - data;

  if (bpf_skb_change_tail(ctx, old_len + TCP_MAXLEN - hdr->tcp_len, 0))

test_tcp_custom_syncookie.c has the same comment. The argument is a u32,
so the compiler adds in 32 bits and in the order that it likes,
data_end - (data + tcp_len) + 60:

  151: w9 += w6                    ; R9=pkt()
  152: w8 -= w9                    ; R8=pkt_end()
  153: w8 += 60

exhandler_kern.c had 'if (!work && !func)' with two pointers, which is
'r2 |= r1': 'R2 pointer |= pointer prohibited'. Commit 44df171a10f8
("selftests/bpf: Workaround a verifier issue for test exhandler") wrote
it in another way. The C code wants a number in both cases.

Commit 72c0681c0e75 ("bpf: Allow bitwise ops, shifts and mul/div on
pointers with CAP_PERFMON") made a number of the result of every op on
a pointer and a number but += and -=. 32-bit -= and pointer -= pointer
have always made one. That leaves 32-bit += and the other ops on two
pointers. Do the same for them: the result is an unknown number. The
prog is rejected where it uses the number as a pointer:

  w0 += 0                          ; R0=map_value -> scalar
  *(u64 *)(r0 + 0) = r1            ; R0 invalid mem access 'scalar'

64-bit += and -= of a pointer and a number keep the pointer, as before.
Nothing changes without CAP_PERFMON.

Three tests go through the result of such an op: they are rejected by
the next insn. 'unpriv: add pointer to pointer' and
verifier_scev/ptr_map_value_alu32_latch do nothing with it: they load.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/verifier.c                                | 12 ++++++------
 .../selftests/bpf/progs/verifier_array_access.c      |  4 ++--
 tools/testing/selftests/bpf/progs/verifier_scev.c    |  3 +--
 tools/testing/selftests/bpf/progs/verifier_unpriv.c  |  4 ++--
 .../selftests/bpf/progs/verifier_value_illegal_alu.c |  4 ++--
 .../selftests/bpf/progs/verifier_value_ptr_arith.c   |  4 ++--
 6 files changed, 15 insertions(+), 16 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8dff321ea723..09a44723bb2e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16911,7 +16911,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 
 	dst_reg = &regs[dst];
 	/*
-	 * Only += and -= keep a pointer. Any other op makes a number of it,
+	 * Only 64-bit += and -= keep a pointer. Any other op makes a number of it,
 	 * which is fine when the program may see values of pointers anyway.
 	 */
 	to_scalar = opcode != BPF_ADD && opcode != BPF_SUB && env->allow_ptr_leaks;
@@ -16927,7 +16927,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 
 	if (BPF_CLASS(insn->code) != BPF_ALU64 && !to_scalar) {
 		/* 32-bit ALU ops on pointers produce (meaningless) scalars */
-		if (opcode == BPF_SUB && env->allow_ptr_leaks) {
+		if (env->allow_ptr_leaks) {
 			bpf_mark_reg_unknown(dst_reg);
 			return 0;
 		}
@@ -18218,11 +18218,11 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
 	if (BPF_SRC(insn->code) == BPF_X) {
 		if (src_reg->type != SCALAR_VALUE) {
 			if (dst_reg->type != SCALAR_VALUE) {
-				/* Combining two pointers by any ALU op yields
-				 * an arbitrary scalar. Disallow all math except
-				 * pointer subtraction
+				/*
+				 * Combining two pointers by any ALU op yields
+				 * an arbitrary scalar.
 				 */
-				if (opcode == BPF_SUB && env->allow_ptr_leaks) {
+				if (env->allow_ptr_leaks) {
 					mark_reg_unknown(env, regs, insn->dst_reg);
 					return 0;
 				}
diff --git a/tools/testing/selftests/bpf/progs/verifier_array_access.c b/tools/testing/selftests/bpf/progs/verifier_array_access.c
index 9333321c3ac8..14d16f35f8ac 100644
--- a/tools/testing/selftests/bpf/progs/verifier_array_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_array_access.c
@@ -311,8 +311,8 @@ l0_%=:	exit;						\
 
 SEC("socket")
 __description("invalid map access into an array with a invalid max check")
-__failure __msg("R0 pointer += pointer")
-__failure_unpriv
+__failure __msg("R0 invalid mem access 'scalar'")
+__failure_unpriv __msg_unpriv("R0 pointer += pointer")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void with_a_invalid_max_check_2(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 07c356fde68d..abe2b1911079 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -3842,10 +3842,9 @@ __naked void ptr_map_value_jmp32(void)
 }
 
 SEC("xdp")
-__failure
+__success
 __log_level(2)
 __msg("loop header at 11, can't compute iterations count")
-__msg("R6 32-bit pointer arithmetic prohibited")
 __naked void ptr_map_value_alu32_latch(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
index b9d96366e656..1d255b04565d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
@@ -140,8 +140,8 @@ __naked void unpriv_add_const_to_pointer(void)
 
 SEC("socket")
 __description("unpriv: add pointer to pointer")
-__failure __msg("R1 pointer += pointer")
-__failure_unpriv
+__success __failure_unpriv __msg_unpriv("R1 pointer += pointer")
+__retval(0)
 __naked void unpriv_add_pointer_to_pointer(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c b/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c
index 209d5be55155..455f2399492b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c
@@ -46,8 +46,8 @@ l0_%=:	exit;						\
 
 SEC("socket")
 __description("map element value illegal alu op, 2")
-__failure __msg("R0 32-bit pointer arithmetic prohibited")
-__failure_unpriv
+__failure __msg("R0 invalid mem access 'scalar'")
+__failure_unpriv __msg_unpriv("R0 32-bit pointer arithmetic prohibited")
 __naked void value_illegal_alu_op_2(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
index b3b701b44550..454b9a92d5c1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
@@ -1156,8 +1156,8 @@ l0_%=:	r0 = 2;						\
 
 SEC("socket")
 __description("map access: value_ptr += value_ptr")
-__failure __msg("R0 pointer += pointer prohibited")
-__failure_unpriv
+__failure __msg("R0 invalid mem access 'scalar'")
+__failure_unpriv __msg_unpriv("R0 pointer += pointer prohibited")
 __naked void access_value_ptr_value_ptr_1(void)
 {
 	asm volatile ("					\
-- 
2.55.0

[PATCH bpf-next v4 38/46] selftests/bpf: Adjust tests to log lines that are gone

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Without this patch 114 subtests of test_progs and 4 tests of test_verifier
fail. Their progs get the old verdict, but the tests match lines of the
verifier log that the kernel does not print any more. Keep the progs. Match
the line of the new log that shows the same fact.

  [P] "bpf: Remove precision tracking"
  [V] "bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth"
  [F] "bpf: Verify programs as a fixpoint over the control flow graph"

  the tests matched        they match now
  -----------------------  ------------------------------------------------
  [P] "mark_precise: ..",  The value of the scalar where the prog uses it:
  462 lines                "R2=-8".
  [V] "N: safe", 23 lines  "from M to N: within". If no state covers the
                           other one: "joined", and the join.
  [F] "id=1" of a scalar,  No id: the verifier clears an id that only one
  12 lines                 live register has.

The first row is verifier_precision.c, _subprog_precision.c, _spill_fill.c,
_scalar_ids.c, stack_arg_precision.c and verifier/precise.c. The second row
is verifier_bounds_step.c, _search_pruning.c, _live_stack.c: the tests of
regsafe(), stacksafe() and range_within(). The target of a jump is verified
before the fall-through. Thus the tests in which the first of two states
must cover the second one swap their branches.

Three verdicts change. short_loop1 was "too large". Now one state at the
loop head covers all iterations, and it loads. 2 unprivileged progs of
verifier_value_ptr_arith.c load: r1 = [0, 3] and r1 = [0, 7] are joined in
front of 'r1 += r0', and the insn gets one limit for its mask. 8 progs are
rejected as before, with a different message. BPF_F_TEST_STATE_FREQ has no
effect. Remove it from the tests that change.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/progs/aggregate_ret_func.c  |  15 -
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |   7 -
 .../testing/selftests/bpf/progs/dynptr_fail.c |   4 +-
 .../selftests/bpf/progs/exceptions_assert.c   |  24 +-
 .../selftests/bpf/progs/stack_arg_precision.c |  48 +--
 .../selftests/bpf/progs/test_global_func15.c  |   5 -
 .../bpf/progs/verifier_array_access.c         |  14 +-
 .../bpf/progs/verifier_bounds_deduction.c     |   2 +-
 .../bpf/progs/verifier_bounds_step.c          |  73 ++--
 .../bpf/progs/verifier_linked_scalars.c       |   7 +-
 .../selftests/bpf/progs/verifier_live_stack.c |  34 +-
 .../bpf/progs/verifier_map_ret_val.c          |   2 +-
 .../bpf/progs/verifier_mem_size_reg.c         |   8 +-
 .../selftests/bpf/progs/verifier_precision.c  | 186 +++------
 .../selftests/bpf/progs/verifier_scalar_ids.c | 329 +++++-----------
 .../bpf/progs/verifier_search_pruning.c       |  40 +-
 .../selftests/bpf/progs/verifier_spill_fill.c | 224 ++++-------
 .../selftests/bpf/progs/verifier_spin_lock.c  |   4 +-
 .../selftests/bpf/progs/verifier_stack_ptr.c  |  13 +-
 .../bpf/progs/verifier_subprog_precision.c    | 353 +++---------------
 .../bpf/progs/verifier_value_or_null.c        |   2 +-
 .../bpf/progs/verifier_value_ptr_arith.c      |   6 +-
 .../selftests/bpf/progs/verifier_var_off.c    |   3 -
 .../testing/selftests/bpf/verifier/precise.c  |  69 +---
 24 files changed, 399 insertions(+), 1073 deletions(-)

diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
index f0077d951a25..2671323255cd 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -121,14 +121,6 @@ static __naked __noinline u128 static_agg_precise(void)
 SEC("tc")
 __load_if_JITed()
 __success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
-__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
 __naked int aggregate_ret_static_precise(void)
 {
 	asm volatile (
@@ -150,13 +142,6 @@ __naked int aggregate_ret_static_precise(void)
 SEC("tc")
 __load_if_JITed()
 __success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
 __naked int aggregate_ret_global_precise(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
index 0361a7b42c9b..102156175ba0 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -31,13 +31,6 @@ SEC("tc")
 __arch_x86_64 __arch_arm64
 __load_if_JITed()
 __success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
 __naked int aggregate_ret_kfunc_precise(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c
index 7f8a38c37d9e..2425e05a12a7 100644
--- a/tools/testing/selftests/bpf/progs/dynptr_fail.c
+++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c
@@ -127,7 +127,7 @@ static int missing_release_callback_fn(__u32 index, void *data)
 
 /* A callback cannot return with the last dynptr for a referenced resource. */
 SEC("?raw_tp")
-__failure __msg("cannot overwrite referenced dynptr")
+__failure __msg("Unreleased reference id=1")
 int ringbuf_missing_release_callback(void *ctx)
 {
 	bpf_loop(10, missing_release_callback_fn, NULL, 0);
@@ -782,7 +782,7 @@ int dynptr_pruning_overwrite(struct __sk_buff *ctx)
 }
 
 SEC("?tc")
-__success __msg("12: safe") __log_level(2)
+__success __msg("to 12: within") __log_level(2)
 int dynptr_pruning_stacksafe(struct __sk_buff *ctx)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/exceptions_assert.c b/tools/testing/selftests/bpf/progs/exceptions_assert.c
index e4abf4172fca..67b51f728af2 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_assert.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_assert.c
@@ -29,32 +29,32 @@ check_assert(s64, ==, eq_llong_min, LLONG_MIN);
 __msg("R{{.}}=0x7fffffffffffffff")
 check_assert(s64, ==, eq_llong_max, LLONG_MAX);
 
-__msg("R{{.}}=scalar(id=1,smax=0x7ffffffe)")
+__msg("R{{.}}=scalar(smax=0x7ffffffe)")
 check_assert(s64, <, lt_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
 check_assert(s64, <, lt_zero, 0);
-__msg("R{{.}}=scalar(id=1,smax=0xffffffff7fffffff")
+__msg("R{{.}}=scalar(smax=0xffffffff7fffffff")
 check_assert(s64, <, lt_neg, INT_MIN);
 
-__msg("R{{.}}=scalar(id=1,smax=0x7fffffff)")
+__msg("R{{.}}=scalar(smax=0x7fffffff)")
 check_assert(s64, <=, le_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smax=0)")
+__msg("R{{.}}=scalar(smax=0)")
 check_assert(s64, <=, le_zero, 0);
-__msg("R{{.}}=scalar(id=1,smax=0xffffffff80000000")
+__msg("R{{.}}=scalar(smax=0xffffffff80000000")
 check_assert(s64, <=, le_neg, INT_MIN);
 
-__msg("R{{.}}=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >, gt_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >, gt_zero, 0);
-__msg("R{{.}}=scalar(id=1,smin=0xffffffff80000001")
+__msg("R{{.}}=scalar(smin=0xffffffff80000001")
 check_assert(s64, >, gt_neg, INT_MIN);
 
-__msg("R{{.}}=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >=, ge_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >=, ge_zero, 0);
-__msg("R{{.}}=scalar(id=1,smin=0xffffffff80000000")
+__msg("R{{.}}=scalar(smin=0xffffffff80000000")
 check_assert(s64, >=, ge_neg, INT_MIN);
 
 SEC("?tc")
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_precision.c b/tools/testing/selftests/bpf/progs/stack_arg_precision.c
index c9564776d733..6c6f58a67eff 100644
--- a/tools/testing/selftests/bpf/progs/stack_arg_precision.c
+++ b/tools/testing/selftests/bpf/progs/stack_arg_precision.c
@@ -23,7 +23,7 @@ int __btf_kfunc_gen(struct __sk_buff *ctx)
 }
 
 /*
- * Test precision backtracking across bpf-to-bpf call for kfunc stack arg.
+ * Test that the range of a stack arg is known across bpf-to-bpf call for kfunc stack arg.
  * subprog_call_mem_kfunc receives a size as incoming stack arg (arg6)
  * and forwards it as mem__sz (arg7) to bpf_kfunc_call_stack_arg_mem.
  */
@@ -52,52 +52,12 @@ static long subprog_call_mem_kfunc(long a, long b, long c, long d, long e, long
 }
 
 SEC("tc")
-__description("stack_arg: precision backtracking across bpf2bpf call for kfunc")
+__description("stack_arg: range across bpf2bpf call for kfunc")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 9 to 11: joined")
+__msg("13: (79) r1 = *(u64 *)(r11 +8)        ; R1=scalar(smin=umin=smin32=umin32=4,smax=umax=smax32=umax32=6,var_off=(0x4; 0x2),step=0+2)")
 __btf_func_path("btf__stack_arg_precision.bpf.o")
-__msg("mark_precise: frame1: last_idx 26 first_idx 13 subseq_idx -1")
-__msg("mark_precise: frame1: regs= stack= before 25: (b7) r5 = 5")
-__msg("mark_precise: frame1: regs= stack= before 24: (b7) r4 = 4")
-__msg("mark_precise: frame1: regs= stack= before 23: (b7) r3 = 3")
-__msg("mark_precise: frame1: regs= stack= before 22: (b7) r2 = 2")
-__msg("mark_precise: frame1: regs= stack= before 21: (b7) r1 = 1")
-__msg("mark_precise: frame1: regs= stack= before 20: (7b) *(u64 *)(r11 -16) = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 19: (7b) *(u64 *)(r11 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 18: (07) r2 += -8")
-__msg("mark_precise: frame1: regs=r1 stack= before 17: (bf) r2 = r10")
-__msg("mark_precise: frame1: regs=r1 stack= before 16: (7b) *(u64 *)(r10 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 14: (18) r2 = 0x807060504030201")
-__msg("mark_precise: frame1: regs=r1 stack= before 13: (79) r1 = *(u64 *)(r11 +8)")
-__msg("mark_precise: frame1: parent state regs= stack=:  frame1: R10=fp0")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame1: last_idx 11 first_idx 11 subseq_idx 13")
-__msg("mark_precise: frame1: regs= stack= before 11: (85) call pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx 11")
-__msg("mark_precise: frame0: regs= stack= before 9: (05) goto pc+1")
-__msg("mark_precise: frame0: regs= stack= before 8: (7a) *(u64 *)(r11 -8) = 4")
-__msg("mark_precise: frame1: last_idx 26 first_idx 13 subseq_idx -1 ")
-__msg("mark_precise: frame1: regs= stack= before 25: (b7) r5 = 5")
-__msg("mark_precise: frame1: regs= stack= before 24: (b7) r4 = 4")
-__msg("mark_precise: frame1: regs= stack= before 23: (b7) r3 = 3")
-__msg("mark_precise: frame1: regs= stack= before 22: (b7) r2 = 2")
-__msg("mark_precise: frame1: regs= stack= before 21: (b7) r1 = 1")
-__msg("mark_precise: frame1: regs= stack= before 20: (7b) *(u64 *)(r11 -16) = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 19: (7b) *(u64 *)(r11 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 18: (07) r2 += -8")
-__msg("mark_precise: frame1: regs=r1 stack= before 17: (bf) r2 = r10")
-__msg("mark_precise: frame1: regs=r1 stack= before 16: (7b) *(u64 *)(r10 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 14: (18) r2 = 0x807060504030201")
-__msg("mark_precise: frame1: regs=r1 stack= before 13: (79) r1 = *(u64 *)(r11 +8)")
-__msg("mark_precise: frame1: parent state regs= stack=:  frame1: R10=fp0")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame1: last_idx 11 first_idx 11 subseq_idx 13 ")
-__msg("mark_precise: frame1: regs= stack= before 11: (85) call pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R10=fp0")
-__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx 11 ")
-__msg("mark_precise: frame0: regs= stack= before 10: (7a) *(u64 *)(r11 -8) = 6")
 __naked void stack_arg_precision_bpf2bpf(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/test_global_func15.c b/tools/testing/selftests/bpf/progs/test_global_func15.c
index 201cc000b3f4..497f09dc8876 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func15.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func15.c
@@ -24,12 +24,7 @@ int global_func15(struct __sk_buff *skb)
 }
 
 SEC("cgroup_skb/ingress")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
 __failure
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (b7) r0 = 1")
-/* check that branch code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7")
 __msg("At program exit the register R0 has ")
 __naked int global_func15_tricky_pruning(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_array_access.c b/tools/testing/selftests/bpf/progs/verifier_array_access.c
index 14d16f35f8ac..7f98822889ca 100644
--- a/tools/testing/selftests/bpf/progs/verifier_array_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_array_access.c
@@ -248,7 +248,7 @@ l0_%=:	exit;						\
 SEC("socket")
 __description("invalid map access into an array with no floor check")
 __failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void array_with_no_floor_check(void)
 {
@@ -280,7 +280,7 @@ l0_%=:	exit;						\
 SEC("socket")
 __description("invalid map access into an array with a invalid max check")
 __failure __msg("invalid access to map value, value_size=48 off=44 size=8")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 max value is outside of the allowed memory range")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void with_a_invalid_max_check_1(void)
 {
@@ -541,8 +541,7 @@ l0_%=:	exit;						\
 
 SEC("socket")
 __description("valid map access into an array using constant without nullness")
-__success __retval(4) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(4)
 unsigned int an_array_with_a_constant_no_nullness(void)
 {
 	/* Need 8-byte alignment for spill tracking */
@@ -557,9 +556,7 @@ unsigned int an_array_with_a_constant_no_nullness(void)
 
 SEC("socket")
 __description("valid multiple map access into an array using constant without nullness")
-__success __retval(8) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -16) = {{(0|r[0-9])}}")
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(8)
 unsigned int multiple_array_with_a_constant_no_nullness(void)
 {
 	__u32 __attribute__((aligned(8))) key = 1;
@@ -595,8 +592,7 @@ unsigned int an_array_with_a_32bit_constant_0_no_nullness(void)
 
 SEC("socket")
 __description("valid map access into a pcpu array using constant without nullness")
-__success __retval(4) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(4)
 unsigned int a_pcpu_array_with_a_constant_no_nullness(void)
 {
 	__u32 __attribute__((aligned(8))) key = 1;
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
index 260a6df264e3..a474c9978a35 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
@@ -102,7 +102,7 @@ l0_%=:	r0 -= r1;					\
 SEC("socket")
 __description("check deducing bounds from const, 7")
 __failure __msg("dereference of modified ctx ptr")
-__msg_unpriv("R1 has pointer with unsupported alu operation")
+__msg_unpriv("dereference of modified ctx ptr")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void deducing_bounds_from_const_7(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
index d824a5d48a39..51b703058cf7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
@@ -371,14 +371,17 @@ struct {
 	__type(value, struct step_val);
 } step_map SEC(".maps");
 
-/* Old register [4..130, step 2] should prune cur register [8..68, step 4]. */
+/*
+ * Register [8..68, step 4] is within register [4..130, step 2]: one state stands for both
+ * and nothing is lost.
+ */
 SEC("socket")
 __success __log_level(2)
 __msg("7: (27) r6 *= 4                       ; R6=scalar({{.*}}umin32=8,{{.*}}umax32=68,{{.*}},step=0+4)")
 __msg("10: (27) r7 *= 2                      ; R7=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2)")
 __msg("11: (25) if r0 > 0x2a goto pc+1")
-__msg("from 11 to 13: safe")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("13: R6=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2) R10=fp0")
+__msg("processed 18 insns")
 __naked void step_prune_hit_multiple(void)
 {
 	asm volatile ("					\
@@ -394,11 +397,11 @@ __naked void step_prune_hit_multiple(void)
 	r7 += 2;					\
 	r7 *= 2;					\
 	if r0 > 42 goto 1f;	/* can't predict */	\
-	r6 = r7;		/* step=2 explored first, step=4 explored next */ \
+	r6 = r7;		/* step=2 here, step=4 on the other path */ \
 1:	r0 = r10;					\
 	r6 = -r6;					\
 	r0 += r6;					\
-	*(u8 *)(r0 + 0) = 7;	/* force r6 precise */	\
+	*(u8 *)(r0 + 0) = 7;				\
 	exit;						\
 "	:
 	: __imm(bpf_get_prandom_u32)
@@ -406,17 +409,16 @@ __naked void step_prune_hit_multiple(void)
 }
 
 /*
- * Old register [0..189, step 3] should not prune cur register [0..30, step 2].
- * Bounds and tnum containment allow pruning; step divisibility must prevent it.
+ * Register [0..30, step 2] is not within register [0..189, step 3].
+ * Bounds and tnum say that it is; step divisibility must prevent it.
+ * The states are joined and the result has no step.
  */
 SEC("socket")
 __success __log_level(2)
 __msg("6: (27) r6 *= 2                       ; R6=scalar({{.*}}smin32=0,{{.*}}umax32=30,{{.*}},step=0+2)")
 __msg("8: (27) r7 *= 3                       ; R7=scalar({{.*}}smin32=0,{{.*}}umax32=189,{{.*}},step=0+3)")
 __msg("9: (25) if r0 > 0x2a goto pc+1")
-__msg("17: (95) exit")
-__msg("from 9 to 11: R6=scalar({{.*}},step=0+2)")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("11: R6=scalar({{.*}}umax32=189,var_off=(0x0; 0xff)) R10=fp0")
 __naked void step_prune_miss_non_multiple(void)
 {
 	asm volatile ("					\
@@ -430,13 +432,13 @@ __naked void step_prune_miss_non_multiple(void)
 	r7 &= 0x3f;					\
 	r7 *= 3;					\
 	if r0 > 42 goto 1f;	/* can't predict */	\
-	r6 = r7;		/* step=3 explored first, step=2 explored next */ \
+	r6 = r7;		/* step=3 here, step=2 on the other path */ \
 1:							\
 	r2 = r10;					\
 	r6 += 1;					\
 	r6 = -r6;					\
 	r2 += r6;					\
-	*(u8 *)(r2 + 0) = 7;	/* force r6 precise */	\
+	*(u8 *)(r2 + 0) = 7;				\
 	r0 = 0;						\
 	exit;						\
 "	:
@@ -445,17 +447,17 @@ __naked void step_prune_miss_non_multiple(void)
 }
 
 /*
- * Constant current register lying on the cached line: cached is {0,3,6,...}
- * (step 3, base 0), current is the constant 6. range_within() takes the
- * constant branch: imod(6, 3) == base 0, so cur is on the line and the
- * (precise) state is pruned -> "safe".
+ * Constant register lying on the line of the other state: that is {0,3,6,...}
+ * (step 3, base 0), this is the constant 6. range_within() takes the
+ * constant branch: imod(6, 3) == base 0, so the constant is on the line and
+ * the state with the line stands for both.
  */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("14: (27) r1 *= 3")		/* cached path: step 3 line */
-__msg("16: (b7) r1 = 6")		/* current path: const 6, on the line */
-__msg("17: safe")			/* pruned at join: imod(6, 3) == 0 */
+__msg("16: (b7) r1 = 6")		/* const 6, on the line */
+__msg("14: (27) r1 *= 3")		/* step 3 line */
+__msg("from 15 to 17: joined")
+__msg("17: R1=scalar({{.*}},step=0+3) R7=")	/* imod(6, 3) == 0 */
 __naked void step_prune_hit_const_on_line(void)
 {
 	asm volatile ("					\
@@ -472,13 +474,13 @@ __naked void step_prune_hit_const_on_line(void)
 	r1 = r6;					\
 	r1 &= 0xff;					\
 	if r6 > 0 goto l_cur_%=;			\
-	r1 *= 3;			/* old: step 3 */	\
+	r1 *= 3;			/* step 3 */	\
 	goto l_join_%=;					\
 l_cur_%=:						\
-	r1 = 6;				/* cur: const on line */	\
+	r1 = 6;				/* const on line */	\
 l_join_%=:						\
 	r0 = r7;					\
-	r0 += r1;			/* r1 forced precise */	\
+	r0 += r1;					\
 	r2 = *(u8 *)(r0 + 0);				\
 l_out_%=:						\
 	r0 = 0;						\
@@ -491,22 +493,21 @@ l_out_%=:						\
 }
 
 /*
- * Constant current register NOT on the cached line: cached is {0,3,6,...}
- * (step 3, base 0), current is the constant 7. imod(7, 3) == 1 != base 0,
- * so range_within() fails and the join is traversed again. A power-of-two
+ * Constant register NOT on the line of the other state: that is {0,3,6,...}
+ * (step 3, base 0), this is the constant 7. imod(7, 3) == 1 != base 0,
+ * so range_within() fails and the states are joined. A power-of-two
  * step is avoided on purpose: with step 3 the tnum is loose enough to admit
  * 7, so imod() is
  * the sole check that rejects it. No failure shape is possible here: 7 is
- * within the line's bounds and tnum, and the cached line path already
- * verifies the whole outro, so an eager prune could not miss an error.
+ * within the line's bounds and tnum.
  */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("14: (27) r1 *= 3")		/* cached path: step 3 line */
-__msg("16: (b7) r1 = 7")		/* current path: const 7, off the line */
-/* not pruned: current continues past the join with the constant offset 7 */
-__msg("19: R0=map_value({{.*}}imm=7) R1=7")
+__msg("16: (b7) r1 = 7")		/* const 7, off the line */
+__msg("14: (27) r1 *= 3")		/* step 3 line */
+/* the result stands for 7 too: it has no step */
+__msg("from 15 to 17: joined")
+__msg("17: R1=scalar({{.*}}var_off=(0x0; 0x3ff)) R7=")
 __naked void step_prune_miss_const_off_line(void)
 {
 	asm volatile ("					\
@@ -523,13 +524,13 @@ __naked void step_prune_miss_const_off_line(void)
 	r1 = r6;					\
 	r1 &= 0xff;					\
 	if r6 > 0 goto l_cur_%=;			\
-	r1 *= 3;			/* old: step 3 */	\
+	r1 *= 3;			/* step 3 */	\
 	goto l_join_%=;					\
 l_cur_%=:						\
-	r1 = 7;				/* cur: const off line */	\
+	r1 = 7;				/* const off line */	\
 l_join_%=:						\
 	r0 = r7;					\
-	r0 += r1;			/* r1 forced precise */	\
+	r0 += r1;					\
 	r2 = *(u8 *)(r0 + 0);				\
 l_out_%=:						\
 	r0 = 0;						\
diff --git a/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c b/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c
index d571fbfc86a3..caae67feaf5b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c
+++ b/tools/testing/selftests/bpf/progs/verifier_linked_scalars.c
@@ -668,12 +668,16 @@ __naked void add_const_base_id_pruning(void)
 	call %[bpf_get_prandom_u32];				\
 	r6 = r0;						\
 	r6 &= 1;						\
-	if r6 >= 1 goto l_true_%=;				\
+	if r6 < 1 goto l_false_%=;				\
+	goto l_true_%=;						\
 								\
+l_false_%=:							\
 	/* False branch (explored first, old state) */		\
 	call %[bpf_get_prandom_u32];				\
 	r2 = r0;						\
 	r2 &= 0xff;		/* R2 = scalar(id=A) [0,255] */	\
+	r4 = r0;						\
+	r4 &= 0xff;		/* R4 = scalar [0,255], id=0 */	\
 	r3 = r2;		/* R3 linked to R2 (id=A) */	\
 	r3 += 10;		/* R3 id=A|ADD_CONST, delta=10 */\
 	r6 = 0;							\
@@ -703,6 +707,7 @@ l_merge_%=:							\
 	r9 += r3;		/* fp-16+[0,5] vs fp-16+[0,255] */\
 	*(u8*)(r9 + 0) = r6;	/* within 16B vs past fp */	\
 l_exit_%=:							\
+	r4 = r4;		/* R4 is live at the merge */	\
 	r0 = 0;							\
 	exit;							\
 "	:
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 2ae5c60150dd..795956bd4f9f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -368,7 +368,6 @@ static __used __naked void arg_track_join_convergence_subprog(void)
 }
 
 SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
 __log_level(2)
 /* fp0-8 consumed at insn 9, dead by insn 11. stack_def at insn 4 kills slots 0,1. */
 __msg("4: (7b) *(u64 *)(r10 -8) = r0        ; def: fp0-8")
@@ -380,9 +379,9 @@ __msg("10: (b7) r0 = 0{{$}}")
 __msg("11: (05) goto pc+0{{$}}")
 __msg("12: (95) exit")
 /*
- * Checkpoint at goto +0 fires because fp0-8 is dead → state pruning.
+ * fp0-8 is dead after goto +0, so one state stands for both.
  */
-__msg("12: safe")
+__msg("from 11 to 12: within")
 __naked void caller_stack_pruning(void)
 {
 	asm volatile (
@@ -497,7 +496,6 @@ struct {
 
 SEC("socket")
 __failure __msg("invalid read from stack R2 off=-4096 size=8")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked unsigned long caller_stack_write_tail_call(void)
 {
         asm volatile (
@@ -855,7 +853,7 @@ static __used __naked void maybe_clobber_key(void)
 
 /*
  * Demonstrate that subprog arg spill/reload breaks arg tracking,
- * inflating caller stack liveness and preventing state pruning.
+ * inflating caller stack liveness and keeping states apart.
  *
  * modifier2(fp-24) has two paths: one writes a scalar to *(r1+8)
  * = caller fp-16, the other leaves it as zero.  After modifier2
@@ -871,16 +869,15 @@ static __used __naked void maybe_clobber_key(void)
  * the reader never touches it.
  *
  * Result: at modifier2's return point two states with different
- * fp-16 values cannot be pruned.
+ * fp-16 values are kept apart.
  *
- * With correct (or old dynamic) liveness fp-16 is dead at that
- * point and the states prune → "6: safe" appears in the log.
+ * With correct liveness fp-16 is dead at that point and one state is
+ * within the other one.
  */
 SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
 __log_level(2)
 __success
-__msg("6: safe")
+__msg("from 16 to 6: within")
 __naked void spill_reload_inflates_stack_liveness(void)
 {
 	asm volatile (
@@ -1366,31 +1363,32 @@ __naked void half_spi_clean_preserves_stack_zero(void)
  * conditional block with helper call and temporary stack spill,
  * spill dead after merge.
  *
- * Path A (fall-through): spill r6 to fp-8 across helper call
- * Path B (branch taken): skip the block entirely
- * At merge (insn 6): fp-8 is dead (never read after merge)
+ * Path A (branch taken, verified first): spill r6 to fp-8 across helper call
+ * Path B (fall-through): skip the block entirely
+ * At merge (insn 7): fp-8 is dead (never read after merge)
  *
  * Static liveness marks fp-8 dead at merge. clean_verifier_state()
  * converts path A's STACK_SPILL to STACK_INVALID. Path B has
- * STACK_INVALID. stacksafe() matches -> path B pruned -> "6: safe".
+ * STACK_INVALID. stacksafe() matches -> one state stands for both paths.
  */
 SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
 __success
 __log_level(2)
-__msg("6: safe")
+__msg("from 3 to 7: within")
 __naked void dead_spill_at_merge_enables_pruning(void)
 {
 	asm volatile (
 	"call %[bpf_get_prandom_u32];"
 	"r6 = 7;"
-	"if r0 != 0 goto l_skip%=;"
+	"if r0 == 0 goto l_spill%=;"
+	"goto l_skip%=;"
+"l_spill%=:"
 	/* conditional block: spill, call, reload */
 	"*(u64 *)(r10 - 8) = r6;"
 	"call %[bpf_get_prandom_u32];"
 	"r6 = *(u64 *)(r10 - 8);"
 "l_skip%=:"
-	/* fp-8 dead. Path B pruned here -> "6: safe" */
+	/* fp-8 dead. The states of both paths are the same here */
 	"r0 = r6;"
 	"exit;"
 	:
diff --git a/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c b/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
index 1639628b832d..2b65ab508596 100644
--- a/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
+++ b/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
@@ -83,7 +83,7 @@ l0_%=:	exit;						\
 SEC("socket")
 __description("sometimes access memory with incorrect alignment")
 __failure __msg("R0 invalid mem access")
-__msg_unpriv("R0 leaks addr")
+__msg_unpriv("R0 invalid mem access")
 __flag(BPF_F_STRICT_ALIGNMENT)
 __naked void access_memory_with_incorrect_alignment_2(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
index 7e24706a764e..aba9510a1c08 100644
--- a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
@@ -7,13 +7,9 @@
 
 char _license[] SEC("license") = "GPL";
 
-/*
- * The __szk size of a kfunc memory/size pair must be marked precise even when
- * the nullable buffer is passed as NULL.
- */
+/* The __szk size of a kfunc memory/size pair when the nullable buffer is passed as NULL */
 SEC("?tc")
-__success __log_level(2)
-__msg("mark_precise: frame0: regs=r4 stack= before")
+__success
 int dynptr_slice_null_buf_size_precise(struct __sk_buff *skb)
 {
 	struct bpf_dynptr dptr;
diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c
index 71ac80a984d8..f0f9b62494ca 100644
--- a/tools/testing/selftests/bpf/progs/verifier_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_precision.c
@@ -16,10 +16,7 @@ struct {
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0xfffffff8 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (87) r2 = -r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 8")
+__msg("1: (87) r2 = -r2                      ; R2=-8")
 __naked int bpf_neg(void)
 {
 	asm volatile (
@@ -36,10 +33,7 @@ __naked int bpf_neg(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (d4) r2 = le16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (d4) r2 = le16 r2                  ; R2=0")
 __naked int bpf_end_to_le(void)
 {
 	asm volatile (
@@ -57,10 +51,7 @@ __naked int bpf_end_to_le(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (dc) r2 = be16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (dc) r2 = be16 r2                  ; R2=0")
 __naked int bpf_end_to_be(void)
 {
 	asm volatile (
@@ -82,10 +73,7 @@ __naked int bpf_end_to_be(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (d7) r2 = bswap16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (d7) r2 = bswap16 r2               ; R2=0")
 __naked int bpf_end_bswap(void)
 {
 	asm volatile (
@@ -104,10 +92,8 @@ __naked int bpf_end_bswap(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (db) r2 = load_acquire((u64 *)(r10 -8))")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("2: (db) r2 = load_acquire((u64 *)(r10 -8)){{ *}}; R2=8 R10=fp0 fp-8=8")
+__msg("4: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_load_acquire(void)
 {
 	asm volatile (
@@ -115,7 +101,7 @@ __naked int bpf_load_acquire(void)
 	"*(u64 *)(r10 - 8) = r1;"
 	".8byte %[load_acquire_insn];" /* r2 = load_acquire((u64 *)(r10 - 8)); */
 	"r3 = r10;"
-	"r3 += r2;" /* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -126,10 +112,8 @@ __naked int bpf_load_acquire(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r2 = r10")
-__msg("mark_precise: frame0: regs=r1 stack= before 2: (79) r1 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (db) store_release((u64 *)(r10 -8), r1)")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("1: (db) store_release((u64 *)(r10 -8), r1){{ *}}; R1=8 R10=fp0 fp-8=8")
+__msg("4: (0f) r2 += r1                      ; R1=8 R2=fp8")
 __naked int bpf_store_release(void)
 {
 	asm volatile (
@@ -137,7 +121,7 @@ __naked int bpf_store_release(void)
 	".8byte %[store_release_insn];" /* store_release((u64 *)(r10 - 8), r1); */
 	"r1 = *(u64 *)(r10 - 8);"
 	"r2 = r10;"
-	"r2 += r1;" /* mark_precise */
+	"r2 += r1;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -150,24 +134,7 @@ __naked int bpf_store_release(void)
 #endif /* v4 instruction */
 
 SEC("?raw_tp")
-__success __log_level(2)
-/*
- * Without the bug fix there will be no history between "last_idx 3 first_idx 3"
- * and "parent state regs=" lines. "R0=6" parts are here to help anchor
- * expected log messages to the one specific mark_chain_precision operation.
- *
- * This is quite fragile: if verifier checkpointing heuristic changes, this
- * might need adjusting.
- */
-__msg("2: (47) r0 |= 0                       ; R0=6")
-__msg("3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (47) r0 |= 0")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 2")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R0=P4")
-__msg("3: R0=6")
+__success
 __naked int state_loop_first_last_equal(void)
 {
 	asm volatile (
@@ -206,10 +173,6 @@ __success __log_level(2)
 __msg("8: (bd) if r2 <= r10 goto pc+3")
 __msg("9: (35) if r1 >= 0xffe3fff8 goto pc+0")
 __msg("10: (b5) if r2 <= 0x8 goto pc+0")
-__msg("mark_precise: frame1: last_idx 10 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame1: regs=r2 stack= before 9: (35) if r1 >= 0xffe3fff8 goto pc+0")
-__msg("mark_precise: frame1: regs=r2 stack= before 8: (bd) if r2 <= r10 goto pc+3")
-__msg("mark_precise: frame1: regs=r2 stack= before 7: (05) goto pc+0")
 __naked void bpf_cond_op_r10(void)
 {
 	asm volatile (
@@ -225,9 +188,6 @@ __success __log_level(2)
 __msg("3: (bf) r3 = r10")
 __msg("4: (bd) if r3 <= r2 goto pc+1")
 __msg("5: (b5) if r2 <= 0x8 goto pc+2")
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bd) if r3 <= r2 goto pc+1")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r3 = r10")
 __naked void bpf_cond_op_not_r10(void)
 {
 	asm volatile (
@@ -245,9 +205,6 @@ SEC("lsm.s/socket_connect")
 __success __log_level(2)
 __msg("0: (b7) r0 = 1                        ; R0=1")
 __msg("1: (84) w0 = -w0                      ; R0=0xffffffff")
-__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (84) w0 = -w0")
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
 __naked int bpf_neg_2(void)
 {
 	/*
@@ -280,9 +237,6 @@ SEC("lsm.s/socket_connect")
 __success __log_level(2)
 __msg("0: (b7) r0 = 1                        ; R0=1")
 __msg("1: (87) r0 = -r0                      ; R0=-1")
-__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (87) r0 = -r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
 __naked int bpf_neg_4(void)
 {
 	/*
@@ -313,11 +267,8 @@ __naked int bpf_neg_5(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_add_precision(void)
 {
 	asm volatile (
@@ -326,7 +277,7 @@ __naked int bpf_atomic_fetch_add_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_add_insn];"	/* r2 = atomic_fetch_add(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -337,11 +288,8 @@ __naked int bpf_atomic_fetch_add_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_xchg((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_xchg((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_xchg_precision(void)
 {
 	asm volatile (
@@ -350,7 +298,7 @@ __naked int bpf_atomic_xchg_precision(void)
 	"r2 = 0;"
 	".8byte %[xchg_insn];"		/* r2 = atomic_xchg(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -361,11 +309,8 @@ __naked int bpf_atomic_xchg_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_or((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_or((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_or_precision(void)
 {
 	asm volatile (
@@ -374,7 +319,7 @@ __naked int bpf_atomic_fetch_or_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_or_insn];"	/* r2 = atomic_fetch_or(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -385,11 +330,8 @@ __naked int bpf_atomic_fetch_or_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_and((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_and((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_and_precision(void)
 {
 	asm volatile (
@@ -398,7 +340,7 @@ __naked int bpf_atomic_fetch_and_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_and_insn];"	/* r2 = atomic_fetch_and(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -409,11 +351,8 @@ __naked int bpf_atomic_fetch_and_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_xor((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_xor((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2                      ; R2=8 R3=fp8")
 __naked int bpf_atomic_fetch_xor_precision(void)
 {
 	asm volatile (
@@ -422,7 +361,7 @@ __naked int bpf_atomic_fetch_xor_precision(void)
 	"r2 = 0;"
 	".8byte %[fetch_xor_insn];"	/* r2 = atomic_fetch_xor(*(u64 *)(r10 - 8), r2) */
 	"r3 = r10;"
-	"r3 += r2;"			/* mark_precise */
+	"r3 += r2;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -433,12 +372,8 @@ __naked int bpf_atomic_fetch_xor_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r0 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2){{ *}}; R0=8 R2=0 R10=fp0 fp-8=mmmmmmmm")
+__msg("6: (0f) r3 += r0                      ; R0=8 R3=fp8")
 __naked int bpf_atomic_cmpxchg_precision(void)
 {
 	asm volatile (
@@ -448,7 +383,7 @@ __naked int bpf_atomic_cmpxchg_precision(void)
 	"r2 = 0;"
 	".8byte %[cmpxchg_insn];"	/* r0 = atomic_cmpxchg(*(u64 *)(r10 - 8), r0, r2) */
 	"r3 = r10;"
-	"r3 += r0;"			/* mark_precise */
+	"r3 += r0;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -457,19 +392,15 @@ __naked int bpf_atomic_cmpxchg_precision(void)
 	: __clobber_all);
 }
 
-/* Regression test for dual precision: Both the fetched value (r2) and
- * a reread of the same stack slot (r3) are tracked for precision. After
- * the atomic operation, the stack slot is STACK_MISC. Thus, the ldx at
- * insn 4 does NOT set INSN_F_STACK_ACCESS. Precision for the stack slot
- * propagates solely through the atomic fetch's load side (insn 3).
+/*
+ * Both the fetched value (r2) and a reread of the same stack slot (r3) are used.
+ * The fetched value is known. After the atomic operation, the stack slot is
+ * STACK_MISC, so what is reread is not.
  */
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r2,r3 stack= before 4: (79) r3 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("5: (bf) r4 = r2                       ; R2=8 R4=8")
+__msg("6: (0f) r4 += r3                      ; R3=scalar() R4=scalar()")
 __naked int bpf_atomic_fetch_add_dual_precision(void)
 {
 	asm volatile (
@@ -482,7 +413,7 @@ __naked int bpf_atomic_fetch_add_dual_precision(void)
 	"r4 += r3;"
 	"r4 &= 7;"
 	"r5 = r10;"
-	"r5 += r4;"			/* mark_precise */
+	"r5 += r4;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -493,12 +424,8 @@ __naked int bpf_atomic_fetch_add_dual_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0,r3 stack= before 5: (79) r3 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r0 = 8")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("6: (bf) r4 = r0                       ; R0=8 R4=8")
+__msg("7: (0f) r4 += r3                      ; R3=scalar() R4=scalar()")
 __naked int bpf_atomic_cmpxchg_dual_precision(void)
 {
 	asm volatile (
@@ -512,7 +439,7 @@ __naked int bpf_atomic_cmpxchg_dual_precision(void)
 	"r4 += r3;"
 	"r4 &= 7;"
 	"r5 = r10;"
-	"r5 += r4;"			/* mark_precise */
+	"r5 += r4;"
 	"r0 = 0;"
 	"exit;"
 	:
@@ -523,9 +450,7 @@ __naked int bpf_atomic_cmpxchg_dual_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 10: (57) r1 &= 7")
-__msg("mark_precise: frame0: regs=r1 stack= before 9: (db) r1 = atomic64_fetch_add((u64 *)(r0 +0), r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("9: (db) r1 = atomic64_fetch_add((u64 *)(r0 +0), r1){{ *}}; R0=map_value({{.*}}) R1=scalar()")
 __naked int bpf_atomic_fetch_add_map_precision(void)
 {
 	asm volatile (
@@ -540,7 +465,7 @@ __naked int bpf_atomic_fetch_add_map_precision(void)
 	".8byte %[fetch_add_insn];"	/* r1 = atomic_fetch_add(*(u64 *)(r0 + 0), r1) */
 	"r1 &= 7;"
 	"r2 = r10;"
-	"r2 += r1;"			/* mark_precise */
+	"r2 += r1;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -553,9 +478,7 @@ __naked int bpf_atomic_fetch_add_map_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (57) r0 &= 7")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (db) r0 = atomic64_cmpxchg((u64 *)(r6 +0), r0, r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("11: (db) r0 = atomic64_cmpxchg((u64 *)(r6 +0), r0, r1){{ *}}; R0=scalar() R1=0")
 __naked int bpf_atomic_cmpxchg_map_precision(void)
 {
 	asm volatile (
@@ -572,7 +495,7 @@ __naked int bpf_atomic_cmpxchg_map_precision(void)
 	".8byte %[cmpxchg_insn];"	/* r0 = atomic_cmpxchg(*(u64 *)(r6 + 0), r0, r1) */
 	"r0 &= 7;"
 	"r2 = r10;"
-	"r2 += r0;"			/* mark_precise */
+	"r2 += r0;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -585,9 +508,7 @@ __naked int bpf_atomic_cmpxchg_map_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 10: (57) r1 &= 7")
-__msg("mark_precise: frame0: regs=r1 stack= before 9: (c3) r1 = atomic_fetch_add((u32 *)(r0 +0), r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("9: (c3) r1 = atomic_fetch_add((u32 *)(r0 +0), r1){{ *}}; R0=map_value({{.*}}) R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
 __naked int bpf_atomic_fetch_add_32bit_precision(void)
 {
 	asm volatile (
@@ -602,7 +523,7 @@ __naked int bpf_atomic_fetch_add_32bit_precision(void)
 	".8byte %[fetch_add_insn];"	/* r1 = atomic_fetch_add(*(u32 *)(r0 + 0), r1) */
 	"r1 &= 7;"
 	"r2 = r10;"
-	"r2 += r1;"			/* mark_precise */
+	"r2 += r1;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -615,9 +536,7 @@ __naked int bpf_atomic_fetch_add_32bit_precision(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (57) r0 &= 7")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (c3) r0 = atomic_cmpxchg((u32 *)(r6 +0), r0, r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("11: (c3) r0 = atomic_cmpxchg((u32 *)(r6 +0), r0, r1){{ *}}; R0=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R1=0")
 __naked int bpf_atomic_cmpxchg_32bit_precision(void)
 {
 	asm volatile (
@@ -634,7 +553,7 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void)
 	".8byte %[cmpxchg_insn];"	/* r0 = atomic_cmpxchg(*(u32 *)(r6 + 0), r0, r1) */
 	"r0 &= 7;"
 	"r2 = r10;"
-	"r2 += r0;"			/* mark_precise */
+	"r2 += r0;"
 	"1: r0 = 0;"
 	"exit;"
 	:
@@ -648,11 +567,9 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void)
 /*
  * Verification takes two paths: with r1 being scalar zero on path (1)
  * and with r1 being some other scalar on path (2).
- * Check that the verifier does not use checkpoints created
- * on path (1) to prune path (2).
+ * Check that the state of path (1) does not stand for path (2).
  */
 SEC("?tc")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("R1 type=scalar expected=fp")
 __naked int null_mem_arg_zero_size(void)
 {
@@ -689,11 +606,9 @@ __weak int subprog_mem_arg(int *p)
 /*
  * Verification takes two paths: with r1 being scalar zero on path (1)
  * and with r1 being some other scalar on path (2).
- * Check that the verifier does not use checkpoints created
- * on path (1) to prune path (2).
+ * Check that the state of path (1) does not stand for path (2).
  */
 SEC("?raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("R1 type=scalar expected=fp")
 __naked int null_mem_arg_global_subprog(void)
 {
@@ -711,9 +626,8 @@ __naked int null_mem_arg_global_subprog(void)
 		: __clobber_all);
 }
 
-/* Same as above, check that path with r3 == 0 does not prune the path with r3 != 0 */
+/* Same as above, check that path with r3 == 0 does not stand for the path with r3 != 0 */
 SEC("?tc")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("R3 type=scalar expected=fp")
 int null_kfunc_arg_dynptr_slice(struct __sk_buff *skb)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
index 256547048cc4..9ea61b162736 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
@@ -11,27 +11,22 @@ struct {
 	__type(value, long long);
 } map_hash_8b SEC(".maps");
 
-/* Check that precision marks propagate through scalar IDs.
+/* scalar in [0, max] that shares id with other registers, as it is in the log */
+#define LINKED(id, max, mask) \
+	"scalar(id=" id ",smin=smin32=0,smax=umax=smax32=umax32=" #max ",var_off=(0x0; " #mask "))"
+#define ALONE(max, mask) \
+	"scalar(smin=smin32=0,smax=umax=smax32=umax32=" #max ",var_off=(0x0; " #mask "))"
+
+/*
  * Registers r{0,1,2} have the same scalar ID.
- * Range information is propagated for scalars sharing same ID.
- * Check that precision mark for r0 causes precision marks for r{1,2}
- * when range information is propagated for 'if <reg> <op> <const>' insn.
+ * Range information is propagated for scalars sharing same ID
+ * for 'if <reg> <op> <const>' insn.
+ * The registers share the ID in the join of both branches too.
  */
 SEC("socket")
 __success __log_level(2)
-/* first 'if' branch */
-__msg("6: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 4: (25) if r1 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2 stack=:")
-__msg("frame0: regs=r0,r1,r2 stack= before 3: (bf) r2 = r0")
-/* second 'if' branch */
-__msg("from 4 to 5: ")
-__msg("6: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 5: (bf) r3 = r10")
-__msg("frame0: regs=r0 stack= before 4: (25) if r1 > 0x7 goto pc+0")
-/* parent state already has r{0,1,2} as precise */
-__msg("frame0: parent state regs= stack=:")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("4: (25) if r1 > 0x7 goto pc+0         ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("5: R0=" LINKED("1", 255, 0xff) " R1=" LINKED("1", 255, 0xff) " R2=" LINKED("1", 255, 0xff))
 __naked void linked_regs_bpf_k(void)
 {
 	asm volatile (
@@ -42,9 +37,6 @@ __naked void linked_regs_bpf_k(void)
 	"r1 = r0;"
 	"r2 = r0;"
 	"if r1 > 7 goto +0;"
-	/* force r0 to be precise, this eventually marks r1 and r2 as
-	 * precise as well because of shared IDs
-	 */
 	"r3 = r10;"
 	"r3 += r0;"
 	/* Mark r1 and r2 as alive. */
@@ -57,16 +49,14 @@ __naked void linked_regs_bpf_k(void)
 	: __clobber_all);
 }
 
-/* Registers r{0,1,2} share same ID when 'if r1 > ...' insn is processed,
- * check that verifier marks r{1,2} as precise while backtracking
- * 'if r1 > ...' with r0 already marked.
+/*
+ * Registers r{0,1,2} share same ID when 'if r1 > r3' insn is processed,
+ * check that range information is propagated to all of them.
  */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("frame0: regs=r0 stack= before 5: (2d) if r1 > r3 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3 stack=:")
-__msg("frame0: regs=r0,r1,r2,r3 stack= before 4: (b7) r3 = 7")
+__msg("5: (2d) if r1 > r3 goto pc+0          ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("6: R0=" LINKED("1", 255, 0xff) " R1=" LINKED("1", 255, 0xff) " R2=" LINKED("1", 255, 0xff))
 __naked void linked_regs_bpf_x_src(void)
 {
 	asm volatile (
@@ -78,9 +68,6 @@ __naked void linked_regs_bpf_x_src(void)
 	"r2 = r0;"
 	"r3 = 7;"
 	"if r1 > r3 goto +0;"
-	/* force r0 to be precise, this eventually marks r1 and r2 as
-	 * precise as well because of shared IDs
-	 */
 	"r4 = r10;"
 	"r4 += r0;"
 	/* Mark r1 and r2 as alive. */
@@ -93,16 +80,11 @@ __naked void linked_regs_bpf_x_src(void)
 	: __clobber_all);
 }
 
-/* Registers r{0,1,2} share same ID when 'if r1 > r3' insn is processed,
- * check that verifier marks r{0,1,2} as precise while backtracking
- * 'if r1 > r3' with r3 already marked.
- */
+/* Same as linked_regs_bpf_x_src(), r3 is what is used after the jump. */
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("frame0: regs=r3 stack= before 5: (2d) if r1 > r3 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3 stack=:")
-__msg("frame0: regs=r0,r1,r2,r3 stack= before 4: (b7) r3 = 7")
+__msg("5: (2d) if r1 > r3 goto pc+0          ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("7: (0f) r4 += r3                      ; R3=7 R4=fp7")
 __naked void linked_regs_bpf_x_dst(void)
 {
 	asm volatile (
@@ -114,9 +96,6 @@ __naked void linked_regs_bpf_x_dst(void)
 	"r2 = r0;"
 	"r3 = 7;"
 	"if r1 > r3 goto +0;"
-	/* force r0 to be precise, this eventually marks r1 and r2 as
-	 * precise as well because of shared IDs
-	 */
 	"r4 = r10;"
 	"r4 += r3;"
 	/* Mark r1 and r2 as alive. */
@@ -131,16 +110,13 @@ __naked void linked_regs_bpf_x_dst(void)
 }
 
 /* Same as linked_regs_bpf_k, but break one of the
- * links, note that r1 is absent from regs=... in __msg below.
+ * links, note that r1 is absent from the first __msg below.
  */
 SEC("socket")
 __success __log_level(2)
-__msg("7: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 6: (bf) r3 = r10")
-__msg("frame0: parent state regs=r0 stack=:")
-__msg("frame0: regs=r0 stack= before 5: (25) if r0 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r2 stack=:")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* r1 is not there */
+__msg("5: (25) if r0 > 0x7 goto pc+0         ; R0=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7) "{{$}}")
+__msg("8: (bf) r1 = r1                       ; R1=0")
 __naked void linked_regs_broken_link(void)
 {
 	asm volatile (
@@ -155,9 +131,6 @@ __naked void linked_regs_broken_link(void)
 	 */
 	"r1 = 0;"
 	"if r0 > 7 goto +0;"
-	/* force r0 to be precise,
-	 * this eventually marks r2 as precise because of shared IDs
-	 */
 	"r3 = r10;"
 	"r3 += r0;"
 	/* Mark r1 and r2 as alive. */
@@ -170,50 +143,16 @@ __naked void linked_regs_broken_link(void)
 	: __clobber_all);
 }
 
-/* Check that precision marks propagate through scalar IDs.
- * Use the same scalar ID in multiple stack frames, check that
- * precision information is propagated up the call stack.
+/*
+ * Use the same scalar ID in multiple stack frames. A static function is verified
+ * on its own: registers of the callers are not there when r1 is compared, so it has no ID.
+ * They share their IDs after the calls like before, if there is more than one left.
  */
 SEC("socket")
 __success __log_level(2)
-__msg("17: (0f) r2 += r1")
-/* Current state */
-__msg("frame2: last_idx 17 first_idx 14 subseq_idx -1 ")
-__msg("frame2: regs=r1 stack= before 16: (bf) r2 = r10")
-__msg("frame2: parent state regs=r1 stack=")
-__msg("frame1: parent state regs= stack=")
-__msg("frame0: parent state regs= stack=")
-/* Parent state */
-__msg("frame2: last_idx 13 first_idx 13 subseq_idx 14 ")
-__msg("frame2: regs=r1 stack= before 13: (25) if r1 > 0x7 goto pc+0")
-__msg("frame2: parent state regs=r1 stack=")
-/* frame1.r{6,7} are marked because mark_precise_scalar_ids()
- * looks for all registers with frame2.r1.id in the current state
- */
-__msg("frame1: parent state regs=r6,r7 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame2: last_idx 9 first_idx 9 subseq_idx 13")
-__msg("frame2: regs=r1 stack= before 9: (85) call pc+3")
-/* frame1.r1 is marked because of backtracking of call instruction */
-__msg("frame1: parent state regs=r1,r6,r7 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame1: last_idx 8 first_idx 7 subseq_idx 9")
-__msg("frame1: regs=r1,r6,r7 stack= before 8: (bf) r7 = r1")
-__msg("frame1: regs=r1,r6 stack= before 7: (bf) r6 = r1")
-__msg("frame1: parent state regs=r1 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame1: last_idx 4 first_idx 4 subseq_idx 7")
-__msg("frame1: regs=r1 stack= before 4: (85) call pc+2")
-__msg("frame0: parent state regs=r1,r6 stack=")
-/* Parent state */
-__msg("frame0: last_idx 3 first_idx 1 subseq_idx 4")
-__msg("frame0: regs=r1,r6 stack= before 3: (bf) r6 = r0")
-__msg("frame0: regs=r0,r1 stack= before 2: (bf) r1 = r0")
-__msg("frame0: regs=r0 stack= before 1: (57) r0 &= 255")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("13: (25) if r1 > 0x7 goto pc+0        ; R1=" ALONE(7, 0x7))
+__msg("10: R0=0 R6=scalar(id={{.*}}umax32=255,{{.*}} R7=scalar(id={{.*}}umax32=255,")
+__msg("5: R0=0 R6=" ALONE(255, 0xff))
 __naked void precision_many_frames(void)
 {
 	asm volatile (
@@ -254,11 +193,6 @@ void precision_many_frames__bar(void)
 	"if r1 > 7 goto +0;"
 	"r6 = 0;" /* mark r6 as live */
 	"r7 = 0;" /* mark r7 as live */
-	/* force r1 to be precise, this eventually marks:
-	 * - bar frame r1
-	 * - foo frame r{1,6,7}
-	 * - main frame r{1,6}
-	 */
 	"r2 = r10;"
 	"r2 += r1;"
 	"r0 = 0;"
@@ -266,23 +200,13 @@ void precision_many_frames__bar(void)
 	::: __clobber_all);
 }
 
-/* Check that scalars with the same IDs are marked precise on stack as
+/*
+ * Check that range information is propagated to scalars with the same IDs on stack as
  * well as in registers.
  */
 SEC("socket")
 __success __log_level(2)
-__msg("12: (0f) r2 += r1")
-/* foo frame */
-__msg("frame1: regs=r1 stack= before 11: (bf) r2 = r10")
-__msg("frame1: regs=r1 stack= before 10: (25) if r1 > 0x7 goto pc+0")
-__msg("frame1: regs=r1 stack=-8,-16 before 9: (7b) *(u64 *)(r10 -16) = r1")
-__msg("frame1: regs=r1 stack=-8 before 8: (7b) *(u64 *)(r10 -8) = r1")
-__msg("frame1: regs=r1 stack= before 4: (85) call pc+3")
-/* main frame */
-__msg("frame0: regs=r1 stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r1")
-__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
-__msg("frame0: regs=r0 stack= before 1: (57) r0 &= 255")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("10: (25) if r1 > 0x7 goto pc+0        ; R1=scalar({{.*}}umax32=7,{{.*}} fp-8=scalar({{.*}}umax32=7,{{.*}} fp-16=scalar({{.*}}umax32=7,")
 __naked void precision_stack(void)
 {
 	asm volatile (
@@ -311,10 +235,6 @@ void precision_stack__foo(void)
 	"*(u64*)(r10 - 8) = r1;"
 	"*(u64*)(r10 - 16) = r1;"
 	"if r1 > 7 goto +0;"
-	/* force r1 to be precise, this eventually marks:
-	 * - foo frame r1,fp{-8,-16}
-	 * - main frame r1,fp{-8}
-	 */
 	"r2 = r10;"
 	"r2 += r1;"
 	"r0 = *(u64*)(r10 - 8);"
@@ -329,20 +249,9 @@ void precision_stack__foo(void)
 SEC("socket")
 __success __log_level(2)
 /* r{6,7} */
-__msg("12: (0f) r3 += r7")
-__msg("frame0: regs=r7 stack= before 11: (bf) r3 = r10")
-__msg("frame0: regs=r7 stack= before 9: (25) if r7 > 0x7 goto pc+0")
-/* ... skip some insns ... */
-__msg("frame0: regs=r6,r7 stack= before 3: (bf) r7 = r0")
-__msg("frame0: regs=r0,r6 stack= before 2: (bf) r6 = r0")
+__msg("9: (25) if r7 > 0x7 goto pc+0         ; R6=" LINKED("1", 7, 0x7) " R7=" LINKED("1", 7, 0x7) "{{$}}")
 /* r{8,9} */
-__msg("13: (0f) r3 += r9")
-__msg("frame0: regs=r9 stack= before 12: (0f) r3 += r7")
-/* ... skip some insns ... */
-__msg("frame0: regs=r9 stack= before 10: (25) if r9 > 0x7 goto pc+0")
-__msg("frame0: regs=r8,r9 stack= before 7: (bf) r9 = r0")
-__msg("frame0: regs=r0,r8 stack= before 6: (bf) r8 = r0")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("10: (25) if r9 > 0x7 goto pc+0        ; R8=" LINKED("2", 7, 0x7) " R9=" LINKED("2", 7, 0x7) "{{$}}")
 __naked void precision_two_ids(void)
 {
 	asm volatile (
@@ -360,13 +269,11 @@ __naked void precision_two_ids(void)
 	"r9 = r0;"
 	/* clear r0 id */
 	"r0 = 0;"
-	/* propagate equal scalars precision */
+	/* propagate range information to equal scalars */
 	"if r7 > 7 goto +0;"
 	"if r9 > 7 goto +0;"
 	"r3 = r10;"
-	/* force r7 to be precise, this also marks r6 */
 	"r3 += r7;"
-	/* force r9 to be precise, this also marks r8 */
 	"r3 += r9;"
 	"r6 = r6;" /* mark r6 as live */
 	"r8 = r8;" /* mark r8 as live */
@@ -378,7 +285,6 @@ __naked void precision_two_ids(void)
 
 SEC("socket")
 __success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 /*
  * check that r5 is unlinked after 'if', collect_linked_regs() can't tie
  * more than 5 registers for a single insn and the register compared by
@@ -386,9 +292,6 @@ __flag(BPF_F_TEST_STATE_FREQ)
  */
 __msg("7: (25) if r5 > 0x7 goto pc+0         ; R5=scalar(smin=")
 __msg("12: (bf) r5 = r5                      ; R5=scalar(id=2")
-/* check that r{0-4} are marked precise after 'if' */
-__msg("frame0: regs=r0 stack= before 7: (25) if r5 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3,r4 stack=:")
 __naked void linked_regs_too_many_regs(void)
 {
 	asm volatile (
@@ -410,9 +313,6 @@ __naked void linked_regs_too_many_regs(void)
 	"r4 = r4;"
 	/* make r5 appear in the log */
 	"r5 = r5;"
-	/* force r0 to be precise,
-	 * this would cause r{0-4} to be precise because of shared IDs
-	 */
 	"r7 = r10;"
 	"r7 += r0;"
 	"r0 = 0;"
@@ -424,10 +324,7 @@ __naked void linked_regs_too_many_regs(void)
 
 SEC("socket")
 __failure __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("regs=r7 stack= before 5: (3d) if r8 >= r0")
-__msg("parent state regs=r0,r7,r8")
-__msg("regs=r0,r7,r8 stack= before 4: (25) if r0 > 0x1")
+__msg("5: (3d) if r8 >= r0 goto pc+3         ; R0=scalar(umin=1) R7=scalar(id=1,umax=0xfffffffffffffffe) R8=scalar(id=1,umax=0xfffffffffffffffe)")
 __msg("div by zero")
 __naked void linked_regs_broken_link_2(void)
 {
@@ -438,13 +335,11 @@ __naked void linked_regs_broken_link_2(void)
 	"call %[bpf_get_prandom_u32];"
 	"if r0 > 1 goto +0;"
 	/* r7.id == r8.id,
-	 * thus r7 precision implies r8 precision,
-	 * which implies r0 precision because of the conditional below.
+	 * thus what is learned about r8 is learned about r7
 	 */
 	"if r8 >= r0 goto 1f;"
 	/* break id relation between r7 and r8 */
 	"r8 += r8;"
-	/* make r7 precise */
 	"if r7 == 0 goto 1f;"
 	"r0 /= 0;"
 "1:"
@@ -455,13 +350,11 @@ __naked void linked_regs_broken_link_2(void)
 	: __clobber_all);
 }
 
-/* Check that mark_chain_precision() for one of the conditional jump
- * operands does not trigger equal scalars precision propagation.
- */
+/* A conditional jump that is predicted says nothing new about its operands. */
 SEC("socket")
 __success __log_level(2)
-__msg("3: (25) if r1 > 0x100 goto pc+0")
-__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
+__msg("3: (25) if r1 > 0x100 goto pc+0       ; R1=" LINKED("1", 255, 0xff) "{{$}}")
+__msg("processed 6 insns")
 __naked void cjmp_no_linked_regs_trigger(void)
 {
 	asm volatile (
@@ -470,9 +363,7 @@ __naked void cjmp_no_linked_regs_trigger(void)
 	"r0 &= 0xff;"
 	/* tie r0.id == r1.id */
 	"r1 = r0;"
-	/* the jump below would be predicted, thus r1 would be marked precise,
-	 * this should not imply precision mark for r0
-	 */
+	/* the jump below is predicted */
 	"if r1 > 256 goto +0;"
 	"r0 = 0;"
 	"exit;"
@@ -497,7 +388,6 @@ __naked void cjmp_no_linked_regs_trigger(void)
  */
 SEC("socket")
 __failure __msg("register with unbounded min value")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void check_ids_in_regsafe(void)
 {
 	asm volatile (
@@ -539,12 +429,9 @@ __naked void check_ids_in_regsafe(void)
  *   (2) r6{.id=B}, r7{.id=A}, r8{.id=B}
  *
  * Where (2) is not safe, as "r7 > 4" check won't propagate range for it.
- * This example would be considered safe without changes to
- * mark_chain_precision() to track scalar values with equal IDs.
  */
 SEC("socket")
 __failure __msg("register with unbounded min value")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void check_ids_in_regsafe_2(void)
 {
 	asm volatile (
@@ -595,24 +482,24 @@ __naked void check_ids_in_regsafe_2(void)
  * If such IDs *are* generated the 'l1' below would be reached in
  * two states:
  *
- *   (1) r1{.id=A}, r2{.id=A}
- *   (2) r1{.id=C}, r2{.id=C}
+ *   (1) r3{.id=A}, r4{.id=A}, which is there first
+ *   (2) r3{.id=B}, r4{.id=C}
  *
- * Thus forcing 'if r1 == r2' verification twice.
+ * and (2) is not within (1).
  */
 SEC("socket")
 __success __log_level(2)
-__msg("14: (1d) if r3 == r4 goto pc+0")
-__msg("frame 0: propagating r3,r4")
-__msg("14: safe")
+__msg("from 14 to 15: within")
+__msg("15: (1d) if r3 == r4 goto pc+0        ; R3=0 R4=0")
 __msg("processed 18 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void no_scalar_id_for_const(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
 	/* unpredictable jump */
-	"if r0 > 7 goto l0_%=;"
+	"if r0 <= 7 goto l2_%=;"
+	"goto l0_%=;"
+"l2_%=:"
 	/* possibly generate same scalar ids for r3 and r4 */
 	"r1 = 0;"
 	"r1 ^= r1;" /* prevent bpf_prune_dead_branches from folding the branch */
@@ -629,7 +516,7 @@ __naked void no_scalar_id_for_const(void)
 	"r3 = r1;"
 	"r4 = r2;"
 "l1_%=:"
-	/* predictable jump, marks r3 and r4 precise */
+	/* predictable jump */
 	"if r3 == r4 goto +0;"
 	"r0 = 0;"
 	"exit;"
@@ -641,17 +528,17 @@ __naked void no_scalar_id_for_const(void)
 /* Same as no_scalar_id_for_const() but for 32-bit values */
 SEC("socket")
 __success __log_level(2)
-__msg("14: (1e) if w3 == w4 goto pc+0")
-__msg("frame 0: propagating r3,r4")
-__msg("14: safe")
+__msg("from 14 to 15: within")
+__msg("15: (1e) if w3 == w4 goto pc+0        ; R3=0 R4=0")
 __msg("processed 18 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void no_scalar_id_for_const32(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
 	/* unpredictable jump */
-	"if r0 > 7 goto l0_%=;"
+	"if r0 <= 7 goto l2_%=;"
+	"goto l0_%=;"
+"l2_%=:"
 	/* possibly generate same scalar ids for r3 and r4 */
 	"w1 = 0;"
 	"w1 ^= w1;" /* prevent bpf_prune_dead_branches from folding the branch */
@@ -668,7 +555,7 @@ __naked void no_scalar_id_for_const32(void)
 	"w3 = w1;"
 	"w4 = w2;"
 "l1_%=:"
-	/* predictable jump, marks r3 and r4 precise */
+	/* predictable jump */
 	"if w3 == w4 goto +0;"
 	"r0 = 0;"
 	"exit;"
@@ -677,20 +564,20 @@ __naked void no_scalar_id_for_const32(void)
 	: __clobber_all);
 }
 
-/* Check that unique scalar IDs are ignored when new verifier state is
- * compared to cached verifier state. For this test:
- * - cached state has no id on r1
- * - new state has a unique id on r1
+/*
+ * Check that unique scalar IDs are ignored when a verifier state is
+ * compared to the state that is there. For this test:
+ * - the state that is there had a unique id on r1
+ * - the state that arrives has no id on r1
  */
 SEC("socket")
 __success __log_level(2)
 __msg("6: (25) if r6 > 0x7 goto pc+1")
 __msg("7: (57) r1 &= 255")
-__msg("8: (bf) r2 = r10")
-__msg("from 6 to 8: safe")
-__msg("processed 12 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
-__naked void ignore_unique_scalar_ids_cur(void)
+__msg("from 7 to 8: within")
+__msg("8: R0=0 R1=" ALONE(255, 0xff))
+__msg("processed 11 insns")
+__naked void ignore_unique_scalar_ids_old(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
@@ -706,8 +593,8 @@ __naked void ignore_unique_scalar_ids_cur(void)
 	"r1 &= 0xff;"
 "l0_%=:"
 	/* get here in two states:
-	 * - first: r1 has no id (cached state)
-	 * - second: r1 has a unique id (should be considered equivalent)
+	 * - first: r1 has a unique id
+	 * - second: r1 has no id (should be considered to be within the first)
 	 */
 	"r2 = r10;"
 	"r2 += r1;"
@@ -717,20 +604,20 @@ __naked void ignore_unique_scalar_ids_cur(void)
 	: __clobber_all);
 }
 
-/* Check that unique scalar IDs are ignored when new verifier state is
- * compared to cached verifier state. For this test:
- * - cached state has a unique id on r1
- * - new state has no id on r1
+/*
+ * Check that unique scalar IDs are ignored when a verifier state is
+ * compared to the state that is there. For this test:
+ * - the state that is there has no id on r1
+ * - the state that arrives has a unique id on r1
  */
 SEC("socket")
 __success __log_level(2)
 __msg("6: (25) if r6 > 0x7 goto pc+1")
 __msg("7: (05) goto pc+1")
-__msg("9: (bf) r2 = r10")
-__msg("9: safe")
-__msg("processed 13 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
-__naked void ignore_unique_scalar_ids_old(void)
+__msg("from 7 to 9: within")
+__msg("9: R0=0 R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R10=fp0")
+__msg("processed 12 insns")
+__naked void ignore_unique_scalar_ids_cur(void)
 {
 	asm volatile (
 	"call %[bpf_ktime_get_ns];"
@@ -748,8 +635,8 @@ __naked void ignore_unique_scalar_ids_old(void)
 	"r1 &= 0xff;"
 "l0_%=:"
 	/* get here in two states:
-	 * - first: r1 has a unique id (cached state)
-	 * - second: r1 has no id (should be considered equivalent)
+	 * - first: r1 has no id
+	 * - second: r1 has a unique id (should be considered to be within the first)
 	 */
 	"r2 = r10;"
 	"r2 += r1;"
@@ -759,16 +646,14 @@ __naked void ignore_unique_scalar_ids_old(void)
 	: __clobber_all);
 }
 
-/* Check that two registers with 0 scalar IDs in a verified state can be mapped
- * to the same scalar ID in current state.
+/*
+ * Check that two registers with 0 scalar IDs in the state that is there can be mapped
+ * to the same scalar ID in the state that arrives.
  */
 SEC("socket")
 __success __log_level(2)
-/* The states should be equivalent on reaching insn 12.
- */
-__msg("12: safe")
-__msg("processed 17 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 10 to 11: within")
+__msg("processed 15 insns")
 __naked void two_nil_old_ids_one_cur_id(void)
 {
 	asm volatile (
@@ -783,16 +668,14 @@ __naked void two_nil_old_ids_one_cur_id(void)
 	"r7 *= 1;"
 	"r0 = 0;"
 	/* Maybe make r{6,7} IDs identical */
-	"if r6 > r7 goto l0_%=;"
-	"goto l1_%=;"
-"l0_%=:"
+	"if r6 <= r7 goto l1_%=;"
 	"r6 = r7;"
 "l1_%=:"
-	/* Mark r{6,7} precise.
+	/*
 	 * Get here in two states:
-	 * - first:  r6{.id=0}, r7{.id=0} (cached state)
+	 * - first:  r6{.id=0}, r7{.id=0}
 	 * - second: r6{.id=A}, r7{.id=A}
-	 * Verifier considers such states equivalent.
+	 * Verifier considers the second one to be within the first one.
 	 * Thus "exit;" would be verified only once.
 	 */
 	"r2 = r10;"
@@ -804,18 +687,14 @@ __naked void two_nil_old_ids_one_cur_id(void)
 	: __clobber_all);
 }
 
-/* Check that two different scalar IDs in a verified state can't be
- * mapped to the same scalar ID in current state.
+/*
+ * Check that two different scalar IDs in the state that is there can't be
+ * mapped to the same scalar ID in the state that arrives.
  */
 SEC("socket")
 __success __log_level(2)
-/* The exit instruction should be reachable from two states,
- * use two matches and "processed .. insns" to ensure this.
- */
-__msg("16: (95) exit")
-__msg("16: (95) exit")
-__msg("processed 22 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__not_msg("from 10 to 11: within")
+__msg("11: R0=0 R6=" ALONE(255, 0xff) " R7=" ALONE(255, 0xff))
 __naked void two_old_ids_one_cur_id(void)
 {
 	asm volatile (
@@ -830,24 +709,22 @@ __naked void two_old_ids_one_cur_id(void)
 	"r9 = r0;"
 	"r0 = 0;"
 	/* Maybe make r{6,7} IDs identical */
-	"if r6 > r7 goto l0_%=;"
-	"goto l1_%=;"
-"l0_%=:"
+	"if r6 <= r7 goto l1_%=;"
 	"r6 = r7;"
 "l1_%=:"
-	/* Mark r{6,7} precise.
+	/*
 	 * Get here in two states:
-	 * - first:  r6{.id=A}, r7{.id=B} (cached state)
-	 * - second: r6{.id=A}, r7{.id=A}
-	 * Currently we don't want to consider such states equivalent.
-	 * Thus "exit;" would be verified twice.
+	 * - first:  r6{.id=A}, r7{.id=B}
+	 * - second: r6{.id=B}, r7{.id=B}
+	 * Currently we don't want to consider the second one to be within
+	 * the first one. Thus they are joined.
 	 */
 	"r2 = r10;"
 	"r2 += r6;"
 	"r2 += r7;"
 	/*
 	 * keep r8 and r9 live, otherwise r6->id and r7->id
-	 * will become singular and reset to zero before if r6 > r7
+	 * will become singular and reset to zero before if r6 <= r7
 	 */
 	"r9 += r8;"
 	"exit;"
@@ -878,10 +755,6 @@ __log_level(2)
 __msg("4: (77) r1 >>= 32                     ; R1=0")
 __msg("5: (bf) r0 = r1                       ; R0=0 R1=0")
 __msg("6: (95) exit")
-__msg("from 3 to 4")
-__msg("4: (77) r1 >>= 32                     ; R1=0")
-__msg("5: (bf) r0 = r1                       ; R0=0 R1=0")
-__msg("6: (95) exit")
 /* Verify that statements to randomize upper half of r1 had not been
  * generated.
  */
diff --git a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
index f40e57251e94..9ff29a981a41 100644
--- a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
+++ b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
@@ -96,7 +96,7 @@ l1_%=:	exit;						\
 SEC("socket")
 __description("varlen_map_value_access pruning")
 __failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void varlen_map_value_access_pruning(void)
 {
@@ -110,8 +110,9 @@ __naked void varlen_map_value_access_pruning(void)
 	if r0 == 0 goto l0_%=;				\
 	r1 = *(u64*)(r0 + 0);				\
 	w2 = %[max_entries];				\
-	if r2 s> r1 goto l1_%=;				\
-	w1 = 0;						\
+	if r2 s<= r1 goto l3_%=;			\
+	goto l1_%=;					\
+l3_%=:	w1 = 0;						\
 l1_%=:	w1 <<= 2;					\
 	r0 += r1;					\
 	goto l2_%=;					\
@@ -193,7 +194,7 @@ l0_%=:	exit;						\
 
 SEC("tracepoint")
 __description("precision tracking for u32 spill/fill")
-__failure __msg("R0 min value is outside of the allowed memory range")
+__failure __msg("invalid access to map value, value_size=8 off=32 size=4")
 __naked void tracking_for_u32_spill_fill(void)
 {
 	asm volatile ("					\
@@ -202,7 +203,7 @@ __naked void tracking_for_u32_spill_fill(void)
 	w6 = 32;					\
 	if r0 == 0 goto l0_%=;				\
 	w6 = 4;						\
-l0_%=:	/* Additional insns to introduce a pruning point. */\
+l0_%=:	/* Additional insns to have one more place where paths meet. */\
 	call %[bpf_get_prandom_u32];			\
 	r3 = 0;						\
 	r3 = 0;						\
@@ -239,28 +240,21 @@ __naked void for_u32_spills_u64_fill(void)
 	call %[bpf_get_prandom_u32];			\
 	r6 = r0;					\
 	w7 = 0xffffffff;				\
-	/* Additional insns to introduce a pruning point. */\
-	r3 = 1;						\
-	r3 = 1;						\
-	r3 = 1;						\
-	r3 = 1;						\
+	r9 = 0xffffffff ll;				\
+	/* Additional insns to have a place where paths meet. */\
 	call %[bpf_get_prandom_u32];			\
 	if r0 == 0 goto l0_%=;				\
 	r3 = 1;						\
-l0_%=:	w3 /= 0;					\
+l0_%=:	r3 = 1;						\
 	/* u32 spills, u64 fill */			\
 	*(u32*)(r10 - 4) = r6;				\
 	*(u32*)(r10 - 8) = r7;				\
 	r8 = *(u64*)(r10 - 8);				\
 	/* if r8 != X goto pc+1  r8 known in fallthrough branch */\
-	if r8 != 0xffffffff goto l1_%=;			\
+	if r8 != r9 goto l1_%=;				\
 	r3 = 1;						\
-l1_%=:	/* if r8 == X goto pc+1  condition always true on first\
-	 * traversal, so starts backtracking to mark r8 as requiring\
-	 * precision. r7 marked as needing precision. r6 not marked\
-	 * since it's not tracked.			\
-	 */						\
-	if r8 == 0xffffffff goto l2_%=;			\
+l1_%=:	/* if r8 == X goto pc+1 */			\
+	if r8 == r9 goto l2_%=;				\
 	/* fails if r8 correctly marked unknown after fill. */\
 	w3 /= 0;					\
 l2_%=:	r0 = 0;						\
@@ -272,7 +266,8 @@ l2_%=:	r0 = 0;						\
 
 SEC("socket")
 __description("allocated_stack")
-__success __msg("processed 15 insns")
+/* every insn once: the state that wrote to the stack is within the state that did not */
+__success __msg("processed 14 insns")
 __success_unpriv __msg_unpriv("") __log_level(1) __retval(0)
 __naked void allocated_stack(void)
 {
@@ -337,12 +332,9 @@ l0_%=:	r1 = 42;					\
 	: __clobber_all);
 }
 
-/* Without checkpoint forcibly inserted at the back-edge a loop this
- * test would take a very long time to verify.
- */
+/* The number of trips is not known */
 SEC("kprobe")
-__failure __log_level(4)
-__msg("BPF program is too large.")
+__success
 __naked void short_loop1(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
index 4d1374885f6d..34c551f97e61 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
@@ -518,15 +518,6 @@ __msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0")
  */
 __msg("10: (71) r2 = *(u8 *)(r10 -9)         ; R2=0 R10=fp0 fp-16=0")
 __msg("11: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (71) r2 = *(u8 *)(r10 -1)")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 4: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (b7) r0 = 0")
 __naked void partial_stack_load_preserves_zeros(void)
 {
 	asm volatile (
@@ -600,8 +591,6 @@ __success
 __msg("2: (62) *(u32 *)(r10 -4) = 0          ; R10=fp0 fp-8=0000????")
 __msg("4: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0000????")
 __msg("5: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (71) r2 = *(u8 *)(r10 -1)")
 __naked void partial_stack_load_preserves_partial_zeros(void)
 {
 	asm volatile (
@@ -637,7 +626,6 @@ __naked void partial_stack_load_preserves_partial_zeros(void)
 SEC("raw_tp")
 __log_level(2)
 __success
-__msg("mark_precise: frame0: regs= stack=-8")
 __msg("R2=0")
 __naked void stack_load_preserves_mixed_zero_and_zero_spill(void)
 {
@@ -663,50 +651,21 @@ __naked void stack_load_preserves_mixed_zero_and_zero_spill(void)
 char two_byte_buf[2] SEC(".data.two_byte_buf");
 
 SEC("raw_tp")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2)
 __success
-/* make sure fp-8 is IMPRECISE fake register spill */
+/* make sure fp-8 is fake register spill */
 __msg("3: (7a) *(u64 *)(r10 -8) = 1          ; R10=fp0 fp-8=1")
-/* and fp-16 is spilled IMPRECISE const reg */
+/* and fp-16 is spilled const reg */
 __msg("5: (7b) *(u64 *)(r10 -16) = r0        ; R0=1 R10=fp0 fp-16=1")
 /* validate load from fp-8, which was initialized using BPF_ST_MEM */
 __msg("8: (79) r2 = *(u64 *)(r10 -8)         ; R2=1 R10=fp0 fp-8=1")
-__msg("9: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-/* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7a) *(u64 *)(r10 -8) = 1")
-__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("9: (0f) r1 += r2                      ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 /* validate load from fp-16, which was initialized using BPF_STX_MEM */
 __msg("12: (79) r2 = *(u64 *)(r10 -16)       ; R2=1 R10=fp0 fp-16=1")
-__msg("13: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (79) r2 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-/* now both fp-8 and fp-16 are precise, very good */
-__msg("mark_precise: frame0: parent state regs= stack=-16:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("13: (0f) r1 += r2                     ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 __naked void stack_load_preserves_const_precision(void)
 {
 	asm volatile (
-		/* establish checkpoint with state that has no stack slots;
-		 * if we bubble up to this state without finding desired stack
-		 * slot, then it's a bug and should be caught
-		 */
 		"goto +0;"
 
 		/* fp-8 is const 1 *fake* register */
@@ -716,7 +675,6 @@ __naked void stack_load_preserves_const_precision(void)
 		"r0 = 1;"
 		"*(u64 *)(r10 -16) = r0;"
 
-		/* force checkpoint to check precision marks preserved in parent states */
 		"goto +0;"
 
 		/* load single U64 from aligned FAKE_REG=1 slot */
@@ -740,48 +698,21 @@ __naked void stack_load_preserves_const_precision(void)
 }
 
 SEC("raw_tp")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2)
 __success
 /* make sure fp-8 is 32-bit FAKE subregister spill */
 __msg("3: (62) *(u32 *)(r10 -8) = 1          ; R10=fp0 fp-8=????1")
-/* but fp-16 is spilled IMPRECISE zero const reg */
+/* and fp-16 is spilled const reg */
 __msg("5: (63) *(u32 *)(r10 -16) = r0        ; R0=1 R10=fp0 fp-16=????1")
 /* validate load from fp-8, which was initialized using BPF_ST_MEM */
 __msg("8: (61) r2 = *(u32 *)(r10 -8)         ; R2=1 R10=fp0 fp-8=????1")
-__msg("9: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (62) *(u32 *)(r10 -8) = 1")
-__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("9: (0f) r1 += r2                      ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 /* validate load from fp-16, which was initialized using BPF_STX_MEM */
 __msg("12: (61) r2 = *(u32 *)(r10 -16)       ; R2=1 R10=fp0 fp-16=????1")
-__msg("13: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (61) r2 = *(u32 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-16:  R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("13: (0f) r1 += r2                     ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
 __naked void stack_load_preserves_const_precision_subreg(void)
 {
 	asm volatile (
-		/* establish checkpoint with state that has no stack slots;
-		 * if we bubble up to this state without finding desired stack
-		 * slot, then it's a bug and should be caught
-		 */
 		"goto +0;"
 
 		/* fp-8 is const 1 *fake* SUB-register */
@@ -791,7 +722,6 @@ __naked void stack_load_preserves_const_precision_subreg(void)
 		"r0 = 1;"
 		"*(u32 *)(r10 -16) = r0;"
 
-		/* force checkpoint to check precision marks preserved in parent states */
 		"goto +0;"
 
 		/* load single U32 from aligned FAKE_REG=1 slot */
@@ -1118,41 +1048,37 @@ l0_%=:	r1 >>= 32;					\
 	: __clobber_all);
 }
 
-/* stacksafe(): check if stack spill of an imprecise scalar in old state
- * is considered equivalent to STACK_{MISC,INVALID} in cur state.
+/*
+ * stacksafe(): the state with a stack spill of a scalar is there (label 1 is
+ * looked at first), the state with STACK_{MISC,INVALID} arrives.
+ * The first one is within the second one, which stands for both.
  */
 SEC("socket")
 __success __log_level(2)
+__msg("8: R0=scalar() R10=fp0 fp-8=??mm??mm")
 __msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-/* STACK_INVALID should prevent verifier in unpriv mode from
- * considering states equivalent and force an error on second
- * verification path (entry - label 1 - label 2).
- */
+__msg("processed 10 insns")
+/* STACK_INVALID is an error in unpriv mode */
 __failure_unpriv
 __msg_unpriv("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg_unpriv("9: (95) exit")
-__msg_unpriv("8: (79) r1 = *(u64 *)(r10 -8)")
 __msg_unpriv("invalid read from stack off -8+2 size 8")
-__flag(BPF_F_TEST_STATE_FREQ)
 __naked void old_imprecise_scalar_vs_cur_stack_misc(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
-	/* conjure scalar at fp-8 */
-	"r0 = 42;"
-	"*(u64*)(r10 - 8) = r0;"
-	"goto 2f;"
-"1:"
 	/* conjure STACK_{MISC,INVALID} at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u16*)(r10 - 8) = r0;"
 	"*(u16*)(r10 - 4) = r0;"
+	"goto 2f;"
+"1:"
+	/* conjure scalar at fp-8 */
+	"r0 = 42;"
+	"*(u64*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r1 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
@@ -1160,71 +1086,69 @@ __naked void old_imprecise_scalar_vs_cur_stack_misc(void)
 	: __clobber_all);
 }
 
-/* stacksafe(): check that stack spill of a precise scalar in old state
- * is not considered equivalent to STACK_MISC in cur state.
+/*
+ * stacksafe(): check that the state with STACK_MISC is not considered to be
+ * within the state with a stack spill of a known scalar, which is there.
  */
 SEC("socket")
 __success __log_level(2)
-/* verifier should visit 'if r1 == 0x2a ...' two times:
- * - once for path entry - label 2;
- * - once for path entry - label 1 - label 2.
- */
-__msg("if r1 == 0x2a goto pc+0")
-__msg("if r1 == 0x2a goto pc+0")
-__msg("processed 15 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* what is loaded is not known to be 42 */
+__msg("8: (79) r1 = *(u64 *)(r10 -8)         ; R1=scalar() R10=fp0 fp-8=mmmmmmmm")
+__msg("9: (15) if r1 == 0x2a goto pc+0       ; R1=scalar()")
+__msg("processed 12 insns")
 __naked void old_precise_scalar_vs_cur_stack_misc(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
-	/* conjure scalar at fp-8 */
-	"r0 = 42;"
-	"*(u64*)(r10 - 8) = r0;"
-	"goto 2f;"
-"1:"
 	/* conjure STACK_MISC at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u64*)(r10 - 8) = r0;"
 	"*(u32*)(r10 - 4) = r0;"
+	"goto 2f;"
+"1:"
+	/* conjure scalar at fp-8 */
+	"r0 = 42;"
+	"*(u64*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should not be considered safe on second visit */
 	"r1 = *(u64*)(r10 - 8);"
-	/* use r1 in precise context */
+	/* the value of r1 matters */
 	"if r1 == 42 goto +0;"
+	/* r0 is dead where the states meet: they differ in fp-8 only */
+	"r0 = 0;"
 	"exit;"
 	:
 	: __imm(bpf_ktime_get_ns)
 	: __clobber_all);
 }
 
-/* stacksafe(): check if STACK_MISC in old state is considered
- * equivalent to stack spill of a scalar in cur state.
+/*
+ * stacksafe(): check if the state with a stack spill of a scalar is considered
+ * to be within the state with STACK_MISC, which is there.
  */
 SEC("socket")
 __success  __log_level(2)
+__msg("from 4 to 8: within")
 __msg("8: (79) r0 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 10 insns")
 __naked void old_stack_misc_vs_cur_scalar(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
+	/* conjure scalar at fp-8 */
+	"r0 = 42;"
+	"*(u64*)(r10 - 8) = r0;"
+	"goto 2f;"
+"1:"
 	/* conjure STACK_{MISC,INVALID} at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u16*)(r10 - 8) = r0;"
 	"*(u16*)(r10 - 4) = r0;"
-	"goto 2f;"
-"1:"
-	/* conjure scalar at fp-8 */
-	"r0 = 42;"
-	"*(u64*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r0 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
@@ -1232,21 +1156,16 @@ __naked void old_stack_misc_vs_cur_scalar(void)
 	: __clobber_all);
 }
 
-/* stacksafe(): check that STACK_MISC in old state is not considered
- * equivalent to stack spill of a non-scalar in cur state.
+/*
+ * stacksafe(): the state with STACK_MISC is there, the state with a stack spill
+ * of a non-scalar arrives. A program that may read uninitialized stack may
+ * take a pointer for a number: the second state is within the first one.
  */
 SEC("socket")
 __success  __log_level(2)
-/* verifier should process exit instructions twice:
- * - once for path entry - label 2;
- * - once for path entry - label 1 - label 2.
- */
-__msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("9: (95) exit")
-__msg("from 2 to 7")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 4 to 8: within")
+__msg("8: (79) r1 = *(u64 *)(r10 -8)         ; R1=scalar()")
+__msg("processed 10 insns")
 __naked void old_stack_misc_vs_cur_ctx_ptr(void)
 {
 	asm volatile(
@@ -1255,16 +1174,16 @@ __naked void old_stack_misc_vs_cur_ctx_ptr(void)
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
+	/* conjure context pointer in fp-8 */
+	"*(u64*)(r10 - 8) = r9;"
+	"goto 2f;"
+"1:"
 	/* conjure STACK_MISC at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u64*)(r10 - 8) = r0;"
 	"*(u32*)(r10 - 4) = r0;"
-	"goto 2f;"
-"1:"
-	/* conjure context pointer in fp-8 */
-	"*(u64*)(r10 - 8) = r9;"
 "2:"
-	/* read fp-8, should not be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r1 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
@@ -1310,35 +1229,34 @@ __naked void stack_noperfmon_spill_32bit_onto_64bit_slot(void)
 }
 
 /*
- * stacksafe(): check if 32-bit scalar spill in old state is considered
- * equivalent to STACK_MISC in cur state.
+ * stacksafe(): the state with a 32-bit spill of an unknown scalar is there, the
+ * state with STACK_MISC arrives. The second one is within the first one.
  * 32-bit scalar spill creates slot[0-3] = STACK_MISC, slot[4-7] = STACK_SPILL.
  * Without 32-bit spill support in stacksafe(), the STACK_SPILL vs STACK_MISC
- * mismatch at slot[4] causes pruning to fail.
+ * mismatch at slot[4] says that it is not.
  */
 SEC("socket")
 __success __log_level(2)
+__msg("from 5 to 8: within")
 __msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 10 insns")
 __naked void old_imprecise_scalar32_vs_cur_stack_misc(void)
 {
 	asm volatile(
 	/* get a random value for branching */
 	"call %[bpf_ktime_get_ns];"
 	"if r0 == 0 goto 1f;"
-	/* conjure 32-bit scalar spill at fp-8 */
-	"r0 = 42;"
-	"*(u32*)(r10 - 8) = r0;"
-	"goto 2f;"
-"1:"
 	/* conjure STACK_MISC at fp-8 */
 	"call %[bpf_ktime_get_ns];"
 	"*(u16*)(r10 - 8) = r0;"
 	"*(u16*)(r10 - 6) = r0;"
+	"goto 2f;"
+"1:"
+	/* conjure 32-bit scalar spill at fp-8 */
+	"call %[bpf_ktime_get_ns];"
+	"*(u32*)(r10 - 8) = r0;"
 "2:"
-	/* read fp-8, should be considered safe on second visit */
+	/* read fp-8, it's looked at once */
 	"r1 = *(u64*)(r10 - 8);"
 	"exit;"
 	:
diff --git a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
index d9d7b05cf6d2..f4e7f47e4cdd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
@@ -469,9 +469,9 @@ l3_%=:	r1 = r7;					\
 
 SEC("cgroup/skb")
 __description("spin_lock: regsafe() check_ids() similar id mappings")
-__success __msg("29: safe")
+__success __msg("to 29: within")
 __failure_unpriv __msg_unpriv("")
-__log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2) __retval(0)
 __naked void check_ids_similar_id_mappings(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
index e6a9021b197b..e9194a560fbd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
@@ -587,14 +587,13 @@ __naked void stack_check_size_512_with_may_goto(void)
 #endif
 
 /*
- * Verify that old PTR_TO_STACK state is considered a super-set of
- * new PTR_TO_STACK state when the new offset is within the old range,
- * e.g. old [-72, -16] vs new {-16}.
+ * Verify that a PTR_TO_STACK state is considered a super-set of
+ * another PTR_TO_STACK state when the offset of that is within its range,
+ * e.g. [-72, -16] vs {-16}.
  */
 SEC("socket")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 /*
  * r7 is widened to [-72, -16] at the loop header (insn 5),
  * the loop body sees [-64, -8] after 'r7 += 8'.
@@ -604,9 +603,9 @@ __msg("R7=fp(smin=smin32=-64,smax=smax32=-8")
 /* 'r7 = r8' sets r7 to fp-16; its ANY SCEV is not clamped. */
 __msg("R7=fp-16 R8=fp-16 loop_stack=5")
 __not_msg("clamping r7")
-/* back-edge state is pruned because {-16} is within [-72, -16]. */
-__msg("from 8 to 5: safe")
-__msg("processed 12 insns")
+/* back-edge state goes no further because {-16} is within [-72, -16]. */
+__msg("from 8 to 5: within")
+__msg("processed 11 insns")
 __naked void stack_ptr_subrange_in_loop(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
index dc0c7034c04f..041b7a71fb3f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
@@ -39,31 +39,21 @@ static unsigned long callback_subprog()
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("7: (0f) r1 += r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r0                      ; R0=12 R1=map_value(map=.data.vals,ks=4,vs=16,imm=12)")
 __naked int subprog_result_precise(void)
 {
 	asm volatile (
 		"r6 = 3;"
-		/* pass r6 through r1 into subprog to get it back as r0;
-		 * this whole chain will have to be marked as precise later
-		 */
+		/* pass r6 through r1 into subprog to get it back as r0 */
 		"r1 = r6;"
 		"call identity_subprog;"
 		/* now use subprog's returned value (which is a
-		 * r6 -> r1 -> r0 chain), as index into vals array, forcing
-		 * all of that to be known precisely
+		 * r6 -> r1 -> r0 chain), as index into vals array
 		 */
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* here r0->r1->r6 chain is forced to be precise and has to be
-		 * propagated back to the beginning, including through the
+		/*
+		 * here r0 has to be known: the constant went through the
 		 * subprog call
 		 */
 		"r1 += r0;"
@@ -98,26 +88,18 @@ static unsigned long sneaky_fp_leaking_subprog()
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
-__msg("mark_precise: frame0: regs=r0 stack= before 10: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 9: (bf) r0 = (s8)r10")
-__msg("7: R0=scalar")
+__msg("3: R0=scalar() R6=")
+__msg("6: (0f) r1 += r0                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int fp_precise_subprog_result(void)
 {
 	asm volatile (
 		"call fp_leaking_subprog;"
 		/* use subprog's returned value (which is derived from r10=fp
-		 * register), as index into vals array, forcing all of that to
-		 * be known precisely
+		 * register, so it's unknown), as index into vals array
 		 */
 		"r0 &= 3;"
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* force precision marking */
 		"r1 += r0;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -129,31 +111,18 @@ __naked int fp_precise_subprog_result(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = (s8)r1")
-/* here r1 is marked precise, even though it's fp register, but that's fine
- * because by the time we get out of subprogram it has to be derived from r10
- * anyways, at which point we'll break precision chain
- */
-__msg("mark_precise: frame1: regs=r1 stack= before 9: (bf) r1 = r10")
-__msg("7: R0=scalar")
+__msg("3: R0=scalar() R6=")
+__msg("6: (0f) r1 += r0                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int sneaky_fp_precise_subprog_result(void)
 {
 	asm volatile (
 		"call sneaky_fp_leaking_subprog;"
 		/* use subprog's returned value (which is derived from r10=fp
-		 * register), as index into vals array, forcing all of that to
-		 * be known precisely
+		 * register, so it's unknown), as index into vals array
 		 */
 		"r0 &= 3;"
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* force precision marking */
 		"r1 += r0;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -165,35 +134,25 @@ __naked int sneaky_fp_precise_subprog_result(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("9: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0")
-__msg("mark_precise: frame0: regs=r0 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 7: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (a5) if r0 < 0x4 goto pc+1")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (85) call pc+7")
+__msg("9: (0f) r1 += r0                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int global_subprog_result_precise(void)
 {
 	asm volatile (
 		"r6 = 3;"
 		/* pass r6 through r1 into subprog to get it back as r0;
-		 * given global_identity_subprog is global, precision won't
-		 * propagate all the way back to r6
+		 * given global_identity_subprog is global, nothing is known
+		 * about what it returns
 		 */
 		"r1 = r6;"
 		"call global_identity_subprog;"
 		/* now use subprog's returned value (which is unknown now, so
-		 * we need to clamp it), as index into vals array, forcing r0
-		 * to be marked precise (with no effect on r6, though)
+		 * we need to clamp it), as index into vals array
 		 */
 		"if r0 < %[vals_arr_sz] goto 1f;"
 		"r0 = %[vals_arr_sz] - 1;"
 	"1:"
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* here r0 is forced to be precise and has to be
-		 * propagated back to the global subprog call, but it
-		 * shouldn't go all the way to mark r6 as precise
-		 */
 		"r1 += r0;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -213,11 +172,9 @@ static unsigned long loop_callback_bad()
 		"if r0 s> 1000 goto 1f;"
 		"r0 = 0;"
 	"1:"
-		"goto +0;" /* checkpoint */
+		"goto +0;"
 		/* bpf_loop() expects [0, 1] values, so branch above skipping
-		 * r0 = 0; should lead to a failure, but if exit instruction
-		 * doesn't enforce r0's precision, this callback will be
-		 * successfully verified
+		 * r0 = 0; should lead to a failure
 		 */
 		"exit;"
 		:
@@ -228,14 +185,8 @@ static unsigned long loop_callback_bad()
 
 SEC("?raw_tp")
 __failure __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame1: regs=r0 stack= before 11: (b7) r0 = 0")
-/* check that we have branch code path doing its own validation */
-__msg("from 10 to 12: frame1: R0=scalar(smin=umin=1001")
-/* check that branch code path marks r0 as precise, before failing */
-__msg("mark_precise: frame1: regs=r0 stack= before 9: (85) call bpf_get_prandom_u32#7")
-__msg("At callback return the register R0 has smin=1001 should have been in [0, 1]")
+/* 0 of the fallthrough code path doesn't stand for what the branch code path has */
+__msg("At callback return the register R0 has {{.*}} should have been in [0, 1]")
 __naked int callback_precise_return_fail(void)
 {
 	asm volatile (
@@ -256,41 +207,15 @@ __naked int callback_precise_return_fail(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-/* First simulated path does not include callback body,
- * r1 and r4 are always precise for bpf_loop() calls.
- */
 __msg("9: (85) call bpf_loop#181")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: parent state regs=r4 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r4 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: parent state regs=r1 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r1 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs=r1 stack= before 7: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs=r1 stack= before 6: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs=r1 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* r6 precision propagation */
-__msg("14: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 14 first_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 12: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (25) if r6 > 0x3 goto pc+4")
-__msg("mark_precise: frame0: regs=r0,r6 stack= before 10: (bf) r6 = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 9: (85) call bpf_loop")
-/* State entering callback body popped from states stack */
-__msg("from 9 to 17: frame1:")
 __msg("17: frame1: R10=fp0 cb")
 __msg("17: (b7) r0 = 0")
 __msg("18: (95) exit")
 __msg("returning from callee:")
 __msg("to caller at 9:")
-__msg("frame 0: propagating r1,r3,r4")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit")
-__msg("from 18 to 9: safe")
+/* the callback changes nothing for the caller */
+__msg("from 18 to 9: within")
+__msg("14: (0f) r1 += r6                     ; R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R6=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
 __naked int callback_result_precise(void)
 {
 	asm volatile (
@@ -309,9 +234,7 @@ __naked int callback_result_precise(void)
 		"if r6 > 3 goto 1f;"
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the bpf_loop() call, but not beyond
-		 */
+		/* here the range of r6 has to be known */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 	"1:"
@@ -326,15 +249,7 @@ __naked int callback_result_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("7: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 7 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 10: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs= stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_callee_saved_reg_precise(void)
 {
 	asm volatile (
@@ -348,9 +263,7 @@ __naked int parent_callee_saved_reg_precise(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -362,13 +275,7 @@ __naked int parent_callee_saved_reg_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("7: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 7 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_callee_saved_reg_precise_global(void)
 {
 	asm volatile (
@@ -382,9 +289,7 @@ __naked int parent_callee_saved_reg_precise_global(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -396,36 +301,14 @@ __naked int parent_callee_saved_reg_precise_global(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-/* First simulated path does not include callback body */
-__msg("12: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 12 first_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 10: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 9: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs=r6 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (b7) r1 = 1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* State entering callback body popped from states stack */
-__msg("from 9 to 15: frame1:")
+__msg("12: (0f) r1 += r6                     ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __msg("15: frame1: R10=fp0 cb")
 __msg("15: (b7) r0 = 0")
 __msg("16: (95) exit")
 __msg("returning from callee:")
 __msg("to caller at 9:")
-/* r1, r4 are always precise for bpf_loop(),
- * r6 was marked before backtracking to callback body.
- */
-__msg("frame 0: propagating r1,r3,r4,r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0")
-__msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs= stack=:")
-__msg("from 16 to 9: safe")
+/* r6 is what it was before the callback */
+__msg("from 16 to 9: within")
 __naked int parent_callee_saved_reg_precise_with_callback(void)
 {
 	asm volatile (
@@ -442,9 +325,7 @@ __naked int parent_callee_saved_reg_precise_with_callback(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) callback call
-		 */
+		/* here r6 has to be known: the callback doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -458,19 +339,7 @@ __naked int parent_callee_saved_reg_precise_with_callback(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("9: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 6")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8:")
-__msg("mark_precise: frame0: last_idx 13 first_idx 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 13: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 12: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs= stack= before 5: (85) call pc+6")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("9: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_stack_slot_precise(void)
 {
 	asm volatile (
@@ -492,9 +361,7 @@ __naked int parent_stack_slot_precise(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -506,15 +373,7 @@ __naked int parent_stack_slot_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("9: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (85) call pc+6")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("9: (0f) r1 += r6                      ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __naked int parent_stack_slot_precise_global(void)
 {
 	asm volatile (
@@ -536,9 +395,7 @@ __naked int parent_stack_slot_precise_global(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the subprog call doesn't change it */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -550,38 +407,14 @@ __naked int parent_stack_slot_precise_global(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-/* First simulated path does not include callback body */
-__msg("14: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 14 first_idx 10")
-__msg("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 12: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 10: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs= stack=-8:")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0 subseq_idx 10")
-__msg("mark_precise: frame0: regs= stack=-8 before 9: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 8: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* State entering callback body popped from states stack */
-__msg("from 10 to 17: frame1:")
+__msg("14: (0f) r1 += r6                     ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
 __msg("17: frame1: R10=fp0 cb")
 __msg("17: (b7) r0 = 0")
 __msg("18: (95) exit")
 __msg("returning from callee:")
 __msg("to caller at 10:")
-/* r1, r4 are always precise for bpf_loop(),
- * fp-8 was marked before backtracking to callback body.
- */
-__msg("frame 0: propagating r1,r3,r4,fp-8")
-__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0")
-__msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181")
-__msg("mark_precise: frame0: parent state regs= stack=:")
-__msg("from 18 to 10: safe")
+/* fp-8 is what it was before the callback */
+__msg("from 18 to 10: within")
 __naked int parent_stack_slot_precise_with_callback(void)
 {
 	asm volatile (
@@ -604,9 +437,7 @@ __naked int parent_stack_slot_precise_with_callback(void)
 
 		"r6 *= 4;"
 		"r1 = %[vals];"
-		/* here r6 is forced to be precise and has to be propagated
-		 * back to the beginning, handling (and ignoring) subprog call
-		 */
+		/* here r6 has to be known: the callback doesn't change fp-8 */
 		"r1 += r6;"
 		"r0 = *(u32 *)(r1 + 0);"
 		"exit;"
@@ -621,27 +452,18 @@ __naked int parent_stack_slot_precise_with_callback(void)
 __noinline __used
 static __u64 subprog_with_precise_arg(__u64 x)
 {
-	return vals[x]; /* x is forced to be precise */
+	return vals[x]; /* x has to be known */
 }
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("8: (0f) r2 += r1")
-__msg("mark_precise: frame1: last_idx 8 first_idx 0")
-__msg("mark_precise: frame1: regs=r1 stack= before 6: (18) r2 = ")
-__msg("mark_precise: frame1: regs=r1 stack= before 5: (67) r1 <<= 2")
-__msg("mark_precise: frame1: regs=r1 stack= before 2: (85) call pc+2")
-__msg("mark_precise: frame0: regs=r1 stack= before 1: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 0: (b7) r6 = 3")
+__msg("8: (0f) r2 += r1                      ; R1=12 R2=map_value(map=.data.vals,ks=4,vs=16,imm=12)")
 __naked int subprog_arg_precise(void)
 {
 	asm volatile (
 		"r6 = 3;"
 		"r1 = r6;"
-		/* subprog_with_precise_arg expects its argument to be
-		 * precise, so r1->r6 will be marked precise from inside the
-		 * subprog
-		 */
+		/* subprog_with_precise_arg needs to know its argument */
 		"call subprog_with_precise_arg;"
 		"r0 += r6;"
 		"exit;"
@@ -666,7 +488,7 @@ static __u64 subprog_spill_reg_precise(void)
 		 * verifier log output)
 		 */
 		"*(u64 *)(r10 - 16) = r2;"
-		/* use both spills as return result to propagete precision everywhere */
+		/* use both spills as return result */
 		"r0 = *(u64 *)(r10 - 16);"
 		"r2 = *(u64 *)(r1 + 0);"
 		"r0 += r2;"
@@ -676,32 +498,14 @@ static __u64 subprog_spill_reg_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("10: (0f) r1 += r7")
-__msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8")
-__msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4")
-__msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
-__msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2")
-__msg("mark_precise: frame1: regs= stack= before 16: (79) r2 = *(u64 *)(r1 +0)")
-__msg("mark_precise: frame1: regs= stack= before 15: (79) r0 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame1: regs= stack= before 14: (7b) *(u64 *)(r10 -16) = r2")
-__msg("mark_precise: frame1: regs= stack= before 13: (7b) *(u64 *)(r1 +0) = r2")
-__msg("mark_precise: frame1: regs=r2 stack= before 6: (85) call pc+6")
-__msg("mark_precise: frame0: regs=r2 stack= before 5: (bf) r2 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 1")
+__msg("10: (0f) r1 += r7                     ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=4) R7=4")
 __naked int subprog_spill_into_parent_stack_slot_precise(void)
 {
 	asm volatile (
 		"r6 = 1;"
 
 		/* pass pointer to stack slot and r6 to subprog;
-		 * r6 will be marked precise and spilled into fp-8 slot, which
-		 * also should be marked precise
+		 * r6 will be spilled into fp-8 slot
 		 */
 		"r1 = r10;"
 		"r1 += -8;"
@@ -716,8 +520,9 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
 
 		"r7 *= 4;"
 		"r1 = %[vals];"
-		/* here r7 is forced to be precise and has to be propagated
-		 * back to the beginning, handling subprog call and logic
+		/*
+		 * here r7 has to be known: it's what the subprog spilled into
+		 * the stack slot of its caller
 		 */
 		"r1 += r7;"
 		"r0 = *(u32 *)(r1 + 0);"
@@ -730,32 +535,12 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("17: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 17 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 16: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 15: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 14: (79) r0 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 13: (7b) *(u64 *)(r7 -8) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (79) r0 = *(u64 *)(r8 +16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (7b) *(u64 *)(r8 +16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 10: (79) r0 = *(u64 *)(r7 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 8: (07) r8 += -32")
-__msg("mark_precise: frame0: regs=r0 stack= before 7: (bf) r8 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 6: (07) r7 += -8")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r7 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 21: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 20: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (85) call pc+15")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 1")
+__msg("17: (0f) r1 += r0                     ; R0=4 R1=map_value(map=.data.vals,ks=4,vs=16,imm=4)")
 __naked int stack_slot_aliases_precision(void)
 {
 	asm volatile (
 		"r6 = 1;"
-		/* pass r6 through r1 into subprog to get it back as r0;
-		 * this whole chain will have to be marked as precise later
-		 */
+		/* pass r6 through r1 into subprog to get it back as r0 */
 		"r1 = r6;"
 		"call identity_subprog;"
 		/* let's setup two registers that are aliased to r10 */
@@ -777,11 +562,11 @@ __naked int stack_slot_aliases_precision(void)
 		"r0 = *(u64 *)(r8 + 16);"
 		"*(u64 *)(r7 - 8) = r0;"
 		"r0 = *(u64 *)(r10 - 16);"
-		/* get ready to use r0 as an index into array to force precision */
+		/* get ready to use r0 as an index into array */
 		"r0 *= 4;"
 		"r1 = %[vals];"
-		/* here r0->r1->r6 chain is forced to be precise and has to be
-		 * propagated back to the beginning, including through the
+		/*
+		 * here r0 has to be known: the constant went through the
 		 * subprog call and all the stack spills and loads
 		 */
 		"r1 += r0;"
@@ -789,7 +574,7 @@ __naked int stack_slot_aliases_precision(void)
 		"exit;"
 		:
 		: __imm_ptr(vals)
-		: __clobber_common, "r6"
+		: __clobber_common, "r6", "r7", "r8"
 	);
 }
 
@@ -820,15 +605,11 @@ static unsigned long identity_tail_call(void)
 SEC("?raw_tp")
 __failure __log_level(2)
 __msg("13: (85) call bpf_tail_call#12")
-__msg("mark_precise: frame1: last_idx 13 first_idx 0 subseq_idx -1 ")
 __msg("returning from callee:")
 __msg("frame1: R0=scalar() R6=3 R10=fp0")
 __msg("to caller at 4:")
 __msg("R0=scalar() R6=map_value(map=.data.vals,ks=4,vs=16) R10=fp0")
 __msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: parent state regs=r0 stack=:  R0=Pscalar() R6=map_value(map=.data.vals,ks=4,vs=16) R10=fp0")
 __msg("math between map_value pointer and register with unbounded min value is not allowed")
 __naked int subprog_result_tail_call(void)
 {
@@ -860,36 +641,22 @@ static int ld_abs_subprog(void)
 }
 
 /*
- * Buggy verifier did not properly backtrack early subprogram exit
- * modelled for BPF_LD | BPF_ABS instruction, causing a segfault.
+ * BPF_LD | BPF_ABS instruction is modelled as an early exit of the subprogram.
+ * r7 is known on both paths.
  */
 SEC("socket")
 __success
 __log_level(2)
-/* early exit path */
-__msg("3: (0f) r1 += r7")
-__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r7 stack= before 9: (20) r0 = *(u32 *)skb[0]")
-__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1")
-__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1")
-__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8")
-/* fallthrough path */
-__msg("3: (0f) r1 += r7")
-__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r7 stack= before 10: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 9: (20) r0 = *(u32 *)skb[0]")
-__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1")
-__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1")
-__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8")
+/* the early exit and the exit of the subprogram give the same state */
+__msg("from 10 to 2: within")
+__msg("3: (0f) r1 += r7                      ; R1=fp-8 R7=-8")
 __naked int ld_abs_backtrack_both_paths(void)
 {
 	asm volatile (
 		"r7 = -8;"
 		"call ld_abs_subprog;"
 		"r1 = r10;"
-		"r1 += r7;" /* mark r7 as precise */
+		"r1 += r7;"
 		"*(u64 *)(r1 + 0) = 0;"
 		"r0 = 0;"
 		"exit;"
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_or_null.c b/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
index 8ff668a242eb..2a5bdac24925 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
@@ -183,7 +183,7 @@ l1_%=:	exit;						\
 SEC("socket")
 __description("invalid map access from else condition")
 __failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
 __flag(BPF_F_ANY_ALIGNMENT)
 __naked void map_access_from_else_condition(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
index 454b9a92d5c1..dbf6da950324 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
@@ -225,8 +225,7 @@ l2_%=:	r0 = 1;						\
 
 SEC("socket")
 __description("map access: known scalar += value_ptr unknown vs unknown (lt)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
 __retval(1)
 __naked void ptr_unknown_vs_unknown_lt(void)
 {
@@ -270,8 +269,7 @@ l2_%=:	r0 = 1;						\
 
 SEC("socket")
 __description("map access: known scalar += value_ptr unknown vs unknown (gt)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
 __retval(1)
 __naked void ptr_unknown_vs_unknown_gt(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index 399884911ea5..d92215c5b923 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -63,7 +63,6 @@ SEC("cgroup/skb")
 __description("variable-offset stack read preserves spilled zero")
 __success
 __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8")
 __msg("R3=0")
 __retval(0)
 __naked void stack_read_var_off_preserves_spilled_zero(void)
@@ -89,7 +88,6 @@ SEC("cgroup/skb")
 __description("variable-offset stack read preserves spilled zero across slots")
 __success
 __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8,-16")
 __msg("R3=0")
 __retval(0)
 __naked void stack_read_var_off_preserves_spilled_zero_across_slots(void)
@@ -116,7 +114,6 @@ SEC("cgroup/skb")
 __description("variable-offset stack read preserves partial spilled zero")
 __success
 __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8")
 __msg("R3=0")
 __retval(0)
 __naked void stack_read_var_off_preserves_partial_spilled_zero(void)
diff --git a/tools/testing/selftests/bpf/verifier/precise.c b/tools/testing/selftests/bpf/verifier/precise.c
index a9242103dc47..de9d35ed8a4b 100644
--- a/tools/testing/selftests/bpf/verifier/precise.c
+++ b/tools/testing/selftests/bpf/verifier/precise.c
@@ -36,26 +36,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
 	.fixup_map_array_48b = { 1 },
-	.result = VERBOSE_ACCEPT,
-	.errstr =
-	"mark_precise: frame0: last_idx 26 first_idx 20\
-	mark_precise: frame0: regs=r2 stack= before 25\
-	mark_precise: frame0: regs=r2 stack= before 24\
-	mark_precise: frame0: regs=r2 stack= before 23\
-	mark_precise: frame0: regs=r2 stack= before 22\
-	mark_precise: frame0: regs=r2 stack= before 20\
-	mark_precise: frame0: parent state regs=r2 stack=:\
-	mark_precise: frame0: last_idx 19 first_idx 10\
-	mark_precise: frame0: regs=r2 stack= before 19\
-	mark_precise: frame0: regs=r9 stack= before 18\
-	mark_precise: frame0: regs=r8,r9 stack= before 17\
-	mark_precise: frame0: regs=r0,r9 stack= before 15\
-	mark_precise: frame0: regs=r0,r9 stack= before 14\
-	mark_precise: frame0: regs=r9 stack= before 13\
-	mark_precise: frame0: regs=r9 stack= before 12\
-	mark_precise: frame0: regs=r9 stack= before 11\
-	mark_precise: frame0: regs=r9 stack= before 10\
-	mark_precise: frame0: parent state regs= stack=:",
+	.result = ACCEPT,
 },
 {
 	"precise: test 2",
@@ -95,24 +76,8 @@
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
 	.fixup_map_array_48b = { 1 },
-	.result = VERBOSE_ACCEPT,
+	.result = ACCEPT,
 	.flags = BPF_F_TEST_STATE_FREQ,
-	.errstr =
-	"26: (85) call bpf_probe_read_kernel#113\
-	mark_precise: frame0: last_idx 26 first_idx 22\
-	mark_precise: frame0: regs=r2 stack= before 25\
-	mark_precise: frame0: regs=r2 stack= before 24\
-	mark_precise: frame0: regs=r2 stack= before 23\
-	mark_precise: frame0: regs=r2 stack= before 22\
-	mark_precise: frame0: parent state regs=r2 stack=:\
-	mark_precise: frame0: last_idx 20 first_idx 20\
-	mark_precise: frame0: regs=r2 stack= before 20\
-	mark_precise: frame0: parent state regs=r2 stack=:\
-	mark_precise: frame0: last_idx 19 first_idx 17\
-	mark_precise: frame0: regs=r2 stack= before 19\
-	mark_precise: frame0: regs=r9 stack= before 18\
-	mark_precise: frame0: regs=r8,r9 stack= before 17\
-	mark_precise: frame0: parent state regs= stack=:",
 },
 {
 	"precise: cross frame pruning",
@@ -153,17 +118,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_XDP,
 	.flags = BPF_F_TEST_STATE_FREQ,
-	.errstr = "mark_precise: frame0: last_idx 5 first_idx 5\
-	mark_precise: frame0: parent state regs=r4 stack=:\
-	mark_precise: frame0: last_idx 4 first_idx 2\
-	mark_precise: frame0: regs=r4 stack= before 4\
-	mark_precise: frame0: regs=r4 stack= before 3\
-	mark_precise: frame0: last_idx 5 first_idx 5\
-	mark_precise: frame0: parent state regs=r0 stack=:\
-	mark_precise: frame0: last_idx 4 first_idx 2\
-	mark_precise: frame0: regs=r0 stack= before 4\
-	5: R0=-1 R4=0",
-	.result = VERBOSE_ACCEPT,
+	.result = ACCEPT,
 	.retval = -1,
 },
 {
@@ -182,23 +137,7 @@
 	},
 	.prog_type = BPF_PROG_TYPE_XDP,
 	.flags = BPF_F_TEST_STATE_FREQ,
-	.errstr = "mark_precise: frame0: last_idx 7 first_idx 7\
-	mark_precise: frame0: parent state regs=r4 stack=:\
-	mark_precise: frame0: last_idx 6 first_idx 4\
-	mark_precise: frame0: regs=r4 stack= before 6: (b7) r0 = -1\
-	mark_precise: frame0: regs=r4 stack= before 5: (79) r4 = *(u64 *)(r10 -8)\
-	mark_precise: frame0: regs= stack=-8 before 4: (7b) *(u64 *)(r3 -8) = r0\
-	mark_precise: frame0: parent state regs=r0 stack=:\
-	mark_precise: frame0: last_idx 3 first_idx 3\
-	mark_precise: frame0: regs=r0 stack= before 3: (55) if r3 != 0x7b goto pc+0\
-	mark_precise: frame0: regs=r0 stack= before 2: (bf) r3 = r10\
-	mark_precise: frame0: regs=r0 stack= before 1: (57) r0 &= 255\
-	mark_precise: frame0: parent state regs=r0 stack=:\
-	mark_precise: frame0: last_idx 0 first_idx 0\
-	mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7\
-	mark_precise: frame0: last_idx 7 first_idx 7\
-	mark_precise: frame0: parent state regs= stack=:",
-	.result = VERBOSE_ACCEPT,
+	.result = ACCEPT,
 	.retval = -1,
 },
 {
-- 
2.55.0

Re: [PATCH bpf-next v4 38/46] selftests/bpf: Adjust tests to log lines that are gone

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 6003 bytes --]

> selftests/bpf: Adjust tests to log lines that are gone

TL;DR:
- Test weakening: several selftests lose log checks without the promised
  replacement, so some only check the verdict
- Changelog omits five changed files and still says BPF_F_TEST_STATE_FREQ was
  removed from changed tests
- Possible duplicate verifier/precise.c test

>     selftests/bpf: Adjust tests to log lines that are gone
>
>     Without this patch 114 subtests of test_progs and 4 tests of test_verifier
>     fail. Their progs get the old verdict, but the tests match lines of the
>     verifier log that the kernel does not print any more. Keep the progs. Match
>     the line of the new log that shows the same fact.
>
>       [P] "bpf: Remove precision tracking"
>       [V] "bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth"
>       [F] "bpf: Verify programs as a fixpoint over the control flow graph"
>
>       the tests matched        they match now
>       -----------------------  ------------------------------------------------
>       [P] "mark_precise: ..",  The value of the scalar where the prog uses it:
>       462 lines                "R2=-8".
>       [V] "N: safe", 23 lines  "from M to N: within". If no state covers the
>                                other one: "joined", and the join.
>       [F] "id=1" of a scalar,  No id: the verifier clears an id that only one
>       12 lines                 live register has.
>
>     The first row is verifier_precision.c, _subprog_precision.c, _spill_fill.c,
>     _scalar_ids.c, stack_arg_precision.c and verifier/precise.c. The second row
>     is verifier_bounds_step.c, _search_pruning.c, _live_stack.c: the tests of
>     regsafe(), stacksafe() and range_within(). The target of a jump is verified
>     before the fall-through. Thus the tests in which the first of two states
>     must cover the second one swap their branches.
>
>     Three verdicts change. short_loop1 was "too large". Now one state at the
>     loop head covers all iterations, and it loads. 2 unprivileged progs of
>     verifier_value_ptr_arith.c load: r1 = [0, 3] and r1 = [0, 7] are joined in
>     front of 'r1 += r0', and the insn gets one limit for its mask. 8 progs are
>     rejected as before, with a different message. BPF_F_TEST_STATE_FREQ has no
>     effect. Remove it from the tests that change.
>
>     Signed-off-by: Alexei Starovoitov <ast@kernel.org>

The changelog does not seem to match the diff in two places.

First, the list of files for the "mark_precise" row:

> The first row is verifier_precision.c, _subprog_precision.c, _spill_fill.c,
> _scalar_ids.c, stack_arg_precision.c and verifier/precise.c.

This leaves out five files whose mark_precise checks the patch also
deletes: aggregate_ret_func.c (2 tests), aggregate_ret_kfunc.c,
test_global_func15.c, verifier_array_access.c (3 tests) and
verifier_var_off.c (3 tests).

Second, the changelog says:

> Match the line of the new log that shows the same fact.

That is done in stack_arg_precision.c and verifier_subprog_precision.c.  In
aggregate_ret_func.c, aggregate_ret_kfunc.c, test_global_func15.c,
verifier_array_access.c and all four changed tests of verifier/precise.c,
the log expectations (and __log_level(2) / VERBOSE_ACCEPT) are just deleted
and nothing replaces them.  The "precise: cross frame pruning" errstr even
ended with the value line "5: R0=-1 R4=0", and that line was dropped too.

In aggregate_ret_static_precise() the remaining verdict does not depend on
the fact the test was written for.  The callee sets "r2 = 4", but the
caller does:

    r6 = r2; r6 &= 7; r1 = r10; r1 += -8; r1 += r6;

followed by a 1-byte store.  That load succeeds privileged for any readable
R2, so the test no longer shows that the second half of the aggregate
return is known in the caller.  Would a line such as the R6=4 state show it?

The comments next to these tests also still describe removed machinery:
"/* ptr += scalar marks r6 (hence R2) precise */" in aggregate_ret_func.c
and aggregate_ret_kfunc.c, "its value is made precise below" in
aggregate_ret_func.c, and the "checkpoint", "precision tracking for R0" and
"is_state_equal() logic" comments in test_global_func15.c.  This patch
rewrote the matching comments in stack_arg_precision.c and
verifier_subprog_precision.c.

Could the changelog list every file and say which tests now check only the
verdict, or could those tests get the replacement log line the message
promises?

> diff --git a/tools/testing/selftests/bpf/verifier/precise.c b/tools/testing/selftests/bpf/verifier/precise.c
> index a9242103dc47..de9d35ed8a4b 100644
> --- a/tools/testing/selftests/bpf/verifier/precise.c
> +++ b/tools/testing/selftests/bpf/verifier/precise.c

[ ... ]

> @@ -95,24 +76,8 @@
>  	},
>  	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
>  	.fixup_map_array_48b = { 1 },
> -	.result = VERBOSE_ACCEPT,
> +	.result = ACCEPT,
>  	.flags = BPF_F_TEST_STATE_FREQ,
> -	.errstr =
> -	"26: (85) call bpf_probe_read_kernel#113\

[ ... ]

The changelog says "BPF_F_TEST_STATE_FREQ has no effect. Remove it from the
tests that change."  Three of the four verifier/precise.c tests changed here
keep the flag: "precise: test 2" (above), "precise: ST zero to stack insn
is supported" and "precise: STX insn causing spi > allocated_stack".

Also, "precise: test 1" and "precise: test 2" used to differ only in their
errstr.  Now both have the same .insns, .prog_type, .fixup_map_array_48b
and .result = ACCEPT, and differ only by a flag that does nothing.  Does
test_verifier now run the same case twice under two names?

Should the flag be dropped from the changed tests as the message says, and
"precise: test 2" merged with or removed in favor of "precise: test 1", or
should the commit message be reworded?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

[PATCH bpf-next v4 39/46] selftests/bpf: Adjust tests to new verdicts, messages and order

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Without this patch 143 subtests of test_progs and 3 tests of test_verifier
fail. No prog that loaded is rejected. 25 progs that were rejected load.

  [F] "bpf: Verify programs as a fixpoint over the control flow graph"
  [O] "bpf: Verify static subprogs on their own"
  [R] "bpf: Allow recursive subprogs"

  before                          now
  ------------------------------  -----------------------------------------
  "infinite loop detected",       [F] 19 progs load, most run and return 0:
  "too large", "8193 jumps"       8 iter_remake_*, loop3.bpf.o, .. One
                                  state covers all iterations, and may_goto
                                  stops the loop.
  the same, the loop has no exit  [F] "loop at insn N never exits": 6 progs
  "recursive call from",          [R] 6 progs load: callx_*_recursion,
  "frames is too deep"            bounded_recursion, .. It needs the JIT:
                                  __arch_x86_64, __arch_arm64.
  "header_count is [0..3]", then  [F] "[0..5]": two bounds are joined in
  "[0..5]"                        front of a counted loop.
  the fall-through is verified    flow_compute_order(): the target of
  first                           a jump is. Swap the branches, or expect
                                  the other one of two errors.
  "0: R1=ctx() R10=fp0"           [F] "0: R10=fp0": only what is live
  scc column, "depth=",           gone, "(limit 0)": the patches that
  "(limit 1000000)"               remove the old pass
  r1 is an argument               [O] only if the callee reads it

The last row: the callees in verifier_live_stack.c and _liveness_exp.c read
the pointer that they get. unsafe_on_2nd_iter also wrote 8 bytes to
'char buf[4]', and the verifier finds that error first: the buffer gets
8 bytes. bpf_loop_iter_limit_nested compares two sums with their maximum:
the verifier knows their range, not the set of their values.

New tests of [R] stand next to the old tests of the same limits. What it
rejects: test_global_func3.c, verifier_tailcall.c,
"calls: a timer callback calls the main prog", "arena stack: the kernel
frame of a recursion counts in the depth of the stack". What runs:
"arena stack: recursion ends in the stack", and
callx_no_recursion_after_all, which gets no check at run time.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../bpf/prog_tests/bpf_verif_scale.c          |   4 +-
 .../selftests/bpf/prog_tests/log_buf.c        |   4 +-
 .../selftests/bpf/prog_tests/log_fixup.c      |   6 +-
 .../selftests/bpf/prog_tests/reg_bounds.c     |   5 +-
 ...f_qdisc_fail__invalid_dynptr_cross_frame.c |   4 +-
 .../selftests/bpf/progs/cpumask_failure.c     |   2 +-
 tools/testing/selftests/bpf/progs/iters.c     | 114 ++++++--------
 .../selftests/bpf/progs/iters_state_safety.c  |   8 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |  12 +-
 .../bpf/progs/mem_rdonly_untrusted.c          |  16 +-
 .../selftests/bpf/progs/test_global_func3.c   | 125 ++++++++++++++++
 .../selftests/bpf/progs/timer_failure.c       |   9 +-
 .../selftests/bpf/progs/verifier_align.c      |  10 +-
 .../selftests/bpf/progs/verifier_and.c        |   4 +-
 .../bpf/progs/verifier_arena_stack.c          |  82 +++++++++++
 .../selftests/bpf/progs/verifier_bounds.c     |   6 +-
 .../selftests/bpf/progs/verifier_callx.c      |  71 +++++++--
 .../bpf/progs/verifier_callx_rodata.c         |   4 +-
 .../selftests/bpf/progs/verifier_cfg.c        |   8 +-
 .../bpf/progs/verifier_global_subprogs.c      |  11 +-
 .../bpf/progs/verifier_helper_value_access.c  |   8 +-
 .../selftests/bpf/progs/verifier_idoms.c      |   9 +-
 .../bpf/progs/verifier_iterating_callbacks.c  |  17 ++-
 .../selftests/bpf/progs/verifier_live_stack.c |   3 +-
 .../bpf/progs/verifier_liveness_exp.c         |   7 +-
 .../bpf/progs/verifier_loop_hierarchy.c       | 125 ++++++++--------
 .../selftests/bpf/progs/verifier_loop_latch.c |   2 +-
 .../selftests/bpf/progs/verifier_loops1.c     |  16 +-
 .../selftests/bpf/progs/verifier_movsx.c      |   3 +-
 .../selftests/bpf/progs/verifier_scev.c       | 139 ++++++++----------
 .../bpf/progs/verifier_subprog_insn_stats.c   |  16 +-
 .../selftests/bpf/progs/verifier_tailcall.c   |  27 ++++
 .../selftests/bpf/progs/verifier_uninit.c     |   2 +-
 .../selftests/bpf/progs/verifier_unpriv.c     |  23 +--
 .../bpf/progs/verifier_unpriv_perf.c          |   4 +-
 tools/testing/selftests/bpf/verifier/calls.c  |  46 +++++-
 36 files changed, 615 insertions(+), 337 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
index 902a70d5afeb..2dc37090d7a4 100644
--- a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
@@ -154,9 +154,9 @@ void test_verif_scale_loop2()
 	scale_test("loop2.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
 }
 
-void test_verif_scale_loop3_fail()
+void test_verif_scale_loop3()
 {
-	scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */);
+	scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
 }
 
 void test_verif_scale_loop4()
diff --git a/tools/testing/selftests/bpf/prog_tests/log_buf.c b/tools/testing/selftests/bpf/prog_tests/log_buf.c
index d6f14a232002..4f821fdc4a02 100644
--- a/tools/testing/selftests/bpf/prog_tests/log_buf.c
+++ b/tools/testing/selftests/bpf/prog_tests/log_buf.c
@@ -125,7 +125,7 @@ static void obj_load_log_buf(void)
 	ASSERT_OK_PTR(strstr(libbpf_log_buf, "libbpf: prog 'bad_prog': -- BEGIN PROG LOAD LOG --"),
 		      "libbpf_log_correct");
 	ASSERT_STREQ(obj_log_buf, "", "obj_log__empty");
-	ASSERT_STREQ(good_log_buf, "processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n",
+	ASSERT_STREQ(good_log_buf, "processed 4 insns (limit 0) max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n",
 		     "good_log_ok");
 	ASSERT_STREQ(bad_log_buf, "", "bad_log_empty");
 
@@ -179,7 +179,7 @@ static void bpf_prog_load_log_buf(void)
 	opts.log_level = 2;
 	fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "good_prog", "GPL",
 			   good_prog_insns, good_prog_insn_cnt, &opts);
-	ASSERT_OK_PTR(strstr(log_buf, "0: R1=ctx() R10=fp0"), "good_log_2");
+	ASSERT_OK_PTR(strstr(log_buf, "0: R10=fp0"), "good_log_2");
 	ASSERT_GE(fd, 0, "good_fd2");
 	if (fd >= 0)
 		close(fd);
diff --git a/tools/testing/selftests/bpf/prog_tests/log_fixup.c b/tools/testing/selftests/bpf/prog_tests/log_fixup.c
index 90a98e23be61..89a7f5947081 100644
--- a/tools/testing/selftests/bpf/prog_tests/log_fixup.c
+++ b/tools/testing/selftests/bpf/prog_tests/log_fixup.c
@@ -41,7 +41,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
 				  "struct task_struct___bad.fake_field (0:1 @ offset 4)\n",
 				  "log_buf_part2");
 		ASSERT_HAS_SUBSTR(log_buf,
-				  "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n",
+				  "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n",
 				  "log_buf_end");
 		break;
 	case TRUNC_PARTIAL:
@@ -50,7 +50,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
 				  "struct task_struct___bad.fake_field (0:1 @ offset 4)\n",
 				  "log_buf_part2");
 		/* we shouldn't get full end of BPF verifier log */
-		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n"),
+		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n"),
 			    "log_buf_end");
 		break;
 	case TRUNC_FULL:
@@ -58,7 +58,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
 		ASSERT_NULL(strstr(log_buf, "struct task_struct___bad.fake_field (0:1 @ offset 4)\n"),
 			    "log_buf_part2");
 		/* we shouldn't get full end of BPF verifier log */
-		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n"),
+		ASSERT_NULL(strstr(log_buf, "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n"),
 			    "log_buf_end");
 		break;
 	}
diff --git a/tools/testing/selftests/bpf/prog_tests/reg_bounds.c b/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
index 6231258b26cc..09f7930733be 100644
--- a/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
+++ b/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
@@ -1276,7 +1276,7 @@ static int parse_range_cmp_log(const char *log_buf, struct case_spec spec,
 		{true_pos + 1,  7, "R7=", true2_reg},
 	};
 	char buf[32];
-	const char *p = log_buf, *q;
+	const char *p, *q;
 	int i, err;
 
 	for (i = 0; i < 4; i++) {
@@ -1291,8 +1291,9 @@ static int parse_range_cmp_log(const char *log_buf, struct case_spec spec,
 		 * Different verifier passes may print
 		 *   18: (bf) r0 = r6
 		 * as well, but never followed by ';'.
+		 * The branches are in the log in the order the verifier looked at them.
 		 */
-		q = p;
+		q = log_buf;
 		while ((q = strstr(q, buf)) != NULL) {
 			const char *s = q + strlen(buf);
 
diff --git a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
index 2e23b8593af9..1d1684d2cf3b 100644
--- a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
+++ b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
@@ -26,8 +26,10 @@ int BPF_PROG(invalid_dynptr_cross_frame, struct sk_buff *skb, struct Qdisc *sch,
 	bpf_dynptr_from_skb((struct __sk_buff *)skb, 0, &ptr);
 
 	hdr = bpf_dynptr_slice(&ptr, 0, NULL, sizeof(*hdr));
-	if (!hdr)
+	if (!hdr) {
+		bpf_kfree_skb(skb);
 		return NET_XMIT_DROP;
+	}
 
 	free_skb(skb);
 
diff --git a/tools/testing/selftests/bpf/progs/cpumask_failure.c b/tools/testing/selftests/bpf/progs/cpumask_failure.c
index c89c88db39d1..715218d32105 100644
--- a/tools/testing/selftests/bpf/progs/cpumask_failure.c
+++ b/tools/testing/selftests/bpf/progs/cpumask_failure.c
@@ -45,7 +45,7 @@ int BPF_PROG(test_alloc_no_release, struct task_struct *task, u64 clone_flags)
 }
 
 SEC("tp_btf/task_newtask")
-__failure __msg("NULL pointer passed to trusted R1")
+__failure __msg("R1 type=scalar expected=ptr_, trusted_ptr_, rcu_ptr_")
 int BPF_PROG(test_alloc_double_release, struct task_struct *task, u64 clone_flags)
 {
 	struct bpf_cpumask *cpumask;
diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c
index ae845ad747bf..b9677d30e692 100644
--- a/tools/testing/selftests/bpf/progs/iters.c
+++ b/tools/testing/selftests/bpf/progs/iters.c
@@ -47,7 +47,7 @@ int iter_err_unsafe_c_loop(const void *ctx)
 }
 
 SEC("?raw_tp")
-__failure __msg("unbounded memory access")
+__failure __msg("math between map_value pointer and register with unbounded min value is not allowed")
 int iter_err_unsafe_asm_loop(const void *ctx)
 {
 	struct bpf_iter_num it;
@@ -878,19 +878,15 @@ __naked int delayed_read_mark(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
+__msg("21: (79) r8 = *(u64 *)(r0 +0)")
+__msg("misaligned stack access off -33+0 size 8")
 __naked int delayed_precision_mark(void)
 {
 	/* This is equivalent to C program below.
-	 * The test is similar to delayed_iter_mark but verifies that incomplete
-	 * precision don't fool verifier.
-	 * The call to bpf_iter_num_next() is reachable with r7 values -16 and -32.
-	 * State with r7=-16 is visited first and follows r6 != 42 ... continue branch.
-	 * At this point iterator next() call is reached with r7 that has no read
-	 * and precision marks.
-	 * Loop body with r7=-32 would only be visited if verifier would decide to continue
-	 * with second loop iteration. Absence of precision mark on r7 might affect state
-	 * equivalent logic used for iterator convergence tracking.
+	 * The test is similar to delayed_read_mark.
+	 * The call to bpf_iter_num_next() is reachable with r7 values -16 and -33.
+	 * The loop body with r7=-33 is only looked at if the verifier goes on
+	 * with the second iteration of the loop.
 	 *
 	 * r8 = 0
 	 * fp[-16] = 0
@@ -899,7 +895,7 @@ __naked int delayed_precision_mark(void)
 	 * bpf_iter_num_new(&fp[-8], 0, 10)
 	 * while (bpf_iter_num_next(&fp[-8])) {
 	 *   if (r6 != 42) {
-	 *     r7 = -32
+	 *     r7 = -33
 	 *     r6 = bpf_get_prandom_u32()
 	 *     continue;
 	 *   }
@@ -957,22 +953,17 @@ __naked int delayed_precision_mark(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("misaligned stack access off -25+0 size 8")
 __naked int loop_state_deps1(void)
 {
 	/* This is equivalent to C program below.
 	 *
 	 * The case turns out to be tricky in a sense that:
-	 * - states with c=-25 are explored only on a second iteration
-	 *   of the outer loop;
-	 * - states with read+precise mark on c are explored only on
-	 *   second iteration of the inner loop and in a state which
-	 *   is pushed to states stack first.
+	 * - c is -25 only on a second iteration of the outer loop;
+	 * - c is used only on a second iteration of the inner loop.
 	 *
-	 * Depending on the details of iterator convergence logic
-	 * verifier might stop states traversal too early and miss
-	 * unsafe c=-25 memory access.
+	 * The verifier might stop too early and miss unsafe c=-25
+	 * memory access.
 	 *
 	 *   j = iter_new();		 // fp[-16]
 	 *   a = 0;			 // r6
@@ -1082,23 +1073,18 @@ __naked int loop_state_deps1(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("34: (0f) r0 += r8")
+__msg("misaligned stack access off -25+0 size 8")
 __naked int loop_state_deps2(void)
 {
 	/* This is equivalent to C program below.
 	 *
 	 * The case turns out to be tricky in a sense that:
-	 * - states with read+precise mark on c are explored only on a second
-	 *   iteration of the first inner loop and in a state which is pushed to
-	 *   states stack first.
-	 * - states with c=-25 are explored only on a second iteration of the
-	 *   second inner loop and in a state which is pushed to states stack
-	 *   first.
+	 * - c is used only on a second iteration of the first inner loop;
+	 * - c is -25 only on a second iteration of the second inner loop.
 	 *
-	 * Depending on the details of iterator convergence logic
-	 * verifier might stop states traversal too early and miss
-	 * unsafe c=-25 memory access.
+	 * The verifier might stop too early and miss unsafe c=-25
+	 * memory access.
 	 *
 	 *   j = iter_new();             // fp[-16]
 	 *   a = 0;                      // r6
@@ -1259,49 +1245,38 @@ __naked int loop_state_deps2(void)
 
 SEC("?raw_tp")
 __failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("misaligned stack access off -25+0 size 8")
 __naked int loop_state_deps3(void)
 {
 	/* This is equivalent to a C program below.
 	 *
-	 *   if (random() != 24) {       // assume false branch is placed first
+	 *   if (random() != 24) {
 	 *     i = iter_new();           // fp[-8]
 	 *     while (iter_next(i));
 	 *     iter_destroy(i);
 	 *     return;
 	 *   }
 	 *
-	 *   for (i = 10; i > 0; i--);   // increase dfs_depth for child states
+	 *   for (i = 10; i > 0; i--);
 	 *
 	 *   i = iter_new();             // fp[-8]
 	 *   b = -24;                    // r8
-	 *   for (;;) {                  // checkpoint (L)
-	 *     if (iter_next(i))         // checkpoint (N)
+	 *   for (;;) {
+	 *     if (iter_next(i))
 	 *       break;
-	 *     if (random() == 77) {     // assume false branch is placed first
+	 *     if (random() == 77) {
 	 *       *(u64 *)(r10 + b) = 7;  // this is not safe when b == -25
 	 *       iter_destroy(i);
 	 *       return;
 	 *     }
-	 *     if (random() == 42) {     // assume false branch is placed first
+	 *     if (random() == 42) {
 	 *       b = -25;
 	 *     }
 	 *   }
 	 *   iter_destroy(i);
 	 *
-	 * In case of a buggy verifier first loop might poison
-	 * env->cur_state->loop_entry with a state having 0 branches
-	 * and small dfs_depth. This would trigger NOT_EXACT states
-	 * comparison for some states within second loop.
-	 * Specifically, checkpoint (L) might be problematic if:
-	 * - branch with '*(u64 *)(r10 + b) = 7' is not explored yet;
-	 * - checkpoint (L) is first reached in state {b=-24};
-	 * - traversal is pruned at checkpoint (N) setting checkpoint's (L)
-	 *   branch count to 0, thus making it eligible for use in pruning;
-	 * - checkpoint (L) is next reached in state {b=-25},
-	 *   this would cause NOT_EXACT comparison with a state {b=-24}
-	 *   while 'b' is not marked precise yet.
+	 * What the verifier learned in the first two loops must not make it
+	 * take {b=-25} at the head of the third one for {b=-24}.
 	 */
 	asm volatile (
 		"call %[bpf_get_prandom_u32];"
@@ -1884,7 +1859,6 @@ __naked int absent_mark_in_the_middle_state2(void)
 }
 
 SEC("?raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure __msg("misaligned stack access off -31+0 size 8")
 __naked int absent_mark_in_the_middle_state3(void)
 {
@@ -2014,8 +1988,8 @@ static int loop1_wrapper(void)
  * but adapted for use with bpf_loop().
  */
 SEC("raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
-__failure __msg("math between fp pointer and register with unbounded min value is not allowed")
+/* -32 or -31 */
+__failure __msg("misaligned stack access off (0xffffffffffffffe0; 0x1)+0 size 8")
 __naked void absent_mark_in_the_middle_state4(void)
 {
 	/*
@@ -2159,8 +2133,7 @@ static int loop_cb5(int i, __u64 *ctx)
 }
 
 SEC("raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
-__failure __msg("memory access is {{.*}} and is outside of the object of size 64")
+__failure __msg("invalid access to map value, value_size=64 off=402000 size=4")
 __naked void loop_counter_precision_2nd_iter(void)
 {
 	asm volatile (
@@ -2174,9 +2147,8 @@ __naked void loop_counter_precision_2nd_iter(void)
 		"r3 += -8;"
 		"r4 = 0;"
 		/*
-		 * Explore with nr_loops=1 on a first path and nr_loops=2 on a second path.
-		 * Buggy verifier did not propagate r1 precision properly,
-		 * and thus checkpoints created for nr_loops=1 case matched nr_loops=2 case.
+		 * nr_loops is 1 on one path and 2 on the other.
+		 * The second call of the callback must not be lost.
 		 */
 		"call %[bpf_loop];"
 		"r0 = 0;"
@@ -2190,12 +2162,12 @@ __naked void loop_counter_precision_2nd_iter(void)
 
 /*
  * An iterator that is destroyed and created again inside its loop starts from
- * the beginning and the loop never ends. In the first four progs the first
- * prune point after bpf_iter_num_new() is the call of bpf_iter_num_next().
+ * the beginning and the loop never ends. bpf_iter_num_next() is not what stops such
+ * loop: the verifier adds may_goto. The progs spin until it's out of budget.
  */
 SEC("socket")
 __description("iter: remake in the loop, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_jmp(void)
 {
 	asm volatile (
@@ -2256,7 +2228,7 @@ __used __naked static void iter_remake(void)
 
 SEC("socket")
 __description("iter: remake in a callee, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_callee_jmp(void)
 {
 	asm volatile (
@@ -2295,7 +2267,7 @@ __naked void iter_remake_callee_jmp(void)
 
 SEC("socket")
 __description("iter: remake after next, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_after_next_jmp(void)
 {
 	asm volatile (
@@ -2336,7 +2308,7 @@ __naked void iter_remake_after_next_jmp(void)
 
 SEC("socket")
 __description("iter: two iterators, remake each after its next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __naked void iter_remake_two_jmp(void)
 {
 	asm volatile (
@@ -2398,7 +2370,7 @@ __naked void iter_remake_two_jmp(void)
 /* Same loops with a prune point in front of bpf_iter_num_next() */
 SEC("socket")
 __description("iter: remake in the loop")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_inline(void)
 {
@@ -2439,7 +2411,7 @@ __naked void iter_remake_inline(void)
 
 SEC("socket")
 __description("iter: remake in a callee")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_callee(void)
 {
@@ -2475,7 +2447,7 @@ __naked void iter_remake_callee(void)
 
 SEC("socket")
 __description("iter: remake after next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_after_next(void)
 {
@@ -2515,7 +2487,7 @@ __naked void iter_remake_after_next(void)
 
 SEC("socket")
 __description("iter: remake when drained")
-__failure __msg("infinite loop detected")
+__success __retval(0)
 __flag(BPF_F_TEST_STATE_FREQ)
 __naked void iter_remake_drained(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/iters_state_safety.c b/tools/testing/selftests/bpf/progs/iters_state_safety.c
index b2ba4609a31e..fa9604f0a173 100644
--- a/tools/testing/selftests/bpf/progs/iters_state_safety.c
+++ b/tools/testing/selftests/bpf/progs/iters_state_safety.c
@@ -30,7 +30,7 @@ int force_clang_to_emit_btf_for_externs(void *ctx)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("fp-8=iter_num(id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(id=1,state=active)")
 int create_and_destroy(void *ctx)
 {
 	struct bpf_iter_num iter;
@@ -207,8 +207,8 @@ static __noinline void subprog_with_iter(void)
 SEC("?raw_tp")
 __failure
 /* ensure there was a call to subprog, which might happen without __noinline */
-__msg("returning from callee:")
-__msg("Unreleased reference id=1")
+__msg("0: (85) call pc+2")
+__msg("Unreleased reference id=2 alloc_insn=7")
 int leak_iter_from_subprog_fail(void *ctx)
 {
 	subprog_with_iter();
@@ -218,7 +218,7 @@ int leak_iter_from_subprog_fail(void *ctx)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("fp-8=iter_num(id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(id=1,state=active)")
 int valid_stack_reuse(void *ctx)
 {
 	struct bpf_iter_num iter;
diff --git a/tools/testing/selftests/bpf/progs/iters_testmod_seq.c b/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
index d00888f6687a..88ce4b7a9c33 100644
--- a/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
+++ b/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
@@ -20,8 +20,8 @@ __s64 res_empty;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_empty(const void *ctx)
 {
@@ -38,8 +38,8 @@ __s64 res_full;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_full(const void *ctx)
 {
@@ -58,8 +58,8 @@ static volatile int zero = 0;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_truncated(const void *ctx)
 {
diff --git a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
index b5cf85e2a19a..c6c1c3767a75 100644
--- a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
+++ b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
@@ -219,9 +219,9 @@ int mixed_mem_untrusted_btf_id_type(void *ctx)
 	 * returning 0.
 	 */
 	asm volatile (
-	"r7 = %[p];"
-	"if %[zero] != 0 goto +1;"
 	"r7 = %[q];"
+	"if %[zero] == 0 goto +1;"
+	"r7 = %[p];"
 	"r8 = *(u64 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
@@ -263,9 +263,9 @@ int mixed_mem_btf_id_type(void *ctx)
 	 * gotten on its own.
 	 */
 	asm volatile (
-	"r7 = %[p];"
-	"if %[zero] != 0 goto +1;"
 	"r7 = %[q];"
+	"if %[zero] == 0 goto +1;"
+	"r7 = %[p];"
 	"r8 = *(u32 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
@@ -305,9 +305,9 @@ int mixed_rdonly_mem_btf_id_type(void *ctx)
 	 * only its PTR_UNTRUSTED variant is.
 	 */
 	asm volatile (
-	"r7 = %[p];"
-	"if %[zero] != 0 goto +1;"
 	"r7 = %[q];"
+	"if %[zero] == 0 goto +1;"
+	"r7 = %[p];"
 	"r8 = *(u32 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
@@ -340,9 +340,9 @@ int mixed_mem_mem_type(void *ctx)
 	 * hence without the BPF_PROBE_MEM rewrite the other path needs.
 	 */
 	asm volatile (
-	"r7 = %[q];"
-	"if %[zero] == 0 goto +1;"
 	"r7 = %[p];"
+	"if %[zero] != 0 goto +1;"
+	"r7 = %[q];"
 	"r8 = *(u64 *)(r7 + 0);"
 	"%[v] = r8;"
 	: [v]"=r"(v)
diff --git a/tools/testing/selftests/bpf/progs/test_global_func3.c b/tools/testing/selftests/bpf/progs/test_global_func3.c
index b332704b395c..b916355c5e59 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func3.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func3.c
@@ -205,6 +205,131 @@ int global_func3_32_frames(struct __sk_buff *skb)
 	return f31(skb);
 }
 
+volatile int again;
+
+static __attribute__ ((noinline))
+int calls_itself(struct __sk_buff *skb)
+{
+	if (again)
+		return (calls_itself(skb) >> 1) ^ 1;
+	return f16(skb);
+}
+
+#define FRAME(name, next)			\
+static __attribute__ ((noinline))		\
+int name(struct __sk_buff *skb)			\
+{						\
+	return next(skb);			\
+}
+
+FRAME(g1, calls_itself)
+FRAME(g2, g1)
+FRAME(g3, g2)
+FRAME(g4, g3)
+FRAME(g5, g4)
+FRAME(g6, g5)
+FRAME(g7, g6)
+FRAME(g8, g7)
+FRAME(g9, g8)
+FRAME(g10, g9)
+FRAME(g11, g10)
+FRAME(g12, g11)
+FRAME(g13, g12)
+FRAME(g14, g13)
+FRAME(g15, g14)
+FRAME(g16, g15)
+
+/*
+ * 16 frames above the subprog that calls itself and 16 below it. The call of it
+ * with nothing above it comes first.
+ */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__failure __msg("the call stack of {{[0-9]+}} frames is too deep")
+int global_func3_frames_around_recursion(struct __sk_buff *skb)
+{
+	return calls_itself(skb) + g16(skb);
+}
+
+static __attribute__ ((noinline))
+int depth(int n)
+{
+	return n > 0 ? (depth(n - 1) + 1) & 255 : 0;
+}
+
+SEC("kprobe")
+__arch_x86_64
+__arch_arm64
+__failure __msg("recursive depth() in a tracing program")
+int global_func3_recursion_in_tracing_prog(void *ctx)
+{
+	return depth(again + 7);
+}
+
+__attribute__ ((noinline))
+int redirects(struct __sk_buff *skb, int n)
+{
+	int ret = n > 0 ? redirects(skb, n - 1) : 0;
+
+	return ret + bpf_clone_redirect(skb, 1, 0);
+}
+
+/* The prog would run again on top of the frames of its recursion */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__failure __msg("recursive redirects() in a program that calls bpf_clone_redirect()")
+int global_func3_recursion_redirects(struct __sk_buff *skb)
+{
+	return redirects(skb, again + 7);
+}
+
+static __attribute__ ((noinline))
+int depth_to(int n, long *out)
+{
+	*out = depth(n);
+	return 0;
+}
+
+/* The pointer to the stack is left over in r2 at the call of depth(), which doesn't read r2 */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+int global_func3_recursion_stale_stack_pointer(struct __sk_buff *skb)
+{
+	long x = 0;
+
+	depth_to(again + 7, &x);
+	return x;
+}
+
+/* It writes to the stack of its caller, or a compiler hands down the value and no pointer */
+static __attribute__ ((noinline))
+long hands_down_its_stack(long *up, int n)
+{
+	long mine = n;
+
+	*up += 1;
+	if (n <= 0)
+		return mine;
+	return hands_down_its_stack(&mine, n - 1) + mine;
+}
+
+/* What a subprog does with the stack of its callers is not found for chains without an end */
+SEC("tc")
+__arch_x86_64
+__arch_arm64
+__failure
+__msg("recursive hands_down_its_stack() passes a pointer to the stack to hands_down_its_stack()")
+int global_func3_recursion_stack_pointer(struct __sk_buff *skb)
+{
+	long x = skb->len;
+
+	return hands_down_its_stack(&x, skb->mark & 7);
+}
+
 #define PTR_FRAME(name, next)			\
 static __attribute__ ((noinline))		\
 long name(long *p)				\
diff --git a/tools/testing/selftests/bpf/progs/timer_failure.c b/tools/testing/selftests/bpf/progs/timer_failure.c
index 0538269101ca..84ea861575fc 100644
--- a/tools/testing/selftests/bpf/progs/timer_failure.c
+++ b/tools/testing/selftests/bpf/progs/timer_failure.c
@@ -43,14 +43,7 @@ static unsigned long timer_cb_ret_bad()
 }
 
 SEC("fentry/bpf_fentry_test1")
-__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __failure
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before")
-__msg(": (85) call bpf_get_prandom_u32#7") /* anchor message */
-/* check that branch code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before ") __msg(": (85) call bpf_get_prandom_u32#7")
 __msg("should have been in [0, 0]")
 long BPF_PROG2(test_bad_ret, int, a)
 {
@@ -80,7 +73,7 @@ static int timer_cb_infinite_loop(void *map, int *key, struct elem *val)
 }
 
 SEC("fentry/bpf_fentry_test1")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn {{[0-9]+}} never exits")
 long BPF_PROG2(test_infinite_loop_cb, int, a)
 {
 	struct bpf_timer *timer;
diff --git a/tools/testing/selftests/bpf/progs/verifier_align.c b/tools/testing/selftests/bpf/progs/verifier_align.c
index 0083ef8b8ba7..aecc5279dbaf 100644
--- a/tools/testing/selftests/bpf/progs/verifier_align.c
+++ b/tools/testing/selftests/bpf/progs/verifier_align.c
@@ -13,7 +13,7 @@
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}} R3=2")
 __msg("1: {{.*}} R3=4")
 __msg("2: {{.*}} R3=8")
@@ -35,7 +35,7 @@ __naked void mov(void)
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}}R3=1")
 __msg("1: {{.*}}R3=2")
 __msg("2: {{.*}}R3=4")
@@ -69,7 +69,7 @@ __naked void shift(void)
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}}R3=4")
 __msg("1: {{.*}}R3=8")
 __msg("2: {{.*}}R3=10")
@@ -93,7 +93,7 @@ __naked void addsub(void)
 SEC("tc")
 __success __log_level(2)
 __flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
 __msg("0: {{.*}}R3=7")
 __msg("1: {{.*}}R3=7")
 __msg("2: {{.*}}R3=14")
@@ -532,7 +532,7 @@ __msg("9: {{.*}} R6={{[^)]*}}var_off=(0x0; 0x3c)")
 /* Adding 14 makes R6 be (4n+2) */
 __msg("10: {{.*}} R6={{[^)]*}}var_off=(0x2; 0x7c)")
 /* Subtracting from packet pointer overflows ubounds */
-__msg("13: R5={{[^)]*}}var_off=(0xffffffffffffff82; 0x7c)")
+__msg("12: {{.*}} R5={{[^)]*}}var_off=(0xffffffffffffff82; 0x7c)")
 /* New unknown value in R7 is (4n), >= 76 */
 __msg("14: {{.*}} R7={{[^)]*}}var_off=(0x0; 0x7fc)")
 /* Adding it to packet pointer gives nice bounds again */
diff --git a/tools/testing/selftests/bpf/progs/verifier_and.c b/tools/testing/selftests/bpf/progs/verifier_and.c
index 2b4fdca162be..fb64ea9928f1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_and.c
+++ b/tools/testing/selftests/bpf/progs/verifier_and.c
@@ -38,9 +38,9 @@ __naked void invalid_and_of_negative_number(void)
 	r1 &= -4;					\
 	r1 <<= 2;					\
 	r0 += r1;					\
-l0_%=:	r1 = %[test_val_foo];				\
+	r1 = %[test_val_foo];				\
 	*(u64*)(r0 + 0) = r1;				\
-	exit;						\
+l0_%=:	exit;						\
 "	:
 	: __imm(bpf_map_lookup_elem),
 	  __imm_addr(map_hash_48b),
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
index 1d676da4b5ea..b46e30a9c67f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
@@ -199,6 +199,88 @@ __naked void call_through(void)
 	: __clobber_all);
 }
 
+unsigned long first_r12, last_r12;
+
+/* 1024 bytes in the stack in arena, nothing in the kernel stack */
+static __naked __noinline __used long goes_down(void)
+{
+	asm volatile (
+	".8byte %[adj];"
+	".8byte %[mov];"	/* r1 = r12 */
+	"r2 = %[last_r12] ll;"
+	"*(u64 *)(r2 + 0) = r1;"
+	"call goes_down;"
+	"exit;"
+	:
+	: __imm_addr(last_r12),
+	  __imm_insn(adj, ADJ(1024)),
+	  __imm_insn(mov, MOV(BPF_REG_1))
+	: __clobber_all);
+}
+
+/* The check of a recursion looks at the kernel stack. It fails before 32K of arena are used. */
+SEC("syscall")
+__description("arena stack: recursion ends in the stack")
+__success __retval(1)
+__naked void recursion(void)
+{
+	asm volatile (
+	".8byte %[adj];"
+	".8byte %[mov];"	/* r1 = r12 */
+	"r2 = %[first_r12] ll;"
+	"*(u64 *)(r2 + 0) = r1;"
+	"call goes_down;"
+	"r1 = %[last_r12] ll;"
+	"r1 = *(u64 *)(r1 + 0);"
+	"r2 = %[first_r12] ll;"
+	"r2 = *(u64 *)(r2 + 0);"
+	"r1 -= r2;"
+	"r0 = 0;"
+	"if r1 < 8192 goto +2;"
+	"if r1 > 32736 goto +1;"
+	"r0 = 1;"
+	"exit;"
+	:
+	: __imm_addr(first_r12),
+	  __imm_addr(last_r12),
+	  __imm_insn(adj, ADJ(32)),
+	  __imm_insn(mov, MOV(BPF_REG_1))
+	: __clobber_all);
+}
+
+static __naked __noinline __used long large_frame(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 1792) = 0;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __naked __noinline __used long goes_down_with_leaf(void)
+{
+	asm volatile (
+	".8byte %[adj];"
+	"call large_frame;"
+	"call goes_down_with_leaf;"
+	"exit;"
+	:
+	: __imm_insn(adj, ADJ(1024))
+	: __clobber_all);
+}
+
+/* goes_down_with_leaf() gets a frame in the kernel stack. What it calls comes on top. */
+SEC("syscall")
+__description("arena stack: the kernel frame of a recursion counts in the depth of the stack")
+__failure __msg("combined stack size of 3 calls is")
+__naked void recursion_above_large_frame(void)
+{
+	asm volatile (
+	"call goes_down_with_leaf;"
+	"exit;"
+	::: __clobber_all);
+}
+
 SEC("syscall")
 __description("arena stack: helpers don't change r12")
 __success __retval(3)
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c
index 05efa9f4a522..6fede778971c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c
@@ -1797,8 +1797,8 @@ l1_%=:	r0 = 1;				\
 
 SEC("socket")
 __description("jset on same register, scalar value unknown branch 1")
-__msg("3: (b7) r0 = 0 {{.*}} R0=0")
 __msg("5: (b7) r0 = 1 {{.*}} R0=1")
+__msg("3: (b7) r0 = 0 {{.*}} R0=0")
 __success __log_level(2)
 __naked void jset_on_same_register_3(void *ctx)
 {
@@ -1818,8 +1818,8 @@ l1_%=:	r0 = 1;				\
 
 SEC("socket")
 __description("jset on same register, scalar value unknown branch 2")
-__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __msg("6: (b7) r0 = 1 {{.*}} R0=1")
+__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __success __log_level(2)
 __naked void jset_on_same_register_4(void *ctx)
 {
@@ -1840,8 +1840,8 @@ l1_%=:	r0 = 1;				\
 
 SEC("socket")
 __description("jset on same register, scalar value unknown branch 3")
-__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __msg("6: (b7) r0 = 1 {{.*}} R0=1")
+__msg("4: (b7) r0 = 0 {{.*}} R0=0")
 __success __log_level(2)
 __naked void jset_on_same_register_5(void *ctx)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index f3f8ea67eb7e..afe4990cd027 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -243,8 +243,7 @@ __naked void callx_scratches_args(void)
 /* return value of the callee is tracked */
 SEC("socket")
 __success __log_level(2)
-__msg("to caller at 4:")
-__msg("R0=6")
+__msg("4: R0=6 R10=fp0")
 __naked void callx_retval_is_tracked(void)
 {
 	asm volatile (
@@ -541,9 +540,11 @@ static unsigned long self(void)
 	);
 }
 
-/* unbounded recursion is caught by the main verification pass */
+/* unbounded recursion is stopped at run time */
 SEC("socket")
-__failure __msg("frames is too deep")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
 __naked void callx_unbounded_recursion(void)
 {
 	asm volatile (
@@ -569,12 +570,10 @@ static unsigned long countdown(void)
 	);
 }
 
-/*
- * The depth of the recursion is known to the main verification pass,
- * but recursive calls are not allowed regardless.
- */
 SEC("socket")
-__failure __msg("recursive call from countdown() to countdown()")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
 __naked void callx_bounded_recursion(void)
 {
 	asm volatile (
@@ -621,7 +620,9 @@ static unsigned long pong(void)
 }
 
 SEC("socket")
-__failure __msg("recursive call from")
+__arch_x86_64
+__arch_arm64
+__success
 __naked void callx_mutual_recursion(void)
 {
 	asm volatile (
@@ -636,6 +637,49 @@ __naked void callx_mutual_recursion(void)
 		: __clobber_all);
 }
 
+/* calls_arg(fn) { return fn(); } */
+__naked __noinline __used
+static unsigned long calls_arg(void)
+{
+	asm volatile (
+		"callx r1;"
+		"exit;"
+	);
+}
+
+__naked __noinline __used
+static unsigned long returns_7(void)
+{
+	asm volatile (
+		"r0 = 7;"
+		"exit;"
+	);
+}
+
+/*
+ * The address of calls_arg() is taken: its callx could call it. The main pass finds that it
+ * calls returns_7(). There is no cycle, so calls_arg() doesn't check the stack at run time.
+ */
+SEC("socket")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__xlated("...")
+__xlated("exit")
+__xlated("callx r1")
+__naked void callx_no_recursion_after_all(void)
+{
+	asm volatile (
+		"r2 = %[calls_arg] ll;"
+		"r1 = %[returns_7] ll;"
+		"callx r2;"
+		"exit;"
+		:
+		: __imm_addr(calls_arg),
+		  __imm_addr(returns_7)
+		: __clobber_all);
+}
+
 __naked __noinline __used
 static unsigned long use_stack_304(void)
 {
@@ -944,11 +988,8 @@ long long vals[] SEC(".data.vals") = {1, 2, 3, 4};
 
 SEC("socket")
 __success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 11: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (8d) callx r2")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("5: R0=3 R10=fp0")
+__msg("8: (0f) r1 += r0                      ; R0=24 R1=map_value(map=.data.vals,ks=4,vs=32,imm=24)")
 __retval(4)
 __naked void callx_precision(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 642071fa7498..8bb74b579203 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -669,7 +669,9 @@ static unsigned long selfcall(void)
 }
 
 SEC("socket")
-__failure __msg("recursive call from selfcall() to selfcall()")
+__arch_x86_64
+__arch_arm64
+__success
 __naked void callx_rodata_recursion(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_cfg.c b/tools/testing/selftests/bpf/progs/verifier_cfg.c
index b429fa7a08f5..e97e89ba3d68 100644
--- a/tools/testing/selftests/bpf/progs/verifier_cfg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_cfg.c
@@ -138,8 +138,8 @@ l0_%=:	r1 = r0;					\
 
 SEC("socket")
 __description("conditional loop")
-__failure __msg("infinite loop detected")
-__msg_unpriv("back-edge")
+__success
+__failure_unpriv __msg_unpriv("back-edge")
 __naked void conditional_loop(void)
 {
 	asm volatile ("					\
@@ -175,7 +175,7 @@ l1_%=:	r8 += r4;					\
 
 SEC("socket")
 __description("unconditional loop after conditional jump")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 2 never exits")
 __failure_unpriv __msg_unpriv("back-edge from insn 3 to 2")
 __naked void uncond_loop_after_cond_jmp(void)
 {
@@ -201,7 +201,7 @@ static unsigned long never_ending_subprog()
 SEC("socket")
 __description("unconditional loop after conditional jump")
 /* infinite loop is detected *after* check_cfg() */
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 6 never exits")
 __naked void uncond_loop_in_subprog_after_cond_jmp(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
index 5e8461b2167d..100d8307c3bd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
+++ b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
@@ -557,12 +557,13 @@ __noinline int global_calls_loop(int x)
 
 /*
  * loop_cb_calls_static() -> static_calls_global() -> global_calls_loop() ->
- * bpf_loop() -> loop_cb_calls_static() is an unbounded recursion that the
- * main verification pass can't see, because it doesn't follow calls of global
- * functions. None of the functions use stack.
+ * bpf_loop() -> loop_cb_calls_static() is an unbounded recursion through a global
+ * function and a helper. It's stopped at run time. None of the functions use stack.
  */
-SEC("?raw_tp")
-__failure __msg("recursive call from global_calls_loop() to loop_cb_calls_static()")
+SEC("?socket")
+__arch_x86_64
+__arch_arm64
+__success
 int recursion_via_global_func_and_callback(const void *ctx)
 {
 	return loop_cb_calls_static(0, NULL);
diff --git a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
index c6603a118fdc..1b13d3a8dcfa 100644
--- a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
@@ -717,12 +717,12 @@ __naked void bounds_check_using_bad_access_1(void)
 	*(u64*)(r2 + 0) = r1;				\
 	r1 = %[map_hash_48b] ll;			\
 	call %[bpf_map_lookup_elem];			\
-	if r0 == 0 goto l0_%=;				\
+	if r0 == 0 goto l1_%=;				\
 	r1 = r0;					\
 	r3 = *(u32*)(r0 + 0);				\
 	if r3 < 32 goto l1_%=;				\
 	r1 += r3;					\
-l0_%=:	r0 = 0;						\
+	r0 = 0;						\
 	*(u8*)(r1 + 0) = r0;				\
 	r0 = 0;						\
 	exit;						\
@@ -775,12 +775,12 @@ __naked void bounds_check_using_bad_access_2(void)
 	*(u64*)(r2 + 0) = r1;				\
 	r1 = %[map_hash_48b] ll;			\
 	call %[bpf_map_lookup_elem];			\
-	if r0 == 0 goto l0_%=;				\
+	if r0 == 0 goto l1_%=;				\
 	r1 = r0;					\
 	r3 = *(u32*)(r0 + 0);				\
 	if r3 <= 32 goto l1_%=;				\
 	r1 += r3;					\
-l0_%=:	r0 = 0;						\
+	r0 = 0;						\
 	*(u8*)(r1 + 0) = r0;				\
 	r0 = 0;						\
 	exit;						\
diff --git a/tools/testing/selftests/bpf/progs/verifier_idoms.c b/tools/testing/selftests/bpf/progs/verifier_idoms.c
index fdee6bced77b..91d9f9aae8e9 100644
--- a/tools/testing/selftests/bpf/progs/verifier_idoms.c
+++ b/tools/testing/selftests/bpf/progs/verifier_idoms.c
@@ -10,7 +10,7 @@
  * The immediate dominator is printed by log_program() as the numeric column
  * immediately before the "<insn#>:" field of every "Program dump" line:
  *
- *   Program dump (scc? loop_header? idom insn#: live_regs_before):
+ *   Program dump (loop_header? idom insn#: live_regs_before):
  *            -1   0: ....... (b7) r0 = 0     <- idom(0) = -1 (subprog entry)
  *             0   1: ....... (07) r0 += 1    <- idom(1) = 0
  *             ^^^^^^
@@ -307,17 +307,16 @@ __naked void entry_is_loop_header(void)
 /*
  * Self-loop: insn 1 is its own predecessor. idom(1) must be the pre-header
  * (insn 0); the self-edge is ignored (idoms[pred] == -1 on the first pass, then
- * idoms_intersect(a == b) short-circuits). Program is rejected later for an
- * infinite loop, but the dump (and idoms) are printed before that.
+ * idoms_intersect(a == b) short-circuits). The loop can spin forever: the
+ * verifier adds may_goto to it.
  */
 SEC("socket")
-__failure
+__success
 __log_level(2)
 __msg("Program dump")
 __msg("{{.*}} -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
 __msg("{{.*}}  0   1: {{.*}} (a5) if r0 < 0xa goto pc-1")
 __msg("{{.*}}  1   2: {{.*}} (95) exit")
-__msg("infinite loop detected")
 __naked void self_loop(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
index 4dd98acc6436..a3e31aa25fe1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
+++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
@@ -45,7 +45,7 @@ SEC("?raw_tp")
 __failure __msg("R1 type=scalar expected=fp")
 int unsafe_on_2nd_iter(void *unused)
 {
-	char buf[4];
+	char buf[8];
 	struct buf_context loop_ctx = { .buf = buf };
 
 	bpf_loop(100, unsafe_on_2nd_iter_cb, &loop_ctx, 0);
@@ -94,7 +94,7 @@ static int loop_detection_cb(__u32 idx, struct num_context *ctx)
 }
 
 SEC("?raw_tp")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn {{[0-9]+}} never exits")
 int loop_detection(void *unused)
 {
 	struct num_context loop_ctx = { .i = 0 };
@@ -246,10 +246,9 @@ int bpf_loop_iter_limit_nested(void *unused)
 	/* This makes 'c' zero, but neither clang nor verifier know it. */
 	c /= 10;
 	/* Make sure that verifier does not visit 'impossible' states:
-	 * enumerate all possible callback visit masks.
+	 * one more call of any of the callbacks gives more than that.
 	 */
-	if (a != 0 && a != 1 && a != 11 && a != 101 && a != 111 &&
-	    b != 0 && b != 1 && b != 11 && b != 101 && b != 111)
+	if (a > 111 || b > 111)
 		asm volatile ("r0 /= 0;" ::: "r0");
 	return 1000 * a + b + c;
 }
@@ -291,9 +290,13 @@ static __naked void iter_limit_bug_cb(void)
 	);
 }
 
+/*
+ * may_goto doesn't leave the loop, so it's not what stops it. The address is greater than 10:
+ * the loop spins until may_goto that the verifier adds ends the prog.
+ */
 int tmp_var;
 SEC("socket")
-__failure __msg("infinite loop detected at insn 2")
+__success __retval(0)
 __naked void jgt_imm64_and_may_goto(void)
 {
 	asm volatile ("			\
@@ -310,7 +313,7 @@ l0_%=:	.byte 0xe5; /* may_goto */	\
 }
 
 SEC("socket")
-__failure __msg("infinite loop detected at insn 1")
+__success __retval(0)
 __naked void may_goto_self(void)
 {
 	asm volatile ("			\
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 795956bd4f9f..32bab3cb3fcb 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -2844,7 +2844,7 @@ __naked void merge_preserves_may_write(void)
 	"if r0 == 0 goto 1f;"
 	/* chain A: r2 = map value (ARG_NONE, not stack) */
 	"r1 = r10;"
-	"r1 += -16;"				/* unused fp anchor */
+	"r1 += -16;"				/* fp anchor */
 	"r2 = r0;"
 	"call mvs_mid;"
 	/* chain B: r2 = caller stack slot fp-8 */
@@ -2872,6 +2872,7 @@ static __used __naked void mvs_leaf(void)
 {
 	asm volatile (
 	"*(u64 *)(r2 + 0) = 42;"
+	"r0 = r1;"				/* a register that is not read is no argument */
 	"r0 = 0;"
 	"exit;"
 	::: __clobber_all);
diff --git a/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c b/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
index 72646fa2745e..3f2a60f19319 100644
--- a/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
+++ b/tools/testing/selftests/bpf/progs/verifier_liveness_exp.c
@@ -32,8 +32,12 @@ char _license[] SEC("license") = "GPL";
 			"r1 += -" #off ";"	\
 			"call " #fn ";"
 
-/* 50 calls, each with a distinct FP offset: -8, -16, ... -400 */
+/*
+ * 50 calls, each with a distinct FP offset: -8, -16, ... -400.
+ * The function reads r1 first: a register that the callee doesn't read is no argument.
+ */
 #define CALLS_50(fn)							\
+	"r0 = r1;"							\
 	C(fn,   8) C(fn,  16) C(fn,  24) C(fn,  32) C(fn,  40)		\
 	C(fn,  48) C(fn,  56) C(fn,  64) C(fn,  72) C(fn,  80)		\
 	C(fn,  88) C(fn,  96) C(fn, 104) C(fn, 112) C(fn, 120)		\
@@ -50,6 +54,7 @@ __naked __noinline __used
 static unsigned long exp_sub7(void)
 {
 	asm volatile (
+		"r0 = r1;"
 		"r0 = 0;"
 		"exit;"
 		::: __clobber_all);
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
index 0608e983d7fc..4f4e04d50bc7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
@@ -22,10 +22,10 @@ __success
 __log_level(2)
 __msg("loop at 1{{$}}")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
-__msg("          2   3: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("      2   3: {{.*}} (95) exit")
 __naked void loop_single(void)
 {
 	asm volatile ("					\
@@ -43,13 +43,13 @@ __log_level(2)
 __msg("loop at 1{{$}}")
 __msg("loop at 4{{$}}")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  2       0   1: {{.*}} (07) r0 += 1")
-__msg("  2   1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
-__msg("          2   3: {{.*}} (b7) r1 = 0")
-__msg("  1       3   4: {{.*}} (07) r1 += 1")
-__msg("  1   4   4   5: {{.*}} (a5) if r1 < 0xa goto pc-2")
-__msg("          5   6: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("      2   3: {{.*}} (b7) r1 = 0")
+__msg("      3   4: {{.*}} (07) r1 += 1")
+__msg("  4   4   5: {{.*}} (a5) if r1 < 0xa goto pc-2")
+__msg("      5   6: {{.*}} (95) exit")
 __naked void loop_two_independent(void)
 {
 	asm volatile ("					\
@@ -70,13 +70,13 @@ __log_level(2)
 __msg("loop at 1{{$}}")
 __msg("loop at 3, nested in 1")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")			/* outer loop header */
-__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")			/* outer loop insn */
-__msg("  1   1   2   3: {{.*}} (07) r1 += 1")			/* inner loop header */
-__msg("  1   3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")	/* inner loop insn */
-__msg("  1   1   4   5: {{.*}} (a5) if r0 < 0xa goto pc-5")	/* outer loop insn */
-__msg("          5   6: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")			/* outer loop header */
+__msg("  1   1   2: {{.*}} (b7) r1 = 0")			/* outer loop insn */
+__msg("  1   2   3: {{.*}} (07) r1 += 1")			/* inner loop header */
+__msg("  3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")	/* inner loop insn */
+__msg("  1   4   5: {{.*}} (a5) if r0 < 0xa goto pc-5")	/* outer loop insn */
+__msg("      5   6: {{.*}} (95) exit")
 __naked void loop_nested(void)
 {
 	asm volatile ("					\
@@ -95,12 +95,12 @@ SEC("socket")
 __success
 __log_level(2)
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (85) call bpf_get_prandom_u32")
-__msg("          0   1: {{.*}} (25) if r0 > 0x0 goto pc+2")
-__msg("          1   2: {{.*}} (b7) r0 = 2")
-__msg("          2   3: {{.*}} (05) goto pc+1")
-__msg("          1   4: {{.*}} (b7) r0 = 3")
-__msg("          1   5: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (85) call bpf_get_prandom_u32")
+__msg("      0   1: {{.*}} (25) if r0 > 0x0 goto pc+2")
+__msg("      1   2: {{.*}} (b7) r0 = 2")
+__msg("      2   3: {{.*}} (05) goto pc+1")
+__msg("      1   4: {{.*}} (b7) r0 = 3")
+__msg("      1   5: {{.*}} (95) exit")
 __naked void fwd_edges_no_loop(void)
 {
 	asm volatile ("					\
@@ -123,16 +123,16 @@ __msg("loop at 1{{$}}")
 __msg("loop at 3, nested in 1")
 __msg("loop at 6, nested in 1")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")
-__msg("  1   1   2   3: {{.*}} (07) r1 += 1")
-__msg("  1   3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
-__msg("  1   1   4   5: {{.*}} (b7) r2 = 0")
-__msg("  1   1   5   6: {{.*}} (07) r2 += 1")
-__msg("  1   6   6   7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
-__msg("  1   1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
-__msg("          8   9: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (b7) r1 = 0")
+__msg("  1   2   3: {{.*}} (07) r1 += 1")
+__msg("  3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
+__msg("  1   4   5: {{.*}} (b7) r2 = 0")
+__msg("  1   5   6: {{.*}} (07) r2 += 1")
+__msg("  6   6   7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
+__msg("  1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg("      8   9: {{.*}} (95) exit")
 __naked void loop_nested_siblings(void)
 {
 	asm volatile ("					\
@@ -157,16 +157,16 @@ __msg("loop at 1{{$}}")
 __msg("loop at 3, nested in 1")
 __msg("loop at 5, nested in 3")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")
-__msg("  1   1   2   3: {{.*}} (07) r1 += 1")
-__msg("  1   3   3   4: {{.*}} (b7) r2 = 0")
-__msg("  1   3   4   5: {{.*}} (07) r2 += 1")
-__msg("  1   5   5   6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
-__msg("  1   3   6   7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
-__msg("  1   1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
-__msg("          8   9: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (b7) r1 = 0")
+__msg("  1   2   3: {{.*}} (07) r1 += 1")
+__msg("  3   3   4: {{.*}} (b7) r2 = 0")
+__msg("  3   4   5: {{.*}} (07) r2 += 1")
+__msg("  5   5   6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
+__msg("  3   6   7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
+__msg("  1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg("      8   9: {{.*}} (95) exit")
 __naked void loop_three_levels(void)
 {
 	asm volatile ("					\
@@ -189,14 +189,14 @@ __success
 __log_level(2)
 __msg("loop at 1")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (b7) r0 = 0")
-__msg("  1       0   1: {{.*}} (07) r0 += 1")
-__msg("  1   1   1   2: {{.*}} (bf) r1 = r0")
-__msg("  1   1   2   3: {{.*}} (25) if r1 > 0x5 goto pc+1")
-__msg("  1   1   3   4: {{.*}} (b7) r1 = 1")
-__msg("  1   1   3   5: {{.*}} (0f) r0 += r1")
-__msg("  1   1   5   6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
-__msg("          6   7: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (b7) r0 = 0")
+__msg("      0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   2: {{.*}} (bf) r1 = r0")
+__msg("  1   2   3: {{.*}} (25) if r1 > 0x5 goto pc+1")
+__msg("  1   3   4: {{.*}} (b7) r1 = 1")
+__msg("  1   3   5: {{.*}} (0f) r0 += r1")
+__msg("  1   5   6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
+__msg("      6   7: {{.*}} (95) exit")
 __naked void loop_with_if_else(void)
 {
 	asm volatile ("					\
@@ -217,15 +217,15 @@ __success
 __log_level(2)
 __msg("loop at 3, irreducible")
 __msg("Program dump")
-__msg("         -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
-__msg("          0   1: {{.*}} (b7) r1 = 0")
-__msg("          1   2: {{.*}} (25) if r0 > 0x5 goto pc+2")
-__msg("  1       2   3: {{.*}} (b7) r1 = 1")
-__msg("  1   3   3   4: {{.*}} (05) goto pc+1")
-__msg("          2   5: {{.*}} (b7) r1 = 2")
-__msg("  1   3   2   6: {{.*}} (0f) r0 += r1")
-__msg("  1   3   6   7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
-__msg("          7   8: {{.*}} (95) exit")
+__msg("     -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("      0   1: {{.*}} (b7) r1 = 0")
+__msg("      1   2: {{.*}} (25) if r0 > 0x5 goto pc+2")
+__msg("      2   3: {{.*}} (b7) r1 = 1")
+__msg("  3   3   4: {{.*}} (05) goto pc+1")
+__msg("      2   5: {{.*}} (b7) r1 = 2")
+__msg("  3   2   6: {{.*}} (0f) r0 += r1")
+__msg("  3   6   7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
+__msg("      7   8: {{.*}} (95) exit")
 __naked void loop_irreducible(void)
 {
 	asm volatile ("					\
@@ -244,10 +244,9 @@ __naked void loop_irreducible(void)
 }
 
 SEC("socket")
-__failure
+__success
 __log_level(2)
 __msg("loop at 1")
-__msg("infinite loop detected at insn 1")
 __naked void loop_self(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
index 6dd46edd8ed0..e8daf791b4c0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -120,7 +120,7 @@ BAD("down from a range, one unsigned test, one too many")
 LOOP(down_unsigned_range_bad, "if r7 > 2048 goto 9f;", "w7 += -1; if w7 < 2049 goto 1b;")
 
 /* below 0 a 32-bit counter is 0xffffffff for a 64-bit jump */
-BAD("down, a 64-bit test with a bound above 32 bits")
+NO_COUNT("down, a 64-bit test with a bound above 32 bits")
 LOOP(down_unsigned_wide_bound, "w8 = 2047; r9 = 0x100000800 ll;", "w8 += -1; if r8 < r9 goto 1b;")
 
 BAD("down, an unsigned and a signed 64-bit test")
diff --git a/tools/testing/selftests/bpf/progs/verifier_loops1.c b/tools/testing/selftests/bpf/progs/verifier_loops1.c
index 48a966cda199..18067907579e 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loops1.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loops1.c
@@ -123,7 +123,7 @@ l0_%=:	exit;						\
 
 SEC("tracepoint")
 __description("infinite loop after a conditional jump")
-__failure __msg("program is too large")
+__failure __msg("loop at insn 2 never exits")
 __naked void loop_after_a_conditional_jump(void)
 {
 	asm volatile ("					\
@@ -135,10 +135,11 @@ l0_%=:	exit;						\
 "	::: __clobber_all);
 }
 
-SEC("tracepoint")
+SEC("xdp")
 __description("bounded recursion")
-__failure
-__msg("recursive call from")
+__arch_x86_64
+__arch_arm64
+__success __retval(4)
 __naked void bounded_recursion(void)
 {
 	asm volatile ("					\
@@ -163,7 +164,7 @@ l0_%=:	call bounded_recursion__1;			\
 
 SEC("tracepoint")
 __description("infinite loop in two jumps")
-__failure __msg("loop detected")
+__failure __msg("loop at insn 1 never exits")
 __naked void infinite_loop_in_two_jumps(void)
 {
 	asm volatile ("					\
@@ -174,9 +175,10 @@ l0_%=:	if r0 < 4 goto l1_%=;				\
 "	::: __clobber_all);
 }
 
-SEC("tracepoint")
+/* It never ends on its own. may_goto that the verifier adds ends the prog. */
+SEC("xdp")
 __description("infinite loop: three-jump trick")
-__failure __msg("loop detected")
+__success __retval(0)
 __naked void infinite_loop_three_jump_trick(void)
 {
 	asm volatile ("					\
diff --git a/tools/testing/selftests/bpf/progs/verifier_movsx.c b/tools/testing/selftests/bpf/progs/verifier_movsx.c
index 195b27a51224..2f4a678c31c2 100644
--- a/tools/testing/selftests/bpf/progs/verifier_movsx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_movsx.c
@@ -226,7 +226,8 @@ l0_%=:							\
 
 SEC("socket")
 __description("MOV32SX, S8, var_off u32_max")
-__failure __msg("infinite loop detected")
+__success __log_level(2)
+__msg("w7 = (s8)w3 {{.*}}R7=scalar({{.*}}smin32=-128,smax32=127,var_off=(0x0; 0xffffffff))")
 __failure_unpriv __msg_unpriv("back-edge from insn 2 to 0")
 __naked void mov64sx_s32_varoff_1(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index abe2b1911079..d89ffe90c6bc 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -8,6 +8,7 @@
 #include "../../../include/linux/filter.h"
 #include "bpf_misc.h"
 #include "bpf_kfuncs.h"
+#include "bpf_compiler.h"
 
 #define COND_TEST(name, _initial, _step, _bound, msg, body)                     \
 SEC("xdp")                                                                      \
@@ -238,7 +239,7 @@ __naked void meet_agrees(void)
 }
 
 SEC("xdp")
-__failure
+__success __retval(0)
 __log_level(2)
 __msg_next("scev at header 1:")
 __msg_next("  r6=(any r6 (+ r6 1)) / ?")
@@ -1234,23 +1235,20 @@ RANGE_TEST(backedges_first_is_a_range, "", 0, 3, 1, 0, 2,
 RANGE_TEST(backedges_two_variables, "", 0, 3, 1, 0, 1,
 	   BACKEDGES("r6 += 1; r7 += 1", "r9 & 1", "r6 < 3", "r7 < 6"),
 	   __msg("header_count is [0..6] ") __msg("widening r6 to 0..8 "))
-/* Explore with a bound of 3 first, of 5 next. The second state is not equal to the first. */
+/* A bound of 3 and a bound of 5 are joined in front of the loop */
 #define PRECISE_BOUND(reg)									\
 	"r4 = 2; r5 = 2; " reg " = 5; if r7 == 0 goto +1; " reg " = 3; r0 = 0;"		\
 	BACKEDGES("r6 += 1", "r9 & 1", "r6 < r4", "r6 < r5")
-#define EXPLORED_TWICE										\
-	__flag(BPF_F_TEST_STATE_FREQ)								\
-	__msg("header_count is [0..3] ") __msg("header_count is [0..5] ")
-RANGE_TEST(backedges_bound1_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r4"), EXPLORED_TWICE)
-RANGE_TEST(backedges_bound0_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r5"), EXPLORED_TWICE)
+#define EXPLORED_ONCE	__msg("header_count is [0..5] ") __not_msg("header_count is [0..3] ")
+RANGE_TEST(backedges_bound1_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r4"), EXPLORED_ONCE)
+RANGE_TEST(backedges_bound0_is_precise, "", 0, 0, 1, 0, 1, PRECISE_BOUND("r5"), EXPLORED_ONCE)
 
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 10 ")
 __msg("loop header at 1, widening r0 to 0..9 step 1")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
 __naked void post_cond_jlt(void)
 {
 	asm volatile ("					\
@@ -1262,8 +1260,8 @@ __naked void post_cond_jlt(void)
 }
 
 /*
- * Explore with r7=10 first, r7=4096 next.
- * Should trigger invalid stack read if r7 is marked precise.
+ * r7 is 10 on one path to the loop and 4096 on the other.
+ * Should trigger invalid stack read if the larger one counts.
  *
  *   r0 = 0;
  *   r6 = 0;
@@ -1278,11 +1276,7 @@ __naked void post_cond_jlt(void)
 SEC("xdp")
 __failure
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("loop header at 6, header_count is 10 ")
-__msg("mark_precise: frame0: parent state regs=r6,r7 stack=:")
-__msg("loop header at 6, widening r0 to 0..9 step 1")
-__msg("loop header at 6, header_count is 4096 ")
+__msg("loop header at 6, header_count is [0..4096] ")
 __msg("loop header at 6, widening r0 to 0..4095 step 1")
 __msg("invalid variable-offset read from stack R2")
 __naked void mark_bound_precise(void)
@@ -1308,8 +1302,8 @@ __naked void mark_bound_precise(void)
 }
 
 /*
- * Explore with r6=4086 first, r6=0 next.
- * Should trigger invalid stack read if r6 is marked precise.
+ * r6 is 4086 on one path to the loop and 0 on the other.
+ * Should trigger invalid stack read if the smaller one counts.
  *
  *   r0 = 0;
  *   r6 = bpf_get_prandom_u32() == 42 ? 4086 : 0;
@@ -1323,11 +1317,7 @@ __naked void mark_bound_precise(void)
 SEC("xdp")
 __failure
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("loop header at 5, header_count is 10 ")
-__msg("mark_precise: frame0: parent state regs=r6 stack=:")
-__msg("loop header at 5, widening r0 to 0..9 step 1")
-__msg("loop header at 5, header_count is 4096 ")
+__msg("loop header at 5, header_count is [0..4096] ")
 __msg("loop header at 5, widening r0 to 0..4095 step 1")
 __msg("invalid variable-offset read from stack R2")
 __naked void mark_base_precise(void)
@@ -1541,7 +1531,7 @@ __msg("scev at header 1:")
 __msg_next("r6=(+ r6 1) / (linear r6 1)")
 __msg_next("*fp-8=(any *fp-8 7) / (any *fp-8 7)")
 __msg("loop header at 1, widening r6 to 0..2 step 1")
-__msg("2: (7a) *(u64 *)(r10 -8) = 7          ; R10=fp0 fp-8=7 loop_stack=1")
+__msg("3: (79) r0 = *(u64 *)(r10 -8)         ; R0=scalar() R10=fp0 loop_stack=1")
 __naked void widen_any_unallocated_slot(void)
 {
 	asm volatile ("					\
@@ -1819,11 +1809,10 @@ BOUND_IN_MAP_TEST(bound_in_data, data_bound, 16)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 3, header_count is 8")
 __msg("loop header at 3, widening r0 to 2..9 step 1")
 __msg("3: R0=scalar(smin=umin=smin32=umin32=2,smax=umax=smax32=umax32=9,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void post_cond_jlt_with_base(void)
 {
 	asm volatile ("					\
@@ -1910,11 +1899,10 @@ NO_ZEXT32_TEST(arsh_by_32, 0x7ffffffe, "r7 <<= 32; r7 s>>= 32;")
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 11")
 __msg("loop header at 1, widening r0 to 0..10 step 1")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
 __naked void post_cond_jle(void)
 {
 	asm volatile ("					\
@@ -1928,11 +1916,10 @@ __naked void post_cond_jle(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 10")
 __msg("loop header at 1, widening r0 to 0..9 step 1")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=9,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void post_cond_jge(void)
 {
 	asm volatile ("					\
@@ -1947,11 +1934,10 @@ __naked void post_cond_jge(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 1, header_count is 11")
 __msg("loop header at 1, widening r0 to 0..10 step 1")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void pre_cond_jge(void)
 {
 	asm volatile ("					\
@@ -1966,7 +1952,6 @@ __naked void pre_cond_jge(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(+ r0 1) / (linear r0 1)")
 __msg_next(" scev at latch 2:")
@@ -1975,10 +1960,9 @@ __msg("loop header at 1, widening r0")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
 __msg("1: (07) r0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
 __msg("2: (55) if r0 != 0x3 goto pc-2")
-__msg("3: (95) exit")
 __msg("loop header at 1, clamping r0")
-__msg("from 2 to 1: safe")
-__msg("processed 5 insns")
+__msg("3: (95) exit")
+__msg("processed 4 insns")
 __naked void post_cond_jne(void)
 {
 	asm volatile ("					\
@@ -1992,16 +1976,15 @@ __naked void post_cond_jne(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(+ r0 -1) / (linear r0 -1)")
 __msg_next(" scev at latch 2:")
 __msg_next("  r0=(+ r0 -1) / (linear (+ r0 -1) -1)")
 __msg("loop header at 1, header_count is 3 ")
 __msg("loop header at 1, widening r0 to 1..3 step 1")
-__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) loop_stack=1")
+__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) R10=fp0 loop_stack=1")
 __msg("loop header at 1, clamping r0 to 1..3 step 1")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
 __naked void post_cond_jne_neg_step(void)
 {
 	asm volatile ("					\
@@ -2025,7 +2008,7 @@ __msg("loop header at 2, widening r1 to 0..4 step 1")
 __msg("loop header at 2, widening r3 to 0..4 step 1")
 __msg("3: (55) if r3 != 0x5 goto pc+1 {{.*}}; R3=5")
 __msg("loop header at 2, clamping r3 to 0..4 step 1")
-__msg("from 6 to 2: safe")
+__msg("from 6 to 2: within")
 __naked void clamp_dependent_overshoot(void)
 {
 	asm volatile ("					\
@@ -2051,7 +2034,7 @@ __msg("loop header at 2, widening r1 to 0..4 step 1")
 __msg("loop header at 2, widening r3 to 0..4 step 1")
 __msg("3: (56) if w3 != 0x5 goto pc+1 {{.*}}; R3=5")
 __msg("loop header at 2, clamping r3 to 0..4 step 1")
-__msg("from 6 to 2: safe")
+__msg("from 6 to 2: within")
 __naked void clamp_dependent_overshoot32(void)
 {
 	asm volatile ("					\
@@ -2070,7 +2053,6 @@ __naked void clamp_dependent_overshoot32(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(+ r0 1) / (linear r0 1)")
 __msg_next(" scev at latch 1:")
@@ -2081,11 +2063,10 @@ __msg("1: (15) if r0 == 0x3 goto pc+2        ; R0=scalar(smin=smin32=0,smax=umax
 __msg("2: (07) r0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
 __msg("3: (05) goto pc-3")
 __msg("loop header at 1, clamping r0")
-__msg("1: safe")
 __msg("from 1 to 4: R0=3")
 __msg("4: R0=3")
 __msg("4: (95) exit")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
 __naked void pre_cond_je1(void)
 {
 	asm volatile ("					\
@@ -2100,7 +2081,6 @@ __naked void pre_cond_je1(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 1:")
 __msg_next("  r0=(zext32 (+ r0 1)) / (zext32 (linear r0 1))")
 __msg_next(" scev at latch 2:")
@@ -2109,10 +2089,9 @@ __msg("loop header at 1, widening r0")
 __msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
 __msg("1: (04) w0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
 __msg("2: (56) if w0 != 0x3 goto pc-2")
-__msg("3: (95) exit")
 __msg("loop header at 1, clamping r0")
-__msg("from 2 to 1: safe")
-__msg("processed 5 insns")
+__msg("3: (95) exit")
+__msg("processed 4 insns")
 __naked void post_cond_jne32(void)
 {
 	asm volatile ("					\
@@ -2386,7 +2365,6 @@ __naked void loop_around_301_exits(void)
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg_next("scev at header 11:")
 __msg_next("  r0=(+ r0 1) / (linear r0 1)")
 __msg_next("  r1=(+ r1 2) / (linear r1 2)")
@@ -2396,18 +2374,15 @@ __msg_next("  r1=(+ r1 2) / (linear (+ r1 2) 2)")
 __msg("loop header at 11, widening r0")
 __msg("loop header at 11, widening r1")
 __msg("11: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=7,var_off=(0x0; 0x7)) R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=14,var_off=(0x0; 0xe),step=0+2)")
-/* loop exit */
 __msg("16: (a5) if r0 < 0x8 goto pc-6")
-__msg("exiting loop 11")
-__msg("17: (95) exit")
 /* second iteration */
 __msg("loop header at 11, clamping r0")
 __msg("loop header at 11, clamping r1")
 /* iteration convergence */
-__msg("from 16 to 11: safe")
 __not_msg("{{^}}11:")
-/* map lookup error path */
-__msg("from 7 to 17: safe")
+/* loop exit */
+__msg("exiting loop 11")
+__msg("17: (95) exit")
 __naked void correlated_regs(void)
 {
 	asm volatile ("					\
@@ -2441,15 +2416,14 @@ __naked void correlated_regs(void)
  *     k += 1
  *     k <<= 1   // make SCEV construction not possible
  *     k >>= 1
- * map[k] = 1    // make k precise
+ * map[k] = 1    // k has to be known
  */
 SEC("xdp")
 __success
 __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
 __msg("19: (72) *(u8 *)(r4 +0) = 1           ; R4=map_value(id={{.*}},map=map,ks=4,vs=1024,imm=16)")
 __not_msg("19: ")
-__msg("processed 106 insns")
+__msg("processed 105 insns")
 __naked void nested_loops_precise_var1(void)
 {
 	asm volatile ("					\
@@ -2488,8 +2462,7 @@ __log_level(2)
 __msg("loop header at 2, header_count is 100 ")
 __msg("loop header at 4, header_count is [0..100] ")
 __msg("loop header at 6, header_count is [0..100] ")
-__msg("processed 20 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 14 insns")
 __naked void nested_loop_with_two_exits(void)
 {
 	asm volatile ("					\
@@ -2550,14 +2523,14 @@ __msg("loop at 5")
 __msg("loop at 6, nested in 5")
 __msg("loop at 11, irreducible")
 __msg("loop at 12, nested in 11")
-/* entry via if r0 == 5 goto d_%= false branch */
+/* entry via if r0 == 5 goto d_%= true branch */
 __msg("loop header at 12, header_count is 3 ")
 __msg("loop header at 12, widening r9 to 0..2 step 1")
-__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) loop_stack=11,12")
-/* entry via if r0 == 5 goto d_%= true branch */
+/* entry via if r0 == 5 goto d_%= false branch */
 __msg("loop header at 12, header_count is 3 ")
 __msg("loop header at 12, widening r9 to 0..2 step 1")
-__msg("from 8 to 12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 loop_stack=11,12")
+__msg("12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 iter(11)=0 loop_stack=11,12")
+__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 iter(11)=0 loop_stack=11,12")
 __naked void enter_nested_loop_from_side(void)
 {
 	asm volatile ("					\
@@ -3037,7 +3010,7 @@ __log_level(2)
 __flag(BPF_F_TEST_STATE_FREQ)
 __msg("loop header at 3, widening r6 to 0..9 step 1")
 __msg("loop header at 3, clamping r6 to 0..9 step 1")
-__msg("12: safe")
+__msg("from 12 to 6: within")
 __naked void clamp_callee_state_pruning(void)
 {
 	asm volatile ("					\
@@ -3204,11 +3177,14 @@ RANGE_TEST(spill32_first_is_a_range, "", 0, 3, 1, 6, 6,
 RANGE_TEST(spill32_first_is_a_range_pre, "", 0, 3, 1, 6, 6,
 	   SPILL32_LOOP(SPILL32_LOAD "if r1 >= r7 goto 2f;" SPILL32_INC "goto 1b;"),
 	   SPILL32_COUNTED(7, "0..6"))
-/* Explore with 2 in the slot first, with 0 next. The second state is not equal to the first. */
+/* 2 and 0 in the slot are joined in front of the loop */
 RANGE_TEST(spill32_first_is_precise, "", 0, 0, 1, 0, 1,
 	   "if r7 == 0 goto +1; r6 = 2;" SPILL32_LOOP(SPILL32_INC "if w1 != 5 goto 1b;"),
-	   __flag(BPF_F_TEST_STATE_FREQ)
-	   __msg("header_count is [0..3] ") __msg("header_count is [0..5] "))
+	   __msg("header_count is [0..5] ") __not_msg("header_count is [0..3] "))
+/* The slot holds 64 bits when the loop starts. Nothing reads the high half: it's dead. */
+RANGE_TEST(spill32_first_is_64bit, "", 0, 0, 1, 5, 5,
+	   "*(u64 *)(r10 - 8) = r6; 1:" LIMIT_ITERATIONS SPILL32_INC "if w1 != w7 goto 1b;",
+	   SPILL32_COUNTED(5, "0..4"))
 /* The counter makes the address of a spill. Such a loop is not summarized, as with a register. */
 PARTIAL_SPILL_TEST(spill32_is_a_stack_offset, u32, 4,
 	"can't widen *fp-8, expr is (spill32 (+ *fp-8 8)), requires exact stack-offset tracking",
@@ -3224,11 +3200,7 @@ PARTIAL_SPILL_TEST(spill32_unknown_is_a_stack_offset, u32, 1000000,
 /* A step above S16_MAX has no range, as with a register */
 PARTIAL_SPILL_TEST(spill32_large_step, u32, 4, "can't widen *fp-8, expr is (spill32 ",
 	SPILL32_0, SPILL32_LOAD "w1 += 0x8000; *(u32 *)(r10 - 8) = r1;")
-/* No formula: the slot holds 64 bits when the loop starts, */
-RANGE_TEST(spill32_first_is_64bit, "", 0, 0, 1, 5, 5,
-	   "*(u64 *)(r10 - 8) = r6; 1:" LIMIT_ITERATIONS SPILL32_INC "if w1 != w7 goto 1b;",
-	   NO_COUNT)
-/* the load extends the sign, */
+/* No formula: the load extends the sign, */
 RANGE_TEST(spill32_sext_load, "", 0, 0, 1, 5, 5,
 	   SPILL32_LOOP(SPILL32_INC "r1 = *(s32 *)(r10 - 8); if r1 != r7 goto 1b;"), NO_COUNT)
 /* the sum is made from a different slot. */
@@ -4298,12 +4270,23 @@ exit_%=:						\
  *        } while (r8 < 3);
  *    }
  *    return 0;
+ *
+ * The outer loop gets may_goto in front of the jump that leaves it, the inner loop doesn't.
+ * At run time it's may_goto that ends the prog.
  */
 SEC("xdp")
-__failure
-__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("The sequence of 8193 jumps is too complex.")
+__success __retval(0)
+__xlated("call unknown")
+__xlated("...")
+__xlated("if r0 == 0x2a goto pc+4")
+__xlated("r8 = 0")
+__xlated("r8 += 1")
+__xlated("if r8 < 0x3 goto pc-2")
+__xlated("goto pc-{{[0-9]+}}")
+__xlated("r0 = 0")
+__xlated("exit")
+__xlated("r0 = 0")
+__xlated("exit")
 __naked void nested_terminating_prune(void)
 {
 	asm volatile ("					\
@@ -4678,7 +4661,7 @@ __naked void widen_pkt_pointer_negative_offset(void)
 
 /* The access is in the second iteration only: the verifier gets to it after the clamp */
 SEC("tc")
-__failure __msg("invalid access to packet, off=56 size=2, R6(id=3,off=56,r=56)")
+__failure __msg("invalid access to packet, off=56 size=2, R6(id={{[0-9]+}},off=56,r=56)")
 __naked void clamp_pkt_pointer_step_lowers_umax(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
index 8f6082fdb5c8..e7636b86cc29 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
@@ -127,13 +127,13 @@ static __u64 stats_async_schedule(struct bpf_map *map, __u32 *key,
 SEC("?raw_tp")
 __success __log_level(4)
 /*
- * self: 9 + 7 + 2 = 18
- * totals: leaf 2, scheduler 7, main root 18
+ * self: 8 + 7 + 2 = 17
+ * totals: leaf 2, scheduler 7, main root 17
  */
-__msg("subprog 0 (stats_async_direct) main insns_self 9 insns_total 18 stack 0")
+__msg("subprog 0 (stats_async_direct) main insns_self 8 insns_total 17 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_schedule) static insns_self 7 insns_total 7 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_leaf) static insns_self 2 insns_total 2 stack 0")
-__msg("processed 18 insns")
+__msg("processed 17 insns")
 __naked int stats_async_direct(void)
 {
 	asm volatile (
@@ -194,14 +194,14 @@ static __u64 stats_async_nested_schedule(struct bpf_map *map, __u32 *key,
 SEC("?raw_tp")
 __success __log_level(4)
 /*
- * self: 9 + 7 + 7 + 2 = 25
- * totals: leaf 2, outer 7, scheduler 7, main root 25
+ * self: 8 + 7 + 7 + 2 = 24
+ * totals: leaf 2, outer 7, scheduler 7, main root 24
  */
-__msg("subprog 0 (stats_async_nested) main insns_self 9 insns_total 25 stack 0")
+__msg("subprog 0 (stats_async_nested) main insns_self 8 insns_total 24 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_nested_schedule) static insns_self 7 insns_total 7 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_outer) static insns_self 7 insns_total 7 stack 0")
 __msg("subprog {{[0-9]+}} (stats_async_nested_leaf) static insns_self 2 insns_total 2 stack 0")
-__msg("processed 25 insns")
+__msg("processed 24 insns")
 __naked int stats_async_nested(void)
 {
 	asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
index 42b90b692dc5..5ba41dca35ea 100644
--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
@@ -148,6 +148,33 @@ __naked void tail_call_caller_stack_ok(void)
 "	::: __clobber_all);
 }
 
+static __noinline int tail_calls(struct __sk_buff *skb)
+{
+	bpf_tail_call_static(skb, &jmp_table, 0);
+	return skb->len;
+}
+
+static __noinline int calls_itself(struct __sk_buff *skb, int n)
+{
+	int ret = 0;
+
+	if (n)
+		ret = calls_itself(skb, n - 1);
+	return ret + tail_calls(skb);
+}
+
+/* The frames of a subprog that calls itself stay on the stack too: up to a half of it. */
+SEC("tc")
+__description("tail call from subprog below a recursive subprog")
+__arch_x86_64
+__arch_arm64
+__failure
+__msg("tail_calls are not allowed when call stack of previous frames is {{[0-9]+}} bytes")
+int tail_call_below_recursion(struct __sk_buff *skb)
+{
+	return calls_itself(skb, 2);
+}
+
 SEC("tc")
 __description("tail call from subprog with 256 bytes of caller stack")
 __failure
diff --git a/tools/testing/selftests/bpf/progs/verifier_uninit.c b/tools/testing/selftests/bpf/progs/verifier_uninit.c
index 691018a46049..345223a4525b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_uninit.c
+++ b/tools/testing/selftests/bpf/progs/verifier_uninit.c
@@ -9,7 +9,7 @@
 SEC("socket")
 __description("read uninitialized register")
 __failure __msg("R2 !read_ok")
-__msg("R2 has never been initialized on this path")
+__msg("R2 is not readable here. It may never have been initialized")
 __failure_unpriv
 __naked void read_uninitialized_register(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
index 1d255b04565d..62f998216d4b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
@@ -394,8 +394,8 @@ __naked void fill_of_different_pointers_stx(void)
 	r2 = r10;					\
 	r2 += -16;					\
 	*(u64*)(r6 + 0) = r2;				\
-l0_%=:	if r1 != 0 goto l1_%=;				\
-	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
+l0_%=:	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	r1 = *(u64*)(r6 + 0);				\
 	*(u32*)(r1 + %[__sk_buff_mark]) = r3;		\
 	r0 = 0;						\
@@ -420,8 +420,8 @@ __naked void fill_of_different_pointers_st(void)
 	r2 = r10;					\
 	r2 += -16;					\
 	*(u64*)(r6 + 0) = r2;				\
-l0_%=:	if r1 != 0 goto l1_%=;				\
-	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
+l0_%=:	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	r1 = *(u64*)(r6 + 0);				\
 	.8byte %[st_mem];				\
 	r0 = 0;						\
@@ -435,7 +435,8 @@ l1_%=:	r1 = *(u64*)(r6 + 0);				\
 
 SEC("tc")
 __description("unpriv: spill/fill of different pointers stx - ctx and sock")
-__failure __msg("type=ctx expected=sock")
+/* bpf_sk_release(skb) is an error too: "type=ctx expected=sock". This one is found first. */
+__failure __msg("R1 invalid mem access 'sock_or_null'")
 __naked void pointers_stx_ctx_and_sock(void)
 {
 	asm volatile ("					\
@@ -449,10 +450,10 @@ __naked void pointers_stx_ctx_and_sock(void)
 	r6 += -8;					\
 	r1 = r8;					\
 	/* if (skb == NULL) *target = sock; */		\
-	if r1 == 0 goto l0_%=;				\
+	if r1 != 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = skb; */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	/* struct __sk_buff *skb = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
@@ -490,10 +491,10 @@ __naked void different_pointers_stx_leak_sock(void)
 	r6 += -8;					\
 	r1 = r8;					\
 	/* if (skb == NULL) *target = sock; */		\
-	if r1 == 0 goto l0_%=;				\
+	if r1 != 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = skb; */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	/* struct __sk_buff *skb = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
@@ -526,8 +527,8 @@ __naked void stx_sock_and_ctx_read(void)
 	/* if (skb) *target = skb */			\
 	if r1 == 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = sock */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
 l1_%=:	/* struct bpf_sock *sk = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
@@ -565,8 +566,8 @@ __naked void stx_sock_and_ctx_write(void)
 	/* if (skb) *target = skb */			\
 	if r1 == 0 goto l0_%=;				\
 	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
 l0_%=:	/* else *target = sock */			\
-	if r1 != 0 goto l1_%=;				\
 	*(u64*)(r6 + 0) = r2;				\
 l1_%=:	/* struct bpf_sock *sk = *target; */		\
 	r1 = *(u64*)(r6 + 0);				\
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
index 4d77407a0a79..5363093a4a5c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
@@ -17,8 +17,8 @@ __naked void fill_of_different_pointers_ldx(void)
 	r2 = r10;					\
 	r2 += %[__imm_0];				\
 	*(u64*)(r6 + 0) = r2;				\
-l0_%=:	if r1 != 0 goto l1_%=;				\
-	*(u64*)(r6 + 0) = r1;				\
+	goto l1_%=;					\
+l0_%=:	*(u64*)(r6 + 0) = r1;				\
 l1_%=:	r1 = *(u64*)(r6 + 0);				\
 	r1 = *(u64*)(r1 + %[sample_period]);		\
 	r0 = 0;						\
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index f9d71e01ddf4..bb2b9e8c71df 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -455,8 +455,47 @@
 	BPF_EXIT_INSN(),
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
-	.errstr = "recursive call",
+	.errstr = "calls itself",
+	.result = REJECT,
+},
+{
+	"calls: a timer callback calls the main prog",
+	.insns = {
+	/* bpf_timer_set_callback(bpf_map_lookup_elem(&timer_map, &(int){0}), cb); */
+	BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+	BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+	BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+	BPF_LD_MAP_FD(BPF_REG_1, 0),
+	BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
+	BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4),
+	BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+	BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, BPF_REG_2, BPF_PSEUDO_FUNC, 0, 4),
+	BPF_RAW_INSN(0, 0, 0, 0, 0),
+	BPF_EMIT_CALL(BPF_FUNC_timer_set_callback),
+	BPF_MOV64_IMM(BPF_REG_0, 0),
+	BPF_EXIT_INSN(),
+	/* cb */
+	BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, -14),
+	BPF_MOV64_IMM(BPF_REG_0, 0),
+	BPF_EXIT_INSN(),
+	},
+	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+	.fixup_map_timer = { 3 },
+	.errstr = "recursive call of the main program",
 	.result = REJECT,
+	/* libbpf makes the main prog global, and then the arguments of the call do not match */
+	.func_info = { { 0, 4 /* main_prog */ }, { 13, 5 /* cb */ } },
+	.func_info_cnt = 2,
+	.btf_strings = "\0int\0ctx\0main_prog\0cb",
+	.btf_types = {
+	/* 1: int            */ BTF_TYPE_INT_ENC(1, BTF_INT_SIGNED, 0, 32, 4),
+	/* 2: void*          */ BTF_PTR_ENC(0),
+	/* 3: int __(void *) */ BTF_FUNC_PROTO_ENC(1, 1),
+				BTF_FUNC_PROTO_ARG_ENC(5, 2),
+	/* 4: main_prog      */ BTF_FUNC_ENC(9, 3),
+	/* 5: cb             */ BTF_FUNC_ENC(19, 3),
+	BTF_END_RAW
+	}
 },
 {
 	"calls: conditional call",
@@ -559,8 +598,7 @@
 	BPF_EXIT_INSN(),
 	},
 	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
-	.errstr = "infinite loop detected",
-	.result = REJECT,
+	.result = ACCEPT,
 },
 {
 	"calls: using r0 returned by callee",
@@ -812,7 +850,7 @@
 	BPF_EXIT_INSN(),
 	},
 	.prog_type = BPF_PROG_TYPE_TRACEPOINT,
-	.errstr = "recursive call",
+	.errstr = "calls itself",
 	.result = REJECT,
 },
 {
-- 
2.55.0

Re: [PATCH bpf-next v4 39/46] selftests/bpf: Adjust tests to new verdicts, messages and order

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] selftests/bpf: ineffective __not_msg() window in
  correlated_regs()
--

commit 7cea750c7f8c80faefbaba10e236762a682e25c2
Author: Alexei Starovoitov <ast@kernel.org>

selftests/bpf: Adjust tests to new verdicts, messages and order

This commit updates existing BPF selftests to match the new verifier
verdicts, log messages, and branch exploration order, and adds tests for
recursive subprogs.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
> index abe2b19110790..d89ffe90c6bc3 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_scev.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
[ ... ]
> @@ -2396,18 +2374,15 @@ __msg_next("  r1=(+ r1 2) / (linear (+ r1 2) 2)")
>  __msg("loop header at 11, widening r0")
>  __msg("loop header at 11, widening r1")
>  __msg("11: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=7,var_off=(0x0; 0x7)) R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=14,var_off=(0x0; 0xe),step=0+2)")
> -/* loop exit */
>  __msg("16: (a5) if r0 < 0x8 goto pc-6")
> -__msg("exiting loop 11")
> -__msg("17: (95) exit")
>  /* second iteration */
>  __msg("loop header at 11, clamping r0")
>  __msg("loop header at 11, clamping r1")
>  /* iteration convergence */
> -__msg("from 16 to 11: safe")
>  __not_msg("{{^}}11:")
> -/* map lookup error path */
> -__msg("from 7 to 17: safe")
> +/* loop exit */
> +__msg("exiting loop 11")
> +__msg("17: (95) exit")
>  __naked void correlated_regs(void)

[Severity: Medium]
As reported by bot+bpf-ci@kernel.org on v2 [1], does moving
__msg("exiting loop 11") right after __not_msg("{{^}}11:") make the
convergence check in correlated_regs() a no-op?

match_negative_msgs() only searches between the preceding positive match
("loop header at 11, clamping r1") and the next one ("exiting loop 11").
Both messages are emitted inside the same do_check_block() call for
instruction 16: flow_drain() delivers the 16 -> 11 backedge (logging the
clamping lines and queueing any non-converged state as pending), and
do_check_block() immediately delivers the fall-through edge to instruction
17 (logging "exiting loop 11").

Only in the next do_check() iteration does bpf_flow_next() dequeue pending
states, so if loop 11 failed to converge, instruction 11 would be logged
after "exiting loop 11" and before "17: (95) exit", outside the window
checked by __not_msg("{{^}}11:").  Placing __not_msg("{{^}}11:") after
__msg("exiting loop 11") (or matching __msg("from 16 to 11: within"))
would cover the span where a non-converged second iteration is printed.

[1] https://lore.kernel.org/r/53c16606b21ca4743d6f2dcc8f31619c9e070d7ca0a729fcb0ae723bb1c04ee5@mail.kernel.org

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=39

[PATCH bpf-next v4 40/46] selftests/bpf: Test states that are not joined

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

  ret = lookup(&p);      /* ret = -ENOENT and p is not set, or */
  if (ret < 0)           /* ret = 0 and p is set               */
          goto drop;
  use(p);

When the compiler inlines lookup(), the two states meet in front of the
jump. A verifier that follows paths keeps them apart and loads the prog.
Their join, ret = [-2, 0], gets to use(p) without p. Thus the verifier does
not join states when a check ahead can depend on the difference. Each test
has one such difference, in the form that a real prog has.

  [J] "bpf: Add join of registers and stack slots"
  [K] "bpf: Find the registers that hold a number or a pointer to arena"
  [A] "bpf: Find the registers and stack slots that later checks depend on"
  [N] "bpf: Decide which states are not joined"
  [M] "bpf: Merge the state that arrives into the set at an insn"
  [F] "bpf: Verify programs as a fixpoint over the control flow graph"

  verifier_..     the two states differ in                   as in
  --------------  -----------------------------------------  --------------
  jumps_ahead.c   [N] a number that a jump ahead tests, the  Cilium,
                  ranges have a gap: -155 and 0              profiler,
                  [N] two such numbers that go together      test_seg6_loop
                  [N] a register that only one state sets
                  [N] 0 or not 0, in one of two slots        xdp_synproxy
                  [N] a flag, 0 and 1, that a jump in this
                  block or the next one tests
                  [M] a register that only a loop sets
  ptr_off.c       [A] a constant that is added to a pointer
                  [J] how much of the packet is checked
                  [K] the offset of one of two fields behind
                  bpf_core_cast(), bpf_cast_to_kern_ctx()
                  [K] 20 or 40 bytes of a packet whose       test_cls_
                  context was in a struct                    redirect
  var_off.c       [A] a constant offset in the stack
  mem_size_reg.c  [A] a size that a helper or a kfunc must
                  know

States that stay multiply. The verifier must join what no check can tell
apart: -134 and -135 ([N]), NULL or not in an array that a loop reads
(bpfjailer), an offset in a map value that has only numbers ([K]). 20 flags
and 12 offsets give 2^20 and 2^12 states: the rules apply only while less
than half of each limit on states is used ([N]). 4 calls that each return
one of 33 numbers give 33^4: bpf_flow_exit() widens an exit state that was
joined 16 times.

[F]: maybe_fork_scalars() makes -1 and 0 of [-1, 0]. The state with 0 must
not go back to an insn where [-1, 0] covers it (verifier_bounds.c). 1 and 2
are joined at exit: a prog that can return 2 must not attach to ingress
(cgroup_skb_egress_ret.c).

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../bpf/prog_tests/cgroup_skb_egress_ret.c    |  23 +
 .../selftests/bpf/prog_tests/verifier.c       |   4 +
 .../bpf/progs/cgroup_skb_egress_ret.c         |  12 +
 .../selftests/bpf/progs/verifier_bounds.c     | 108 +++
 .../bpf/progs/verifier_jumps_ahead.c          | 670 ++++++++++++++++++
 .../bpf/progs/verifier_mem_size_reg.c         |  48 ++
 .../selftests/bpf/progs/verifier_ptr_off.c    | 279 ++++++++
 .../selftests/bpf/progs/verifier_var_off.c    | 203 ++++++
 8 files changed, 1347 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_ptr_off.c

diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c b/tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
new file mode 100644
index 000000000000..e261caac35e6
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/cgroup_skb_egress_ret.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "cgroup_helpers.h"
+#include "cgroup_skb_egress_ret.skel.h"
+
+void test_cgroup_skb_egress_ret(void)
+{
+	struct cgroup_skb_egress_ret *skel;
+	int cg, fd;
+
+	cg = test__join_cgroup("/cgroup_skb_egress_ret");
+	if (!ASSERT_OK_FD(cg, "join_cgroup"))
+		return;
+	skel = cgroup_skb_egress_ret__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "open_and_load"))
+		goto out;
+	fd = bpf_program__fd(skel->progs.one_or_two);
+	ASSERT_EQ(bpf_prog_attach(fd, cg, BPF_CGROUP_INET_INGRESS, 0), -EINVAL, "ingress");
+	ASSERT_OK(bpf_prog_attach(fd, cg, BPF_CGROUP_INET_EGRESS, 0), "egress");
+	cgroup_skb_egress_ret__destroy(skel);
+out:
+	close(cg);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index d41339ef00d8..2f34a34e3d54 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -69,6 +69,7 @@
 #include "verifier_iterating_callbacks.skel.h"
 #include "verifier_jeq_infer_not_null.skel.h"
 #include "verifier_jit_convergence.skel.h"
+#include "verifier_jumps_ahead.skel.h"
 #include "verifier_kfunc_packet_access.skel.h"
 #include "verifier_kfunc_uninit.skel.h"
 #include "verifier_kfunc_uninit_multi.skel.h"
@@ -107,6 +108,7 @@
 #include "verifier_precision.skel.h"
 #include "verifier_prevent_map_lookup.skel.h"
 #include "verifier_private_stack.skel.h"
+#include "verifier_ptr_off.skel.h"
 #include "verifier_ptr_to_buf.skel.h"
 #include "verifier_raw_stack.skel.h"
 #include "verifier_raw_tp_writable.skel.h"
@@ -267,6 +269,7 @@ void test_verifier_int_ptr(void)              { RUN(verifier_int_ptr); }
 void test_verifier_iterating_callbacks(void)  { RUN(verifier_iterating_callbacks); }
 void test_verifier_jeq_infer_not_null(void)   { RUN(verifier_jeq_infer_not_null); }
 void test_verifier_jit_convergence(void)      { RUN(verifier_jit_convergence); }
+void test_verifier_jumps_ahead(void)          { RUN(verifier_jumps_ahead); }
 void test_verifier_kfunc_packet_access(void)  { RUN_TESTS(verifier_kfunc_packet_access); }
 void test_verifier_kfunc_uninit(void)         { RUN_TESTS(verifier_kfunc_uninit); }
 void test_verifier_kfunc_uninit_multi(void)   { RUN_TESTS(verifier_kfunc_uninit_multi); }
@@ -317,6 +320,7 @@ void test_verifier_percpu_addr(void)          { RUN(verifier_percpu_addr); }
 void test_verifier_precision(void)            { RUN(verifier_precision); }
 void test_verifier_prevent_map_lookup(void)   { RUN(verifier_prevent_map_lookup); }
 void test_verifier_private_stack(void)        { RUN(verifier_private_stack); }
+void test_verifier_ptr_off(void)              { RUN(verifier_ptr_off); }
 void test_verifier_ptr_to_buf(void)           { RUN(verifier_ptr_to_buf); }
 void test_verifier_raw_stack(void)            { RUN(verifier_raw_stack); }
 void test_verifier_raw_tp_writable(void)      { RUN(verifier_raw_tp_writable); }
diff --git a/tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c b/tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
new file mode 100644
index 000000000000..107f61d77538
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/cgroup_skb_egress_ret.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+/* Only a prog at egress returns 2. At exit the verifier has 1 and 2 in one state. */
+SEC("cgroup_skb/egress")
+int one_or_two(struct __sk_buff *skb)
+{
+	return skb->len > 100 ? 2 : 1;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c
index 6fede778971c..28d48e5fbc4f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c
@@ -2159,6 +2159,74 @@ l0_%=:	r0 = 0;						\
 	: __clobber_all);
 }
 
+/*
+ * -1 and 0 meet at the insn that forks. The state with [-1, 0] is kept there. The
+ * pushed state is within it, so it's not there that it is pushed to.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: AND where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_at_join_rejects_oob(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u64*)(r10 - 8) = r1;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map_hash_8b] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l0_%=;				\
+	r9 = r0;					\
+	r1 = *(u64*)(r9 + 0);				\
+	r6 = -1;					\
+	if r1 == 0 goto l1_%=;				\
+	r6 = 0;						\
+l1_%=:	r6 &= 8;					\
+	/* r6 is 8 (current) or 0 (pushed) */		\
+	r6 ^= 8;					\
+	/* pushed path: r6 = 8, OOB for value_size=8 */	\
+	r9 += r6;					\
+	r0 = *(u8*)(r9 + 0);				\
+l0_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: 32-bit AND of a 32-bit range where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and32_scalar32_fork_at_join_rejects_oob(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u64*)(r10 - 8) = r1;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map_hash_8b] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l0_%=;				\
+	r9 = r0;					\
+	r1 = *(u64*)(r9 + 0);				\
+	w6 = -1;					\
+	if r1 == 0 goto l1_%=;				\
+	r6 = 0;						\
+l1_%=:	w6 &= 8;					\
+	/* r6 is 8 (current) or 0 (pushed) */		\
+	r6 ^= 8;					\
+	/* pushed path: r6 = 8, OOB for value_size=8 */	\
+	r9 += r6;					\
+	r0 = *(u8*)(r9 + 0);				\
+l0_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
 /*
  * The insn that forks is the head of a loop that scalar evolution counted. The state with
  * [-1, 0] is kept there, and the state with 0 that is split off is within it.
@@ -2200,6 +2268,46 @@ l0_%=:	r0 = 0;						\
 	: __clobber_all);
 }
 
+#define FORK						\
+	"r6 = *(u64*)(r9 + 0);"				\
+	"r6 s>>= 63;"					\
+	"r6 &= 4;"					\
+	"*(u64*)(r9 + 0) = r6;"
+#define FORK4 FORK FORK FORK FORK
+
+/*
+ * The pushed states of one block that don't differ in what is live are one state: 16 forks
+ * are not 2^16 states.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: 16 forks in a block")
+__success __retval(0)
+__log_level(4)
+__msg("processed {{[0-9][0-9]?[0-9]?}} insns")
+__naked void and_scalar_16_forks_in_a_block(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64*)(r10 - 8) = r1;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"r1 = %[map_hash_8b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto l0_%=;"
+	"r9 = r0;"
+	FORK4 FORK4 FORK4 FORK4
+"l0_%=:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
+#undef FORK4
+#undef FORK
+
 /*
  * Last jump can be detected as always taken because the intersection of R5 and
  * R7 32bit tnums produces a constant that isn't within R7's s32 bounds.
diff --git a/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c b/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
new file mode 100644
index 000000000000..66d8a7c4a6a0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
@@ -0,0 +1,670 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/*
+ * A join must not make up what neither state has where a jump ahead can see it:
+ *	ret = lookup(&p);
+ *	if (ret < 0)
+ *		goto drop;
+ *	use(p);
+ * When lookup() is inlined (ret = -ENOENT, p is not set) and (ret = 0, p is set) meet in front of
+ * the jump.
+ */
+
+SEC("socket")
+__description("jumps ahead: a register is set when ret is 0")
+__success __retval(0)
+__naked void ret_and_register(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = -2;"
+		"if r0 == 0 goto 1f;"
+		"r8 = 0;"
+		"w6 = 0;"
+	"1:"
+		"w7 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r0 = r8;"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("jumps ahead: a register is not set and ret doesn't tell")
+__failure __msg("R8 !read_ok")
+__naked void ret_does_not_tell(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = 1;"
+		"if r0 == 0 goto 1f;"
+		"r8 = 0;"
+		"w6 = 0;"
+	"1:"
+		"w7 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r0 = r8;"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ * As in bpf_host.o of cilium:
+ *	err = helper();
+ *	ret = err < 0 ? -155 : 0;
+ *	if (err < 0)
+ *		goto out;
+ *	...
+ * out:
+ *	if (ret < 0)
+ *		goto drop;
+ * The states differ in what two jumps look at, and in nothing else.
+ * (err < 0, ret = 0) is neither of them.
+ */
+SEC("socket")
+__description("jumps ahead: two numbers that go together")
+__success
+__naked void two_numbers(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = 0;"
+		"if w0 s> -1 goto 1f;"
+		"w6 = -155;"
+	"1:"
+		"w7 = 0;"
+		"if w0 s< 0 goto 2f;"
+		"r0 = 0;"
+		"exit;"
+	"2:"
+		"if w6 s< 0 goto 3f;"
+		"r0 = r8;"
+	"3:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/* -134 and -135 go the same way */
+SEC("socket")
+__description("jumps ahead: states that go the same way are joined")
+__success __log_level(2)
+__msg("from 4 to 7: joined")
+__msg("7: R6=scalar({{.*}}smin32=-135,smax32=-134{{.*}}) R10=fp0")
+__msg("from 6 to 7: R6=0 R8=0 R10=fp0")
+__naked void same_way(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w6 = -134;"
+		"if r0 == 1 goto 1f;"
+		"w6 = -135;"
+		"if r0 == 2 goto 1f;"
+		"r8 = 0;"
+		"w6 = 0;"
+	"1:"
+		"w7 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r0 = r8;"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ *	ret = idx < 8 ? 0 : -EINVAL;
+ *	if (ret < 0)
+ *		goto drop;
+ *	use(buf[idx]);
+ * 0 and -EINVAL leave a gap, so the states are not joined, and idx keeps its range.
+ */
+SEC("socket")
+__description("jumps ahead: ret vouches for the range of another number")
+__success
+__naked void ret_and_range(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"call %[bpf_get_prandom_u32];"
+		"r7 = r0;"
+		"w6 = 0;"
+		"if r7 < 8 goto 1f;"
+		"w6 = -22;"
+	"1:"
+		"w8 = 0;"
+		"if w6 s< 0 goto 2f;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r1 += r7;"
+		"r0 = *(u8 *)(r1 + 0);"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ *	ok = i < 8 && x == 2;
+ *	if (x == 7)
+ *		n++;
+ *	if (ok)
+ *		use(buf[i]);
+ * A compiler keeps ok in a register: "w6 &= 1; if w6 == 0". (ok = 0, i >= 8), (ok = 1, i < 8) and
+ * (ok = 0, i < 8) meet in front of "if (x == 7)". 0 and 1 leave no gap, and i < 8 comes with
+ * either. They are not joined when the jump that tells 0 from 1 ends this basic block or the next.
+ */
+#define FLAG_SET(max, no, yes)			\
+	"r1 = 0;"				\
+	"*(u64 *)(r10 - 8) = r1;"		\
+	"call %[bpf_get_prandom_u32];"		\
+	"r7 = r0;"				\
+	"call %[bpf_get_prandom_u32];"		\
+	"w6 = " #no ";"				\
+	"if r7 > " #max " goto +3;"		\
+	"w6 = " #yes ";"			\
+	"if r0 == 2 goto +1;"			\
+	"w6 = " #no ";"
+#define FLAG_USE(unless)				\
+	"if " #unless " goto +4;"		\
+	"r1 = r10;"				\
+	"r1 += -8;"				\
+	"r1 += r7;"				\
+	"r0 = *(u8 *)(r1 + 0);"
+#define FLAG_PROG(name, max, no, yes, rest)	\
+__naked void name(void)				\
+{						\
+	asm volatile (				\
+	FLAG_SET(max, no, yes)			\
+	rest					\
+	"exit;"					\
+	:: __imm(bpf_get_prandom_u32) : __clobber_all); \
+}
+
+SEC("socket")
+__description("jumps ahead: a flag, another jump, the test of the flag")
+__success
+FLAG_PROG(flag_next_block, 7, 0, 1, "if r0 != 7 goto +1; r0 = 0; w6 &= 1;" FLAG_USE(w6 == 0))
+
+SEC("socket")
+__description("jumps ahead: a flag that says too much")
+__failure __msg("invalid variable-offset read from stack R1")
+FLAG_PROG(flag_next_block_bad, 8, 0, 1, "if r0 != 7 goto +1; r0 = 0; w6 &= 1;" FLAG_USE(w6 == 0))
+
+/* "f = i < 8 ? in[i] : 0; ... if (f) use(buf[i]);" A compiler makes a flag of "f == 0" */
+SEC("socket")
+__description("jumps ahead: a flag that is 1 for no")
+__success
+FLAG_PROG(flag_for_no, 7, 1, 0, "if r0 != 7 goto +1; r0 = 0; w6 &= 1;" FLAG_USE(w6 != 0))
+
+SEC("socket")
+__description("jumps ahead: a flag is tested twice")
+__success
+FLAG_PROG(flag_twice, 7, 0, 1, FLAG_USE(w6 == 0) "r0 = 0;" FLAG_USE(w6 == 0))
+
+SEC("socket")
+__description("jumps ahead: the other value of a flag that is tested twice")
+__failure __msg("math between fp pointer and register with unbounded min value")
+FLAG_PROG(flag_twice_bad, 7, 0, 1, FLAG_USE(w6 == 0) "w6 ^= 1;" FLAG_USE(w6 == 0))
+
+SEC("socket")
+__description("jumps ahead: a flag is spilled, filled, copied and extended")
+__success
+FLAG_PROG(flag_moves, 7, 0, 1, "*(u64 *)(r10 - 16) = r6; r1 = *(u64 *)(r10 - 16);"
+			       "r2 = r1; r2 <<= 32; r2 >>= 32;" FLAG_USE(r2 == 0))
+
+/* if and else: two ends of basic blocks on every way to the test */
+SEC("socket")
+__description("jumps ahead: the test of a flag is too far")
+__success __log_level(2)
+__msg("from 8 to 10: joined")
+FLAG_PROG(flag_too_far, 7, 0, 1, "if r0 != 7 goto +2; r0 = 0; goto +1; r0 = 1;"
+				 "if w6 == 0 goto +1; r0 = 2;")
+
+SEC("socket")
+__description("jumps ahead: the jump tests another register than the flag")
+__success __log_level(2)
+__msg("from 8 to 10: joined")
+FLAG_PROG(flag_not_tested, 7, 0, 1, "if r0 == 1 goto +1; r0 = r6;")
+
+/* the first values of a counter */
+SEC("socket")
+__description("jumps ahead: 1 and 2 are no flag")
+__success __log_level(2)
+__msg("from 8 to 10: joined")
+FLAG_PROG(flag_1_and_2, 7, 1, 2, "if w6 == 1 goto +1; r0 = 0;")
+
+/*
+ * A register that is set in one state and not in the other: these states are not joined,
+ * whatever it is that tells them apart later.
+ */
+SEC("socket")
+__description("set or not: a register is set when a flag is 1, flag + 4 is tested")
+__success
+__naked void flag_add(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r6 = 0;"
+	"if r0 == 0 goto 1f;"
+	"r8 = 0;"
+	"r6 = 1;"
+"1:"
+	"r1 = r6;"
+	"r1 += 4;"
+	"if r1 != 5 goto 2f;"
+	"r0 = r8;"
+	"exit;"
+"2:"
+	"r0 = 0;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* The first iteration sets r7 and the flag. The states meet at the head of the loop. */
+SEC("socket")
+__description("set or not: a loop sets a register and a flag")
+__success
+__naked void set_in_a_loop(void)
+{
+	asm volatile (
+	"r6 = 0;"
+"1:"
+	".8byte %[may_goto];"
+	"if r6 == 0 goto 2f;"
+	"r8 = r7;"
+"2:"
+	"call %[bpf_get_prandom_u32];"
+	"r7 = r0;"
+	"r6 = 1;"
+	"goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 6, 0))
+	: __clobber_all);
+}
+
+/*
+ * 20 flags, 0 or 5, all are looked at later: 2^20 ways, and 0 and 5 leave a gap.
+ * States are kept apart while there is room. After that they are joined.
+ */
+#define PICK_FLAG(slot)					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r1 = 0;"					\
+	"if r0 == 0 goto +1;"				\
+	"r1 = 5;"					\
+	"*(u64*)(r10 - " #slot ") = r1;"
+#define USE_FLAG(slot)					\
+	"r1 = *(u64*)(r10 - " #slot ");"		\
+	"if r1 == 0 goto +1;"				\
+	"r6 += 1;"
+
+SEC("socket")
+__description("jumps ahead: 20 flags that leave a gap")
+__success
+__naked void many_flags_gap(void)
+{
+	asm volatile (
+	"r6 = 0;"
+	PICK_FLAG(8) PICK_FLAG(16) PICK_FLAG(24) PICK_FLAG(32) PICK_FLAG(40) PICK_FLAG(48)
+	PICK_FLAG(56) PICK_FLAG(64) PICK_FLAG(72) PICK_FLAG(80) PICK_FLAG(88) PICK_FLAG(96)
+	PICK_FLAG(104) PICK_FLAG(112) PICK_FLAG(120) PICK_FLAG(128) PICK_FLAG(136) PICK_FLAG(144)
+	PICK_FLAG(152) PICK_FLAG(160)
+	USE_FLAG(8) USE_FLAG(16) USE_FLAG(24) USE_FLAG(32) USE_FLAG(40) USE_FLAG(48)
+	USE_FLAG(56) USE_FLAG(64) USE_FLAG(72) USE_FLAG(80) USE_FLAG(88) USE_FLAG(96)
+	USE_FLAG(104) USE_FLAG(112) USE_FLAG(120) USE_FLAG(128) USE_FLAG(136) USE_FLAG(144)
+	USE_FLAG(152) USE_FLAG(160)
+	"r0 = r6;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* 33 numbers, each with an exit of its own */
+static __naked __noinline __used int ret_exits(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = r0;"
+	"if r1 != 0 goto +2; r0 = 0; exit;"
+	"if r1 != 1 goto +2; r0 = 3; exit;"
+	"if r1 != 2 goto +2; r0 = 6; exit;"
+	"if r1 != 3 goto +2; r0 = 9; exit;"
+	"if r1 != 4 goto +2; r0 = 12; exit;"
+	"if r1 != 5 goto +2; r0 = 15; exit;"
+	"if r1 != 6 goto +2; r0 = 18; exit;"
+	"if r1 != 7 goto +2; r0 = 21; exit;"
+	"if r1 != 8 goto +2; r0 = 24; exit;"
+	"if r1 != 9 goto +2; r0 = 27; exit;"
+	"if r1 != 10 goto +2; r0 = 30; exit;"
+	"if r1 != 11 goto +2; r0 = 33; exit;"
+	"if r1 != 12 goto +2; r0 = 36; exit;"
+	"if r1 != 13 goto +2; r0 = 39; exit;"
+	"if r1 != 14 goto +2; r0 = 42; exit;"
+	"if r1 != 15 goto +2; r0 = 45; exit;"
+	"if r1 != 16 goto +2; r0 = 48; exit;"
+	"if r1 != 17 goto +2; r0 = 51; exit;"
+	"if r1 != 18 goto +2; r0 = 54; exit;"
+	"if r1 != 19 goto +2; r0 = 57; exit;"
+	"if r1 != 20 goto +2; r0 = 60; exit;"
+	"if r1 != 21 goto +2; r0 = 63; exit;"
+	"if r1 != 22 goto +2; r0 = 66; exit;"
+	"if r1 != 23 goto +2; r0 = 69; exit;"
+	"if r1 != 24 goto +2; r0 = 72; exit;"
+	"if r1 != 25 goto +2; r0 = 75; exit;"
+	"if r1 != 26 goto +2; r0 = 78; exit;"
+	"if r1 != 27 goto +2; r0 = 81; exit;"
+	"if r1 != 28 goto +2; r0 = 84; exit;"
+	"if r1 != 29 goto +2; r0 = 87; exit;"
+	"if r1 != 30 goto +2; r0 = 90; exit;"
+	"if r1 != 31 goto +2; r0 = 93; exit;"
+	"r0 = 99;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* 33^4 ways: not all of them are looked at */
+SEC("socket")
+__description("jumps ahead: one of 33 numbers out of a function, 4 times")
+__success
+__naked void rets_33_4_times(void)
+{
+	asm volatile (
+	"call ret_exits; r6 = r0;"
+	"call ret_exits; r7 = r0;"
+	"call ret_exits; r8 = r0;"
+	"call ret_exits; r9 = r0;"
+	"r0 = 0;"
+	"if r6 != 3 goto +1; r0 += 1;"
+	"if r7 != 6 goto +1; r0 += 1;"
+	"if r8 != 9 goto +1; r0 += 1;"
+	"if r9 != 12 goto +1; r0 += 1;"
+	"exit;"
+	::: __clobber_all);
+}
+
+#define PICK_ANY(slot)					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r0 <<= 32;"					\
+	"r7 = r0;"					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r0 |= r7;"					\
+	"*(u64*)(r10 - " #slot ") = r0;"
+
+/*
+ * As in bpfj_jailer_free() of bpfjailer:
+ *	for (i = 0; i < 8; i++) {
+ *		p = a[i];
+ *		if (!p)
+ *			continue;
+ *		put(p);
+ *	}
+ * The jump finds out about a[i] too, which nothing looks at again. All of a[] is live while
+ * the loop goes on, because it is read through a pointer that moves. The states have to be
+ * joined behind the jump, or there are 2^8 of them. They are, because a slot that is filled
+ * through another register than r10 doesn't count as one that a jump ahead looks at.
+ */
+SEC("socket")
+__description("0 or not: an array on the stack that a loop goes through")
+__success __log_level(4)
+__msg("max_states_per_insn {{[0-9]}} total_states")
+__naked void array_of_nulls(void)
+{
+	asm volatile (
+	"r6 = 0;"
+	PICK_ANY(8) PICK_ANY(16) PICK_ANY(24) PICK_ANY(32)
+	PICK_ANY(40) PICK_ANY(48) PICK_ANY(56) PICK_ANY(64)
+	"r8 = r10;"
+	"r8 += -64;"
+	"r7 = 0;"
+"1:"
+	"r1 = *(u64*)(r8 + 0);"
+	"if r1 == 0 goto 2f;"
+	"r6 += 1;"
+"2:"
+	"r8 += 8;"
+	"r7 += 1;"
+	"if r7 < 8 goto 1b;"
+	"r0 = r6;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * As in xdp_synproxy_kern.c: hdr->ipv4 or hdr->ipv6 is there, the other one is 0.
+ *	if (hdr->ipv4) ... else if (hdr->ipv6) ...
+ * (0, x) and (y, 0) are not joined: (?, ?) has both, and r9 is not set.
+ */
+SEC("socket")
+__description("0 or not: one of two things is there")
+__success
+__naked void one_of_two(void)
+{
+	asm volatile (
+	PICK_ANY(8)
+	"r6 = r0;"
+	"r7 = 0;"
+	"call %[bpf_get_prandom_u32];"
+	"if r0 == 0 goto 1f;"
+	"r7 = r6;"
+	"r6 = 0;"
+"1:"
+	"r0 = 0;"
+	"if r6 == 0 goto 2f;"
+	"if r7 == 0 goto 2f;"
+	"r0 = r9;"
+"2:"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * As in bpf_sock.o of cilium:
+ *	err = xlate();
+ *	if (err == -EHOSTUNREACH || err == -ENOMEM) {
+ *		bpf_set_retval(err);
+ *		return 0;
+ *	}
+ *	return 1;
+ * A compiler computes the condition where err is set. 0 and 1 touch, and it's not a jump that
+ * looks at err.
+ */
+SEC("cgroup/connect4")
+__description("jumps ahead: a flag goes together with what bpf_set_retval() takes")
+__success
+__naked void flag_and_retval(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"w8 = 0;"
+		"w1 = 1;"
+		"if r0 == 0 goto 1f;"
+		"w8 = 1;"
+		"w1 = -113;"
+	"1:"
+		"w0 = 1;"
+		"if w8 == 0 goto 2f;"
+		"call %[bpf_set_retval];"
+		"w0 = 0;"
+	"2:"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32), __imm(bpf_set_retval)
+		: __clobber_all
+	);
+}
+
+char buf[8];
+
+/* As in test_seg6_loop.c without 32-bit insns: the copy is tested, the original is used */
+SEC("socket")
+__description("jumps ahead: the jump looks at a copy that two shifts extend")
+__success
+__naked void extended_copy(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"r8 = 0;"
+		"if r0 == 0 goto 1f;"
+		"r8 = 5;"
+	"1:"
+		"r7 = r8;"
+		"r7 <<= 32;"
+		"r7 >>= 32;"
+		"if r7 == 0 goto 2f;"
+		"r8 += -5;"
+		"r1 = %[buf] ll;"
+		"r1 += r8;"
+		"r0 = *(u8 *)(r1 + 0);"
+		"exit;"
+	"2:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32), __imm_addr(buf)
+		: __clobber_all
+	);
+}
+
+static __naked __noinline __used __u64 index_or_minus_1(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"r1 = r0;"
+		"r0 = 0xffffffff ll;"
+		"if r1 > 7 goto 1f;"
+		"r0 = r1;"
+	"1:"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+/*
+ * As in profiler.inc.h without 32-bit insns:
+ *	int index = find();
+ *	if (index == -1)
+ *		return 0;
+ *	use(array[index]);
+ * The numbers meet at the exit of find(). -1 doesn't fit in the jump, it is in a register.
+ */
+SEC("socket")
+__description("jumps ahead: the caller looks at what the function returns")
+__success
+__naked void caller_looks(void)
+{
+	asm volatile (
+		"call index_or_minus_1;"
+		"r0 <<= 32;"
+		"r1 = r0;"
+		"r1 >>= 32;"
+		"r2 = 0xffffffff ll;"
+		"if r1 == r2 goto 1f;"
+		"r0 s>>= 32;"
+		"r1 = %[buf] ll;"
+		"r1 += r0;"
+		"r0 = *(u8 *)(r1 + 0);"
+		"exit;"
+	"1:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(buf)
+		: __clobber_all
+	);
+}
+
+/*
+ * A state that arrives is compared with the states that it may be related to. Its shape tells:
+ * the same pointers in the same places at the same constant offsets. These states are related
+ * and don't look alike. They have to find each other.
+ */
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, char[48]);
+} shape_map SEC(".maps");
+
+/*
+ * 16 states with R9 at 16 places of the stack get to "1:". With that many states at an insn
+ * their shapes are looked at. Then comes one more, which @last makes of the 16th. The two are
+ * related and don't have the same shape.
+ */
+#define SHAPE_16_PLACES	"8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128"
+#define SHAPE_MANY(first, last, use)					\
+	asm volatile (							\
+		first							\
+		"r1 = 0;"						\
+		".irp off," SHAPE_16_PLACES ";"				\
+		"*(u64 *)(r10 - \\off) = r1;"				\
+		".endr;"						\
+		"call %[bpf_get_prandom_u32];"				\
+		"r6 = r0;"						\
+		"call %[bpf_get_prandom_u32];"				\
+		"r7 = r0;"						\
+		"r7 &= 1;"						\
+		".irp off," SHAPE_16_PLACES ";"				\
+		"r9 = r10;"						\
+		"r9 += -\\off;"						\
+		"if r6 == \\off goto 1f;"				\
+		".endr;"						\
+		last							\
+	"1:"								\
+		use							\
+		"r0 = *(u64 *)(r9 + 0);"				\
+	"2:"								\
+		"r0 = 0;"						\
+		"exit;"							\
+		:							\
+		: __imm(bpf_get_prandom_u32), __imm(bpf_map_lookup_elem),	\
+		  __imm_addr(shape_map),				\
+		  __imm_const(data, offsetof(struct __sk_buff, data)),	\
+		  __imm_const(data_end, offsetof(struct __sk_buff, data_end))	\
+		: __clobber_all						\
+	)
+
+SEC("socket")
+__description("shape: nothing in a register stands for a pointer")
+__success __log_level(2)
+__msg("processed 135 insns")
+__naked void shape_unset(void)
+{
+	SHAPE_MANY("",
+		   "r8 = r10;",
+		   "if r7 < 2 goto +1;"
+		   "r0 = *(u64 *)(r8 - 8);");
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
index aba9510a1c08..5a4a29b30b7a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
@@ -2,11 +2,59 @@
 
 #include <vmlinux.h>
 #include <bpf/bpf_helpers.h>
+#include <bpf/bpf_endian.h>
 #include "bpf_misc.h"
 #include "bpf_kfuncs.h"
 
 char _license[] SEC("license") = "GPL";
 
+/*
+ * A size that has to be a known constant is one of two constants, and a compiler makes one call
+ * of the two. The constants must not be joined on the way to the call.
+ */
+SEC("?tc")
+__description("one of two constants: __szk, the header of IPv4 or of IPv6")
+__success
+int dynptr_slice_two_sizes(struct __sk_buff *skb)
+{
+	char buf[sizeof(struct ipv6hdr)];
+	__u32 sz = sizeof(struct iphdr);
+	struct bpf_dynptr dptr;
+	__u8 *p;
+
+	bpf_dynptr_from_skb(skb, 0, &dptr);
+
+	if (skb->protocol != bpf_htons(0x0800))
+		sz = sizeof(struct ipv6hdr);
+	p = bpf_dynptr_slice(&dptr, 14, buf, sz);
+	if (p)
+		return p[0] >> 4;
+	return 0;
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+SEC("?tc")
+__description("one of two constants: the size of bpf_ringbuf_reserve()")
+__success
+int ringbuf_reserve_two_sizes(struct __sk_buff *skb)
+{
+	__u32 sz = 8;
+	char *p;
+
+	if (skb->len > 100)
+		sz = 16;
+	p = bpf_ringbuf_reserve(&ringbuf, sz, 0);
+	if (!p)
+		return 0;
+	p[7] = 1;
+	bpf_ringbuf_discard(p, 0);
+	return 0;
+}
+
 /* The __szk size of a kfunc memory/size pair when the nullable buffer is passed as NULL */
 SEC("?tc")
 __success
diff --git a/tools/testing/selftests/bpf/progs/verifier_ptr_off.c b/tools/testing/selftests/bpf/progs/verifier_ptr_off.c
new file mode 100644
index 000000000000..260f7ea50b9f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_ptr_off.c
@@ -0,0 +1,279 @@
+// SPDX-License-Identifier: GPL-2.0
+/* One of two constants is added to a pointer. A compiler makes one add and one load of it. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_misc.h"
+#include "bpf_kfuncs.h"
+
+struct with_timer {
+	int before[4];
+	struct bpf_timer timer;
+	int after[4];
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct with_timer);
+} timers SEC(".maps");
+
+__u64 in_bss[8];
+
+#define __imm_pkt								\
+	__imm_const(data, offsetof(struct __sk_buff, data)),			\
+	__imm_const(data_end, offsetof(struct __sk_buff, data_end))
+
+/*
+ *	len = v6 ? 40 : 20;
+ *	if (data + len > data_end) return 0;
+ *	if (v6) use(data[39]);
+ * 1 and 2 touch, so nothing else keeps the states apart.
+ */
+SEC("tc")
+__description("offset of a pointer: the length of a header goes with its kind")
+__success __retval(0)
+__naked void pkt_len_and_kind(void)
+{
+	asm volatile (
+	"r6 = r1;"
+	"call %[bpf_get_prandom_u32];"
+	"r8 = 1;"
+	"r9 = 20;"
+	"if r0 == 0 goto 1f;"
+	"r8 = 2;"
+	"r9 = 40;"
+"1:	r1 = *(u32 *)(r6 + %[data]);"
+	"r3 = *(u32 *)(r6 + %[data_end]);"
+	"r2 = r1;"
+	"r2 += r9;"
+	"if r2 > r3 goto 2f;"
+	"if r8 != 2 goto 2f;"
+	"r0 = *(u8 *)(r1 + 39);"
+"2:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+/* The same with c->skb->data: the context comes out of a struct on the stack */
+SEC("tc")
+__description("offset of a pointer: the packet of a context that was in a struct")
+__success __retval(0)
+__naked void pkt_of_ctx_in_struct(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = r1;"
+	"r7 = r10;"
+	"r7 += -8;"
+	"call %[bpf_get_prandom_u32];"
+	"r8 = 1;"
+	"r9 = 20;"
+	"if r0 == 0 goto 1f;"
+	"r8 = 2;"
+	"r9 = 40;"
+"1:	r6 = *(u64 *)(r7 + 0);"
+	"r1 = *(u32 *)(r6 + %[data]);"
+	"r3 = *(u32 *)(r6 + %[data_end]);"
+	"r2 = r1;"
+	"r2 += r9;"
+	"if r2 > r3 goto 2f;"
+	"if r8 != 2 goto 2f;"
+	"r0 = *(u8 *)(r1 + 39);"
+"2:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+/* data - 1 or data + 30000. A range of the two plus 1 would vouch for 30001 bytes behind data. */
+SEC("tc")
+__description("offset of a pointer: one of two places, none of them in the packet")
+__failure __msg("invalid access to packet, off=30000 size=1, R1(id=0,off=30000,r=0)")
+__naked void pkt_two_places(void)
+{
+	asm volatile (
+	"r6 = r1;"
+	"call %[bpf_get_prandom_u32];"
+	"r1 = *(u32 *)(r6 + %[data]);"
+	"r3 = *(u32 *)(r6 + %[data_end]);"
+	"r2 = r1;"
+	"if r0 == 0 goto 1f;"
+	"r2 += 30000;"
+	"goto 2f;"
+"1:	r2 += -1;"
+"2:	r2 += 1;"
+	"if r2 > r3 goto 3f;"
+	"r0 = *(u8 *)(r1 + 30000);"
+"3:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+/* A range of offsets would go over the timer */
+SEC("socket")
+__description("offset of a pointer: in front of a timer or behind it")
+__success __retval(0)
+__naked void around_timer(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u32 *)(r10 - 4) = r1;"
+	"r2 = r10;"
+	"r2 += -4;"
+	"r1 = %[timers] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 2f;"
+	"r6 = r0;"
+	"call %[bpf_get_prandom_u32];"
+	"r1 = %[before];"
+	"if r0 == 0 goto 1f;"
+	"r1 = %[after];"
+"1:	r6 += r1;"
+	"r0 = *(u32 *)(r6 + 0);"
+"2:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm(bpf_map_lookup_elem), __imm_addr(timers),
+	  __imm_const(before, offsetof(struct with_timer, before[3])),
+	  __imm_const(after, offsetof(struct with_timer, after[0]))
+	: __clobber_all);
+}
+
+/* Nothing looks at the offset but for its ends: the states are joined */
+SEC("socket")
+__description("offset of a pointer: a map value that holds nothing but numbers")
+__success __retval(0) __log_level(2)
+__msg("from 4 to 5: R1=scalar({{.*}}smin32=umin32=8,smax=umax=smax32=umax32=48{{.*}}) R10=fp0")
+__msg("max_states_per_insn 1 ")
+__naked void plain_value(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = 8;"
+	"r2 = 0;"
+	"if r0 == 0 goto 1f;"
+	"r1 = 48;"
+"1:	r6 = %[in_bss] ll;"
+	"r6 += r1;"
+	"r0 = *(u64 *)(r6 + 0);"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_addr(in_bss)
+	: __clobber_all);
+}
+
+const volatile struct {
+	int size;
+	int kind;
+} cfgs[2] = { { 8, 1 }, { 32, 2 } };
+
+/* What is at a known offset of a read-only map is known */
+SEC("tc")
+__description("offset of a pointer: one of two elements of a read-only map")
+__success
+int size_of_two(struct __sk_buff *skb)
+{
+	char buf[32];
+
+	return bpf_skb_load_bytes(skb, 0, buf, (skb->mark ? &cfgs[0] : &cfgs[1])->size);
+}
+
+/* The prototypes of the two kfuncs say void *. The verifier makes a pointer to a struct of it. */
+SEC("tc")
+__description("offset of a pointer: one of two fields behind bpf_core_cast()")
+__success
+int field_behind_core_cast(struct __sk_buff *skb)
+{
+	struct task_struct *task = bpf_core_cast((void *)bpf_get_current_task(), struct task_struct);
+
+	return skb->mark ? task->pid : task->tgid;
+}
+
+SEC("tc")
+__description("offset of a pointer: one of two fields behind bpf_cast_to_kern_ctx()")
+__success
+int field_behind_kern_ctx(struct __sk_buff *ctx)
+{
+	struct sk_buff *skb = bpf_cast_to_kern_ctx(ctx);
+
+	return ctx->mark ? skb->len : skb->data_len;
+}
+
+static __naked __noinline __used void random(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Two states enter the loop, one with each offset. r9 is not what it was when they come around.
+ * Each of them is joined with the state at the head of the loop that it comes from.
+ */
+SEC("tp_btf/sched_switch")
+__description("offset of a pointer: two offsets enter a loop")
+__success
+__naked void two_offsets_in_a_loop(void)
+{
+	asm volatile (
+	"r6 = *(u64 *)(r1 + 8);"
+	"call %[bpf_get_prandom_u32];"
+	"r7 = %[pid];"
+	"if r0 == 0 goto 1f;"
+	"r7 = %[tgid];"
+"1:	r9 = 0;"
+"2:	if r9 == 7 goto 3f;"
+	"r1 = r6;"
+	"r1 += r7;"
+	"r0 = *(u32 *)(r1 + 0);"
+	"call random;"
+	"r9 = r0;"
+	"goto 2b;"
+"3:	r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm_const(pid, offsetof(struct task_struct, pid)),
+	  __imm_const(tgid, offsetof(struct task_struct, tgid))
+	: __clobber_all);
+}
+
+/* The offset is what changes in the loop: it's forgotten, and the loop ends */
+SEC("tc")
+__description("offset of a pointer: the offset changes in a loop")
+__success
+__naked void offset_changes_in_a_loop(void)
+{
+	asm volatile (
+	"r6 = r1;"
+	"r7 = 0;"
+"1:	r1 = *(u32 *)(r6 + %[data]);"
+	"r2 = *(u32 *)(r6 + %[data_end]);"
+	"r7 &= 0xff;"
+	"r1 += r7;"
+	"r3 = r1;"
+	"r3 += 4;"
+	"if r3 > r2 goto 2f;"
+	"r0 = *(u32 *)(r1 + 0);"
+"2:	call %[bpf_get_prandom_u32];"
+	"r7 += 3;"
+	"if r0 != 7 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32), __imm_pkt
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index d92215c5b923..4cfdedca8890 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -12,6 +12,13 @@ struct {
 	__type(value, long long);
 } map_hash_8b SEC(".maps");
 
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, long long);
+} map_array_8b SEC(".maps");
+
 SEC("lwt_in")
 __description("variable-offset ctx access")
 __failure __msg("variable ctx access var_off=(0x0; 0x4)")
@@ -310,6 +317,202 @@ __naked void stack_write_clobbers_spilled_regs(void)
 	: __clobber_all);
 }
 
+/*
+ * if (x) a[0] = p; else a[1] = p; with one store. The paths meet in front of it.
+ * The offset is known on each of them.
+ */
+SEC("tc")
+__description("pointer is spilled at one of two offsets")
+__success __retval(0)
+__naked void ptr_at_one_of_two_offsets(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u32*)(r10 - 4) = r1;				\
+	r2 = r10;					\
+	r2 += -4;					\
+	r1 = %[map_array_8b] ll;			\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l2_%=;				\
+	r6 = r0;					\
+	r1 = 0;						\
+	*(u64*)(r10 - 16) = r1;				\
+	*(u64*)(r10 - 24) = r1;				\
+	call %[bpf_get_prandom_u32];			\
+	r7 = 0;						\
+	r8 = 1;						\
+	if r0 > 100 goto l0_%=;				\
+	r7 = 8;						\
+	r8 = 2;						\
+l0_%=:	r2 = r10;					\
+	r2 += -24;					\
+	r2 += r7;					\
+	*(u64*)(r2 + 0) = r6;				\
+	if r8 != 1 goto l1_%=;				\
+	r3 = *(u64*)(r10 - 24);				\
+	r0 = *(u64*)(r3 + 0);				\
+	exit;						\
+l1_%=:	r3 = *(u64*)(r10 - 16);				\
+	r0 = *(u64*)(r3 + 0);				\
+	exit;						\
+l2_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_array_8b)
+	: __clobber_all);
+}
+
+#define PICK_OFF(slot)					\
+	"call %[bpf_get_prandom_u32];"			\
+	"r1 = 0;"					\
+	"if r0 > 100 goto +1;"				\
+	"r1 = 8;"					\
+	"*(u64*)(r10 - " #slot ") = r1;"
+
+#define USE_OFF(slot)					\
+	"r1 = *(u64*)(r10 - " #slot ");"		\
+	"r2 = r10;"					\
+	"r2 += -16;"					\
+	"r2 += r1;"					\
+	"*(u64*)(r2 + 0) = r0;"
+
+/* 4096 combinations of offsets that are known: not all of them are looked at */
+SEC("tc")
+__description("12 times one of two offsets")
+__success __retval(0)
+__naked void many_offsets(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64*)(r10 - 8) = r1;"
+	"*(u64*)(r10 - 16) = r1;"
+	PICK_OFF(24) PICK_OFF(32) PICK_OFF(40) PICK_OFF(48)
+	PICK_OFF(56) PICK_OFF(64) PICK_OFF(72) PICK_OFF(80)
+	PICK_OFF(88) PICK_OFF(96) PICK_OFF(104) PICK_OFF(112)
+	USE_OFF(24) USE_OFF(32) USE_OFF(40) USE_OFF(48)
+	USE_OFF(56) USE_OFF(64) USE_OFF(72) USE_OFF(80)
+	USE_OFF(88) USE_OFF(96) USE_OFF(104) USE_OFF(112)
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym;
+extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym;
+extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym;
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+
+#define ITER_NEW	"r1 = r10; r1 += -24; w2 = 0; w3 = 100; call %[bpf_iter_num_new];"
+#define ITER_NEXT	"r1 = r10; r1 += -24; call %[bpf_iter_num_next];"
+#define ITER_DESTROY	"r1 = r10; r1 += -24; call %[bpf_iter_num_destroy];"
+#define __imm_iter	__imm(bpf_iter_num_new), __imm(bpf_iter_num_next), \
+			__imm(bpf_iter_num_destroy)
+
+/*
+ *	u8 stack[10], depth = 0;
+ *	loop {
+ *		if (x) { if (depth >= 10) break; stack[depth++] = 1; }
+ *		else { depth--; if (depth >= 10) break; y = stack[depth]; }
+ *	}
+ * depth is in a 32-bit register. 'depth & 255' is tested, and computed anew for the index.
+ */
+#define PUSH_OR_POP					\
+		"r1 = *(u64 *)(r6 + 0);"		\
+		"if r1 == 0 goto 2f;"			\
+		"w7 = w8;"				\
+		"w7 &= 255;"				\
+		"if w7 > 9 goto 9f;"			\
+		"r1 = r8;"				\
+		"r1 &= 255;"				\
+		"r2 = r10;"				\
+		"r2 += -10;"				\
+		"r2 += r1;"				\
+		"*(u8 *)(r2 + 0) = r1;"			\
+		"w8 += 1;"				\
+		"goto 1b;"				\
+	"2:"						\
+		"w8 += -1;"				\
+		"w2 = w8;"				\
+		"w2 &= 255;"				\
+		"if w2 > 9 goto 9f;"			\
+		"r2 = r8;"				\
+		"r2 &= 255;"				\
+		"r3 = r10;"				\
+		"r3 += -10;"				\
+		"r3 += r2;"				\
+		"r9 = *(u8 *)(r3 + 0);"			\
+		"goto 1b;"
+
+SEC("tc")
+__description("depth of a stack on the stack: a copy is tested")
+__success __retval(0)
+__naked void depth_copy_is_tested(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 16) = r1;"
+		"w8 = 0;"
+		ITER_NEW
+	"1:"
+		ITER_NEXT
+		"if r0 == 0 goto 9f;"
+		PUSH_OR_POP
+	"9:"
+		ITER_DESTROY
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(zero), __imm_iter
+		: __clobber_all);
+}
+
+SEC("tc")
+__description("depth of a stack on the stack: no bound")
+__failure __msg("invalid write to stack R2 off=0 size=1")
+__naked void depth_without_a_bound(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 16) = r1;"
+		"w8 = 0;"
+		ITER_NEW
+	"1:"
+		ITER_NEXT
+		"if r0 == 0 goto 9f;"
+		"r1 = r8;"
+		"r1 &= 255;"
+		"r2 = r10;"
+		"r2 += -10;"
+		"r2 += r1;"
+		"*(u8 *)(r2 + 0) = r1;"
+		"w8 += 1;"
+		"goto 1b;"
+	"9:"
+		ITER_DESTROY
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_iter
+		: __clobber_all);
+}
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+	bpf_iter_num_new(0, 0, 0);
+	bpf_iter_num_next(0);
+	bpf_iter_num_destroy(0);
+}
+
 SEC("sockops")
 __description("indirect variable-offset stack access, unbounded")
 __failure __msg("invalid unbounded variable-offset write to stack R4")
-- 
2.55.0

[PATCH bpf-next v4 41/46] selftests/bpf: Test static subprogs that are verified on their own

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier followed each call into the subprog, with the frames of the
callers in the state. Now it verifies most static subprogs for the states
at their first insn, without these frames. A call waits, and each exit
state goes to all calls that wait for this entry state. What can go wrong:
a call does not get a result that is its own, or the subprog changes what
a caller holds and the caller does not see it.

  [E] "bpf: Add the sets of states at an insn and the entries of functions"
  [M] "bpf: Move states along edges, into loops and subprogs and out of
      them"
  [B] "bpf: Make the state after a call from the caller's state and an
      exit state"
  [O] "bpf: Verify static subprogs on their own"

  verifier_..         the case                                        pins
  ------------------  ----------------------------------------------  ----
  calls_in_a_row.c    20 calls pass kind 3, one call passes kind 1:   [E]
                      the 20 entry states are joined, the one stays
                      'x = inc(x); x = inc(x);' loads and returns 2   [M]
                      3 entry states meet in the subprog in 2 steps:  [E]
                      the exit state is for all 3 calls, if they      [B]
                      wait together or come one after the other
                      The argument is r6 + 10, linked to r6 of the    [O]
                      caller: the 10 is not for the subprog's links
  subprog_callers.c   The subprog gives up the lock or the RCU        [O]
                      section of its caller and takes it again.
                      3 calls wait when the verifier finds it out.
                      It is called, and it is a timer callback too.   [O]
                      Recursive subprogs change packet data or        [O]
                      a tree in the callback of bpf_rbtree_add().
                      Ids at the return: an inner map from each of    [B]
                      2 calls, a reference 2 calls down, too many ids.
                      Two calls pass packet pointers whose offsets    [O]
                      have different ranges.
                      The subprog loads data_end itself and checks    [O]
                      the packet pointer of its caller.
  struct_ops_         The subprog reads an argument with __ref from   [O]
  refcounted_subprog  ctx while the prog holds the reference.
  stack_arg.c         The stack arguments that a subprog got are not  [O]
                      the ones that it passes on. A subprog does the
                      NULL check of a stack argument for its caller.
                      A stack argument is linked in the same way.     [O]

15 of the 25 progs are unsafe and must be rejected: only one call gets
a number that is out of bounds, only the code after one call uses what the
subprog took away.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../prog_tests/test_struct_ops_refcounted.c   |   2 +
 .../selftests/bpf/prog_tests/verifier.c       |   4 +
 .../bpf/progs/struct_ops_refcounted_subprog.c |  30 +
 .../bpf/progs/verifier_calls_in_a_row.c       | 223 +++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  | 127 ++++
 .../bpf/progs/verifier_subprog_callers.c      | 543 ++++++++++++++++++
 6 files changed, 929 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_callers.c

diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
index db79ac9b98d3..e33e4cd1e157 100644
--- a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
+++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
@@ -1,6 +1,7 @@
 #include <test_progs.h>
 
 #include "struct_ops_refcounted.skel.h"
+#include "struct_ops_refcounted_subprog.skel.h"
 #include "struct_ops_refcounted_fail__ref_leak.skel.h"
 #include "struct_ops_refcounted_fail__global_subprog.skel.h"
 #include "struct_ops_refcounted_fail__tail_call.skel.h"
@@ -9,6 +10,7 @@
 void test_struct_ops_refcounted(void)
 {
 	RUN_TESTS(struct_ops_refcounted);
+	RUN_TESTS(struct_ops_refcounted_subprog);
 	RUN_TESTS(struct_ops_refcounted_fail__ref_leak);
 	RUN_TESTS(struct_ops_refcounted_fail__global_subprog);
 	RUN_TESTS(struct_ops_refcounted_fail__tail_call);
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 2f34a34e3d54..3f3b9b48523f 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -36,6 +36,7 @@
 #include "verifier_btf_flex_array.skel.h"
 #include "verifier_btf_unreliable_prog.skel.h"
 #include "verifier_call_large_imm.skel.h"
+#include "verifier_calls_in_a_row.skel.h"
 #include "verifier_callx.skel.h"
 #include "verifier_callx_rodata.skel.h"
 #include "verifier_cfg.skel.h"
@@ -130,6 +131,7 @@
 #include "verifier_stack_arg_order.skel.h"
 #include "verifier_stack_ptr.skel.h"
 #include "verifier_store_release.skel.h"
+#include "verifier_subprog_callers.skel.h"
 #include "verifier_subprog_insn_stats.skel.h"
 #include "verifier_subprog_precision.skel.h"
 #include "verifier_subprog_topo.skel.h"
@@ -236,6 +238,7 @@ void test_verifier_btf_ctx_access(void)       { RUN(verifier_btf_ctx_access); }
 void test_verifier_btf_flex_array(void)       { RUN(verifier_btf_flex_array); }
 void test_verifier_btf_unreliable_prog(void)  { RUN(verifier_btf_unreliable_prog); }
 void test_verifier_call_large_imm(void)       { RUN(verifier_call_large_imm); }
+void test_verifier_calls_in_a_row(void)       { RUN(verifier_calls_in_a_row); }
 void test_verifier_callx(void)                { RUN_JITED(verifier_callx); }
 void test_verifier_callx_rodata(void)         { RUN_JITED(verifier_callx_rodata); }
 void test_verifier_cfg(void)                  { RUN(verifier_cfg); }
@@ -342,6 +345,7 @@ void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }
 void test_verifier_stack_arg_order(void)      { RUN(verifier_stack_arg_order); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
 void test_verifier_store_release(void)        { RUN(verifier_store_release); }
+void test_verifier_subprog_callers(void)      { RUN(verifier_subprog_callers); }
 void test_verifier_subprog_insn_stats(void)   { RUN(verifier_subprog_insn_stats); }
 void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
 void test_verifier_subprog_topo(void)        { RUN(verifier_subprog_topo); }
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c b/tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
new file mode 100644
index 000000000000..c5524bce975b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/struct_ops_refcounted_subprog.c
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+static __noinline int pid_of(unsigned long long *ctx)
+{
+	struct task_struct *task = (void *)ctx[1];
+
+	return task->pid;
+}
+
+/* A static subprog loads the referenced argument from ctx, like the main prog. */
+SEC("struct_ops/test_refcounted")
+__success
+int refcounted_subprog(unsigned long long *ctx)
+{
+	int pid = pid_of(ctx);
+
+	bpf_task_release((void *)ctx[1]);
+	return pid & 1;
+}
+
+SEC(".struct_ops.link")
+struct bpf_testmod_ops testmod_refcounted_subprog = {
+	.test_refcounted = (void *)refcounted_subprog,
+};
diff --git a/tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c b/tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
new file mode 100644
index 000000000000..401256bc34b0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_calls_in_a_row.c
@@ -0,0 +1,223 @@
+// SPDX-License-Identifier: GPL-2.0
+/* A static subprog that is called from many places */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* gcc makes a copy of a function for an argument that is the same constant in most calls */
+#if defined(__GNUC__) && !defined(__clang__)
+#define __noclone __attribute__((noclone))
+#else
+#define __noclone
+#endif
+
+volatile int sel;
+char arr[64];
+
+#define SITES_4(call, n)  call(n) call(n + 1) call(n + 2) call(n + 3)
+#define SITES_20(call)    SITES_4(call, 0) SITES_4(call, 4) SITES_4(call, 8) \
+			  SITES_4(call, 12) SITES_4(call, 16)
+
+/*
+ * The states of the 20 calls are joined at the entry of the subprog one by one: x grows. The state
+ * of the first call stays apart from them, however many joins it takes.
+ *
+ * kind 1: idx is an index. kind 3: idx is something else. 2 is not there.
+ */
+static __noinline __noclone int use(int kind, unsigned int idx, int x)
+{
+	if (kind == 1)
+		return arr[idx] + x;
+	return idx + x;
+}
+
+SEC("socket")
+__description("20 calls in a row, a number that a jump tests")
+__success
+int many_sites_gap(void *ctx)
+{
+	int r = use(1, 5, 100);
+
+#define USE(n) if (sel == n) r += use(3, 1000, n);
+	SITES_20(USE)
+	return r;
+}
+
+/* The second call gets what the first call returns */
+static __noinline __noclone int inc(int x)
+{
+	return x + 1;
+}
+
+SEC("socket")
+__description("x = inc(x), two times in a row")
+__success __retval(2)
+int inc_two_times(void *ctx)
+{
+	int x = 0;
+
+	x = inc(x);
+	x = inc(x);
+	return x;
+}
+
+char four[4] SEC(".data.four") = { 10, 11, 12, 13 };
+
+/*
+ * For the caller the argument is r6 + 10. The subprog has no r6: r1 is linked to nothing there.
+ * r1 is 6 .. 9 and r2 is r1 + 4, that is 10 .. 13, not r1 + 4 - 10.
+ */
+#define FOUR_AT_R1_PLUS_4(off)			\
+	"r2 = r1;"				\
+	"r2 += 4;"				\
+	"r0 = 0;"				\
+	"if r1 < 6 goto 1f;"			\
+	"if r1 > 9 goto 1f;"			\
+	"r3 = %[four] ll;"			\
+	"r3 += r2;"				\
+	"r0 = *(u8 *)(r3 " #off ");"		\
+"1:"						\
+	"exit;"
+
+static __naked __noinline __used int four_at_0_to_3(void)
+{
+	asm volatile (FOUR_AT_R1_PLUS_4(- 10) :: __imm_addr(four) : __clobber_all);
+}
+
+static __naked __noinline __used int four_at_10_to_13(void)
+{
+	asm volatile (FOUR_AT_R1_PLUS_4(+ 0) :: __imm_addr(four) : __clobber_all);
+}
+
+#define CALL_WITH_R6_PLUS_10(func)		\
+	"call %[bpf_get_prandom_u32];"		\
+	"r6 = r0;"				\
+	"r1 = r6;"				\
+	"r1 += 10;"				\
+	"call " #func ";"			\
+	"r0 += r6;"				\
+	"exit;"
+
+SEC("socket")
+__description("the argument is linked to a register of the caller")
+__success
+__naked void arg_is_linked(void)
+{
+	asm volatile (CALL_WITH_R6_PLUS_10(four_at_0_to_3)
+		      :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+SEC("socket")
+__description("the argument is linked to a register of the caller, out of bounds")
+__failure __msg("invalid access to map value, value_size=4 off=10 size=1")
+__naked void arg_is_linked_bad(void)
+{
+	asm volatile (CALL_WITH_R6_PLUS_10(four_at_10_to_13)
+		      :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * r1 is 1, 3 or 5, and the jumps look at it: the states of three calls stay apart at the first
+ * insn. Those of 1 and 3 meet behind the second jump. What becomes of them meets that of 5 behind
+ * the first one. From there on one state stands for the three, and what comes out is for
+ * all three calls.
+ * Returns r2 + 3, r2 + 2, r2.
+ */
+static __naked __noinline __used int three_kinds(void)
+{
+	asm volatile (
+	"if r1 == 5 goto 2f;"
+	"if r1 == 3 goto 1f;"
+	"r2 += 1;"
+"1:"
+	"r2 += 2;"
+"2:"
+	"r0 = r2;"
+	"exit;"
+	::: __clobber_all);
+}
+
+#define CALL(kind, v)				\
+	"r1 = " #kind ";"			\
+	"r2 = " #v ";"				\
+	"call three_kinds;"
+
+#define LOAD					\
+	"r1 = %[four] ll;"			\
+	"r1 += r0;"				\
+	"r0 = *(u8 *)(r1 + 0);"
+
+/* The three calls wait at the same time */
+#define BRANCHES(v1, load1, v3, load3, v5, load5)	\
+	asm volatile (				\
+	"r6 = *(u32 *)(r1 + %[len]);"		\
+	"if r6 == 1 goto 1f;"			\
+	"if r6 == 3 goto 3f;"			\
+	CALL(5, v5)				\
+	load5					\
+	"exit;"					\
+"1:"						\
+	CALL(1, v1)				\
+	load1					\
+	"exit;"					\
+"3:"						\
+	CALL(3, v3)				\
+	load3					\
+	"exit;"					\
+	:					\
+	: __imm_addr(four),			\
+	  __imm_const(len, offsetof(struct __sk_buff, len))	\
+	: __clobber_all)
+
+SEC("tc")
+__description("three calls wait, their states meet in two steps")
+__success __retval(10)
+__naked void three_calls_wait(void)
+{
+	BRANCHES(0, LOAD, 0, LOAD, 0, LOAD);
+}
+
+SEC("tc")
+__description("three calls wait, their states meet in two steps, the first one gets 7")
+__failure __msg("invalid access to map value, value_size=4 off=7 size=1")
+__naked void three_calls_wait_bad_1(void)
+{
+	BRANCHES(4, LOAD, 0, "", 0, "");
+}
+
+#define ADD	LOAD "r6 += r0;"
+
+/* A call is done with before the next one is made */
+#define ROW(k1, v1, load1, k2, v2, load2, k3, v3, load3)	\
+	asm volatile (				\
+	"r6 = 0;"				\
+	CALL(k1, v1)				\
+	load1					\
+	CALL(k2, v2)				\
+	load2					\
+	CALL(k3, v3)				\
+	load3					\
+	"r0 = r6;"				\
+	"exit;"					\
+	:					\
+	: __imm_addr(four)			\
+	: __clobber_all)
+
+SEC("tc")
+__description("three calls in a row, their states meet in two steps")
+__success __retval(35)
+__naked void three_calls_in_a_row(void)
+{
+	ROW(1, 0, ADD, 3, 0, ADD, 5, 0, ADD);
+}
+
+SEC("tc")
+__description("three calls in a row, the second one gets 6")
+__failure __msg("invalid access to map value, value_size=4 off=6 size=1")
+__naked void three_calls_in_a_row_bad_2(void)
+{
+	ROW(1, 0, "", 3, 4, ADD, 5, 0, "");
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index 146cdf81bbc4..e3f5e4d14b75 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -445,6 +445,133 @@ __naked void stack_arg_shared_store(void)
 	);
 }
 
+/* What it stores is what it got: the path that stores nothing looks the same at the call. */
+static __naked __noinline __used long stores_same_on_one_path(long a, long b, long c, long d,
+							      long e, long f, long g)
+{
+	asm volatile (
+	"r6 = *(u64 *)(r11 + 8);"
+	"r7 = *(u64 *)(r11 + 16);"
+	"if r1 == 0 goto l0_%=;"
+	"*(u64 *)(r11 - 8) = r6;"
+	"*(u64 *)(r11 - 16) = r7;"
+"l0_%=:"
+	"r1 = 0;"
+	"r2 = 0;"
+	"r3 = 0;"
+	"r4 = 0;"
+	"r5 = 0;"
+	"call subprog_7args;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: incoming args are not taken for outgoing args, same values")
+__failure __msg("callee expects 7 args, stack arg1 is not initialized")
+__naked void incoming_for_outgoing_same(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = r0;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"*(u64 *)(r11 - 8) = 6;"
+	"*(u64 *)(r11 - 16) = 7;"
+	"call stores_same_on_one_path;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/* r1 is 6 .. 9 and r2 is r1 + 4, that is 10 .. 13: r9, which is not set, is not read */
+static __naked __noinline __used long sixth_plus_4(long a, long b, long c, long d, long e, long f)
+{
+	asm volatile (
+	"r1 = *(u64 *)(r11 + 8);"
+	"r2 = r1;"
+	"r2 += 4;"
+	"r0 = 0;"
+	"if r1 < 6 goto l0_%=;"
+	"if r1 > 9 goto l0_%=;"
+	"if r2 > 9 goto l0_%=;"
+	"r0 = r9;"
+"l0_%=:"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* For the caller the argument is r6 + 10. The subprog has no r6. */
+SEC("tc")
+__description("stack_arg: the argument is linked to a register of the caller")
+__success
+__naked void stack_arg_is_linked(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r6 = r0;"
+	"r1 = r6;"
+	"r1 += 10;"
+	"*(u64 *)(r11 - 8) = r1;"
+	"r1 = 1;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"call sixth_plus_4;"
+	"r0 += r6;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+static __naked __noinline __used long sixth_is_there(long a, long b, long c, long d, long e,
+						     long *f)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r11 + 8);"
+	"if r0 == 0 goto l0_%=;"
+	"r0 = 1;"
+"l0_%=:"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: subprog does the NULL check for its caller")
+__success
+__naked void null_check_of_stack_arg(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"r1 = %[map_hash_8b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"r6 = r0;"
+	"r1 = 1;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"*(u64 *)(r11 - 8) = r6;"
+	"call sixth_is_there;"
+	"if r0 == 0 goto l0_%=;"
+	"r0 = *(u64 *)(r6 + 0);"
+"l0_%=:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_8b)
+	: __clobber_all);
+}
+
 /* r1 - r5 and all 19 slots. Its prototype has no parameters: what it takes is seen in its code. */
 __naked __noinline __used
 static unsigned long subprog_24args(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_callers.c b/tools/testing/selftests/bpf/progs/verifier_subprog_callers.c
new file mode 100644
index 000000000000..48338c0ae467
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_callers.c
@@ -0,0 +1,543 @@
+// SPDX-License-Identifier: GPL-2.0
+/* A static subprog and its callers: what they hold, where they run */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+#define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8)))
+
+struct elem {
+	struct bpf_list_node l;
+	__u64 data;
+};
+
+private(A) struct bpf_spin_lock lock_a;
+private(A) struct bpf_list_head head_a __contains(elem, l);
+
+int pid;
+
+static __noinline void relock_a(void)
+{
+	bpf_spin_unlock(&lock_a);
+	bpf_spin_lock(&lock_a);
+}
+
+/* Somebody else can pop and free the node while the lock is not held. */
+SEC("tc")
+__description("subprog drops the lock of its caller for a while, non-owning reference")
+__failure __msg("invalid mem access 'scalar'")
+int relock_non_owning_ref(struct __sk_buff *skb)
+{
+	struct elem *n;
+
+	n = bpf_obj_new(typeof(*n));
+	if (!n)
+		return 0;
+	bpf_spin_lock(&lock_a);
+	bpf_list_push_front(&head_a, &n->l);
+	relock_a();
+	n->data = 42;
+	bpf_spin_unlock(&lock_a);
+	return 0;
+}
+
+static __noinline void rcu_window(void)
+{
+	bpf_rcu_read_unlock();
+	bpf_rcu_read_lock();
+}
+
+SEC("?fentry.s/" SYS_PREFIX "sys_getpgid")
+__description("subprog leaves the RCU section of its caller for a while")
+__failure __msg("R1 must be a rcu pointer")
+int rcu_window_rcu_ptr(void *ctx)
+{
+	struct task_struct *task = bpf_get_current_task_btf(), *parent;
+
+	bpf_rcu_read_lock();
+	parent = task->real_parent;
+	if (!parent)
+		goto out;
+	rcu_window();
+	parent = bpf_task_acquire(parent);
+	if (parent) {
+		pid = parent->pid;
+		bpf_task_release(parent);
+	}
+out:
+	bpf_rcu_read_unlock();
+	return 0;
+}
+
+struct timer_elem {
+	struct bpf_timer timer;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer_elem);
+} timers SEC(".maps");
+
+static __noinline int leaves_preempt_off(void *map, int *key, struct timer_elem *val)
+{
+	bpf_preempt_disable();
+	return 0;
+}
+
+/* The call and the callback have one frame each. The callback returns to the kernel. */
+SEC("tc")
+__description("subprog is called, then it is made a timer callback")
+__failure __msg("cannot be used inside bpf_preempt_disable-ed region")
+int call_then_timer_callback(struct __sk_buff *skb)
+{
+	struct timer_elem *val = bpf_map_lookup_elem(&timers, &(int){0});
+
+	if (!val)
+		return 0;
+	leaves_preempt_off(NULL, NULL, NULL);
+	bpf_preempt_enable();
+	bpf_timer_set_callback(&val->timer, leaves_preempt_off);
+	return 0;
+}
+
+/* return p + 14 > ctx->data_end ? -22 : 0; */
+static __naked __noinline __used int has_eth(struct xdp_md *ctx, void *p)
+{
+	asm volatile (
+	"r3 = *(u32 *)(r1 + %[data_end]);"
+	"r2 += 14;"
+	"r0 = -22;"
+	"if r2 > r3 goto 1f;"
+	"r0 = 0;"
+"1:	exit;"
+	:
+	: __imm_const(data_end, offsetof(struct xdp_md, data_end))
+	: __clobber_all);
+}
+
+/* The caller learns how much of the packet is there. */
+SEC("?xdp")
+__description("subprog compares a packet pointer of its caller with the end that it loads itself")
+__success
+__naked void subprog_loads_data_end(void)
+{
+	asm volatile (
+	"r6 = *(u32 *)(r1 + %[data]);"
+	"r2 = r6;"
+	"call has_eth;"
+	"if r0 != 0 goto 1f;"
+	"r0 = *(u8 *)(r6 + 13);"
+"1:	r0 = 1;"
+	"exit;"
+	:
+	: __imm_const(data, offsetof(struct xdp_md, data))
+	: __clobber_all);
+}
+
+struct tree_elem {
+	long key;
+	struct bpf_rb_node node;
+};
+
+private(D) struct bpf_spin_lock lock_d;
+private(D) struct bpf_rb_root root_d __contains(tree_elem, node);
+
+static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct tree_elem *elem_a = container_of(a, struct tree_elem, node);
+	struct tree_elem *elem_b = container_of(b, struct tree_elem, node);
+
+	return elem_a->key < elem_b->key;
+}
+
+static __noinline long removes_and_recurses(struct tree_elem *e, int n)
+{
+	struct bpf_rb_node *node;
+
+	if (n <= 0)
+		return 0;
+	node = bpf_rbtree_remove(&root_d, &e->node);
+	if (node)
+		bpf_rbtree_add(&root_d, node, less);
+	n = removes_and_recurses(e, n - 1);
+	barrier_var(n);
+	return n + 1;
+}
+
+static bool less_changes_tree(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct tree_elem *elem_a = container_of(a, struct tree_elem, node);
+	struct tree_elem *elem_b = container_of(b, struct tree_elem, node);
+
+	removes_and_recurses(elem_b, 1);
+	return elem_a->key < elem_b->key;
+}
+
+/* bpf_rbtree_add() walks the tree while it calls the callback. */
+SEC("?tc")
+__description("recursive subprog changes the tree, it is called from the callback of rbtree_add")
+__arch_x86_64
+__arch_arm64
+__failure __msg("bpf_rbtree_remove not allowed in rbtree cb")
+int recursion_removes_in_rbtree_callback(void *ctx)
+{
+	struct tree_elem *e;
+
+	e = bpf_obj_new(typeof(*e));
+	if (!e)
+		return 0;
+	bpf_spin_lock(&lock_d);
+	bpf_rbtree_add(&root_d, &e->node, less_changes_tree);
+	bpf_spin_unlock(&lock_d);
+	return 0;
+}
+
+/*
+ * What a subprog does is what its callees do. The walk over the graph is done with second()
+ * before it is done with first(), which second() calls: the prog calls second() and then
+ * first(), so first() is looked at first, and in it the call that comes last.
+ */
+static __noinline int pulls_data(struct __sk_buff *skb)
+{
+	return bpf_skb_pull_data(skb, 0);
+}
+
+static __noinline int pulls_first(struct __sk_buff *skb, int n);
+
+static __noinline int pulls_second(struct __sk_buff *skb, int n)
+{
+	if (n <= 0)
+		return 0;
+	return pulls_first(skb, n - 1) + 1;
+}
+
+static __noinline int pulls_first(struct __sk_buff *skb, int n)
+{
+	int r = pulls_data(skb);
+
+	return r + pulls_second(skb, n);
+}
+
+SEC("?tc")
+__description("recursive static subprogs, the one that is explored last changes packet data")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid mem access 'scalar'")
+int recursion_changes_pkt_data(struct __sk_buff *skb)
+{
+	char *p = (void *)(long)skb->data, *end = (void *)(long)skb->data_end;
+	int r;
+
+	if (p + 1 > end)
+		return 0;
+	r = pulls_second(skb, 1);
+	*p = 42;
+	r += pulls_first(skb, 0);
+	return r & 1;
+}
+
+static __naked __noinline __used long returns_third(void)
+{
+	asm volatile (
+	"r0 = r1;"
+	"r0 = r2;"
+	"r0 = r3;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* What comes back has the range of r6, not the range of the pointer of the first call. */
+#define PKT_POINTERS_OF_TWO_RANGES(func)	\
+	asm volatile (				\
+	"r9 = *(u32 *)(r1 + %[data]);"		\
+	"r8 = *(u32 *)(r1 + %[data_end]);"	\
+	"call %[bpf_get_prandom_u32];"		\
+	"r7 = r0;"				\
+	"r0 &= 0xff;"				\
+	"r1 = 0;"				\
+	"r2 = 3;"				\
+	"r3 = r9;"				\
+	"r3 += r0;"				\
+	"call " #func ";"			\
+	"r7 &= 1;"				\
+	"r6 = r9;"				\
+	"r6 += r7;"				\
+	"r1 = 0;"				\
+	"r2 = 3;"				\
+	"r3 = r6;"				\
+	"call " #func ";"			\
+	"if r0 == 0 goto 1f;"			\
+	"if r0 > r8 goto 1f;"			\
+	"r0 = *(u8 *)(r6 + 100);"		\
+"1:"						\
+	"r0 = 0;"				\
+	"exit;"					\
+	:					\
+	: __imm(bpf_get_prandom_u32),		\
+	  __imm_const(data, offsetof(struct __sk_buff, data)),		\
+	  __imm_const(data_end, offsetof(struct __sk_buff, data_end))	\
+	: __clobber_all)
+
+SEC("tc")
+__description("two calls, the offset of the packet pointer has a smaller range in the second one")
+__failure __msg("invalid access to packet")
+__naked void pkt_pointers_of_two_ranges(void)
+{
+	PKT_POINTERS_OF_TWO_RANGES(returns_third);
+}
+
+static __naked __noinline __used long returns_third_r2_calls_down(void)
+{
+	asm volatile (
+	"if r2 == 0 goto 1f;"
+	"r2 += -1;"
+	"call returns_third_r2_calls_down;"
+	"exit;"
+"1:"
+	"r0 = r3;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* A recursive subprog that gets such a pointer needs the frames of its callers too. */
+SEC("?tc")
+__description("two calls of a recursive subprog, the second packet pointer has the smaller range")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to packet")
+__naked void pkt_pointers_of_two_ranges_recursion(void)
+{
+	PKT_POINTERS_OF_TWO_RANGES(returns_third_r2_calls_down);
+}
+
+static __naked __noinline __used long tcp_sock_of(void)
+{
+	asm volatile (
+	"call %[bpf_tcp_sock];"
+	"exit;"
+	:
+	: __imm(bpf_tcp_sock)
+	: __clobber_all);
+}
+
+static __naked __noinline __used long calls_tcp_sock_of(void)
+{
+	asm volatile (
+	"call tcp_sock_of;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* The caller of tcp_sock_of() has the reference from its own caller. */
+SEC("tc")
+__description("reference goes down two calls, what is made from it is used after the release")
+__failure __msg("R7 invalid mem access 'scalar'")
+__naked void reference_two_calls_down(void)
+{
+	asm volatile (
+	"r2 = 0;"
+	"*(u64 *)(r10 - 8) = r2;"
+	"*(u64 *)(r10 - 16) = r2;"
+	"r2 = r10;"
+	"r2 += -16;"
+	"r3 = 12;"
+	"r4 = 0;"
+	"r5 = 0;"
+	"call %[bpf_sk_lookup_tcp];"
+	"if r0 == 0 goto 1f;"
+	"r6 = r0;"
+	"r1 = r6;"
+	"call calls_tcp_sock_of;"
+	"r7 = r0;"
+	"r1 = r6;"
+	"call %[bpf_sk_release];"
+	"if r7 == 0 goto 1f;"
+	"r0 = *(u32 *)(r7 + 0);"
+"1:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_sk_lookup_tcp),
+	  __imm(bpf_sk_release)
+	: __clobber_all);
+}
+
+struct inner_timers {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer_elem);
+} inner_timers SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
+	__uint(max_entries, 2);
+	__type(key, int);
+	__array(values, struct inner_timers);
+} outer_timers SEC(".maps");
+
+static __naked __noinline __used long inner_map(void)
+{
+	asm volatile (
+	"*(u32 *)(r10 - 4) = r1;"
+	"r2 = r10;"
+	"r2 += -4;"
+	"r1 = %[outer_timers] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(outer_timers)
+	: __clobber_all);
+}
+
+#define TWO_INNER_MAPS				\
+	"r1 = 0;"				\
+	"call inner_map;"			\
+	"if r0 == 0 goto 1f;"			\
+	"r6 = r0;"				\
+	"r1 = 1;"				\
+	"call inner_map;"			\
+	"if r0 == 0 goto 1f;"			\
+	"r7 = r0;"
+
+/* The timer is in the map in r0. It is fine only if that is the map in r6. */
+#define INIT_TIMER_WITH_R6			\
+	"r1 = 0;"				\
+	"*(u32 *)(r10 - 4) = r1;"		\
+	"r2 = r10;"				\
+	"r2 += -4;"				\
+	"r1 = r0;"				\
+	"call %[bpf_map_lookup_elem];"		\
+	"if r0 == 0 goto 1f;"			\
+	"r1 = r0;"				\
+	"r2 = r6;"				\
+	"r3 = 1;"				\
+	"call %[bpf_timer_init];"		\
+"1:"						\
+	"r0 = 0;"				\
+	"exit;"
+
+#define INNER_MAP_IMMS				\
+	__imm(bpf_get_prandom_u32),		\
+	__imm(bpf_map_lookup_elem),		\
+	__imm(bpf_timer_init)
+
+/* Each call returns another map. */
+SEC("tc")
+__description("subprog returns an inner map, timer of one map is set up with another one")
+__failure __msg("doesn't match map pointer in R2")
+__naked void inner_maps_of_two_calls(void)
+{
+	asm volatile (
+	TWO_INNER_MAPS
+	INIT_TIMER_WITH_R6
+	:
+	: INNER_MAP_IMMS
+	: __clobber_all);
+}
+
+/* As many as there is room for ids at the return, and one more. */
+static __naked __noinline __used void saves_irq_flags_many_times(void)
+{
+	asm volatile (
+	".rept 9153;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call %[bpf_local_irq_save];"
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	".endr;"
+	"exit;"
+	:
+	: __imm(bpf_local_irq_save)
+	: __clobber_all);
+}
+
+SEC("tc")
+__description("subprog comes back with more ids than there is room for")
+__failure __msg("too many ids at the call at insn")
+__log_level(4)
+int too_many_ids_at_the_return(void *ctx)
+{
+	unsigned long flags;
+
+	saves_irq_flags_many_times();
+	bpf_local_irq_save(&flags);
+	bpf_local_irq_restore(&flags);
+	return 0;
+}
+
+static __naked __noinline __used void leaves_rcu_section(void)
+{
+	asm volatile (
+	"call %[bpf_rcu_read_unlock];"
+	"call %[bpf_rcu_read_lock];"
+	"exit;"
+	:
+	: __imm(bpf_rcu_read_unlock),
+	  __imm(bpf_rcu_read_lock)
+	: __clobber_all);
+}
+
+static __naked __noinline __used int returns_zero(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/*
+ * Three calls wait for the subprog. When it's known what the subprog does to the RCU section of
+ * its callers each of them stops to wait and is verified in the context of its caller.
+ * Then one more call waits.
+ */
+#define RCU_BRANCHES(then1, then2, then3)	\
+	asm volatile (				\
+	"r6 = *(u32 *)(r1 + %[len]);"		\
+	"call %[bpf_rcu_read_lock];"		\
+	"if r6 == 1 goto 1f;"			\
+	"if r6 == 2 goto 2f;"			\
+	"call leaves_rcu_section;"		\
+	then3					\
+	"goto 3f;"				\
+"1:"						\
+	"call leaves_rcu_section;"		\
+	then1					\
+	"goto 3f;"				\
+"2:"						\
+	"call leaves_rcu_section;"		\
+	then2					\
+"3:"						\
+	"call %[bpf_rcu_read_unlock];"		\
+	"call returns_zero;"			\
+	"exit;"					\
+	:					\
+	: __imm(bpf_rcu_read_lock),		\
+	  __imm(bpf_rcu_read_unlock),		\
+	  __imm_const(len, offsetof(struct __sk_buff, len))	\
+	: __clobber_all)
+
+SEC("tc")
+__description("three calls stop to wait")
+__success __retval(0)
+__naked void three_calls_stop_to_wait(void)
+{
+	RCU_BRANCHES("", "", "");
+}
+
+SEC("tc")
+__description("three calls stop to wait, what follows the first one")
+__failure __msg("R7 !read_ok")
+__naked void three_calls_stop_to_wait_bad_1(void)
+{
+	RCU_BRANCHES("r0 = r7;", "", "");
+}
+
+char _license[] SEC("license") = "GPL";
-- 
2.55.0

[PATCH bpf-next v4 42/46] selftests/bpf: Test loops that are counted, walked or widened

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier has three methods for a loop:
 - Counted: scalar evolution gives the number of iterations and a range for
   each register that has a closed form. The loop gets no may_goto.
 - Walked: iteration by iteration, with the real values.
 - Widened: what returns to the head is joined with the state that it comes
   from. A bound that grows moves to a fixed stop, until nothing changes.
The wrong method rejects a valid prog or costs thousands of insns. A wrong
state at the head loads an unsafe prog.

  [W] "bpf: Add union and widening of the range of a register"
  [J] "bpf: Add join of registers and stack slots"
  [A] "bpf: Find the registers and stack slots that later checks depend on"
  [E] "bpf: Merge the state that arrives into the set at an insn"
  [M] "bpf: Move states along edges, into loops and subprogs and out of
      them"
  [L] "bpf: Enter and leave loops in the main pass, build flow.c"
  [C] "bpf: Keep the count of a loop when a register has no closed form"
  [G] "bpf: Tell the passes over the graph where an added may_goto goes"
  [1] "bpf: Keep the first iteration of a counted loop apart"
  [R] "bpf: What a call returns in each iteration is like what the loop
      loads"
  [T] "bpf: Find where the jumps of a loop stop a number"
  [B] "bpf: A loop is one iteration shorter behind the jump that counts it"
  [P] "bpf: Give the places of a back edge to the jumps that count"
  [Y] "bpf: Count a loop whose exit tests a copy of the counter"

  verifier_..        the loop                     the verifier must
  -----------------  ---------------------------  -------------------------
  loop_need.c        is counted, 'x = x * 3 + 1'  [C] walk it if a jump, an
                     has no closed form           access or a call ahead
                                                  depends on x ([A]), else
                                                  widen x
                     is too long for a walk,      [1] verify the first
                     'idx = table[idx]',          iteration on its own
                     'x = rnd() & 15'             [R] the same
  loop_as_is.c       is not counted               [M] join the state that
                                                  ends a walk with a state
                                                  of this walk only
                     or bpf_loop(): two states    [E] join what comes back
                     get there, with 0 and 8      with the state that it
                                                  comes from
                     'if (..) n--', from 3        [W] stop n at 0
                     'sum += x', no jump depends  [J] forget sum at once:
                     on sum, in 64 and 32 bits    two passes
                     a jump compares sum with     [J] stop sum at S32_MAX
                     a register
                     'if (sum == 77)'             [W] not try S64_MAX after
                                                  U32_MAX
                     the largest of numbers       [W] keep the lower bound
                     'off -= 4' until off is -4   [W] keep the step of 4
                     'if (n < 8) a[n++] = j',     [T] stop n at 8, off at
                     'off += l' below 8000        7999
                     is counted, too long for     [C] stop n at 8, add no
                     a walk, 'if (n < 8) n++'     may_goto
  loop_walk.c        has two entries (bpfjailer)  [L] walk it to its end
                     each iteration decides       [L] go on with the walk
                     a jump that leaves the loop
                     'while (rnd()) n++'          [L] give up at once
                     has two iterations, a flag   [L] walk both
                     for a counter, and a call
                     only a flag that went both   [L] give up
                     ways decides its jump
                     32 states enter, each needs  [L] count the insns of
                     a walk of 300 insns          each walk, not of all
                     is too long for a walk       [L] not try it again
  loop_latch.c       ends when a flag is false:   [G] count it by the jump
                     'more = ++i < N && cond',    that clears the flag
                     20 insns, a jump or calls
                     in front of 'while (more)'
                     the same with 40 jumps, an   [G] count it
                     error code or a size
                     from that jump a path leads  [G] not count it
                     to another back edge, or
                     into a nested loop
                     the flag is a constant only  [G] not count it
                     without an added may_goto
                     a load changes the flag,     [G] not count it
                     101 of 64 - 127 goes on
                     a store behind the jump      [B] one iteration less
                     that counts                  for the pointer
                     that jump leaves by way of   [B] not shorten the range
                     a flag, or on both edges
                     five statuses of calls are   [P] count it, by way of
                     nearer to the back edge      a flag too
                     'while (buf[i++] && i < N)'  [Y] count it if the two
                     tests a copy of i            are equal at the entry
  value_ptr_arith.c  'p = &v[p->next]'            [J] keep [0, 31] next to
                                                  0: it covers the rest
                     'p += 1'                     [J] reject
  scev.c             takes a reference in each    [L] reject, or load
                     iteration, or in one
  iterating_         bpf_loop() is the head of    [L] verify the callback
  callbacks.c        a walked loop                in the last iteration too

In most tests the number is an index after the loop, where no jump limits
it: its range at the head decides.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/verifier.c       |   6 +
 .../bpf/progs/verifier_iterating_callbacks.c  |  53 ++
 .../selftests/bpf/progs/verifier_loop_as_is.c | 513 ++++++++++++++++++
 .../selftests/bpf/progs/verifier_loop_latch.c | 300 +++++++++-
 .../selftests/bpf/progs/verifier_loop_need.c  | 372 +++++++++++++
 .../selftests/bpf/progs/verifier_loop_walk.c  | 306 +++++++++++
 .../selftests/bpf/progs/verifier_scev.c       | 130 +++++
 .../bpf/progs/verifier_value_ptr_arith.c      |  55 ++
 8 files changed, 1733 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_need.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_loop_walk.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 3f3b9b48523f..978d6d7773ca 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -84,7 +84,10 @@
 #include "verifier_liveness_exp.skel.h"
 #include "verifier_load_acquire.skel.h"
 #include "verifier_loop_hierarchy.skel.h"
+#include "verifier_loop_as_is.skel.h"
 #include "verifier_loop_latch.skel.h"
+#include "verifier_loop_need.skel.h"
+#include "verifier_loop_walk.skel.h"
 #include "verifier_loops1.skel.h"
 #include "verifier_lwt.skel.h"
 #include "verifier_map_in_map.skel.h"
@@ -286,7 +289,10 @@ void test_verifier_linked_scalars(void)       { RUN(verifier_linked_scalars); }
 void test_verifier_live_stack(void)           { RUN(verifier_live_stack); }
 void test_verifier_liveness_exp(void)         { RUN(verifier_liveness_exp); }
 void test_verifier_loop_hierarchy(void)       { RUN(verifier_loop_hierarchy); }
+void test_verifier_loop_as_is(void)           { RUN(verifier_loop_as_is); }
 void test_verifier_loop_latch(void)           { RUN(verifier_loop_latch); }
+void test_verifier_loop_need(void)            { RUN(verifier_loop_need); }
+void test_verifier_loop_walk(void)            { RUN(verifier_loop_walk); }
 void test_verifier_loops1(void)               { RUN(verifier_loops1); }
 void test_verifier_lwt(void)                  { RUN(verifier_lwt); }
 void test_verifier_map_in_map(void)           { RUN(verifier_map_in_map); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
index a3e31aa25fe1..e30f7676081f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
+++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
@@ -290,6 +290,59 @@ static __naked void iter_limit_bug_cb(void)
 	);
 }
 
+static __naked __noinline __used void set_flag_cb(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"*(u64 *)(r2 + 0) = r1;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all
+	);
+}
+
+/*
+ * bpf_loop() is the head of a loop whose iterations are looked at one by one, because of r1.
+ * The callback comes back to it in the same iteration, the last one included.
+ */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callback_returns_to_loop_head(void)
+{
+	asm volatile (
+	"r7 = 0;"
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r1 = 1;"
+	"r2 = %[set_flag_cb] ll;"
+	"r3 = r10;"
+	"r3 += -8;"
+	"r4 = 0;"
+	"1:"
+	"call %[bpf_loop];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"if r1 == 0 goto 2f;"
+	"if r7 != 3 goto 2f;"
+	"r10 = 0;"
+	"2:"
+	"r7 += 1;"
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r1 = r7;"
+	"r2 = %[set_flag_cb] ll;"
+	"r3 = r10;"
+	"r3 += -8;"
+	"r4 = 0;"
+	"if r7 < 4 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_addr(set_flag_cb),
+	  __imm(bpf_loop)
+	: __clobber_all
+	);
+}
+
 /*
  * may_goto doesn't leave the loop, so it's not what stops it. The address is greater than 10:
  * the loop spins until may_goto that the verifier adds ends the prog.
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_as_is.c b/tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
new file mode 100644
index 000000000000..fffc53e17b5d
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_as_is.c
@@ -0,0 +1,513 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The state that enters a loop that is not counted is left as it is. So is the state that ends
+ * a walk through the iterations of such a loop. What stands for many iterations is a join of states
+ * that were seen, not a guess that is written into one of them.
+ */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u64[9]);
+} map SEC(".maps");
+
+#define LOOKUP(reg)							\
+	"r1 = 0;"							\
+	"*(u32 *)(r10 - 4) = r1;"					\
+	"r2 = r10;"							\
+	"r2 += -4;"							\
+	"r1 = %[map] ll;"						\
+	"call %[bpf_map_lookup_elem];"					\
+	#reg " = r0;"							\
+	"if r0 == 0 goto 9f;"
+
+#define END								\
+"9:"									\
+	"r0 = 0;"							\
+	"exit;"								\
+	:								\
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)			\
+	: __clobber_all
+
+/* END with more operands. may_goto leaves the loop: @off insns ahead */
+#define END_WITH(more)							\
+"9:"									\
+	"r0 = 0;"							\
+	"exit;"								\
+	:								\
+	: __imm(bpf_map_lookup_elem), __imm_addr(map), more		\
+	: __clobber_all
+#define MAY_GOTO(off) __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0))
+
+#define OK(text)	SEC("socket") __description("as is: " text) __success __retval(0)
+#define BAD(text)	SEC("socket") __description("as is: " text) __failure
+
+/*
+ *	do { n += 3; } while (v[0]);
+ * Two iterations are looked at as they are. The second one ends the walk: its jump goes both
+ * ways. Its state is joined with the one before, nothing stops n, so n is any number.
+ * What comes back next is within that. The number of insns tells that there is no fourth pass.
+ * The loop gets may_goto when the walk ends, and all is done again: two times 24 insns.
+ */
+OK("the state that ends a walk is joined with the one before")
+__log_level(1) __msg("processed 48 insns")
+__naked void two_passes(void)
+{
+	asm volatile (
+	LOOKUP(r7)
+	"r6 = 0;"
+"1:"
+	"r6 += 3;"
+	"r1 = *(u32 *)(r7 + 0);"
+	"if r1 != 0 goto 1b;"
+	"r6 &= 63;"
+	"r7 += r6;"
+	"r0 = *(u8 *)(r7 + 8);"
+	END);
+}
+
+/*
+ *	x = (v[0] & 1) << 62; do { x |= x >> 1; if (x > 1 << 62) break; } while (v[1]);
+ * The range at the head is 0..1 << 62 in every pass, the lower half is 0, and one more bit is not
+ * known: bit 61, 60 .. 32. That is not followed bit by bit, 30 passes and more in a nest of loops:
+ * the known bits are what the range says after the first one.
+ */
+OK("known bits are not lost one by one")
+__log_level(1) __msg("processed 74 insns")
+__naked void bits_one_by_one(void)
+{
+	asm volatile (
+	LOOKUP(r7)
+	"r6 = *(u64 *)(r7 + 0);"
+	"r6 &= 1;"
+	"r6 <<= 62;"
+	"r8 = 1;"
+	"r8 <<= 62;"
+"1:"
+	"r1 = r6;"
+	"r1 >>= 1;"
+	"r6 |= r1;"
+	"if r6 > r8 goto 9f;"
+	"r1 = *(u64 *)(r7 + 8);"
+	"if r1 != 0 goto 1b;"
+	"r6 >>= 60;"
+	"r7 += r6;"
+	"r0 = *(u8 *)(r7 + 0);"
+	END);
+}
+
+/*
+ *	if (x < 1000000) return; do { if (x && !v[2]) x--; } while (v[1]);
+ * x is 1000000..U64_MAX, then 999999..U64_MAX. As a signed number it's S64_MIN..S64_MAX all the
+ * time: no signed bound moves, and the range does, a million times if nothing is done about it.
+ */
+OK("a range moves and its signed bounds do not")
+__log_level(1) __msg("processed 68 insns")
+__naked void unsigned_bound_moves(void)
+{
+	asm volatile (
+	LOOKUP(r7)
+	"r6 = *(u64 *)(r7 + 0);"
+	"if r6 < 1000000 goto 9f;"
+"1:"
+	"if r6 == 0 goto 2f;"
+	"r1 = *(u64 *)(r7 + 16);"
+	"if r1 != 0 goto 2f;"
+	"r6 += -1;"
+"2:"
+	"r1 = *(u64 *)(r7 + 8);"
+	"if r1 != 0 goto 1b;"
+	"r6 &= 7;"
+	"r7 += r6;"
+	"r0 = *(u8 *)(r7 + 0);"
+	END);
+}
+
+/*
+ *	n = 3; while (v[0]) { if (v[0] != 1 && n) n--; }
+ * n goes from 3 to 0. The bound that moves at the head of the loop is put at 0, the nearest of the
+ * fixed stops, and that holds. Without it n goes on to fall, and n & 63 is up to 63.
+ */
+OK("a number that falls stops at 0")
+__naked void stops_at_0(void)
+{
+	asm volatile (
+	LOOKUP(r6)
+	"r7 = 3;"
+"1:"
+	"r1 = *(u64 *)(r6 + 0);"
+	"if r1 == 0 goto 3f;"
+	"if r1 == 1 goto 2f;"
+	"if r7 == 0 goto 2f;"
+	"r7 += -1;"
+"2:"
+	"goto 1b;"
+"3:"
+	"r7 &= 63;"
+	"r6 += r7;"
+	"r0 = *(u8 *)(r6 + 68);"
+	END);
+}
+
+/*
+ *	sum = 0; n = 0; max = v[1]; if (max > 64) return;
+ *	while (can_loop) { @test sum += (u8)v[0]; if (!(u8)v[0]) break; }
+ *	@use ((u8 *)v)[sum]
+ * The loop has may_goto, @off insns long, so it is neither counted nor walked. The number of insns
+ * tells how many times it is looked at.
+ */
+#define SUM(name, test, add, off, use)					\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r7)							\
+	"r6 = 0;"							\
+	"r9 = 0;"							\
+	"r8 = *(u64 *)(r7 + 8);"					\
+	"if r8 s> 64 goto 9f;"						\
+"1:"									\
+	".8byte %[may_goto];"						\
+	test								\
+	"r1 = *(u8 *)(r7 + 0);"						\
+	add								\
+	"if r1 != 0 goto 1b;"						\
+	use								\
+	"r7 += r6;"							\
+	"r0 = *(u8 *)(r7 + 0);"						\
+	END_WITH(MAY_GOTO(off)));					\
+}
+
+/* No jump of the loop depends on the sum, no stop holds it: two passes, with 0 and any number */
+OK("a sum is any number at once")
+__log_level(1) __msg("processed 25 insns")
+SUM(sum_two_passes, "", "r6 += r1;", 3, "r6 &= 63;")
+
+BAD("a sum is any number at once, and no index")
+__msg("register with unbounded min value is not allowed")
+SUM(sum_two_passes_bad, "", "r6 += r1;", 3, "")
+
+/* sum = x + sum + x; if (x != 3) n -= x; */
+OK("a sum has many forms")
+__log_level(1) __msg("processed 36 insns")
+SUM(sum_forms, "",
+    "r2 = r1; r2 += r6; r6 = r2; r6 += r1; if r1 == 3 goto +1; r9 -= r1;", 8, "r6 += r9; r6 &= 63;")
+
+/* A test of the low half is sufficient behind the loop */
+OK("a sum in 32 bits is any number of 32 bits at once")
+__log_level(1) __msg("processed 25 insns")
+SUM(sum32_two_passes, "", "w6 += w1;", 3, "if w6 > 63 goto 9f;")
+
+/* 0..S32_MAX at the head, 0..63 behind the jump. Of any number the jump leaves S64_MIN..63. */
+OK("a sum that a jump compares with a register is not forgotten")
+SUM(sum_below_max, "if r6 s>= r8 goto 9f; r2 = r7; r2 += r6; r0 = *(u8 *)(r2 + 0);",
+    "r6 += r1;", 7, "r6 &= 63;")
+
+/* The same jump in a loop in the loop: do { if (sum >= max) return; } while (((u8 *)v)[sum]); */
+OK("a sum that a jump of a nested loop compares with a register is not forgotten")
+SUM(sum_below_max_nested,
+    "2: if r6 s>= r8 goto 9f; r2 = r7; r2 += r6; r0 = *(u8 *)(r2 + 0); if r0 != 0 goto 2b;",
+    "r6 += r1;", 8, "r6 &= 63;")
+
+/* 0, 0..S32_MAX, 0..U32_MAX, any number. 0..S64_MAX would be one more pass, and wraps. */
+OK("a sum that gets past U32_MAX is any number")
+__log_level(1) __msg("processed 37 insns")
+SUM(sum_past_u32_max, "if r6 == 77 goto +0;", "r6 += r1;", 4, "r6 &= 63;")
+
+/* 0, S32_MIN..0, any number */
+OK("a sum that gets past S32_MIN is any number")
+__log_level(1) __msg("processed 32 insns")
+SUM(sum_past_s32_min, "if r6 == 77 goto +0;", "r6 -= r1;", 4, "r6 &= 63;")
+
+/*
+ *	while (can_loop) { x = v[1] >> 1; if (x > sum) sum = x; if (!(u8)v[0]) break; }
+ * 0..S64_MAX is what arrives. No guess has failed: the lower bound stays, the signed test needs it.
+ */
+OK("the largest of numbers up to S64_MAX is not negative")
+SUM(largest, "r2 = *(u64 *)(r7 + 8); r2 >>= 1; if r2 <= r6 goto +1; r6 = r2;", "", 6,
+    "if r6 s> 63 goto 9f;")
+
+/*
+ *	n = 0; while (v[0]) { if (v[0] != 1 && n < 8) n++; }
+ *	((u8 *)v)[n + off]
+ * n grows in some iterations only, and no edge of the jump that looks at it leaves the loop. n is
+ * an index behind the loop, where no jump has cut it: what counts is its range at the head.
+ * 7 goes on to the add, so 8 comes back at most.
+ * @load, @store: n lives in a register, or in 32 bits of a stack slot.
+ */
+#define GROWS(name, load, store, off)					\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r6)							\
+	"r7 = 0;"							\
+	store								\
+"1:"									\
+	"r1 = *(u64 *)(r6 + 0);"					\
+	"if r1 == 0 goto 3f;"						\
+	"if r1 == 1 goto 2f;"						\
+	load								\
+	"if r7 >= 8 goto 2f;"						\
+	"r7 += 1;"							\
+	store								\
+"2:"									\
+	"goto 1b;"							\
+"3:"									\
+	load								\
+	"r6 += r7;"							\
+	"r0 = *(u8 *)(r6 + " #off ");"					\
+	END);								\
+}
+
+OK("a jump of the loop stops a number at 8")
+GROWS(stops_at_8, "", "", 63)
+
+BAD("a jump of the loop stops a number at 8, one byte too far")
+__msg("invalid access to map value, value_size=72 off=72 size=1")
+GROWS(stops_at_8_bad, "", "", 64)
+
+OK("a jump of the loop stops a number in a stack slot at 8")
+GROWS(slot_stops_at_8, "r7 = *(u32 *)(r10 - 16);", "*(u32 *)(r10 - 16) = r7;", 63)
+
+/*
+ *	n = 0; for (i = 0; i < 100000; i++) { if (v[0] != 1 && n < 8) n++; }
+ *	((u8 *)v)[n + off]
+ * The loop has a count and is too long for a walk. The count tells what i is and that there is
+ * no need for may_goto. It has nothing to say about n, which is joined and widened as above.
+ */
+#define COUNTED_GROWS(name, off)					\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r6)							\
+	"r7 = 0;"							\
+	"r8 = 0;"							\
+"1:"									\
+	"r1 = *(u64 *)(r6 + 0);"					\
+	"if r1 == 1 goto 2f;"						\
+	"if r7 >= 8 goto 2f;"						\
+	"r7 += 1;"							\
+"2:"									\
+	"r8 += 1;"							\
+	"if r8 < 100000 goto 1b;"					\
+	"r6 += r7;"							\
+	"r0 = *(u8 *)(r6 + " #off ");"					\
+	END);								\
+}
+
+OK("a jump of a counted loop stops a number at 8")
+__xlated("...")
+__xlated("r8 += 1")
+__xlated("if r8 < 0x186a0 goto pc-{{[0-9]+}}")
+COUNTED_GROWS(counted_stops_at_8, 63)
+
+BAD("a jump of a counted loop stops a number at 8, one byte too far")
+__msg("invalid access to map value, value_size=72 off=72 size=1")
+COUNTED_GROWS(counted_stops_at_8_bad, 64)
+
+/* 8 comes back to the head of the inner loop, and goes from there to the head of the outer one */
+OK("a jump of the inner loop stops a number at 8")
+int inner_loop_stops_at_8(void *ctx)
+{
+	__u32 key = 0;
+	__u64 n = 0, *v;
+	int i, j;
+
+	v = bpf_map_lookup_elem(&map, &key);
+	if (!v)
+		return 0;
+	bpf_for(i, 0, 100) {
+		bpf_for(j, 0, 100) {
+			if (*(volatile __u64 *)v == 1)
+				continue;
+			if (n < 8)
+				v[n++] = 0;
+		}
+	}
+	return v[n];
+}
+
+/*
+ *	err = 0; n = 0; @init
+ *	while (can_loop) { n += 3; @body if (!v[0]) break; }
+ *	@test never();
+ * The loop has may_goto, @off insns long, so it is neither counted nor walked.
+ */
+#define ERR_LOOP(name, init, body, off, test)				\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r7)							\
+	"r6 = 0;"							\
+	"r8 = 0;"							\
+	init								\
+"1:"									\
+	".8byte %[may_goto];"						\
+	"r8 += 3;"							\
+	body								\
+	"r1 = *(u64 *)(r7 + 0);"					\
+	"if r1 != 0 goto 1b;"						\
+	test								\
+	"r0 = r5;"							\
+"2:"									\
+	"r8 &= 7;"							\
+	"r7 += r8;"							\
+	"r0 = *(u8 *)(r7 + 0);"						\
+	END_WITH(MAY_GOTO(off)));					\
+}
+
+/*
+ *	if (v[1]) err = 8; ... if (err != 0 && err != 8) never();
+ * Two states enter the loop. What comes around it with 0 is for the state with 0. Next to 8 it
+ * looks like a number that grew.
+ */
+#define TWO_ENTRIES(name, write)					\
+ERR_LOOP(name, "r1 = *(u64 *)(r7 + 8); if r1 == 0 goto 1f; r6 = 8;",	\
+	 write, 4, "if r6 == 0 goto 2f; if r6 == 8 goto 2f;")
+
+OK("what comes around a loop is for the state that it comes from")
+TWO_ENTRIES(two_entries, "r1 = r8;")
+
+BAD("what comes around a loop is for the state that it comes from, the loop writes")
+__msg("R5 !read_ok")
+TWO_ENTRIES(two_entries_bad, "r6 = r8;")
+
+#define COUNT(name, write)						\
+static __naked __noinline __used int name(void)				\
+{									\
+	asm volatile (							\
+	"r1 = *(u64 *)(r2 + 0);"					\
+	"r1 += 1;"							\
+	"*(u64 *)(r2 + 0) = r1;"					\
+	write								\
+	"r0 = 0;"							\
+	"exit;"								\
+	::: __clobber_all);						\
+}
+
+COUNT(count, "")
+COUNT(count_and_write, "*(u64 *)(r2 + 8) = r1;")
+
+/* The same with bpf_loop(100, @cb, &n, 0) in place of the loop. err is in the slot above n. */
+#define TWO_CALLS(name, cb)						\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r7)							\
+	"r1 = 0;"							\
+	"*(u64 *)(r10 - 16) = r1;"					\
+	"*(u64 *)(r10 - 8) = r1;"					\
+	"r1 = *(u64 *)(r7 + 8);"					\
+	"if r1 == 0 goto 1f;"						\
+	"r1 = 8;"							\
+	"*(u64 *)(r10 - 8) = r1;"					\
+"1:"									\
+	"r1 = 100;"							\
+	"r2 = " #cb " ll;"						\
+	"r3 = r10;"							\
+	"r3 += -16;"							\
+	"r4 = 0;"							\
+	"call %[bpf_loop];"						\
+	"r6 = *(u64 *)(r10 - 8);"					\
+	"if r6 == 0 goto 9f;"						\
+	"if r6 == 8 goto 9f;"						\
+	"r0 = r5;"							\
+	END_WITH(__imm(bpf_loop)));					\
+}
+
+OK("what a callback comes back with is for the state that it comes from")
+TWO_CALLS(two_calls, count)
+
+BAD("what a callback comes back with is for the state that it comes from, it writes")
+__msg("R5 !read_ok")
+TWO_CALLS(two_calls_bad, count_and_write)
+
+/*
+ *	for (i = n; i >= 0; i--) sum += v[i];
+ * A compiler counts in bytes and leaves the loop one element below the array:
+ *	off = n * 8; do { sum += *(u64 *)(v + off); off -= @step; } while (off != -8);
+ * n is a range, so the loop is not counted. Only -8 leaves it. The offset doesn't get past -8
+ * because it is a multiple of 8, and that has to be known at the head of the loop.
+ */
+#define DOWN(name, step)						\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r7)							\
+	"r6 = *(u32 *)(r7 + 64);"					\
+	"if r6 > 7 goto 9f;"						\
+	"r6 <<= 3;"							\
+"1:"									\
+	"r2 = r7;"							\
+	"r2 += r6;"							\
+	"r0 = *(u64 *)(r2 + 0);"					\
+	"r6 += -" #step ";"						\
+	"if r6 != -8 goto 1b;"						\
+	END);								\
+}
+
+OK("down by 8 bytes, left at -8")
+DOWN(down_by_8, 8)
+
+/* 16, 4, -8 is fine, 8, -4 is not */
+BAD("down by 12 bytes, left at -8")
+__msg("min value is negative")
+DOWN(down_by_12, 12)
+
+static __naked __noinline __used void nothing(void)
+{
+	asm volatile ("r0 = 0; exit;" ::: __clobber_all);
+}
+
+/*
+ *	if (v[0]) { n = 7; have = 1; }
+ *	nothing();
+ *	i = 0; do { if (v[0] != 3) i++; } while (i < 8 && v[0] != 2);
+ *	if (have) return n;
+ * (have = 0, n is not set) and (have = 1, n = 7) wait for the function. What comes out of it goes
+ * to both at once, and the insn behind the call is the head of the loop: two walks begin, and no
+ * insn is looked at in between. The state that ends one of them is joined with a state of
+ * that walk. With a state of the other one it would be (have = 0 or 1, n is not set).
+ */
+#define TWO_WALKS(name, test)						\
+__naked void name(void)							\
+{									\
+	asm volatile (							\
+	LOOKUP(r8)							\
+	"r9 = 0;"							\
+	"r1 = *(u32 *)(r8 + 0);"					\
+	"if r1 != 0 goto 1f;"						\
+	"r7 = 0;"							\
+	"goto 2f;"							\
+"1:"									\
+	"r6 = 7;"							\
+	"r7 = 1;"							\
+"2:"									\
+	"call nothing;"							\
+"3:"									\
+	"r1 = *(u32 *)(r8 + 0);"					\
+	"if r1 == 3 goto +1;"						\
+	"r9 += 1;"							\
+	"if r9 > 7 goto 4f;"						\
+	"if r1 == 1 goto 3b;"						\
+	"if r1 != 2 goto 3b;"						\
+"4:"									\
+	test								\
+	"r0 = r6;"							\
+	"exit;"								\
+	END);								\
+}
+
+OK("two walks begin at the same time")
+TWO_WALKS(two_walks_at_once, "if r7 == 0 goto 9f;")
+
+/* n is read when it is not set */
+BAD("two walks begin at the same time, the register is not set")
+__msg("R6 !read_ok")
+TWO_WALKS(two_walks_at_once_bad, "if r7 != 0 goto 9f;")
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
index e8daf791b4c0..12c1cb2f0384 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_latch.c
@@ -21,6 +21,16 @@ struct {
 	__type(value, char[NR]);
 } map SEC(".maps");
 
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+
+/* BTF of a kfunc that only asm refers to is not generated */
+void __kfunc_btf_root(void)
+{
+	bpf_rcu_read_lock();
+	bpf_rcu_read_unlock();
+}
+
 #define OK(text, count)								\
 	SEC("socket") __description("latch: " text) __success __log_level(2)	\
 	__msg("header_count is " count)
@@ -53,7 +63,8 @@ __naked void name(void)								\
 9:	r0 = 0;									\
 	exit;									\
 "	:									\
-	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map), \
+	  __imm(bpf_rcu_read_lock), __imm(bpf_rcu_read_unlock)			\
 	: __clobber_all);							\
 }
 
@@ -87,6 +98,96 @@ LOOP(ptr_up, "r8 = r6; r9 = r6; r9 += 8; r6 += 2040;", PTR_UP(0))
 BAD("pointers, the first one is in front of the value")
 LOOP(ptr_up_from_below, "r8 = r6; r9 = r6; r9 += 7; r6 += 2040;", PTR_UP(-2))
 
+/*
+ * The store behind the jump that counts is not reached in the last iteration. r9 is linear at the
+ * jump, also when it moved in front of it, and has the range of one iteration less.
+ */
+#define AT_TOP(n)	"if w8 == " #n " goto 9f; *(u8 *)(r9 + 0) = r1; r9 += 1; w8 += 1;"
+#define MOVED(n)	"r9 += 1; if w8 == " #n " goto 9f; *(u8 *)(r9 + 0) = r1; w8 += 1;"
+
+OK("a store behind the jump that counts", "5")
+LOOP(behind, "w8 = 0; r9 = r6; r9 += 2044;", AT_TOP(4) "goto 1b;")
+
+BAD("a store behind the jump that counts, one too many")
+LOOP(behind_bad, "w8 = 0; r9 = r6; r9 += 2044;", AT_TOP(5) "goto 1b;")
+
+OK("a store behind the jump that counts, the pointer has moved", "5")
+LOOP(behind_moved, "w8 = 0; r9 = r6; r9 += 2043;", MOVED(4) "goto 1b;")
+
+BAD("a store behind the jump that counts, the pointer has moved too far")
+LOOP(behind_moved_bad, "w8 = 0; r9 = r6; r9 += 2044;", MOVED(4) "goto 1b;")
+
+OK("a store behind the jump that counts two back edges", "5")
+LOOP(behind_two, "w8 = 0; r9 = r6; r9 += 2044;", AT_TOP(4) "if r7 == 7 goto 1b; goto 1b;")
+
+BAD("a store behind the jump that counts two back edges, one too many")
+LOOP(behind_two_bad, "w8 = 0; r9 = r6; r9 += 2044;", AT_TOP(5) "if r7 == 7 goto 1b; goto 1b;")
+
+/*
+ * Both edges of the first jump that counts leave the loop: the one that is not taken at once, the
+ * other one by way of a flag: w2 is 1 or 3. The store behind it is reached in the last iteration.
+ */
+#define BOTH_LEAVE								\
+	"r9 += 1; w8 += 1; call %[bpf_get_prandom_u32]; if r0 == 0 goto 2f;"	\
+	"if w8 >= 2 goto 3f; goto 9f;"						\
+	"3: *(u8 *)(r9 + 0) = r0; w2 = 1; if r7 == 3 goto 4f; w2 = 3;"		\
+	"4: if w2 == 0 goto 1b; goto 9f;"					\
+	"2: if w8 < 4 goto 1b;"
+
+OK("a store behind a jump that leaves on both edges", "[0..4]")
+LOOP(behind_both, "w8 = 0; r9 = r6; r9 += 2043;", BOTH_LEAVE)
+
+BAD("a store behind a jump that leaves on both edges, one too far")
+LOOP(behind_both_bad, "w8 = 0; r9 = r6; r9 += 2044;", BOTH_LEAVE)
+
+/* The statuses of five calls are nearer to the back edge than the jump that counts */
+#define STATUS		"call %[bpf_get_prandom_u32]; if r0 == 0 goto 9f;"
+#define FAR_UP(n)	"if w8 == " #n " goto 9f; w8 += 1;"			\
+			STATUS STATUS STATUS STATUS STATUS "goto 1b;"
+
+OK("behind five jumps that don't count", "[0..2048]")
+LOOP(far_up, "w8 = 0;", FAR_UP(2047))
+
+BAD("behind five jumps that don't count, one too many")
+LOOP(far_up_bad, "w8 = 0;", FAR_UP(2048))
+
+/*
+ * The same in C: each frame appends up to 32 words. The index has no formula. Its range follows
+ * from the number of frames: the iterations are looked at one by one.
+ */
+u64 word;
+
+#define READ(dst)	bpf_probe_read_kernel(&dst, sizeof(dst), &word)
+#define FRAMES(name, max)							\
+int name(void *ctx)								\
+{										\
+	int i, j, n, idx = 0, frames = 0;					\
+	u64 a, b, c, d, e, *out;						\
+	u32 zero = 0;								\
+										\
+	out = bpf_map_lookup_elem(&map, &zero);					\
+	if (!out)								\
+		return -1;							\
+	bpf_for(i, 0, 100) {							\
+		if (frames == max)						\
+			break;							\
+		if (READ(a) || READ(b) || READ(c) || READ(d) || READ(e))	\
+			break;							\
+		n = 32 - (a & 1);						\
+		for (j = 0; j < n; j++)						\
+			out[idx + j] = b + c + d + e;				\
+		idx += n;							\
+		frames++;							\
+	}									\
+	return idx;								\
+}
+
+OK("frames of a stack walker", "[0..9]") __retval(256)
+FRAMES(eight_frames, 8)
+
+BAD("frames of a stack walker, one too many")
+FRAMES(nine_frames, 9)
+
 /* an int goes down to -1, or up from -N to 0: the distance is a 32-bit number */
 #define DOWN_TO_M1	"w8 += -1; if w8 != -1 goto 1b;"
 #define UP_TO_0		"w8 += 1; if w8 != 0 goto 1b;"
@@ -127,6 +228,75 @@ BAD("down, an unsigned and a signed 64-bit test")
 LOOP(down_two_tests, "w8 = 4; r6 += 2043;",
      "w8 += -1; if r8 < 5 goto 2f; if r8 s>= 5 goto 1b; goto 9f; 2: if r8 < 5 goto 1b;")
 
+/*
+ * 'while (buf[i++] && i < N)': the jump tests r9, a copy of the counter r8 from before the
+ * increment. It is made in front of the loop and at the end of each iteration.
+ */
+#define COPY(n)	"r8 += 1; if r9 > " #n " goto 8f; r9 = r8; if r7 != 0 goto 1b; 8:"
+/* one more store, at the index that the counter is behind the loop */
+#define AT_R8	"r5 += r8; *(u8 *)(r5 + 0) = r1;"
+
+OK("a copy of the counter", "[0..2048]")
+LOOP(copy, "r8 = 0; r9 = r8;", COPY(2046))
+
+BAD("a copy of the counter, one too many")
+LOOP(copy_bad, "r8 = 0; r9 = r8;", COPY(2047))
+
+BAD("a copy of the counter, another number at the entry")
+LOOP(copy_other_const, "r8 = 2047; r9 = 0; r6 += 2047;", COPY(2046))
+
+OK("a copy of the counter, a range", "[0..2]")
+LOOP(copy_range, "r5 = r6; r8 = r7; r8 &= 1; r8 += 2045; r9 = r8;", COPY(2045) AT_R8)
+
+BAD("a copy of the counter, another number of the same range at the entry")
+LOOP(copy_other_range,
+     "r5 = r6; r8 = r7; r8 &= 1; r8 += 2045; r9 = r7; r9 >>= 1; r9 &= 1; r9 += 2045;",
+     COPY(2045) AT_R8)
+
+NO_COUNT("a copy of the counter that is one behind")
+LOOP(copy_behind, "r8 = 0; r9 = r8;",
+     "r8 += 1; if r9 > 2046 goto 9f; r9 = r8; r9 += -1; if r7 != 0 goto 1b;")
+
+/* what the jump says about the copy holds for the counter */
+OK("a copy of the counter, the jump bounds both", "[0..2048]")
+LOOP(copy_bounds_both, "r5 = r6; if r7 > 2047 goto 9f; r8 = r7; r9 = r8;",
+     "if r9 > 2046 goto 8f; r8 += 1; r9 = r8; if r7 != 0 goto 1b; 8:" AT_R8)
+
+/*
+ * Behind the jump that counts the counter has moved on and the copy has not: at the store r9 is
+ * r8 - 1, or r8 + 1 when they go down. A test of r8 is no test of r9 there.
+ */
+#define NOT_EQUAL(step, leave, skip, off)					\
+	"r8 += " #step "; if r9 " leave " goto 9f; if r8 " skip " goto 2f;"	\
+	"r4 = r5; r4 += r9; *(u8 *)(r4 " off ") = r1; 2: r9 = r8; if r7 != 0 goto 1b;"
+
+OK("a copy of the counter, one less behind the jump", "[0..2048]")
+LOOP(copy_less, "r5 = r6; r8 = 0; r9 = r8;", NOT_EQUAL(1, "> 2046", "< 2", "- 1"))
+
+BAD("a copy of the counter, one less behind the jump, not equal")
+LOOP(copy_less_bad, "r5 = r6; r8 = 0; r9 = r8;", NOT_EQUAL(1, "> 2046", "< 2", "- 2"))
+
+OK("a copy of the counter, one more behind the jump", "[0..2047]")
+LOOP(copy_more, "r5 = r6; r8 = 2047; r9 = r8;", NOT_EQUAL(-1, "s< 2", "> 2045", "+ 1"))
+
+BAD("a copy of the counter, one more behind the jump, not equal")
+LOOP(copy_more_bad, "r5 = r6; r8 = 2047; r9 = r8;", NOT_EQUAL(-1, "s< 2", "> 2045", "+ 2"))
+
+/*
+ * r4 is what r9 was at the head. It is r9 that has to be equal to the counter when the loop is
+ * entered, not r4. 64 insns more: too long for a walk.
+ */
+#define X4(insns)	insns insns insns insns
+#define OF_COPY		"r3 = r4; r4 = r9; r8 += 1; if r8 > 2047 goto 9f;"	\
+			"r3 = r4; r3 &= 4095; r3 += r5; *(u8 *)(r3 + 0) = r1;"	\
+			X4(X4(X4("r3 = 0;"))) "r9 = r8; if r7 != 0 goto 1b;"
+
+OK("a copy of a copy of the counter", "[0..2048]")
+LOOP(copy_of_copy, "r5 = r6; r8 = 0; r9 = 0; r4 = 0;", OF_COPY)
+
+BAD("a copy of what is no copy of the counter")
+LOOP(copy_of_no_copy, "r5 = r6; r8 = 0; r9 = r7; r9 &= 4095; r4 = 0;", OF_COPY)
+
 /* 'while (n--)': the jump that counts sets a flag, the jump that leaves the loop tests it */
 #define FLAG(mid)	"w8 += 1; w2 = 1; if w8 == 0 goto 2f; w2 = 0; 2: " mid " if w2 != 1 goto 1b;"
 
@@ -136,6 +306,15 @@ LOOP(flag, "w8 = -2048;", FLAG("r3 = r6;"))
 BAD("a flag, one too many")
 LOOP(flag_bad, "w8 = -2049;", FLAG("r3 = r6;"))
 
+/* A flag leaves the loop later: what is behind the jump runs in the last iteration too */
+#define STORE		"*(u8 *)(r9 + 0) = r1; r9 += 1;"
+
+OK("a flag, a store behind the jump that counts", "[0..2048]")
+LOOP(flag_behind, "w8 = -2048; r9 = r6;", FLAG(STORE))
+
+BAD("a flag, a store behind the jump that counts, one too far")
+LOOP(flag_behind_bad, "w8 = -2048; r9 = r6; r9 += 1;", FLAG(STORE))
+
 OK("a flag, a store behind it", "[0..2048]")
 LOOP(flag_store, "w8 = -2048;",
      "w8 += 1; w2 = 1; *(u32 *)(r10 - 4) = r8; if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
@@ -144,7 +323,84 @@ OK("a flag, a call behind it", "[0..2048]")
 LOOP(flag_call, "w8 = -2048;",
      "w8 += 1; w9 = 1; if w8 == 0 goto 2f; w9 = 0; 2: call %[bpf_get_prandom_u32]; if w9 != 1 goto 1b;")
 
-/* in these the loop goes on when the counter gets to 0 */
+OK("a flag that is set in front of another jump", "[0..2048]")
+LOOP(flag_far, "w8 = -2048;",
+     "w8 += 1; w2 = 1; if r7 != 0 goto 3f; r3 = r6;"
+     "3: if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
+
+/*
+ * 'do { ...; more = ++i < N && cond; work; } while (more);': a compiler keeps the flag when the
+ * work is more than some 10 insns. The flag has two reasons to be 0.
+ */
+#define MORE(count, mid)							\
+	count "w9 = 1; if w7 != 0 goto 2f; w9 = 0; 2: " mid "w9 &= 1; if w9 != 0 goto 1b;"
+#define UP_TO(n)	"w8 += 1; w9 = 0; if w8 > " #n " goto 2f;"
+#define WORK4		"r1 = *(u8 *)(r10 - 4); r1 += 1; *(u8 *)(r10 - 4) = r1; r1 = r6;"
+#define WORK20		WORK4 WORK4 WORK4 WORK4 WORK4
+#define RND		"call %[bpf_get_prandom_u32];"
+
+OK("more: work", "[0..2048]")
+LOOP(more_work, "w8 = 0;", MORE(UP_TO(2047), WORK20))
+
+BAD("more: work, one too many")
+LOOP(more_work_bad, "w8 = 0;", MORE(UP_TO(2048), WORK20))
+
+OK("more: work with a jump in it", "[0..2048]")
+LOOP(more_jump, "w8 = 0;", MORE(UP_TO(2047), WORK4 "if r7 > 5 goto 4f;" WORK20 "4:"))
+
+OK("more: work with calls in it", "[0..2048]")
+LOOP(more_calls, "w8 = 0;", MORE(UP_TO(2047), RND WORK20 RND))
+
+OK("more: the counter is spilled", "[0..2048]")
+LOOP(more_spill, "r8 = 0; *(u64 *)(r10 - 16) = r8;",
+     MORE("r8 = *(u64 *)(r10 - 16); w8 += 1; w9 = 0; *(u64 *)(r10 - 16) = r8;"
+	  "if w8 > 2047 goto 2f;", RND "w8 = w0;" WORK20 RND))
+
+#define NOT_TAKEN(n)	"w8 += 1; if w8 < " #n " goto 4f; w9 = 0; goto 2f; 4:"
+
+OK("more: the edge that is not taken leaves", "[0..2048]")
+LOOP(more_not_taken, "w8 = 0;", MORE(NOT_TAKEN(2048), WORK20))
+
+BAD("more: the edge that is not taken leaves, one too many")
+LOOP(more_not_taken_bad, "w8 = 0;", MORE(NOT_TAKEN(2049), WORK20))
+
+#define JUMP		"if r7 > 5 goto 4f;" WORK4 "4:"
+#define JUMP8		JUMP JUMP JUMP JUMP JUMP JUMP JUMP JUMP
+
+OK("more: work with 40 jumps in it", "[0..2048]")
+LOOP(more_jumps, "w8 = 0;", MORE(UP_TO(2047), JUMP8 JUMP8 JUMP8 JUMP8 JUMP8))
+
+OK("more: behind five jumps that don't count", "[0..2048]")
+LOOP(more_far, "w8 = 0;", MORE(UP_TO(2047), STATUS STATUS STATUS STATUS STATUS))
+
+BAD("more: behind five jumps that don't count, one too many")
+LOOP(more_far_bad, "w8 = 0;", MORE(UP_TO(2048), STATUS STATUS STATUS STATUS STATUS))
+
+OK("more: the flag is 0 or 2 where two paths meet", "[0..2048]")
+LOOP(more_even, "w8 = 0;", MORE(UP_TO(2047), "if r7 > 5 goto 4f; w9 += 2; 4:"))
+
+OK("a flag, a jump on the way out knows a constant", "[0..2048]")
+LOOP(flag_const, "w8 = -2048;",
+     "w8 += 1; w2 = 1; if w8 == 0 goto 2f; w2 = 0;"
+     "2: w3 = 0; if w2 != 0 goto 3f; goto 1b; 3: if w3 == 2 goto 1b;")
+
+/* 'do { ...; err = -E2BIG; if (++i < N) err = f(); work; } while (!err);' */
+#define ERR(last, other, test)							\
+	"w8 += 1; w9 = " #last "; if w8 > 2047 goto 2f; w9 = " #other "; 2: " WORK20 test
+
+OK("an error code", "[0..2048]")
+LOOP(err, "w8 = 0;", ERR(-7, 0, "if w9 == 0 goto 1b;"))
+
+OK("an error code, a signed test", "[0..2048]")
+LOOP(err_signed, "w8 = 0;", ERR(-7, 0, "if w9 s>= 0 goto 1b;"))
+
+OK("a size", "[0..2048]")
+LOOP(size, "w8 = 0;", ERR(128, 0, "if w9 <= 63 goto 1b;"))
+
+OK("a bit of a size", "[0..2048]")
+LOOP(size_bit, "w8 = 0;", ERR(130, 0, "if w9 & 128 goto 9f; goto 1b;"))
+
+/* in these the loop goes on when the counter gets to its last value */
 NO_COUNT("a flag that a call overwrites")
 LOOP(flag_call_r0, "w8 = -2048;",
      "w8 += 1; w0 = 1; if w8 == 0 goto 2f; w0 = 0; 2: call %[bpf_get_prandom_u32]; if w0 != 1 goto 1b;")
@@ -158,6 +414,46 @@ LOOP(flag_changed, "w8 = -2048;", FLAG("w2 &= w7;"))
 NO_COUNT("a flag that is set again on one path")
 LOOP(flag_other_jump, "w8 = -2048;", FLAG("if r7 != 0 goto 4f; w2 = 0; 4:"))
 
+NO_COUNT("a flag that is set again behind the jump that counts")
+LOOP(flag_set_behind, "w8 = 0;",
+     "w8 += 1; w3 = 0; if w8 > 2047 goto 2f; w3 = 5; goto 3f; 2: w3 = 2;"
+     "3: if w3 == 0 goto 9f; if w8 <= w7 goto 1b;")
+
+NO_COUNT("a flag that a load overwrites")
+LOOP(flag_load, "w8 = -2048;", FLAG("r2 = *(u8 *)(r10 - 4);"))
+
+NO_COUNT("the one number of its range that goes on")
+LOOP(err_near, "w8 = 0;", ERR(101, w7, "if w9 == 101 goto 1b;"))
+
+NO_COUNT("a bit that not all numbers of the range have")
+LOOP(size_other_bit, "w8 = 0;", ERR(130, w7, "if w9 & 2 goto 1b;"))
+
+NO_COUNT("a size that is changed")
+LOOP(size_changed, "w8 = 0;", ERR(128, w7, "w9 &= 1; if w9 <= 63 goto 1b;"))
+
+NO_COUNT("an error code of 64 bits, a 64-bit test")
+LOOP(err_wide, "w8 = 0;",
+     "w8 += 1; if w8 > 2047 goto 3f; r9 = r7; r9 <<= 32; r9 s>>= 32; goto 2f;"
+     "3: r9 = -7; 2: if r9 s< 0 goto 1b;")
+
+NO_COUNT("more: a back edge that doesn't test the flag")
+LOOP(more_back_edge, "w8 = 0;",
+     MORE(UP_TO(2047), "if w7 == 3 goto 5f;") "goto 9f; 5: if w8 == 2048 goto 1b;")
+
+/* A nested loop can get may_goto. It goes where a jump of that loop goes, whatever the flag is. */
+NO_COUNT("more: the way out leads through a nested loop")
+LOOP(more_nested, "w8 = 0; call %[bpf_rcu_read_lock];",
+     "w8 += 1; w9 = 0; if w8 > 2047 goto 2f; w9 = 1;"
+     "2: if w9 == 5 goto 4f; if w9 != 0 goto 3f; if w9 == 0 goto 8f; 3: goto 2b;"
+     "4: if w8 <= w7 goto 1b; 8: call %[bpf_rcu_read_unlock];")
+
+/* May_goto at the head of the first loop goes to 7 with w9 = 1. The program gets there with 0. */
+NO_COUNT("more: a flag that is a constant without may_goto")
+LOOP(more_stale, "call %[bpf_rcu_read_lock];" RND "w4 = w0; w8 = 0; w9 = 0;"
+     "5: if w4 == 0 goto 6f; w9 = 0; if w4 == 1 goto 7f; w9 = 1; goto 5b; 6: w9 = 1; goto 5b; 7:",
+     "w5 = w9; w8 += 1; if w8 > 2047 goto 2f; w5 = 1;"
+     "2: if w8 > w7 goto 8f; if w5 != 0 goto 1b; 8: call %[bpf_rcu_read_unlock];")
+
 NO_COUNT("a flag, another way to the jump that counts")
 LOOP(flag_two_ways, "w8 = -2048;",
      "w8 += 1; w2 = 0; if r7 != 0 goto 3f; w2 = 1; 3: if w8 == 0 goto 2f; w2 = 0; 2: if w2 != 1 goto 1b;")
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_need.c b/tools/testing/selftests/bpf/progs/verifier_loop_need.c
new file mode 100644
index 000000000000..281f94bfeb47
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_need.c
@@ -0,0 +1,372 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A counted loop changes a number in a way that has no formula:
+ *	for (i = 0; i < 1000; i++) x = x * 3 + 1;
+ * The verifier can go through the iterations one by one to find out what x is. It does so only
+ * if something ahead looks at x: a jump, an access to memory, a call. Otherwise what comes back
+ * to the head of the loop is joined with what entered and widened: the loop is looked at
+ * a few times, however many iterations it has.
+ *
+ * What looks at a number is found ahead of the main pass, and it is a guess. A use that is missed
+ * rejects a program that would load after a walk. It never loads a program that is not safe.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* a map value of 64 bytes that holds nothing but numbers */
+u64 vals[8];
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, 1);
+} arena SEC(".maps");
+
+#define OK(text)	SEC("socket") __description("need: " text) __success
+#define BAD(text)	SEC("socket") __description("need: " text) __failure
+#define NOT_WALKED	__log_level(2) __msg("nothing looks at it")
+#define WALKED		__log_level(2) __not_msg("nothing looks at it")
+
+/* The read is not allowed */
+#define UNLESS(cond)	"if " cond " goto +1; r0 = *(u64 *)(r10 + 8);"
+
+/* r6 is 1 or 2, and is multiplied by 3 in each of @n iterations. @use comes behind the loop. */
+#define TIMES_3_IN(init, n, use)					\
+	asm volatile (							\
+	init								\
+	"call %[bpf_get_prandom_u32];"					\
+	"r6 = r0;"							\
+	"r6 &= 1;"							\
+	"r6 += 1;"							\
+	"r7 = 0;"							\
+"1:	r6 *= 3;"							\
+	"r7 += 1;"							\
+	"if r7 < " #n " goto 1b;"					\
+	use								\
+	"exit;"								\
+	:: __imm(bpf_get_prandom_u32), __imm_addr(vals) : __clobber_all)
+#define TIMES_3(n, use)	TIMES_3_IN("", n, use "r0 = 0;")
+
+OK("a number that is only stored is not followed")
+NOT_WALKED __msg("processed 17 insns")
+__naked void stored_only(void)
+{
+	TIMES_3(1000, "*(u64 *)(r10 - 8) = r6;");
+}
+
+OK("a jump looks at the number")
+WALKED
+__naked void jump(void)
+{
+	TIMES_3(5, UNLESS("r6 <= 486"));
+}
+
+BAD("a jump looks at the number, which is too large")
+__msg("invalid read from stack")
+__naked void jump_too_large(void)
+{
+	TIMES_3(5, UNLESS("r6 <= 485"));
+}
+
+OK("the number is an offset in a map value that holds numbers")
+WALKED
+__naked void value_offset(void)
+{
+	/* 27 or 54 */
+	TIMES_3(3, "r1 = %[vals] ll; r1 += r6; r0 = *(u8 *)(r1 + 9);");
+}
+
+#define VALUE_AT(reg, off)	"r1 = %[vals] ll; r1 += " #reg "; r0 = *(u8 *)(r1 + " #off ");"
+
+__naked __noinline __used
+static unsigned long value_at_plus_9(void)
+{
+	asm volatile ("r2 = r1;" VALUE_AT(r2, 9) "exit;" :: __imm_addr(vals) : __clobber_all);
+}
+
+OK("the number is an offset in a map value in a function")
+WALKED
+__naked void value_offset_argument(void)
+{
+	TIMES_3(3, "r1 = r6; call value_at_plus_9;");
+}
+
+__naked __noinline __used
+static unsigned long times_27(void)
+{
+	TIMES_3_IN("", 3, "r0 = r6;");
+}
+
+OK("what a function returns is an offset in a map value")
+WALKED
+__naked void value_offset_returned(void)
+{
+	asm volatile ("call times_27;" VALUE_AT(r0, 9) "exit;"
+		      :: __imm_addr(vals) : __clobber_all);
+}
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+struct pair {
+	__u64 lo;	/* r0 */
+	__u64 hi;	/* r2 */
+};
+
+__naked __noinline __used
+static struct pair zero_and_times_27(void)
+{
+	TIMES_3_IN("", 3, "r0 = 0; r2 = r6;");
+}
+
+OK("the second half of what a function returns is an offset in a map value")
+__load_if_JITed() WALKED
+__naked void value_offset_returned_in_r2(void)
+{
+	asm volatile ("call zero_and_times_27;" VALUE_AT(r2, 9) "exit;"
+		      :: __imm_addr(vals) : __clobber_all);
+}
+#endif
+
+__naked __noinline __used
+static unsigned long times_27_there(void)
+{
+	TIMES_3_IN("r8 = r1;", 3, "*(u64 *)(r8 + 0) = r6; r0 = 0;");
+}
+
+#define TIMES_27_AT_FP_8	"r1 = r10; r1 += -8; call times_27_there; r2 = *(u64 *)(r10 - 8);"
+
+OK("a function stores the number in the stack of its caller")
+WALKED
+__naked void stored_for_the_caller(void)
+{
+	asm volatile (TIMES_27_AT_FP_8 VALUE_AT(r2, 9) "exit;"
+		      :: __imm_addr(vals) : __clobber_all);
+}
+
+OK("the number is an offset in the stack")
+WALKED
+__naked void stack_offset(void)
+{
+	/* fp-9 or fp-18 */
+	TIMES_3(2, "*(u64 *)(r10 - 8) = r7; *(u64 *)(r10 - 16) = r7; *(u64 *)(r10 - 24) = r7;"
+		   "r6 = -r6; r1 = r10; r1 += r6; r0 = *(u8 *)(r1 + 0);");
+}
+
+__naked __noinline __used
+static unsigned long times_9(void)
+{
+	TIMES_3_IN("", 2, "r0 = r6;");
+}
+
+/* fp-9 or fp-18, plus @off */
+#define STACK_AT_MINUS_R0(off)						\
+	"*(u64 *)(r10 - 8) = r0; *(u64 *)(r10 - 16) = r0; *(u64 *)(r10 - 24) = r0;"	\
+	"r0 = -r0; r1 = r10; r1 += r0; r0 = *(u8 *)(r1 + " #off ");"
+
+OK("what a function returns is an offset in the stack")
+WALKED
+__naked void stack_offset_returned(void)
+{
+	asm volatile ("call times_9;" STACK_AT_MINUS_R0(0) "exit;" ::: __clobber_all);
+}
+
+/* r6 is 1, and is multiplied by 3 in each of 2 iterations */
+#define NINE(use)							\
+	asm volatile (							\
+	"r6 = 1;"							\
+	"r7 = 0;"							\
+"1:	r6 *= 3;"							\
+	"r7 += 1;"							\
+	"if r7 < 2 goto 1b;"						\
+	use								\
+	"r0 = 0;"							\
+	"exit;"								\
+	:: __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard),	\
+	   __imm_addr(ringbuf) : __clobber_all)
+
+OK("the number is part of a size that has to be a constant")
+WALKED
+__naked void constant_argument(void)
+{
+	NINE("r1 = %[ringbuf] ll; r2 = 7; r2 += r6; r3 = 0; call %[bpf_ringbuf_reserve];"
+	     "if r0 == 0 goto +3; r1 = r0; r2 = 0; call %[bpf_ringbuf_discard];");
+}
+
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+/* Nothing checks where a pointer to arena points to */
+OK("an address in arena is not followed")
+NOT_WALKED
+__naked void arena_address(void)
+{
+	asm volatile (
+	"r1 = %[arena] ll;"
+	"r6 = 1;"
+	"r7 = 0;"
+"1:	r1 = addr_space_cast(r6, 0x0, 0x1);"
+	"r0 = *(u64 *)(r1 + 0);"
+	"r6 *= 3;"
+	"r7 += 1;"
+	"if r7 < 1000 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(arena) : __clobber_all);
+}
+#endif
+
+/*
+ *	p = NULL; for (i = 0; i < 3; i++) { if (have) n += *p; p = &val; have = 1; }
+ * No jump and no add looks at p.
+ */
+OK("the base of a load is 0 in the first iteration")
+WALKED
+__naked void base_of_a_load(void)
+{
+	asm volatile (
+	"r6 = %[vals] ll;"
+	"r7 = 0;"
+	"r8 = 0;"
+	"r9 = 0;"
+"1:	if r9 == 0 goto 2f;"
+	"r0 = *(u64 *)(r8 + 0);"
+"2:	r8 = r6;"
+	"r9 = 1;"
+	"r7 += 1;"
+	"if r7 < 3 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(vals) : __clobber_all);
+}
+
+/* r8 goes on by 0 or 8 bytes. Nothing tells how far but the iterations. */
+OK("a pointer that moves is followed")
+WALKED
+__naked void pointer(void)
+{
+	asm volatile (
+	"r8 = %[vals] ll;"
+	"r7 = 0;"
+"1:	r1 = r7;"
+	"r1 &= 1;"
+	"r1 <<= 3;"
+	"r8 += r1;"
+	"r7 += 1;"
+	"if r7 < 14 goto 1b;"
+	"r0 = *(u64 *)(r8 + 0);"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(vals) : __clobber_all);
+}
+
+/* The sets have 64 slots of the stack. What is deeper is followed. */
+OK("a jump looks at a slot below fp-512")
+__load_if_large_stack()
+WALKED
+__naked void deep_slot(void)
+{
+	asm volatile (
+	"r6 = 2;"
+	"*(u64 *)(r10 - 520) = r6;"
+	"r7 = 0;"
+"1:	r6 = *(u64 *)(r10 - 520);"
+	"r6 *= 3;"
+	"*(u64 *)(r10 - 520) = r6;"
+	"r7 += 1;"
+	"if r7 < 5 goto 1b;"
+	"r6 = *(u64 *)(r10 - 520);"
+	UNLESS("r6 == 486")
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* The iterator is not a number. It is at its end after 3 of 5 iterations. */
+OK("a slot that holds an iterator is followed")
+WALKED
+__naked void iterator_slot(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"r2 = 0;"
+	"r3 = 3;"
+	"call %[bpf_iter_num_new];"
+	"r7 = 0;"
+"1:	r1 = r10;"
+	"r1 += -8;"
+	"call %[bpf_iter_num_next];"
+	"r7 += 1;"
+	"if r7 < 5 goto 1b;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call %[bpf_iter_num_destroy];"
+	"r0 = 0;"
+	"exit;"
+	:: __imm(bpf_iter_num_new), __imm(bpf_iter_num_next), __imm(bpf_iter_num_destroy)
+	: __clobber_all);
+}
+
+/*
+ *	for (i = 0; i < n; i++) { if (i != skip) sum += prev; prev = (x ^ i) * 3; }
+ * prev is not set when the loop is entered. A compiler doesn't set what is not read.
+ */
+#define SUM_OF_PREV(skip, n, use)					\
+	asm volatile (							\
+	"call %[bpf_get_prandom_u32];"					\
+	"r6 = r0;"							\
+	"r7 = 0;"							\
+	"r9 = 0;"							\
+"1:	if r7 == " #skip " goto 2f;"					\
+	"r9 += r8;"							\
+"2:	r8 = r6;"							\
+	"r8 ^= r7;"							\
+	"r8 *= 3;"							\
+	"r7 += 1;"							\
+	"if r7 < " #n " goto 1b;"					\
+	use								\
+	"r0 = 0;"							\
+	"exit;"								\
+	:: __imm(bpf_get_prandom_u32), __imm_addr(vals) : __clobber_all)
+
+OK("what is not set at the entry is set from the second iteration on")
+NOT_WALKED __msg("processed 22 insns")
+__naked void set_in_the_first_iteration(void)
+{
+	SUM_OF_PREV(0, 40, "*(u64 *)(r10 - 8) = r9;");
+}
+
+BAD("what is not set at the entry is read in the first iteration")
+__msg("R8 !read_ok")
+__naked void read_in_the_first_iteration(void)
+{
+	SUM_OF_PREV(1, 40, "*(u64 *)(r10 - 8) = r9;");
+}
+
+/* What a call returns is made anew in every iteration, like what is loaded */
+OK("a long loop carries what a call returns")
+int call_in_a_long_loop(void *ctx)
+{
+	u64 sum = 0;
+	u32 x = 0;
+	int i;
+
+	for (i = 0; i < 20000; i++) {
+		sum += vals[x];
+		x = bpf_get_prandom_u32() & 7;
+	}
+	return sum;
+}
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+	bpf_iter_num_new(0, 0, 0);
+	bpf_iter_num_next(0);
+	bpf_iter_num_destroy(0);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_walk.c b/tools/testing/selftests/bpf/progs/verifier_loop_walk.c
new file mode 100644
index 000000000000..1a6180aceacc
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_walk.c
@@ -0,0 +1,306 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A loop that is not counted is looked at iteration by iteration for a while: some thousand insns.
+ * Each such walk counts the insns that are looked at in it, not those of the walks next to it.
+ */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, u32);
+	__type(value, char[2048]);
+} map SEC(".maps");
+
+/* 0 and 2 stay apart up to the jump that looks at them */
+#define SET(slot)								\
+	"call %[bpf_get_prandom_u32]; r1 = 0; if r0 == 0 goto +1; r1 = 2;"	\
+	"*(u64 *)(r10 - " #slot ") = r1;"
+#define TEST(slot)	"r1 = *(u64 *)(r10 - " #slot "); if r1 == 1 goto 9f;"
+#define PAD4		"r3 = r6; r3 += 1; r4 = r3; r4 += 1;"
+
+/*
+ * 32 states enter the loop one after the other. The counter has 8 bits, which has no formula. The
+ * offset is 1, 2, 4 ...: only a walk tells that it stays in the value. One walk takes 300 insns.
+ */
+#define WALKS(name, trips)							\
+__naked void name(void)								\
+{										\
+	asm volatile ("								\
+	r1 = 0;									\
+	*(u32 *)(r10 - 4) = r1;							\
+	r2 = r10;								\
+	r2 += -4;								\
+	r1 = %[map] ll;								\
+	call %[bpf_map_lookup_elem];						\
+	if r0 == 0 goto 9f;							\
+	r6 = r0;								\
+"	SET(16) SET(24) SET(32) SET(40) SET(48) "				\
+	w8 = 0;									\
+	r9 = 1;									\
+1:	r1 = r6;								\
+	r1 += r9;								\
+	r2 = 0;									\
+	*(u8 *)(r1 + 0) = r2;							\
+	r9 <<= 1;								\
+"	PAD4 PAD4 PAD4 PAD4 "							\
+	w8 += 1;								\
+	w2 = w8;								\
+	w2 &= 255;								\
+	if w2 < " #trips " goto 1b;						\
+"	TEST(16) TEST(24) TEST(32) TEST(40) TEST(48) "				\
+9:	r0 = 0;									\
+	exit;									\
+"	:									\
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __clobber_all);							\
+}
+
+SEC("socket")
+__description("walk: 32 walks at a time")
+__success
+WALKS(walks_32, 11)
+
+/*
+ *	for (i = 7; i >= 0; i--) { if (i >= n) continue; a[i] = 0; }
+ * A compiler tests 'i >= n' in front of the loop too and jumps into the middle of it. Such a loop
+ * has no count. It's walked to its end whichever way its jumps go: no may_goto is added.
+ */
+SEC("socket")
+__description("walk: a loop with two entries")
+__success __retval(0)
+__arch_x86_64
+__arch_arm64
+__xlated("w3 += -1")
+__xlated("if w3 s< 0x0 goto pc+2")
+__xlated("if w3 < w2 goto pc-6")
+__xlated("goto pc-4")
+__xlated("exit")
+__naked void two_entries(void)
+{
+	asm volatile ("								\
+	call %[bpf_get_prandom_u32];						\
+	r2 = r0;								\
+	r2 &= 7;								\
+	r0 = 0;									\
+	w3 = 7;									\
+	if w3 >= w2 goto 2f;							\
+1:	r1 = r10;								\
+	r1 += r3;								\
+	*(u8 *)(r1 - 8) = r0;							\
+2:	w3 += -1;								\
+	if w3 s< 0 goto 3f;							\
+	if w3 < w2 goto 1b;							\
+	goto 2b;								\
+3:	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	for (i = 0; (u8)i < 11; i++) { if (!rnd()) break; v[1 << i] = 0; }
+ * One jump that leaves the loop goes both ways in each iteration, the other one is decided in
+ * each iteration: the loop counts. The walk goes on. Only a walk tells what the offset is.
+ */
+SEC("socket")
+__description("walk: goes on while an iteration decides a jump that leaves the loop")
+__success
+__naked void walk_goes_on(void)
+{
+	asm volatile ("								\
+	r1 = 0;									\
+	*(u32 *)(r10 - 4) = r1;							\
+	r2 = r10;								\
+	r2 += -4;								\
+	r1 = %[map] ll;								\
+	call %[bpf_map_lookup_elem];						\
+	if r0 == 0 goto 9f;							\
+	r6 = r0;								\
+	w8 = 0;									\
+	r9 = 1;									\
+1:	call %[bpf_get_prandom_u32];						\
+	if r0 == 0 goto 9f;							\
+	r1 = r6;								\
+	r1 += r9;								\
+	*(u8 *)(r1 + 0) = r0;							\
+	r9 <<= 1;								\
+	w8 += 1;								\
+	w2 = w8;								\
+	w2 &= 255;								\
+	if w2 < 11 goto 1b;							\
+9:	r0 = 0;									\
+	exit;									\
+"	:									\
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __clobber_all);
+}
+
+static __naked __noinline __used int zero_or_one(void)
+{
+	asm volatile ("								\
+	call %[bpf_get_prandom_u32];						\
+	r0 &= 1;								\
+	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	for (i = 0; i < 2; i++) off += zero_or_one();
+ * A compiler makes a flag of i: there is nothing to count. Both trips are walked although the loop
+ * calls a function. Only a walk tells that off is 2 at most.
+ */
+#define TWO_TRIPS(name, size)							\
+__naked void name(void)								\
+{										\
+	asm volatile ("								\
+	r6 = 1;									\
+	r7 = 0;									\
+1:	call zero_or_one;							\
+	r7 += r0;								\
+	r1 = r6;								\
+	r6 = 0;									\
+	if r1 != 0 goto 1b;							\
+	r1 = r10;								\
+	r1 += r7;								\
+	r0 = 0;									\
+	*(u8 *)(r1 - " #size ") = r0;						\
+	exit;									\
+"	::: __clobber_all);							\
+}
+
+SEC("socket")
+__description("walk: two trips of a loop with a call")
+__success __retval(0)
+TWO_TRIPS(two_trips, 3)
+
+SEC("socket")
+__description("walk: two trips of a loop with a call, off by one")
+__failure __msg("invalid variable-offset write to stack R1")
+TWO_TRIPS(two_trips_off_by_one, 2)
+
+/*
+ *	while (rnd()) n++;
+ * The second iteration decides nothing. The walk gives up at once: less than 100 insns.
+ */
+SEC("socket")
+__description("walk: gives up when an iteration decides no jump that leaves the loop")
+__success __log_level(4) __msg("processed {{..}} insns")
+__naked void walk_gives_up(void)
+{
+	asm volatile ("								\
+	r6 = 0;									\
+1:	call %[bpf_get_prandom_u32];						\
+	if r0 == 0 goto 2f;							\
+	r6 += 1;								\
+	goto 1b;								\
+2:	r0 = 0;									\
+	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	do { n++; more = rnd(); ... } while (more);
+ * A compiler keeps the flag. States with 0 and with 1 are not joined in front of its test, so the
+ * test is decided in each of them. That decides nothing: the walk gives up as above.
+ */
+SEC("socket")
+__description("walk: gives up when only a flag that went both ways decides the jump")
+__success __log_level(4) __msg("processed {{..}} insns")
+__naked void walk_gives_up_flag(void)
+{
+	asm volatile ("								\
+	r6 = 0;									\
+1:	call %[bpf_get_prandom_u32];						\
+	r6 += 1;								\
+	w7 = 1;									\
+	if r0 != 0 goto 2f;							\
+	w7 = 0;									\
+2:	w7 &= 1;								\
+	if w7 != 0 goto 1b;							\
+	r0 = 0;									\
+	exit;									\
+"	:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ *	for (off = 1; !done; off <<= 1) { v[off] = 0; done = off > 512; if (rnd()) n++; }
+ * done is known in every iteration. Another flag went both ways, and was tested before it: the
+ * walk goes on. Only a walk tells what the offset is.
+ */
+SEC("socket")
+__description("walk: goes on when the flag that leaves the loop is known")
+__success
+__naked void walk_goes_on_flag(void)
+{
+	asm volatile ("								\
+	r1 = 0;									\
+	*(u32 *)(r10 - 4) = r1;							\
+	r2 = r10;								\
+	r2 += -4;								\
+	r1 = %[map] ll;								\
+	call %[bpf_map_lookup_elem];						\
+	if r0 == 0 goto 9f;							\
+	r6 = r0;								\
+	r9 = 1;									\
+1:	r1 = r6;								\
+	r1 += r9;								\
+	r2 = 0;									\
+	*(u8 *)(r1 + 0) = r2;							\
+	w8 = 1;									\
+	if r9 > 512 goto 2f;							\
+	w8 = 0;									\
+2:	r9 <<= 1;								\
+	call %[bpf_get_prandom_u32];						\
+	w7 = 1;									\
+	if r0 != 0 goto 3f;							\
+	w7 = 0;									\
+3:	if w7 == 0 goto 4f;							\
+	r2 = 0;									\
+4:	if w8 == 0 goto 1b;							\
+9:	r0 = 0;									\
+	exit;									\
+"	:									\
+	: __imm(bpf_map_lookup_elem), __imm(bpf_get_prandom_u32), __imm_addr(map) \
+	: __clobber_all);
+}
+
+static __naked __noinline __used int long_loop(void)
+{
+	asm volatile ("								\
+	w3 = 0;									\
+1:	w3 += 1;								\
+	w2 = w3;								\
+	w2 &= 0xffff;								\
+	if w2 < 5000 goto 1b;							\
+	r0 = 0;									\
+	*(u64 *)(r1 + 0) = r0;							\
+	exit;									\
+"	::: __clobber_all);
+}
+
+/*
+ * The walk doesn't get to the end of the loop. The function is looked at three times, for three
+ * pointers. There is no walk the second and the third time: less than 10000 insns in all.
+ */
+SEC("socket")
+__description("walk: no other try for a loop that is too long")
+__success __log_level(4) __msg("processed {{....}} insns")
+__naked void walk_fails_once(void)
+{
+	asm volatile ("								\
+	r1 = r10;								\
+	r1 += -8;								\
+	call long_loop;								\
+	r1 = r10;								\
+	r1 += -16;								\
+	call long_loop;								\
+	r1 = r10;								\
+	r1 += -24;								\
+	call long_loop;								\
+	r0 = 0;									\
+	exit;									\
+"	::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index d89ffe90c6bc..c1aede6fa4ae 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -4410,6 +4410,136 @@ __naked void post_cond_jlt32_copy(void)
 	: __clobber_all);
 }
 
+/*
+ * The loop loads r6 again in every iteration. It's 0 at the entry and a byte in all iterations
+ * but the first one: the first iteration is verified on its own, the state that comes back to
+ * the head stands for the rest.
+ */
+SEC("socket")
+__success __retval(0)
+__log_level(2)
+__msg("header_count is 10 ")
+__msg("can't widen r6, expr is _")
+__not_msg("widening r8")
+__msg("clamping r8 to 0..9 step 1")
+__naked void index_loaded_in_previous_iteration(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r7 = r0;					\
+	r6 = 0;						\
+	r8 = 0;						\
+1:	r1 = r7;					\
+	r1 += r6;					\
+	r0 = *(u8 *)(r1 + 0);				\
+	r6 = *(u8 *)(r7 + 0);				\
+	r8 += 1;					\
+	if r8 < 10 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * What the first iteration makes of r6 is a guess about the rest. A byte in the first
+ * iteration, 16 bits in the others: the state that comes back to the head next is not within it.
+ */
+SEC("socket")
+__failure __msg("R1 unbounded memory access")
+__naked void first_iteration_loads_less(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 4f;				\
+	r7 = r0;					\
+	r6 = 0;						\
+	r8 = 0;						\
+1:	r1 = r7;					\
+	r1 += r6;					\
+	r0 = *(u8 *)(r1 + 0);				\
+	if r8 != 0 goto 2f;				\
+	r6 = *(u8 *)(r7 + 0);				\
+	goto 3f;					\
+2:	r6 = *(u16 *)(r7 + 0);				\
+3:	r8 += 1;					\
+	if r8 < 10 goto 1b;				\
+4:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * Every iteration takes a reference, only the last one is released. The state that comes back
+ * to the head of the loop has a reference that nothing holds.
+ */
+SEC("socket")
+__failure __msg("Unreleased reference")
+__naked void loop_takes_reference_every_time(void)
+{
+	asm volatile (
+	"r6 = 0;"
+"1:"
+	"r1 = %[ringbuf] ll;"
+	"r2 = 8;"
+	"r3 = 0;"
+	"call %[bpf_ringbuf_reserve];"
+	"if r0 == 0 goto 2f;"
+	"r8 = r0;"
+	"r6 += 1;"
+	"if r6 < 10 goto 1b;"
+	"r1 = r8;"
+	"r2 = 0;"
+	"call %[bpf_ringbuf_discard];"
+"2:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard), __imm_addr(ringbuf)
+	: __clobber_all);
+}
+
+/*
+ * The first iteration that has something to say takes a reference, the rest use it. The state
+ * that comes back to the head has a reference more than the state that entered, once.
+ */
+__u64 nr_vals, vals[64];
+
+SEC("socket")
+__success __retval(0)
+int loop_takes_reference_once(void *ctx)
+{
+	__u64 *sum = NULL;
+	int i;
+
+	bpf_for(i, 0, nr_vals) {
+		if (i >= 64 || !vals[i])
+			continue;
+		if (!sum) {
+			sum = bpf_ringbuf_reserve(&ringbuf, sizeof(*sum), 0);
+			if (!sum)
+				break;
+			*sum = 0;
+		}
+		*sum += vals[i];
+	}
+	if (sum)
+		bpf_ringbuf_submit(sum, 0);
+	return 0;
+}
+
 /*
  * r6 is a pointer to ringbuf memory with a step of 1. bpf_ringbuf_discard() in the first iteration
  * makes a number of it. The offsets 0..1 of the pointer are not the range of this number: it's an
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
index dbf6da950324..5a3893151f64 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
@@ -859,6 +859,61 @@ l0_%=:	r0 = 1;						\
 	: __clobber_all);
 }
 
+/*
+ * A list in a map value: it starts at element 0 and goes on with the element that the current one
+ * names. The offset is 0 in front of the loop and [0, 31] after every trip. The second of
+ * these states at the head of the loop stands for all that follow.
+ */
+SEC("tc")
+__description("map access: the offset of a value pointer grows in the first trip of a loop")
+__success
+__naked void loop_offset_settles(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r2 = r10; r2 += -8;"
+	"r1 = %[map_array_48b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 2f;"
+	"r6 = r0;"
+	"r2 = r6;"
+"1:	r5 = *(u8 *)(r2 + 0);"
+	"r5 &= 31;"
+	"r2 = r6; r2 += r5;"
+	"may_goto 2f;"
+	"goto 1b;"
+"2:	r0 = 0; exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_array_48b)
+	: __clobber_all);
+}
+
+SEC("tc")
+__description("map access: the offset of a value pointer grows in every trip of a loop")
+__failure __msg("invalid access to map value, value_size=48 off=48 size=1")
+__naked void loop_offset_grows(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"r2 = r10; r2 += -8;"
+	"r1 = %[map_array_48b] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 2f;"
+	"r2 = r0;"
+"1:	r5 = *(u8 *)(r2 + 0);"
+	"r2 += 1;"
+	"may_goto 2f;"
+	"goto 1b;"
+"2:	r0 = 0; exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_array_48b)
+	: __clobber_all);
+}
+
 SEC("socket")
 __description("map access: value_ptr += known scalar, 4")
 __success __success_unpriv __retval(1)
-- 
2.55.0

[PATCH bpf-next v4 43/46] selftests/bpf: Test progs that run out of budget

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

The verifier does not prove that a prog ends. It adds may_goto to each loop
that it cannot bound, and a check to the entry of each recursive subprog.
A prog is out of budget when may_goto times out or the check fails. No
other test gets there: a mistake in the added code rejects nothing, it
locks up a cpu, or the prog goes on along an edge that nobody verified.
Load and run such progs. Each invocation must return, and the return value
shows where the prog went on.

  [G] "bpf: Add may_goto to the loops that the verifier marks"
  [P] "bpf: Order the insns, find what stops a loop and where may_goto
      can go"
  [F] "bpf: Verify programs as a fixpoint over the control flow graph"
  [R] "bpf: Limit depth and time of recursive subprogs at run time",
      "bpf: Allow recursive subprogs"

  the prog                                        out of budget
  ----------------------------------------------  -------------------------
  a loop, the prog holds nothing                  [G] the prog ends
  the same, the last exit is dead code            [F] the prog ends
  a loop under a lock                             [G] the loop exits
  the same, the code after it needs p != NULL     [G] reject
  a loop in a subprog                             [G] the loop exits
  the same, the call has unwind_goto              [G] the subprog unwinds
  a landing pad does not release a lock           [G] reject
  all registers are live at the added may_goto    [G] none of them changes
  a loop has no exit, a nest of 17 loops          [F] [P] reject
  may_goto of the prog does not leave the loop,   [P] may_goto at the head,
  a jump to itself, a loop with two entries       the prog ends
  may_goto at the head, in a subprog, the code    [F] the subprog returns 0
  behind the loop reads what only the loop sets
  an iterator does not stop the loop around it    [F] the prog ends
  recursion is too deep, no unwind_goto           [R] the subprog returns 0
  the kernel stack where such a recursion ends    [R] a half of it is free
  recursion is too deep, calls have unwind_goto   [R] the landing pad runs
  recursion is not deep, but makes 2^60 calls     [R] the prog returns
  recursion through bpf_loop(), or in a loop      [R] the prog returns
  recursion through a global subprog and a table  [R] reject
  a timer callback arms itself again              [R] no recursion

The edge of an added may_goto is an edge of the prog. 10 more progs are
unsafe only on this edge: a caller that counts on what only the loop sets,
a bound of the next loop, a loop that is counted for one state or call and
gets may_goto for a different one ([P], [F]). They must be rejected.

verifier_budget.c is serial, like timed_may_goto: its progs spin for 250 ms
in an RCU read-side critical section, one after the other. It takes 5 sec.
verifier_callx.c gets a test of packet pointers after callx of an argument.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/verifier.c       |    3 +
 .../selftests/bpf/progs/verifier_budget.c     | 1915 +++++++++++++++++
 .../selftests/bpf/progs/verifier_callx.c      |   33 +
 .../bpf/progs/verifier_callx_rodata.c         |   51 +
 4 files changed, 2002 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_budget.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 978d6d7773ca..71ccd1e94bb2 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -31,6 +31,7 @@
 #include "verifier_bpf_get_stack.skel.h"
 #include "verifier_bpf_trap.skel.h"
 #include "verifier_bswap.skel.h"
+#include "verifier_budget.skel.h"
 #include "verifier_btf_array_access.skel.h"
 #include "verifier_btf_ctx_access.skel.h"
 #include "verifier_btf_flex_array.skel.h"
@@ -236,6 +237,8 @@ void test_verifier_bounds_mix_sign_unsign(void) { RUN(verifier_bounds_mix_sign_u
 void test_verifier_bpf_get_stack(void)        { RUN(verifier_bpf_get_stack); }
 void test_verifier_bpf_trap(void)             { RUN(verifier_bpf_trap); }
 void test_verifier_bswap(void)                { RUN(verifier_bswap); }
+/* Serial: many progs spin for 250 ms in an RCU read-side critical section, see timed_may_goto.c */
+void serial_test_verifier_budget(void)        { RUN(verifier_budget); }
 void test_verifier_btf_array_access(void)     { RUN(verifier_btf_array_access); }
 void test_verifier_btf_ctx_access(void)       { RUN(verifier_btf_ctx_access); }
 void test_verifier_btf_flex_array(void)       { RUN(verifier_btf_flex_array); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_budget.c b/tools/testing/selftests/bpf/progs/verifier_budget.c
new file mode 100644
index 000000000000..61c22ac58c1e
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_budget.c
@@ -0,0 +1,1915 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/*
+ * The verifier adds may_goto to a loop that it cannot count and a check of
+ * the stack to a recursive subprog. What the program does when it is out of budget:
+ * - it unwinds, when unwind_goto follows every call of the subprog and the subprog
+ *   holds nothing of its own. The pads of the callers are verified.
+ * - the main prog and an async callback end
+ * - otherwise it leaves the loop, or returns 0 from the subprog
+ *
+ * It takes unwinding, per-cpu insns and timed may_goto in JIT. Some of the progs never end
+ * on their own.
+ */
+#define unwind_goto(off) ".8byte %[insn_unwind_goto_" #off "];"
+#define unwind_exit ".8byte %[insn_unwind_exit];"
+
+#define __imm_unwind_goto(off) \
+	__imm_insn(insn_unwind_goto_##off, \
+		   BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, off, 0))
+#define __imm_unwind_exit \
+	__imm_insn(insn_unwind_exit, \
+		   BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+
+extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym;
+extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym;
+extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym;
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+unsigned long one = 1;
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, long);
+} empty SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 2);
+	__type(key, int);
+	__type(value, long[2]);
+} pairs SEC(".maps");
+
+SEC("socket")
+__description("budget: a loop in the main prog ends it")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in the main prog ends it, the last exit is dead code")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main_dead_exit(void)
+{
+	asm volatile (
+		"r6 = 0;"
+		"if r6 != 0 goto 2f;"
+		"r6 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+	"2:"
+		"r0 = 2;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in the main prog that holds something is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main_rcu(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"call %[bpf_rcu_read_lock];"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+/* 0 is not what this prog may return */
+SEC("cgroup/recvmsg4")
+__description("budget: a loop in the main prog that cannot return 0 is left")
+__arch_x86_64
+__arch_arm64
+__success
+__naked void loop_in_main_retval(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+/*
+ * do { p = lookup(); } while (!p); return *p;
+ * may_goto must not go where the loop goes when it's over: p may be NULL. Something is held, so
+ * it cannot end the prog either: there is no way out.
+ */
+SEC("socket")
+__description("budget: something is held and the code behind the loop counts on its exit")
+__arch_x86_64
+__arch_arm64
+__failure __msg("R0 invalid mem access 'map_value_or_null'")
+__naked void exit_of_loop_checks_null_rcu(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u32 *)(r10 - 4) = r1;"
+		"call %[bpf_rcu_read_lock];"
+	"1:"
+		"r2 = r10;"
+		"r2 += -4;"
+		"r1 = %[empty] ll;"
+		"call %[bpf_map_lookup_elem];"
+		"if r0 == 0 goto 1b;"
+		"r6 = *(u64 *)(r0 + 0);"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(empty),
+		  __imm(bpf_map_lookup_elem),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long spins(void)
+{
+	asm volatile (
+		"r2 = %[one] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(one)
+	);
+}
+
+/* the same, for the tests that call it another way */
+__naked __noinline __used
+static unsigned long spins_caught(void)
+{
+	asm volatile (
+		"r2 = %[one] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(one)
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog is left when there is no unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog(void)
+{
+	asm volatile (
+		"call spins;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog unwinds when there is unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(42)
+__naked void loop_in_subprog_caught(void)
+{
+	asm volatile (
+		"call spins_caught;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long spins_caught3(void)
+{
+	asm volatile (
+		"r2 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+	);
+}
+
+/* Nothing in the subprog unwinds. The pad is looked at because of may_goto that is added. */
+SEC("socket")
+__description("budget: the pad forgets what the caller holds")
+__arch_x86_64
+__arch_arm64
+__failure __msg("cannot be used inside bpf_rcu_read_lock-ed region")
+__naked void pad_forgets(void)
+{
+	asm volatile (
+		"call %[bpf_rcu_read_lock];"
+		"call spins_caught3;"
+		unwind_goto(4)
+		"r6 = r0;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = r6;"
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(4),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+/* There is no pad for what the subprog took itself */
+__naked __noinline __used
+static unsigned long spins_locked(void)
+{
+	asm volatile (
+		"call %[bpf_rcu_read_lock];"
+		"r2 = %[one] ll;"
+	"1:"
+		"r1 = *(u64 *)(r2 + 0);"
+		"if r1 != 0 goto 1b;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = 7;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+	);
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog that holds something is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog_locked(void)
+{
+	asm volatile (
+		"call spins_locked;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+/* r1: how deep to go. Returns the sum of 1 .. r1 */
+__naked __noinline __used
+static unsigned long sum_to(void)
+{
+	asm volatile (
+		"if r1 != 0 goto 1f;"
+		"r0 = 0;"
+		"exit;"
+	"1:"
+		"r6 = r1;"
+		"r1 += -1;"
+		"call sum_to;"
+		"r0 += r6;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion")
+__arch_x86_64
+__arch_arm64
+__success __retval(55)
+__naked void recursion(void)
+{
+	asm volatile (
+		"r1 = 10;"
+		"call sum_to;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * The one that is called too deep returns 0: 1000 + 999 + ... down to where it was: more than
+ * 1000, less than 500500
+ */
+SEC("socket")
+__description("budget: recursion that is too deep returns 0 when there is no unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_too_deep(void)
+{
+	asm volatile (
+		"r1 = 1000;"
+		"call sum_to;"
+		"r1 = r0;"
+		"r0 = 0;"
+		"if r1 <= 1000 goto +2;"
+		"if r1 >= 500500 goto +1;"
+		"r0 = 1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+unsigned long lowest_r10;
+
+__naked __noinline __used
+static unsigned long goes_down(void)
+{
+	asm volatile (
+		"r2 = %[lowest_r10] ll;"
+		"*(u64 *)(r2 + 0) = r10;"
+		"call goes_down;"
+		"exit;"
+		:
+		: __imm_addr(lowest_r10)
+		: __clobber_all
+	);
+}
+
+/*
+ * The stack of a task is aligned to its size, 16K at least. r10 of the last frame whose body
+ * runs is right above the middle.
+ */
+SEC("socket")
+__description("budget: recursion ends in the middle of the kernel stack")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_ends_in_the_middle(void)
+{
+	asm volatile (
+		"call goes_down;"
+		"r1 = %[lowest_r10] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r1 += 64;"
+		"r1 &= 8191;"
+		"r0 = 1;"
+		"if r1 < 256 goto +1;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(lowest_r10)
+		: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long has_large_frame(void)
+{
+	asm volatile (
+		"*(u64 *)(r10 - 1024) = r1;"
+		"call has_large_frame;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* The limit on the stack of a chain of calls has the frame of a recursive subprog in it */
+SEC("socket")
+__description("budget: one frame of a recursive subprog counts in the depth of the stack")
+__arch_x86_64
+__arch_arm64
+__failure __msg("combined stack size of 2 calls is 2560. Too large")
+__naked void recursion_with_large_frames(void)
+{
+	asm volatile (
+		"*(u64 *)(r10 - 1536) = r1;"
+		"call has_large_frame;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+__naked __noinline __used
+static unsigned long sum_to_caught(void)
+{
+	asm volatile (
+		"if r1 != 0 goto 1f;"
+		"r0 = 0;"
+		"exit;"
+	"1:"
+		"r6 = r1;"
+		"r1 += -1;"
+		"call sum_to_caught;"
+		unwind_goto(2)
+		"r0 += r6;"
+		"exit;"
+		"r0 = 0;"
+		unwind_exit
+		:
+		: __imm_unwind_goto(2),
+		  __imm_unwind_exit
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion with unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(55)
+__naked void recursion_caught(void)
+{
+	asm volatile (
+		"r1 = 10;"
+		"call sum_to_caught;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion that is too deep unwinds")
+__arch_x86_64
+__arch_arm64
+__success __retval(42)
+__naked void recursion_too_deep_caught(void)
+{
+	asm volatile (
+		"r1 = 1000;"
+		"call sum_to_caught;"
+		unwind_goto(1)
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(1)
+		: __clobber_all
+	);
+}
+
+/* r1: how deep to go. It calls itself twice: 2^r1 calls */
+__naked __noinline __used
+static unsigned long calls_itself_twice(void)
+{
+	asm volatile (
+		"if r1 != 0 goto 1f;"
+		"r0 = 1;"
+		"exit;"
+	"1:"
+		"r6 = r1;"
+		"r1 += -1;"
+		"call calls_itself_twice;"
+		"r7 = r0;"
+		"r1 = r6;"
+		"r1 += -1;"
+		"call calls_itself_twice;"
+		"r0 += r7;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: recursion that is not deep, 2^10 calls")
+__arch_x86_64
+__arch_arm64
+__success __retval(1024)
+__naked void recursion_tree(void)
+{
+	asm volatile (
+		"r1 = 10;"
+		"call calls_itself_twice;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* 61 frames fit in the stack. The check begins with may_goto: the time is limited too. */
+SEC("socket")
+__description("budget: recursion that is not deep, 2^60 calls")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void recursion_tree_forever(void)
+{
+	asm volatile (
+		"r1 = 60;"
+		"call calls_itself_twice;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* The time slice is for an invocation of the prog, not for the call that starts a recursion */
+SEC("socket")
+__description("budget: a loop that calls a subprog that calls itself 2^60 times")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_calls_recursion_tree(void)
+{
+	asm volatile (
+		"r8 = %[one] ll;"
+	"1:"
+		"r1 = 60;"
+		"call calls_itself_twice;"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/*
+ * Every frame loops forever and calls itself in every trip. The call that finds no room in the
+ * stack returns at once. When the time slice is over every frame that is on the stack leaves
+ * its loop.
+ */
+__naked __noinline __used
+static unsigned long calls_itself_in_a_loop(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+	"1:"
+		"call calls_itself_in_a_loop;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(one)
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog that calls itself in a loop that never ends")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_in_a_loop(void)
+{
+	asm volatile (
+		"call calls_itself_in_a_loop;"
+		"r0 = 1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * calls_loop() -> bpf_loop() -> loop_cb() -> calls_loop(): the frames of the helper are on
+ * the stack too
+ */
+static int loop_cb(__u32 i, void *ctx);
+
+static __noinline int calls_loop(int x)
+{
+	bpf_loop(1, loop_cb, NULL, zero);
+	return 0;
+}
+
+static __noinline int loop_cb(__u32 i, void *ctx)
+{
+	return calls_loop(i);
+}
+
+SEC("socket")
+__description("budget: recursion through a helper that calls a callback")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+int recursion_through_callback(void *ctx)
+{
+	return calls_loop(0);
+}
+
+struct timer {
+	struct bpf_timer t;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer);
+} timers SEC(".maps");
+
+__naked __noinline __used
+static int arms_itself(void)
+{
+	asm volatile (
+		"r1 = r3;"
+		"r2 = %[arms_itself] ll;"
+		"call %[bpf_timer_set_callback];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(arms_itself), __imm(bpf_timer_set_callback)
+		: __clobber_all
+	);
+}
+
+/* The helper keeps the callback for later. There is nothing to count, on any architecture. */
+SEC("socket")
+__description("budget: a callback that arms itself again is not a recursion")
+__success
+__xlated("...")
+__xlated("exit")
+__xlated("r1 = r3")
+__naked void callback_arms_itself(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u32 *)(r10 - 4) = r1;"
+		"r2 = r10;"
+		"r2 += -4;"
+		"r1 = %[timers] ll;"
+		"call %[bpf_map_lookup_elem];"
+		"if r0 == 0 goto 1f;"
+		"r1 = r0;"
+		"r2 = %[arms_itself] ll;"
+		"call %[bpf_timer_set_callback];"
+	"1:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(timers), __imm_addr(arms_itself), __imm(bpf_map_lookup_elem),
+		  __imm(bpf_timer_set_callback)
+		: __clobber_all
+	);
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, long);
+} hash SEC(".maps");
+
+static __noinline int sleeps_at_the_bottom(int n, void *user_ptr)
+{
+	char buf[8];
+
+	if (n)
+		return sleeps_at_the_bottom(n - 1, user_ptr) + 1;
+	return bpf_copy_from_user(buf, sizeof(buf), user_ptr);
+}
+
+static int calls_what_sleeps(void *map, void *key, void *value, void *ctx)
+{
+	return sleeps_at_the_bottom(1, (void *)zero) & 1;
+}
+
+SEC("syscall")
+__description("budget: recursion in a sleepable prog can sleep")
+__arch_x86_64
+__arch_arm64
+__success
+int recursion_sleeps(void *ctx)
+{
+	return sleeps_at_the_bottom(1, (void *)zero) & 1;
+}
+
+/* The frames of the callers are not there when a recursive subprog is verified */
+SEC("syscall")
+__description("budget: recursion in a callback that runs under rcu_read_lock() cannot sleep")
+__arch_x86_64
+__arch_arm64
+__failure __msg("sleepable helper bpf_copy_from_user#148 in callback running under rcu_read_lock")
+int recursion_sleeps_in_rcu_callback(void *ctx)
+{
+	bpf_for_each_map_elem(&hash, calls_what_sleeps, NULL, 0);
+	return 0;
+}
+
+/*
+ * may_goto of the program stops the loop that it leaves. This one goes to itself
+ * when it is out of budget. The loop spins until may_goto that is added ends the prog.
+ */
+SEC("socket")
+__description("budget: may_goto that doesn't leave the loop")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_stays_in_loop(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+	"1:"
+		".8byte %[may_goto_self];"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm_insn(may_goto_self, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, -1, 0))
+		: __clobber_all
+	);
+}
+
+/* The jump to itself has to go through may_goto that is added in front of it */
+SEC("socket")
+__description("budget: a conditional jump to itself")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void jump_to_itself(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r1 = *(u64 *)(r6 + 0);"
+	"1:"
+		"if r1 != 0 goto 1b;"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/* A cycle with two entries: there is no insn that every way into it goes through */
+SEC("socket")
+__description("budget: a loop with two entries")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_with_two_entries(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 2f;"
+	"1:"
+		"r2 = *(u64 *)(r6 + 0);"
+		"if r2 == 0 goto 3f;"
+	"2:"
+		"r3 = *(u64 *)(r6 + 0);"
+		"if r3 != 0 goto 1b;"
+	"3:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/*
+ * bpf_iter_num_next() stops the loop that the program leaves when there is nothing left.
+ * This one goes on. The iterator is held, so the loop is left when it is out of budget.
+ */
+SEC("socket")
+__description("budget: a loop that goes on when its iterator is drained")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_after_iter_is_drained(void)
+{
+	asm volatile (
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 10;"
+		"call %[bpf_iter_num_new];"
+		"r6 = %[one] ll;"
+	"1:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy)
+		: __clobber_all
+	);
+}
+
+/*
+ * bpf_iter_num_next() stops the loops that the program entered after it made the iterator.
+ * The loop around bpf_for() makes a new one in every trip, so it gets may_goto. It ends on its own
+ * after a time that is much longer than the budget.
+ */
+unsigned long trips, sum;
+unsigned long many = 1UL << 32;
+
+SEC("socket")
+__description("budget: the iterator of bpf_for() doesn't stop the loop around it")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+int loop_around_bpf_for(void *ctx)
+{
+	volatile unsigned long *cnt = &trips, *max = &many;
+	int i;
+
+	do {
+		bpf_for(i, 0, 2)
+			sum += i;
+		(*cnt)++;
+	} while (*cnt < *max);
+	return 1;
+}
+
+/*
+ * The first time the outer loop counts on bpf_iter_num_next(): its trips are joined, r7 is not
+ * known and the store is refused. All is done again, the loop is walked, r7 is 27 and the store is
+ * dead code. Where the first time got to doesn't count.
+ */
+SEC("socket")
+__description("budget: an insn that is reached only before all is done again is dead code")
+__arch_x86_64
+__arch_arm64
+__success
+__xlated("...")
+__xlated("if r6 < 0x3 goto pc-{{[0-9]+}}")
+__xlated("r0 = 0")
+__xlated("exit")
+__naked void insn_of_dropped_round(void)
+{
+	asm volatile (
+		"r6 = 0;"
+		"r7 = 1;"
+	"1:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 10;"
+		"call %[bpf_iter_num_new];"
+	"2:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"if r0 != 0 goto 2b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		"r7 *= 3;"
+		"r6 += 1;"
+		"if r6 == 1 goto 1b;"
+		"if r6 < 3 goto 1b;"
+		"if r7 == 27 goto 3f;"
+		"r1 = 0;"
+		"*(u64 *)(r1 + 0) = r1;"
+	"3:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy)
+		: __clobber_all
+	);
+}
+
+/*
+ * The loop at 3: spins. It has no jump that every trip goes through, so may_goto is at its head.
+ * The first way out of it, from 2: to 1:, is the back edge of the loop at 1:, which
+ * has no may_goto of its own: bpf_iter_num_next() stops it. From the head of 3: to 1: there is
+ * no bpf_iter_num_next().
+ */
+SEC("socket")
+__description("budget: may_goto at the head of a loop leaves the loops around it too")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void guard_at_head_of_nested_loop(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r9 = %[zero] ll;"
+		"r7 = 0;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 2;"
+		"call %[bpf_iter_num_new];"
+		"call %[bpf_rcu_read_lock];"
+		"r7 += 1;"
+		"goto 1f;"
+	"2:"
+		"r7 += 1;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 != 0 goto 4f;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 != 0 goto 4f;"
+	"1:"
+		"r7 += 1;"
+		"r1 = *(u64 *)(r6 + 0);"
+	"5:"
+		"if r1 == 0 goto 5b;"
+	"3:"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 != 0 goto +0;"
+	"4:"
+		"r7 += 1;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 == 0 goto 3b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"if r0 == 0 goto 6f;"
+		"r1 = *(u64 *)(r9 + 0);"
+		"if r1 == 0 goto 2b;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 4b;"
+	"6:"
+		"call %[bpf_rcu_read_unlock];"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(one),
+		  __imm_addr(zero),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+#define NEST(n, body)	#n ": call %[bpf_get_prandom_u32];" body "if r0 == " #n " goto " #n "b;"
+#define NEST4(a, b, c, d, body)	NEST(a, NEST(b, NEST(c, NEST(d, body))))
+
+/* It's refused before anything goes up the nest from each of its insns */
+SEC("socket")
+__description("budget: a nest of 17 loops")
+__failure __msg("Too many nested loops at")
+__naked void nest_of_17_loops(void)
+{
+	asm volatile (
+		NEST4(1, 2, 3, 4, NEST4(5, 6, 7, 8, NEST4(9, 10, 11, 12,
+		NEST4(13, 14, 15, 16, NEST(17, "")))))
+		"r0 = 0;"
+		"exit;"
+		:: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/*
+ * The loops below spin and have no jump that every trip goes through, so may_goto is at the head.
+ * It goes to the code behind the loop, past what the loop makes for that code, with the registers
+ * and the stack as they are at the head.
+ *
+ * r6 is 5 on every edge of the program that gets behind the loop. At the head it's 7.
+ */
+__naked __noinline __used
+static unsigned long spins_and_makes(void)
+{
+	asm volatile (
+		"r8 = %[one] ll;"
+		"r6 = 7;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"r6 = 5;"
+		"if r1 != 2 goto 1b;"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop keeps a register that the loop sets")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void guard_at_head_keeps_reg(void)
+{
+	asm volatile (
+		"call spins_and_makes;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* Only the loop sets r6: there is no way from the head to the code behind the loop */
+__naked __noinline __used
+static unsigned long spins_and_makes_unset(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"r6 = 5;"
+		"if r1 != 2 goto 1b;"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, a register that only the loop sets")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void guard_at_head_unset_reg(void)
+{
+	asm volatile (
+		"call spins_and_makes_unset;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, the caller counts on what the loop sets")
+__failure __msg("R9 !read_ok")
+__naked void guard_at_head_unset_reg_caller(void)
+{
+	asm volatile (
+		"call spins_and_makes_unset;"
+		"if r0 == 5 goto 1f;"
+		"r0 = r9;"
+	"1:"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* fp-8 is 0 on every edge of the program that gets behind the loop. It's dead at the head. */
+__naked __noinline __used
+static unsigned long spins_and_makes_slot(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+		"r7 = 1;"
+		"*(u64 *)(r10 - 8) = r7;"
+		"r7 = 0;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"*(u64 *)(r10 - 8) = r7;"
+		"if r1 != 2 goto 1b;"
+		"r0 = *(u64 *)(r10 - 8);"
+		"r0 <<= 3;"
+		"r8 += r0;"
+		"r0 = *(u64 *)(r8 + 0);"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, a stack slot that is dead there")
+__failure __msg("slot poisoned by dead code elimination")
+__naked void guard_at_head_dead_slot(void)
+{
+	asm volatile (
+		"call spins_and_makes_slot;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * The low half of r7 is live at the head of the loop, all of it where may_goto goes. On the edges
+ * of the program that get there it's 3. 'w7 &= 7' has to clear the upper half.
+ */
+__naked __noinline __used
+static unsigned long spins_with_half_of_reg(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+		"r7 = *(u64 *)(r8 + 0);"
+		"w7 &= 7;"
+	"1:"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if w1 == w7 goto 1b;"
+		"r7 = 3;"
+		"if r1 == 2 goto 2f;"
+		"w7 = 5;"
+		"goto 1b;"
+	"2:"
+		"r0 = r7;"
+		"r0 >>= 32;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, one half of a register is live there")
+__arch_x86_64
+__arch_arm64
+__flag(BPF_F_TEST_RND_HI32)
+__success __retval(0)
+__naked void guard_at_head_half_of_reg(void)
+{
+	asm volatile (
+		"call spins_with_half_of_reg;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * From the head of a loop to the code behind it is not an edge of the program. What holds on
+ * every edge of the program that gets there doesn't hold when may_goto does.
+ */
+const volatile unsigned long four = 4;
+char five_bytes[5] SEC(".data.five_bytes");
+
+/*
+ * r7 points to 'four' on every edge. At the head of the first loop it points to @at_head.
+ * The second loop goes on while r6 is less than what r7 points to.
+ */
+#define SPINS_THEN_COUNTS(at_head)			\
+		"r8 = %[zero] ll;"			\
+		"r7 = %[" #at_head "] ll;"		\
+	"1:"						\
+		"r2 = *(u64 *)(r7 + 0);"		\
+		"r1 = *(u64 *)(r8 + 0);"		\
+		"if r1 == 1 goto 1b;"			\
+		"r7 = %[four] ll;"			\
+		"if r1 == 2 goto 2f;"			\
+		"r7 = %[" #at_head "] ll;"		\
+		"goto 1b;"				\
+	"2:"						\
+		"r6 = 0;"				\
+	"3:"						\
+		"r2 = *(u64 *)(r7 + 0);"		\
+		"if r6 >= r2 goto 4f;"			\
+		"r1 = %[five_bytes] ll;"		\
+		"r1 += r6;"				\
+		"r3 = 1;"				\
+		"*(u8 *)(r1 + 0) = r3;"			\
+		"r6 += 1;"				\
+		"goto 3b;"				\
+	"4:"						\
+		"r0 = r6;"				\
+		"exit;"					\
+		:					\
+		: __imm_addr(zero),			\
+		  __imm_addr(one),			\
+		  __imm_addr(four),			\
+		  __imm_addr(five_bytes)		\
+		: __clobber_all
+
+__naked __noinline __used
+static unsigned long spins_then_counts_to_anything(void)
+{
+	asm volatile (SPINS_THEN_COUNTS(one));
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, another bound for the next loop")
+__arch_x86_64
+__arch_arm64
+__failure __msg("math between map_value pointer and register with unbounded min value")
+__naked void guard_at_head_bound_of_next_loop(void)
+{
+	asm volatile (
+		"call spins_then_counts_to_anything;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * r7 is fp-8 on every edge and fp-16 at the head of the first loop. The second loop
+ * counts in fp-16 and stores a number that nobody knows through r7.
+ */
+__naked __noinline __used
+static unsigned long spins_then_counts_on_stack(void)
+{
+	asm volatile (
+		"r8 = %[zero] ll;"
+		"r3 = 0;"
+		"*(u64 *)(r10 - 8) = r3;"
+		"*(u64 *)(r10 - 16) = r3;"
+		"r7 = r10;"
+		"r7 += -16;"
+	"1:"
+		"r2 = *(u64 *)(r7 + 0);"
+		"r1 = *(u64 *)(r8 + 0);"
+		"if r1 == 1 goto 1b;"
+		"r7 = r10;"
+		"r7 += -8;"
+		"if r1 == 2 goto 2f;"
+		"r7 = r10;"
+		"r7 += -16;"
+		"goto 1b;"
+	"2:"
+		"r3 = 0;"
+		"*(u64 *)(r10 - 16) = r3;"
+	"3:"
+		"r6 = *(u64 *)(r10 - 16);"
+		"if r6 == 4 goto 4f;"
+		"r1 = %[five_bytes] ll;"
+		"r1 += r6;"
+		"r3 = 1;"
+		"*(u8 *)(r1 + 0) = r3;"
+		"r6 += 1;"
+		"*(u64 *)(r10 - 16) = r6;"
+		"r3 = *(u64 *)(r8 + 0);"
+		"*(u64 *)(r7 + 0) = r3;"
+		"goto 3b;"
+	"4:"
+		"r0 = r6;"
+		"exit;"
+		:
+		: __imm_addr(zero),
+		  __imm_addr(five_bytes)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto at the head of a loop, another pointer to the stack")
+__arch_x86_64
+__arch_arm64
+__failure __msg("math between map_value pointer and register with unbounded min value")
+__naked void guard_at_head_stack_pointer(void)
+{
+	asm volatile (
+		"call spins_then_counts_on_stack;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * Two states that have no join enter a loop. One has no count, so the loop gets may_goto.
+ * The other one has a count. may_goto is there when it runs too, and a loop in front
+ * may have used up the time slice. Then it leaves the loop in the first trip.
+ *
+ * r7 = the value, 16 bytes, and r8 = count, or r7 = 0 and r8 = anything
+ */
+#define VALUE_AND_COUNT_OR_NEITHER(count)		\
+		"r1 = 0;"				\
+		"*(u32 *)(r10 - 4) = r1;"		\
+		"r2 = r10;"				\
+		"r2 += -4;"				\
+		"r1 = %[pairs] ll;"			\
+		"call %[bpf_map_lookup_elem];"		\
+		"if r0 == 0 goto 9f;"			\
+		"r9 = %[zero] ll;"			\
+		"r2 = *(u64 *)(r9 + 0);"		\
+		"if r2 == 0 goto 3f;"			\
+		"r7 = r0;"				\
+		"r8 = " #count ";"			\
+		"goto 4f;"				\
+	"3:"						\
+		"r7 = 0;"				\
+		"r8 = *(u64 *)(r9 + 0);"		\
+	"4:"
+
+/* It's in the value when r6 is what it is at the end of the loop that was counted: last */
+#define LOAD_IF_VALUE(last)				\
+		"if r7 == 0 goto 9f;"			\
+		"r2 = " #last ";"			\
+		"r2 -= r6;"				\
+		"r2 &= 1;"				\
+		"r2 <<= 4;"				\
+		"r7 += r2;"				\
+		"r0 = *(u64 *)(r7 + 0);"		\
+		"exit;"					\
+	"9:"						\
+		"r0 = 0;"				\
+		"exit;"					\
+		:					\
+		: __imm_addr(pairs),			\
+		  __imm_addr(zero),			\
+		  __imm(bpf_map_lookup_elem)		\
+		: __clobber_all
+
+__naked __noinline __used
+static unsigned long counted_or_not(void)
+{
+	asm volatile (
+		VALUE_AND_COUNT_OR_NEITHER(10)
+		"r6 = 0;"
+	"1:"
+		"r6 += 1;"
+		"if r6 < r8 goto 1b;"
+		LOAD_IF_VALUE(10)
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto that another state needs, the loop is counted")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void guard_in_counted_loop(void)
+{
+	asm volatile (
+		"call counted_or_not;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* 2 4 8 16 is not counted. Constants decide the jump. */
+__naked __noinline __used
+static unsigned long walked_or_not(void)
+{
+	asm volatile (
+		VALUE_AND_COUNT_OR_NEITHER(16)
+		"r6 = 1;"
+	"1:"
+		"r6 <<= 1;"
+		"if r6 < r8 goto 1b;"
+		"r6 >>= 1;"
+		LOAD_IF_VALUE(8)
+	);
+}
+
+SEC("socket")
+__description("budget: may_goto that another state needs, constants decide the exit of the loop")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void guard_in_walked_loop(void)
+{
+	asm volatile (
+		"call walked_or_not;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/* i = 0; do { i++; } while (i < n); return v ? v[((10 - i) & 1) * 2] : 0; */
+__naked __noinline __used
+static unsigned long counts(unsigned long n, long *v)
+{
+	asm volatile (
+		"r0 = 0;"
+	"1:"
+		"r0 += 1;"
+		"if r0 < r1 goto 1b;"
+		"if r2 == 0 goto 2f;"
+		"r3 = 10;"
+		"r3 -= r0;"
+		"r3 &= 1;"
+		"r3 <<= 4;"
+		"r2 += r3;"
+		"r0 = *(u64 *)(r2 + 0);"
+	"2:"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+#define LOOKUP_PAIR					\
+		"r1 = 0;"				\
+		"*(u32 *)(r10 - 4) = r1;"		\
+		"r2 = r10;"				\
+		"r2 += -4;"				\
+		"r1 = %[pairs] ll;"			\
+		"call %[bpf_map_lookup_elem];"		\
+		"if r0 == 0 goto 9f;"			\
+		"r6 = r0;"
+
+SEC("socket")
+__description("budget: a subprog is called without a count, then with a count")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void two_callers(void)
+{
+	asm volatile (
+		LOOKUP_PAIR
+		"r1 = %[zero] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r2 = 0;"
+		"call counts;"
+		"r1 = 10;"
+		"r2 = r6;"
+		"call counts;"
+	"9:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(pairs),
+		  __imm_addr(zero),
+		  __imm(bpf_map_lookup_elem)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog is called with a count, then without a count")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void two_callers_counted_first(void)
+{
+	asm volatile (
+		LOOKUP_PAIR
+		"r1 = 10;"
+		"r2 = r6;"
+		"call counts;"
+		"r1 = %[zero] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r2 = 0;"
+		"call counts;"
+	"9:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(pairs),
+		  __imm_addr(zero),
+		  __imm(bpf_map_lookup_elem)
+		: __clobber_all
+	);
+}
+
+/* A global subprog is verified after the main prog */
+__naked __noinline __used
+int counts_anything(void)
+{
+	asm volatile (
+		"r1 = %[zero] ll;"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r2 = 0;"
+		"call counts;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog is called with a count, then without a count by a global subprog")
+__arch_x86_64
+__arch_arm64
+__failure __msg("invalid access to map value, value_size=16 off=16 size=8")
+__naked void two_callers_two_passes(void)
+{
+	asm volatile (
+		LOOKUP_PAIR
+		"r1 = 10;"
+		"r2 = r6;"
+		"call counts;"
+		"call counts_anything;"
+	"9:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(pairs),
+		  __imm(bpf_map_lookup_elem)
+		: __clobber_all
+	);
+}
+
+/*
+ * if (x) lock(); do { p = lookup(); } while (!p); if (x) { unlock(); return 0; } return *p;
+ * looks(0) holds nothing: may_goto unwinds. looks(1) cannot, so may_goto leaves the loop, in
+ * looks(0) too, and p is NULL.
+ */
+__naked __noinline __used
+static unsigned long looks(unsigned long x)
+{
+	asm volatile (
+		"r6 = r1;"
+		"if r6 == 0 goto 1f;"
+		"call %[bpf_rcu_read_lock];"
+	"1:"
+		"r1 = 0;"
+		"*(u32 *)(r10 - 4) = r1;"
+		"r2 = r10;"
+		"r2 += -4;"
+		"r1 = %[empty] ll;"
+		"call %[bpf_map_lookup_elem];"
+		"if r0 == 0 goto 1b;"
+		"if r6 == 0 goto 2f;"
+		"call %[bpf_rcu_read_unlock];"
+		"r0 = 0;"
+		"exit;"
+	"2:"
+		"r0 = *(u64 *)(r0 + 0);"
+		"exit;"
+		:
+		: __imm_addr(empty),
+		  __imm(bpf_map_lookup_elem),
+		  __imm(bpf_rcu_read_lock),
+		  __imm(bpf_rcu_read_unlock)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a subprog finds out late that something is held")
+__arch_x86_64
+__arch_arm64
+__failure __msg("R0 invalid mem access 'map_value_or_null'")
+__naked void unwinds_then_cannot(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"call looks;"
+		unwind_goto(5)
+		"r1 = 1;"
+		"call looks;"
+		unwind_goto(2)
+		"r0 = 0;"
+		"exit;"
+		"r0 = 42;"
+		"exit;"
+		:
+		: __imm_unwind_goto(5),
+		  __imm_unwind_goto(2)
+		: __clobber_all
+	);
+}
+
+SEC("socket")
+__description("budget: a loop without an exit")
+__arch_x86_64
+__arch_arm64
+__failure __msg("never exits")
+__naked void loop_without_exit(void)
+{
+	asm volatile (
+		"r0 = 0;"
+	"1:"
+		"r0 += 1;"
+		"goto 1b;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * may_goto keeps the address of its count in a register that is dead. What is live in front
+ * of the insn that gets may_goto is live in front of may_goto.
+ */
+SEC("socket")
+__description("budget: may_goto in front of a jump keeps R0")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void guard_keeps_r0(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"r0 = 7;"
+	"1:"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 != 0 goto 1b;"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+#define ALL_REGS_LIVE_IN_LOOP(var)			\
+		"r9 = %[" #var "] ll;"			\
+		"r0 = 1;"				\
+		"r2 = 2;"				\
+		"r3 = 3;"				\
+		"r4 = 4;"				\
+		"r5 = 5;"				\
+		"r6 = 6;"				\
+		"r7 = 7;"				\
+		"r8 = 8;"				\
+	"1:"						\
+		"r1 = *(u64 *)(r9 + 0);"		\
+		"if r1 != 0 goto 1b;"			\
+		"r0 += r1;"				\
+		"r0 += r2;"				\
+		"r0 += r3;"				\
+		"r0 += r4;"				\
+		"r0 += r5;"				\
+		"r0 += r6;"				\
+		"r0 += r7;"				\
+		"r0 += r8;"				\
+		"r9 = *(u64 *)(r9 + 0);"		\
+		"r0 += r9;"				\
+		"exit;"					\
+		:					\
+		: __imm_addr(var)			\
+		: __clobber_all
+
+/* No register is dead: R0 waits in the stack */
+SEC("socket")
+__description("budget: may_goto where all registers are live")
+__arch_x86_64
+__arch_arm64
+__success __retval(36)
+__xlated("...")
+__xlated("*(u64 *)(r10 -8) = r0")
+__xlated("r0 = 0x{{.*}}")
+__xlated("r0 = &(void __percpu *)(r0)")
+__xlated("r13 = *(u64 *)(r0 +0)")
+__xlated("...")
+__xlated("*(u64 *)(r0 +0) = r13")
+__xlated("r0 = *(u64 *)(r10 -8)")
+__xlated("if r1 != 0x0 goto pc-{{[0-9]+}}")
+__naked void guard_all_regs_live(void)
+{
+	asm volatile (
+		ALL_REGS_LIVE_IN_LOOP(zero)
+	);
+}
+
+__naked __noinline __used
+static unsigned long spins_all_regs_live(void)
+{
+	asm volatile (
+		ALL_REGS_LIVE_IN_LOOP(one)
+	);
+}
+
+/* The time is up: the loop is left with r1 = 1 and all the other registers as they are */
+SEC("socket")
+__description("budget: may_goto where all registers are live, out of budget")
+__arch_x86_64
+__arch_arm64
+__success __retval(38)
+__naked void guard_all_regs_live_spins(void)
+{
+	asm volatile (
+		"call spins_all_regs_live;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * A loop that has no count is looked at trip by trip for a while. A state that is within a state of
+ * an earlier trip is dropped, wherever that is. Then nothing says that the loop ends, it gets
+ * may_goto, and all is done again. None of the loops below ends on its own.
+ */
+SEC("socket")
+__description("budget: every second trip leaves everything as it was")
+__arch_x86_64
+__arch_arm64
+__success __retval(0) __log_level(2)
+__msg("processed 25 insns")
+__naked void trips_of_period_two(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r7 = 1;"
+	"1:"
+		"r7 ^= 3;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 == 0 goto 2f;"
+		"if r7 != 7 goto 1b;"
+	"2:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/*
+ * r7 is 0 at the head in the first trip and 5 in the second one. It's dead at 2, where the second
+ * trip is dropped. Nothing comes back to the head that is within what is there.
+ */
+SEC("socket")
+__description("budget: a trip is dropped in the middle of the loop")
+__arch_x86_64
+__arch_arm64
+__success __retval(0) __log_level(2)
+__msg("processed 33 insns")
+__naked void trip_dropped_in_the_middle(void)
+{
+	asm volatile (
+		"r6 = %[one] ll;"
+		"r7 = 0;"
+	"1:"
+		"if r7 == 7 goto 3f;"
+		"r1 = *(u64 *)(r6 + 0);"
+		"if r1 == 0 goto 3f;"
+		"if r1 == 2 goto 2f;"
+		"r1 = 0;"
+	"2:"
+		"r7 = 5;"
+		"goto 1b;"
+	"3:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(one)
+		: __clobber_all
+	);
+}
+
+/* n ? zero_at_the_bottom(n - 1) : zero */
+__naked __noinline __used
+static unsigned long zero_at_the_bottom(unsigned long n)
+{
+	asm volatile (
+		"if r1 == 0 goto 1f;"
+		"r1 += -1;"
+		"call zero_at_the_bottom;"
+		"exit;"
+	"1:"
+		"r1 = %[zero] ll;"
+		"r0 = *(u64 *)(r1 + 0);"
+		"exit;"
+		:
+		: __imm_addr(zero)
+		: __clobber_all
+	);
+}
+
+/*
+ * The call of a recursive subprog waits for what comes out of it. The call of the second trip is
+ * within the one that waits. A recursive subprog that has no time or no stack left returns 0 at
+ * once, so that doesn't stop the loop.
+ */
+SEC("socket")
+__description("budget: a trip is dropped at a call that waits")
+__arch_x86_64
+__arch_arm64
+__success __retval(0) __log_level(2)
+__msg("processed 27 insns")
+__naked void trip_dropped_at_a_call(void)
+{
+	asm volatile (
+		"r7 = 0;"
+	"1:"
+		"if r7 == 7 goto 2f;"
+		"r1 = 3;"
+		"call zero_at_the_bottom;"
+		"r7 = 5;"
+		"if r0 == 0 goto 1b;"
+	"2:"
+		"r0 = 1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * Every trip goes through bpf_iter_num_next(), so the loop is taken for bounded at first, and
+ * through may_goto of the program. The iterator is made in the loop, and may_goto stays in it:
+ * neither of them stops it. The program doesn't end on its own.
+ */
+SEC("socket")
+__description("budget: may_goto that stays in the loop doesn't stop it")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_stays_in_the_loop(void)
+{
+	asm volatile (
+		"r6 = %[zero] ll;"
+		"r6 = *(u64 *)(r6 + 0);"
+		"if r6 != 0 goto 4f;"
+	"1:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"r2 = 0;"
+		"r3 = 2;"
+		"call %[bpf_iter_num_new];"
+	"2:"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_next];"
+		"if r0 != 0 goto 2b;"
+		"r1 = r10;"
+		"r1 += -8;"
+		"call %[bpf_iter_num_destroy];"
+		".8byte %[may_goto];"
+		"if r6 == 0 goto 1b;"
+	"4:"
+		"r0 = 1;"
+		"exit;"
+		:
+		: __imm_addr(zero),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy),
+		  __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 0 /* offset */, 0))
+		: __clobber_all
+	);
+}
+
+/* No state is within another one: the walk gets to the end of the loop, no may_goto */
+SEC("socket")
+__description("budget: ten trips that differ")
+__success __retval(10) __log_level(2)
+__msg("processed 34 insns")
+__xlated("r8 += 1")
+__xlated("r7 <<= 1")
+__xlated("if r7 != 0x400 goto pc-3")
+__naked void ten_trips_that_differ(void)
+{
+	asm volatile (
+		"r7 = 1;"
+		"r8 = 0;"
+	"1:"
+		"r8 += 1;"
+		"r7 <<= 1;"
+		"if r7 != 1024 goto 1b;"
+		"r0 = r8;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * BTF FUNC records are not generated for kfuncs referenced
+ * from inline assembly. These records are necessary for
+ * libbpf to link the program. The function below is a hack
+ * to ensure that BTF FUNC records are generated.
+ */
+void __kfunc_btf_root(void)
+{
+	bpf_rcu_read_lock();
+	bpf_rcu_read_unlock();
+	bpf_iter_num_new(0, 0, 0);
+	bpf_iter_num_next(0);
+	bpf_iter_num_destroy(0);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index afe4990cd027..5f0d9e545656 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -637,6 +637,39 @@ __naked void callx_mutual_recursion(void)
 		: __clobber_all);
 }
 
+static __noinline __used int pulls_data(struct __sk_buff *skb)
+{
+	return bpf_skb_pull_data(skb, 0);
+}
+
+/*
+ * The prog loads the address of the function, apply() calls it. The packet
+ * pointer of the prog is gone after the call of apply().
+ */
+SEC("tc")
+__failure __msg("R6 invalid mem access 'scalar'")
+__naked void callx_fn_as_arg_changes_pkt_data(void)
+{
+	asm volatile (
+		"r2 = r1;"
+		"r6 = *(u32 *)(r1 + %[data]);"
+		"r7 = *(u32 *)(r1 + %[data_end]);"
+		"r0 = 0;"
+		"r3 = r6;"
+		"r3 += 1;"
+		"if r3 > r7 goto 1f;"
+		"r1 = %[pulls_data] ll;"
+		"call apply;"
+		"*(u8 *)(r6 + 0) = 42;"
+		"r0 = 0;"
+	"1:	exit;"
+		:
+		: __imm_addr(pulls_data),
+		  __imm_const(data, offsetof(struct __sk_buff, data)),
+		  __imm_const(data_end, offsetof(struct __sk_buff, data_end))
+		: __clobber_all);
+}
+
 /* calls_arg(fn) { return fn(); } */
 __naked __noinline __used
 static unsigned long calls_arg(void)
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 8bb74b579203..2e7e65949497 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -198,6 +198,57 @@ __naked void callx_rodata_array_of_structs_data(void)
 		: __clobber_all);
 }
 
+int calls_what_calls_it(void);
+
+__naked __noinline __used
+static unsigned long calls_global(void)
+{
+	asm volatile (
+		"call calls_what_calls_it;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all);
+}
+
+__naked __noinline __used
+int calls_what_calls_it(void)
+{
+	asm volatile (
+		RODATA(cb, ".quad calls_global;")
+		"r2 = cb_%= ll;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 1;"
+		"r3 = 0;"
+		"r4 = 0;"
+		"call %[bpf_loop];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_loop)
+		: __clobber_all);
+}
+
+/*
+ * No subprog takes the address of the callback, and calls of global functions are not
+ * followed. It's a recursion all the same, and nothing checks the stack.
+ */
+SEC("socket")
+__failure __msg("recursive call from calls_global() to calls_what_calls_it()")
+__naked void callx_rodata_callback_recursion(void)
+{
+	asm volatile (
+		FUNC_TABLE2(tbl, add1, add2)
+		"call calls_what_calls_it;"
+		/* libbpf resolves the pointers for progs that have callx */
+		"r2 = tbl_%= ll;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 1;"
+		"callx r2;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all);
+}
+
 /* but not the data and the pointer with the same load */
 SEC("socket")
 __failure __msg("overlaps with a pointer to a function")
-- 
2.55.0

[PATCH bpf-next v4 44/46] selftests/bpf: Test callx of one of a set of functions

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

'ops[i](x)' in C, and each '{}' of a format string in Rust, is a load from
a table of functions at a variable offset, and callx. The load gives one
register that points to one of N functions.

  [S] "bpf: Track a pointer to one of a set of functions"
  [L] "bpf: callx reads what the subprogs whose address is taken read"

  file, test                     the rule that it pins
  -----------------------------  ------------------------------------------
  verifier_callx_rodata.c, callx_rodata_..
   one_of_two                    [S] The log has R2=func(one_of=2). callx
                                 calls the function that the index selects.
   callback_one_of_two           [S] bpf_loop() gets the register: each
                                 function is verified as the callback.
   one_of_two_at_loop_head       [S] callx is the head of a loop. The prog
                                 is unsafe in the fourth iteration with one
                                 of the two functions: rejected.
  verifier_callx_many.c, callx_rodata_..
   400_functions                 [S] 3 states that have no join get to the
                                 load: 3 states at an insn. One state for
                                 each function is 1200, the limit is 1024.
   4400_sets                     [S] More than 4096 sets: rejected.
  verifier_stack_arg.c
   callx_one_of_two_not_written  [S] The caller does not store the stack
                                 arguments that both functions take:
                                 rejected.
  verifier_callx.c
   callx_reads_declared_args     [L] r2 is live at callx: the callee does
                                 not read it, but its BTF declares it.

one_of_two, 400_functions and 4400_sets fail without [S], and
callx_reads_declared_args fails without [L]. The other three tests pass in
both cases. They examine the places where a set becomes one function.

libbpf adds the functions of a table to each prog of the object that has
callx. Thus the 400 functions are in a file of their own.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_callx.c      |  24 ++++
 .../selftests/bpf/progs/verifier_callx_many.c | 113 ++++++++++++++++++
 .../bpf/progs/verifier_callx_rodata.c         | 112 +++++++++++++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  |  54 +++++++++
 5 files changed, 305 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_many.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 71ccd1e94bb2..0db43480cd99 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -39,6 +39,7 @@
 #include "verifier_call_large_imm.skel.h"
 #include "verifier_calls_in_a_row.skel.h"
 #include "verifier_callx.skel.h"
+#include "verifier_callx_many.skel.h"
 #include "verifier_callx_rodata.skel.h"
 #include "verifier_cfg.skel.h"
 #include "verifier_cgroup_inv_retcode.skel.h"
@@ -246,6 +247,7 @@ void test_verifier_btf_unreliable_prog(void)  { RUN(verifier_btf_unreliable_prog
 void test_verifier_call_large_imm(void)       { RUN(verifier_call_large_imm); }
 void test_verifier_calls_in_a_row(void)       { RUN(verifier_calls_in_a_row); }
 void test_verifier_callx(void)                { RUN_JITED(verifier_callx); }
+void test_verifier_callx_many(void)           { RUN_JITED(verifier_callx_many); }
 void test_verifier_callx_rodata(void)         { RUN_JITED(verifier_callx_rodata); }
 void test_verifier_cfg(void)                  { RUN(verifier_cfg); }
 void test_verifier_cgroup_inv_retcode(void)   { RUN(verifier_cgroup_inv_retcode); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index 5f0d9e545656..a2bc466791f3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -985,6 +985,30 @@ __naked void callx_callee_reads_caller_stack_nested(void)
 		: __clobber_all);
 }
 
+__naked __noinline __used
+static unsigned long reads_one_of_two(long a, long b)
+{
+	asm volatile (
+		"r0 = r1;"
+		"exit;"
+		::: __clobber_all);
+}
+
+/* r2 is live at callx: the BTF of the callee, which a tracer counts on, declares it */
+SEC("socket")
+__success __retval(1) __log_level(2)
+__msg(".123...... (8d) callx r3")
+__naked void callx_reads_declared_args(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = reads_one_of_two ll;"
+		"callx r3;"
+		"exit;"
+		::: __clobber_all);
+}
+
 /* registers that are constant before callx are not constant after it */
 SEC("socket")
 __success __retval(1)
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_many.c b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
new file mode 100644
index 000000000000..23c8948c24b8
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * callx through a table of many functions. libbpf appends the functions that read-only
+ * data points to to every prog of the object that has callx, hence a file of its own.
+ */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../../../include/linux/filter.h"
+
+#ifdef CAN_USE_CALLX
+
+/*
+ * Read-only data with pointers to functions, where the compiler puts tables
+ * of functions, structures of operations and vtables. libbpf resolves
+ * a pointer to the offset of the function in the program and the kernel
+ * recognizes it by that value.
+ */
+#define DATA(SECTION, NAME, ...)				\
+	".pushsection " SECTION ",@progbits;"			\
+	".balign 8;"						\
+	#NAME "_%=:"						\
+	__VA_ARGS__						\
+	".type " #NAME "_%=, @object;"				\
+	".size " #NAME "_%=, .-" #NAME "_%=;"			\
+	".popsection;"
+
+#define RODATA(NAME, ...) DATA(".rodata,\"a\"", NAME, __VA_ARGS__)
+
+#define REP10(M, p) M(p##0) M(p##1) M(p##2) M(p##3) M(p##4) M(p##5) M(p##6) M(p##7) M(p##8) M(p##9)
+#define REP100(M, p) REP10(M, p##0) REP10(M, p##1) REP10(M, p##2) REP10(M, p##3) REP10(M, p##4) \
+		     REP10(M, p##5) REP10(M, p##6) REP10(M, p##7) REP10(M, p##8) REP10(M, p##9)
+#define REP400(M) REP100(M, 0) REP100(M, 1) REP100(M, 2) REP100(M, 3)
+
+#define MANY_FUNC(n)						\
+__naked __noinline __used static unsigned long many##n(void)	\
+{								\
+	asm volatile ("r0 = r1; r0 += 1; exit;");		\
+}
+#define MANY_QUAD(n) ".quad many" #n ";"
+
+REP400(MANY_FUNC)
+
+/* many<n> or what is 'mask' bytes behind it in the table: 400 sets that were not there before */
+#define SET_LOAD(n) "r2 = *(u64 *)(r7 + 8 * (1" #n " - 1000));"
+#define SETS400(mask) "r7 = r6; r7 &= " #mask "; r7 += r8;" REP400(SET_LOAD)
+
+SEC("socket")
+__failure __msg("more than 4096 sets of functions")
+__naked void callx_rodata_4400_sets(void)
+{
+	asm volatile (
+		RODATA(twice, REP400(MANY_QUAD) REP400(MANY_QUAD))
+		"call %[bpf_get_prandom_u32];"
+		"r6 = r0;"
+		"r8 = twice_%= ll;"
+		SETS400(8) SETS400(16) SETS400(32) SETS400(64) SETS400(128) SETS400(256)
+		SETS400(512) SETS400(1024) SETS400(2048) SETS400(24) SETS400(40)
+		"r1 = 10;"
+		"callx r2;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
+/* 3 states that have no join get to a table of 400 functions: 3 states, and not 1200 */
+SEC("socket")
+__success __retval(11)
+__naked void callx_rodata_400_functions(void)
+{
+	asm volatile (
+		RODATA(big, REP400(MANY_QUAD))
+		"r7 = r1;"
+		"call %[bpf_get_prandom_u32];"
+		"if r0 == 1 goto 1f;"
+		"r7 = r10;"
+		"if r0 == 2 goto 1f;"
+		"r7 = 0;"
+	"1:"
+		"call %[bpf_get_prandom_u32];"
+		"r0 &= 511;"
+		"if r0 > 399 goto 2f;"
+		"r0 <<= 3;"
+		"r2 = big_%= ll;"
+		"r2 += r0;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 10;"
+		"callx r2;"
+		"if r7 != 0 goto +0;"
+		"exit;"
+	"2:"
+		"r0 = 11;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
+#else
+
+SEC("socket")
+__skip("callx needs arch JIT and compiler support")
+__success
+int dummy(void *ctx)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 2e7e65949497..3545c2a2e8f1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -171,6 +171,118 @@ __naked void callx_rodata_array_of_structs(void)
 		: __clobber_all);
 }
 
+/* one state with one of two functions, and not two states */
+SEC("socket")
+__success __retval(11)
+__log_level(2)
+__msg("r2 = *(u64 *)(r2 +0)          ; R2=func(one_of=2)")
+__naked void callx_rodata_one_of_two(void)
+{
+	asm volatile (
+		FUNC_TABLE2(tbl, add1, add2)
+		"call %[bpf_get_prandom_u32];"
+		"r6 = r0;"
+		"r6 &= 1;"
+		"r3 = r6;"
+		"r3 <<= 3;"
+		"r2 = tbl_%= ll;"
+		"r2 += r3;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r1 = 10;"
+		"callx r2;"
+		"r0 -= r6;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_stop(void)
+{
+	asm volatile (
+		"r0 = 1;"
+		"exit;"
+	);
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_go_on(void)
+{
+	asm volatile (
+		"r0 = 0;"
+		"exit;"
+	);
+}
+
+/* a helper needs to know the callback */
+SEC("socket")
+__success __retval(0)
+__naked void callx_rodata_callback_one_of_two(void)
+{
+	asm volatile (
+		FUNC_TABLE2(cbs, loop_cb_stop, loop_cb_go_on)
+		"call %[bpf_get_prandom_u32];"
+		"r0 &= 1;"
+		"r0 <<= 3;"
+		"r2 = cbs_%= ll;"
+		"r2 += r0;"
+		"r2 = *(u64 *)(r2 + 0);"
+		"r6 = r2;"
+		"r1 = 2;"
+		"r3 = 0;"
+		"r4 = 0;"
+		"call %[bpf_loop];"
+		/* libbpf resolves the pointers for progs that have callx */
+		"callx r6;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32), __imm(bpf_loop)
+		: __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long not_three(void)
+{
+	asm volatile (
+		"r0 = 0;"
+		"if r1 != 3 goto +1;"
+		"r10 = 0;"
+		"exit;"
+	);
+}
+
+/*
+ * callx is the head of a loop whose iterations are looked at one by one, because of r1.
+ * The states with one function each are in the iteration of the state with both.
+ */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callx_rodata_one_of_two_at_loop_head(void)
+{
+	asm volatile (
+		FUNC_TABLE2(tbl, add1, not_three)
+		"call %[bpf_get_prandom_u32];"
+		"r0 &= 1;"
+		"r0 <<= 3;"
+		"r6 = tbl_%= ll;"
+		"r6 += r0;"
+		"r6 = *(u64 *)(r6 + 0);"
+		"r7 = 0;"
+		"r1 = 0;"
+	"l_%=:"
+		"callx r6;"
+		"r7 += 1;"
+		"r1 = r7;"
+		"if r7 < 4 goto l_%=;"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32)
+		: __clobber_all);
+}
+
 /* and the data of the element that the index selects can be read */
 SEC("socket")
 __success __retval(16)
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index e3f5e4d14b75..c642675dd92b 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -572,6 +572,60 @@ __naked void null_check_of_stack_arg(void)
 	: __clobber_all);
 }
 
+static __naked __noinline __used long adds_arg6_arg7(long a, long b, long c, long d, long e,
+						     long f, long g)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r11 + 8);"
+	"r1 = *(u64 *)(r11 + 16);"
+	"r0 += r1;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __naked __noinline __used long subs_arg6_arg7(long a, long b, long c, long d, long e,
+						     long f, long g)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r11 + 8);"
+	"r1 = *(u64 *)(r11 + 16);"
+	"r0 -= r1;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: callx of one of two functions, args are not written")
+__failure __msg("callee expects 7 args, stack arg1 is not initialized")
+__naked void callx_one_of_two_not_written(void)
+{
+	asm volatile (
+	".pushsection .rodata,\"a\",@progbits;"
+	".balign 8;"
+"two_funcs_%=:"
+	".quad adds_arg6_arg7;"
+	".quad subs_arg6_arg7;"
+	".type two_funcs_%=, @object;"
+	".size two_funcs_%=, .-two_funcs_%=;"
+	".popsection;"
+	"call %[bpf_get_prandom_u32];"
+	"r0 &= 1;"
+	"r0 <<= 3;"
+	"r2 = two_funcs_%= ll;"
+	"r2 += r0;"
+	"r6 = *(u64 *)(r2 + 0);"
+	"r1 = 1;"
+	"r2 = 2;"
+	"r3 = 3;"
+	"r4 = 4;"
+	"r5 = 5;"
+	"callx r6;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
 /* r1 - r5 and all 19 slots. Its prototype has no parameters: what it takes is seen in its code. */
 __naked __noinline __used
 static unsigned long subprog_24args(void)
-- 
2.55.0

[PATCH bpf-next v4 45/46] selftests/bpf: Test pointers that become numbers and subprogs that return void

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Three patches let an insn pass that was refused. The prog is rejected where
it uses the result for more than it is.

"bpf: Allow tail calls and LD_ABS in subprogs that return void":
verifier_tailcall.c, verifier_ld_ind.c. A subprog that returns void has
a tail call: it loads. Its caller counts on the 1 that the subprog leaves
in r0 when the tail call is not taken: rejected, r0 is what the other prog
returns. The same for LD_ABS, which returns 0 when it fails. A subprog that
returns a pointer is refused with the message of check_btf_func().

"bpf: A part of a pointer that is spilled is a number":
verifier_spill_fill.c. 4 bytes of ctx are spilled and filled: a number with
CAP_PERFMON, 'invalid size of register spill' without. Both halves are
spilled and filled as one: the value is that of ctx, a load through it is
rejected. ctx is spilled, then 4 bytes of ctx are stored over the spill:
the slot has the same bytes as before and is not a spilled pointer any
more.

"bpf: 32-bit add to a pointer and pointer op pointer make a number":
verifier_value_illegal_alu.c. 'w0 += 1' on a pointer to a map value loads
with CAP_PERFMON and is refused without. 'r1 = r10; r1 |= r10' has the
value of r10: a store through it is rejected.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/progs/verifier_ld_ind.c     | 15 ++++
 .../selftests/bpf/progs/verifier_spill_fill.c | 47 +++++++++++++
 .../selftests/bpf/progs/verifier_tailcall.c   | 69 +++++++++++++++++++
 .../bpf/progs/verifier_value_illegal_alu.c    | 38 ++++++++++
 4 files changed, 169 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_ld_ind.c b/tools/testing/selftests/bpf/progs/verifier_ld_ind.c
index b31a5e558409..0071dc466668 100644
--- a/tools/testing/selftests/bpf/progs/verifier_ld_ind.c
+++ b/tools/testing/selftests/bpf/progs/verifier_ld_ind.c
@@ -319,6 +319,21 @@ __naked void ld_abs_void_subprog(void)
 	::: __clobber_all);
 }
 
+SEC("socket")
+__description("ld_abs: void return subprog, early exit on ld_abs failure")
+__failure __msg("R9 !read_ok")
+__naked void ld_abs_void_subprog_early_exit(void)
+{
+	asm volatile (
+	"call ldabs_void_subprog;"
+	"if r0 != 0 goto l_exit_%=;"
+	"r0 = r9;"
+	"l_exit_%=:"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
 __naked __noinline __used
 static void ldind_void_subprog(void)
 {
diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
index 34c551f97e61..0baa8e06b8d1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
@@ -171,6 +171,53 @@ __naked void check_corrupted_spill_fill_msb(void)
 "	::: __clobber_all);
 }
 
+SEC("socket")
+__description("32-bit spill of a pointer")
+__success __failure_unpriv __msg_unpriv("invalid size of register spill")
+__retval(0)
+__naked void spill_32bit_of_pointer(void)
+{
+	asm volatile ("					\
+	*(u32*)(r10 - 8) = r1;				\
+	r0 = *(u32*)(r10 - 8);				\
+	r0 -= r0;					\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("socket")
+__description("32-bit spills of both halves of a pointer")
+__failure __msg("R0 invalid mem access 'scalar'")
+__failure_unpriv __msg_unpriv("invalid size of register spill")
+__naked void spill_32bit_of_pointer_twice(void)
+{
+	asm volatile ("					\
+	*(u32*)(r10 - 8) = r1;				\
+	r1 >>= 32;					\
+	*(u32*)(r10 - 4) = r1;				\
+	/* R0 has the value of R1(ctx). It's a number */\
+	r0 = *(u64*)(r10 - 8);				\
+	r0 = *(u32*)(r0 + 0);				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("socket")
+__description("32-bit spill of a pointer over its 64-bit spill")
+__failure __msg("R0 invalid mem access 'scalar'")
+__failure_unpriv __msg_unpriv("attempt to corrupt spilled")
+__naked void spill_32bit_of_pointer_over_spill(void)
+{
+	asm volatile ("					\
+	*(u64*)(r10 - 8) = r1;				\
+	/* the same bytes. The spill of R1(ctx) is gone */\
+	*(u32*)(r10 - 8) = r1;				\
+	r0 = *(u64*)(r10 - 8);				\
+	r0 = *(u32*)(r0 + 0);				\
+	exit;						\
+"	::: __clobber_all);
+}
+
 SEC("tc")
 __description("Spill and refill a u32 const scalar.  Offset to skb->data")
 __success __retval(0)
diff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
index 5ba41dca35ea..81be4ef8c321 100644
--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
@@ -50,6 +50,75 @@ static int subprog_tail_call(void)
 	: __clobber_all);
 }
 
+__used __naked
+static void void_subprog_tail_call(void)
+{
+	asm volatile ("			\
+	r2 = %[jmp_table] ll;		\
+	r3 = 0;				\
+	call %[bpf_tail_call];		\
+	r0 = 1;				\
+	exit;				\
+"	:
+	: __imm(bpf_tail_call),
+	  __imm_addr(jmp_table)
+	: __clobber_all);
+}
+
+SEC("tc")
+__description("tail call in a subprog that returns void")
+__success
+__naked void tail_call_in_void_subprog(void)
+{
+	asm volatile ("			\
+	call void_subprog_tail_call;	\
+	r0 = 0;				\
+	exit;				\
+"	::: __clobber_all);
+}
+
+/* r0 is what the other program returns when the tail call is taken */
+SEC("tc")
+__description("tail call in a subprog that returns void, the caller counts on r0")
+__failure __msg("R9 !read_ok")
+__naked void tail_call_in_void_subprog_r0(void)
+{
+	asm volatile ("			\
+	call void_subprog_tail_call;	\
+	if r0 == 1 goto +1;		\
+	r0 = r9;			\
+	r0 = 0;				\
+	exit;				\
+"	::: __clobber_all);
+}
+
+__used __naked
+static void *ptr_subprog_tail_call(void)
+{
+	asm volatile ("			\
+	r2 = %[jmp_table] ll;		\
+	r3 = 0;				\
+	call %[bpf_tail_call];		\
+	r0 = 0;				\
+	exit;				\
+"	:
+	: __imm(bpf_tail_call),
+	  __imm_addr(jmp_table)
+	: __clobber_all);
+}
+
+SEC("tc")
+__description("tail call in a subprog that returns a pointer")
+__failure __msg("tail_call is only allowed in functions that return 'int' or 'void'")
+__naked void tail_call_in_ptr_subprog(void)
+{
+	asm volatile ("			\
+	call ptr_subprog_tail_call;	\
+	r0 = 0;				\
+	exit;				\
+"	::: __clobber_all);
+}
+
 /* The exception callback has no ctx, so its tail call can only be dead code */
 __used __noinline
 int exception_cb_tail_call(__u64 cookie)
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c b/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c
index 455f2399492b..2fcc0d08f9e7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c
@@ -330,6 +330,44 @@ l0_%=:	r0 = 0;						\
 	: __clobber_all);
 }
 
+SEC("socket")
+__description("32-bit add to a pointer")
+__success __retval(0)
+__failure_unpriv __msg_unpriv("R0 32-bit pointer arithmetic prohibited")
+__naked void ptr_add32(void)
+{
+	asm volatile ("					\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = 0;						\
+	*(u64*)(r2 + 0) = r1;				\
+	r1 = %[map_hash_48b] ll;			\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l0_%=;				\
+	w0 += 1;					\
+l0_%=:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map_hash_48b)
+	: __clobber_all);
+}
+
+SEC("socket")
+__description("or of two pointers")
+__failure __msg("R1 invalid mem access 'scalar'")
+__failure_unpriv __msg_unpriv("R1 pointer |= pointer prohibited")
+__naked void ptr_or_ptr(void)
+{
+	asm volatile ("					\
+	r1 = r10;					\
+	r1 |= r10;					\
+	r0 = 0;						\
+	*(u64*)(r1 - 8) = r0;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
 SEC("flow_dissector")
 __description("flow_keys illegal alu op with variable offset")
 __failure __msg("R7 pointer arithmetic on flow_keys prohibited")
-- 
2.55.0

[PATCH bpf-next v4 46/46] selftests/bpf: Test progs without bpf_for(), zero, can_loop and such

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

Loops in BPF progs are written for the verifier, not for the reader:

  bpf_qdisc_fifo.c       bpf_for(i, 0, sch->q.qlen) {
  arena_htab.c           for (i = zero; i < 100000 && can_loop; i++) {
  rbtree_search_kptr.c   while (rb_n && can_loop) {
  tailcall_bpf2bpf2.c    barrier_var(ret); return ret;

progs/plain_c_X.c has progs of progs/X.c in plain C, with the maps,
globals, functions and headers that they use:

  bpf_for(i, a, b)               for (i = a; i < b; i++)
  bpf_repeat(n)                  for (int i = 0; i < n; i++)
  i = zero                       i = 0
  cond && can_loop               cond
  while (can_loop)               while (1)
  barrier_var(x); cond_break;    removed
  static __always_inline         static

Only progs that bpf-next rejects in this form are taken. It says:

  12  The sequence of 8193 jumps is too complex
   7  infinite loop detected
   6  BPF program is too large. Processed 1000001 insn
   6  tail_call / LD_ABS is only allowed in functions that return 'int'
   3  invalid size of register spill

bpf-next rejects all 34, five of them in some flavours of test_progs only.
With this series they load.

RUN_TESTS() loads each prog, and runs it where the result is known.
arena_list, bpf_ma_ttrace and dynptr_success have short drivers.
Nothing bounds a walk of a list or a tree: the verifier adds may_goto.
Five functions of xdp_lb_bench.c become subprogs, two with stack args.
Three progs are unsafe with and without the macros: they keep __failure
and __msg(). With GCC 11 of the 34 are skipped: __skip() says why.
pyperf600_bpf_loop and loop_takes_reference_once are still rejected in
plain C: left out.

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 .../selftests/bpf/prog_tests/plain_c.c        | 160 ++++
 .../selftests/bpf/progs/plain_c_arena_htab.c  | 222 ++++++
 .../bpf/progs/plain_c_arena_htab_asm.c        |   5 +
 .../selftests/bpf/progs/plain_c_arena_list.c  | 181 +++++
 .../selftests/bpf/progs/plain_c_arena_race.c  |  45 ++
 .../bpf/progs/plain_c_arena_strsearch.c       | 263 +++++++
 .../bpf/progs/plain_c_bpf_ma_ttrace.c         |  53 ++
 .../bpf/progs/plain_c_bpf_nop_bench.c         |  75 ++
 .../bpf/progs/plain_c_bpf_qdisc_fifo.c        | 132 ++++
 .../bpf/progs/plain_c_bpf_qdisc_fq.c          | 728 ++++++++++++++++++
 .../bpf/progs/plain_c_cpumask_success.c       |  83 ++
 .../bpf/progs/plain_c_dynptr_success.c        | 175 +++++
 .../selftests/bpf/progs/plain_c_iters.c       | 147 ++++
 .../bpf/progs/plain_c_mem_rdonly_untrusted.c  |  24 +
 .../bpf/progs/plain_c_mptcp_subflow.c         | 116 +++
 .../bpf/progs/plain_c_rbtree_search.c         | 136 ++++
 .../bpf/progs/plain_c_rbtree_search_kptr.c    | 264 +++++++
 .../bpf/progs/plain_c_strobemeta_bpf_loop.c   | 583 ++++++++++++++
 .../bpf/progs/plain_c_tailcall_bpf2bpf2.c     |  40 +
 .../plain_c_tailcall_bpf2bpf_hierarchy1.c     |  44 ++
 .../plain_c_tailcall_bpf2bpf_hierarchy2.c     |  81 ++
 .../plain_c_tailcall_bpf2bpf_hierarchy3.c     |  74 ++
 .../bpf/progs/plain_c_tailcall_callback.c     |  80 ++
 .../bpf/progs/plain_c_test_wakeup_source.c    |  93 +++
 .../plain_c_tracing_stack_arg_tailcall_tgt.c  |  55 ++
 .../bpf/progs/plain_c_verifier_arena_large.c  | 122 +++
 .../plain_c_verifier_iterating_callbacks.c    |  25 +
 .../bpf/progs/plain_c_verifier_loop_as_is.c   |  40 +
 .../bpf/progs/plain_c_xdp_lb_bench.c          | 723 +++++++++++++++++
 29 files changed, 4769 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/plain_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_htab.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_htab_asm.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_list.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_race.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_arena_strsearch.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_ma_ttrace.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_nop_bench.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fifo.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fq.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_cpumask_success.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_dynptr_success.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_iters.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_mem_rdonly_untrusted.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_mptcp_subflow.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_rbtree_search.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_rbtree_search_kptr.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_strobemeta_bpf_loop.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf2.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy1.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy2.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy3.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tailcall_callback.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_tracing_stack_arg_tailcall_tgt.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_verifier_arena_large.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_verifier_iterating_callbacks.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_verifier_loop_as_is.c
 create mode 100644 tools/testing/selftests/bpf/progs/plain_c_xdp_lb_bench.c

diff --git a/tools/testing/selftests/bpf/prog_tests/plain_c.c b/tools/testing/selftests/bpf/prog_tests/plain_c.c
new file mode 100644
index 000000000000..b7305118a454
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/plain_c.c
@@ -0,0 +1,160 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * progs/plain_c_X.c has the progs of progs/X.c that needed bpf_for(), bpf_repeat(),
+ * 'i = zero', barrier_var(), can_loop or cond_break to get past the verifier, without these.
+ */
+
+#include <test_progs.h>
+
+/* types of globals in the skeletons */
+#include "bpf_arena_list.h"
+
+#include "plain_c_arena_htab.skel.h"
+#include "plain_c_arena_htab_asm.skel.h"
+#include "plain_c_arena_list.skel.h"
+#include "plain_c_arena_race.skel.h"
+#include "plain_c_arena_strsearch.skel.h"
+#include "plain_c_bpf_ma_ttrace.skel.h"
+#include "plain_c_bpf_nop_bench.skel.h"
+#include "plain_c_bpf_qdisc_fifo.skel.h"
+#include "plain_c_bpf_qdisc_fq.skel.h"
+#include "plain_c_cpumask_success.skel.h"
+#include "plain_c_dynptr_success.skel.h"
+#include "plain_c_iters.skel.h"
+#include "plain_c_mem_rdonly_untrusted.skel.h"
+#include "plain_c_mptcp_subflow.skel.h"
+#include "plain_c_rbtree_search.skel.h"
+#include "plain_c_rbtree_search_kptr.skel.h"
+#include "plain_c_strobemeta_bpf_loop.skel.h"
+#include "plain_c_tailcall_bpf2bpf2.skel.h"
+#include "plain_c_tailcall_bpf2bpf_hierarchy1.skel.h"
+#include "plain_c_tailcall_bpf2bpf_hierarchy2.skel.h"
+#include "plain_c_tailcall_bpf2bpf_hierarchy3.skel.h"
+#include "plain_c_tailcall_callback.skel.h"
+#include "plain_c_test_wakeup_source.skel.h"
+#include "plain_c_tracing_stack_arg_tailcall_tgt.skel.h"
+#include "plain_c_verifier_arena_large.skel.h"
+#include "plain_c_verifier_iterating_callbacks.skel.h"
+#include "plain_c_verifier_loop_as_is.skel.h"
+#include "plain_c_xdp_lb_bench.skel.h"
+
+#define RUN(name) RUN_TESTS(plain_c_##name)
+
+/* prog_tests/arena_list.c: add 1000 elems, delete them, compare the sums */
+static void arena_list_add_del(void)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct plain_c_arena_list *skel;
+	int cnt = 1000, sum = cnt * (cnt - 1) / 2;
+
+	skel = plain_c_arena_list__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "open_and_load"))
+		return;
+	skel->bss->cnt = cnt;
+	ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_add), &opts),
+		  "add");
+	if (skel->bss->skip) {
+		test__skip();
+		goto out;
+	}
+	ASSERT_EQ(skel->arena->arena_sum, sum, "sum of elems");
+	ASSERT_EQ(skel->arena->test_val, cnt + 1, "num of elems");
+	ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_del), &opts),
+		  "del");
+	ASSERT_EQ(skel->bss->list_sum, sum, "sum of deleted elems");
+out:
+	plain_c_arena_list__destroy(skel);
+}
+
+/* prog_tests/bpf_ma_ttrace.c: the loop finds one cache per cpu */
+static void ma_ttrace_run(void)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct plain_c_bpf_ma_ttrace *skel;
+
+	skel = plain_c_bpf_ma_ttrace__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "open_and_load"))
+		return;
+	skel->bss->nr_cpus = libbpf_num_possible_cpus();
+	ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.check_ttrace), &opts),
+		  "test_run");
+	ASSERT_OK(opts.retval, "retval");
+	ASSERT_EQ(skel->bss->nr_caches, skel->bss->nr_cpus, "nr_caches");
+	plain_c_bpf_ma_ttrace__destroy(skel);
+}
+
+/* prog_tests/dynptr.c: run each prog, it leaves 0 in 'err' */
+static void dynptr_run(void)
+{
+	char data[90000];
+	LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = data, .repeat = 1,
+		    .data_size_in = getpagesize() == 65536 ? sizeof(data) : 5000);
+	struct plain_c_dynptr_success *skel;
+	struct bpf_program *prog;
+
+	skel = plain_c_dynptr_success__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+	bpf_object__for_each_program(prog, skel->obj)
+		bpf_program__set_autoload(prog, true);
+	if (!ASSERT_OK(plain_c_dynptr_success__load(skel), "load"))
+		goto out;
+	bpf_object__for_each_program(prog, skel->obj) {
+		skel->bss->err = 0;
+		ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(prog), &opts), "test_run");
+		ASSERT_EQ(skel->bss->err, 0, bpf_program__name(prog));
+	}
+out:
+	plain_c_dynptr_success__destroy(skel);
+}
+
+void test_plain_c_arena_htab(void) { RUN(arena_htab); }
+void test_plain_c_arena_htab_asm(void) { RUN(arena_htab_asm); }
+
+void test_plain_c_arena_list(void)
+{
+	RUN(arena_list);
+	if (test__start_subtest("run"))
+		arena_list_add_del();
+}
+
+void test_plain_c_arena_race(void) { RUN(arena_race); }
+void test_plain_c_arena_strsearch(void) { RUN(arena_strsearch); }
+
+void test_plain_c_bpf_ma_ttrace(void)
+{
+	RUN(bpf_ma_ttrace);
+	if (test__start_subtest("run"))
+		ma_ttrace_run();
+}
+
+void test_plain_c_bpf_nop_bench(void) { RUN(bpf_nop_bench); }
+void test_plain_c_bpf_qdisc_fifo(void) { RUN(bpf_qdisc_fifo); }
+void test_plain_c_bpf_qdisc_fq(void) { RUN(bpf_qdisc_fq); }
+void test_plain_c_cpumask_success(void) { RUN(cpumask_success); }
+
+void test_plain_c_dynptr_success(void)
+{
+	RUN(dynptr_success);
+	if (test__start_subtest("run"))
+		dynptr_run();
+}
+
+void test_plain_c_iters(void) { RUN(iters); }
+void test_plain_c_mem_rdonly_untrusted(void) { RUN(mem_rdonly_untrusted); }
+void test_plain_c_mptcp_subflow(void) { RUN(mptcp_subflow); }
+void test_plain_c_rbtree_search(void) { RUN(rbtree_search); }
+void test_plain_c_rbtree_search_kptr(void) { RUN(rbtree_search_kptr); }
+void test_plain_c_strobemeta_bpf_loop(void) { RUN(strobemeta_bpf_loop); }
+void test_plain_c_tailcall_bpf2bpf2(void) { RUN(tailcall_bpf2bpf2); }
+void test_plain_c_tailcall_bpf2bpf_hierarchy1(void) { RUN(tailcall_bpf2bpf_hierarchy1); }
+void test_plain_c_tailcall_bpf2bpf_hierarchy2(void) { RUN(tailcall_bpf2bpf_hierarchy2); }
+void test_plain_c_tailcall_bpf2bpf_hierarchy3(void) { RUN(tailcall_bpf2bpf_hierarchy3); }
+void test_plain_c_tailcall_callback(void) { RUN(tailcall_callback); }
+void test_plain_c_test_wakeup_source(void) { RUN(test_wakeup_source); }
+void test_plain_c_tracing_stack_arg_tailcall_tgt(void) { RUN(tracing_stack_arg_tailcall_tgt); }
+void test_plain_c_verifier_arena_large(void) { RUN(verifier_arena_large); }
+void test_plain_c_verifier_iterating_callbacks(void) { RUN(verifier_iterating_callbacks); }
+void test_plain_c_verifier_loop_as_is(void) { RUN(verifier_loop_as_is); }
+void test_plain_c_xdp_lb_bench(void) { RUN(xdp_lb_bench); }
diff --git a/tools/testing/selftests/bpf/progs/plain_c_arena_htab.c b/tools/testing/selftests/bpf/progs/plain_c_arena_htab.c
new file mode 100644
index 000000000000..10a50b5058dd
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_arena_htab.c
@@ -0,0 +1,222 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
+/* arena_htab.c, bpf_arena_htab.h and bpf_arena_list.h in plain C */
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, 100); /* number of pages */
+} arena SEC(".maps");
+
+#include <errno.h>
+#include "bpf_arena_alloc.h"
+#include <bpf_arena_common.h>
+
+struct arena_list_node;
+
+typedef struct arena_list_node __arena arena_list_node_t;
+
+struct arena_list_node {
+	arena_list_node_t *next;
+	arena_list_node_t * __arena *pprev;
+};
+
+struct arena_list_head {
+	struct arena_list_node __arena *first;
+};
+typedef struct arena_list_head __arena arena_list_head_t;
+
+#define list_entry(ptr, type, member) arena_container_of(ptr, type, member)
+
+#define list_entry_safe(ptr, type, member) \
+	({ typeof(*ptr) * ___ptr = (ptr); \
+	 ___ptr ? ({ cast_kern(___ptr); list_entry(___ptr, type, member); }) : NULL; \
+	 })
+
+/* Safely walk link list elements. Deletion of elements is allowed. */
+#define list_for_each_entry(pos, head, member)					\
+	for (void * ___tmp = (pos = list_entry_safe((head)->first,		\
+						    typeof(*(pos)), member),	\
+			      (void *)0);					\
+	     pos && ({ ___tmp = (void *)pos->member.next; 1; });                \
+	     pos = list_entry_safe((void __arena *)___tmp, typeof(*(pos)), member))
+
+static inline void list_add_head(arena_list_node_t *n, arena_list_head_t *h)
+{
+	arena_list_node_t *first = h->first, * __arena *tmp;
+
+	cast_user(first);
+	cast_kern(n);
+	WRITE_ONCE(n->next, first);
+	cast_kern(first);
+	if (first) {
+		tmp = &n->next;
+		cast_user(tmp);
+		WRITE_ONCE(first->pprev, tmp);
+	}
+	cast_user(n);
+	WRITE_ONCE(h->first, n);
+
+	tmp = &h->first;
+	cast_user(tmp);
+	cast_kern(n);
+	WRITE_ONCE(n->pprev, tmp);
+}
+
+static inline void __list_del(arena_list_node_t *n)
+{
+	arena_list_node_t *next = n->next;
+	arena_list_node_t * __arena *pprev = n->pprev;
+
+	cast_user(next);
+	cast_kern(pprev);
+	WRITE_ONCE(*pprev, next);
+	if (next) {
+		cast_user(pprev);
+		cast_kern(next);
+		WRITE_ONCE(next->pprev, pprev);
+	}
+}
+
+#define POISON_POINTER_DELTA 0
+
+#define LIST_POISON1  ((void __arena *) 0x100 + POISON_POINTER_DELTA)
+#define LIST_POISON2  ((void __arena *) 0x122 + POISON_POINTER_DELTA)
+
+static inline void list_del(arena_list_node_t *n)
+{
+	__list_del(n);
+	n->next = LIST_POISON1;
+	n->pprev = LIST_POISON2;
+}
+
+struct htab_bucket {
+	struct arena_list_head head;
+};
+typedef struct htab_bucket __arena htab_bucket_t;
+
+struct htab {
+	htab_bucket_t *buckets;
+	int n_buckets;
+};
+
+static inline htab_bucket_t *__select_bucket(struct htab __arena *htab, __u32 hash)
+{
+	htab_bucket_t *b = htab->buckets;
+
+	cast_kern(b);
+	return &b[hash & (htab->n_buckets - 1)];
+}
+
+static inline arena_list_head_t *select_bucket(struct htab __arena *htab, __u32 hash)
+{
+	return &__select_bucket(htab, hash)->head;
+}
+
+struct hashtab_elem {
+	int hash;
+	int key;
+	int value;
+	struct arena_list_node hash_node;
+};
+typedef struct hashtab_elem __arena hashtab_elem_t;
+
+static hashtab_elem_t *lookup_elem_raw(arena_list_head_t *head, __u32 hash, int key)
+{
+	hashtab_elem_t *l;
+
+	list_for_each_entry(l, head, hash_node)
+		if (l->hash == hash && l->key == key)
+			return l;
+
+	return NULL;
+}
+
+static int htab_hash(int key)
+{
+	return key;
+}
+
+__weak int htab_update_elem(struct htab __arena *htab, int key, int value)
+{
+	hashtab_elem_t *l_new = NULL, *l_old;
+	arena_list_head_t *head;
+
+	cast_kern(htab);
+	head = select_bucket(htab, key);
+	l_old = lookup_elem_raw(head, htab_hash(key), key);
+
+	l_new = bpf_alloc(sizeof(*l_new));
+	if (!l_new)
+		return -ENOMEM;
+	l_new->key = key;
+	l_new->hash = htab_hash(key);
+	l_new->value = value;
+
+	list_add_head(&l_new->hash_node, head);
+	if (l_old) {
+		list_del(&l_old->hash_node);
+		bpf_free(l_old);
+	}
+	return 0;
+}
+
+void htab_init(struct htab __arena *htab)
+{
+	void __arena *buckets = bpf_arena_alloc_pages(&arena, NULL, 2, NUMA_NO_NODE, 0);
+
+	cast_user(buckets);
+	htab->buckets = buckets;
+	htab->n_buckets = 2 * PAGE_SIZE / sizeof(struct htab_bucket);
+}
+
+void __arena *htab_for_user;
+bool skip = false;
+
+char __arena arr1[100000];
+char arr2[1000];
+
+SEC("syscall")
+__success __retval(0)
+#if !defined(__clang__) && defined(BPF_ARENA_FORCE_ASM)
+__skip("GCC calls htab_init(): a global function gets an arena pointer as a scalar")
+#endif
+int arena_htab_llvm(void *ctx)
+{
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)
+	struct htab __arena *htab;
+	char __arena *arr = arr1;
+	__u64 i;
+
+	htab = bpf_alloc(sizeof(*htab));
+	cast_kern(htab);
+	htab_init(htab);
+
+	/* first run. No old elems in the table */
+	for (i = 0; i < 100000; i++) {
+		htab_update_elem(htab, i, i);
+		arr[i] = i;
+	}
+
+	/* should replace some elems with new ones */
+	for (i = 0; i < 1000; i++) {
+		htab_update_elem(htab, i, i);
+		/* Access mem to make the verifier use bounded loop logic */
+		arr2[i] = i;
+	}
+	cast_user(htab);
+	htab_for_user = htab;
+#else
+	skip = true;
+#endif
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_arena_htab_asm.c b/tools/testing/selftests/bpf/progs/plain_c_arena_htab_asm.c
new file mode 100644
index 000000000000..0b74223e8c8c
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_arena_htab_asm.c
@@ -0,0 +1,5 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
+#define BPF_ARENA_FORCE_ASM
+#define arena_htab_llvm arena_htab_asm
+#include "plain_c_arena_htab.c"
diff --git a/tools/testing/selftests/bpf/progs/plain_c_arena_list.c b/tools/testing/selftests/bpf/progs/plain_c_arena_list.c
new file mode 100644
index 000000000000..996488b6eabc
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_arena_list.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
+/* arena_list.c and bpf_arena_list.h in plain C */
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, 100); /* number of pages */
+#ifdef __TARGET_ARCH_arm64
+	__ulong(map_extra, 0x1ull << 32); /* start of mmap() region */
+#else
+	__ulong(map_extra, 0x1ull << 44); /* start of mmap() region */
+#endif
+} arena SEC(".maps");
+
+#include "bpf_arena_alloc.h"
+#include <bpf_arena_common.h>
+
+struct arena_list_node;
+
+typedef struct arena_list_node __arena arena_list_node_t;
+
+struct arena_list_node {
+	arena_list_node_t *next;
+	arena_list_node_t * __arena *pprev;
+};
+
+struct arena_list_head {
+	struct arena_list_node __arena *first;
+};
+typedef struct arena_list_head __arena arena_list_head_t;
+
+#define list_entry(ptr, type, member) arena_container_of(ptr, type, member)
+
+#define list_entry_safe(ptr, type, member) \
+	({ typeof(*ptr) * ___ptr = (ptr); \
+	 ___ptr ? ({ cast_kern(___ptr); list_entry(___ptr, type, member); }) : NULL; \
+	 })
+
+/* Safely walk link list elements. Deletion of elements is allowed. */
+#define list_for_each_entry(pos, head, member)					\
+	for (void * ___tmp = (pos = list_entry_safe((head)->first,		\
+						    typeof(*(pos)), member),	\
+			      (void *)0);					\
+	     pos && ({ ___tmp = (void *)pos->member.next; 1; });                \
+	     pos = list_entry_safe((void __arena *)___tmp, typeof(*(pos)), member))
+
+__attribute__((unused))
+static inline void list_add_head(arena_list_node_t *n, arena_list_head_t *h)
+{
+	arena_list_node_t *first = h->first, * __arena *tmp;
+
+	cast_user(first);
+	cast_kern(n);
+	WRITE_ONCE(n->next, first);
+	cast_kern(first);
+	if (first) {
+		tmp = &n->next;
+		cast_user(tmp);
+		WRITE_ONCE(first->pprev, tmp);
+	}
+	cast_user(n);
+	WRITE_ONCE(h->first, n);
+
+	tmp = &h->first;
+	cast_user(tmp);
+	cast_kern(n);
+	WRITE_ONCE(n->pprev, tmp);
+}
+
+static inline void __list_del(arena_list_node_t *n)
+{
+	arena_list_node_t *next = n->next;
+	arena_list_node_t * __arena *pprev = n->pprev;
+
+	cast_user(next);
+	cast_kern(pprev);
+	WRITE_ONCE(*pprev, next);
+	if (next) {
+		cast_user(pprev);
+		cast_kern(next);
+		WRITE_ONCE(next->pprev, pprev);
+	}
+}
+
+#define POISON_POINTER_DELTA 0
+
+#define LIST_POISON1  ((void __arena *) 0x100 + POISON_POINTER_DELTA)
+#define LIST_POISON2  ((void __arena *) 0x122 + POISON_POINTER_DELTA)
+
+__attribute__((unused))
+static inline void list_del(arena_list_node_t *n)
+{
+	__list_del(n);
+	n->next = LIST_POISON1;
+	n->pprev = LIST_POISON2;
+}
+
+struct elem {
+	struct arena_list_node node;
+	__u64 value;
+};
+
+struct arena_list_head __arena *list_head;
+int list_sum;
+int cnt;
+bool skip = false;
+const volatile bool nonsleepable = false;
+
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+long __arena arena_sum;
+int __arena test_val = 1;
+struct arena_list_head __arena global_head;
+#else
+long arena_sum SEC(".addr_space.1");
+int test_val SEC(".addr_space.1");
+#endif
+
+void bpf_rcu_read_lock(void) __ksym;
+void bpf_rcu_read_unlock(void) __ksym;
+
+SEC("syscall")
+__success
+int arena_list_add(void *ctx)
+{
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+	__u64 i;
+
+	list_head = &global_head;
+
+	for (i = 0; i < cnt; i++) {
+		struct elem __arena *n = bpf_alloc(sizeof(*n));
+
+		test_val++;
+		n->value = i;
+		arena_sum += i;
+		list_add_head(&n->node, list_head);
+	}
+#else
+	skip = true;
+#endif
+	return 0;
+}
+
+SEC("syscall")
+__success
+int arena_list_del(void *ctx)
+{
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+	struct elem __arena *n;
+	int sum = 0;
+
+	/* Take rcu_read_lock to test non-sleepable context */
+	if (nonsleepable)
+		bpf_rcu_read_lock();
+
+	arena_sum = 0;
+	list_for_each_entry(n, list_head, node) {
+		sum += n->value;
+		arena_sum += n->value;
+		list_del(&n->node);
+		bpf_free(n);
+	}
+	list_sum = sum;
+
+	if (nonsleepable)
+		bpf_rcu_read_unlock();
+#else
+	skip = true;
+#endif
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_arena_race.c b/tools/testing/selftests/bpf/progs/plain_c_arena_race.c
new file mode 100644
index 000000000000..ea200b857a8a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_arena_race.c
@@ -0,0 +1,45 @@
+// SPDX-License-Identifier: GPL-2.0
+/* arena_race.c in plain C */
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+#include "bpf_misc.h"
+
+int target_tid;
+int pause_on_zap;
+int zap_entered;
+int release;
+int timed_out;
+
+static void wait_for_release(void)
+{
+	while (!*(volatile int *)&release)
+		;
+	if (!*(volatile int *)&release)
+		timed_out = 1;
+}
+
+/*
+ * Pause a free while it zaps user mappings: after it cleared the kernel PTEs
+ * and flushed the TLB, but before it releases the range. zap_vma_range() is
+ * generic MM code, so it is an fentry target even where
+ * flush_tlb_kernel_range() is inline (arm64).
+ */
+SEC("?fentry/zap_vma_range")
+__success
+int BPF_PROG(trace_zap, struct vm_area_struct *vma, unsigned long address,
+	     unsigned long size)
+{
+	if ((__u32)bpf_get_current_pid_tgid() != target_tid)
+		return 0;
+	zap_entered = 1;
+	if (pause_on_zap)
+		wait_for_release();
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_arena_strsearch.c b/tools/testing/selftests/bpf/progs/plain_c_arena_strsearch.c
new file mode 100644
index 000000000000..f05cf023c4d2
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_arena_strsearch.c
@@ -0,0 +1,263 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+/* arena_strsearch.c and bpf_arena_strsearch.h in plain C */
+#include <vmlinux.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, 100); /* number of pages */
+} arena SEC(".maps");
+
+#include <bpf_arena_common.h>
+
+__noinline int bpf_arena_strlen(const char __arena *s)
+{
+	const char __arena *sc;
+
+	for (sc = s; *sc != '\0'; ++sc)
+		;
+	return sc - s;
+}
+
+/**
+ * glob_match - Shell-style pattern matching, like !fnmatch(pat, str, 0)
+ * @pat: Shell-style pattern to match, e.g. "*.[ch]".
+ * @str: String to match.  The pattern must match the entire string.
+ *
+ * Perform shell-style glob matching, returning true (1) if the match
+ * succeeds, or false (0) if it fails.  Equivalent to !fnmatch(@pat, @str, 0).
+ *
+ * Pattern metacharacters are ?, *, [ and \.
+ * (And, inside character classes, !, - and ].)
+ *
+ * This is small and simple implementation intended for device blacklists
+ * where a string is matched against a number of patterns.  Thus, it
+ * does not preprocess the patterns.  It is non-recursive, and run-time
+ * is at most quadratic: strlen(@str)*strlen(@pat).
+ *
+ * An example of the worst case is glob_match("*aaaaa", "aaaaaaaaaa");
+ * it takes 6 passes over the pattern before matching the string.
+ *
+ * Like !fnmatch(@pat, @str, 0) and unlike the shell, this does NOT
+ * treat / or leading . specially; it isn't actually used for pathnames.
+ *
+ * Note that according to glob(7) (and unlike bash), character classes
+ * are complemented by a leading !; this does not support the regex-style
+ * [^a-z] syntax.
+ *
+ * An opening bracket without a matching close is matched literally.
+ */
+__noinline bool glob_match(char const __arena *pat, char const __arena *str)
+{
+	/*
+	 * Backtrack to previous * on mismatch and retry starting one
+	 * character later in the string.  Because * matches all characters
+	 * (no exception for /), it can be easily proved that there's
+	 * never a need to backtrack multiple levels.
+	 */
+	char const __arena *back_pat = NULL, *back_str;
+
+	/*
+	 * Loop over each token (character or class) in pat, matching
+	 * it against the remaining unmatched tail of str.  Return false
+	 * on mismatch, or true after matching the trailing nul bytes.
+	 */
+	for (;;) {
+		unsigned char c = *str++;
+		unsigned char d = *pat++;
+
+		switch (d) {
+		case '?':	/* Wildcard: anything but nul */
+			if (c == '\0')
+				return false;
+			break;
+		case '*':	/* Any-length wildcard */
+			if (*pat == '\0')	/* Optimize trailing * case */
+				return true;
+			back_pat = pat;
+			back_str = --str;	/* Allow zero-length match */
+			break;
+		case '[': {	/* Character class */
+			bool match = false, inverted = (*pat == '!');
+			char const __arena *class = pat + inverted;
+			unsigned char a = *class++;
+
+			/*
+			 * Iterate over each span in the character class.
+			 * A span is either a single character a, or a
+			 * range a-b.  The first span may begin with ']'.
+			 */
+			do {
+				unsigned char b = a;
+
+				if (a == '\0')	/* Malformed */
+					goto literal;
+
+				if (class[0] == '-' && class[1] != ']') {
+					b = class[1];
+
+					if (b == '\0')
+						goto literal;
+
+					class += 2;
+					/* Any special action if a > b? */
+				}
+				match |= (a <= c && c <= b);
+			} while ((a = *class++) != ']');
+
+			if (match == inverted)
+				goto backtrack;
+			pat = class;
+			}
+			break;
+		case '\\':
+			d = *pat++;
+			__attribute__((__fallthrough__));
+		default:	/* Literal character */
+literal:
+			if (c == d) {
+				if (d == '\0')
+					return true;
+				break;
+			}
+backtrack:
+			if (c == '\0' || !back_pat)
+				return false;	/* No point continuing */
+			/* Try again from last *, one character later in str. */
+			pat = back_pat;
+			str = ++back_str;
+			break;
+		}
+	}
+	return false;
+}
+
+static bool test(char const __arena *pat, char const __arena *str, bool expected)
+{
+	bool match = glob_match(pat, str);
+	bool success = match == expected;
+
+	/* bpf_printk("glob_match %s %s res %d ok %d", pat, str, match, success); */
+	return success;
+}
+
+/*
+ * The tests are all jammed together in one array to make it simpler
+ * to place that array in the .init.rodata section.  The obvious
+ * "array of structures containing char *" has no way to force the
+ * pointed-to strings to be in a particular section.
+ *
+ * Anyway, a test consists of:
+ * 1. Expected glob_match result: '1' or '0'.
+ * 2. Pattern to match: null-terminated string
+ * 3. String to match against: null-terminated string
+ *
+ * The list of tests is terminated with a final '\0' instead of
+ * a glob_match result character.
+ */
+static const char __arena glob_tests[] =
+	/* Some basic tests */
+	"1" "a\0" "a\0"
+	"0" "a\0" "b\0"
+	"0" "a\0" "aa\0"
+	"0" "a\0" "\0"
+	"1" "\0" "\0"
+	"0" "\0" "a\0"
+	/* Simple character class tests */
+	"1" "[a]\0" "a\0"
+	"0" "[a]\0" "b\0"
+	"0" "[!a]\0" "a\0"
+	"1" "[!a]\0" "b\0"
+	"1" "[ab]\0" "a\0"
+	"1" "[ab]\0" "b\0"
+	"0" "[ab]\0" "c\0"
+	"1" "[!ab]\0" "c\0"
+	"1" "[a-c]\0" "b\0"
+	"0" "[a-c]\0" "d\0"
+	/* Corner cases in character class parsing */
+	"1" "[a-c-e-g]\0" "-\0"
+	"0" "[a-c-e-g]\0" "d\0"
+	"1" "[a-c-e-g]\0" "f\0"
+	"1" "[]a-ceg-ik[]\0" "a\0"
+	"1" "[]a-ceg-ik[]\0" "]\0"
+	"1" "[]a-ceg-ik[]\0" "[\0"
+	"1" "[]a-ceg-ik[]\0" "h\0"
+	"0" "[]a-ceg-ik[]\0" "f\0"
+	"0" "[!]a-ceg-ik[]\0" "h\0"
+	"0" "[!]a-ceg-ik[]\0" "]\0"
+	"1" "[!]a-ceg-ik[]\0" "f\0"
+	/* Simple wild cards */
+	"1" "?\0" "a\0"
+	"0" "?\0" "aa\0"
+	"0" "??\0" "a\0"
+	"1" "?x?\0" "axb\0"
+	"0" "?x?\0" "abx\0"
+	"0" "?x?\0" "xab\0"
+	/* Asterisk wild cards (backtracking) */
+	"0" "*??\0" "a\0"
+	"1" "*??\0" "ab\0"
+	"1" "*??\0" "abc\0"
+	"1" "*??\0" "abcd\0"
+	"0" "??*\0" "a\0"
+	"1" "??*\0" "ab\0"
+	"1" "??*\0" "abc\0"
+	"1" "??*\0" "abcd\0"
+	"0" "?*?\0" "a\0"
+	"1" "?*?\0" "ab\0"
+	"1" "?*?\0" "abc\0"
+	"1" "?*?\0" "abcd\0"
+	"1" "*b\0" "b\0"
+	"1" "*b\0" "ab\0"
+	"0" "*b\0" "ba\0"
+	"1" "*b\0" "bb\0"
+	"1" "*b\0" "abb\0"
+	"1" "*b\0" "bab\0"
+	"1" "*bc\0" "abbc\0"
+	"1" "*bc\0" "bc\0"
+	"1" "*bc\0" "bbc\0"
+	"1" "*bc\0" "bcbc\0"
+	/* Multiple asterisks (complex backtracking) */
+	"1" "*ac*\0" "abacadaeafag\0"
+	"1" "*ac*ae*ag*\0" "abacadaeafag\0"
+	"1" "*a*b*[bc]*[ef]*g*\0" "abacadaeafag\0"
+	"0" "*a*b*[ef]*[cd]*g*\0" "abacadaeafag\0"
+	"1" "*abcd*\0" "abcabcabcabcdefg\0"
+	"1" "*ab*cd*\0" "abcabcabcabcdefg\0"
+	"1" "*abcd*abcdef*\0" "abcabcdabcdeabcdefg\0"
+	"0" "*abcd*\0" "abcabcabcabcefg\0"
+	"0" "*ab*cd*\0" "abcabcabcabcefg\0";
+
+SEC("syscall")
+__success __retval(0) __skip_without_arena
+int arena_strsearch(void *ctx)
+{
+	unsigned successes = 0;
+	unsigned n = 0;
+	char const __arena *p = glob_tests;
+
+	/*
+	 * Tests are jammed together in a string.  The first byte is '1'
+	 * or '0' to indicate the expected outcome, or '\0' to indicate the
+	 * end of the tests.  Then come two null-terminated strings: the
+	 * pattern and the string to match it against.
+	 */
+	while (*p) {
+		bool expected = *p++ & 1;
+		char const __arena *pat = p;
+
+		p += bpf_arena_strlen(p) + 1;
+		successes += test(pat, p, expected);
+		p += bpf_arena_strlen(p) + 1;
+		n++;
+	}
+
+	n -= successes;
+	/* bpf_printk("glob: %u self-tests passed, %u failed\n", successes, n); */
+
+	return n ? -1 : 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_bpf_ma_ttrace.c b/tools/testing/selftests/bpf/progs/plain_c_bpf_ma_ttrace.c
new file mode 100644
index 000000000000..71df2e037f5b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_bpf_ma_ttrace.c
@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: GPL-2.0
+/* bpf_ma_ttrace.c in plain C */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_kfuncs.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(map_flags, BPF_F_NO_PREALLOC);
+	__uint(max_entries, 4096);
+	__type(key, int);
+	__type(value, long);
+} htab SEC(".maps");
+
+extern const void __per_cpu_offset __ksym;
+
+int nr_cpus;
+int nr_caches;
+int in_progress;
+int not_freed;
+
+/* Look at bpf_mem_cache of every cpu that htab allocates its elements from */
+SEC("syscall")
+__success
+int check_ttrace(void *ctx)
+{
+	struct bpf_htab *h = bpf_core_cast(&htab, struct bpf_htab);
+	unsigned long cache = (unsigned long)BPF_CORE_READ(h, ma.cache);
+	const unsigned long *offsets = &__per_cpu_offset;
+	struct bpf_mem_cache *c;
+	unsigned long off;
+	int cpu;
+
+	nr_caches = 0;
+	in_progress = 0;
+	not_freed = 0;
+	for (cpu = 0; cpu < nr_cpus; cpu++) {
+		if (bpf_probe_read_kernel(&off, sizeof(off), offsets + cpu))
+			return -1;
+		c = bpf_core_cast((void *)(cache + off), struct bpf_mem_cache);
+		if (c->unit_size)
+			nr_caches++;
+		if (c->call_rcu_ttrace_in_progress.counter)
+			in_progress++;
+		if (c->free_by_rcu_ttrace.first || c->waiting_for_gp_ttrace.first)
+			not_freed++;
+	}
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_bpf_nop_bench.c b/tools/testing/selftests/bpf/progs/plain_c_bpf_nop_bench.c
new file mode 100644
index 000000000000..fdb7260b0fdf
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_bpf_nop_bench.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+/* bpf_nop_bench.c and bench_bpf_timing.bpf.h in plain C */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+#include <stdbool.h>
+#include "bpf_misc.h"
+
+#ifndef BENCH_NR_SAMPLES
+#define BENCH_NR_SAMPLES	4096
+#endif
+#ifndef BENCH_NR_CPUS
+#define BENCH_NR_CPUS		256
+#endif
+#define BENCH_CPU_MASK		(BENCH_NR_CPUS - 1)
+
+__u64 timing_samples[BENCH_NR_CPUS][BENCH_NR_SAMPLES];
+__u32 timing_idx[BENCH_NR_CPUS];
+
+volatile __u32 batch_iters;
+volatile __u32 timing_enabled;
+
+static void bench_record_sample(__u64 elapsed_ns)
+{
+	__u32 cpu, idx;
+
+	if (!timing_enabled)
+		return;
+
+	cpu = bpf_get_smp_processor_id() & BENCH_CPU_MASK;
+	idx = timing_idx[cpu];
+
+	if (idx >= BENCH_NR_SAMPLES)
+		return;
+
+	timing_samples[cpu][idx] = elapsed_ns;
+	timing_idx[cpu] = idx + 1;
+}
+
+/*
+ * @body:  expression to time; return value (int) stored in __bench_result.
+ * @reset: undo body's side-effects so each iteration starts identically.
+ *         May reference __bench_result.  Use ({}) for empty reset.
+ *
+ * Runs batch_iters timed iterations, then one untimed iteration whose
+ * return value the macro evaluates to (for validation).
+ */
+#define BENCH_BPF_LOOP(body, reset) ({					\
+	__u64 __bench_start = bpf_ktime_get_ns();			\
+	__u32 __bench_i;						\
+	int __bench_result;						\
+									\
+	for (__bench_i = 0;						\
+	     __bench_i < batch_iters;					\
+	     __bench_i++) {						\
+		__bench_result = (body);				\
+		reset;							\
+	}								\
+									\
+	bench_record_sample(bpf_ktime_get_ns() - __bench_start);	\
+									\
+	__bench_result = (body);					\
+	__bench_result;							\
+})
+
+SEC("syscall")
+__success __retval(0)
+int bench_nop(void *ctx)
+{
+	return BENCH_BPF_LOOP(0, ({}));
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fifo.c b/tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fifo.c
new file mode 100644
index 000000000000..846385204a29
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fifo.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/* bpf_qdisc_fifo.c in plain C */
+
+#include <vmlinux.h>
+#include "bpf_experimental.h"
+#include "bpf_qdisc_common.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+struct skb_node {
+	struct sk_buff __kptr * skb;
+	struct bpf_list_node node;
+};
+
+private(A) struct bpf_spin_lock q_fifo_lock;
+private(A) struct bpf_list_head q_fifo __contains(skb_node, node);
+
+bool init_called;
+
+SEC("struct_ops/bpf_fifo_enqueue")
+__auxiliary
+int BPF_PROG(bpf_fifo_enqueue, struct sk_buff *skb, struct Qdisc *sch,
+	     struct bpf_sk_buff_ptr *to_free)
+{
+	struct skb_node *skbn;
+	u32 pkt_len;
+
+	if (sch->q.qlen == sch->limit)
+		goto drop;
+
+	skbn = bpf_obj_new(typeof(*skbn));
+	if (!skbn)
+		goto drop;
+
+	pkt_len = qdisc_pkt_len(skb);
+
+	sch->q.qlen++;
+	skb = bpf_kptr_xchg(&skbn->skb, skb);
+	if (skb)
+		bpf_qdisc_skb_drop(skb, to_free);
+
+	bpf_spin_lock(&q_fifo_lock);
+	bpf_list_push_back(&q_fifo, &skbn->node);
+	bpf_spin_unlock(&q_fifo_lock);
+
+	sch->qstats.backlog += pkt_len;
+	return NET_XMIT_SUCCESS;
+drop:
+	bpf_qdisc_skb_drop(skb, to_free);
+	return NET_XMIT_DROP;
+}
+
+SEC("struct_ops/bpf_fifo_dequeue")
+__auxiliary
+struct sk_buff *BPF_PROG(bpf_fifo_dequeue, struct Qdisc *sch)
+{
+	struct bpf_list_node *node;
+	struct sk_buff *skb = NULL;
+	struct skb_node *skbn;
+
+	bpf_spin_lock(&q_fifo_lock);
+	node = bpf_list_pop_front(&q_fifo);
+	bpf_spin_unlock(&q_fifo_lock);
+	if (!node)
+		return NULL;
+
+	skbn = container_of(node, struct skb_node, node);
+	skb = bpf_kptr_xchg(&skbn->skb, skb);
+	bpf_obj_drop(skbn);
+	if (!skb)
+		return NULL;
+
+	sch->qstats.backlog -= qdisc_pkt_len(skb);
+	bpf_qdisc_bstats_update(sch, skb);
+	sch->q.qlen--;
+
+	return skb;
+}
+
+SEC("struct_ops/bpf_fifo_init")
+__auxiliary
+int BPF_PROG(bpf_fifo_init, struct Qdisc *sch, struct nlattr *opt,
+	     struct netlink_ext_ack *extack)
+{
+	sch->limit = 1000;
+	init_called = true;
+	return 0;
+}
+
+SEC("struct_ops/bpf_fifo_reset")
+__success
+void BPF_PROG(bpf_fifo_reset, struct Qdisc *sch)
+{
+	struct bpf_list_node *node;
+	struct skb_node *skbn;
+	int i;
+
+	for (i = 0; i < sch->q.qlen; i++) {
+		struct sk_buff *skb = NULL;
+
+		bpf_spin_lock(&q_fifo_lock);
+		node = bpf_list_pop_front(&q_fifo);
+		bpf_spin_unlock(&q_fifo_lock);
+
+		if (!node)
+			break;
+
+		skbn = container_of(node, struct skb_node, node);
+		skb = bpf_kptr_xchg(&skbn->skb, skb);
+		if (skb)
+			bpf_kfree_skb(skb);
+		bpf_obj_drop(skbn);
+	}
+	sch->q.qlen = 0;
+}
+
+SEC("struct_ops")
+__auxiliary
+void BPF_PROG(bpf_fifo_destroy, struct Qdisc *sch)
+{
+}
+
+SEC(".struct_ops")
+struct Qdisc_ops fifo = {
+	.enqueue   = (void *)bpf_fifo_enqueue,
+	.dequeue   = (void *)bpf_fifo_dequeue,
+	.init      = (void *)bpf_fifo_init,
+	.reset     = (void *)bpf_fifo_reset,
+	.destroy   = (void *)bpf_fifo_destroy,
+	.id        = "bpf_fifo",
+};
diff --git a/tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fq.c b/tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fq.c
new file mode 100644
index 000000000000..e4e40ceb07cc
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_bpf_qdisc_fq.c
@@ -0,0 +1,728 @@
+// SPDX-License-Identifier: GPL-2.0
+/* bpf_qdisc_fq.c in plain C */
+
+#include <vmlinux.h>
+#include <errno.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include "bpf_qdisc_common.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+#define NSEC_PER_USEC 1000L
+#define NSEC_PER_SEC 1000000000L
+
+#define NUM_QUEUE (1 << 20)
+
+struct fq_bpf_data {
+	u32 quantum;
+	u32 initial_quantum;
+	u32 flow_refill_delay;
+	u32 flow_plimit;
+	u64 horizon;
+	u32 orphan_mask;
+	u32 timer_slack;
+	u64 time_next_delayed_flow;
+	u64 unthrottle_latency_ns;
+	u8 horizon_drop;
+	u32 new_flow_cnt;
+	u32 old_flow_cnt;
+	u64 ktime_cache;
+};
+
+enum {
+	CLS_RET_PRIO	= 0,
+	CLS_RET_NONPRIO = 1,
+	CLS_RET_ERR	= 2,
+};
+
+struct skb_node {
+	u64 tstamp;
+	struct sk_buff __kptr * skb;
+	struct bpf_rb_node node;
+};
+
+struct fq_flow_node {
+	int credit;
+	u32 qlen;
+	u64 age;
+	u64 time_next_packet;
+	struct bpf_list_node list_node;
+	struct bpf_rb_node rb_node;
+	struct bpf_rb_root queue __contains(skb_node, node);
+	struct bpf_spin_lock lock;
+	struct bpf_refcount refcount;
+};
+
+struct dequeue_nonprio_ctx {
+	bool stop_iter;
+	u64 expire;
+	u64 now;
+};
+
+struct remove_flows_ctx {
+	bool gc_only;
+	u32 reset_cnt;
+	u32 reset_max;
+};
+
+struct unset_throttled_flows_ctx {
+	bool unset_all;
+	u64 now;
+};
+
+struct fq_stashed_flow {
+	struct fq_flow_node __kptr * flow;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__type(key, __u64);
+	__type(value, struct fq_stashed_flow);
+	__uint(max_entries, NUM_QUEUE);
+} fq_nonprio_flows SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__type(key, __u64);
+	__type(value, struct fq_stashed_flow);
+	__uint(max_entries, 1);
+} fq_prio_flows SEC(".maps");
+
+private(A) struct bpf_spin_lock fq_delayed_lock;
+private(A) struct bpf_rb_root fq_delayed __contains(fq_flow_node, rb_node);
+
+private(B) struct bpf_spin_lock fq_new_flows_lock;
+private(B) struct bpf_list_head fq_new_flows __contains(fq_flow_node, list_node);
+
+private(C) struct bpf_spin_lock fq_old_flows_lock;
+private(C) struct bpf_list_head fq_old_flows __contains(fq_flow_node, list_node);
+
+private(D) struct fq_bpf_data q;
+
+/* Wrapper for bpf_kptr_xchg that expects NULL dst */
+static void bpf_kptr_xchg_back(void *map_val, void *ptr)
+{
+	void *ret;
+
+	ret = bpf_kptr_xchg(map_val, ptr);
+	if (ret)
+		bpf_obj_drop(ret);
+}
+
+static bool skbn_tstamp_less(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct skb_node *skbn_a;
+	struct skb_node *skbn_b;
+
+	skbn_a = container_of(a, struct skb_node, node);
+	skbn_b = container_of(b, struct skb_node, node);
+
+	return skbn_a->tstamp < skbn_b->tstamp;
+}
+
+static bool fn_time_next_packet_less(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct fq_flow_node *flow_a;
+	struct fq_flow_node *flow_b;
+
+	flow_a = container_of(a, struct fq_flow_node, rb_node);
+	flow_b = container_of(b, struct fq_flow_node, rb_node);
+
+	return flow_a->time_next_packet < flow_b->time_next_packet;
+}
+
+static void
+fq_flows_add_head(struct bpf_list_head *head, struct bpf_spin_lock *lock,
+		  struct fq_flow_node *flow, u32 *flow_cnt)
+{
+	bpf_spin_lock(lock);
+	bpf_list_push_front(head, &flow->list_node);
+	bpf_spin_unlock(lock);
+	*flow_cnt += 1;
+}
+
+static void
+fq_flows_add_tail(struct bpf_list_head *head, struct bpf_spin_lock *lock,
+		  struct fq_flow_node *flow, u32 *flow_cnt)
+{
+	bpf_spin_lock(lock);
+	bpf_list_push_back(head, &flow->list_node);
+	bpf_spin_unlock(lock);
+	*flow_cnt += 1;
+}
+
+static void
+fq_flows_remove_front(struct bpf_list_head *head, struct bpf_spin_lock *lock,
+		      struct bpf_list_node **node, u32 *flow_cnt)
+{
+	bpf_spin_lock(lock);
+	*node = bpf_list_pop_front(head);
+	bpf_spin_unlock(lock);
+	*flow_cnt -= 1;
+}
+
+static bool
+fq_flows_is_empty(struct bpf_list_head *head, struct bpf_spin_lock *lock)
+{
+	bool empty;
+
+	bpf_spin_lock(lock);
+	empty = bpf_list_empty(head);
+	bpf_spin_unlock(lock);
+
+	return empty;
+}
+
+/*
+ * flow->age is used to denote the state of the flow (not-detached, detached, throttled)
+ * as well as the timestamp when the flow is detached.
+ *
+ * 0: not-detached
+ * 1 - (~0ULL-1): detached
+ * ~0ULL: throttled
+ */
+static void fq_flow_set_detached(struct fq_flow_node *flow)
+{
+	flow->age = bpf_jiffies64();
+}
+
+static bool fq_flow_is_detached(struct fq_flow_node *flow)
+{
+	return flow->age != 0 && flow->age != ~0ULL;
+}
+
+static bool sk_listener(struct sock *sk)
+{
+	return (1 << sk->__sk_common.skc_state) & (TCPF_LISTEN | TCPF_NEW_SYN_RECV);
+}
+
+static void fq_gc(void);
+
+static int fq_new_flow(void *flow_map, struct fq_stashed_flow **sflow, u64 hash)
+{
+	struct fq_stashed_flow tmp = {};
+	struct fq_flow_node *flow;
+	int ret;
+
+	flow = bpf_obj_new(typeof(*flow));
+	if (!flow)
+		return -ENOMEM;
+
+	flow->credit = q.initial_quantum,
+	flow->qlen = 0,
+	flow->age = 1,
+	flow->time_next_packet = 0,
+
+	ret = bpf_map_update_elem(flow_map, &hash, &tmp, 0);
+	if (ret == -ENOMEM || ret == -E2BIG) {
+		fq_gc();
+		bpf_map_update_elem(&fq_nonprio_flows, &hash, &tmp, 0);
+	}
+
+	*sflow = bpf_map_lookup_elem(flow_map, &hash);
+	if (!*sflow) {
+		bpf_obj_drop(flow);
+		return -ENOMEM;
+	}
+
+	bpf_kptr_xchg_back(&(*sflow)->flow, flow);
+	return 0;
+}
+
+static int
+fq_classify(struct sk_buff *skb, struct fq_stashed_flow **sflow)
+{
+	struct sock *sk = skb->sk;
+	int ret = CLS_RET_NONPRIO;
+	u64 hash = 0;
+
+	if ((skb->priority & TC_PRIO_MAX) == TC_PRIO_CONTROL) {
+		*sflow = bpf_map_lookup_elem(&fq_prio_flows, &hash);
+		ret = CLS_RET_PRIO;
+	} else {
+		if (!sk || sk_listener(sk)) {
+			hash = bpf_skb_get_hash(skb) & q.orphan_mask;
+			/*
+			 * Avoid collision with an existing flow hash, which
+			 * only uses the lower 32 bits of hash, by setting the
+			 * upper half of hash to 1.
+			 */
+			hash |= (1ULL << 32);
+		} else if (sk->__sk_common.skc_state == TCP_CLOSE) {
+			hash = bpf_skb_get_hash(skb) & q.orphan_mask;
+			hash |= (1ULL << 32);
+		} else {
+			hash = sk->__sk_common.skc_hash;
+		}
+		*sflow = bpf_map_lookup_elem(&fq_nonprio_flows, &hash);
+	}
+
+	if (!*sflow)
+		ret = fq_new_flow(&fq_nonprio_flows, sflow, hash) < 0 ?
+		      CLS_RET_ERR : CLS_RET_NONPRIO;
+
+	return ret;
+}
+
+static bool fq_packet_beyond_horizon(struct sk_buff *skb)
+{
+	return (s64)skb->tstamp > (s64)(q.ktime_cache + q.horizon);
+}
+
+SEC("struct_ops/bpf_fq_enqueue")
+__auxiliary
+int BPF_PROG(bpf_fq_enqueue, struct sk_buff *skb, struct Qdisc *sch,
+	     struct bpf_sk_buff_ptr *to_free)
+{
+	struct fq_flow_node *flow = NULL, *flow_copy;
+	struct fq_stashed_flow *sflow;
+	u64 time_to_send, jiffies;
+	struct skb_node *skbn;
+	int ret;
+
+	if (sch->q.qlen >= sch->limit)
+		goto drop;
+
+	if (!skb->tstamp) {
+		time_to_send = q.ktime_cache = bpf_ktime_get_ns();
+	} else {
+		if (fq_packet_beyond_horizon(skb)) {
+			q.ktime_cache = bpf_ktime_get_ns();
+			if (fq_packet_beyond_horizon(skb)) {
+				if (q.horizon_drop)
+					goto drop;
+
+				skb->tstamp = q.ktime_cache + q.horizon;
+			}
+		}
+		time_to_send = skb->tstamp;
+	}
+
+	ret = fq_classify(skb, &sflow);
+	if (ret == CLS_RET_ERR)
+		goto drop;
+
+	flow = bpf_kptr_xchg(&sflow->flow, flow);
+	if (!flow)
+		goto drop;
+
+	if (ret == CLS_RET_NONPRIO) {
+		if (flow->qlen >= q.flow_plimit) {
+			bpf_kptr_xchg_back(&sflow->flow, flow);
+			goto drop;
+		}
+
+		if (fq_flow_is_detached(flow)) {
+			flow_copy = bpf_refcount_acquire(flow);
+
+			jiffies = bpf_jiffies64();
+			if ((s64)(jiffies - (flow_copy->age + q.flow_refill_delay)) > 0) {
+				if (flow_copy->credit < q.quantum)
+					flow_copy->credit = q.quantum;
+			}
+			flow_copy->age = 0;
+			fq_flows_add_tail(&fq_new_flows, &fq_new_flows_lock, flow_copy,
+					  &q.new_flow_cnt);
+		}
+	}
+
+	skbn = bpf_obj_new(typeof(*skbn));
+	if (!skbn) {
+		bpf_kptr_xchg_back(&sflow->flow, flow);
+		goto drop;
+	}
+
+	skbn->tstamp = skb->tstamp = time_to_send;
+
+	sch->qstats.backlog += qdisc_pkt_len(skb);
+
+	skb = bpf_kptr_xchg(&skbn->skb, skb);
+	if (skb)
+		bpf_qdisc_skb_drop(skb, to_free);
+
+	bpf_spin_lock(&flow->lock);
+	bpf_rbtree_add(&flow->queue, &skbn->node, skbn_tstamp_less);
+	bpf_spin_unlock(&flow->lock);
+
+	flow->qlen++;
+	bpf_kptr_xchg_back(&sflow->flow, flow);
+
+	sch->q.qlen++;
+	return NET_XMIT_SUCCESS;
+
+drop:
+	bpf_qdisc_skb_drop(skb, to_free);
+	sch->qstats.drops++;
+	return NET_XMIT_DROP;
+}
+
+static int fq_unset_throttled_flows(u32 index, struct unset_throttled_flows_ctx *ctx)
+{
+	struct bpf_rb_node *node = NULL;
+	struct fq_flow_node *flow;
+
+	bpf_spin_lock(&fq_delayed_lock);
+
+	node = bpf_rbtree_first(&fq_delayed);
+	if (!node) {
+		bpf_spin_unlock(&fq_delayed_lock);
+		return 1;
+	}
+
+	flow = container_of(node, struct fq_flow_node, rb_node);
+	if (!ctx->unset_all && flow->time_next_packet > ctx->now) {
+		q.time_next_delayed_flow = flow->time_next_packet;
+		bpf_spin_unlock(&fq_delayed_lock);
+		return 1;
+	}
+
+	node = bpf_rbtree_remove(&fq_delayed, &flow->rb_node);
+
+	bpf_spin_unlock(&fq_delayed_lock);
+
+	if (!node)
+		return 1;
+
+	flow = container_of(node, struct fq_flow_node, rb_node);
+	flow->age = 0;
+	fq_flows_add_tail(&fq_old_flows, &fq_old_flows_lock, flow, &q.old_flow_cnt);
+
+	return 0;
+}
+
+static void fq_flow_set_throttled(struct fq_flow_node *flow)
+{
+	flow->age = ~0ULL;
+
+	if (q.time_next_delayed_flow > flow->time_next_packet)
+		q.time_next_delayed_flow = flow->time_next_packet;
+
+	bpf_spin_lock(&fq_delayed_lock);
+	bpf_rbtree_add(&fq_delayed, &flow->rb_node, fn_time_next_packet_less);
+	bpf_spin_unlock(&fq_delayed_lock);
+}
+
+static void fq_check_throttled(u64 now)
+{
+	struct unset_throttled_flows_ctx ctx = {
+		.unset_all = false,
+		.now = now,
+	};
+	unsigned long sample;
+
+	if (q.time_next_delayed_flow > now)
+		return;
+
+	sample = (unsigned long)(now - q.time_next_delayed_flow);
+	q.unthrottle_latency_ns -= q.unthrottle_latency_ns >> 3;
+	q.unthrottle_latency_ns += sample >> 3;
+
+	q.time_next_delayed_flow = ~0ULL;
+	bpf_loop(NUM_QUEUE, fq_unset_throttled_flows, &ctx, 0);
+}
+
+static struct sk_buff*
+fq_dequeue_nonprio_flows(u32 index, struct dequeue_nonprio_ctx *ctx)
+{
+	u64 time_next_packet, time_to_send;
+	struct bpf_rb_node *rb_node;
+	struct sk_buff *skb = NULL;
+	struct bpf_list_head *head;
+	struct bpf_list_node *node;
+	struct bpf_spin_lock *lock;
+	struct fq_flow_node *flow;
+	struct skb_node *skbn;
+	bool is_empty;
+	u32 *cnt;
+
+	if (q.new_flow_cnt) {
+		head = &fq_new_flows;
+		lock = &fq_new_flows_lock;
+		cnt = &q.new_flow_cnt;
+	} else if (q.old_flow_cnt) {
+		head = &fq_old_flows;
+		lock = &fq_old_flows_lock;
+		cnt = &q.old_flow_cnt;
+	} else {
+		if (q.time_next_delayed_flow != ~0ULL)
+			ctx->expire = q.time_next_delayed_flow;
+		goto break_loop;
+	}
+
+	fq_flows_remove_front(head, lock, &node, cnt);
+	if (!node)
+		goto break_loop;
+
+	flow = container_of(node, struct fq_flow_node, list_node);
+	if (flow->credit <= 0) {
+		flow->credit += q.quantum;
+		fq_flows_add_tail(&fq_old_flows, &fq_old_flows_lock, flow, &q.old_flow_cnt);
+		return NULL;
+	}
+
+	bpf_spin_lock(&flow->lock);
+	rb_node = bpf_rbtree_first(&flow->queue);
+	if (!rb_node) {
+		bpf_spin_unlock(&flow->lock);
+		is_empty = fq_flows_is_empty(&fq_old_flows, &fq_old_flows_lock);
+		if (head == &fq_new_flows && !is_empty) {
+			fq_flows_add_tail(&fq_old_flows, &fq_old_flows_lock, flow, &q.old_flow_cnt);
+		} else {
+			fq_flow_set_detached(flow);
+			bpf_obj_drop(flow);
+		}
+		return NULL;
+	}
+
+	skbn = container_of(rb_node, struct skb_node, node);
+	time_to_send = skbn->tstamp;
+
+	time_next_packet = (time_to_send > flow->time_next_packet) ?
+		time_to_send : flow->time_next_packet;
+	if (ctx->now < time_next_packet) {
+		bpf_spin_unlock(&flow->lock);
+		flow->time_next_packet = time_next_packet;
+		fq_flow_set_throttled(flow);
+		return NULL;
+	}
+
+	rb_node = bpf_rbtree_remove(&flow->queue, rb_node);
+	bpf_spin_unlock(&flow->lock);
+
+	if (!rb_node)
+		goto add_flow_and_break;
+
+	skbn = container_of(rb_node, struct skb_node, node);
+	skb = bpf_kptr_xchg(&skbn->skb, skb);
+	bpf_obj_drop(skbn);
+
+	if (!skb)
+		goto add_flow_and_break;
+
+	flow->credit -= qdisc_skb_cb(skb)->pkt_len;
+	flow->qlen--;
+
+add_flow_and_break:
+	fq_flows_add_head(head, lock, flow, cnt);
+
+break_loop:
+	ctx->stop_iter = true;
+	return skb;
+}
+
+static struct sk_buff *fq_dequeue_prio(void)
+{
+	struct fq_flow_node *flow = NULL;
+	struct fq_stashed_flow *sflow;
+	struct bpf_rb_node *rb_node;
+	struct sk_buff *skb = NULL;
+	struct skb_node *skbn;
+	u64 hash = 0;
+
+	sflow = bpf_map_lookup_elem(&fq_prio_flows, &hash);
+	if (!sflow)
+		return NULL;
+
+	flow = bpf_kptr_xchg(&sflow->flow, flow);
+	if (!flow)
+		return NULL;
+
+	bpf_spin_lock(&flow->lock);
+	rb_node = bpf_rbtree_first(&flow->queue);
+	if (!rb_node) {
+		bpf_spin_unlock(&flow->lock);
+		goto out;
+	}
+
+	skbn = container_of(rb_node, struct skb_node, node);
+	rb_node = bpf_rbtree_remove(&flow->queue, &skbn->node);
+	bpf_spin_unlock(&flow->lock);
+
+	if (!rb_node)
+		goto out;
+
+	skbn = container_of(rb_node, struct skb_node, node);
+	skb = bpf_kptr_xchg(&skbn->skb, skb);
+	bpf_obj_drop(skbn);
+
+out:
+	bpf_kptr_xchg_back(&sflow->flow, flow);
+
+	return skb;
+}
+
+SEC("struct_ops/bpf_fq_dequeue")
+__success
+struct sk_buff *BPF_PROG(bpf_fq_dequeue, struct Qdisc *sch)
+{
+	struct dequeue_nonprio_ctx cb_ctx = {};
+	struct sk_buff *skb = NULL;
+	int i;
+
+	if (!sch->q.qlen)
+		goto out;
+
+	skb = fq_dequeue_prio();
+	if (skb)
+		goto dequeue;
+
+	q.ktime_cache = cb_ctx.now = bpf_ktime_get_ns();
+	fq_check_throttled(q.ktime_cache);
+	for (i = 0; i < sch->limit; i++) {
+		skb = fq_dequeue_nonprio_flows(i, &cb_ctx);
+		if (cb_ctx.stop_iter)
+			break;
+	};
+
+	if (skb) {
+dequeue:
+		sch->q.qlen--;
+		sch->qstats.backlog -= qdisc_pkt_len(skb);
+		bpf_qdisc_bstats_update(sch, skb);
+		return skb;
+	}
+
+	if (cb_ctx.expire)
+		bpf_qdisc_watchdog_schedule(sch, cb_ctx.expire, q.timer_slack);
+out:
+	return NULL;
+}
+
+static int fq_remove_flows_in_list(u32 index, void *ctx)
+{
+	struct bpf_list_node *node;
+	struct fq_flow_node *flow;
+
+	bpf_spin_lock(&fq_new_flows_lock);
+	node = bpf_list_pop_front(&fq_new_flows);
+	bpf_spin_unlock(&fq_new_flows_lock);
+	if (!node) {
+		bpf_spin_lock(&fq_old_flows_lock);
+		node = bpf_list_pop_front(&fq_old_flows);
+		bpf_spin_unlock(&fq_old_flows_lock);
+		if (!node)
+			return 1;
+	}
+
+	flow = container_of(node, struct fq_flow_node, list_node);
+	bpf_obj_drop(flow);
+
+	return 0;
+}
+
+extern unsigned CONFIG_HZ __kconfig;
+
+/* limit number of collected flows per round */
+#define FQ_GC_MAX 8
+#define FQ_GC_AGE (3*CONFIG_HZ)
+
+static bool fq_gc_candidate(struct fq_flow_node *flow)
+{
+	u64 jiffies = bpf_jiffies64();
+
+	return fq_flow_is_detached(flow) &&
+	       ((s64)(jiffies - (flow->age + FQ_GC_AGE)) > 0);
+}
+
+static int
+fq_remove_flows(struct bpf_map *flow_map, u64 *hash,
+		struct fq_stashed_flow *sflow, struct remove_flows_ctx *ctx)
+{
+	if (sflow->flow &&
+	    (!ctx->gc_only || fq_gc_candidate(sflow->flow))) {
+		bpf_map_delete_elem(flow_map, hash);
+		ctx->reset_cnt++;
+	}
+
+	return ctx->reset_cnt < ctx->reset_max ? 0 : 1;
+}
+
+static void fq_gc(void)
+{
+	struct remove_flows_ctx cb_ctx = {
+		.gc_only = true,
+		.reset_cnt = 0,
+		.reset_max = FQ_GC_MAX,
+	};
+
+	bpf_for_each_map_elem(&fq_nonprio_flows, fq_remove_flows, &cb_ctx, 0);
+}
+
+SEC("struct_ops/bpf_fq_reset")
+__auxiliary
+void BPF_PROG(bpf_fq_reset, struct Qdisc *sch)
+{
+	struct unset_throttled_flows_ctx utf_ctx = {
+		.unset_all = true,
+	};
+	struct remove_flows_ctx rf_ctx = {
+		.gc_only = false,
+		.reset_cnt = 0,
+		.reset_max = NUM_QUEUE,
+	};
+	struct fq_stashed_flow *sflow;
+	u64 hash = 0;
+
+	sch->q.qlen = 0;
+	sch->qstats.backlog = 0;
+
+	bpf_for_each_map_elem(&fq_nonprio_flows, fq_remove_flows, &rf_ctx, 0);
+
+	rf_ctx.reset_cnt = 0;
+	bpf_for_each_map_elem(&fq_prio_flows, fq_remove_flows, &rf_ctx, 0);
+	fq_new_flow(&fq_prio_flows, &sflow, hash);
+
+	bpf_loop(NUM_QUEUE, fq_remove_flows_in_list, NULL, 0);
+	q.new_flow_cnt = 0;
+	q.old_flow_cnt = 0;
+
+	bpf_loop(NUM_QUEUE, fq_unset_throttled_flows, &utf_ctx, 0);
+}
+
+SEC("struct_ops/bpf_fq_init")
+__auxiliary
+int BPF_PROG(bpf_fq_init, struct Qdisc *sch, struct nlattr *opt,
+	     struct netlink_ext_ack *extack)
+{
+	struct net_device *dev = sch->dev_queue->dev;
+	u32 psched_mtu = dev->mtu + dev->hard_header_len;
+	struct fq_stashed_flow *sflow;
+	u64 hash = 0;
+
+	if (fq_new_flow(&fq_prio_flows, &sflow, hash) < 0)
+		return -ENOMEM;
+
+	sch->limit = 10000;
+	q.initial_quantum = 10 * psched_mtu;
+	q.quantum = 2 * psched_mtu;
+	q.flow_refill_delay = 40;
+	q.flow_plimit = 100;
+	q.horizon = 10ULL * NSEC_PER_SEC;
+	q.horizon_drop = 1;
+	q.orphan_mask = 1024 - 1;
+	q.timer_slack = 10 * NSEC_PER_USEC;
+	q.time_next_delayed_flow = ~0ULL;
+	q.unthrottle_latency_ns = 0ULL;
+	q.new_flow_cnt = 0;
+	q.old_flow_cnt = 0;
+
+	return 0;
+}
+
+SEC("struct_ops")
+__auxiliary
+void BPF_PROG(bpf_fq_destroy, struct Qdisc *sch)
+{
+}
+
+SEC(".struct_ops")
+struct Qdisc_ops fq = {
+	.enqueue   = (void *)bpf_fq_enqueue,
+	.dequeue   = (void *)bpf_fq_dequeue,
+	.reset     = (void *)bpf_fq_reset,
+	.init      = (void *)bpf_fq_init,
+	.destroy   = (void *)bpf_fq_destroy,
+	.id        = "bpf_fq",
+};
diff --git a/tools/testing/selftests/bpf/progs/plain_c_cpumask_success.c b/tools/testing/selftests/bpf/progs/plain_c_cpumask_success.c
new file mode 100644
index 000000000000..fb1032386d99
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_cpumask_success.c
@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
+/* cpumask_success.c in plain C */
+
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+
+#include "bpf_misc.h"
+#include "cpumask_common.h"
+
+char _license[] SEC("license") = "GPL";
+
+int pid, nr_cpus;
+
+static bool is_test_task(void)
+{
+	int cur_pid = bpf_get_current_pid_tgid() >> 32;
+
+	return pid == cur_pid;
+}
+
+/* Mask is guaranteed to be large enough for bpf_cpumask_t. */
+#define CPUMASK_TEST_MASKLEN (sizeof(cpumask_t))
+
+/* Add an extra word for the test_populate_reject_unaligned test. */
+u64 bits[CPUMASK_TEST_MASKLEN / 8 + 1];
+
+SEC("tp_btf/task_newtask")
+__success
+int BPF_PROG(test_populate, struct task_struct *task, u64 clone_flags)
+{
+	struct bpf_cpumask *mask;
+	bool bit;
+	int ret;
+	int i;
+
+	if (!is_test_task())
+		return 0;
+
+	/* Set only odd bits. */
+	__builtin_memset(bits, 0xaa, CPUMASK_TEST_MASKLEN);
+
+	mask = bpf_cpumask_create();
+	if (!mask) {
+		err = 1;
+		return 0;
+	}
+
+	/* Pass the entire bits array, the kfunc will only copy the valid bits. */
+	ret = bpf_cpumask_populate(mask, bits, CPUMASK_TEST_MASKLEN);
+	if (ret) {
+		err = 2;
+		goto out;
+	}
+
+	/*
+	 * Test is there to appease the verifier. We cannot directly
+	 * access NR_CPUS, the upper bound for nr_cpus, so we infer
+	 * it from the size of cpumask_t.
+	 */
+	if (nr_cpus < 0 || nr_cpus > CPUMASK_TEST_MASKLEN * 8) {
+		err = 3;
+		goto out;
+	}
+
+	for (i = 0; i < nr_cpus; i++) {
+		/* Odd-numbered bits should be set, even ones unset. */
+		bit = bpf_cpumask_test_cpu(i, (const struct cpumask *)mask);
+		if (bit == (i % 2 != 0))
+			continue;
+
+		err = 4;
+		break;
+	}
+
+out:
+	bpf_cpumask_release(mask);
+
+	return 0;
+}
+
+#undef CPUMASK_TEST_MASKLEN
diff --git a/tools/testing/selftests/bpf/progs/plain_c_dynptr_success.c b/tools/testing/selftests/bpf/progs/plain_c_dynptr_success.c
new file mode 100644
index 000000000000..2c1c4f7c38a1
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_dynptr_success.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2022 Facebook */
+/* dynptr_success.c in plain C */
+
+#include <vmlinux.h>
+#include <string.h>
+#include <stdbool.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+#include "errno.h"
+
+#define PAGE_SIZE_64K 65536
+
+char _license[] SEC("license") = "GPL";
+
+int err;
+
+static inline int bpf_memcmp(const char *a, const char *b, u32 size)
+{
+	int i;
+
+	for (i = 0; i < size; i++) {
+		if (a[i] != b[i])
+			return a[i] < b[i] ? -1 : 1;
+	}
+	return 0;
+}
+
+static int fill_xdp(struct bpf_dynptr *ptr, __u32 start, __u32 n)
+{
+	__u8 b;
+	int i;
+
+	for (i = 0; i < n; i++) {
+		b = (start + i) % 251;
+		if (bpf_dynptr_write(ptr, start + i, &b, 1, 0))
+			return 1;
+	}
+	return 0;
+}
+
+static int check_xdp(struct bpf_dynptr *ptr, __u32 at, __u32 n, __u32 from)
+{
+	__u8 b;
+	int i;
+
+	for (i = 0; i < n; i++) {
+		if (bpf_dynptr_read(&b, 1, ptr, at + i, 0))
+			return 1;
+		if (b != (__u8)((from + i) % 251))
+			return 1;
+	}
+	return 0;
+}
+
+SEC("?xdp")
+__success
+int test_dynptr_copy_xdp_overlap(struct xdp_md *xdp)
+{
+	void *data_end = (void *)(long)xdp->data_end;
+	void *data = (void *)(long)xdp->data;
+	__u32 off = data_end - data - 1000;
+	struct bpf_dynptr ptr;
+
+	bpf_dynptr_from_xdp(xdp, 0, &ptr);
+	if (fill_xdp(&ptr, off, 1300)) {
+		err = 1;
+		return XDP_DROP;
+	}
+	err = bpf_dynptr_copy(&ptr, off + 100, &ptr, off, 1200);
+	if (err)
+		return XDP_DROP;
+	if (check_xdp(&ptr, off + 100, 1200, off))
+		err = 2;
+	return XDP_DROP;
+}
+
+SEC("?xdp")
+__success
+int test_dynptr_copy_xdp_overlap_src_linear(struct xdp_md *xdp)
+{
+	void *data_end = (void *)(long)xdp->data_end;
+	void *data = (void *)(long)xdp->data;
+	__u32 lin = data_end - data;
+	struct bpf_dynptr ptr;
+
+	bpf_dynptr_from_xdp(xdp, 0, &ptr);
+	if (fill_xdp(&ptr, lin - 64, 118)) {
+		err = 1;
+		return XDP_DROP;
+	}
+	err = bpf_dynptr_copy(&ptr, lin - 10, &ptr, lin - 64, 64);
+	if (err)
+		return XDP_DROP;
+	if (check_xdp(&ptr, lin - 10, 64, lin - 64))
+		err = 2;
+	return XDP_DROP;
+}
+
+SEC("?xdp")
+__success
+int test_dynptr_copy_xdp_overlap_dst_frag(struct xdp_md *xdp)
+{
+	void *data_end = (void *)(long)xdp->data_end;
+	void *data = (void *)(long)xdp->data;
+	__u32 lin = data_end - data;
+	struct bpf_dynptr ptr;
+
+	bpf_dynptr_from_xdp(xdp, 0, &ptr);
+	if (fill_xdp(&ptr, lin - 10, 84)) {
+		err = 1;
+		return XDP_DROP;
+	}
+	err = bpf_dynptr_copy(&ptr, lin + 10, &ptr, lin - 10, 64);
+	if (err)
+		return XDP_DROP;
+	if (check_xdp(&ptr, lin + 10, 64, lin - 10))
+		err = 2;
+	return XDP_DROP;
+}
+
+#define DYNPTR_MEMSET_VAL 42
+
+#define min_t(type, x, y) ({		\
+	type __x = (x);			\
+	type __y = (y);			\
+	__x < __y ? __x : __y; })
+
+SEC("xdp")
+__success
+int test_dynptr_memset_xdp_chunks(struct xdp_md *xdp)
+{
+	u32 data_sz, chunk_sz, offset = 0;
+	const int max_chunks = 200;
+	struct bpf_dynptr ptr_xdp;
+	char expected_buf[32];
+	char buf[32];
+	int i;
+
+	__builtin_memset(expected_buf, DYNPTR_MEMSET_VAL, sizeof(expected_buf));
+
+	/* ptr_xdp is backed by non-contiguous memory */
+	bpf_dynptr_from_xdp(xdp, 0, &ptr_xdp);
+	data_sz = bpf_dynptr_size(&ptr_xdp);
+
+	err = bpf_dynptr_memset(&ptr_xdp, 0, data_sz, DYNPTR_MEMSET_VAL);
+	if (err) {
+		/*
+		 * bpf_dynptr_memset() eventually called bpf_xdp_pointer()
+		 * where if data_sz is greater than 0xffff, -EFAULT will be
+		 * returned. For 64K page size, data_sz is greater than
+		 * 64K, so error is expected and let us zero out error and
+		 * return success.
+		 */
+		if (data_sz >= PAGE_SIZE_64K)
+			err = 0;
+		goto out;
+	}
+
+	for (i = 0; i < max_chunks; i++) {
+		offset = i * sizeof(buf);
+		if (offset >= data_sz)
+			goto out;
+		chunk_sz = min_t(u32, sizeof(buf), data_sz - offset);
+		err = bpf_dynptr_read(&buf, chunk_sz, &ptr_xdp, offset, 0);
+		if (err)
+			goto out;
+		err = bpf_memcmp(buf, expected_buf, sizeof(buf));
+		if (err)
+			goto out;
+	}
+out:
+	return XDP_DROP;
+}
diff --git a/tools/testing/selftests/bpf/progs/plain_c_iters.c b/tools/testing/selftests/bpf/progs/plain_c_iters.c
new file mode 100644
index 000000000000..9c639d6423e8
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_iters.c
@@ -0,0 +1,147 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */
+/* iters.c in plain C */
+
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_compiler.h"
+
+int my_pid;
+
+#ifdef REAL_TEST
+#define MY_PID_GUARD() if (my_pid != (bpf_get_current_pid_tgid() >> 32)) return 0
+#else
+#define MY_PID_GUARD() ({ })
+#endif
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__type(key, int);
+	__type(value, int);
+	__uint(max_entries, 1000);
+} hash_map SEC(".maps");
+
+SEC("?raw_tp")
+__failure
+__msg("invalid mem access 'map_value_or_null'")
+int iter_err_too_permissive2(const void *ctx)
+{
+	int *map_val = NULL;
+	int key = 0;
+
+	MY_PID_GUARD();
+
+	map_val = bpf_map_lookup_elem(&hash_map, &key);
+	if (!map_val)
+		return 0;
+
+	for (int i = 0; i < 1000000; i++) {
+		map_val = bpf_map_lookup_elem(&hash_map, &key);
+	}
+
+	*map_val = 123;
+
+	return 0;
+}
+
+SEC("?raw_tp")
+__failure
+__msg("invalid mem access 'map_value_or_null'")
+int iter_err_too_permissive3(const void *ctx)
+{
+	int *map_val = NULL;
+	int key = 0;
+	bool found = false;
+
+	MY_PID_GUARD();
+
+	for (int i = 0; i < 1000000; i++) {
+		map_val = bpf_map_lookup_elem(&hash_map, &key);
+		found = true;
+	}
+
+	if (found)
+		*map_val = 123;
+
+	return 0;
+}
+
+__u32 upper, select_n, result;
+__u64 global;
+
+static __noinline bool nest_2(char *str)
+{
+	/*
+	 * some insns (including branch insns) to ensure stacksafe() is triggered
+	 * in nest_2(). This way, stacksafe() can compare frame associated with nest_1().
+	 */
+	if (str[0] == 't')
+		return true;
+	if (str[1] == 'e')
+		return true;
+	if (str[2] == 's')
+		return true;
+	if (str[3] == 't')
+		return true;
+	return false;
+}
+
+static __noinline bool nest_1(int n)
+{
+	/* case 0: allocate stack, case 1: no allocate stack */
+	switch (n) {
+	case 0: {
+		char comm[16];
+
+		if (bpf_get_current_comm(comm, 16))
+			return false;
+		return nest_2(comm);
+	}
+	case 1:
+		return nest_2((char *)&global);
+	default:
+		return false;
+	}
+}
+
+SEC("raw_tp")
+__success __retval(0)
+int iter_subprog_check_stacksafe(const void *ctx)
+{
+	long i;
+
+	for (i = 0; i < upper; i++) {
+		if (!nest_1(select_n)) {
+			result = 1;
+			return 0;
+		}
+	}
+
+	result = 2;
+	return 0;
+}
+
+SEC("raw_tp")
+__success
+int clean_live_states(const void *ctx)
+{
+	char buf[1];
+	int i, j, k, l, m, n, o;
+
+	for (i = 0; i < 10; i++)
+	for (j = 0; j < 10; j++)
+	for (k = 0; k < 10; k++)
+	for (l = 0; l < 10; l++)
+	for (m = 0; m < 10; m++)
+	for (n = 0; n < 10; n++)
+	for (o = 0; o < 10; o++) {
+		if (unlikely(bpf_get_prandom_u32()))
+			buf[0] = 42;
+		bpf_printk("%s", buf);
+	}
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_mem_rdonly_untrusted.c b/tools/testing/selftests/bpf/progs/plain_c_mem_rdonly_untrusted.c
new file mode 100644
index 000000000000..7e94b575e5eb
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_mem_rdonly_untrusted.c
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* mem_rdonly_untrusted.c in plain C */
+
+#include <vmlinux.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_misc.h"
+#include "bpf_kfuncs.h"
+
+SEC("socket")
+__success
+int offset_not_tracked(void *ctx)
+{
+	int *p, i, s;
+
+	p = bpf_rdonly_cast(0, 0);
+	s = 0;
+	for (i = 0; i < 1000 * 1000 * 1000; i++) {
+		p++;
+		s += *p;
+	}
+	return s;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_mptcp_subflow.c b/tools/testing/selftests/bpf/progs/plain_c_mptcp_subflow.c
new file mode 100644
index 000000000000..b1d0cdedc534
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_mptcp_subflow.c
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2020, Tessares SA. */
+/* Copyright (c) 2024, Kylin Software */
+/* mptcp_subflow.c and mptcp_bpf.h in plain C */
+
+#include "bpf_tracing_net.h"
+
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+
+/* list helpers from include/linux/list.h */
+static inline int list_is_head(const struct list_head *list,
+			       const struct list_head *head)
+{
+	return list == head;
+}
+
+#define list_entry(ptr, type, member)					\
+	container_of(ptr, type, member)
+
+#define list_first_entry(ptr, type, member)				\
+	list_entry((ptr)->next, type, member)
+
+#define list_next_entry(pos, member)					\
+	list_entry((pos)->member.next, typeof(*(pos)), member)
+
+#define list_entry_is_head(pos, head, member)				\
+	list_is_head(&pos->member, (head))
+
+#define list_for_each_entry(pos, head, member)				\
+	for (pos = list_first_entry(head, typeof(*pos), member);	\
+	     !list_entry_is_head(pos, head, member);			\
+	     pos = list_next_entry(pos, member))
+
+/* mptcp helpers from protocol.h */
+#define mptcp_for_each_subflow(__msk, __subflow)			\
+	list_for_each_entry(__subflow, &((__msk)->conn_list), node)
+
+static struct sock *
+mptcp_subflow_tcp_sock(const struct mptcp_subflow_context *subflow)
+{
+	return subflow->tcp_sock;
+}
+
+char _license[] SEC("license") = "GPL";
+
+char cc[TCP_CA_NAME_MAX] = "reno";
+int pid;
+
+static int _check_getsockopt_subflow_mark(struct mptcp_sock *msk, struct bpf_sockopt *ctx)
+{
+	struct mptcp_subflow_context *subflow;
+	int i = 0;
+
+	mptcp_for_each_subflow(msk, subflow) {
+		struct sock *ssk;
+
+		ssk = mptcp_subflow_tcp_sock(bpf_core_cast(subflow,
+							   struct mptcp_subflow_context));
+
+		if (ssk->sk_mark != ++i) {
+			ctx->retval = -2;
+			break;
+		}
+	}
+
+	return 1;
+}
+
+static int _check_getsockopt_subflow_cc(struct mptcp_sock *msk, struct bpf_sockopt *ctx)
+{
+	struct mptcp_subflow_context *subflow;
+
+	mptcp_for_each_subflow(msk, subflow) {
+		struct inet_connection_sock *icsk;
+		struct sock *ssk;
+
+		ssk = mptcp_subflow_tcp_sock(bpf_core_cast(subflow,
+							   struct mptcp_subflow_context));
+		icsk = bpf_core_cast(ssk, struct inet_connection_sock);
+
+		if (ssk->sk_mark == 2 &&
+		    __builtin_memcmp(icsk->icsk_ca_ops->name, cc, TCP_CA_NAME_MAX)) {
+			ctx->retval = -2;
+			break;
+		}
+	}
+
+	return 1;
+}
+
+SEC("cgroup/getsockopt")
+__success
+int _getsockopt_subflow(struct bpf_sockopt *ctx)
+{
+	struct bpf_sock *sk = ctx->sk;
+	struct mptcp_sock *msk;
+
+	if (bpf_get_current_pid_tgid() >> 32 != pid)
+		return 1;
+
+	if (!sk || sk->protocol != IPPROTO_MPTCP ||
+	    (!(ctx->level == SOL_SOCKET && ctx->optname == SO_MARK) &&
+	     !(ctx->level == SOL_TCP && ctx->optname == TCP_CONGESTION)))
+		return 1;
+
+	msk = bpf_core_cast(sk, struct mptcp_sock);
+	if (msk->pm.extra_subflows != 1) {
+		ctx->retval = -1;
+		return 1;
+	}
+
+	if (ctx->optname == SO_MARK)
+		return _check_getsockopt_subflow_mark(msk, ctx);
+	return _check_getsockopt_subflow_cc(msk, ctx);
+}
diff --git a/tools/testing/selftests/bpf/progs/plain_c_rbtree_search.c b/tools/testing/selftests/bpf/progs/plain_c_rbtree_search.c
new file mode 100644
index 000000000000..d3bfbde6f90e
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_rbtree_search.c
@@ -0,0 +1,136 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+/* rbtree_search.c in plain C */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+struct node_data {
+	struct bpf_refcount ref;
+	struct bpf_rb_node r0;
+	struct bpf_rb_node r1;
+	int key0;
+	int key1;
+};
+
+#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
+private(A) struct bpf_spin_lock glock0;
+private(A) struct bpf_rb_root groot0 __contains(node_data, r0);
+
+private(B) struct bpf_spin_lock glock1;
+private(B) struct bpf_rb_root groot1 __contains(node_data, r1);
+
+#define rb_entry(ptr, type, member) container_of(ptr, type, member)
+#define NR_NODES 16
+
+static bool less0(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct node_data *node_a;
+	struct node_data *node_b;
+
+	node_a = rb_entry(a, struct node_data, r0);
+	node_b = rb_entry(b, struct node_data, r0);
+
+	return node_a->key0 < node_b->key0;
+}
+
+static bool less1(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct node_data *node_a;
+	struct node_data *node_b;
+
+	node_a = rb_entry(a, struct node_data, r1);
+	node_b = rb_entry(b, struct node_data, r1);
+
+	return node_a->key1 < node_b->key1;
+}
+
+SEC("syscall")
+__success __retval(0)
+#ifndef __clang__
+__skip("GCC walks the array on the stack with a pointer")
+#endif
+long rbtree_search(void *ctx)
+{
+	struct bpf_rb_node *rb_n, *rb_m, *gc_ns[NR_NODES];
+	long lookup_key = NR_NODES / 2;
+	struct node_data *n, *m;
+	int i, nr_gc = 0;
+
+	for (i = 0; i < NR_NODES; i++) {
+		n = bpf_obj_new(typeof(*n));
+		if (!n)
+			return __LINE__;
+
+		m = bpf_refcount_acquire(n);
+
+		n->key0 = i;
+		m->key1 = i;
+
+		bpf_spin_lock(&glock0);
+		bpf_rbtree_add(&groot0, &n->r0, less0);
+		bpf_spin_unlock(&glock0);
+
+		bpf_spin_lock(&glock1);
+		bpf_rbtree_add(&groot1, &m->r1, less1);
+		bpf_spin_unlock(&glock1);
+	}
+
+	n = NULL;
+	bpf_spin_lock(&glock0);
+	rb_n = bpf_rbtree_root(&groot0);
+	while (1) {
+		if (!rb_n) {
+			bpf_spin_unlock(&glock0);
+			return __LINE__;
+		}
+
+		n = rb_entry(rb_n, struct node_data, r0);
+		if (lookup_key == n->key0)
+			break;
+		if (nr_gc < NR_NODES)
+			gc_ns[nr_gc++] = rb_n;
+		if (lookup_key < n->key0)
+			rb_n = bpf_rbtree_left(&groot0, rb_n);
+		else
+			rb_n = bpf_rbtree_right(&groot0, rb_n);
+	}
+
+	if (!n || lookup_key != n->key0) {
+		bpf_spin_unlock(&glock0);
+		return __LINE__;
+	}
+
+	for (i = 0; i < nr_gc; i++) {
+		rb_n = gc_ns[i];
+		gc_ns[i] = bpf_rbtree_remove(&groot0, rb_n);
+	}
+
+	m = bpf_refcount_acquire(n);
+	bpf_spin_unlock(&glock0);
+
+	for (i = 0; i < nr_gc; i++) {
+		rb_n = gc_ns[i];
+		if (rb_n) {
+			n = rb_entry(rb_n, struct node_data, r0);
+			bpf_obj_drop(n);
+		}
+	}
+
+	if (!m)
+		return __LINE__;
+
+	bpf_spin_lock(&glock1);
+	rb_m = bpf_rbtree_remove(&groot1, &m->r1);
+	bpf_spin_unlock(&glock1);
+	bpf_obj_drop(m);
+	if (!rb_m)
+		return __LINE__;
+	bpf_obj_drop(rb_entry(rb_m, struct node_data, r1));
+
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_rbtree_search_kptr.c b/tools/testing/selftests/bpf/progs/plain_c_rbtree_search_kptr.c
new file mode 100644
index 000000000000..5808d873fedd
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_rbtree_search_kptr.c
@@ -0,0 +1,264 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 KylinSoft Corporation. */
+/* rbtree_search_kptr.c in plain C */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+#define NR_NODES 16
+
+struct node_data {
+	int data;
+};
+
+struct tree_node {
+	struct bpf_rb_node node;
+	u64 key;
+	struct node_data __kptr * node_data;
+};
+
+struct tree_node_ref {
+	struct bpf_refcount ref;
+	struct bpf_rb_node node;
+	u64 key;
+	struct node_data __kptr * node_data;
+};
+
+#define private(name) SEC(".data." #name) __hidden __aligned(8)
+
+private(A) struct bpf_rb_root root __contains(tree_node, node);
+private(A) struct bpf_spin_lock lock;
+
+private(B) struct bpf_rb_root root_r __contains(tree_node_ref, node);
+private(B) struct bpf_spin_lock lock_r;
+
+static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct tree_node *node_a, *node_b;
+
+	node_a = container_of(a, struct tree_node, node);
+	node_b = container_of(b, struct tree_node, node);
+
+	return node_a->key < node_b->key;
+}
+
+SEC("syscall")
+__success __retval(0)
+#ifndef __clang__
+__skip("GCC's layout: the may_goto that the verifier adds leaves the loop with a reference")
+#endif
+long rbtree_search_kptr(void *ctx)
+{
+	struct tree_node *tnode;
+	struct bpf_rb_node *rb_n;
+	struct node_data __kptr * node_data;
+	int lookup_key  = NR_NODES / 2;
+	int lookup_data = NR_NODES / 2;
+	int i, data, ret = 0;
+
+	for (i = 0; i < NR_NODES; i++) {
+		tnode = bpf_obj_new(typeof(*tnode));
+		if (!tnode)
+			return __LINE__;
+
+		node_data = bpf_obj_new(typeof(*node_data));
+		if (!node_data) {
+			bpf_obj_drop(tnode);
+			return __LINE__;
+		}
+
+		tnode->key = i;
+		node_data->data = i;
+
+		node_data = bpf_kptr_xchg(&tnode->node_data, node_data);
+		if (node_data)
+			bpf_obj_drop(node_data);
+
+		bpf_spin_lock(&lock);
+		bpf_rbtree_add(&root, &tnode->node, less);
+		bpf_spin_unlock(&lock);
+	}
+
+	bpf_spin_lock(&lock);
+	rb_n = bpf_rbtree_root(&root);
+	while (rb_n) {
+		tnode = container_of(rb_n, struct tree_node, node);
+		node_data = bpf_kptr_xchg(&tnode->node_data, NULL);
+		if (!node_data) {
+			ret = __LINE__;
+			goto fail;
+		}
+
+		data = node_data->data;
+		node_data = bpf_kptr_xchg(&tnode->node_data, node_data);
+		if (node_data) {
+			bpf_spin_unlock(&lock);
+			bpf_obj_drop(node_data);
+			return __LINE__;
+		}
+
+		if (lookup_key == tnode->key) {
+			if (data == lookup_data)
+				break;
+
+			ret = __LINE__;
+			goto fail;
+		}
+
+		if (lookup_key < tnode->key)
+			rb_n = bpf_rbtree_left(&root, rb_n);
+		else
+			rb_n = bpf_rbtree_right(&root, rb_n);
+	}
+	bpf_spin_unlock(&lock);
+
+	while (1) {
+		bpf_spin_lock(&lock);
+		rb_n = bpf_rbtree_first(&root);
+		if (!rb_n) {
+			bpf_spin_unlock(&lock);
+			return 0;
+		}
+
+		rb_n = bpf_rbtree_remove(&root, rb_n);
+		if (!rb_n) {
+			ret = __LINE__;
+			goto fail;
+		}
+		bpf_spin_unlock(&lock);
+
+		tnode = container_of(rb_n, struct tree_node, node);
+
+		node_data = bpf_kptr_xchg(&tnode->node_data, NULL);
+		if (node_data)
+			bpf_obj_drop(node_data);
+
+		bpf_obj_drop(tnode);
+	}
+
+	return 0;
+fail:
+	bpf_spin_unlock(&lock);
+	return ret;
+}
+
+static bool less_r(struct bpf_rb_node *a, const struct bpf_rb_node *b)
+{
+	struct tree_node_ref *node_a, *node_b;
+
+	node_a = container_of(a, struct tree_node_ref, node);
+	node_b = container_of(b, struct tree_node_ref, node);
+
+	return node_a->key < node_b->key;
+}
+
+SEC("syscall")
+__success __retval(0)
+#ifndef __clang__
+__skip("GCC's layout: the may_goto that the verifier adds leaves the loop with a reference")
+#endif
+long rbtree_search_kptr_ref(void *ctx)
+{
+	struct tree_node_ref *tnode_r, *tnode_m;
+	struct bpf_rb_node *rb_n;
+	struct node_data __kptr * node_data;
+	int lookup_key  = NR_NODES / 2;
+	int lookup_data = NR_NODES / 2;
+	int i, data, ret = 0;
+
+	for (i = 0; i < NR_NODES; i++) {
+		tnode_r = bpf_obj_new(typeof(*tnode_r));
+		if (!tnode_r)
+			return __LINE__;
+
+		node_data = bpf_obj_new(typeof(*node_data));
+		if (!node_data) {
+			bpf_obj_drop(tnode_r);
+			return __LINE__;
+		}
+
+		tnode_r->key = i;
+		node_data->data = i;
+
+		node_data = bpf_kptr_xchg(&tnode_r->node_data, node_data);
+		if (node_data)
+			bpf_obj_drop(node_data);
+
+		/* Unused reference */
+		tnode_m = bpf_refcount_acquire(tnode_r);
+		if (!tnode_m)
+			return __LINE__;
+
+		bpf_spin_lock(&lock_r);
+		bpf_rbtree_add(&root_r, &tnode_r->node, less_r);
+		bpf_spin_unlock(&lock_r);
+
+		bpf_obj_drop(tnode_m);
+	}
+
+	bpf_spin_lock(&lock_r);
+	rb_n = bpf_rbtree_root(&root_r);
+	while (rb_n) {
+		tnode_r = container_of(rb_n, struct tree_node_ref, node);
+		node_data = bpf_kptr_xchg(&tnode_r->node_data, NULL);
+		if (!node_data) {
+			ret = __LINE__;
+			goto fail;
+		}
+
+		data = node_data->data;
+		node_data = bpf_kptr_xchg(&tnode_r->node_data, node_data);
+		if (node_data) {
+			bpf_spin_unlock(&lock_r);
+			bpf_obj_drop(node_data);
+			return __LINE__;
+		}
+
+		if (lookup_key == tnode_r->key) {
+			if (data == lookup_data)
+				break;
+
+			ret = __LINE__;
+			goto fail;
+		}
+
+		if (lookup_key < tnode_r->key)
+			rb_n = bpf_rbtree_left(&root_r, rb_n);
+		else
+			rb_n = bpf_rbtree_right(&root_r, rb_n);
+	}
+	bpf_spin_unlock(&lock_r);
+
+	while (1) {
+		bpf_spin_lock(&lock_r);
+		rb_n = bpf_rbtree_first(&root_r);
+		if (!rb_n) {
+			bpf_spin_unlock(&lock_r);
+			return 0;
+		}
+
+		rb_n = bpf_rbtree_remove(&root_r, rb_n);
+		if (!rb_n) {
+			ret = __LINE__;
+			goto fail;
+		}
+		bpf_spin_unlock(&lock_r);
+
+		tnode_r = container_of(rb_n, struct tree_node_ref, node);
+
+		node_data = bpf_kptr_xchg(&tnode_r->node_data, NULL);
+		if (node_data)
+			bpf_obj_drop(node_data);
+
+		bpf_obj_drop(tnode_r);
+	}
+
+	return 0;
+fail:
+	bpf_spin_unlock(&lock_r);
+	return ret;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_strobemeta_bpf_loop.c b/tools/testing/selftests/bpf/progs/plain_c_strobemeta_bpf_loop.c
new file mode 100644
index 000000000000..b4f53c42e098
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_strobemeta_bpf_loop.c
@@ -0,0 +1,583 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021 Facebook */
+// Copyright (c) 2019 Facebook
+/* strobemeta_bpf_loop.c and strobemeta.h in plain C */
+
+#define STROBE_MAX_INTS 2
+#define STROBE_MAX_STRS 25
+#define STROBE_MAX_MAPS 100
+#define STROBE_MAX_MAP_ENTRIES 20
+
+#include <stdint.h>
+#include <stddef.h>
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <linux/ptrace.h>
+#include <linux/sched.h>
+#include <linux/types.h>
+#include <bpf/bpf_helpers.h>
+
+#include "bpf_compiler.h"
+#include "bpf_misc.h"
+
+#ifdef __clang__
+
+typedef uint32_t pid_t;
+struct task_struct {};
+
+#define TASK_COMM_LEN 16
+#define PERF_MAX_STACK_DEPTH 127
+
+#define STACK_TABLE_EPOCH_SHIFT 20
+#define STROBE_MAX_STR_LEN 1
+#define STROBE_MAX_CFGS 32
+#define READ_MAP_VAR_PAYLOAD_CAP					\
+	((1 + STROBE_MAX_MAP_ENTRIES * 2) * STROBE_MAX_STR_LEN)
+#define STROBE_MAX_PAYLOAD						\
+	(STROBE_MAX_STRS * STROBE_MAX_STR_LEN +				\
+	 STROBE_MAX_MAPS * READ_MAP_VAR_PAYLOAD_CAP)
+
+struct strobe_value_header {
+	/*
+	 * meaning depends on type:
+	 * 1. int: 0, if value not set, 1 otherwise
+	 * 2. str: 1 always, whether value is set or not is determined by ptr
+	 * 3. map: 1 always, pointer points to additional struct with number
+	 *    of entries (up to STROBE_MAX_MAP_ENTRIES)
+	 */
+	uint16_t len;
+	/*
+	 * _reserved might be used for some future fields/flags, but we always
+	 * want to keep strobe_value_header to be 8 bytes, so BPF can read 16
+	 * bytes in one go and get both header and value
+	 */
+	uint8_t _reserved[6];
+};
+
+/*
+ * strobe_value_generic is used from BPF probe only, but needs to be a union
+ * of strobe_value_int/strobe_value_str/strobe_value_map
+ */
+struct strobe_value_generic {
+	struct strobe_value_header header;
+	union {
+		int64_t val;
+		void *ptr;
+	};
+};
+
+struct strobe_map_entry {
+	const char* key;
+	const char* val;
+};
+
+/*
+ * Map of C-string key/value pairs with fixed maximum capacity. Each map has
+ * corresponding int64 ID, which application can use (or ignore) in whatever
+ * way appropriate. Map is "write-only", there is no way to get data out of
+ * map. Map is intended to be used to provide metadata for profilers and is
+ * not to be used for internal in-app communication. All methods are
+ * thread-safe.
+ */
+struct strobe_map_raw {
+	/*
+	 * general purpose unique ID that's up to application to decide
+	 * whether and how to use; for request metadata use case id is unique
+	 * request ID that's used to match metadata with stack traces on
+	 * Strobelight backend side
+	 */
+	int64_t id;
+	/* number of used entries in map */
+	int64_t cnt;
+	/*
+	 * having volatile doesn't change anything on BPF side, but clang
+	 * emits warnings for passing `volatile const char *` into
+	 * bpf_probe_read_user_str that expects just `const char *`
+	 */
+	const char* tag;
+	/*
+	 * key/value entries, each consisting of 2 pointers to key and value
+	 * C strings
+	 */
+	struct strobe_map_entry entries[STROBE_MAX_MAP_ENTRIES];
+};
+
+#define TLS_LOCAL_EXEC 0
+
+/*
+ * structure that universally represents TLS location (both for static
+ * executables and shared libraries)
+ */
+struct strobe_value_loc {
+	/*
+	 * tls_mode defines what TLS mode was used for particular metavariable:
+	 * - -1 (TLS_NOT_SET) - no metavariable;
+	 * - 0 (TLS_LOCAL_EXEC) - Local Executable mode;
+	 * - 1 (TLS_IMM_EXEC) - Immediate Executable mode;
+	 * - 2 (TLS_GENERAL_DYN) - General Dynamic mode;
+	 * Local Dynamic mode is not yet supported, because never seen in
+	 * practice.  Mode defines how offset field is interpreted. See
+	 * calc_location() in below for details.
+	 */
+	int64_t tls_mode;
+	/*
+	 * TLS_LOCAL_EXEC: offset from thread pointer (fs:0 for x86-64,
+	 * tpidr_el0 for aarch64).
+	 * TLS_IMM_EXEC: absolute address of GOT entry containing offset
+	 * from thread pointer;
+	 * TLS_GENERAL_DYN: absolute address of double GOT entry
+	 * containing tls_index_t struct;
+	 */
+	int64_t offset;
+};
+
+struct strobemeta_cfg {
+	int64_t req_meta_idx;
+	struct strobe_value_loc int_locs[STROBE_MAX_INTS];
+	struct strobe_value_loc str_locs[STROBE_MAX_STRS];
+	struct strobe_value_loc map_locs[STROBE_MAX_MAPS];
+};
+
+struct strobe_map_descr {
+	uint64_t id;
+	int16_t tag_len;
+	/*
+	 * cnt <0 - map value isn't set;
+	 * 0 - map has id set, but no key/value entries
+	 */
+	int16_t cnt;
+	/*
+	 * both key_lens[i] and val_lens[i] should be >0 for present key/value
+	 * entry
+	 */
+	uint16_t key_lens[STROBE_MAX_MAP_ENTRIES];
+	uint16_t val_lens[STROBE_MAX_MAP_ENTRIES];
+};
+
+struct strobemeta_payload {
+	/* req_id has valid request ID, if req_meta_valid == 1 */
+	int64_t req_id;
+	uint8_t req_meta_valid;
+	/*
+	 * mask has Nth bit set to 1, if Nth metavar was present and
+	 * successfully read
+	 */
+	uint64_t int_vals_set_mask;
+	int64_t int_vals[STROBE_MAX_INTS];
+	/* len is >0 for present values */
+	uint16_t str_lens[STROBE_MAX_STRS];
+	/* if map_descrs[i].cnt == -1, metavar is not present/set */
+	struct strobe_map_descr map_descrs[STROBE_MAX_MAPS];
+	/*
+	 * payload has compactly packed values of str and map variables in the
+	 * form: strval1\0strval2\0map1key1\0map1val1\0map2key1\0map2val1\0
+	 * (and so on); str_lens[i], key_lens[i] and val_lens[i] determines
+	 * value length
+	 */
+	char payload[STROBE_MAX_PAYLOAD];
+};
+
+struct strobelight_bpf_sample {
+	uint64_t ktime;
+	char comm[TASK_COMM_LEN];
+	pid_t pid;
+	int user_stack_id;
+	int kernel_stack_id;
+	int has_meta;
+	struct strobemeta_payload metadata;
+	/*
+	 * makes it possible to pass (<real payload size> + 1) as data size to
+	 * perf_submit() to avoid perf_submit's paranoia about passing zero as
+	 * size, as it deduces that <real payload size> might be
+	 * **theoretically** zero
+	 */
+	char dummy_safeguard;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+	__uint(max_entries, 32);
+	__uint(key_size, sizeof(int));
+	__uint(value_size, sizeof(int));
+} samples SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
+	__uint(max_entries, 16);
+	__uint(key_size, sizeof(uint32_t));
+	__uint(value_size, sizeof(uint64_t) * PERF_MAX_STACK_DEPTH);
+} stacks_0 SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
+	__uint(max_entries, 16);
+	__uint(key_size, sizeof(uint32_t));
+	__uint(value_size, sizeof(uint64_t) * PERF_MAX_STACK_DEPTH);
+} stacks_1 SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, uint32_t);
+	__type(value, struct strobelight_bpf_sample);
+} sample_heap SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, STROBE_MAX_CFGS);
+	__type(key, pid_t);
+	__type(value, struct strobemeta_cfg);
+} strobemeta_cfgs SEC(".maps");
+
+/* Type for the dtv.  */
+/* https://github.com/lattera/glibc/blob/master/nptl/sysdeps/x86_64/tls.h#L34 */
+typedef union dtv {
+	size_t counter;
+	struct {
+		void* val;
+		bool is_static;
+	} pointer;
+} dtv_t;
+
+/* Partial definition for tcbhead_t */
+/* https://github.com/bminor/glibc/blob/master/sysdeps/x86_64/nptl/tls.h#L42 */
+struct tcbhead {
+	void* tcb;
+	dtv_t* dtv;
+};
+
+/*
+ * TLS module/offset information for shared library case.
+ * For x86-64, this is mapped onto two entries in GOT.
+ * For aarch64, this is pointed to by second GOT entry.
+ */
+struct tls_index {
+	uint64_t module;
+	uint64_t offset;
+};
+
+static void *calc_location(struct strobe_value_loc *loc, void *tls_base)
+{
+	/*
+	 * tls_mode value is:
+	 * - -1 (TLS_NOT_SET), if no metavar is present;
+	 * - 0 (TLS_LOCAL_EXEC), if metavar uses Local Executable mode of TLS
+	 * (offset from fs:0 for x86-64 or tpidr_el0 for aarch64);
+	 * - 1 (TLS_IMM_EXEC), if metavar uses Immediate Executable mode of TLS;
+	 * - 2 (TLS_GENERAL_DYN), if metavar uses General Dynamic mode of TLS;
+	 * This schema allows to use something like:
+	 * (tls_mode + 1) * (tls_base + offset)
+	 * to get NULL for "no metavar" location, or correct pointer for local
+	 * executable mode without doing extra ifs.
+	 */
+	if (loc->tls_mode <= TLS_LOCAL_EXEC) {
+		/*
+		 * static executable is simple, we just have offset from
+		 * tls_base
+		 */
+		void *addr = tls_base + loc->offset;
+		/*
+		 * multiply by (tls_mode + 1) to get NULL, if we have no
+		 * metavar in this slot
+		 */
+		return (void *)((loc->tls_mode + 1) * (int64_t)addr);
+	}
+	/*
+	 * Other modes are more complicated, we need to jump through few hoops.
+	 *
+	 * For immediate executable mode (currently supported only for aarch64):
+	 *  - loc->offset is pointing to a GOT entry containing fixed offset
+	 *  relative to tls_base;
+	 *
+	 * For general dynamic mode:
+	 *  - loc->offset is pointing to a beginning of double GOT entries;
+	 *  - (for aarch64 only) second entry points to tls_index_t struct;
+	 *  - (for x86-64 only) two GOT entries are already tls_index_t;
+	 *  - tls_index_t->module is used to find start of TLS section in
+	 *  which variable resides;
+	 *  - tls_index_t->offset provides offset within that TLS section,
+	 *  pointing to value of variable.
+	 */
+	struct tls_index tls_index;
+	dtv_t *dtv;
+	void *tls_ptr;
+
+	bpf_probe_read_user(&tls_index, sizeof(struct tls_index),
+			    (void *)loc->offset);
+	/* valid module index is always positive */
+	if (tls_index.module > 0) {
+		/* dtv = ((struct tcbhead *)tls_base)->dtv[tls_index.module] */
+		bpf_probe_read_user(&dtv, sizeof(dtv),
+				    &((struct tcbhead *)tls_base)->dtv);
+		dtv += tls_index.module;
+	} else {
+		dtv = NULL;
+	}
+	bpf_probe_read_user(&tls_ptr, sizeof(void *), dtv);
+	/* if pointer has (void *)-1 value, then TLS wasn't initialized yet */
+	if (!tls_ptr || tls_ptr == (void *)-1)
+		return NULL;
+	return tls_ptr + tls_index.offset;
+}
+
+static void read_int_var(struct strobemeta_cfg *cfg,
+			 size_t idx, void *tls_base,
+			 struct strobe_value_generic *value,
+			 struct strobemeta_payload *data)
+{
+	void *location = calc_location(&cfg->int_locs[idx], tls_base);
+	if (!location)
+		return;
+
+	bpf_probe_read_user(value, sizeof(struct strobe_value_generic), location);
+	data->int_vals[idx] = value->val;
+	if (value->header.len)
+		data->int_vals_set_mask |= (1 << idx);
+}
+
+static uint64_t read_str_var(struct strobemeta_cfg *cfg,
+			     size_t idx, void *tls_base,
+			     struct strobe_value_generic *value,
+			     struct strobemeta_payload *data,
+			     size_t off)
+{
+	void *location;
+	uint64_t len;
+
+	data->str_lens[idx] = 0;
+	location = calc_location(&cfg->str_locs[idx], tls_base);
+	if (bpf_probe_read_user(value, sizeof(struct strobe_value_generic), location))
+		return off;
+
+	len = bpf_probe_read_user_str(&data->payload[off], STROBE_MAX_STR_LEN, value->ptr);
+	/*
+	 * if bpf_probe_read_user_str returns error (<0), due to casting to
+	 * unsigned int, it will become big number, so next check is
+	 * sufficient to check for errors AND prove to BPF verifier, that
+	 * bpf_probe_read_user_str won't return anything bigger than
+	 * STROBE_MAX_STR_LEN
+	 */
+	if (len > STROBE_MAX_STR_LEN)
+		return off;
+
+	data->str_lens[idx] = len;
+	return off + len;
+}
+
+static uint64_t read_map_var(struct strobemeta_cfg *cfg,
+			     size_t idx, void *tls_base,
+			     struct strobe_value_generic *value,
+			     struct strobemeta_payload *data,
+			     size_t off)
+{
+	struct strobe_map_descr* descr = &data->map_descrs[idx];
+	struct strobe_map_raw map;
+	void *location;
+	uint64_t len;
+
+	descr->tag_len = 0; /* presume no tag is set */
+	descr->cnt = -1; /* presume no value is set */
+
+	location = calc_location(&cfg->map_locs[idx], tls_base);
+	if (unlikely(bpf_probe_read_user(value, sizeof(struct strobe_value_generic), location)))
+		return off;
+
+	if (unlikely(bpf_probe_read_user(&map, sizeof(struct strobe_map_raw), value->ptr)))
+		return off;
+
+	descr->id = map.id;
+	descr->cnt = map.cnt;
+	if (cfg->req_meta_idx == idx) {
+		data->req_id = map.id;
+		data->req_meta_valid = 1;
+	}
+
+	len = bpf_probe_read_user_str(&data->payload[off], STROBE_MAX_STR_LEN, map.tag);
+	if (len <= STROBE_MAX_STR_LEN) {
+		descr->tag_len = len;
+		off += len;
+	}
+
+	__pragma_loop_unroll
+	for (int i = 0; i < STROBE_MAX_MAP_ENTRIES; ++i) {
+		if (i >= map.cnt)
+			break;
+
+		descr->key_lens[i] = 0;
+		len = bpf_probe_read_user_str(&data->payload[off], STROBE_MAX_STR_LEN,
+					      map.entries[i].key);
+		if (len <= STROBE_MAX_STR_LEN) {
+			descr->key_lens[i] = len;
+			off += len;
+		}
+		descr->val_lens[i] = 0;
+		len = bpf_probe_read_user_str(&data->payload[off], STROBE_MAX_STR_LEN,
+					      map.entries[i].val);
+		if (len <= STROBE_MAX_STR_LEN) {
+			descr->val_lens[i] = len;
+			off += len;
+		}
+	}
+
+	return off;
+}
+
+enum read_type {
+	READ_INT_VAR,
+	READ_MAP_VAR,
+	READ_STR_VAR,
+};
+
+struct read_var_ctx {
+	struct strobemeta_payload *data;
+	void *tls_base;
+	struct strobemeta_cfg *cfg;
+	size_t payload_off;
+	/* value gets mutated */
+	struct strobe_value_generic *value;
+	enum read_type type;
+};
+
+static int read_var_callback(__u64 index, struct read_var_ctx *ctx)
+{
+	switch (ctx->type) {
+	case READ_INT_VAR:
+		if (index >= STROBE_MAX_INTS)
+			return 1;
+		read_int_var(ctx->cfg, index, ctx->tls_base, ctx->value, ctx->data);
+		break;
+	case READ_MAP_VAR:
+		if (index >= STROBE_MAX_MAPS)
+			return 1;
+		if (ctx->payload_off > sizeof(ctx->data->payload) - READ_MAP_VAR_PAYLOAD_CAP)
+			return 1;
+		ctx->payload_off = read_map_var(ctx->cfg, index, ctx->tls_base,
+						ctx->value, ctx->data, ctx->payload_off);
+		break;
+	case READ_STR_VAR:
+		if (index >= STROBE_MAX_STRS)
+			return 1;
+		if (ctx->payload_off > sizeof(ctx->data->payload) - STROBE_MAX_STR_LEN)
+			return 1;
+		ctx->payload_off = read_str_var(ctx->cfg, index, ctx->tls_base,
+						ctx->value, ctx->data, ctx->payload_off);
+		break;
+	}
+	return 0;
+}
+
+/*
+ * read_strobe_meta returns NULL, if no metadata was read; otherwise returns
+ * pointer to *right after* payload ends
+ */
+static void *read_strobe_meta(struct task_struct *task,
+			      struct strobemeta_payload *data)
+{
+	pid_t pid = bpf_get_current_pid_tgid() >> 32;
+	struct strobe_value_generic value = {0};
+	struct strobemeta_cfg *cfg;
+	size_t payload_off;
+	void *tls_base;
+
+	cfg = bpf_map_lookup_elem(&strobemeta_cfgs, &pid);
+	if (!cfg)
+		return NULL;
+
+	data->int_vals_set_mask = 0;
+	data->req_meta_valid = 0;
+	payload_off = 0;
+	/*
+	 * we don't have struct task_struct definition, it should be:
+	 * tls_base = (void *)task->thread.fsbase;
+	 */
+	tls_base = (void *)task;
+
+	struct read_var_ctx ctx = {
+		.cfg = cfg,
+		.tls_base = tls_base,
+		.value = &value,
+		.data = data,
+		.payload_off = 0,
+	};
+	int err;
+
+	ctx.type = READ_INT_VAR;
+	err = bpf_loop(STROBE_MAX_INTS, read_var_callback, &ctx, 0);
+	if (err != STROBE_MAX_INTS)
+		return NULL;
+
+	ctx.type = READ_STR_VAR;
+	err = bpf_loop(STROBE_MAX_STRS, read_var_callback, &ctx, 0);
+	if (err != STROBE_MAX_STRS)
+		return NULL;
+
+	ctx.type = READ_MAP_VAR;
+	err = bpf_loop(STROBE_MAX_MAPS, read_var_callback, &ctx, 0);
+	if (err != STROBE_MAX_MAPS)
+		return NULL;
+
+	payload_off = ctx.payload_off;
+	/* this should not really happen, here only to satisfy verifier */
+	if (payload_off > sizeof(data->payload))
+		payload_off = sizeof(data->payload);
+
+	/*
+	 * return pointer right after end of payload, so it's possible to
+	 * calculate exact amount of useful data that needs to be sent
+	 */
+	return &data->payload[payload_off];
+}
+
+SEC("raw_tracepoint/kfree_skb")
+__success
+int on_event(struct pt_regs *ctx) {
+	pid_t pid =  bpf_get_current_pid_tgid() >> 32;
+	struct strobelight_bpf_sample* sample;
+	struct task_struct *task;
+	uint32_t zero = 0;
+	uint64_t ktime_ns;
+	void *sample_end;
+
+	sample = bpf_map_lookup_elem(&sample_heap, &zero);
+	if (!sample)
+		return 0; /* this will never happen */
+
+	sample->pid = pid;
+	bpf_get_current_comm(&sample->comm, TASK_COMM_LEN);
+	ktime_ns = bpf_ktime_get_ns();
+	sample->ktime = ktime_ns;
+
+	task = (struct task_struct *)bpf_get_current_task();
+	sample_end = read_strobe_meta(task, &sample->metadata);
+	sample->has_meta = sample_end != NULL;
+	sample_end = sample_end ? : &sample->metadata;
+
+	if ((ktime_ns >> STACK_TABLE_EPOCH_SHIFT) & 1) {
+		sample->kernel_stack_id = bpf_get_stackid(ctx, &stacks_1, 0);
+		sample->user_stack_id = bpf_get_stackid(ctx, &stacks_1, BPF_F_USER_STACK);
+	} else {
+		sample->kernel_stack_id = bpf_get_stackid(ctx, &stacks_0, 0);
+		sample->user_stack_id = bpf_get_stackid(ctx, &stacks_0, BPF_F_USER_STACK);
+	}
+
+	uint64_t sample_size = sample_end - (void *)sample;
+	/* should always be true */
+	if (sample_size < sizeof(struct strobelight_bpf_sample))
+		bpf_perf_event_output(ctx, &samples, 0, sample, 1 + sample_size);
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("GCC refuses a function with six arguments, use a dummy test")
+__skip("GCC refuses a function with six arguments")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf2.c b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf2.c
new file mode 100644
index 000000000000..b7b1a43aa94e
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf2.c
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* tailcall_bpf2bpf2.c in plain C */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_legacy.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+static __noinline
+int subprog_tail(struct __sk_buff *skb)
+{
+	int ret = 1;
+
+	if (load_byte(skb, 0))
+		bpf_tail_call_static(skb, &jmp_table, 1);
+	else
+		bpf_tail_call_static(skb, &jmp_table, 0);
+	return ret;
+}
+
+int count = 0;
+
+SEC("tc")
+__success __retval(1)
+#ifndef __clang__
+__skip("GCC does not keep the skb in R6 for LD_ABS")
+#endif
+int classifier_0(struct __sk_buff *skb)
+{
+	count++;
+	return subprog_tail(skb);
+}
+
+char __license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy1.c b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy1.c
new file mode 100644
index 000000000000..a66c972a91ed
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy1.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0
+/* tailcall_bpf2bpf_hierarchy1.c in plain C */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_legacy.h"
+#include "bpf_test_utils.h"
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+int count = 0;
+
+static __noinline
+int subprog_tail(struct __sk_buff *skb)
+{
+	int ret = 0;
+
+	bpf_tail_call_static(skb, &jmp_table, 0);
+	return ret;
+}
+
+SEC("tc")
+__success __retval(1)
+int entry(struct __sk_buff *skb)
+{
+	int ret = 1, ret1, ret2;
+
+	clobber_regs_stack();
+
+	count++;
+	ret1 = subprog_tail(skb);
+	ret2 = subprog_tail(skb);
+	__sink(ret1);
+	__sink(ret2);
+
+	return ret;
+}
+
+char __license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy2.c b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy2.c
new file mode 100644
index 000000000000..443dbe042de4
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy2.c
@@ -0,0 +1,81 @@
+// SPDX-License-Identifier: GPL-2.0
+/* tailcall_bpf2bpf_hierarchy2.c in plain C */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_test_utils.h"
+
+int classifier_0(struct __sk_buff *skb);
+int classifier_1(struct __sk_buff *skb);
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 2);
+	__uint(key_size, sizeof(__u32));
+	__array(values, void (void));
+} jmp_table SEC(".maps") = {
+	.values = {
+		[0] = (void *) &classifier_0,
+		[1] = (void *) &classifier_1,
+	},
+};
+
+int count0 = 0;
+int count1 = 0;
+
+static __noinline
+int subprog_tail0(struct __sk_buff *skb)
+{
+	int ret = 0;
+
+	bpf_tail_call_static(skb, &jmp_table, 0);
+	return ret;
+}
+
+SEC("tc")
+__auxiliary
+int classifier_0(struct __sk_buff *skb)
+{
+	count0++;
+	subprog_tail0(skb);
+	return 0;
+}
+
+static __noinline
+int subprog_tail1(struct __sk_buff *skb)
+{
+	int ret = 0;
+
+	bpf_tail_call_static(skb, &jmp_table, 1);
+	return ret;
+}
+
+SEC("tc")
+__auxiliary
+int classifier_1(struct __sk_buff *skb)
+{
+	int ret;
+
+	count1++;
+	ret = subprog_tail1(skb);
+	__sink(ret);
+	return 0;
+}
+
+SEC("tc")
+__success __retval(33)
+int tailcall_bpf2bpf_hierarchy_2(struct __sk_buff *skb)
+{
+	int ret = 0, ret1, ret2;
+
+	clobber_regs_stack();
+
+	ret1 = subprog_tail0(skb);
+	ret2 = subprog_tail1(skb);
+	__sink(ret1);
+	__sink(ret2);
+	__sink(ret);
+	return (count1 << 16) | count0;
+}
+
+char __license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy3.c b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy3.c
new file mode 100644
index 000000000000..3b01c1b037ef
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_tailcall_bpf2bpf_hierarchy3.c
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0
+/* tailcall_bpf2bpf_hierarchy3.c in plain C */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_test_utils.h"
+
+int classifier_0(struct __sk_buff *skb);
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__array(values, void (void));
+} jmp_table0 SEC(".maps") = {
+	.values = {
+		[0] = (void *) &classifier_0,
+	},
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__array(values, void (void));
+} jmp_table1 SEC(".maps") = {
+	.values = {
+		[0] = (void *) &classifier_0,
+	},
+};
+
+int count = 0;
+
+static __noinline
+int subprog_tail(struct __sk_buff *skb, void *jmp_table)
+{
+	int ret = 0;
+
+	bpf_tail_call_static(skb, jmp_table, 0);
+	return ret;
+}
+
+SEC("tc")
+__auxiliary
+int classifier_0(struct __sk_buff *skb)
+{
+	int ret1, ret2;
+
+	count++;
+	ret1 = subprog_tail(skb, &jmp_table0);
+	ret2 = subprog_tail(skb, &jmp_table1);
+	__sink(ret1);
+	__sink(ret2);
+	return count;
+}
+
+SEC("tc")
+__success __retval(33)
+#ifndef __clang__
+__skip("GCC does not read 'count' again behind the tail calls")
+#endif
+int tailcall_bpf2bpf_hierarchy_3(struct __sk_buff *skb)
+{
+	int ret = 0;
+
+	clobber_regs_stack();
+
+	bpf_tail_call_static(skb, &jmp_table0, 0);
+
+	__sink(ret);
+	return ret;
+}
+
+char __license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_tailcall_callback.c b/tools/testing/selftests/bpf/progs/plain_c_tailcall_callback.c
new file mode 100644
index 000000000000..1be4843bf0e7
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_tailcall_callback.c
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0
+/* tailcall_callback.c in plain C */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "bpf_test_utils.h"
+
+int classifier_0(struct __sk_buff *skb);
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__array(values, void (void));
+} jmp_table SEC(".maps") = {
+	.values = {
+		[0] = (void *) &classifier_0,
+	},
+};
+
+SEC("tc")
+__auxiliary
+int classifier_0(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+static __noinline
+int subprog_tail0(struct __sk_buff *skb)
+{
+	int ret = 0;
+
+	bpf_tail_call_static(skb, &jmp_table, 0);
+	return ret;
+}
+
+static __noinline
+int callback_loop(int index, void **cb_ctx)
+{
+	int ret;
+
+	ret = subprog_tail0(*cb_ctx);
+	return ret ? 1 : 0;
+}
+
+static __noinline
+int callback_empty(int index, void *data)
+{
+	return 0;
+}
+
+/* callback involving subprog with tail call is rejected */
+SEC("tc")
+__failure
+__msg("cannot tail call within callback")
+int tailcall_callback_1(struct __sk_buff *skb)
+{
+	clobber_regs_stack();
+
+	bpf_loop(1, callback_loop, &skb, 0);
+	return 0;
+}
+
+/* subprogs with tailcall do not affect no-tailcall callback */
+SEC("tc")
+__success __retval(0)
+int tailcall_callback_2(struct __sk_buff *skb)
+{
+	int ret;
+
+	clobber_regs_stack();
+
+	ret = subprog_tail0(skb);
+	__sink(ret);
+
+	bpf_loop(1, callback_empty, NULL, 0);
+	return 0;
+}
+
+char __license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c b/tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c
new file mode 100644
index 000000000000..751a7157d7d3
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c
@@ -0,0 +1,93 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2026 Google LLC */
+/* test_wakeup_source.c in plain C */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+#include "wakeup_source.h"
+
+#define MAX_LOOP_ITER 1000
+#define RB_SIZE (16384 * 4)
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, RB_SIZE);
+} rb SEC(".maps");
+
+struct bpf_ws_lock;
+struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void) __ksym;
+void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock) __ksym;
+void *bpf_wakeup_sources_get_head(void) __ksym;
+
+SEC("syscall")
+__success __retval(0)
+int iterate_wakeupsources(void *ctx)
+{
+	struct list_head *head = bpf_wakeup_sources_get_head();
+	struct list_head *pos = head;
+	struct bpf_ws_lock *lock;
+	int i;
+
+	lock = bpf_wakeup_sources_read_lock();
+	if (!lock)
+		return 0;
+
+	for (i = 0; i < MAX_LOOP_ITER; i++) {
+		if (bpf_core_read(&pos, sizeof(pos), &pos->next) || !pos || pos == head)
+			break;
+
+		struct wakeup_event_t *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
+
+		if (!e)
+			break;
+
+		struct wakeup_source *ws = bpf_core_cast(
+				(void *)pos - bpf_core_field_offset(struct wakeup_source, entry),
+				struct wakeup_source);
+		s64 active_time = 0;
+		bool active = BPF_CORE_READ_BITFIELD(ws, active);
+		bool autosleep_enable = BPF_CORE_READ_BITFIELD(ws, autosleep_enabled);
+		s64 last_time = ws->last_time;
+		s64 max_time = ws->max_time;
+		s64 prevent_sleep_time = ws->prevent_sleep_time;
+		s64 total_time = ws->total_time;
+
+		if (active) {
+			s64 curr_time = bpf_ktime_get_ns();
+			s64 prevent_time = ws->start_prevent_time;
+
+			if (curr_time > last_time)
+				active_time = curr_time - last_time;
+
+			total_time += active_time;
+			if (active_time > max_time)
+				max_time = active_time;
+			if (autosleep_enable && curr_time > prevent_time)
+				prevent_sleep_time += curr_time - prevent_time;
+		}
+
+		e->active_count = ws->active_count;
+		e->active_time_ns = active_time;
+		e->event_count = ws->event_count;
+		e->expire_count = ws->expire_count;
+		e->last_time_ns = last_time;
+		e->max_time_ns = max_time;
+		e->prevent_sleep_time_ns = prevent_sleep_time;
+		e->total_time_ns = total_time;
+		e->wakeup_count = ws->wakeup_count;
+
+		if (bpf_probe_read_kernel_str(
+				e->name, WAKEUP_NAME_LEN, ws->name) < 0)
+			e->name[0] = '\0';
+
+		bpf_ringbuf_submit(e, 0);
+	}
+
+	bpf_wakeup_sources_read_unlock(lock);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_tracing_stack_arg_tailcall_tgt.c b/tools/testing/selftests/bpf/progs/plain_c_tracing_stack_arg_tailcall_tgt.c
new file mode 100644
index 000000000000..eb521c8ec321
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_tracing_stack_arg_tailcall_tgt.c
@@ -0,0 +1,55 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+/* tracing_stack_arg_tailcall_tgt.c in plain C */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+__u64 base;
+__u64 ret_stack;
+
+#if defined(__clang__) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+/* g and h are on the stack, for the x86-64 kernel convention too. */
+static __noinline __u64 callee_stack(__u64 a, __u64 b, __u64 c, __u64 d,
+				     __u64 e, __u64 f, __u64 g, __u64 h)
+{
+	return a + b * 2 + c * 3 + d * 4 + e * 5 + f * 6 + g * 7 + h * 8;
+}
+
+/* No stack args of its own, so it can share a program with a tail call. */
+static __noinline __u64 call_stack(__u64 x)
+{
+	return callee_stack(x + 1, x + 2, x + 3, x + 4, x + 5, x + 6, x + 7,
+			    x + 8);
+}
+
+#endif
+
+/* Makes the program tail_call_reachable, though callee_stack() is not. */
+static __noinline int subprog_tail(void *ctx)
+{
+	int ret = 0;
+
+	bpf_tail_call_static(ctx, &jmp_table, 0);
+	return ret;
+}
+
+SEC("syscall")
+__success __retval(0) __arch_x86_64 __arch_arm64
+int run(void *ctx)
+{
+#if defined(__clang__) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+	ret_stack = call_stack(base);
+#endif
+	return subprog_tail(ctx);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_verifier_arena_large.c b/tools/testing/selftests/bpf/progs/plain_c_verifier_arena_large.c
new file mode 100644
index 000000000000..4c5339edb06f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_verifier_arena_large.c
@@ -0,0 +1,122 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
+/* verifier_arena_large.c in plain C */
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+#define ARENA_SIZE (1ull << 32)
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARENA);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, ARENA_SIZE / PAGE_SIZE);
+} arena SEC(".maps");
+
+#define PAGE_CNT 100
+__u8 __arena * __arena page[PAGE_CNT]; /* occupies the first page */
+__u8 __arena *base;
+
+/*
+ * Check that arena's range_tree algorithm allocates pages sequentially
+ * on the first pass and then fills in all gaps on the second pass.
+ */
+__noinline int alloc_pages(int page_cnt, int pages_atonce, bool first_pass,
+		int max_idx, int step)
+{
+	__u8 __arena *pg;
+	int i, pg_idx;
+
+	for (i = 0; i < page_cnt; i++) {
+		pg = bpf_arena_alloc_pages(&arena, NULL, pages_atonce,
+					   NUMA_NO_NODE, 0);
+		if (!pg)
+			return step;
+		pg_idx = (unsigned long) (pg - base) / PAGE_SIZE;
+		if (first_pass) {
+			/* Pages must be allocated sequentially */
+			if (pg_idx != i)
+				return step + 100;
+		} else {
+			/* Allocator must fill into gaps */
+			if (pg_idx >= max_idx || (pg_idx & 1))
+				return step + 200;
+		}
+		*pg = pg_idx;
+		page[pg_idx] = pg;
+	}
+	return 0;
+}
+
+SEC("syscall")
+__success __retval(0) __skip_without_arena
+int big_alloc2(void *ctx)
+{
+	__u8 __arena *pg;
+	int i, err;
+
+	base = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
+	if (!base)
+		return 1;
+	bpf_arena_free_pages(&arena, (void __arena *)base, 1);
+
+	err = alloc_pages(PAGE_CNT, 1, true, PAGE_CNT, 2);
+	if (err)
+		return err;
+
+	/* Clear all even pages */
+	for (i = 0; i < PAGE_CNT; i += 2) {
+		pg = page[i];
+		if (*pg != i)
+			return 3;
+		bpf_arena_free_pages(&arena, (void __arena *)pg, 1);
+		page[i] = NULL;
+	}
+
+	/* Allocate into freed gaps */
+	err = alloc_pages(PAGE_CNT / 2, 1, false, PAGE_CNT, 4);
+	if (err)
+		return err;
+
+	/* Free pairs of pages */
+	for (i = 0; i < PAGE_CNT; i += 4) {
+		pg = page[i];
+		if (*pg != i)
+			return 5;
+		bpf_arena_free_pages(&arena, (void __arena *)pg, 2);
+		page[i] = NULL;
+		barrier();
+		page[i + 1] = NULL;
+	}
+
+	/* Allocate 2 pages at a time into freed gaps */
+	err = alloc_pages(PAGE_CNT / 4, 2, false, PAGE_CNT, 6);
+	if (err)
+		return err;
+
+	/* Check pages without freeing */
+	for (i = 0; i < PAGE_CNT; i += 2) {
+		pg = page[i];
+		if (*pg != i)
+			return 7;
+	}
+
+	pg = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
+
+	if (!pg)
+		return 8;
+	/*
+	 * The first PAGE_CNT pages are occupied. The new page
+	 * must be above.
+	 */
+	if ((pg - base) / PAGE_SIZE < PAGE_CNT)
+		return 9;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/plain_c_verifier_iterating_callbacks.c
new file mode 100644
index 000000000000..64a50586e9d7
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_verifier_iterating_callbacks.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0
+/* verifier_iterating_callbacks.c in plain C */
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+#define ARR_SZ 1000000
+char arr[ARR_SZ];
+
+SEC("socket")
+__success __retval(0xd495cdc0)
+int cond_break1(const void *ctx)
+{
+	unsigned long i;
+	unsigned int sum = 0;
+
+	for (i = 0; i < ARR_SZ; i++)
+		sum += i;
+	for (i = 0; i < ARR_SZ; i++) {
+		sum += i + arr[i];
+	}
+
+	return sum;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_verifier_loop_as_is.c b/tools/testing/selftests/bpf/progs/plain_c_verifier_loop_as_is.c
new file mode 100644
index 000000000000..daa05d2b7b57
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_verifier_loop_as_is.c
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* verifier_loop_as_is.c in plain C */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u64[9]);
+} map SEC(".maps");
+
+/* 8 comes back to the head of the inner loop, and goes from there to the head of the outer one */
+SEC("socket")
+__success __retval(0)
+#ifndef __clang__
+__skip("GCC unrolls the inner loop")
+#endif
+int inner_loop_stops_at_8(void *ctx)
+{
+	__u32 key = 0;
+	__u64 n = 0, *v;
+	int i, j;
+
+	v = bpf_map_lookup_elem(&map, &key);
+	if (!v)
+		return 0;
+	for (i = 0; i < 100; i++) {
+		for (j = 0; j < 100; j++) {
+			if (*(volatile __u64 *)v == 1)
+				continue;
+			if (n < 8)
+				v[n++] = 0;
+		}
+	}
+	return v[n];
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/plain_c_xdp_lb_bench.c b/tools/testing/selftests/bpf/progs/plain_c_xdp_lb_bench.c
new file mode 100644
index 000000000000..1499e53f117f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/plain_c_xdp_lb_bench.c
@@ -0,0 +1,723 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+/* xdp_lb_bench.c and bench_bpf_timing.bpf.h in plain C */
+
+#include <stddef.h>
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/in.h>
+#include <linux/tcp.h>
+#include <linux/udp.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_endian.h>
+#include "bpf_compiler.h"
+#include "xdp_lb_bench_common.h"
+#include "bpf_misc.h"
+
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
+	(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+#ifndef BENCH_NR_SAMPLES
+#define BENCH_NR_SAMPLES	4096
+#endif
+#ifndef BENCH_NR_CPUS
+#define BENCH_NR_CPUS		256
+#endif
+#define BENCH_CPU_MASK		(BENCH_NR_CPUS - 1)
+
+__u64 timing_samples[BENCH_NR_CPUS][BENCH_NR_SAMPLES];
+__u32 timing_idx[BENCH_NR_CPUS];
+
+volatile __u32 batch_iters;
+volatile __u32 timing_enabled;
+
+static void bench_record_sample(__u64 elapsed_ns)
+{
+	__u32 cpu, idx;
+
+	if (!timing_enabled)
+		return;
+
+	cpu = bpf_get_smp_processor_id() & BENCH_CPU_MASK;
+	idx = timing_idx[cpu];
+
+	if (idx >= BENCH_NR_SAMPLES)
+		return;
+
+	timing_samples[cpu][idx] = elapsed_ns;
+	timing_idx[cpu] = idx + 1;
+}
+
+/*
+ * @body:  expression to time; return value (int) stored in __bench_result.
+ * @reset: undo body's side-effects so each iteration starts identically.
+ *         May reference __bench_result.  Use ({}) for empty reset.
+ *
+ * Runs batch_iters timed iterations, then one untimed iteration whose
+ * return value the macro evaluates to (for validation).
+ */
+#define BENCH_BPF_LOOP(body, reset) ({					\
+	__u64 __bench_start = bpf_ktime_get_ns();			\
+	__u32 __bench_i;						\
+	int __bench_result;						\
+									\
+	for (__bench_i = 0;						\
+	     __bench_i < batch_iters;					\
+	     __bench_i++) {						\
+		__bench_result = (body);				\
+		reset;							\
+	}								\
+									\
+	bench_record_sample(bpf_ktime_get_ns() - __bench_start);	\
+									\
+	__bench_result = (body);					\
+	__bench_result;							\
+})
+
+#ifndef IPPROTO_FRAGMENT
+#define IPPROTO_FRAGMENT 44
+#endif
+
+/* jhash helpers */
+
+static inline __u32 rol32(__u32 word, unsigned int shift)
+{
+	return (word << shift) | (word >> ((-shift) & 31));
+}
+
+#define __jhash_mix(a, b, c)			\
+{						\
+	a -= c;  a ^= rol32(c, 4);  c += b;	\
+	b -= a;  b ^= rol32(a, 6);  a += c;	\
+	c -= b;  c ^= rol32(b, 8);  b += a;	\
+	a -= c;  a ^= rol32(c, 16); c += b;	\
+	b -= a;  b ^= rol32(a, 19); a += c;	\
+	c -= b;  c ^= rol32(b, 4);  b += a;	\
+}
+
+#define __jhash_final(a, b, c)			\
+{						\
+	c ^= b; c -= rol32(b, 14);		\
+	a ^= c; a -= rol32(c, 11);		\
+	b ^= a; b -= rol32(a, 25);		\
+	c ^= b; c -= rol32(b, 16);		\
+	a ^= c; a -= rol32(c, 4);		\
+	b ^= a; b -= rol32(a, 14);		\
+	c ^= b; c -= rol32(b, 24);		\
+}
+
+#define JHASH_INITVAL 0xdeadbeef
+
+static inline __u32 __jhash_nwords(__u32 a, __u32 b, __u32 c, __u32 initval)
+{
+	a += initval;
+	b += initval;
+	c += initval;
+	__jhash_final(a, b, c);
+	return c;
+}
+
+static inline __u32 jhash_2words(__u32 a, __u32 b, __u32 initval)
+{
+	return __jhash_nwords(a, b, 0, initval + JHASH_INITVAL + (2 << 2));
+}
+
+static inline __u32 jhash2_4words(const __u32 *k, __u32 initval)
+{
+	__u32 a, b, c;
+
+	a = b = c = JHASH_INITVAL + (4 << 2) + initval;
+
+	a += k[0]; b += k[1]; c += k[2];
+	__jhash_mix(a, b, c);
+
+	a += k[3];
+	__jhash_final(a, b, c);
+
+	return c;
+}
+
+static void ipv4_csum(struct iphdr *iph)
+{
+	__u16 *next_iph = (__u16 *)iph;
+	__u32 csum = 0;
+	int i;
+
+	__pragma_loop_unroll_full
+	for (i = 0; i < (int)(sizeof(*iph) >> 1); i++)
+		csum += *next_iph++;
+
+	csum = (csum & 0xffff) + (csum >> 16);
+	csum = (csum & 0xffff) + (csum >> 16);
+	iph->check = ~csum;
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 64);
+	__type(key, struct vip_definition);
+	__type(value, struct vip_meta);
+} vip_map SEC(".maps");
+
+struct lru_inner_map {
+	__uint(type, BPF_MAP_TYPE_LRU_HASH);
+	__type(key, struct flow_key);
+	__type(value, struct real_pos_lru);
+	__uint(max_entries, DEFAULT_LRU_SIZE);
+} lru_inner SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
+	__type(key, __u32);
+	__type(value, __u32);
+	__uint(max_entries, BENCH_NR_CPUS);
+	__array(values, struct lru_inner_map);
+} lru_mapping SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, CH_RINGS_SIZE);
+	__type(key, __u32);
+	__type(value, __u32);
+} ch_rings SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, MAX_REALS);
+	__type(key, __u32);
+	__type(value, struct real_definition);
+} reals SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, STATS_SIZE);
+	__type(key, __u32);
+	__type(value, struct lb_stats);
+} stats SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, MAX_REALS);
+	__type(key, __u32);
+	__type(value, struct lb_stats);
+} reals_stats SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct ctl_value);
+} ctl_array SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct vip_definition);
+} vip_miss_stats SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, MAX_REALS);
+	__type(key, __u32);
+	__type(value, __u32);
+} lru_miss_stats SEC(".maps");
+
+volatile __u32 flow_mask;
+volatile __u32 cold_lru;
+__u32 batch_gen;
+
+/*
+ * old_eth MUST be read BEFORE writing the outer header because
+ * bpf_xdp_adjust_head makes them overlap.
+ */
+static int encap_v4(struct xdp_md *xdp, __be32 saddr, __be32 daddr,
+		    __u16 payload_len, const __u8 *dst_mac)
+{
+	struct ethhdr *new_eth, *old_eth;
+	void *data, *data_end;
+	struct iphdr *iph;
+
+	if (bpf_xdp_adjust_head(xdp, -(int)sizeof(struct iphdr)))
+		return -1;
+
+	data     = (void *)(long)xdp->data;
+	data_end = (void *)(long)xdp->data_end;
+
+	new_eth = data;
+	iph     = data + sizeof(struct ethhdr);
+	old_eth = data + sizeof(struct iphdr);
+
+	if (new_eth + 1 > data_end || old_eth + 1 > data_end || iph + 1 > data_end)
+		return -1;
+
+	__builtin_memcpy(new_eth->h_source, old_eth->h_dest, sizeof(new_eth->h_source));
+	__builtin_memcpy(new_eth->h_dest, dst_mac, sizeof(new_eth->h_dest));
+	new_eth->h_proto = bpf_htons(ETH_P_IP);
+
+	__builtin_memset(iph, 0, sizeof(*iph));
+	iph->version  = 4;
+	iph->ihl      = sizeof(*iph) >> 2;
+	iph->protocol = IPPROTO_IPIP;
+	iph->tot_len  = bpf_htons(payload_len + sizeof(*iph));
+	iph->ttl      = 64;
+	iph->saddr    = saddr;
+	iph->daddr    = daddr;
+	ipv4_csum(iph);
+
+	return 0;
+}
+
+static int encap_v6(struct xdp_md *xdp, const __be32 saddr[4],
+		    const __be32 daddr[4], __u8 nexthdr, __u16 payload_len,
+		    const __u8 *dst_mac)
+{
+	struct ethhdr *new_eth, *old_eth;
+	void *data, *data_end;
+	struct ipv6hdr *ip6h;
+
+	if (bpf_xdp_adjust_head(xdp, -(int)sizeof(struct ipv6hdr)))
+		return -1;
+
+	data     = (void *)(long)xdp->data;
+	data_end = (void *)(long)xdp->data_end;
+
+	new_eth = data;
+	ip6h    = data + sizeof(struct ethhdr);
+	old_eth = data + sizeof(struct ipv6hdr);
+
+	if (new_eth + 1 > data_end || old_eth + 1 > data_end || ip6h + 1 > data_end)
+		return -1;
+
+	__builtin_memcpy(new_eth->h_source, old_eth->h_dest, sizeof(new_eth->h_source));
+	__builtin_memcpy(new_eth->h_dest, dst_mac, sizeof(new_eth->h_dest));
+	new_eth->h_proto = bpf_htons(ETH_P_IPV6);
+
+	__builtin_memset(ip6h, 0, sizeof(*ip6h));
+	ip6h->version     = 6;
+	ip6h->nexthdr     = nexthdr;
+	ip6h->payload_len = bpf_htons(payload_len);
+	ip6h->hop_limit   = 64;
+	__builtin_memcpy(&ip6h->saddr, saddr, sizeof(ip6h->saddr));
+	__builtin_memcpy(&ip6h->daddr, daddr, sizeof(ip6h->daddr));
+
+	return 0;
+}
+
+static void update_stats(void *map, __u32 key, __u16 bytes)
+{
+	struct lb_stats *st = bpf_map_lookup_elem(map, &key);
+
+	if (st) {
+		st->v1 += 1;
+		st->v2 += bytes;
+	}
+}
+
+static void count_action(int action)
+{
+	struct lb_stats *st;
+	__u32 key;
+
+	if (action == XDP_TX)
+		key = STATS_XDP_TX;
+	else if (action == XDP_PASS)
+		key = STATS_XDP_PASS;
+	else
+		key = STATS_XDP_DROP;
+
+	st = bpf_map_lookup_elem(&stats, &key);
+	if (st)
+		st->v1 += 1;
+}
+
+static bool is_under_flood(void)
+{
+	__u32 key = STATS_NEW_CONN;
+	struct lb_stats *conn_st = bpf_map_lookup_elem(&stats, &key);
+	__u64 cur_time;
+
+	if (!conn_st)
+		return true;
+
+	cur_time = bpf_ktime_get_ns();
+	if ((cur_time - conn_st->v2) > ONE_SEC) {
+		conn_st->v1 = 1;
+		conn_st->v2 = cur_time;
+	} else {
+		conn_st->v1 += 1;
+		if (conn_st->v1 > MAX_CONN_RATE)
+			return true;
+	}
+	return false;
+}
+
+static struct real_definition *connection_table_lookup(void *lru_map,
+						       struct flow_key *flow,
+						       __u32 *out_pos)
+{
+	struct real_pos_lru *dst_lru;
+	struct real_definition *real;
+	__u32 key;
+
+	dst_lru = bpf_map_lookup_elem(lru_map, flow);
+	if (!dst_lru)
+		return NULL;
+
+	/* UDP connections use atime-based timeout instead of FIN/RST */
+	if (flow->proto == IPPROTO_UDP) {
+		__u64 cur_time = bpf_ktime_get_ns();
+
+		if (cur_time - dst_lru->atime > LRU_UDP_TIMEOUT)
+			return NULL;
+		dst_lru->atime = cur_time;
+	}
+
+	key = dst_lru->pos;
+	*out_pos = key;
+	real = bpf_map_lookup_elem(&reals, &key);
+	return real;
+}
+
+static bool get_packet_dst(struct real_definition **real, struct flow_key *flow,
+			   struct vip_meta *vip_info, bool is_v6, void *lru_map,
+			   bool is_rst, __u32 *out_pos)
+{
+	bool under_flood;
+	__u32 hash, ch_key;
+	__u32 *ch_val;
+	__u32 real_pos;
+
+	under_flood = is_under_flood();
+
+	if (is_v6) {
+		__u32 src_hash = jhash2_4words((__u32 *)flow->srcv6, MAX_VIPS);
+
+		hash = jhash_2words(src_hash, flow->ports, CH_RING_SIZE);
+	} else {
+		hash = jhash_2words(flow->src, flow->ports, CH_RING_SIZE);
+	}
+
+	ch_key = CH_RING_SIZE * vip_info->vip_num + hash % CH_RING_SIZE;
+	ch_val = bpf_map_lookup_elem(&ch_rings, &ch_key);
+	if (!ch_val)
+		return false;
+	real_pos = *ch_val;
+
+	*real = bpf_map_lookup_elem(&reals, &real_pos);
+	if (!(*real))
+		return false;
+
+	if (!(vip_info->flags & F_LRU_BYPASS) && !under_flood && !is_rst) {
+		struct real_pos_lru new_lru = { .pos = real_pos };
+
+		if (flow->proto == IPPROTO_UDP)
+			new_lru.atime = bpf_ktime_get_ns();
+		bpf_map_update_elem(lru_map, flow, &new_lru, BPF_ANY);
+	}
+
+	*out_pos = real_pos;
+	return true;
+}
+
+static void update_vip_lru_miss_stats(struct vip_definition *vip, bool is_v6,
+				      __u32 real_idx)
+{
+	struct vip_definition *miss_vip;
+	__u32 key = 0;
+	__u32 *cnt;
+
+	miss_vip = bpf_map_lookup_elem(&vip_miss_stats, &key);
+	if (!miss_vip)
+		return;
+
+	if (is_v6) {
+		if (miss_vip->vipv6[0] != vip->vipv6[0] || miss_vip->vipv6[1] != vip->vipv6[1] ||
+		    miss_vip->vipv6[2] != vip->vipv6[2] || miss_vip->vipv6[3] != vip->vipv6[3])
+			return;
+	} else {
+		if (miss_vip->vip != vip->vip)
+			return;
+	}
+
+	if (miss_vip->port != vip->port || miss_vip->proto != vip->proto)
+		return;
+
+	cnt = bpf_map_lookup_elem(&lru_miss_stats, &real_idx);
+	if (cnt)
+		*cnt += 1;
+}
+
+static __noinline int process_packet(struct xdp_md *xdp)
+{
+	void *data     = (void *)(long)xdp->data;
+	void *data_end = (void *)(long)xdp->data_end;
+	struct ethhdr *eth = data;
+	struct real_definition *dst = NULL;
+	struct vip_definition vip_def = {};
+	struct ctl_value *cval;
+	struct flow_key flow = {};
+	struct vip_meta *vip_info;
+	struct lb_stats *data_stats;
+	struct udphdr *uh;
+	__be32 tnl_src[4];
+	void *lru_map;
+	void *l4;
+	__u16 payload_len;
+	__u32 real_pos = 0, cpu_num, key;
+	__u8 proto;
+	int action = XDP_DROP;
+	bool is_v6, is_syn = false, is_rst = false;
+
+	if (eth + 1 > data_end)
+		goto out;
+
+	if (eth->h_proto == bpf_htons(ETH_P_IPV6)) {
+		is_v6 = true;
+	} else if (eth->h_proto == bpf_htons(ETH_P_IP)) {
+		is_v6 = false;
+	} else {
+		action = XDP_PASS;
+		goto out;
+	}
+
+	if (is_v6) {
+		struct ipv6hdr *ip6h = (void *)(eth + 1);
+
+		if (ip6h + 1 > data_end)
+			goto out;
+		if (ip6h->nexthdr == IPPROTO_FRAGMENT)
+			goto out;
+
+		payload_len = sizeof(struct ipv6hdr) + bpf_ntohs(ip6h->payload_len);
+		proto = ip6h->nexthdr;
+
+		__builtin_memcpy(flow.srcv6, &ip6h->saddr, sizeof(flow.srcv6));
+		__builtin_memcpy(flow.dstv6, &ip6h->daddr, sizeof(flow.dstv6));
+		__builtin_memcpy(vip_def.vipv6, &ip6h->daddr, sizeof(vip_def.vipv6));
+		l4 = (void *)(ip6h + 1);
+	} else {
+		struct iphdr *iph = (void *)(eth + 1);
+
+		if (iph + 1 > data_end)
+			goto out;
+		if (iph->ihl != 5)
+			goto out;
+		if (iph->frag_off & bpf_htons(PCKT_FRAGMENTED))
+			goto out;
+
+		payload_len = bpf_ntohs(iph->tot_len);
+		proto = iph->protocol;
+
+		flow.src    = iph->saddr;
+		flow.dst    = iph->daddr;
+		vip_def.vip = iph->daddr;
+		l4 = (void *)(iph + 1);
+	}
+
+	/* TCP and UDP share the same port layout at offset 0 */
+	if (proto != IPPROTO_TCP && proto != IPPROTO_UDP) {
+		action = XDP_PASS;
+		goto out;
+	}
+
+	uh = l4;
+	if ((void *)(uh + 1) > data_end)
+		goto out;
+	flow.port16[0] = uh->source;
+	flow.port16[1] = uh->dest;
+
+	if (proto == IPPROTO_TCP) {
+		struct tcphdr *th = l4;
+
+		if ((void *)(th + 1) > data_end)
+			goto out;
+		is_syn = th->syn;
+		is_rst = th->rst;
+	}
+
+	flow.proto    = proto;
+	vip_def.port  = flow.port16[1];
+	vip_def.proto = proto;
+
+	vip_info = bpf_map_lookup_elem(&vip_map, &vip_def);
+	if (!vip_info) {
+		action = XDP_PASS;
+		goto out;
+	}
+
+	key = STATS_LRU;
+	data_stats = bpf_map_lookup_elem(&stats, &key);
+	if (!data_stats)
+		goto out;
+	data_stats->v1 += 1;
+
+	cpu_num = bpf_get_smp_processor_id();
+	lru_map = bpf_map_lookup_elem(&lru_mapping, &cpu_num);
+	if (!lru_map)
+		goto out;
+
+	if (!(vip_info->flags & F_LRU_BYPASS) && !is_syn)
+		dst = connection_table_lookup(lru_map, &flow, &real_pos);
+
+	if (!dst) {
+		if (flow.proto == IPPROTO_TCP) {
+			struct lb_stats *miss_st;
+
+			key = STATS_LRU_MISS;
+			miss_st = bpf_map_lookup_elem(&stats, &key);
+			if (miss_st)
+				miss_st->v1 += 1;
+		}
+
+		if (!get_packet_dst(&dst, &flow, vip_info, is_v6, lru_map, is_rst, &real_pos))
+			goto out;
+
+		update_vip_lru_miss_stats(&vip_def, is_v6, real_pos);
+		data_stats->v2 += 1;
+	}
+
+	key = 0;
+	cval = bpf_map_lookup_elem(&ctl_array, &key);
+	if (!cval)
+		goto out;
+
+	update_stats(&stats, vip_info->vip_num, payload_len);
+	update_stats(&reals_stats, real_pos, payload_len);
+
+	if (is_v6) {
+		create_encap_ipv6_src(flow.port16[0], flow.srcv6[0], tnl_src);
+		if (encap_v6(xdp, tnl_src, dst->dstv6, IPPROTO_IPV6, payload_len, cval->mac))
+			goto out;
+	} else if (dst->flags & F_IPV6) {
+		create_encap_ipv6_src(flow.port16[0], flow.src, tnl_src);
+		if (encap_v6(xdp, tnl_src, dst->dstv6, IPPROTO_IPIP, payload_len, cval->mac))
+			goto out;
+	} else {
+		if (encap_v4(xdp, create_encap_ipv4_src(flow.port16[0], flow.src), dst->dst,
+			     payload_len, cval->mac))
+			goto out;
+	}
+
+	action = XDP_TX;
+
+out:
+	count_action(action);
+	return action;
+}
+
+static int strip_encap(struct xdp_md *xdp, const struct ethhdr *saved_eth)
+{
+	void *data = (void *)(long)xdp->data;
+	void *data_end = (void *)(long)xdp->data_end;
+	struct ethhdr *eth = data;
+	int hdr_sz;
+
+	if (eth + 1 > data_end)
+		return -1;
+
+	hdr_sz = (eth->h_proto == bpf_htons(ETH_P_IPV6)) ? (int)sizeof(struct ipv6hdr)
+							 : (int)sizeof(struct iphdr);
+
+	if (bpf_xdp_adjust_head(xdp, hdr_sz))
+		return -1;
+
+	data     = (void *)(long)xdp->data;
+	data_end = (void *)(long)xdp->data_end;
+	eth      = data;
+
+	if (eth + 1 > data_end)
+		return -1;
+
+	__builtin_memcpy(eth, saved_eth, sizeof(*saved_eth));
+	return 0;
+}
+
+static void randomize_src(struct xdp_md *xdp, int saddr_off, __u32 *rand_state)
+{
+	void *data     = (void *)(long)xdp->data;
+	void *data_end = (void *)(long)xdp->data_end;
+	__u32 *saddr   = data + saddr_off;
+
+	*rand_state ^= *rand_state << 13;
+	*rand_state ^= *rand_state >> 17;
+	*rand_state ^= *rand_state << 5;
+
+	if ((void *)(saddr + 1) <= data_end)
+		*saddr = *rand_state & flow_mask;
+}
+
+SEC("xdp")
+__success
+int xdp_lb_bench(struct xdp_md *xdp)
+{
+	void *data     = (void *)(long)xdp->data;
+	void *data_end = (void *)(long)xdp->data_end;
+	struct ethhdr *eth = data;
+	struct ethhdr saved_eth;
+	__u32 rand_state = 0;
+	__u32 batch_hash = 0;
+	int saddr_off = 0;
+	bool is_v6;
+
+	if (eth + 1 > data_end)
+		return XDP_DROP;
+
+	__builtin_memcpy(&saved_eth, eth, sizeof(saved_eth));
+
+	is_v6 = (saved_eth.h_proto == bpf_htons(ETH_P_IPV6));
+
+	saddr_off = sizeof(struct ethhdr) + (is_v6 ? offsetof(struct ipv6hdr, saddr) :
+					     offsetof(struct iphdr, saddr));
+
+	if (flow_mask)
+		rand_state = bpf_get_prandom_u32() | 1;
+
+	if (cold_lru) {
+		__u32 *saddr = data + saddr_off;
+
+		batch_gen++;
+		batch_hash = (batch_gen + bpf_get_smp_processor_id()) * KNUTH_HASH_MULT;
+		if ((void *)(saddr + 1) <= data_end)
+			*saddr ^= batch_hash;
+	}
+
+	return BENCH_BPF_LOOP(
+		process_packet(xdp),
+		({
+			if (__bench_result == XDP_TX) {
+				if (strip_encap(xdp, &saved_eth))
+					return XDP_DROP;
+				if (rand_state)
+					randomize_src(xdp, saddr_off, &rand_state);
+			}
+			if (cold_lru) {
+				void *d = (void *)(long)xdp->data;
+				void *de = (void *)(long)xdp->data_end;
+				__u32 *__sa = d + saddr_off;
+
+				if ((void *)(__sa + 1) <= de)
+					*__sa ^= batch_hash;
+			}
+		})
+	);
+}
+
+#else
+
+SEC("socket")
+__description("stack_arg is not supported by compiler or jit, use a dummy test")
+__skip("stack_arg is not supported by compiler or jit")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
-- 
2.55.0

Re: [PATCH bpf-next v4 46/46] selftests/bpf: Test progs without bpf_for(), zero, can_loop and such

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5488 bytes --]

> selftests/bpf: Test progs without bpf_for(), zero, can_loop and such

TL;DR:
- Test issue: plain_c arena tests are not serial, so parallel runs on KASAN
  VMs may bring back OOM kills
- plain_c wakeup_source test may fail instead of skip where the kfuncs are
  missing (e.g. s390)

> diff --git a/tools/testing/selftests/bpf/prog_tests/plain_c.c b/tools/testing/selftests/bpf/prog_tests/plain_c.c
> new file mode 100644
> index 000000000000..b7305118a454
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/plain_c.c

[ ... ]

> +#define RUN(name) RUN_TESTS(plain_c_##name)

[ ... ]

> +void test_plain_c_arena_htab(void) { RUN(arena_htab); }
> +void test_plain_c_arena_htab_asm(void) { RUN(arena_htab_asm); }
> +
> +void test_plain_c_arena_list(void)
> +{
> +	RUN(arena_list);
> +	if (test__start_subtest("run"))
> +		arena_list_add_del();
> +}
> +
> +void test_plain_c_arena_race(void) { RUN(arena_race); }
> +void test_plain_c_arena_strsearch(void) { RUN(arena_strsearch); }

Should these arena tests be serial_test_plain_c_arena_htab(),
serial_test_plain_c_arena_htab_asm(), serial_test_plain_c_arena_list()
and serial_test_plain_c_arena_strsearch(), to match the tests they mirror?

As written they are test_*(), so test_progs runs them in the parallel
phase.  The same applies to test_plain_c_verifier_arena_large() below.

plain_c_arena_htab, plain_c_arena_htab_asm (it includes
plain_c_arena_htab.c), plain_c_arena_list, plain_c_arena_strsearch and
plain_c_verifier_arena_large all declare a BPF_MAP_TYPE_ARENA map.  Each
RUN() subtest opens and loads the object again, so each creates a new
arena, and the "run" subtest of test_plain_c_arena_list() creates one
more.

Every arena reserves a 4GB kernel VM area:

    arena_map_alloc() kernel/bpf/arena.c:
        kern_vm = get_vm_area(KERN_VM_SZ, VM_SPARSE | VM_USERMAP);

and alloc_vmap_area() populates the KASAN shadow for the whole area
right away:

    mm/vmalloc.c:
        ret = kasan_populate_vmalloc(addr, size, gfp_mask);

On a generic KASAN build that is about 512MB of shadow per live arena.

Commit a14f9ba080c1 ("selftests/bpf: Run arena tests serially") renamed
arena_list, arena_htab, arena_strsearch and the other arena_* tests to
serial_test_*() for this reason.  Its message says:

    When several arena tests run in parallel their shadow adds up,
    exhausts the memory of a small CI VM and trips the OOM killer during
    map creation.

It left verifier_arena* parallel only because those share RUN() suites,
which does not apply here, since each plain_c test is its own top-level
function.

The parallel dispatcher in test_progs.c skips only tests that have
run_serial_test:

    if (!test->should_run || test->run_serial_test)
        continue;

So with test_progs -j on a KASAN VM, several workers can be inside
plain_c_arena_* subtests at once, on top of the parallel verifier_arena*
tests.  Can this bring back the OOM kills that a14f9ba080c1 fixed?

[ ... ]

> +void test_plain_c_test_wakeup_source(void) { RUN(test_wakeup_source); }

[ ... ]

> +void test_plain_c_verifier_arena_large(void) { RUN(verifier_arena_large); }

[ ... ]

> diff --git a/tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c b/tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c
> new file mode 100644
> index 000000000000..751a7157d7d3
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/plain_c_test_wakeup_source.c

[ ... ]

> +struct bpf_ws_lock;
> +struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void) __ksym;
> +void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock) __ksym;
> +void *bpf_wakeup_sources_get_head(void) __ksym;
> +
> +SEC("syscall")
> +__success __retval(0)
> +int iterate_wakeupsources(void *ctx)

A subsystem pattern flags this as potentially concerning: a selftest that
fails, rather than skips, when a feature or config option is missing.

bpf_wakeup_sources_read_lock(), bpf_wakeup_sources_read_unlock() and
bpf_wakeup_sources_get_head() live in drivers/base/power/wakeup.c, which
is built only with CONFIG_PM_SLEEP:

    obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o wakeup_stats.o

PM_SLEEP needs SUSPEND || HIBERNATE_CALLBACKS, and s390 has neither
ARCH_SUSPEND_POSSIBLE nor ARCH_HIBERNATION_POSSIBLE.  The kfunc externs
are not __weak, so on such kernels libbpf fails the object load with
"extern (func ksym) '...': not found in kernel or module BTFs".

The driver is just:

    void test_plain_c_test_wakeup_source(void) { RUN(test_wakeup_source); }

so run_subtest() in test_loader.c takes the
!ASSERT_OK(err, "unexpected_load_failure") path for this __success prog,
and test_progs exits with EXIT_FAILURE.

The original driver, test_wakeup_source() in prog_tests/wakeup_source.c,
handles this case:

    id = btf__find_by_name_kind(btf, "bpf_wakeup_sources_get_head", BTF_KIND_FUNC);
    ...
    if (id < 0) {
        printf("%s:SKIP:bpf_wakeup_sources_get_head kfunc not found in BTF\n", __func__);
        test__skip();
        return;
    }

DENYLIST.s390x has no plain_c_test_wakeup_source entry either.

This is test-only code, and the prog itself matches the original.  Could
the plain C driver keep the same BTF probe and test__skip(), or could the
test be denylisted where the kfuncs cannot exist?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42335829855

Re: [PATCH bpf-next v4 46/46] selftests/bpf: Test progs without bpf_for(), zero, can_loop and such

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] selftests/bpf: missing BTF kfunc skip check in
  test_plain_c_test_wakeup_source()
- [Low] selftests/bpf: missing __skip_without_arena in plain_c
  arena_htab and arena_list
--

commit 803ffb91b4f8392f64289683ec7ad50605335c6c
Author: Alexei Starovoitov <ast@kernel.org>

selftests/bpf: Test progs without bpf_for(), zero, can_loop and such

This commit adds plain-C versions of existing BPF selftest programs
without verifier-oriented loop macros and workarounds, along with test
runners in prog_tests/plain_c.c.

> diff --git a/tools/testing/selftests/bpf/prog_tests/plain_c.c b/tools/testing/selftests/bpf/prog_tests/plain_c.c
> new file mode 100644
> index 0000000000000..b7305118a4541
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/plain_c.c
> @@ -0,0 +1,160 @@

[ ... ]

> +/* prog_tests/arena_list.c: add 1000 elems, delete them, compare the sums */
> +static void arena_list_add_del(void)
> +{
> +	LIBBPF_OPTS(bpf_test_run_opts, opts);
> +	struct plain_c_arena_list *skel;
> +	int cnt = 1000, sum = cnt * (cnt - 1) / 2;
> +
> +	skel = plain_c_arena_list__open_and_load();
> +	if (!ASSERT_OK_PTR(skel, "open_and_load"))
> +		return;
> +	skel->bss->cnt = cnt;
> +	ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_add), &opts),
> +		  "add");
> +	if (skel->bss->skip) {
> +		test__skip();
> +		goto out;
> +	}

[ ... ]

> +void test_plain_c_arena_htab(void) { RUN(arena_htab); }
> +void test_plain_c_arena_htab_asm(void) { RUN(arena_htab_asm); }
> +
> +void test_plain_c_arena_list(void)
> +{
> +	RUN(arena_list);
> +	if (test__start_subtest("run"))
> +		arena_list_add_del();
> +}

[ ... ]

> +void test_plain_c_test_wakeup_source(void) { RUN(test_wakeup_source); }

[Severity: Medium]
Does test_plain_c_test_wakeup_source() fail when CONFIG_PM_SLEEP is not
enabled in the kernel (for example on s390x, where ARCH_SUSPEND_POSSIBLE
and ARCH_HIBERNATION_POSSIBLE are not supported)?

plain_c_test_wakeup_source.c declares bpf_wakeup_sources_read_lock(),
bpf_wakeup_sources_read_unlock(), and bpf_wakeup_sources_get_head() as
strong __ksym externs.  Unlike test_wakeup_source() in
prog_tests/wakeup_source.c, which checks vmlinux BTF for
bpf_wakeup_sources_get_head and calls test__skip() when absent,
test_plain_c_test_wakeup_source() unconditionally invokes
RUN(test_wakeup_source), causing bpf_object__load() to fail with
unresolved kfunc externs.

> diff --git a/tools/testing/selftests/bpf/progs/plain_c_arena_htab.c b/tools/testing/selftests/bpf/progs/plain_c_arena_htab.c
> new file mode 100644
> index 0000000000000..10a50b5058ddb
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/plain_c_arena_htab.c
> @@ -0,0 +1,222 @@

[ ... ]

> +SEC("syscall")
> +__success __retval(0)
> +#if !defined(__clang__) && defined(BPF_ARENA_FORCE_ASM)
> +__skip("GCC calls htab_init(): a global function gets an arena pointer as a scalar")
> +#endif
> +int arena_htab_llvm(void *ctx)
> +{
> +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)

[ ... ]

> +#else
> +	skip = true;
> +#endif
> +	return 0;
> +}

[Severity: Low]
Should arena_htab_llvm() (when BPF_ARENA_FORCE_ASM is not defined) and
arena_list_add() / arena_list_del() in plain_c_arena_list.c be annotated
with __skip_without_arena, similar to plain_c_arena_strsearch.c and
plain_c_verifier_arena_large.c?

Without __BPF_FEATURE_ADDR_SPACE_CAST (such as in test_progs-bpf_gcc or
test_progs-no_alu32), these programs compile to `skip = true; return 0;`,
and RUN(arena_htab) and RUN(arena_list) load the no-op stubs via
test_loader without checking skel->bss->skip, reporting OK instead of
SKIP.  Also, arena_list_add_del() in prog_tests/plain_c.c calls
test__skip() without printing a reason message.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261107033758.1547725-1-alexei.starovoitov@gmail.com?part=46

Re: [PATCH bpf-next v4 00/46] bpf: A verifier that joins states

[patchwork-bot+netdevbpf]

Hello:

This series was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:

On Sat,  7 Nov 2026 03:37:08 +0000 you wrote:
> From: Alexei Starovoitov <ast@kernel.org>
> 
> The verifier follows each path of a prog, and each branch can double the
> number of paths. The work grows with the paths, not with the prog:
>   insns processed                                    today    this series
>   rust-bpf  scx_beerland/beerland_init_task        1000001 *        16430
>   rust-bpf  scx_qmap/qmap_init_cids                 131088 *          196
>   scx       scx_eevdf/eevdf_dispatch                350041          15322
>   Cilium    bpf_host/cil_to_netdev, Oct 2026         64658           7538
>   * "BPF program is too large", "The sequence of 8193 jumps is too complex"
> 
> [...]

Here is the summary with links:
  - [bpf-next,v4,01/46] bpf: Add may_goto to the loops that the verifier marks
    https://git.kernel.org/bpf/bpf-next/c/8ed7285ee075
  - [bpf-next,v4,02/46] bpf: Limit depth and time of recursive subprogs at run time
    https://git.kernel.org/bpf/bpf-next/c/7b7dc9c6e8e9
  - [bpf-next,v4,03/46] bpf: Add union and widening of the range of a register
    https://git.kernel.org/bpf/bpf-next/c/ff3712ef1106
  - [bpf-next,v4,04/46] bpf: Add join of registers and stack slots
    https://git.kernel.org/bpf/bpf-next/c/47388a4c175e
  - [bpf-next,v4,05/46] bpf: Add join of verifier states
    https://git.kernel.org/bpf/bpf-next/c/ff30ca620432
  - [bpf-next,v4,06/46] bpf: Tell which arguments of a call a join must leave alone
    https://git.kernel.org/bpf/bpf-next/c/af6bda3802af
  - [bpf-next,v4,07/46] bpf: Add the data structures of the main pass over the control flow graph
    https://git.kernel.org/bpf/bpf-next/c/0c30e60b214c
  - [bpf-next,v4,08/46] bpf: Order the insns, find what stops a loop and where may_goto can go
    https://git.kernel.org/bpf/bpf-next/c/9075b1169d6b
  - [bpf-next,v4,09/46] bpf: Find the registers that hold a number or a pointer to arena
    https://git.kernel.org/bpf/bpf-next/c/0c5ce2587057
  - [bpf-next,v4,10/46] bpf: Find the registers and stack slots that later checks depend on
    https://git.kernel.org/bpf/bpf-next/c/a8bfc00bacb4
  - [bpf-next,v4,11/46] bpf: Add the sets of states at an insn and the entries of functions
    https://git.kernel.org/bpf/bpf-next/c/8468842a8950
  - [bpf-next,v4,12/46] bpf: Compare states of different iterations of a loop
    https://git.kernel.org/bpf/bpf-next/c/6a3204708703
  - [bpf-next,v4,13/46] bpf: Decide which states are not joined
    https://git.kernel.org/bpf/bpf-next/c/773b03804ab1
  - [bpf-next,v4,14/46] bpf: Merge the state that arrives into the set at an insn
    https://git.kernel.org/bpf/bpf-next/c/9b56d2e47f3a
  - [bpf-next,v4,15/46] bpf: Move states along edges, into loops and subprogs and out of them
    https://git.kernel.org/bpf/bpf-next/c/19d27a501a52
  - [bpf-next,v4,16/46] bpf: Make the state after a call from the caller's state and an exit state
    https://git.kernel.org/bpf/bpf-next/c/8875170bb1d2
  - [bpf-next,v4,17/46] bpf: Enter and leave loops in the main pass, build flow.c
    https://git.kernel.org/bpf/bpf-next/c/a784eb58f12b
  - [bpf-next,v4,18/46] bpf: Tell the passes over the graph where an added may_goto goes
    https://git.kernel.org/bpf/bpf-next/c/51bfd6314dcc
  - [bpf-next,v4,19/46] bpf: Verify programs as a fixpoint over the control flow graph
    https://git.kernel.org/bpf/bpf-next/c/5395aa9bbeb5
  - [bpf-next,v4,20/46] bpf: Keep the count of a loop when a register has no closed form
    https://git.kernel.org/bpf/bpf-next/c/9b202a8c7722
  - [bpf-next,v4,21/46] bpf: Verify static subprogs on their own
    https://git.kernel.org/bpf/bpf-next/c/8fd58560b71e
  - [bpf-next,v4,22/46] bpf: Allow recursive subprogs
    https://git.kernel.org/bpf/bpf-next/c/eaea73907052
  - [bpf-next,v4,23/46] bpf: Remove is_state_visited(), bpf_compute_scc() and iter.depth
    https://git.kernel.org/bpf/bpf-next/c/6771f70a4733
  - [bpf-next,v4,24/46] bpf: Remove precision tracking
    https://git.kernel.org/bpf/bpf-next/c/9f644d937dcf
  - [bpf-next,v4,25/46] bpf: Remove prune points, levels of state comparison and pop_stack()
    https://git.kernel.org/bpf/bpf-next/c/8ecf70baad8b
  - [bpf-next,v4,26/46] bpf: Update comments about paths, pruning and explored states
    https://git.kernel.org/bpf/bpf-next/c/b7bf79dd6f1e
  - [bpf-next,v4,27/46] bpf: Track a pointer to one of a set of functions
    https://git.kernel.org/bpf/bpf-next/c/edcafaec6256
  - [bpf-next,v4,28/46] bpf: callx reads what the subprogs whose address is taken read
    https://git.kernel.org/bpf/bpf-next/c/6d4131627eae
  - [bpf-next,v4,29/46] bpf: Keep the first iteration of a counted loop apart
    https://git.kernel.org/bpf/bpf-next/c/df9b13a441c9
  - [bpf-next,v4,30/46] bpf: What a call returns in each iteration is like what the loop loads
    https://git.kernel.org/bpf/bpf-next/c/89557ca1fc86
  - [bpf-next,v4,31/46] bpf: Find where the jumps of a loop stop a number
    https://git.kernel.org/bpf/bpf-next/c/e2b83f4db687
  - [bpf-next,v4,32/46] bpf: A loop is one iteration shorter behind the jump that counts it
    https://git.kernel.org/bpf/bpf-next/c/a00c8eafb381
  - [bpf-next,v4,33/46] bpf: Give the places of a back edge to the jumps that count
    https://git.kernel.org/bpf/bpf-next/c/95ff2abedf64
  - [bpf-next,v4,34/46] bpf: Count a loop whose exit tests a copy of the counter
    https://git.kernel.org/bpf/bpf-next/c/ab5fa7281144
  - [bpf-next,v4,35/46] bpf: Allow tail calls and LD_ABS in subprogs that return void
    https://git.kernel.org/bpf/bpf-next/c/1dcc217fefba
  - [bpf-next,v4,36/46] bpf: A part of a pointer that is spilled is a number
    https://git.kernel.org/bpf/bpf-next/c/50ea1f0797de
  - [bpf-next,v4,37/46] bpf: 32-bit add to a pointer and pointer op pointer make a number
    https://git.kernel.org/bpf/bpf-next/c/4a3a9c2d6896
  - [bpf-next,v4,38/46] selftests/bpf: Adjust tests to log lines that are gone
    https://git.kernel.org/bpf/bpf-next/c/7e363442447f
  - [bpf-next,v4,39/46] selftests/bpf: Adjust tests to new verdicts, messages and order
    https://git.kernel.org/bpf/bpf-next/c/dd43eeaf99eb
  - [bpf-next,v4,40/46] selftests/bpf: Test states that are not joined
    https://git.kernel.org/bpf/bpf-next/c/8292b648ae99
  - [bpf-next,v4,41/46] selftests/bpf: Test static subprogs that are verified on their own
    https://git.kernel.org/bpf/bpf-next/c/489b946f5f6a
  - [bpf-next,v4,42/46] selftests/bpf: Test loops that are counted, walked or widened
    https://git.kernel.org/bpf/bpf-next/c/d0a8c63bfc66
  - [bpf-next,v4,43/46] selftests/bpf: Test progs that run out of budget
    https://git.kernel.org/bpf/bpf-next/c/bab5cb94006e
  - [bpf-next,v4,44/46] selftests/bpf: Test callx of one of a set of functions
    https://git.kernel.org/bpf/bpf-next/c/d14f973f1559
  - [bpf-next,v4,45/46] selftests/bpf: Test pointers that become numbers and subprogs that return void
    https://git.kernel.org/bpf/bpf-next/c/d13baed274e6
  - [bpf-next,v4,46/46] selftests/bpf: Test progs without bpf_for(), zero, can_loop and such
    https://git.kernel.org/bpf/bpf-next/c/b5ae07f77d27

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html